Source: docs/research/2026-09-03-defensibility/appendices/D-session-commit-linkage.md

Does local session evidence predict commit rework, beyond git history?

Measured 2026-09-03. Read-only: no repo, log, or session file was modified. Only metadata was extracted from session logs (timestamps, cwd, tool names, event types, is_error flags, and short SHAs printed by git commit). No prompt text, assistant text, or file contents were read into the analysis or this report.

Headline: the question could not be answered. Not "no effect" — no measurement. Two independent structural blockers reduce the joint git x session sample to n=10 with a degenerate outcome (10/10 reworked, zero negative examples), so AUC is undefined for both the git-only and the git+session model and no delta exists to report.

1. Method

Session scan

• Roots: ~/.claude/projects/ plus ~/.claude-identities/{api-nozdr,api,personal}/projects/.
• 262 JSONL files (75 top-level sessions + 187 subagent transcripts nested under <session-id>/subagents/). An earlier pass that globbed only projects/*/*.jsonl missed all 187 subagent files; the numbers below use the corrected glob.
• Per line: timestamp, cwd, and message.content[] blocks. tool_use blocks give tool name and (for Bash) whether input.command contains git commit — the command itself was never emitted. tool_result blocks give is_error and are matched back to their tool_use by tool_use_id.
• Friction markers: "The user doesn't want to proceed", "User rejected" (rejection); "Request interrupted by user" (interrupt); plus the top-level toolDenialKind field.
• Commit-SHA extraction: from tool_result text of a git commit call, matching git's commit confirmation line ^\[<branch> <sha>\] <subject> (7-40 hex). Only the SHA was retained.

Linkage

Short SHA -> git rev-parse in the repo root of the session cwd -> full SHA -> test whether the commit is an ancestor of the default branch. For commits not on the default branch, a recovery pass searched default-branch commit messages (since 2026-07-20) for the session commit's subject line, which catches GitHub squash-merges (the squash body lists the branch commits).

Rework

Identical to the prior git-only backtest (/tmp/panout-ideas/backtest/measure.py): take added line ranges from git diff C^ C --unified=0 (skipping lockfiles/binaries/minified, top 20 files by added lines), find the newest default-branch descendant E inside the window, then git blame --reverse --porcelain C..E -L <ranges>. Any attribution other than E means a line was modified or deleted inside the window.

2. Blocker 1: zero time overlap with the git-only backtest

The backtest window is by construction commits 30-180 days old, so every commit in raw.json has a full 30-day forward window. Local session logs begin nine days after the newest raw.json commit. The retention horizon of the Claude Code session store here is ~21 days.

Linkage yield against raw.json: 0 of 2,859 commits (0.0%). Not one repo, not one commit, in either the agent or human class. This is a deterministic consequence of the two date ranges, not a matching failure: 49 distinct commit SHAs were extracted from session logs and none shares a prefix with any of the 2,859 sampled SHAs.

Session-log retention is also short enough that no commit observed in a session is yet 30 days old. The earliest session commit (2026-08-14) reaches a full 30-day window on 2026-09-13; the median one (2026-08-25) on 2026-09-24.

3. Blocker 2: the SHA an agent sees is usually not the SHA that lands

49 SHAs were extracted; 48 still resolve in a local repo (1 was garbage-collected or lived in a deleted worktree).

In the PR-based repo (poe2, 2,811 default-branch commits since 2026-08-01) 0 of 40 session commit SHAs are ancestors of origin/main. GitHub squash-merge rewrites the commit, so the SHA the agent printed at commit time never enters the branch history the backtest measures. Subject-line recovery from squash bodies rescued 8 of those 40 (20%); the other 32 have no trace on origin/main under any subject and are not contained by any remote branch — abandoned, rebased away, or still unmerged after 8-20 days.

This blocker is independent of Blocker 1. Even with perfectly overlapping date ranges, direct SHA linkage would recover only ~13% of session commits, and the recoverable ones are systematically the squash-merged (i.e. landed, reviewed) subset — a selection effect, not a random sample.

Session provenance, for the record: 11 session groups produced commits; 13 of 262 transcripts contain a commit. 36 of 48 commits carry an agent trailer; 47 of 48 are authored by gareth@garethpaul.com.

4. What the 48 linked commits do show (features, well-powered)

Feature values are computed over the parent session plus all its subagent transcripts, counting only events at or before the commit's timestamp.

The proposed predictor is nearly absent from the data. Only 3 of 48 session commits (6.3%) had any rejection or interrupt before the commit. Across all 262 transcripts there are only 53 rejection and 31 interrupt events total, against 15,660 tool calls. Sessions here run in bypassPermissions mode, which removes the approval prompt that generates most rejection events. A binary "friction happened" feature with a 6% base rate cannot carry meaningful signal at any achievable sample size in this corpus.

Tool errors are the opposite: present in 44 of 48 (91.7%), median 4 per commit. That feature has usable variance — it is the outcome, not the feature, that is missing.

5. Rates and AUC (n=10, shortened window)

Since no linked commit is 30 days old, the rework test was re-run at a 7-day window on the landed subset. 11 landed commits had >= 7 days of forward history; 1 (2a4fc35, resume) had no measurable added ranges after skip filters.

Friction split

No contrast is observable. With 2 exposed and 8 unexposed and a saturated outcome, this design has essentially zero power: even a true effect of any size would produce exactly this table.

Tool errors, reworked vs not reworked

AUC

AUC is a rank statistic over (positive, negative) pairs. The outcome vector is all-positive (10 reworked, 0 not), so the pair set is empty and AUC is undefined for any feature set, including git-only. No logistic regression was fit: with 10 observations, one outcome class, and 12 candidate features, the likelihood is unbounded and any reported coefficient would be an artifact. No bootstrap CI on an AUC delta can be produced, because the point estimate does not exist.

6. Is the 10/10 itself informative?

A control sample of default-branch commits from the same repos and the same date band (2026-08-10 to 2026-08-27), measured with the identical 7-day reverse-blame test:

Binomial: P(10/10 reworked | p = 0.641) = 0.0116.

The all-positive outcome is therefore not a window artifact — the 7-day base rate has plenty of negatives. But the difference is confounded and should not be read as an effect:

• 5 of 10 linked commits are squash-merge landings, whose blame surface is the entire PR branch, not the single session commit. A larger added-line surface mechanically raises the chance that something in it is later touched.
• Linked commits are not size-matched to controls (linked median 536 added lines; the control sample's added-line distribution was not recorded), and lines-added is a known correlate of rework.
• Selection: commits are linkable only if they landed, and in poe2 only if their subject survived into a squash body — 20% of that repo's session commits.

With n=10 and p=0.012 against a confounded comparison, this is a curiosity, not a result.

7. Secondary: time-to-rework vs session features (n=10)

All 10 reworked, so the only continuous outcome available is hours-to-first-rework (a lower bound, per the backtest's method note). Spearman rank correlation:

Signs are in the direction "more in-session error -> reworked sooner", but at n=10 the two-sided threshold for rho at alpha=0.05 is about |0.65|; none of these clears it, and six correlations were computed without correction. Treat as unreportable.

8. Sample-size caveat

n is far too small for any delta to be meaningful, and the specific delta requested does not exist. Concretely:

• Joint git x session sample against the original backtest: n = 0.
• Joint sample under a relaxed 7-day window with squash recovery: n = 10, one outcome class.
• Commits carrying the proposed predictor (>= 1 rejection or interrupt): 3 of 48 overall, 2 of 10 in the measurable set.
• To detect a 15 pp rework-rate difference between friction and no-friction commits at 80% power with a 6% exposure rate, the required linked sample is on the order of 10^3-10^4 commits. The entire local session corpus contains 48.

The binding constraints are (i) ~21-day session-log retention against a 30-day rework window, and (ii) squash-merge SHA rewriting, which severs 100% of the SHA-level linkage in the highest-volume repo.

Artifacts

All under /tmp/panout-ideas/linkage/: