Source: docs/research/2026-09-03-defensibility/appendices/H-split-half-stability.md

Is the agent-vs-human rework gap a property of the repo, or a draw?

Measured 2026-09-03 on the 11 public repositories already cloned in /tmp/panout-ideas/public-repos/. Read-only: no repository was modified, nothing was re-cloned, nothing was pushed, no issue or PR was opened, no maintainer was contacted.

Two questions:

• A. Split each repo's eligible commit window into two non-overlapping calendar halves. Does a repo's agent-vs-human R1 rework gap in one half predict its gap in the other?
• B. Re-run the corrective definitions (R1, R4, R4-strict) on bot-free, time-stratified samples. How many of the 11 repos print a gap whose bootstrap CI excludes zero, and how many print per-class agent rework rates a maintainer could actually read?

Scripts in this directory:

• measure_stability.py — per-repo measurement (sampling, split, R1/R2/R3/R4/R4-strict, path class). Derived from /tmp/panout-ideas/public-map/measure_pub.py (classification, skip list, endpoint, blame, path classes) and /tmp/panout-ideas/restricted/measure_restricted.py (R2/R3/R4 clauses).
• analyze.py — tables A1–A5, B1–B4 (_tables.md).
• extra.py — tables B0, B0b (_extra.md).
• build_report.py — composes this file.
• raw/*.json — per-commit output, one file per repo.

Runtime: 11 repos measured in parallel, 9 s (azure-sdk-tools) to 272 s (eliza), ~5 min wall; analysis ~30 s. The sample stayed at 100 per class per half — no reduction to 60 was needed.

1. Method

1.1 Inherited definitions

Unchanged from the two prior runs:

• Agent classification — trailer/marker rule from measure_pub.py: Co-authored-by: naming Claude/Codex/Cursor/Copilot/Devin, Generated with [Claude Code], the robot-emoji marker, a Co-authored-by: address containing claude|copilot|codex|cursoragent|devin-ai-integration, the Copilot bot author address, or a git author literally named Copilot/Claude/Codex. Everything else is human.
• Rework endpoint — E = newest descendant of C on the default branch within 30 days (git rev-list --ancestry-path --until=<C+30d> C..HEAD, first entry).
• Added ranges — git diff C^ C --unified=0 -M, same skip list (lockfiles, minified bundles, binaries, images, fonts, snapshots, .sha256), top 20 files by added lines.
• R1 — git blame --reverse --porcelain -w -M -C C..E -L <ranges> -- <file>. Any added line whose last-surviving commit is not E counts as rework. (R0 was not recomputed: the public-map run showed R0→R1 moves ≤2.1 pts anywhere, and the brief asks for R1.)
• R2 / R3 / R4 / R4-strict — verbatim clauses from measure_restricted.py. The reworking commit is approximated as the first commit on the ancestry path after the blamed last-surviving commit that touches the same file.
  • R2 = R1 and some reworking commit has a different git author email than C.
  • R3 = R1 and some reworking commit either touches a test path or has a subject matching fix|bug|revert|hotfix|regress|broke|typo|wrong|incorrect|crash|error.
  • R4 = R1 and (R2 or R3).
  • R4-strict = R1 and (R2 or fix-word subject) — drops R3's test-path clause.
• Path class — same ordered rules (ci → tests → docs → src → config → other), commit assigned the class holding most of its added lines.

1.2 What changed for this run

1. Bots excluded from both classes. Any commit whose author name or email matches [bot], dependabot, renovate, or github-actions is dropped before sampling. The public-map run left these in the human class; they are 975 of mastra's 4,137 eligible commits and 234 of azure-sdk-tools' 1,076 (table A1).
2. Two halves. The eligible window (commits 180–30 days old, non-merge, non-bot) is cut at the median commit date of that repo's own eligible population, giving two contiguous, non-overlapping calendar intervals with equal commit counts. A pure calendar midpoint was tried first and rejected: microsoft/vscode-copilot-chat's clone has 506 of its 507 eligible commits before its range midpoint (its cloned branch tip is 2026-05-20), and elizaOS/eliza's clone only reaches back to 2026-06-19, so a fixed global midpoint left one half empty in both. Cut dates and observed ranges are in table A1.
3. Random sampling inside each half, seeded (random.Random(1234)), up to 100 agent-trailer and 100 human commits per half. The public-map run took the 100 most recent per class over the whole window, which is exactly the calendar-offset confound its own §6.1 flagged.
4. Part B uses the union of the two half-samples (up to 200 agent + 200 human per repo, bot-free, time-stratified) rather than a fresh un-stratified 100/100. This is a deliberate deviation: it is the same commits already measured for A, it doubles n per class, and it removes the calendar offset. Table B0 shows what that costs relative to the public-map numbers.

1.3 Statistics

• Gap = agent rate − human rate, in percentage points.
• Per-gap CIs: percentile bootstrap, 5,000 resamples (2,000 where noted), resampling commits with replacement within each class. Sampling error only.
• Across-repo correlation: Spearman rho (ties averaged), with a repo-clustered percentile bootstrap (5,000 resamples of the 11 repos with replacement).
• Noise floor (A5): each repo's measured commits are re-split at random into two groups of the same sizes as its real halves, and the across-repo Spearman is recomputed, 2,000 times. This is the correlation a perfectly stable repo property would produce at these sample sizes.

2. Part A — split-half stability

A1. Sampling and window split

A2. R1 gap per half

Across-repo (n=11 repos) half-1 gap vs half-2 gap: Spearman rho = +0.164, Kendall tau = +0.164, Pearson r = +0.231. Repo-clustered percentile bootstrap (5000 resamples of the 11 repos): 95% CI on Spearman rho = [-0.617, +0.898] (share of resamples with rho>0: 64.0%). Sign agreement: 7/11 repos. Mean |half-1 gap - half-2 gap| = 9.6 pts; median = 5.5 pts. Half-1 gaps: temporalio/temporal +20.7, microsoft/aspire +18.8, gradio-app/gradio +15.4, n8n-io/n8n +12.0, github/spec-kit +9.3, elizaOS/eliza +8.6, microsoft/vscode-copilot-chat +6.8, mastra-ai/mastra +2.2, denoland/deno +1.0, Azure/azure-sdk-tools -3.4, microsoft/typescript-go -12.1

Half-2 gaps: microsoft/aspire +16.0, gradio-app/gradio +12.7, mastra-ai/mastra +10.3, n8n-io/n8n +5.6, microsoft/vscode-copilot-chat +1.4, Azure/azure-sdk-tools +1.0, denoland/deno +0.4, elizaOS/eliza -5.4, microsoft/typescript-go -13.4, github/spec-kit -13.6, temporalio/temporal -16.5

A3. Is each half's gap itself distinguishable from zero?

A4. Per-path-class agent R1 rework rate, half 1 vs half 2 (classes with n>=15 in both halves)

Repos with >=2 qualifying classes: 5. Spearman values: microsoft/aspire -1.00 (k=2), n8n-io/n8n +1.00 (k=2), elizaOS/eliza -1.00 (k=2), microsoft/typescript-go -0.50 (k=3), temporalio/temporal +1.00 (k=2). Mean = -0.10.

A5. Noise floor: what would the split-half correlation be with NO time drift?

Control: for each repo, re-split its own measured commits into two groups of the same sizes as the real halves, at random (ignoring commit date), recompute both gaps, then recompute the across-repo Spearman. Under 'the gap is a stable repo property + binomial sampling noise' this control gives the correlation attainable at these sample sizes. Observed time-split rho is compared against that distribution.

Random-split control (2000 reps): median Spearman rho = +0.482, 5th-95th pct = +0.164 .. +0.764. Observed time-split rho = +0.164; it sits at the 5th percentile of the random-split control.

Per-repo: is the change in gap between halves larger than sampling noise?

Half-to-half change exceeds sampling noise in 2 of 11 repos.

A6. Plain answer to A

A repo's half-1 gap does not predict its half-2 gap. Spearman rho = +0.164 across the 11 repos, with a repo-clustered 95% CI of [−0.62, +0.90] that comfortably contains zero, negative values, and near-perfect correlation alike. Signs agree in 7 of 11 repos; under a coin-flip null the expected count is 5.5 and a count of 7 or more occurs 27% of the time, so sign agreement on its own is not evidence of stability. Six of the 22 half-gaps are inside ±5 points, where the sign is arbitrary anyway.

The noise-floor control (A5) is the sharper result. If the gap were a fixed repo property observed through binomial sampling noise, splitting each repo's own commits at random would produce a median rho of +0.48. The observed time-split rho of +0.164 sits at the 5th percentile of that control distribution. So the split-half correlation is not merely weak — it is weaker than sampling noise alone can explain, which is the signature of real drift over calendar time on top of the noise, not of a stable repo trait measured imprecisely.

Two repos change gap by more than sampling error can account for: temporalio/temporal (+20.7 → −16.5, CI on the change +17.9 … +58.4) and github/spec-kit (+9.3 → −13.6, CI +4.7 … +41.3). Both flip sign. In the other 9 the change is inside the noise band, which is consistent either with stability or with the sample being too small to see the drift — those CIs are 30–50 points wide.

The per-class ordering question (A4) is close to unanswerable at this sample size. Only 6 of 11 repos have two or more path classes with ≥15 agent commits in both halves, and in mastra-ai/mastra both qualifying classes are pinned at 100% in half 2, so the correlation is undefined. Of the 5 repos with a defined value, k = 2 classes in four of them and k = 3 in one: rho = −1.00, +1.00, −1.00, −0.50, +1.00 (mean −0.10). With k = 2 the statistic can only take the values ±1, so this is a coin flip reported to two decimal places. The class ordering inside a repo is not measurable at 100 agent commits per half.

3. Part B — corrective definitions on bot-free, time-stratified samples

B0. Comparison with the public-map run (same repos, different sample)

Two conclusions from B0. First, the two repos with the largest gaps in the public-map run lose almost all of it once agent and human commits cover the same calendar span: denoland/deno +42.8 → +4.0 and microsoft/vscode-copilot-chat +25.2 → +4.4. Second, the two repos with the largest negative gaps also shrink: temporalio/temporal −19.9 → −1.4 and github/spec-kit −18.2 → −2.2. The pooled figure barely moves (+2.5 → +3.1) because these cancel — the per-repo spread collapses from a 63-point range (−19.9 … +42.8) to a 30-point range (−13.0 … +17.4) while the centre stays put.

B0b. Clause-level rates (union sample): R2 = reworked by a different author, R3 = reworking commit is a fix or touches tests

B1. R1 / R4 / R4-strict rates and gaps (union of both half-samples, bots excluded)

B2. Verdict counts

Repo-clustered bootstrap on the pooled gap (resample the 11 repos, then commits within):

• R1: gap +3.1 pts, repo-clustered 95% CI -2.7 .. +8.3
• R4: gap +7.1 pts, repo-clustered 95% CI +0.3 .. +12.6
• R4-strict: gap +7.7 pts, repo-clustered 95% CI +1.3 .. +13.5

B2b. Plain answer to the count question

Of the 11 repos, under R1: 3 print a positive gap with a CI excluding zero (aspire +17.4, gradio +14.4, mastra +6.2), 1 prints a negative gap with a CI excluding zero (typescript-go −13.0), and 7 are indeterminate.

Under R4: 4 positive (aspire, spec-kit, mastra, gradio), 1 negative (typescript-go), 6 indeterminate. Under R4-strict: 4 positive (aspire, spec-kit, mastra, gradio), 0 negative, 7 indeterminate.

The pooled gap under R1 (+3.1) has a commit-level CI that just touches zero (−0.1 … +6.2) and a repo-clustered CI that clearly contains it (−2.7 … +8.3). Under R4 (+7.1) and R4-strict (+7.7) the repo-clustered CIs exclude zero (+0.3 … +12.6 and +1.3 … +13.5), but only barely, and both are driven by the R2 clause (B0b): the "reworked by a different author" rate is where agent and human separate most (pooled 47.2% vs 39.9%), and that clause is confounded — a commit is more likely to be reworked by someone else when its author is not the file's usual owner, which differs systematically between agent-trailer PRs and routine maintainer commits.

github/spec-kit is the clearest case of the definition doing the work: R1 gap −2.2 (agent reworked slightly less), R4 gap +21.8. Its human commits are reworked at 60.7% under R1 but only 33.2% under R4 — most human rework there is the same author touching their own lines again, which R4 discards, while agent rework is more often somebody else's fix.

B3. Are the two worst per-class agent rework rates readable?

Worst = highest agent R1 rework rate. Readable = both classes have n>=25 agent commits AND not both rates >95%.

Readable in 2 of 11 repos.

B3b. Plain answer to the readability question

In 9 of 11 repos the two worst per-class agent rework rates are not readable. The failure mode is almost always sample size, not the ceiling: in 9 repos at least one of the top-two classes has fewer than 25 agent commits, and in 4 repos the single worst class has n ≤ 5 (temporalio/temporal reports ci 100% off one commit; elizaOS/eliza reports other 100% off four). Two repos fail on the ceiling as well — mastra-ai/mastra and gradio-app/gradio both have their top two classes pinned at 100%, in a repo where the human baseline is 82–88%, so the ranking carries no information. Only microsoft/aspire (src 77% n=47, tests 75% n=99) and denoland/deno (src 50% n=78, tests 33% n=43) pass both tests, and in both cases the two named classes are simply the two largest classes by volume.

B4. Pooled path-class rates (union sample, R1 / R4-strict)

4. Caveats

1. Clone artefacts limit two repos. elizaOS/eliza's shallow clone only reaches 2026-06-19, so its "180-day window" is really 46 days and both halves are ~23 days. microsoft/vscode-copilot-chat's cloned branch tip is 2026-05-20, so its window is 2026-03-07 … 2026-05-20 and every measured commit is at least 105 days old. Neither repo's halves are comparable in calendar length to the other nine. Repos were not re-cloned, per the brief.
2. One half can be thin in one class. denoland/deno has 23 agent commits in half 2 (its agent activity is concentrated in March–May), gradio-app/gradio 19 in half 1, temporalio/temporal 62 in half 1. Those half-gaps carry ±20–25-point CIs, which mechanically pushes the split-half correlation toward zero — part of the low rho is thin cells, not drift. The A5 control accounts for this (it uses the same cell sizes) and still puts the observed correlation at the 5th percentile.
3. The median-date cut is not a calendar midpoint. Halves have equal commit counts, not equal durations. In steady repos the two are nearly the same (aspire 2026-05-11 vs a midpoint of 2026-05-21); in bursty ones they differ by weeks.
4. Bootstrap CIs cover sampling only. They say nothing about attribution error, squash-merge granularity, or repo-level confounding. The repo-clustered pooled CIs additionally cover between-repo variance, which is why they are wider than the commit-level ones.
5. Trailer attribution is unchanged and still broad. The public-map run showed that on the three Copilot repos, 84–94% of trailer-flagged commits come from PRs a human opened, and that PR-level attribution flips the sign. Nothing here re-tests that; every number in this report is "an agent was in the room", not "an agent shipped this".
6. R2 is a proxy for ownership, not for quality. "Reworked by a different author" depends on how many people touch a file and on whether the original author is still active. R4 and R4-strict inherit that. The restricted-repo report reached the same conclusion on the founder's repos, where R2 was degenerate at 0/351.
7. The reworking commit is approximated, as in measure_restricted.py: blame --reverse names the last commit in which a line survived, and the edit is attributed to the first later commit on the ancestry path touching that file. It is wrong whenever a rename and an edit land in different commits.
8. Ceiling saturation persists. mastra-ai/mastra (94.5%/88.3%) and gradio-app/gradio (96.1%/81.6%) still have almost no dynamic range under R1, and R4-strict barely helps (93.5%/81.6% and 92.1%/73.5%).
9. Random sampling within halves is seeded but single-draw. No repeat draw was taken, so the contribution of sample-selection variance to the per-repo numbers is not separately measured (the bootstrap approximates it).
10. microsoft/typescript-go's newest eligible commit is 2026-07-31, three days short of the 30-day cutoff, because of a gap in its history; immaterial to the result.
11. Path classes are heuristic, unchanged from public-map: spec-kit's templates and eliza's generated files land in config/other in ways a maintainer might not agree with.

5. Summary of findings

• A. Split-half Spearman rho = +0.164, repo-clustered 95% CI [−0.62, +0.90]; signs agree in 7 of 11 repos; median absolute change in gap between halves is 5.5 points, mean 9.6. Against a random-split control whose median rho is +0.48, the observed value sits at the 5th percentile. A repo's gap in one half does not predict its gap in the other.
• A (per class). Only 6 of 11 repos have ≥2 path classes with n≥15 agent commits in both halves, and 4 of the 5 usable correlations are computed on exactly 2 classes, where the statistic is ±1 by construction. Not measurable at this sample size.
• B. With bots excluded and both classes drawn from the same calendar span: R1 — 3 repos positive with CI excluding zero, 1 negative, 7 indeterminate. R4 — 4 positive, 1 negative, 6 indeterminate. R4-strict — 4 positive, 0 negative, 7 indeterminate.
• B (pooled). R1 +3.1 pts (repo-clustered CI −2.7 … +8.3), R4 +7.1 (+0.3 … +12.6), R4-strict +7.7 (+1.3 … +13.5).
• B (readability). The two worst per-class agent rework rates are readable in 2 of 11 repos (microsoft/aspire, denoland/deno). Seven fail on cell size, two additionally on the 95% ceiling.
• Time-stratifying and dropping bots cuts the per-repo spread of the R1 gap from 63 points to 30 points while leaving the pooled value unchanged (+2.5 → +3.1).