Source: docs/research/2026-09-03-defensibility/appendices/G-public-repo-rework-map.md

Does the rework map print the same story on strangers' repos?

Measured 2026-09-03 on 11 public GitHub repositories. Read-only: every repo was cloned into /tmp/panout-ideas/public-repos/, nothing was pushed, no issue or PR was opened, no maintainer was contacted.

Question: on repos the founder has never touched, is the share of commits reworked within 30 days higher for agent-authored commits than for human-authored ones, and is the gap positive, negative, or noise?

Scripts: classify.py (discovery/attribution), measure_pub.py (rework measurement), report.py (tables). Raw per-commit output in raw/*.json.

1. Method

Ported from /tmp/panout-ideas/backtest/measure.py; the R0 definition is byte-for-byte the same test, so public numbers are directly comparable to the founder-repo numbers in /tmp/panout-ideas/backtest/REPORT.md.

Clones. git clone --shallow-since=2026-02-01 --single-branch (full blobs, ~214 days of history — deeper than the 180-day window so that every measured commit has its parent present). Discovery used cheap treeless clones (--filter=tree:0 --no-checkout).

Agent classification (trailer/marker). A commit is agent-authored if subject or body matches Co-authored-by: (Claude|Codex|Cursor|Copilot|Devin), a Co-authored-by: line whose address contains claude|copilot|codex|cursoragent|devin-ai-integration, Generated with [Claude Code], or 🤖 Generated with; or if the git author is the Copilot bot (198982749+Copilot@users.noreply.github.com) or literally named Copilot/Claude/Codex. Everything else is human. This is the same trailer-only rule as the founder backtest.

Population. Default branch (HEAD), commits authored in the last 180 days (since 2026-03-07), merge commits excluded from measurement.

Sample. Commits between 180 and 30 days old, so every commit has a complete 30-day forward window. Stratified: up to 100 most recent agent commits and 100 most recent human commits per repo (the founder backtest used a single un-stratified cap of 400; stratifying guarantees both classes are measurable in repos where one class is rare). Reduced from 300 to 100-per-class to hold total runtime under the budget — this is the one deliberate deviation from the brief, and Section 6 shows what it costs.

Rework test. For commit C: added line ranges from git diff C^ C --unified=0, skipping lockfiles (package-lock.json, yarn.lock, pnpm-lock.yaml, Cargo.lock, poetry.lock, uv.lock, go.sum, composer.lock, Gemfile.lock), minified bundles, binaries, images, fonts, snapshots, and .sha256; top 20 files by added lines. E = newest descendant of C on the default branch within 30 days. Then:

• R0 — git blame --reverse --porcelain C..E -L <ranges> -- <file> (identical to measure.py).
• R1 — same plus -w -M -C (whitespace-, rename- and copy-insensitive).

Reverse blame attributes each line to the last commit in which it survived; any attribution other than E means the line was changed or deleted inside the window. C counts as reworked if any added line fails to survive to E.

Time to first rework. Hours from C to the earliest last-surviving commit. As in the founder backtest this is a lower bound (the edit lands at that commit's child); 0 h means "the next commit touched it".

Path class. Each commit is assigned the class of the bucket holding most of its added lines, using path rules evaluated in order: ci (.github/workflows, .circleci, .gitlab-ci, azure-pipelines, Jenkinsfile, …) → tests (tests/, __tests__/, spec/, e2e/, *_test.go, *.test.ts, test_*.py, *Tests.cs) → docs (.md/.mdx/.rst/.adoc, docs/, website/) → src (known code extensions) → config (.json/.yaml/.toml/.csproj/Dockerfile/…) → other.

2. Selection

Discovery started from gh searches (gh search prs 'Co-authored-by: Claude' --merged, gh search prs --author app/copilot-swe-agent --merged, gh search prs 'head:codex/' --merged, and gh api search/commits with the cloak-preview header). Those searches are not usable for this selection: sorted by recency they return an almost pure long tail of one-person hobby repos (the top hit across 100 merged Claude-trailer PRs had 15 PRs; nothing in the first 40 rows had the commit volume required), and the GitHub search API secondary rate limit repeatedly blocked pagination. So discovery was flipped: 48 well-known, high-traffic public repos were treeless-cloned and their agent share was computed from the commit history itself, which is both cheaper and the same measurement used downstream.

Candidates with ≥300 commits in the last 180 days (48 probed, 44 qualified). Agent share is by the trailer rule above:

Selected 11 (bold above; the brief asked for 10, temporalio/temporal was added for a second Go/server-side data point). Selection rules applied: ≥300 commits/180d, both classes present, prefer 20–80% agent share, real software, language spread.

Only 6 of 11 land inside the 20–80% target band. That band is scarce in public code: of 44 qualifying repos, 8 exceeded 20% and 26 were below 8%. The remaining slots were filled from the 13–17% tier, choosing for language spread (Go, Rust) rather than share.

Excluded despite qualifying: PostHog/posthog, BerriAI/litellm, grafana/grafana (all ≥6k commits in 180 days — clone and measurement cost outside the runtime budget), and repos below ~8% agent share where the agent cell would be too thin to read.

3. Squash merges and PR-level attribution

Nine of the eleven repos squash-merge; Azure/azure-sdk-tools has zero merge commits in 180 days and microsoft/aspire has 25 in 1,922. Every agent contribution therefore lands as one linear commit on the default branch, and the co-author trailer survives into that squash commit — so trailer classification works without PR lookups.

For the three Copilot-agent repos the brief asked for PR-level attribution, so it was done: gh search prs --repo <R> --author app/copilot-swe-agent --merged gave the set of PR numbers opened by the Copilot coding agent (805 / 314 / 71), matched to commits by the trailing (#NNNN) in the squash subject. Result on the measured samples:

The trailer rule has zero false negatives against PR-level authorship, but it is far broader than it: 84–94% of trailer-flagged commits come from PRs a human opened, that merely carry a Co-authored-by: Copilot trailer. The trailer measures "an agent was in the room", not "an agent shipped this". Section 6 shows the two definitions disagree on sign.

4. Results

4.1 Rework within 30 days, agent vs human

CI is a normal-approximation interval on the difference of two proportions; it accounts for sampling only, not for the confounds in Section 6.

4.2 Direction of the gap

R1 gaps, sorted: deno +42.8, vscode-copilot-chat +25.2, gradio +9.2, mastra +8.1, aspire +5.2, typescript-go −1.2, n8n −2.0, azure-sdk-tools −2.8, eliza −5.1, spec-kit −18.2, temporal −19.9.

Pooled gap is +2.5 points with a CI spanning zero. -w -M -C (R1) barely moves anything: the largest R0→R1 change is 2.1 points (deno). Whitespace and rename insensitivity is not what drives these numbers.

For contrast, the founder-repo backtest reported pooled agent 64.6% vs human 51.4% (+13.2), and +21.7 on the founder-only slice. The public pooled gap is one fifth of that and not distinguishable from zero.

4.3 Median hours to first rework (R1, lower bound)

Agent rework arrives 9 hours later than human rework at the median (76.5 h vs 67.7 h), and the per-repo sign splits 5 slower / 6 faster. No usable signal. The founder backtest saw a much larger separation (17.1 h agent vs 1.2 h human), which is a property of a single-committer repo where the author's own next commit lands minutes later.

5. What a maintainer would see: top task classes

Top two path classes by agent rework rate per repo (R1, classes with n≥5 agent commits):

Pooled, by class:

Read as a maintainer would: in 9 of 11 repos the two named classes are simply tests and src, which are also the two largest classes by volume. In 4 repos the top class is a class with fewer than 25 agent commits (other n=6/7, docs n=6/15, config n=20). In mastra and gradio the top classes are pinned at 96–100% for both agent and human commits — the ranking there carries no information because nothing survives 30 days untouched in those monorepos.

The one pooled cut that looks non-obvious is src (+19.1) and config (+19.5) positive while tests (−7.0) and docs (−14.3) are negative. That is the most defensible finding in this report, but it is pooled across repos whose individual gaps run from −20 to +43, so it is an artifact of which repos contribute volume to which class as much as it is a property of agents.

6. Sensitivity: the gap does not survive two obvious controls

6.1 Calendar-period matching

Sampling the 100 most recent commits of each class independently means the two samples cover different calendar periods whenever one class is rarer. The gap is largest exactly where the offset is largest:

Recomputing the gap within calendar-month strata (strata needing ≥10 agent and ≥10 human commits, weighted by stratum size):

Time-matching cuts deno's gap by more than half and flips the pooled gap from +2.5 to −1.3.

6.2 Trailer attribution vs PR-level attribution

Using the strict definition (agent = PR opened by app/copilot-swe-agent) instead of the trailer, on the same measured commits:

The sign flips in all three repos. Commits from PRs the coding agent itself opened are reworked less than everything else; commits merely carrying an agent co-author trailer are reworked more. Cell counts are small (13/25/5), so this is not a claim that agent PRs are better — it is a demonstration that the two attribution rules produce opposite answers on identical commits.

6.3 Within-author

Authors with ≥10 agent-trailer and ≥10 human commits in the same sample (holds the person fixed):

Only five author-repo cells qualify, and they disagree in sign. denoland/deno's entire +42.8 repo-level gap is one contributor: 47 of the 100 sampled agent commits and 29 of the 100 human commits are the same person, whose agent-trailer commits are reworked 63.8% vs 10.3% for their non-trailer commits. Whether that is "the agent produced worse code" or "this person uses the agent on the churny parts (perf work on ext/node stream paths) and hand-writes the stable parts" cannot be separated from git.

7. Sampling detail

"Measured" is below "sampled" where a commit had no descendant within 30 days or touched only skipped files (lockfiles, binaries, minified assets). Azure/azure-sdk-tools loses 78 of 200, mostly dependabot lockfile-only commits.

8. Caveats

1. Attribution is by commit trailer. On the three Copilot repos where it could be checked, the trailer rule flags 5–7× more commits than PR-level agent authorship, and the two rules give opposite signs (6.2). Nothing in git separates "the agent wrote this" from "the agent was open in another pane".
2. Squash merges collapse the unit of work. Nine of eleven repos squash. A squash commit is a whole reviewed PR, so "one commit" means very different amounts of work in deno (one PR ≈ one fix) than in eliza (thousands of small direct commits). Rework rates are not comparable across repos, only within.
3. Reverse blame is line identity, not semantics. A reformat, an unmatched rename, or a neighbouring edit that shifts a line counts as rework; a semantically wrong line nobody touched for 30 days counts as survived. R1 (-w -M -C) was meant to test how much of the signal that explains: it moves nothing (≤2.1 pts anywhere), which means the proxy's noise is not mostly whitespace and renames — it is genuine co-location of edits.
4. Ceiling saturation. In mastra (98%/90%) and gradio (96%/87%) almost every commit is "reworked" within 30 days. In high-churn monorepos the metric has almost no dynamic range, and the per-class ranking in Section 5 is unreadable there.
5. Sample is 100 per class, not 300. Reduced to hold runtime under the budget, as the brief permitted. Per-repo CIs are consequently ±13 points wide on average; only 4 of 11 repos have a CI excluding zero, and month-stratification (6.1) drops several repos to a single usable stratum.
6. The 20–80% agent-share band is rare in public code. Only 8 of 44 qualifying repos reached 20%. Five of the eleven selected sit at 15–17%, so their agent samples reach further back in time than their human samples — the exact confound measured in 6.1.
7. Shallow clones. --shallow-since=2026-02-01 on the default branch only. Commits reachable only from other branches, and the pre-February history, are absent. This affects nothing in the 30–180-day window but means -C (copy detection) cannot see pre-February sources.
8. Bot traffic is in the human class. dependabot, renovate and repo-maintenance bots are not agent-flagged and land in "human". Azure/azure-sdk-tools (240 dependabot commits in 180 days) and github/spec-kit (147 github-actions commits) are the most affected; both show negative or near-zero gaps.
9. Single 30-day window, single snapshot. Measured once, on 2026-09-03. No repeat run, so run-to-run stability of these numbers is unknown.
10. Path classes are heuristic. spec-kit's templates and eliza's generated files land in config/other in ways a maintainer might not agree with.

9. What the numbers say

• Pooled across 11 stranger repos and 2,042 measured commits, agent-trailer commits are reworked within 30 days 54.7% of the time vs 52.1% for human commits: a +2.5-point gap whose 95% interval (−1.8 … +6.9) contains zero.
• Per repo the gap is positive in 5, negative in 3, and inside ±5 points in 3 (R1). The spread runs from −19.9 to +42.8 points.
• Matching agent and human commits by calendar month moves the pooled gap to −1.3.
• Switching from trailer attribution to PR-level attribution flips the sign in all three repos where both were computable.
• The same measurement on the founder's own repos produced +13.2 pooled (+21.7 founder-only).