Source: docs/research/2026-09-03-defensibility/appendices/F-restricted-rework-metric.md

Restricted rework definitions: does the agent-vs-human gap survive?

Measured 2026-09-03, read-only. No repository was modified.

• Script: /tmp/panout-ideas/restricted/measure_restricted.py (R0-R4)
• Variant script with the R3 clause split: /tmp/panout-ideas/restricted/measure_split.py
• Per-commit output: /tmp/panout-ideas/restricted/restricted_raw.json, /tmp/panout-ideas/restricted/restricted_raw_split.json
• Table generator: /tmp/panout-ideas/restricted/report.py -> /tmp/panout-ideas/restricted/_tables.md
• Input sample: the 2,859 commits already measured in /tmp/panout-ideas/backtest/raw.json

Method

Same sample, same repos, same 30-day forward window, same file-skip list (lockfiles, minified assets, binaries, .sha256), same top-20-files-by-added-lines cap, same 400-commits-per-repo cap. Every one of the 2,859 sampled commits was re-measured; no commit was dropped and no down-sampling to 200/repo was needed. Full run time was 191 seconds, so -M -C blame was not a problem.

For each commit C with added line ranges and window endpoint E (newest descendant within 30 days on the default branch):

• R0 - original metric. git blame --reverse --porcelain C..E -L <ranges> -- <file>. Any line whose last-surviving commit is not E counts as reworked. Recomputed from scratch here; it agrees with the stored rw field in raw.json on 2859/2859 (100%) commits, so R1-R4 are measured on an identical baseline.
• R1 - whitespace- and move-insensitive. Same blame with -w -M -C. Whitespace-only edits and lines moved or copied within the same commit no longer break survival.
• R2 - R1 and at least one reworking commit has a different git author email than C.
• R3 - R1 and at least one reworking commit either touches a path matching test|spec|__tests__ (or a *_test.* / *.spec.*-style filename) or has a subject matching fix|bug|revert|hotfix|regress|broke|typo|wrong|incorrect|crash|error (case-insensitive, word boundaries).
• R4 - R1 AND (R2 OR R3).
• R4-strict (added, reported alongside) - R1 AND (R2 OR fix-word subject). Drops the test-path clause of R3, which turned out to be doing most of the work (see Caveat 4).

Approximation for "the reworking commit"

git blame --reverse names the last commit in which a line survived (call it s), not the commit that changed it. The actual edit lands in s's child. This measurement approximates the reworking commit as:

git rev-list --ancestry-path --reverse s..E -- <file>   # first entry

i.e. the earliest commit on the ancestry path after s that touches the same file. If the file was renamed away and that list is empty, it falls back to the first commit after s on the ancestry path regardless of path. This is exact whenever the next commit touching the file is the one that edited the blamed lines, which is the normal case; it can name the wrong commit when a single commit both moves the file and a later commit edits the lines. Author and subject for R2/R3 are read off that approximated commit. All of C's reworking commits across all its files are unioned, so R2/R3 are "any reworking commit qualifies".

1. Pooled rates by metric

All repos, all authors (n = 2,859)

All repos minus the duplicate checkout (n = 2,463)

/code/work/poe-app-platform and /code/work/poe2 are the same git history - all 396 sampled commit SHAs are identical. Pooled numbers that include both double-count 396 commits (323 of them agent). This table drops poe-app-platform.

All repos, bot-authored commits excluded (n = 2,337)

Excludes authors matching [bot], github-actions, or openclaw.local (dependabot, poe-man[bot] release republishes, the work-brain research pipeline). These are all classed "human" by the trailer test but are machine-generated.

Founder-only (author gareth@garethpaul.com / gpj@local), n = 1,493

R2 is degenerate on the founder slice. Agent commits authored by the founder are, by construction, reworked by the founder too (these are single-author repos), so "different author" scores 0/351. The 3.2% human figure comes from a handful of commits reworked by a bot or a collaborator. R2 carries no signal here and R4 on the founder slice is effectively R3.

Dropping the duplicate poe-app-platform checkout changes founder-only numbers by <=0.1pt (only 2 founder commits are in that repo's sample), so the founder tables above stand as-is.

2. Per-repo rates

R1 by repo

R4 by repo

R4-strict by repo (R1 AND (different author OR fix-word subject))

Sign-reversal count by metric (repos with both classes non-empty)

Four of the six R1 reversals (cursor-plugins, research/linear, work-brain, and to a lesser degree car-racer) sit on agent samples of n = 1, 4, 1, and 10 respectively. The only reversals with a substantial agent sample under R1 are poe2 / poe-app-platform (the same repo, n = 323).

3. The three originally-reversed repos, metric by metric

Why poe2 reverses under R0/R1: bot release commits

poe2's 73 "human" commits are half machine-generated. Splitting them:

Every one of the 36 chore(release-cli): republish SDK + CLI (...) commits is 100% "reworked" under R0 and R1, because the next release-republish commit overwrites the same version strings. That is the entire source of the poe2 reversal. Excluding bot authors, the poe2 R1 gap collapses from -19.0 to -1.7 pts (agent 63.2% n=323, human 64.9% n=37) and the R4 gap is +5.5 pts (agent 62.2%, human 56.8%). R3/R4 excludes those bot commits automatically, because one version bump superseding another has no fix-word subject and touches no test file.

car-racer

car-racer's reversal is a ceiling effect on a tiny sample: all 20 human commits are reworked under R0, R1, R3 and R4, so the human rate is pinned at 100% and the agent rate (80%, 8/10) cannot exceed it. Only R4-strict, which drops the test-path clause, separates them (+10.0 pts). With n = 10 agent commits nothing here is statistically meaningful.

4. Bootstrap 95% CI on the founder-only gap (1,000 resamples, agent and human classes resampled independently with replacement, seed 20260903)

All CIs except R2's exclude zero. The founder-only gap is positive and its lower bound stays above +5 pts under every definition tested.

5. Does the gap survive, and do the reversals persist?

The gap survives every filter. Whitespace and move insensitivity (R1) removes very little: it costs the agent rate 2.4 pts and the human rate 1.3 pts pooled, and 0.2 / 1.5 pts on the founder slice. Reformatting and moved code were not what the original metric was counting. Requiring the rework to look corrective (R3/R4) cuts the human floor hard - pooled human rework falls from 51.4% to 28.6%, founder-only from 37.8% to 33.6%, and under R4-strict to 19.9% / 23.0% - while the agent rate falls much less. The gap therefore widens under R3/R4 pooled (+12.1 -> +31.0) and holds roughly flat on the founder slice (+23.0 -> +19.9), with a bootstrap 95% CI of [+14.1, +25.8]. Under the strictest variant it narrows to +11.2 [+5.8, +16.9] but stays positive.

The high floor is partly fixed. The complaint that human commits are "reworked 38-51%" is specific to R0/R1. Under R4 the human floor is 28.6% pooled / 33.6% founder; under R4-strict it is 19.9% / 23.0%. Agent rates fall proportionally less, which is what a discriminating metric should do.

The sign reversals mostly do not persist.

• poe2 and poe-app-platform (the same repository, counted twice): reversal does not persist. R1 keeps it (-19.0 / -18.5) but R4 flips it decisively to +32.1 / +32.9, and R4-strict to +20.3 / +22.8. The R0/R1 reversal is an artifact of 36 bot release-republish commits with a 100% line-churn rate; excluding bots, the R1 gap is -1.7 pts, i.e. gone.
• car-racer: reversal persists under R1, R3 and R4 (-20.0 pts at every one), and flips to +10.0 only under R4-strict. It is a 100%-human-rework ceiling effect on n = 10 agent / n = 20 human commits and should not be read as evidence either way.
• Two further R1 reversals (cursor-plugins n=1 agent, work-brain n=1 agent, research/linear n=4 agent) persist under R4 but rest on one to four agent commits each. research/linear flips positive under R4-strict.
• Under R4-strict a new reversal appears in dynorouter-gateway (-11.7, agent n=44, human n=24) that was not present under R0, R1, R3 or R4.

Net: of the six repos that reverse under R1, four have agent samples of n <= 10. The one large reversal (poe2) is explained by bot commits and disappears under both the corrective filter and simple bot exclusion. R4 leaves no sign reversal backed by a meaningful agent sample.

6. Caveats

1. poe-app-platform and poe2 are the same git history. All 396 sampled SHAs match. Every "all repos pooled" figure in the prior report and in §1 here double-counts them. The dedup rows in §1 are the honest pooled numbers, and the "8 repos" framing in the task is really 7 distinct histories among the 11 that produced measurements.
2. R2 is uninformative for the founder. The founder's agent commits carry the founder's git author email, so "reworked by a different author" is 0/351 by construction. R2 only carries signal in multi-author repos (poe2: agent 28.2% vs human 19.2%). On the founder slice R4 reduces to R3. This is the limitation the task anticipated: git author identity cannot separate "the agent wrote it and I fixed it" from "I wrote it and I fixed it".
3. The reworking commit is approximated, not identified. See Method. Reverse blame names the last commit where a line survived; this measurement takes the next commit on the ancestry path that touches the same file. It is wrong when a file is renamed between s and the real edit, and it collapses to "first commit after s" when the file path disappears entirely.
4. The test-path clause dominates R3, and it is not really a corrective signal. Splitting R3: of all 1,002 agent commits, 22.7% qualify by test path only, 1.2% by fix-word subject only, and 35.6% by both; for humans, 9.8% / 3.1% / 11.7%. So R3 is mostly measuring "the follow-up commit touched a test file", which reads as "the code lives in a tested area and the follow-up was a substantial change", not "someone fixed a bug". R4-strict is reported throughout precisely because it drops this clause; it is the more defensible corrective definition, and it is also where car-racer's reversal disappears and dynorouter-gateway's appears.
5. The fix-word regex is broad in conventional-commit repos. fix(scope): ... is the standard prefix in poe2, jones, dotfiles, and dynorouter. A fix: commit that touches the same file for an unrelated reason scores as corrective rework. The regex also matches error and wrong inside ordinary prose subjects.
6. Bot commits are classed as human. dependabot, poe-man[bot], github-actions[bot], and the poe-strategy-research@openclaw.local pipeline (342 of 400 work-brain commits) have no agent trailer, so the trailer test files them under "human". They are 522 of 1,857 pooled human commits (28%). Bot-excluded rows are given in §1.
7. All caveats 1-10 from /tmp/panout-ideas/backtest/REPORT.md still apply, in particular: rework is a proxy for quality not a measurement of it; attribution is by commit trailer only; six repos were capped at their 400 most recent in-window commits so their rates describe the most recent slice; only the top 20 files by added lines per commit were checked; lockfiles, minified assets and binaries were skipped, so commits touching only skipped files are absent from all denominators.
8. -C without -C -C limits copy detection to files modified in the same commit. A block moved from a file untouched by the reworking commit still registers as rework under R1. Adding -C -C would be substantially slower and was not tried.
9. Small-n repos drive most of the sign table. cursor-plugins (1 agent commit), work-brain (1), research/linear (4), car-racer (10). Their gaps are reported for completeness but no confidence interval would exclude zero.
10. The bootstrap resamples commits, not repositories. Commits within a repo are not independent - a single day's work can produce a run of correlated commits - so the CIs in §4 are narrower than a repo-clustered bootstrap would give.