Appendix I: Retrospective contract backtest on the founder's repos (2026-09-04)
Read-only. Script: scripts/backtest_contracts.py (evaluation) and scripts/analyze_contracts.py (summary). Raw output kept at /tmp/panout-site/analysis/founder.json on the founder's machine; not committed because it names paths and authors.
Why this exists
The round-5 judge said the predictive measurement (does a contract result at commit time predict rework?) could not exist before 2026-10-05 because it needs 30 days of commit-time capture. That is wrong in a useful way. A contract evaluates the diff, the message, and the paths, and all three exist in git for every historical commit. So the four default contracts can be evaluated on every commit made in the last 180 days exactly as the prepare-commit-msg hook would have evaluated them, and joined to the same 30-day rework proxy the audit uses. Because every one of those commits landed, every retrospective failure is what the hook would have recorded as an override.
This appendix does that for the founder's repos. Appendix J repeats it on six public repos that are not the founder's.
Method
- Repos: the 25 git repositories under
/code(excluding.explore,node_modules,vendor). Two checkouts of one history are deduplicated by root commit. 13 repos had no commits in the window or were archives; 12 contributed. - Commits: non-merge commits on the default branch dated between 180 and 30 days ago (2026-03-08 to 2026-08-05), capped at the 600 most recent per repo. Bots excluded (dependabot, renovate, GitHub Actions, release-please,
[bot]authors). - Attribution: agent if the message carries a Claude, Codex, Cursor, Copilot, ChatGPT, Devin or Aider co-author trailer, a "Generated with Claude Code" line, or mentions Codex (same rule as appendix A, for parity).
- Contracts: the four MVP defaults, evaluated by the shipped
panout.evaluate_contractagainst a snapshot built fromgit diff sha^ sha:tests-touch(source paths changed with no test path),no-secrets(secret shapes in added lines),size-guard(more than 400 added lines),agent-attribution(never fails; excluded from prediction). - Rework: line-identity proxy from appendix F definition R1 (
git blame --reverse -w -M -Cagainst the 30-day endpoint), plus the corrective variant R3 (the reworking commit has a fix-word subject or touches a test path). - Uncertainty: 2,000-sample bootstrap on the difference in rework rate between commits that failed and commits that passed each contract.
Headline numbers
| value | |
|---|---|
| commits measured (non-bot) | 3,221 |
| agent-attributed | 934 (29%) |
| exposure: commits with at least one non-skip contract result | 100% |
| retrospective override rate: commits that failed at least one contract | 26% |
| override rate, agent commits | 43% |
| override rate, human commits | 19% |
| per-repo override rate range (12 repos) | 9% to 48% |
The memo needed contracts to fire on roughly 30% of commits for the day-60 test to be powered. The default set fires on 26% of the founder's commits and on 43% of agent commits without any tuning.
Fail rate per contract
| contract | evaluated | fail, all | fail, agent | fail, human |
|---|---|---|---|---|
| tests-touch | 1,316 | 28% | 21% (n=695) | 36% (n=621) |
| no-secrets | 3,204 | 1% | 2% (n=923) | 1% (n=2,281) |
| size-guard | 3,221 | 15% | 29% (n=934) | 10% (n=2,287) |
Agent commits exceed 400 added lines three times as often as the founder's own. Agent commits also touch tests more often than the founder does when touching source, so tests-touch fails less on agent work.
Predictive test, unadjusted
Rework within 30 days, given the contract result at commit time. Line-identity (R1) first, corrective (R3) second.
| contract | population | rework if fail | rework if pass | delta (pp) | 95% CI |
|---|---|---|---|---|---|
| size-guard | all | 85% (489) | 49% (2,708) | +37 | +33 to +40 |
| size-guard | agent | 85% (270) | 56% (653) | +29 | +23 to +35 |
| no-secrets | all | 89% (28) | 54% (3,169) | +35 | +21 to +46 |
| tests-touch | all | 46% (366) | 72% (941) | −26 | −31 to −20 |
| tests-touch | agent | 55% (141) | 74% (548) | −19 | −28 to −10 |
| any contract | all | 68% (827) | 49% (2,370) | +18 | +15 to +22 |
| any contract | agent | 74% (396) | 57% (527) | +17 | +11 to +23 |
Corrective rework (R3) shows the same shape with larger magnitudes: size-guard +45 (CI +41 to +49), any-contract +31 (CI +27 to +35), tests-touch −31 (CI −37 to −26).
Read naively, a contract failure predicts 18 to 31 points more rework and the day-60 test is already passed. That reading is wrong, and the next table is why.
The size confound
Within bins of added lines, the any-fail versus all-pass difference disappears.
| added lines | n | rework if any fail | rework if all pass | delta (pp) |
|---|---|---|---|---|
| 0 to 20 | 428 | 28% (80) | 32% (348) | −5 |
| 21 to 100 | 934 | 44% (124) | 46% (810) | −2 |
| 101 to 400 | 1,346 | 51% (134) | 57% (1,212) | −5 |
| over 400 | 489 | 85% (489) | n/a (0) | n/a |
Every commit over 400 lines fails size-guard by definition, so the last bin has no pass group. In the three bins where a comparison exists, failing a contract predicts slightly less rework, not more. The +18 pooled delta is entirely the fact that big commits get reworked more, which size-guard restates rather than detects.
Per contract, size-adjusted (n-weighted average of within-bin deltas; bins with fewer than five commits on either side are marked thin):
| contract | 0 to 20 | 21 to 100 | 101 to 400 | over 400 | weighted delta | agent-only weighted |
|---|---|---|---|---|---|---|
| tests-touch | −8 (79/58) | −4 (123/201) | −28 (130/321) | −1 (34/361) | −13 | −3 |
| no-secrets | thin | thin | +44 (6/1,340) | +15 (17/472) | +23 | +15 |
| size-guard | thin | thin | thin | thin | not separable from size |
- tests-touch does not predict rework once size is held fixed, and on the raw numbers predicts less of it. Commits that add tests alongside source are the larger, multi-file changes that get revisited; a source-only commit is more often a one-line tweak. As a rework predictor the default is disconfirmed. As a policy a team might still want ("agent may not touch source without touching a test"), it is a preference, not a measurement.
- no-secrets is the only contract that keeps a positive size-adjusted delta (+23 weighted, +15 agent-only), but on 28 failures in 3,221 commits with two bins too thin to compare. Direction is consistent; magnitude is not established.
- size-guard cannot be separated from size by construction. Its 85% rework-if-fail is real and useful to a buyer ("commits over 400 lines are reworked 85% of the time in your repo") but it is a property of size, not of the agent or the contract.
Per-repo view
| repo (anonymised by role) | commits | agent | override rate | agent rework | human rework | gap (pp) |
|---|---|---|---|---|---|---|
| dotfiles | 482 | 128 | 17% | 64% | 31% | +32 |
| team monorepo (minority committer) | 549 | 64 | 29% | 60% | 40% | +20 |
| gateway service | 69 | 44 | 48% | 55% | 38% | +17 |
| router | 182 | 110 | 39% | 65% | 56% | +9 |
| browser | 600 | 52 | 28% | 48% | 40% | +8 |
| poe2 (agent-majority) | 545 | 484 | 41% | 67% | 71% | −4 |
| pooled | 3,221 | 934 | 26% | 65% | 50% | +15 |
The agent-versus-human gap replicates appendices A and F in direction (+15 pooled, range −4 to +32). In dotfiles, size-guard failure predicts +66 points of rework for the founder's own commits but only +15 for agent commits (CI −3 to +32): the founder's large commits are rare and messy, the agent's are routine. That is the kind of per-repo, per-contract, per-author fact the product exists to surface, and it was learnable in about four minutes of git blame with no capture period.
What this changes
- The 30-day wait is not needed to power the predictive test. The backtest runs on install, on any repo, in minutes. The memo's "labels from 2026-10-05" timeline was too conservative for the predictive question; it remains correct for the override question, because whether a human saw and overrode the failure is the one thing history does not record.
- The default contracts measure exposure, not risk. Two of the three predictive defaults carry no information once size is controlled. This is not a failure of the design; it is the design's first result: autonomy per contract means
tests-touchwould never earn trust on these repos, and the product would say so on day one instead of after sixty. - Size is the only strong observational predictor found so far, and it is free. Anyone can compute it. What is not free is knowing which of a team's own contracts, beyond size, carries signal, and that is exactly the per-contract fault-injection and override record the memo already proposes.
- The override rate is the product's exposure guarantee. 26% of all commits and 43% of agent commits would have produced a recorded override under the defaults. The buyer's audit trail is not empty on day one.
Limits
- Retrospective failures are not overrides. The hook records a human committing past a visible failure; history records only that the commit landed. Treat the 26% as the upper bound of what the hook would have written.
- The rework proxy is line identity, with the same high human floor discussed in appendix F (31% to 71% by repo).
- Most repos are the founder's own; the poe2 repo is agent-majority and dominates the agent sample (484 of 934). Appendix J addresses repos the founder does not control.
- Bootstrap intervals treat commits as independent; commits within a repo are not.