CB-WP-0004 T06: retrospective — what actually predicted the outcome

The question was whether converting agent work to deterministic tooling
recovers capacity or the work reappears elsewhere. Neither, cleanly:
tooling recovers capacity where it removes the possibility of doing the
work by hand, and recovers nothing where it merely offers a better
option.

Sorting the five candidates by that test reproduces the results exactly.
env-setup and task-done left no manual path — env-test fails the build if
the friction returns, and task-done refuses to emit an estimate, so the
old way means knowingly writing fiction. Both met. facts-check and
`make status` left grep and heredocs one keystroke away. Both missed.

This is the principle ADR-0003 already used to reject explicit session
markers — nothing that depends on discipline gets to be the primary
index. It governs tooling economics too, and CB-RES-0003 did not apply
it.

Stated confidence sorted the extremes and said nothing in the middle,
where every judgement call lives: both high met, both medium missed, and
the single low also met, so it was miscalibrated at that end too. Future
reviews state the mechanism instead — after this fix, is the manual path
impossible, absurd, or merely inferior? — and only the first two may
carry a cost prediction.

The confound is symmetric and does not explain the split: building
repo.py and task-done.py also required inspection and patching, and those
categories still went to zero. It inflates the size of the miss, not its
location.

The most valuable output is not the 6 points of 38 recovered. It is that
the loop made a falsifiable quantitative forecast, was wrong by 4-5x, and
could say exactly which parts were wrong and why.

CB-WP-0004 status -> done.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

history/260731-mechanical-work-retrospective.md

@@ -0,0 +1,121 @@
+# 2026-07-31 — retrospective: does tooling recover capacity?
+
+CB-WP-0004 T06. The pass built `repo.py`, `task-done.py`, `status.py`,
+`facts.py`, `make env-test`, `make facts-check`, `cb-cost --since`, and
+`facts.toml`; it measured itself in
+[CB-EV-0003](../evidence/CB-EV-0003-mechanical-work.md).
+
+## The question this task was given
+
+> **Does converting agent work to deterministic tooling actually recover
+> capacity, or does the work reappear elsewhere?**
+
+**Neither, cleanly. It recovers capacity where the tool removes the
+possibility of doing the work by hand, and recovers nothing where the
+tool merely offers a better option.** That distinction — not confidence,
+not category size — is what separated the hits from the misses, and it
+was not the distinction the review used.
+
+## Prediction error, per candidate
+
+This is the first pass in the project to make a quantitative prediction
+*before* acting, so the error matters more than the saving.
+
+| # | candidate | stated confidence | predicted | measured (share of pass) | verdict |
+|---|---|---|---|---|---|
+| 1 | environment setup | **high** | <10 turns, $12–15 | 11.5% → **0.6%**, 85 turns → 1 | met |
+| 3+5 | `make task-done` | **high** | ~6 turns, $9–11 | 8.5% → **0.0%**, 46 turns → 0 | met, exceeded |
+| 6 | cb-cost flags | **low** | $2 | 3.4% → **1.0%** | met |
+| 2 | fact registry | **medium** | $6–9 | 10.4% → **9.6%** | missed |
+| 4 | `make status` | **medium** | $4, ~1 turn/session | 5.1% → **21.2%** | missed, worse |
+| | **aggregate** | | **25–30 points** | **6 points** (38.2% → 32.4%) | **missed** |
+
+The aggregate prediction was wrong by roughly 4–5×, in the same direction
+and for a related reason each time.
+
+## Did the stated confidence track reality?
+
+Partly, and not usefully.
+
+- Both **high**-confidence candidates met their prediction. Both were
+  correct.
+- Both **medium**-confidence candidates missed, one of them badly.
+- The single **low**-confidence candidate met its prediction, so
+  confidence was miscalibrated at that end too — it under-predicted.
+
+So confidence sorted the extremes correctly and said nothing useful in
+the middle, which is where every judgement call actually lives. **The
+next review should stop stating a confidence level and state the
+mechanism instead** — see below — because the mechanism was predictive
+and the confidence label was mostly a restatement of how sure the author
+felt.
+
+## The mechanism that actually predicted the outcome
+
+Sorting the five candidates by whether they **removed the work** or
+**offered an alternative** reproduces the results exactly:
+
+| candidate | after the fix, can the old way still happen? | outcome |
+|---|---|---|
+| environment setup | **No.** There is nothing to type; `make env-test` fails the build if the friction returns. | met |
+| `make task-done` | **No.** The hand path is a heredoc plus a hub call; the tool refuses estimates, so using the old way means knowingly writing fiction. | met |
+| cb-cost flags | **Mostly no.** The flags exist; writing a fresh analysis is strictly more work. | met |
+| fact registry | **Yes.** `facts-check` gates *drifted copies*; it does not stop anyone patching markdown by hand. | missed |
+| `make status` | **Yes.** `grep` still works, and is still one keystroke away. | missed |
+
+**Tooling recovers capacity when it makes the manual path impossible or
+absurd. It recovers nothing when it makes the manual path merely
+inferior** — because at the moment of choosing, the manual path is
+specific to the question being asked and the tool is generic.
+
+This is a design rule, not an observation about discipline. ADR-0003
+already rejected explicit session markers on the grounds that *nothing
+which depends on discipline gets to be the primary index*. This pass
+found the same principle governs tooling economics, and the review did
+not apply it.
+
+## The confound, and why it does not dissolve the finding
+
+This pass **built** the tools it measured, so the two misses are exactly
+the two whose tools were under construction: writing `make status`
+requires inspecting what it summarizes, and writing `facts-check`
+requires grepping every duplicated number in the repo. CB-EV-0003 states
+this before defending any number.
+
+But the confound is symmetric and does not explain the split. Building
+`repo.py` and `task-done.py` also required inspection and patching, and
+those categories still went to zero — because after the fix there was
+nothing left to do by hand. The confound inflates the size of the miss;
+it does not explain why *these two* missed.
+
+**The clean test is still owed.** A pass that uses these tools without
+building them is the only run that can confirm or refute the aggregate
+prediction, and it should carry the same three tests.
+
+## What this changes
+
+1. **Reviews state the mechanism, not a confidence level.** For each
+   candidate: *after this fix, is the manual path impossible, absurd, or
+   merely inferior?* Only the first two are predicted to recover
+   anything; the third is proposed as a quality fix, with no cost claim.
+2. **A "merely inferior" tool needs a gate to become a real saving.**
+   `facts-check` closed an error class without closing a cost category,
+   and that is a legitimate result — but the workplan predicted dollars
+   from it, and should not have.
+3. **The relocation test earned its place and should stay.** It came
+   back negative (output per response *fell* 24%), and that negative is
+   only credible because the test could have come back positive. It also
+   forced the pass to disown the halved cost-per-response as a compaction
+   effect, which was the most tempting wrong claim available.
+
+## Honest summary
+
+The pass recovered **6 points of 38**, against a predicted 25–30, on a
+window it admits is confounded. Two of five categories are gone and
+gated; two are unchanged and now understood; one error class gained its
+first executable gate.
+
+The most valuable output is not the saving. It is that **the loop made a
+falsifiable quantitative forecast, was wrong by 4–5×, and could tell
+exactly which parts were wrong and why** — which is the capability a
+loop needs before its cost projections are worth anything.

workplans/CB-WP-0004-mechanical-work.md

@@ -1,7 +1,7 @@
 ---
 id: CB-WP-0004
 title: "Move mechanical turns off the token budget, and prove it worked"
-status: in_progress
+status: done
 state_hub_workstream_id: "6880ac78-d817-41b9-b267-f12ff9deea28"
 ---
 
@@ -241,7 +241,7 @@ InnerLoop's single-source-of-fact rule moves from prose to executable —
 
 ```task
 id: CB-WP-0004-T05
-status: todo
+status: done
 priority: high
 state_hub_task_id: "922346ac-1e99-4e94-9dc6-35c8fa0eddc5"
 ```
@@ -330,3 +330,28 @@ that cannot is guessing with numbers attached.
 Record the prediction error per candidate, and whether the review's stated
 confidence levels (high/medium/low) tracked reality. If they did not, the
 next review should stop stating confidence, or state it differently.
+
+**Recorded — [260731-mechanical-work-retrospective.md](../history/260731-mechanical-work-retrospective.md).**
+
+The answer to the question is neither of the two offered:
+**tooling recovers capacity where it removes the possibility of doing the
+work by hand, and recovers nothing where it merely offers a better
+option.** Sorting the five candidates by that test reproduces the results
+exactly — the two that removed the manual path met their predictions, the
+two that left `grep` and heredocs one keystroke away did not.
+
+Confidence sorted the extremes and said nothing in the middle, which is
+where every judgement call lives: both `high` met, both `medium` missed,
+and the single `low` also met — miscalibrated at that end too. So the
+next review **states the mechanism, not a confidence level**: after this
+fix, is the manual path impossible, absurd, or merely inferior? Only the
+first two may carry a cost prediction.
+
+This is the same principle ADR-0003 used to reject explicit session
+markers — *nothing that depends on discipline gets to be the primary
+index*. It governs tooling economics too, and this review did not apply
+it.
+
+The most valuable output is not the $6 of 38 points recovered. It is that
+the loop made a falsifiable forecast, was wrong by 4–5×, and could say
+exactly which parts were wrong and why.