# 2026-07-31 — retrospective: does tooling recover capacity? CB-WP-0004 T06. The pass built `repo.py`, `task-done.py`, `status.py`, `facts.py`, `make env-test`, `make facts-check`, `cb-cost --since`, and `facts.toml`; it measured itself in [CB-EV-0003](../evidence/CB-EV-0003-mechanical-work.md). ## The question this task was given > **Does converting agent work to deterministic tooling actually recover > capacity, or does the work reappear elsewhere?** **Neither, cleanly. It recovers capacity where the tool removes the possibility of doing the work by hand, and recovers nothing where the tool merely offers a better option.** That distinction — not confidence, not category size — is what separated the hits from the misses, and it was not the distinction the review used. ## Prediction error, per candidate This is the first pass in the project to make a quantitative prediction *before* acting, so the error matters more than the saving. | # | candidate | stated confidence | predicted | measured (share of pass) | verdict | |---|---|---|---|---|---| | 1 | environment setup | **high** | <10 turns, $12–15 | 11.5% → **0.6%**, 85 turns → 1 | met | | 3+5 | `make task-done` | **high** | ~6 turns, $9–11 | 8.5% → **0.0%**, 46 turns → 0 | met, exceeded | | 6 | cb-cost flags | **low** | $2 | 3.4% → **1.0%** | met | | 2 | fact registry | **medium** | $6–9 | 10.4% → **9.6%** | missed | | 4 | `make status` | **medium** | $4, ~1 turn/session | 5.1% → **21.2%** | missed, worse | | | **aggregate** | | **25–30 points** | **6 points** (38.2% → 32.4%) | **missed** | The aggregate prediction was wrong by roughly 4–5×, in the same direction and for a related reason each time. ## Did the stated confidence track reality? Partly, and not usefully. - Both **high**-confidence candidates met their prediction. Both were correct. - Both **medium**-confidence candidates missed, one of them badly. - The single **low**-confidence candidate met its prediction, so confidence was miscalibrated at that end too — it under-predicted. So confidence sorted the extremes correctly and said nothing useful in the middle, which is where every judgement call actually lives. **The next review should stop stating a confidence level and state the mechanism instead** — see below — because the mechanism was predictive and the confidence label was mostly a restatement of how sure the author felt. ## The mechanism that actually predicted the outcome Sorting the five candidates by whether they **removed the work** or **offered an alternative** reproduces the results exactly: | candidate | after the fix, can the old way still happen? | outcome | |---|---|---| | environment setup | **No.** There is nothing to type; `make env-test` fails the build if the friction returns. | met | | `make task-done` | **No.** The hand path is a heredoc plus a hub call; the tool refuses estimates, so using the old way means knowingly writing fiction. | met | | cb-cost flags | **Mostly no.** The flags exist; writing a fresh analysis is strictly more work. | met | | fact registry | **Yes.** `facts-check` gates *drifted copies*; it does not stop anyone patching markdown by hand. | missed | | `make status` | **Yes.** `grep` still works, and is still one keystroke away. | missed | **Tooling recovers capacity when it makes the manual path impossible or absurd. It recovers nothing when it makes the manual path merely inferior** — because at the moment of choosing, the manual path is specific to the question being asked and the tool is generic. This is a design rule, not an observation about discipline. ADR-0003 already rejected explicit session markers on the grounds that *nothing which depends on discipline gets to be the primary index*. This pass found the same principle governs tooling economics, and the review did not apply it. ## The confound, and why it does not dissolve the finding This pass **built** the tools it measured, so the two misses are exactly the two whose tools were under construction: writing `make status` requires inspecting what it summarizes, and writing `facts-check` requires grepping every duplicated number in the repo. CB-EV-0003 states this before defending any number. But the confound is symmetric and does not explain the split. Building `repo.py` and `task-done.py` also required inspection and patching, and those categories still went to zero — because after the fix there was nothing left to do by hand. The confound inflates the size of the miss; it does not explain why *these two* missed. **The clean test is still owed.** A pass that uses these tools without building them is the only run that can confirm or refute the aggregate prediction, and it should carry the same three tests. ## What this changes 1. **Reviews state the mechanism, not a confidence level.** For each candidate: *after this fix, is the manual path impossible, absurd, or merely inferior?* Only the first two are predicted to recover anything; the third is proposed as a quality fix, with no cost claim. 2. **A "merely inferior" tool needs a gate to become a real saving.** `facts-check` closed an error class without closing a cost category, and that is a legitimate result — but the workplan predicted dollars from it, and should not have. 3. **The relocation test earned its place and should stay.** It came back negative (output per response *fell* 24%), and that negative is only credible because the test could have come back positive. It also forced the pass to disown the halved cost-per-response as a compaction effect, which was the most tempting wrong claim available. ## Honest summary The pass recovered **6 points of 38**, against a predicted 25–30, on a window it admits is confounded. Two of five categories are gone and gated; two are unchanged and now understood; one error class gained its first executable gate. The most valuable output is not the saving. It is that **the loop made a falsifiable quantitative forecast, was wrong by 4–5×, and could tell exactly which parts were wrong and why** — which is the capability a loop needs before its cost projections are worth anything.