From 7b4d9e359fcdda3cf4d73cdfaa4151494e28ee80 Mon Sep 17 00:00:00 2001 From: tegwick Date: Fri, 31 Jul 2026 10:28:35 +0200 Subject: [PATCH] =?UTF-8?q?CB-WP-0004=20T06:=20retrospective=20=E2=80=94?= =?UTF-8?q?=20what=20actually=20predicted=20the=20outcome?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The question was whether converting agent work to deterministic tooling recovers capacity or the work reappears elsewhere. Neither, cleanly: tooling recovers capacity where it removes the possibility of doing the work by hand, and recovers nothing where it merely offers a better option. Sorting the five candidates by that test reproduces the results exactly. env-setup and task-done left no manual path — env-test fails the build if the friction returns, and task-done refuses to emit an estimate, so the old way means knowingly writing fiction. Both met. facts-check and `make status` left grep and heredocs one keystroke away. Both missed. This is the principle ADR-0003 already used to reject explicit session markers — nothing that depends on discipline gets to be the primary index. It governs tooling economics too, and CB-RES-0003 did not apply it. Stated confidence sorted the extremes and said nothing in the middle, where every judgement call lives: both high met, both medium missed, and the single low also met, so it was miscalibrated at that end too. Future reviews state the mechanism instead — after this fix, is the manual path impossible, absurd, or merely inferior? — and only the first two may carry a cost prediction. The confound is symmetric and does not explain the split: building repo.py and task-done.py also required inspection and patching, and those categories still went to zero. It inflates the size of the miss, not its location. The most valuable output is not the 6 points of 38 recovered. It is that the loop made a falsifiable quantitative forecast, was wrong by 4-5x, and could say exactly which parts were wrong and why. CB-WP-0004 status -> done. Co-Authored-By: Claude Opus 5 --- .../260731-mechanical-work-retrospective.md | 121 ++++++++++++++++++ workplans/CB-WP-0004-mechanical-work.md | 29 ++++- 2 files changed, 148 insertions(+), 2 deletions(-) create mode 100644 history/260731-mechanical-work-retrospective.md diff --git a/history/260731-mechanical-work-retrospective.md b/history/260731-mechanical-work-retrospective.md new file mode 100644 index 0000000..9d2c435 --- /dev/null +++ b/history/260731-mechanical-work-retrospective.md @@ -0,0 +1,121 @@ +# 2026-07-31 — retrospective: does tooling recover capacity? + +CB-WP-0004 T06. The pass built `repo.py`, `task-done.py`, `status.py`, +`facts.py`, `make env-test`, `make facts-check`, `cb-cost --since`, and +`facts.toml`; it measured itself in +[CB-EV-0003](../evidence/CB-EV-0003-mechanical-work.md). + +## The question this task was given + +> **Does converting agent work to deterministic tooling actually recover +> capacity, or does the work reappear elsewhere?** + +**Neither, cleanly. It recovers capacity where the tool removes the +possibility of doing the work by hand, and recovers nothing where the +tool merely offers a better option.** That distinction — not confidence, +not category size — is what separated the hits from the misses, and it +was not the distinction the review used. + +## Prediction error, per candidate + +This is the first pass in the project to make a quantitative prediction +*before* acting, so the error matters more than the saving. + +| # | candidate | stated confidence | predicted | measured (share of pass) | verdict | +|---|---|---|---|---|---| +| 1 | environment setup | **high** | <10 turns, $12–15 | 11.5% → **0.6%**, 85 turns → 1 | met | +| 3+5 | `make task-done` | **high** | ~6 turns, $9–11 | 8.5% → **0.0%**, 46 turns → 0 | met, exceeded | +| 6 | cb-cost flags | **low** | $2 | 3.4% → **1.0%** | met | +| 2 | fact registry | **medium** | $6–9 | 10.4% → **9.6%** | missed | +| 4 | `make status` | **medium** | $4, ~1 turn/session | 5.1% → **21.2%** | missed, worse | +| | **aggregate** | | **25–30 points** | **6 points** (38.2% → 32.4%) | **missed** | + +The aggregate prediction was wrong by roughly 4–5×, in the same direction +and for a related reason each time. + +## Did the stated confidence track reality? + +Partly, and not usefully. + +- Both **high**-confidence candidates met their prediction. Both were + correct. +- Both **medium**-confidence candidates missed, one of them badly. +- The single **low**-confidence candidate met its prediction, so + confidence was miscalibrated at that end too — it under-predicted. + +So confidence sorted the extremes correctly and said nothing useful in +the middle, which is where every judgement call actually lives. **The +next review should stop stating a confidence level and state the +mechanism instead** — see below — because the mechanism was predictive +and the confidence label was mostly a restatement of how sure the author +felt. + +## The mechanism that actually predicted the outcome + +Sorting the five candidates by whether they **removed the work** or +**offered an alternative** reproduces the results exactly: + +| candidate | after the fix, can the old way still happen? | outcome | +|---|---|---| +| environment setup | **No.** There is nothing to type; `make env-test` fails the build if the friction returns. | met | +| `make task-done` | **No.** The hand path is a heredoc plus a hub call; the tool refuses estimates, so using the old way means knowingly writing fiction. | met | +| cb-cost flags | **Mostly no.** The flags exist; writing a fresh analysis is strictly more work. | met | +| fact registry | **Yes.** `facts-check` gates *drifted copies*; it does not stop anyone patching markdown by hand. | missed | +| `make status` | **Yes.** `grep` still works, and is still one keystroke away. | missed | + +**Tooling recovers capacity when it makes the manual path impossible or +absurd. It recovers nothing when it makes the manual path merely +inferior** — because at the moment of choosing, the manual path is +specific to the question being asked and the tool is generic. + +This is a design rule, not an observation about discipline. ADR-0003 +already rejected explicit session markers on the grounds that *nothing +which depends on discipline gets to be the primary index*. This pass +found the same principle governs tooling economics, and the review did +not apply it. + +## The confound, and why it does not dissolve the finding + +This pass **built** the tools it measured, so the two misses are exactly +the two whose tools were under construction: writing `make status` +requires inspecting what it summarizes, and writing `facts-check` +requires grepping every duplicated number in the repo. CB-EV-0003 states +this before defending any number. + +But the confound is symmetric and does not explain the split. Building +`repo.py` and `task-done.py` also required inspection and patching, and +those categories still went to zero — because after the fix there was +nothing left to do by hand. The confound inflates the size of the miss; +it does not explain why *these two* missed. + +**The clean test is still owed.** A pass that uses these tools without +building them is the only run that can confirm or refute the aggregate +prediction, and it should carry the same three tests. + +## What this changes + +1. **Reviews state the mechanism, not a confidence level.** For each + candidate: *after this fix, is the manual path impossible, absurd, or + merely inferior?* Only the first two are predicted to recover + anything; the third is proposed as a quality fix, with no cost claim. +2. **A "merely inferior" tool needs a gate to become a real saving.** + `facts-check` closed an error class without closing a cost category, + and that is a legitimate result — but the workplan predicted dollars + from it, and should not have. +3. **The relocation test earned its place and should stay.** It came + back negative (output per response *fell* 24%), and that negative is + only credible because the test could have come back positive. It also + forced the pass to disown the halved cost-per-response as a compaction + effect, which was the most tempting wrong claim available. + +## Honest summary + +The pass recovered **6 points of 38**, against a predicted 25–30, on a +window it admits is confounded. Two of five categories are gone and +gated; two are unchanged and now understood; one error class gained its +first executable gate. + +The most valuable output is not the saving. It is that **the loop made a +falsifiable quantitative forecast, was wrong by 4–5×, and could tell +exactly which parts were wrong and why** — which is the capability a +loop needs before its cost projections are worth anything. diff --git a/workplans/CB-WP-0004-mechanical-work.md b/workplans/CB-WP-0004-mechanical-work.md index 04898b4..c39f052 100644 --- a/workplans/CB-WP-0004-mechanical-work.md +++ b/workplans/CB-WP-0004-mechanical-work.md @@ -1,7 +1,7 @@ --- id: CB-WP-0004 title: "Move mechanical turns off the token budget, and prove it worked" -status: in_progress +status: done state_hub_workstream_id: "6880ac78-d817-41b9-b267-f12ff9deea28" --- @@ -241,7 +241,7 @@ InnerLoop's single-source-of-fact rule moves from prose to executable — ```task id: CB-WP-0004-T05 -status: todo +status: done priority: high state_hub_task_id: "922346ac-1e99-4e94-9dc6-35c8fa0eddc5" ``` @@ -330,3 +330,28 @@ that cannot is guessing with numbers attached. Record the prediction error per candidate, and whether the review's stated confidence levels (high/medium/low) tracked reality. If they did not, the next review should stop stating confidence, or state it differently. + +**Recorded — [260731-mechanical-work-retrospective.md](../history/260731-mechanical-work-retrospective.md).** + +The answer to the question is neither of the two offered: +**tooling recovers capacity where it removes the possibility of doing the +work by hand, and recovers nothing where it merely offers a better +option.** Sorting the five candidates by that test reproduces the results +exactly — the two that removed the manual path met their predictions, the +two that left `grep` and heredocs one keystroke away did not. + +Confidence sorted the extremes and said nothing in the middle, which is +where every judgement call lives: both `high` met, both `medium` missed, +and the single `low` also met — miscalibrated at that end too. So the +next review **states the mechanism, not a confidence level**: after this +fix, is the manual path impossible, absurd, or merely inferior? Only the +first two may carry a cost prediction. + +This is the same principle ADR-0003 used to reject explicit session +markers — *nothing that depends on discipline gets to be the primary +index*. It governs tooling economics too, and this review did not apply +it. + +The most valuable output is not the $6 of 38 points recovered. It is that +the loop made a falsifiable forecast, was wrong by 4–5×, and could say +exactly which parts were wrong and why.