--- id: CB-WP-0004 kind: meta title: "Move mechanical turns off the token budget, and prove it worked" status: done state_hub_workstream_id: "6880ac78-d817-41b9-b267-f12ff9deea28" --- # Purpose `research/CB-RES-0003-agent-vs-deterministic.md` measured every turn in both clay-borg sessions by the tool calls it made: ```text MECHANICAL (deduplicated) 290 turns $ 51.26 = 38% of pass ``` **38% of spend went through turns doing work a deterministic tool could do.** The largest single category is `cd` and `export PATH` — 84 turns and $15.33 of pure environment friction. The second is inline heredocs string-patching markdown, which is also the mechanism behind the duplicated-fact-drift error class InnerLoop v1.2 named and could not gate. This workplan converts the five worthwhile categories to classic compute and **measures whether that actually recovered anything**. The predicted recovery is $33–41 per pass, 25–30%. **The control loop is the point, not a formality.** The named failure mode is *relocation*: an agent that can no longer write a heredoc may simply write more prose, and the pass costs the same. `make cost-mix` emits the same categories that produced the baseline, so the claim is falsifiable by the same instrument that made it. A saving that cannot be demonstrated in that table did not happen. Per InnerLoop v1.2, targets here are **provisional until the instrument emits them**, and no target may be moved in the commit that measures it unless the instrument disproved it (§Step 4, correction vs retarget). ## Phase A — The certain wins ## Task: Remove environment friction ```task id: CB-WP-0004-T01 status: done priority: high state_hub_task_id: "3ddfd3e2-8596-4969-a069-09577933fcbc" ``` 84 turns / $15.33, the largest category and the least interesting work in the corpus. `cargo` is not on the default path, so every Rust-touching command carried `export PATH="$HOME/.cargo/bin:$PATH"`, and the shell does not persist `cd`. Fix at the root, not the leaf: `tools/dep-weight.py` already special-cases the missing `cargo` with a helpful error, which is evidence the friction was noticed and patched in the wrong place. Deliver: every `make` target runs from a clean shell with no prefix, from any directory. Document the one-line environment requirement in `README.md` if one remains. Remove the leaf workaround in `dep-weight.py` only if it becomes unreachable — a positive control that never fires is still cheaper than a regression. **Predicted:** environment-setup turns → **< 10** (from 84), **$12–15** recovered. Highest confidence in the review. **Delivered.** `tools/repo.py` resolves the repo root from `__file__` and `cargo` from PATH-then-rustup-locations; every tool imports it, so `REPO` is now stated once rather than four times. The Makefile derives `REPO` from `MAKEFILE_LIST` and resolves `CARGO` the same way. `make env-test` is the positive control — it runs every tool from `/` with `PATH=/usr/bin:/bin`, and is wired into `make all`, so this cannot silently regress and invalidate T05's measurement. The leaf workaround in `dep-weight.py` was kept, not deleted: it is now unreachable on a rustup machine, and `--self-test` asserts `cargo_bin()` resolves unaided. Per the task text, a control that never fires is cheaper than a regression. `loop-lint` failed on `repo.py` immediately — a new tool with a positive control but no `--self-test` entry point. Second time the gate has caught its own pass's work within the hour. ## Task: `make task-done` — one command for a task close ```task id: CB-WP-0004-T02 status: done priority: high state_hub_task_id: "8a1b59c7-6318-47ba-8cbc-83bb44dd221f" ``` Merges two categories: workplan status edits (21 turns, $4.06) and hub task-status calls (25 turns, $7.46). `make task-done T=CB-WP-0004-T02` must: 1. flip `status: todo` → `done` in the workplan file, **failing loudly on an unknown or already-done task** — the heredocs it replaces silently no-op'd on a typo; 2. read that task's **measured** cost and tokens from `cb-cost --by-task`; 3. push the hub event with the real numbers. The third point is the one that matters beyond cost. Every `update_task_status` in this project so far carried **hand-typed token estimates**, in a repo whose central finding is that estimated token counts are worthless. The hub currently holds fiction produced by the exact habit CB-WP-0002 disproved. **Predicted:** those 46 turns → **~6**, **$9–11** recovered, and the hub stops holding estimates. Add `--self-test` per InnerLoop v1.1. **Delivered.** `tools/task-done.py` + `make task-done T=`. It refuses on an unknown id, a typo'd id, an already-done task, a task with no `state_hub_task_id`, and — the one that matters — **a task with no measured spend**, rather than reporting an estimate. `cb-cost` gained `by_task_detail` (cost, response count, model histogram, and token components per task), and `task-done` imports cb-cost rather than parsing its printed table, so the hub number is not a copy that can drift. **The positive control found a real defect before the tool was used once.** Attribution keyed on a bare `T\d\d` from the commit subject, so `CB-WP-0002 T01`, `CB-WP-0003 T01` and `CB-WP-0004 T01` all landed in one bucket. The self-test reported **$12.10** for "T01"; the qualified figure is **$2.33** — a 5.2× overstatement that would have been pushed to the hub as a measured number, reproducing the fiction this task exists to end, in a new form. Fixed by `task_label()`: qualified subjects (`CB-WP-0004 T01`) key on the full id, unqualified ones stay bare and are never retro-assigned to a workplan. The pinned $93.15 benchmark is unchanged, confirming historical attribution was not disturbed. That is the **fourth** instance of trusted arithmetic (TA) — a number believed because it was produced by a program rather than by hand. ## Task: `make status` — one-shot orientation ```task id: CB-WP-0004-T03 status: done priority: medium state_hub_task_id: "5466a510-37a5-4491-b93e-509cd400cc23" ``` 49 turns / $6.87 of `grep`/`ls`/`wc` answering "what is the state of this repo". Replace with one command printing: active workplan and task counts, gate results, open spend since the last commit (CB-01), provisional item ages, and any `loop-lint` findings. Also shrinks cold-start context, which `specs/SessionShape.md` §3 measures at ~51k for a fresh session — the artifacts a new session reads to orient are exactly what this prints. **Predicted:** ~10 orientation turns → **1 per session**, **$4** recovered. Confidence medium: some inspection is genuinely exploratory and will not disappear, and the review says so. **Delivered.** `tools/status.py` + `make status`, 22 lines of output: commit and working-tree state, every workplan with its task counts, the next todo task with its heading, spend since the last commit against CB-01/CB-02, and the three fast gates. Two deliberate constraints. It **does not build** — a status command that takes two minutes gets replaced by `ls` within a day — and it **states its own limit** in the output (`slow gates not run here`), so a green `status` cannot be misread as a green `make all`. The self-test asserts the output stays under 40 lines, because the failure mode for an orientation tool is becoming another thing to skim. Its positive control refuses the confidently-empty report: a parser that found zero workplans and zero tasks would print a clean, wrong picture, which is worse than the greps it replaces. That control also caught a dot-all regex that returned twenty paragraphs of a previous task's prose as the "task heading". ## Phase B — The one that also closes an error class ## Task: Fact registry and `make facts-check` ```task id: CB-WP-0004-T04 status: done priority: high state_hub_task_id: "b5dfbf1e-619b-4e64-8463-49ff2ce38b48" ``` 75 turns / $13.86 of heredocs opening a markdown file, string-replacing a number, and writing it back — the mechanism behind **duplicated-fact drift**, the fourth error class, which InnerLoop v1.2 states as prose and cannot currently gate. Two instances on record: a price sheet inlined into a spec went stale within an hour of the real sheet changing, and the acceptance figure $92.21 → $92.87 → $93.32 → $93.15 had to be chased across a survey, a workplan, and an evidence file on every move. Deliver a registry where a number appearing in more than one artifact is declared once — **generated by the instrument that measures it wherever possible**, not hand-maintained — plus `make facts-check` failing when a committed artifact disagrees with it. **The trap to avoid, stated up front:** a hand-maintained registry moves the problem rather than solving it, and would itself become a copy that drifts. If generation from instruments proves impractical, deliver only the *check* (detect the same number stated differently in two artifacts) and say so — a gate with no generator still closes the class. **Predicted:** **$6–9** recovered, plus DFD's first executable gate. Confidence medium; this is the hardest task here and the most valuable. **Delivered — both halves, not just the check.** `facts.toml` holds 15 facts and is **generated** by `make facts-gen` from cb-cost, dep-weight and rule-coverage; the file opens with `# GENERATED — do not edit` and the self-test asserts that line is still there. The trap named in this task — a hand-maintained registry that becomes another drifting copy — is closed by `facts-check` re-running the instruments and failing if the committed registry disagrees with them. A stale registry cannot certify stale artifacts. An artifact quoting a fact tags it: `**$93.15** `. 17 occurrences across 5 artifacts are now under the gate. **Falsified before being believed.** Changing `specs/CostAccounting.md` line 158 from $93.15 to $92.87 — the exact historical drift — produced exit 1 and `specs/CostAccounting.md:158 claims fact:pinned_total but does not state $93.15`. The gate was tested against the class it exists to catch, on a real artifact, not only in its self-test. **What it does not close, stated rather than implied.** 22 untagged literal copies remain, across `specs/InnerLoop.md`, `specs/GameKernel.md`, `research/CB-RES-0002` and the older workplans. They are **reported, not failed**: tagging is opt-in, a number can legitimately recur, and a gate that fires on coincidence gets routed around. Naming the uncovered surface is more useful than claiming the class is closed. InnerLoop's single-source-of-fact rule moves from prose to executable — **v1.3**. ## Phase C — Prove it, or withdraw the claim ## Task: Control loop — measure recovery and test for relocation ```task id: CB-WP-0004-T05 status: done priority: high state_hub_task_id: "922346ac-1e99-4e94-9dc6-35c8fa0eddc5" ``` The task this workplan exists for. Run `make cost-mix` over the sessions that executed T01–T04 and commit `evidence/CB-EV-0003-mechanical-work.md` comparing against the committed baseline: | category | baseline turns | baseline $ | predicted | measured | verdict | |---|---|---|---|---|---| | environment setup | 84 | $15.33 | <10 turns | | | | ad-hoc text patching | 75 | $13.86 | $6–9 saved | | | | hub task status + workplan edit | 46 | $11.52 | ~6 turns | | | | orientation / inspect | 49 | $6.87 | $4 saved | | | | **mechanical total** | **290** | **$51.26** | **$33–41 saved** | | | **Three tests, all of which must be reported:** 1. **Did the mechanical turns disappear?** Per-category, against prediction. An unmet prediction is reported unmet, not retargeted. 2. **Did they relocate?** Total pass cost and non-mechanical turn counts must be compared too. If mechanical turns fell and prose turns rose by as much, the saving is zero and this workplan failed — that is the result to publish. 3. **Did quality hold?** `make all` green, and the same class of findings still surfacing. A cheaper pass that catches fewer errors is worse, and the loop has no metric for this yet — record the judgment explicitly rather than implying the cost number settles it. Normalize per unit of work, not per session: passes differ in size, so report **mechanical share of pass cost** (baseline: 38%) alongside absolute dollars. **Measured — [CB-EV-0003](../evidence/CB-EV-0003-mechanical-work.md).** `cb-cost` gained `--since`, so the baseline window (`--pin 578dcbe`, 662 responses, $135.60) and this pass (`--since 578dcbe`, 83 responses, $7.82) are two disjoint windows over the same transcripts. **Mechanical share fell 38.2% → 32.4%. Six points, against a predicted 25–30. The aggregate prediction is reported unmet.** Two categories met their prediction and two did not: - environment setup **11.5% → 0.6%** of pass (85 turns → 1) — met - hub + workplan closes **8.5% → 0.0%** (46 turns → 0) — met - text patching **10.4% → 9.6%** — not met - orientation **5.1% → 21.2%** — not met, *worse* **The confound is stated in the evidence before any defence:** this is the pass that *built* the tools, and the two categories that missed are precisely the two whose tools were under construction. Writing `make status` requires inspecting what it summarizes. The clean test is the next pass, which uses the tools without building them — that test is carried forward, not waived. **Relocation: not supported.** Output tokens per response *fell* 896 → 681. The rise in output's share of cost is a shrinking denominator (cache-read fell as context shrank), not more prose. Cost per response halved, and the evidence explicitly refuses to claim it: that is compaction (mean context 232,982 → 117,822), not tooling. **Quality held.** `make all` green with two new gates, and four findings surfaced this pass — three of them caught by controls written this pass, including a 5.2× attribution error that would have reached the hub as a measured number. ## Task: Retrospective ```task id: CB-WP-0004-T06 status: done priority: low state_hub_task_id: "47892c23-ef09-4910-a9af-327494289e05" ``` The question to answer honestly: **does converting agent work to deterministic tooling actually recover capacity, or does the work reappear elsewhere?** This is the first pass in the project to make a *quantitative prediction before acting*. Whether the prediction held is more informative than the saving itself — a loop that can forecast its own economics can plan; one that cannot is guessing with numbers attached. Record the prediction error per candidate, and whether the review's stated confidence levels (high/medium/low) tracked reality. If they did not, the next review should stop stating confidence, or state it differently. **Recorded — [260731-mechanical-work-retrospective.md](../history/260731-mechanical-work-retrospective.md).** The answer to the question is neither of the two offered: **tooling recovers capacity where it removes the possibility of doing the work by hand, and recovers nothing where it merely offers a better option.** Sorting the five candidates by that test reproduces the results exactly — the two that removed the manual path met their predictions, the two that left `grep` and heredocs one keystroke away did not. Confidence sorted the extremes and said nothing in the middle, which is where every judgement call lives: both `high` met, both `medium` missed, and the single `low` also met — miscalibrated at that end too. So the next review **states the mechanism, not a confidence level**: after this fix, is the manual path impossible, absurd, or merely inferior? Only the first two may carry a cost prediction. This is the same principle ADR-0003 used to reject explicit session markers — *nothing that depends on discipline gets to be the primary index*. It governs tooling economics too, and this review did not apply it. The most valuable output is not the $6 of 38 points recovered. It is that the loop made a falsifiable forecast, was wrong by 4–5×, and could say exactly which parts were wrong and why.