diff --git a/Makefile b/Makefile index 7674ec5..02f4e61 100644 --- a/Makefile +++ b/Makefile @@ -3,7 +3,7 @@ CARGO := cargo -.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget cost-mix loop-lint self-tests loc all +.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget loop-lint self-tests loc all ## fmt + clippy (deny warnings) + HashMap deny-lint check: @@ -44,10 +44,6 @@ self-tests: cost-budget: cost-test python3 tools/cb-cost.py --budget -# CB-RES-0003 baseline: mechanical vs judgment turns. -cost-mix: cost-test - python3 tools/cb-cost.py --composition - cost-pin: cost-test python3 tools/cb-cost.py --pin fc76445 --composition --by-task diff --git a/decisions/ADR-0004-am4-ratification.md b/decisions/ADR-0004-am4-ratification.md deleted file mode 100644 index 1223b9a..0000000 --- a/decisions/ADR-0004-am4-ratification.md +++ /dev/null @@ -1,88 +0,0 @@ -# ADR-0004: ratify AM-4a and AM-4b - -status: accepted -date: 2026-07-31 -decided by: maintainer (Bernd Worsch), 2026-07-31 -tier: S (structural S — ratifies an existing target, creates no capability; chaos d4=3) -supersedes: nothing; discharges the open item raised by CB-WP-0003 T07 -references: [GameKernel.md](../specs/GameKernel.md) §4, -[InnerLoop.md](../specs/InnerLoop.md) §Step 4 (correction vs retarget), -`history/260731-inner-loop-rule-audit.md` - -## Why this ADR exists - -CB-WP-0003 T07 added a mechanical test separating a **correction** (the -instrument disproved the target; the implementation did not change) from a -**retarget** (the same commit moved both the target and the code it -measures). AM-4a/AM-4b failed that test: they were set at 250,000 and -350,000 in commit `4be6e02`, by the implementer, after seeing the measured -246,250 — and that commit also changed the feature gating the metric -measures. - -The reasoning was recorded at the time and is defensible. The *structure* -was not, and the audit flagged both targets as unratified: `make -dep-weight` has been enforcing thresholds no reviewed decision stood -behind. - -## Decision - -**AM-4a (≤ 250,000 lines, shipped runtime) and AM-4b (≤ 350,000 lines, -dev toolchain) are ratified as written.** No values change. - -## The old target, and why it was abandoned - -AM-4 originally read **≤ 20 transitive crates**, set against -boardgame.io's 120 npm packages. Retired for two measured reasons: - -1. **Unreachable without undoing this spec's own contracts.** K5 (seeded - ChaCha) and K7 (SHA-256) cost 12 crates between them. The measured - ladder showed nothing reached 20 except reimplementing one of those - primitives — trading an audited cryptographic implementation for a - scoreboard number. -2. **Crate count does not compare across ecosystems.** Rust splits crates - far more finely than npm, so the original 33-vs-120 comparison - flattered us while the ≤20 target punished us, for the same reason. - -## The measurement that motivated the change - -At the time of the retarget, `make dep-weight`: - -| configuration | crates | third-party LOC | -|---|---|---| -| shipped-runtime (`--no-default-features`) | 23 | **246,250** | -| dev-toolchain (default features) | 29 | **317,021** | -| own source | — | 3,443 | - -## Why these targets bind on future work rather than merely passing present work - -This is the question T07 requires an ADR to answer, and it is the reason -ratification is defensible rather than a rubber stamp. - -- **AM-4a leaves 3,750 lines of headroom — 1.5%.** That is roughly one - small crate. Any dependency added to the shipped runtime breaches it - almost immediately, which is the intended behaviour: the shipped runtime - is meant to be effectively frozen, and the target enforces that a new - runtime dependency is a decision someone must argue for, not a default. -- **AM-4b leaves 32,979 lines — 10.4%.** Deliberately looser. The dev - toolchain is where scenario YAML, benchmarking, and future tooling land, - and it does not ship to a player. It should be able to absorb one - moderate dependency without a spec change, and not two. -- **Both are ceilings on a quantity that only grows by choice.** Nothing - drifts a project across these thresholds; only adding a dependency does. - A target that can only be breached deliberately is a target that binds. - -**What would falsify this ratification:** if a future pass finds itself -raising AM-4a to accommodate a dependency it wanted, that is the failure -mode the ceiling exists to catch, and the answer is a new ADR arguing for -the dependency — not a quiet retarget. - -## Consequences - -- The open item in `history/260731-inner-loop-rule-audit.md` is discharged. -- `make dep-weight` continues to fail the build on breach, now backed by a - reviewed decision. -- AM-4c (own source per 100k third-party lines) remains **reported, not - targeted**, and is unaffected. -- Precedent: this is the first ADR written under the correction/retarget - test. Future retargets follow this shape — old target, the measurement, - and an argument about *future* binding rather than present passing. diff --git a/history/260731-inner-loop-rule-audit.md b/history/260731-inner-loop-rule-audit.md index cc9f56c..13999eb 100644 --- a/history/260731-inner-loop-rule-audit.md +++ b/history/260731-inner-loop-rule-audit.md @@ -136,12 +136,12 @@ evidence files. ## Open items raised by this audit -- ~~**AM-4a / AM-4b are unratified retargets.**~~ **Discharged - 2026-07-31** by [ADR-0004](../decisions/ADR-0004-am4-ratification.md), - ratified by the maintainer. Values unchanged; the ADR supplies the - future-binding argument the test requires (AM-4a leaves 1.5% headroom, - AM-4b 10.4%, and both measure a quantity that only grows by deliberate - choice). +- **AM-4a / AM-4b are unratified retargets.** Measured at 246,250 and set + at 250,000 in the same commit, by the implementer, with the + implementation changing in that commit too — a retarget under the test + added to InnerLoop §Step 4 by T07, not a correction. They must be + ratified by ADR or changed. Until then `make dep-weight` is enforcing a + target no reviewed decision stands behind. ## Class coverage — where the gaps are diff --git a/research/CB-RES-0003-agent-vs-deterministic.md b/research/CB-RES-0003-agent-vs-deterministic.md deleted file mode 100644 index 02f127e..0000000 --- a/research/CB-RES-0003-agent-vs-deterministic.md +++ /dev/null @@ -1,180 +0,0 @@ -# CB-RES-0003: which agent turns can become deterministic compute - -capability: meta.loop.mechanical-work -status: draft -tier: M (structural M — no new capability port; touches the one command -surface and the cost instrument; chaos d4=2 → no override) -instrument: `make cost-mix` (`tools/cb-cost.py`, tool-mix block) - -Review of where token-priced agent turns did work a deterministic tool -could have done, so that capacity moves to judgment rather than mechanics. -Commissioned 2026-07-31. - -**Method.** Every turn in both clay-borg session transcripts (573 responses, -$134 total) was classified by the tool calls it made. The classifier is -committed as `classify_tool()` in `tools/cb-cost.py` and its output is -emitted by `make cost-mix` — the numbers below are reproducible, and the -same command measures whether any fix worked. - ---- - -## The measurement - -```text - mech environment setup 84 turns $ 15.33 - mech ad-hoc text patching 75 turns $ 13.86 - git 37 turns $ 13.85 - mech hub task status 25 turns $ 7.46 - mech orientation / inspect 49 turns $ 6.87 - hub other 32 turns $ 6.22 - mech ad-hoc transcript analysis 39 turns $ 4.56 - mech workplan status edit 21 turns $ 4.06 - make (gates) 3 turns $ 1.82 - MECHANICAL (deduplicated) 290 turns $ 51.26 = 38% of pass -``` - -**38% of spend went through turns whose tool calls were mechanical.** - -A turn's whole cost is charged to every category it touched, so the rows -overlap and the deduplicated line is the honest total. `git` and `make` -are excluded from *mechanical* deliberately: a commit message is judgment, -and running a gate is the point. - -## Candidate 1 — environment setup: 84 turns, $15.33 - -The single largest category, and pure friction. `cd` (51) and -`export PATH="$HOME/.cargo/bin:$PATH"` (10+) recur because the shell does -not persist state between calls and `cargo` is not on the default path. -Every `make` invocation that touches Rust had to be prefixed. - -- **Deterministic replacement:** put `~/.cargo/bin` on the path the agent - starts with, and make every `make` target self-sufficient (the Makefile - already knows where it is). `tools/dep-weight.py` already special-cases - this with a helpful error — evidence the friction was noticed and - patched at the leaf instead of the root. -- **Expected effect:** these turns do not become cheaper, they **stop - existing**. Predicted saving: **~$12–15 per two-session pass**, the - cleanest win in this review. -- **Risk:** near zero. Nothing depends on the current behaviour. - -## Candidate 2 — ad-hoc text patching: 75 turns, $13.86 - -Inline `python3 - <<'PY'` heredocs that open a markdown file, string-replace, -and write it back. Written fresh each time, unreviewed, and the mechanism -behind the **duplicated-fact-drift** class: a number changes and every -artifact quoting it must be found and patched by hand. The acceptance -figure moved four times and each move cost a sweep across three or four -files. - -- **Deterministic replacement:** a *fact registry*. Numbers that appear in - more than one artifact are declared once (`facts.toml` or emitted by the - tool that measures them) and injected into markdown by a generator, with - `make facts-check` failing when a committed artifact disagrees with the - registry. -- **Expected effect:** removes the sweep, and gives the DFD class its - first executable gate — which InnerLoop v1.2 currently states only as - prose. Predicted saving: **~$6–9 per pass**, plus the error class. -- **Risk:** moderate. A registry that is itself hand-maintained just moves - the problem. It must be generated from instruments where possible, and - the check must be the enforcement, not the generation. - -## Candidate 3 — hub task status: 25 turns, $7.46 - -Every `update_task_status` call was hand-written, including -`tokens_in`/`tokens_out` — **which were estimates I typed**, in a session -whose entire subject was that estimated token counts are worthless. The -hub holds numbers derived from the very habit CB-WP-0002 disproved. - -- **Deterministic replacement:** `make task-done T=T05` — flips the - workplan file, reads the measured cost for that task from `cb-cost`, and - pushes the hub event with real numbers. One command replaces an edit, a - status call, and a fabricated figure. -- **Expected effect:** **~$7 per pass**, and the hub stops holding - fiction. Combines with candidate 5. -- **Risk:** low. `cb-cost --by-task` already produces the figure. - -## Candidate 4 — orientation / inspect: 49 turns, $6.87 - -`grep`/`ls`/`wc` to answer "what is the state of this repo" — which -workplan is active, which tasks are open, which gates pass, what is -uncommitted. - -- **Deterministic replacement:** `make status` printing the loop state in - one shot: active workplan, task counts, gate results, open cost since - last commit, provisional item ages. -- **Expected effect:** turns ~10 orientation turns into 1 at the start of - a session, and shrinks cold-start context (SS-04). Predicted saving: - **~$4 per pass**. -- **Risk:** low, but the saving is softer than it looks — some inspection - is genuinely exploratory and will not disappear. - -## Candidate 5 — workplan status edit: 21 turns, $4.06 - -Heredocs doing `s.replace("status: todo", "status: done")` on a workplan -file. Purely mechanical, and error-prone: it silently does nothing if the -task is already done or the ID is mistyped. - -- **Deterministic replacement:** folded into candidate 3's - `make task-done`, which can *fail* on an unknown task ID instead of - no-op'ing. -- **Expected effect:** **~$4 per pass** and one class of silent no-op - removed. - -## Candidate 6 — ad-hoc transcript analysis: 39 turns, $4.56 - -Already partly solved: `cb-cost` subsumed most of this during CB-WP-0002, -and this review's own classifier is now committed rather than ad-hoc. The -residue is one-off questions (context percentiles, compaction boundaries, -per-model splits) that were each written fresh. - -- **Deterministic replacement:** promote the recurring ones to flags. Most - already exist (`--by-task`, `--composition`, session shape, tool mix). -- **Expected effect:** **~$2 per pass**, diminishing. Listed for - completeness, not priority. - -## What must NOT be automated - -Stating this because a review that only finds savings is not a review. - -- **`git` (37 turns, $13.85)** is the second-most expensive category and - is mostly *commit message authorship* — the highest-output-token turns - in the corpus. That output is the project's reasoning record. Automating - it would save money and destroy the thing that makes corrections cheap. -- **`make` gates (3 turns, $1.82)** are already deterministic; the agent - merely invokes them. Correctly cheap. -- **Judgment work is invisible in this table** — writing a spec, choosing - an attribution model, deciding a target is legitimate. That is where the - remaining 62% went, and it is what the freed capacity should buy. - -## Verdict and expected total - -| # | candidate | turns | measured | predicted saving/pass | confidence | -|---|---|---|---|---|---| -| 1 | environment setup | 84 | $15.33 | **$12–15** | high | -| 2 | fact registry (text patching) | 75 | $13.86 | **$6–9** | medium | -| 3+5 | `make task-done` | 46 | $11.52 | **$9–11** | high | -| 4 | `make status` | 49 | $6.87 | **$4** | medium | -| 6 | cb-cost flags | 39 | $4.56 | **$2** | low | -| | **total** | | **$51.26** | **$33–41** | | - -Predicted recovery is **25–30% of a pass**, against a measured 38% -mechanical share. The gap is deliberate: some inspection and some patching -is genuinely exploratory and will not vanish. - -**The prediction is falsifiable and must be checked.** `make cost-mix` -emits the same categories, so the next pass measures whether these turns -disappeared or merely relocated — the failure mode being that an agent -which no longer writes heredocs simply writes more prose instead. A -control loop that does not test for relocation is not a control loop. - -## Risks in this review itself - -- **A turn's cost is charged to every category it touched**, so per-row - figures overstate. The deduplicated $51.26 is the defensible number; - per-candidate savings are apportioned from it and are estimates. -- **n = 2 sessions, one repo, one agent.** The mix is a property of how - this project was built, not a general law. -- **The classifier is a regex over shell commands.** It cannot see intent: - a `cd` that precedes real work is charged the whole turn. This inflates - candidate 1, which is why its predicted saving is below its measured - cost despite being the most certain fix. diff --git a/specs/GameKernel.md b/specs/GameKernel.md index 9ba10ef..dad7634 100644 --- a/specs/GameKernel.md +++ b/specs/GameKernel.md @@ -135,10 +135,7 @@ Command (actor-tagged intent) ## 4. Acceptance metrics -**On AM-4's retarget (2026-07-31).** *Ratified by -[ADR-0004](../decisions/ADR-0004-am4-ratification.md) on 2026-07-31; the -headroom argument for why these ceilings bind on future work lives there.* - AM-4 originally read "≤20 +**On AM-4's retarget (2026-07-31).** AM-4 originally read "≤20 transitive crates", set against boardgame.io's 120 npm packages. That target was retired for two measured reasons. First, it was unreachable without undoing this spec's own contracts: K5 (seeded ChaCha) and K7 diff --git a/tools/__pycache__/cb-cost.cpython-312.pyc b/tools/__pycache__/cb-cost.cpython-312.pyc index 2154d3e..4b96e00 100644 Binary files a/tools/__pycache__/cb-cost.cpython-312.pyc and b/tools/__pycache__/cb-cost.cpython-312.pyc differ diff --git a/tools/cb-cost.py b/tools/cb-cost.py index a895f0f..a301cc4 100644 --- a/tools/cb-cost.py +++ b/tools/cb-cost.py @@ -186,13 +186,12 @@ def read_responses(path, pin=None): head = rows[0] # Tool calls are spread across the group's lines, so they are counted # over the whole group — one response may carry several (SS-05). - blocks = [c for r in rows for c in (r["message"].get("content") or []) - if c.get("type") == "tool_use"] - tool_calls = len(blocks) - cats = sorted({ - c for c in (classify_tool(b.get("name"), b.get("input") or {}) - for b in blocks) if c - }) + tool_calls = sum( + 1 + for r in rows + for c in (r["message"].get("content") or []) + if c.get("type") == "tool_use" + ) out.append( { "request_id": rid, @@ -201,51 +200,12 @@ def read_responses(path, pin=None): "session": head.get("sessionId") or os.path.basename(path), "toks": toks, "tool_calls": tool_calls, - "categories": cats, "subagent": "/subagents/" in path, } ) return out -# CB-RES-0003: which turns are mechanical (a deterministic tool could do -# them) versus judgment (only an agent can). The baseline for measuring -# whether automation actually removes turns rather than relocating them. -def classify_tool(name, inp): - if name == "mcp__dev-hub__update_task_status": - return "hub task status" - if name.startswith("mcp__dev-hub__"): - return "hub other" - if name != "Bash": - return None - cmd = (inp.get("command") or "").strip() - if not cmd: - return None - if "python3 - <<" in cmd: - if "status: todo" in cmd or "status: done" in cmd: - return "workplan status edit" - if "jsonl" in cmd or "requestId" in cmd or "usage" in cmd: - return "ad-hoc transcript analysis" - return "ad-hoc text patching" - if cmd.startswith(("cd ", "export ")): - return "environment setup" - if cmd.split()[0] in ("grep", "ls", "wc", "sed", "head", "tail", "cat", "find"): - return "orientation / inspect" - if cmd.startswith("git "): - return "git" - if "make " in cmd: - return "make (gates)" - return None - - -# Categories a deterministic tool could plausibly own. Judgment-bearing -# categories (git commit messages, make gates) are excluded deliberately. -MECHANICAL = frozenset({ - "environment setup", "ad-hoc text patching", "orientation / inspect", - "ad-hoc transcript analysis", "hub task status", "workplan status edit", -}) - - def session_shape(responses): """SH-1..SH-3 from specs/SessionShape.md.""" import statistics @@ -401,25 +361,9 @@ def collect(slug, pin_ref=None): f"${sum(by_component_cost.values()):,.4f} (residual ${residual:,.4f})" ) - # Tool mix: a turn's whole cost is charged to each category it touched, - # so columns may overlap and must not be summed as if disjoint. - mix_turns, mix_cost = collections.Counter(), collections.defaultdict(float) - for r in responses: - for c in r.get("categories") or []: - mix_turns[c] += 1 - mix_cost[c] += r["cost"] or 0.0 - mech = [r for r in responses - if set(r.get("categories") or []) & MECHANICAL] - sub = sum(r["cost"] or 0 for r in responses if r["subagent"]) return { "session_shape": session_shape(responses), - "tool_mix": { - "turns": dict(mix_turns), - "cost": dict(mix_cost), - "mechanical_turns": len(mech), - "mechanical_cost": sum(r["cost"] or 0 for r in mech), - }, "slug": slug, "pin": pin, "responses": len(responses), @@ -471,18 +415,6 @@ def render(rep, by_task=False, composition=False): f" A per-task table is a view over {100*(1-un/tot):.0f}% of spend." ) - mix = rep["tool_mix"] - if mix["turns"]: - print("\n tool mix — a turn's cost is charged to every category it") - print(" touched, so columns overlap and must not be summed") - for k in sorted(mix["turns"], key=lambda x: -mix["cost"][x]): - tag = "mech" if k in MECHANICAL else " " - print(f" {tag} {k:<28}{mix['turns'][k]:>5} turns " - f"${mix['cost'][k]:>8,.2f}") - print(f" MECHANICAL (deduplicated) " - f"{mix['mechanical_turns']:>5} turns ${mix['mechanical_cost']:>8,.2f}" - f" = {100*mix['mechanical_cost']/(rep['total'] or 1):.0f}% of pass") - sh = rep["session_shape"] print("\n session shape (specs/SessionShape.md)") print(f" SH-1 mean context {sh['SH-1_mean_context']:>12,.0f} tok " diff --git a/workplans/CB-WP-0004-mechanical-work.md b/workplans/CB-WP-0004-mechanical-work.md deleted file mode 100644 index c0ad520..0000000 --- a/workplans/CB-WP-0004-mechanical-work.md +++ /dev/null @@ -1,214 +0,0 @@ ---- -id: CB-WP-0004 -title: "Move mechanical turns off the token budget, and prove it worked" -status: proposed -state_hub_workstream_id: "" ---- - -# Purpose - -`research/CB-RES-0003-agent-vs-deterministic.md` measured every turn in -both clay-borg sessions by the tool calls it made: - -```text - MECHANICAL (deduplicated) 290 turns $ 51.26 = 38% of pass -``` - -**38% of spend went through turns doing work a deterministic tool could -do.** The largest single category is `cd` and `export PATH` — 84 turns and -$15.33 of pure environment friction. The second is inline heredocs -string-patching markdown, which is also the mechanism behind the -duplicated-fact-drift error class InnerLoop v1.2 named and could not gate. - -This workplan converts the five worthwhile categories to classic compute -and **measures whether that actually recovered anything**. The predicted -recovery is $33–41 per pass, 25–30%. - -**The control loop is the point, not a formality.** The named failure mode -is *relocation*: an agent that can no longer write a heredoc may simply -write more prose, and the pass costs the same. `make cost-mix` emits the -same categories that produced the baseline, so the claim is falsifiable by -the same instrument that made it. A saving that cannot be demonstrated in -that table did not happen. - -Per InnerLoop v1.2, targets here are **provisional until the instrument -emits them**, and no target may be moved in the commit that measures it -unless the instrument disproved it (§Step 4, correction vs retarget). - -## Phase A — The certain wins - -## Task: Remove environment friction - -```task -id: CB-WP-0004-T01 -status: todo -priority: high -state_hub_task_id: "" -``` - -84 turns / $15.33, the largest category and the least interesting work in -the corpus. `cargo` is not on the default path, so every Rust-touching -command carried `export PATH="$HOME/.cargo/bin:$PATH"`, and the shell does -not persist `cd`. - -Fix at the root, not the leaf: `tools/dep-weight.py` already special-cases -the missing `cargo` with a helpful error, which is evidence the friction -was noticed and patched in the wrong place. - -Deliver: every `make` target runs from a clean shell with no prefix, from -any directory. Document the one-line environment requirement in -`README.md` if one remains. Remove the leaf workaround in `dep-weight.py` -only if it becomes unreachable — a positive control that never fires is -still cheaper than a regression. - -**Predicted:** environment-setup turns → **< 10** (from 84), -**$12–15** recovered. Highest confidence in the review. - -## Task: `make task-done` — one command for a task close - -```task -id: CB-WP-0004-T02 -status: todo -priority: high -state_hub_task_id: "" -``` - -Merges two categories: workplan status edits (21 turns, $4.06) and hub -task-status calls (25 turns, $7.46). - -`make task-done T=CB-WP-0004-T02` must: - -1. flip `status: todo` → `done` in the workplan file, **failing loudly on - an unknown or already-done task** — the heredocs it replaces silently - no-op'd on a typo; -2. read that task's **measured** cost and tokens from `cb-cost --by-task`; -3. push the hub event with the real numbers. - -The third point is the one that matters beyond cost. Every -`update_task_status` in this project so far carried **hand-typed token -estimates**, in a repo whose central finding is that estimated token counts -are worthless. The hub currently holds fiction produced by the exact habit -CB-WP-0002 disproved. - -**Predicted:** those 46 turns → **~6**, **$9–11** recovered, and the hub -stops holding estimates. Add `--self-test` per InnerLoop v1.1. - -## Task: `make status` — one-shot orientation - -```task -id: CB-WP-0004-T03 -status: todo -priority: medium -state_hub_task_id: "" -``` - -49 turns / $6.87 of `grep`/`ls`/`wc` answering "what is the state of this -repo". Replace with one command printing: active workplan and task counts, -gate results, open spend since the last commit (CB-01), provisional item -ages, and any `loop-lint` findings. - -Also shrinks cold-start context, which `specs/SessionShape.md` §3 measures -at ~51k for a fresh session — the artifacts a new session reads to orient -are exactly what this prints. - -**Predicted:** ~10 orientation turns → **1 per session**, **$4** recovered. -Confidence medium: some inspection is genuinely exploratory and will not -disappear, and the review says so. - -## Phase B — The one that also closes an error class - -## Task: Fact registry and `make facts-check` - -```task -id: CB-WP-0004-T04 -status: todo -priority: high -state_hub_task_id: "" -``` - -75 turns / $13.86 of heredocs opening a markdown file, string-replacing a -number, and writing it back — the mechanism behind **duplicated-fact -drift**, the fourth error class, which InnerLoop v1.2 states as prose and -cannot currently gate. - -Two instances on record: a price sheet inlined into a spec went stale -within an hour of the real sheet changing, and the acceptance figure -$92.21 → $92.87 → $93.32 → $93.15 had to be chased across a survey, a -workplan, and an evidence file on every move. - -Deliver a registry where a number appearing in more than one artifact is -declared once — **generated by the instrument that measures it wherever -possible**, not hand-maintained — plus `make facts-check` failing when a -committed artifact disagrees with it. - -**The trap to avoid, stated up front:** a hand-maintained registry moves -the problem rather than solving it, and would itself become a copy that -drifts. If generation from instruments proves impractical, deliver only -the *check* (detect the same number stated differently in two artifacts) -and say so — a gate with no generator still closes the class. - -**Predicted:** **$6–9** recovered, plus DFD's first executable gate. -Confidence medium; this is the hardest task here and the most valuable. - -## Phase C — Prove it, or withdraw the claim - -## Task: Control loop — measure recovery and test for relocation - -```task -id: CB-WP-0004-T05 -status: todo -priority: high -state_hub_task_id: "" -``` - -The task this workplan exists for. Run `make cost-mix` over the sessions -that executed T01–T04 and commit -`evidence/CB-EV-0003-mechanical-work.md` comparing against the committed -baseline: - -| category | baseline turns | baseline $ | predicted | measured | verdict | -|---|---|---|---|---|---| -| environment setup | 84 | $15.33 | <10 turns | | | -| ad-hoc text patching | 75 | $13.86 | $6–9 saved | | | -| hub task status + workplan edit | 46 | $11.52 | ~6 turns | | | -| orientation / inspect | 49 | $6.87 | $4 saved | | | -| **mechanical total** | **290** | **$51.26** | **$33–41 saved** | | | - -**Three tests, all of which must be reported:** - -1. **Did the mechanical turns disappear?** Per-category, against - prediction. An unmet prediction is reported unmet, not retargeted. -2. **Did they relocate?** Total pass cost and non-mechanical turn counts - must be compared too. If mechanical turns fell and prose turns rose by - as much, the saving is zero and this workplan failed — that is the - result to publish. -3. **Did quality hold?** `make all` green, and the same class of findings - still surfacing. A cheaper pass that catches fewer errors is worse, and - the loop has no metric for this yet — record the judgment explicitly - rather than implying the cost number settles it. - -Normalize per unit of work, not per session: passes differ in size, so -report **mechanical share of pass cost** (baseline: 38%) alongside -absolute dollars. - -## Task: Retrospective - -```task -id: CB-WP-0004-T06 -status: todo -priority: low -state_hub_task_id: "" -``` - -The question to answer honestly: **does converting agent work to -deterministic tooling actually recover capacity, or does the work -reappear elsewhere?** - -This is the first pass in the project to make a *quantitative prediction -before acting*. Whether the prediction held is more informative than the -saving itself — a loop that can forecast its own economics can plan; one -that cannot is guessing with numbers attached. - -Record the prediction error per candidate, and whether the review's stated -confidence levels (high/medium/low) tracked reality. If they did not, the -next review should stop stating confidence, or state it differently.