diff --git a/Makefile b/Makefile index 02f4e61..7674ec5 100644 --- a/Makefile +++ b/Makefile @@ -3,7 +3,7 @@ CARGO := cargo -.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget loop-lint self-tests loc all +.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget cost-mix loop-lint self-tests loc all ## fmt + clippy (deny warnings) + HashMap deny-lint check: @@ -44,6 +44,10 @@ self-tests: cost-budget: cost-test python3 tools/cb-cost.py --budget +# CB-RES-0003 baseline: mechanical vs judgment turns. +cost-mix: cost-test + python3 tools/cb-cost.py --composition + cost-pin: cost-test python3 tools/cb-cost.py --pin fc76445 --composition --by-task diff --git a/research/CB-RES-0003-agent-vs-deterministic.md b/research/CB-RES-0003-agent-vs-deterministic.md new file mode 100644 index 0000000..02f127e --- /dev/null +++ b/research/CB-RES-0003-agent-vs-deterministic.md @@ -0,0 +1,180 @@ +# CB-RES-0003: which agent turns can become deterministic compute + +capability: meta.loop.mechanical-work +status: draft +tier: M (structural M — no new capability port; touches the one command +surface and the cost instrument; chaos d4=2 → no override) +instrument: `make cost-mix` (`tools/cb-cost.py`, tool-mix block) + +Review of where token-priced agent turns did work a deterministic tool +could have done, so that capacity moves to judgment rather than mechanics. +Commissioned 2026-07-31. + +**Method.** Every turn in both clay-borg session transcripts (573 responses, +$134 total) was classified by the tool calls it made. The classifier is +committed as `classify_tool()` in `tools/cb-cost.py` and its output is +emitted by `make cost-mix` — the numbers below are reproducible, and the +same command measures whether any fix worked. + +--- + +## The measurement + +```text + mech environment setup 84 turns $ 15.33 + mech ad-hoc text patching 75 turns $ 13.86 + git 37 turns $ 13.85 + mech hub task status 25 turns $ 7.46 + mech orientation / inspect 49 turns $ 6.87 + hub other 32 turns $ 6.22 + mech ad-hoc transcript analysis 39 turns $ 4.56 + mech workplan status edit 21 turns $ 4.06 + make (gates) 3 turns $ 1.82 + MECHANICAL (deduplicated) 290 turns $ 51.26 = 38% of pass +``` + +**38% of spend went through turns whose tool calls were mechanical.** + +A turn's whole cost is charged to every category it touched, so the rows +overlap and the deduplicated line is the honest total. `git` and `make` +are excluded from *mechanical* deliberately: a commit message is judgment, +and running a gate is the point. + +## Candidate 1 — environment setup: 84 turns, $15.33 + +The single largest category, and pure friction. `cd` (51) and +`export PATH="$HOME/.cargo/bin:$PATH"` (10+) recur because the shell does +not persist state between calls and `cargo` is not on the default path. +Every `make` invocation that touches Rust had to be prefixed. + +- **Deterministic replacement:** put `~/.cargo/bin` on the path the agent + starts with, and make every `make` target self-sufficient (the Makefile + already knows where it is). `tools/dep-weight.py` already special-cases + this with a helpful error — evidence the friction was noticed and + patched at the leaf instead of the root. +- **Expected effect:** these turns do not become cheaper, they **stop + existing**. Predicted saving: **~$12–15 per two-session pass**, the + cleanest win in this review. +- **Risk:** near zero. Nothing depends on the current behaviour. + +## Candidate 2 — ad-hoc text patching: 75 turns, $13.86 + +Inline `python3 - <<'PY'` heredocs that open a markdown file, string-replace, +and write it back. Written fresh each time, unreviewed, and the mechanism +behind the **duplicated-fact-drift** class: a number changes and every +artifact quoting it must be found and patched by hand. The acceptance +figure moved four times and each move cost a sweep across three or four +files. + +- **Deterministic replacement:** a *fact registry*. Numbers that appear in + more than one artifact are declared once (`facts.toml` or emitted by the + tool that measures them) and injected into markdown by a generator, with + `make facts-check` failing when a committed artifact disagrees with the + registry. +- **Expected effect:** removes the sweep, and gives the DFD class its + first executable gate — which InnerLoop v1.2 currently states only as + prose. Predicted saving: **~$6–9 per pass**, plus the error class. +- **Risk:** moderate. A registry that is itself hand-maintained just moves + the problem. It must be generated from instruments where possible, and + the check must be the enforcement, not the generation. + +## Candidate 3 — hub task status: 25 turns, $7.46 + +Every `update_task_status` call was hand-written, including +`tokens_in`/`tokens_out` — **which were estimates I typed**, in a session +whose entire subject was that estimated token counts are worthless. The +hub holds numbers derived from the very habit CB-WP-0002 disproved. + +- **Deterministic replacement:** `make task-done T=T05` — flips the + workplan file, reads the measured cost for that task from `cb-cost`, and + pushes the hub event with real numbers. One command replaces an edit, a + status call, and a fabricated figure. +- **Expected effect:** **~$7 per pass**, and the hub stops holding + fiction. Combines with candidate 5. +- **Risk:** low. `cb-cost --by-task` already produces the figure. + +## Candidate 4 — orientation / inspect: 49 turns, $6.87 + +`grep`/`ls`/`wc` to answer "what is the state of this repo" — which +workplan is active, which tasks are open, which gates pass, what is +uncommitted. + +- **Deterministic replacement:** `make status` printing the loop state in + one shot: active workplan, task counts, gate results, open cost since + last commit, provisional item ages. +- **Expected effect:** turns ~10 orientation turns into 1 at the start of + a session, and shrinks cold-start context (SS-04). Predicted saving: + **~$4 per pass**. +- **Risk:** low, but the saving is softer than it looks — some inspection + is genuinely exploratory and will not disappear. + +## Candidate 5 — workplan status edit: 21 turns, $4.06 + +Heredocs doing `s.replace("status: todo", "status: done")` on a workplan +file. Purely mechanical, and error-prone: it silently does nothing if the +task is already done or the ID is mistyped. + +- **Deterministic replacement:** folded into candidate 3's + `make task-done`, which can *fail* on an unknown task ID instead of + no-op'ing. +- **Expected effect:** **~$4 per pass** and one class of silent no-op + removed. + +## Candidate 6 — ad-hoc transcript analysis: 39 turns, $4.56 + +Already partly solved: `cb-cost` subsumed most of this during CB-WP-0002, +and this review's own classifier is now committed rather than ad-hoc. The +residue is one-off questions (context percentiles, compaction boundaries, +per-model splits) that were each written fresh. + +- **Deterministic replacement:** promote the recurring ones to flags. Most + already exist (`--by-task`, `--composition`, session shape, tool mix). +- **Expected effect:** **~$2 per pass**, diminishing. Listed for + completeness, not priority. + +## What must NOT be automated + +Stating this because a review that only finds savings is not a review. + +- **`git` (37 turns, $13.85)** is the second-most expensive category and + is mostly *commit message authorship* — the highest-output-token turns + in the corpus. That output is the project's reasoning record. Automating + it would save money and destroy the thing that makes corrections cheap. +- **`make` gates (3 turns, $1.82)** are already deterministic; the agent + merely invokes them. Correctly cheap. +- **Judgment work is invisible in this table** — writing a spec, choosing + an attribution model, deciding a target is legitimate. That is where the + remaining 62% went, and it is what the freed capacity should buy. + +## Verdict and expected total + +| # | candidate | turns | measured | predicted saving/pass | confidence | +|---|---|---|---|---|---| +| 1 | environment setup | 84 | $15.33 | **$12–15** | high | +| 2 | fact registry (text patching) | 75 | $13.86 | **$6–9** | medium | +| 3+5 | `make task-done` | 46 | $11.52 | **$9–11** | high | +| 4 | `make status` | 49 | $6.87 | **$4** | medium | +| 6 | cb-cost flags | 39 | $4.56 | **$2** | low | +| | **total** | | **$51.26** | **$33–41** | | + +Predicted recovery is **25–30% of a pass**, against a measured 38% +mechanical share. The gap is deliberate: some inspection and some patching +is genuinely exploratory and will not vanish. + +**The prediction is falsifiable and must be checked.** `make cost-mix` +emits the same categories, so the next pass measures whether these turns +disappeared or merely relocated — the failure mode being that an agent +which no longer writes heredocs simply writes more prose instead. A +control loop that does not test for relocation is not a control loop. + +## Risks in this review itself + +- **A turn's cost is charged to every category it touched**, so per-row + figures overstate. The deduplicated $51.26 is the defensible number; + per-candidate savings are apportioned from it and are estimates. +- **n = 2 sessions, one repo, one agent.** The mix is a property of how + this project was built, not a general law. +- **The classifier is a regex over shell commands.** It cannot see intent: + a `cd` that precedes real work is charged the whole turn. This inflates + candidate 1, which is why its predicted saving is below its measured + cost despite being the most certain fix. diff --git a/tools/__pycache__/cb-cost.cpython-312.pyc b/tools/__pycache__/cb-cost.cpython-312.pyc index 4b96e00..2154d3e 100644 Binary files a/tools/__pycache__/cb-cost.cpython-312.pyc and b/tools/__pycache__/cb-cost.cpython-312.pyc differ diff --git a/tools/cb-cost.py b/tools/cb-cost.py index a301cc4..a895f0f 100644 --- a/tools/cb-cost.py +++ b/tools/cb-cost.py @@ -186,12 +186,13 @@ def read_responses(path, pin=None): head = rows[0] # Tool calls are spread across the group's lines, so they are counted # over the whole group — one response may carry several (SS-05). - tool_calls = sum( - 1 - for r in rows - for c in (r["message"].get("content") or []) - if c.get("type") == "tool_use" - ) + blocks = [c for r in rows for c in (r["message"].get("content") or []) + if c.get("type") == "tool_use"] + tool_calls = len(blocks) + cats = sorted({ + c for c in (classify_tool(b.get("name"), b.get("input") or {}) + for b in blocks) if c + }) out.append( { "request_id": rid, @@ -200,12 +201,51 @@ def read_responses(path, pin=None): "session": head.get("sessionId") or os.path.basename(path), "toks": toks, "tool_calls": tool_calls, + "categories": cats, "subagent": "/subagents/" in path, } ) return out +# CB-RES-0003: which turns are mechanical (a deterministic tool could do +# them) versus judgment (only an agent can). The baseline for measuring +# whether automation actually removes turns rather than relocating them. +def classify_tool(name, inp): + if name == "mcp__dev-hub__update_task_status": + return "hub task status" + if name.startswith("mcp__dev-hub__"): + return "hub other" + if name != "Bash": + return None + cmd = (inp.get("command") or "").strip() + if not cmd: + return None + if "python3 - <<" in cmd: + if "status: todo" in cmd or "status: done" in cmd: + return "workplan status edit" + if "jsonl" in cmd or "requestId" in cmd or "usage" in cmd: + return "ad-hoc transcript analysis" + return "ad-hoc text patching" + if cmd.startswith(("cd ", "export ")): + return "environment setup" + if cmd.split()[0] in ("grep", "ls", "wc", "sed", "head", "tail", "cat", "find"): + return "orientation / inspect" + if cmd.startswith("git "): + return "git" + if "make " in cmd: + return "make (gates)" + return None + + +# Categories a deterministic tool could plausibly own. Judgment-bearing +# categories (git commit messages, make gates) are excluded deliberately. +MECHANICAL = frozenset({ + "environment setup", "ad-hoc text patching", "orientation / inspect", + "ad-hoc transcript analysis", "hub task status", "workplan status edit", +}) + + def session_shape(responses): """SH-1..SH-3 from specs/SessionShape.md.""" import statistics @@ -361,9 +401,25 @@ def collect(slug, pin_ref=None): f"${sum(by_component_cost.values()):,.4f} (residual ${residual:,.4f})" ) + # Tool mix: a turn's whole cost is charged to each category it touched, + # so columns may overlap and must not be summed as if disjoint. + mix_turns, mix_cost = collections.Counter(), collections.defaultdict(float) + for r in responses: + for c in r.get("categories") or []: + mix_turns[c] += 1 + mix_cost[c] += r["cost"] or 0.0 + mech = [r for r in responses + if set(r.get("categories") or []) & MECHANICAL] + sub = sum(r["cost"] or 0 for r in responses if r["subagent"]) return { "session_shape": session_shape(responses), + "tool_mix": { + "turns": dict(mix_turns), + "cost": dict(mix_cost), + "mechanical_turns": len(mech), + "mechanical_cost": sum(r["cost"] or 0 for r in mech), + }, "slug": slug, "pin": pin, "responses": len(responses), @@ -415,6 +471,18 @@ def render(rep, by_task=False, composition=False): f" A per-task table is a view over {100*(1-un/tot):.0f}% of spend." ) + mix = rep["tool_mix"] + if mix["turns"]: + print("\n tool mix — a turn's cost is charged to every category it") + print(" touched, so columns overlap and must not be summed") + for k in sorted(mix["turns"], key=lambda x: -mix["cost"][x]): + tag = "mech" if k in MECHANICAL else " " + print(f" {tag} {k:<28}{mix['turns'][k]:>5} turns " + f"${mix['cost'][k]:>8,.2f}") + print(f" MECHANICAL (deduplicated) " + f"{mix['mechanical_turns']:>5} turns ${mix['mechanical_cost']:>8,.2f}" + f" = {100*mix['mechanical_cost']/(rep['total'] or 1):.0f}% of pass") + sh = rep["session_shape"] print("\n session shape (specs/SessionShape.md)") print(f" SH-1 mean context {sh['SH-1_mean_context']:>12,.0f} tok " diff --git a/workplans/CB-WP-0004-mechanical-work.md b/workplans/CB-WP-0004-mechanical-work.md new file mode 100644 index 0000000..c0ad520 --- /dev/null +++ b/workplans/CB-WP-0004-mechanical-work.md @@ -0,0 +1,214 @@ +--- +id: CB-WP-0004 +title: "Move mechanical turns off the token budget, and prove it worked" +status: proposed +state_hub_workstream_id: "" +--- + +# Purpose + +`research/CB-RES-0003-agent-vs-deterministic.md` measured every turn in +both clay-borg sessions by the tool calls it made: + +```text + MECHANICAL (deduplicated) 290 turns $ 51.26 = 38% of pass +``` + +**38% of spend went through turns doing work a deterministic tool could +do.** The largest single category is `cd` and `export PATH` — 84 turns and +$15.33 of pure environment friction. The second is inline heredocs +string-patching markdown, which is also the mechanism behind the +duplicated-fact-drift error class InnerLoop v1.2 named and could not gate. + +This workplan converts the five worthwhile categories to classic compute +and **measures whether that actually recovered anything**. The predicted +recovery is $33–41 per pass, 25–30%. + +**The control loop is the point, not a formality.** The named failure mode +is *relocation*: an agent that can no longer write a heredoc may simply +write more prose, and the pass costs the same. `make cost-mix` emits the +same categories that produced the baseline, so the claim is falsifiable by +the same instrument that made it. A saving that cannot be demonstrated in +that table did not happen. + +Per InnerLoop v1.2, targets here are **provisional until the instrument +emits them**, and no target may be moved in the commit that measures it +unless the instrument disproved it (§Step 4, correction vs retarget). + +## Phase A — The certain wins + +## Task: Remove environment friction + +```task +id: CB-WP-0004-T01 +status: todo +priority: high +state_hub_task_id: "" +``` + +84 turns / $15.33, the largest category and the least interesting work in +the corpus. `cargo` is not on the default path, so every Rust-touching +command carried `export PATH="$HOME/.cargo/bin:$PATH"`, and the shell does +not persist `cd`. + +Fix at the root, not the leaf: `tools/dep-weight.py` already special-cases +the missing `cargo` with a helpful error, which is evidence the friction +was noticed and patched in the wrong place. + +Deliver: every `make` target runs from a clean shell with no prefix, from +any directory. Document the one-line environment requirement in +`README.md` if one remains. Remove the leaf workaround in `dep-weight.py` +only if it becomes unreachable — a positive control that never fires is +still cheaper than a regression. + +**Predicted:** environment-setup turns → **< 10** (from 84), +**$12–15** recovered. Highest confidence in the review. + +## Task: `make task-done` — one command for a task close + +```task +id: CB-WP-0004-T02 +status: todo +priority: high +state_hub_task_id: "" +``` + +Merges two categories: workplan status edits (21 turns, $4.06) and hub +task-status calls (25 turns, $7.46). + +`make task-done T=CB-WP-0004-T02` must: + +1. flip `status: todo` → `done` in the workplan file, **failing loudly on + an unknown or already-done task** — the heredocs it replaces silently + no-op'd on a typo; +2. read that task's **measured** cost and tokens from `cb-cost --by-task`; +3. push the hub event with the real numbers. + +The third point is the one that matters beyond cost. Every +`update_task_status` in this project so far carried **hand-typed token +estimates**, in a repo whose central finding is that estimated token counts +are worthless. The hub currently holds fiction produced by the exact habit +CB-WP-0002 disproved. + +**Predicted:** those 46 turns → **~6**, **$9–11** recovered, and the hub +stops holding estimates. Add `--self-test` per InnerLoop v1.1. + +## Task: `make status` — one-shot orientation + +```task +id: CB-WP-0004-T03 +status: todo +priority: medium +state_hub_task_id: "" +``` + +49 turns / $6.87 of `grep`/`ls`/`wc` answering "what is the state of this +repo". Replace with one command printing: active workplan and task counts, +gate results, open spend since the last commit (CB-01), provisional item +ages, and any `loop-lint` findings. + +Also shrinks cold-start context, which `specs/SessionShape.md` §3 measures +at ~51k for a fresh session — the artifacts a new session reads to orient +are exactly what this prints. + +**Predicted:** ~10 orientation turns → **1 per session**, **$4** recovered. +Confidence medium: some inspection is genuinely exploratory and will not +disappear, and the review says so. + +## Phase B — The one that also closes an error class + +## Task: Fact registry and `make facts-check` + +```task +id: CB-WP-0004-T04 +status: todo +priority: high +state_hub_task_id: "" +``` + +75 turns / $13.86 of heredocs opening a markdown file, string-replacing a +number, and writing it back — the mechanism behind **duplicated-fact +drift**, the fourth error class, which InnerLoop v1.2 states as prose and +cannot currently gate. + +Two instances on record: a price sheet inlined into a spec went stale +within an hour of the real sheet changing, and the acceptance figure +$92.21 → $92.87 → $93.32 → $93.15 had to be chased across a survey, a +workplan, and an evidence file on every move. + +Deliver a registry where a number appearing in more than one artifact is +declared once — **generated by the instrument that measures it wherever +possible**, not hand-maintained — plus `make facts-check` failing when a +committed artifact disagrees with it. + +**The trap to avoid, stated up front:** a hand-maintained registry moves +the problem rather than solving it, and would itself become a copy that +drifts. If generation from instruments proves impractical, deliver only +the *check* (detect the same number stated differently in two artifacts) +and say so — a gate with no generator still closes the class. + +**Predicted:** **$6–9** recovered, plus DFD's first executable gate. +Confidence medium; this is the hardest task here and the most valuable. + +## Phase C — Prove it, or withdraw the claim + +## Task: Control loop — measure recovery and test for relocation + +```task +id: CB-WP-0004-T05 +status: todo +priority: high +state_hub_task_id: "" +``` + +The task this workplan exists for. Run `make cost-mix` over the sessions +that executed T01–T04 and commit +`evidence/CB-EV-0003-mechanical-work.md` comparing against the committed +baseline: + +| category | baseline turns | baseline $ | predicted | measured | verdict | +|---|---|---|---|---|---| +| environment setup | 84 | $15.33 | <10 turns | | | +| ad-hoc text patching | 75 | $13.86 | $6–9 saved | | | +| hub task status + workplan edit | 46 | $11.52 | ~6 turns | | | +| orientation / inspect | 49 | $6.87 | $4 saved | | | +| **mechanical total** | **290** | **$51.26** | **$33–41 saved** | | | + +**Three tests, all of which must be reported:** + +1. **Did the mechanical turns disappear?** Per-category, against + prediction. An unmet prediction is reported unmet, not retargeted. +2. **Did they relocate?** Total pass cost and non-mechanical turn counts + must be compared too. If mechanical turns fell and prose turns rose by + as much, the saving is zero and this workplan failed — that is the + result to publish. +3. **Did quality hold?** `make all` green, and the same class of findings + still surfacing. A cheaper pass that catches fewer errors is worse, and + the loop has no metric for this yet — record the judgment explicitly + rather than implying the cost number settles it. + +Normalize per unit of work, not per session: passes differ in size, so +report **mechanical share of pass cost** (baseline: 38%) alongside +absolute dollars. + +## Task: Retrospective + +```task +id: CB-WP-0004-T06 +status: todo +priority: low +state_hub_task_id: "" +``` + +The question to answer honestly: **does converting agent work to +deterministic tooling actually recover capacity, or does the work +reappear elsewhere?** + +This is the first pass in the project to make a *quantitative prediction +before acting*. Whether the prediction held is more informative than the +saving itself — a loop that can forecast its own economics can plan; one +that cannot is guessing with numbers attached. + +Record the prediction error per candidate, and whether the review's stated +confidence levels (high/medium/low) tracked reality. If they did not, the +next review should stop stating confidence, or state it differently.