coulomb/clay-borg
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

CB-RES-0003 + CB-WP-0004: 38% of pass cost is mechanical turns
Some checks failed
ci / check (push) Failing after 4s
Details

Browse source 
Review of where token-priced turns did work a deterministic tool could
do. Method: classify every turn in both transcripts by the tool calls it
made. The classifier is committed in tools/cb-cost.py and emitted by
`make cost-mix`, so the baseline is reproducible and the same command
can later falsify the predictions.

  mech environment setup       84 turns  $15.33
  mech ad-hoc text patching    75 turns  $13.86
       git                     37 turns  $13.85
  mech hub task status         25 turns  $ 7.46
  mech orientation / inspect   49 turns  $ 6.87
       hub other               32 turns  $ 6.22
  mech ad-hoc transcript       39 turns  $ 4.56
  mech workplan status edit    21 turns  $ 4.06
  MECHANICAL (dedup)          290 turns  $51.26  = 38% of pass

Largest category is `cd` and `export PATH` -- pure friction, and
dep-weight.py already patched it at the leaf, which is evidence it was
noticed and fixed in the wrong place. Second is heredocs string-patching
markdown, which is also the mechanism behind duplicated-fact drift, the
error class InnerLoop v1.2 names and cannot gate.

Explicitly NOT automated: git (37 turns, $13.85) is mostly commit
message authorship -- the highest-output turns in the corpus and the
project's reasoning record. Automating it would save money and destroy
what makes corrections cheap.

CB-WP-0004 implements five candidates and predicts $33-41 recovery
(25-30%), below the 38% measured share on purpose: some inspection and
patching is genuinely exploratory.

The control loop is the deliverable, not a formality. T05 tests three
things and must report all: did mechanical turns disappear, did they
RELOCATE into prose, and did quality hold. If mechanical turns fall and
prose rises by as much, the saving is zero and that is the result to
publish.

Also a self-indictment worth recording: every hub update_task_status in
this project carried hand-typed token estimates, in a repo whose central
finding is that estimated token counts are worthless. T02 fixes it by
reading measured values from cb-cost.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 09:49:57 +02:00
parent 0c1eb9ecba
commit 7e21df378a
5 changed files with 473 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

6
Makefile
View file

@ -3,7 +3,7 @@
CARGO := cargo
.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget loop-lint self-tests loc all
.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget cost-mix loop-lint self-tests loc all
## fmt + clippy (deny warnings) + HashMap deny-lint
check:
@ -44,6 +44,10 @@ self-tests:
cost-budget: cost-test
python3 tools/cb-cost.py --budget
# CB-RES-0003 baseline: mechanical vs judgment turns.
cost-mix: cost-test
python3 tools/cb-cost.py --composition
cost-pin: cost-test
python3 tools/cb-cost.py --pin fc76445 --composition --by-task

180
research/CB-RES-0003-agent-vs-deterministic.md Normal file
View file

@ -0,0 +1,180 @@
# CB-RES-0003: which agent turns can become deterministic compute
capability: meta.loop.mechanical-work
status: draft
tier: M (structural M — no new capability port; touches the one command
surface and the cost instrument; chaos d4=2 → no override)
instrument: `make cost-mix` (`tools/cb-cost.py`, tool-mix block)
Review of where token-priced agent turns did work a deterministic tool
could have done, so that capacity moves to judgment rather than mechanics.
Commissioned 2026-07-31.
**Method.** Every turn in both clay-borg session transcripts (573 responses,
$134 total) was classified by the tool calls it made. The classifier is
committed as `classify_tool()` in `tools/cb-cost.py` and its output is
emitted by `make cost-mix` — the numbers below are reproducible, and the
same command measures whether any fix worked.
---
## The measurement
```text
mech environment setup 84 turns $ 15.33
mech ad-hoc text patching 75 turns $ 13.86
git 37 turns $ 13.85
mech hub task status 25 turns $ 7.46
mech orientation / inspect 49 turns $ 6.87
hub other 32 turns $ 6.22
mech ad-hoc transcript analysis 39 turns $ 4.56
mech workplan status edit 21 turns $ 4.06
make (gates) 3 turns $ 1.82
MECHANICAL (deduplicated) 290 turns $ 51.26 = 38% of pass
```
**38% of spend went through turns whose tool calls were mechanical.**
A turn's whole cost is charged to every category it touched, so the rows
overlap and the deduplicated line is the honest total. `git` and `make`
are excluded from *mechanical* deliberately: a commit message is judgment,
and running a gate is the point.
## Candidate 1 — environment setup: 84 turns, $15.33
The single largest category, and pure friction. `cd` (51) and
`export PATH="$HOME/.cargo/bin:$PATH"` (10+) recur because the shell does
not persist state between calls and `cargo` is not on the default path.
Every `make` invocation that touches Rust had to be prefixed.
- **Deterministic replacement:** put `~/.cargo/bin` on the path the agent
starts with, and make every `make` target self-sufficient (the Makefile
already knows where it is). `tools/dep-weight.py` already special-cases
this with a helpful error — evidence the friction was noticed and
patched at the leaf instead of the root.
- **Expected effect:** these turns do not become cheaper, they **stop
existing**. Predicted saving: **~$12–15 per two-session pass**, the
cleanest win in this review.
- **Risk:** near zero. Nothing depends on the current behaviour.
## Candidate 2 — ad-hoc text patching: 75 turns, $13.86
Inline `python3 - <<'PY'` heredocs that open a markdown file, string-replace,
and write it back. Written fresh each time, unreviewed, and the mechanism
behind the **duplicated-fact-drift** class: a number changes and every
artifact quoting it must be found and patched by hand. The acceptance
figure moved four times and each move cost a sweep across three or four
files.
- **Deterministic replacement:** a *fact registry*. Numbers that appear in
more than one artifact are declared once (`facts.toml` or emitted by the
tool that measures them) and injected into markdown by a generator, with
`make facts-check` failing when a committed artifact disagrees with the
registry.
- **Expected effect:** removes the sweep, and gives the DFD class its
first executable gate — which InnerLoop v1.2 currently states only as
prose. Predicted saving: **~$6–9 per pass**, plus the error class.
- **Risk:** moderate. A registry that is itself hand-maintained just moves
the problem. It must be generated from instruments where possible, and
the check must be the enforcement, not the generation.
## Candidate 3 — hub task status: 25 turns, $7.46
Every `update_task_status` call was hand-written, including
`tokens_in`/`tokens_out` — **which were estimates I typed**, in a session
whose entire subject was that estimated token counts are worthless. The
hub holds numbers derived from the very habit CB-WP-0002 disproved.
- **Deterministic replacement:** `make task-done T=T05` — flips the
workplan file, reads the measured cost for that task from `cb-cost`, and
pushes the hub event with real numbers. One command replaces an edit, a
status call, and a fabricated figure.
- **Expected effect:** **~$7 per pass**, and the hub stops holding
fiction. Combines with candidate 5.
- **Risk:** low. `cb-cost --by-task` already produces the figure.
## Candidate 4 — orientation / inspect: 49 turns, $6.87
`grep`/`ls`/`wc` to answer "what is the state of this repo" — which
workplan is active, which tasks are open, which gates pass, what is
uncommitted.
- **Deterministic replacement:** `make status` printing the loop state in
one shot: active workplan, task counts, gate results, open cost since
last commit, provisional item ages.
- **Expected effect:** turns ~10 orientation turns into 1 at the start of
a session, and shrinks cold-start context (SS-04). Predicted saving:
**~$4 per pass**.
- **Risk:** low, but the saving is softer than it looks — some inspection
is genuinely exploratory and will not disappear.
## Candidate 5 — workplan status edit: 21 turns, $4.06
Heredocs doing `s.replace("status: todo", "status: done")` on a workplan
file. Purely mechanical, and error-prone: it silently does nothing if the
task is already done or the ID is mistyped.
- **Deterministic replacement:** folded into candidate 3's
`make task-done`, which can *fail* on an unknown task ID instead of
no-op'ing.
- **Expected effect:** **~$4 per pass** and one class of silent no-op
removed.
## Candidate 6 — ad-hoc transcript analysis: 39 turns, $4.56
Already partly solved: `cb-cost` subsumed most of this during CB-WP-0002,
and this review's own classifier is now committed rather than ad-hoc. The
residue is one-off questions (context percentiles, compaction boundaries,
per-model splits) that were each written fresh.
- **Deterministic replacement:** promote the recurring ones to flags. Most
already exist (`--by-task`, `--composition`, session shape, tool mix).
- **Expected effect:** **~$2 per pass**, diminishing. Listed for
completeness, not priority.
## What must NOT be automated
Stating this because a review that only finds savings is not a review.
- **`git` (37 turns, $13.85)** is the second-most expensive category and
is mostly *commit message authorship* — the highest-output-token turns
in the corpus. That output is the project's reasoning record. Automating
it would save money and destroy the thing that makes corrections cheap.
- **`make` gates (3 turns, $1.82)** are already deterministic; the agent
merely invokes them. Correctly cheap.
- **Judgment work is invisible in this table** — writing a spec, choosing
an attribution model, deciding a target is legitimate. That is where the
remaining 62% went, and it is what the freed capacity should buy.
## Verdict and expected total
| # | candidate | turns | measured | predicted saving/pass | confidence |
|---|---|---|---|---|---|
| 1 | environment setup | 84 | $15.33 | **$12–15** | high |
| 2 | fact registry (text patching) | 75 | $13.86 | **$6–9** | medium |
| 3+5 | `make task-done` | 46 | $11.52 | **$9–11** | high |
| 4 | `make status` | 49 | $6.87 | **$4** | medium |
| 6 | cb-cost flags | 39 | $4.56 | **$2** | low |
| | **total** | | **$51.26** | **$33–41** | |
Predicted recovery is **25–30% of a pass**, against a measured 38%
mechanical share. The gap is deliberate: some inspection and some patching
is genuinely exploratory and will not vanish.
**The prediction is falsifiable and must be checked.** `make cost-mix`
emits the same categories, so the next pass measures whether these turns
disappeared or merely relocated — the failure mode being that an agent
which no longer writes heredocs simply writes more prose instead. A
control loop that does not test for relocation is not a control loop.
## Risks in this review itself
- **A turn's cost is charged to every category it touched**, so per-row
figures overstate. The deduplicated $51.26 is the defensible number;
per-candidate savings are apportioned from it and are estimates.
- **n = 2 sessions, one repo, one agent.** The mix is a property of how
this project was built, not a general law.
- **The classifier is a regex over shell commands.** It cannot see intent:
a `cd` that precedes real work is charged the whole turn. This inflates
candidate 1, which is why its predicted saving is below its measured
cost despite being the most certain fix.

BIN
tools/__pycache__/cb-cost.cpython-312.pyc
View file

Binary file not shown.

80
tools/cb-cost.py
View file

@ -186,12 +186,13 @@ def read_responses(path, pin=None):
head = rows[0]
# Tool calls are spread across the group's lines, so they are counted
# over the whole group — one response may carry several (SS-05).
tool_calls = sum(
1
for r in rows
for c in (r["message"].get("content") or [])
if c.get("type") == "tool_use"
)
blocks = [c for r in rows for c in (r["message"].get("content") or [])
if c.get("type") == "tool_use"]
tool_calls = len(blocks)
cats = sorted({
c for c in (classify_tool(b.get("name"), b.get("input") or {})
for b in blocks) if c
})
out.append(
{
"request_id": rid,
@ -200,12 +201,51 @@ def read_responses(path, pin=None):
"session": head.get("sessionId") or os.path.basename(path),
"toks": toks,
"tool_calls": tool_calls,
"categories": cats,
"subagent": "/subagents/" in path,
}
)
return out
# CB-RES-0003: which turns are mechanical (a deterministic tool could do
# them) versus judgment (only an agent can). The baseline for measuring
# whether automation actually removes turns rather than relocating them.
def classify_tool(name, inp):
if name == "mcp__dev-hub__update_task_status":
return "hub task status"
if name.startswith("mcp__dev-hub__"):
return "hub other"
if name != "Bash":
return None
cmd = (inp.get("command") or "").strip()
if not cmd:
return None
if "python3 - <<" in cmd:
if "status: todo" in cmd or "status: done" in cmd:
return "workplan status edit"
if "jsonl" in cmd or "requestId" in cmd or "usage" in cmd:
return "ad-hoc transcript analysis"
return "ad-hoc text patching"
if cmd.startswith(("cd ", "export ")):
return "environment setup"
if cmd.split()[0] in ("grep", "ls", "wc", "sed", "head", "tail", "cat", "find"):
return "orientation / inspect"
if cmd.startswith("git "):
return "git"
if "make " in cmd:
return "make (gates)"
return None
# Categories a deterministic tool could plausibly own. Judgment-bearing
# categories (git commit messages, make gates) are excluded deliberately.
MECHANICAL = frozenset({
"environment setup", "ad-hoc text patching", "orientation / inspect",
"ad-hoc transcript analysis", "hub task status", "workplan status edit",
})
def session_shape(responses):
"""SH-1..SH-3 from specs/SessionShape.md."""
import statistics
@ -361,9 +401,25 @@ def collect(slug, pin_ref=None):
f"${sum(by_component_cost.values()):,.4f} (residual ${residual:,.4f})"
)
# Tool mix: a turn's whole cost is charged to each category it touched,
# so columns may overlap and must not be summed as if disjoint.
mix_turns, mix_cost = collections.Counter(), collections.defaultdict(float)
for r in responses:
for c in r.get("categories") or []:
mix_turns[c] += 1
mix_cost[c] += r["cost"] or 0.0
mech = [r for r in responses
if set(r.get("categories") or []) & MECHANICAL]
sub = sum(r["cost"] or 0 for r in responses if r["subagent"])
return {
"session_shape": session_shape(responses),
"tool_mix": {
"turns": dict(mix_turns),
"cost": dict(mix_cost),
"mechanical_turns": len(mech),
"mechanical_cost": sum(r["cost"] or 0 for r in mech),
},
"slug": slug,
"pin": pin,
"responses": len(responses),
@ -415,6 +471,18 @@ def render(rep, by_task=False, composition=False):
f" A per-task table is a view over {100*(1-un/tot):.0f}% of spend."
)
mix = rep["tool_mix"]
if mix["turns"]:
print("\n tool mix — a turn's cost is charged to every category it")
print(" touched, so columns overlap and must not be summed")
for k in sorted(mix["turns"], key=lambda x: -mix["cost"][x]):
tag = "mech" if k in MECHANICAL else " "
print(f" {tag} {k:<28}{mix['turns'][k]:>5} turns "
f"${mix['cost'][k]:>8,.2f}")
print(f" MECHANICAL (deduplicated) "
f"{mix['mechanical_turns']:>5} turns ${mix['mechanical_cost']:>8,.2f}"
f" = {100*mix['mechanical_cost']/(rep['total'] or 1):.0f}% of pass")
sh = rep["session_shape"]
print("\n session shape (specs/SessionShape.md)")
print(f" SH-1 mean context {sh['SH-1_mean_context']:>12,.0f} tok "

214
workplans/CB-WP-0004-mechanical-work.md Normal file
View file

@ -0,0 +1,214 @@
---
id: CB-WP-0004
title: "Move mechanical turns off the token budget, and prove it worked"
status: proposed
state_hub_workstream_id: ""
---
# Purpose
`research/CB-RES-0003-agent-vs-deterministic.md` measured every turn in
both clay-borg sessions by the tool calls it made:
```text
MECHANICAL (deduplicated) 290 turns $ 51.26 = 38% of pass
```
**38% of spend went through turns doing work a deterministic tool could
do.** The largest single category is `cd` and `export PATH` — 84 turns and
$15.33 of pure environment friction. The second is inline heredocs
string-patching markdown, which is also the mechanism behind the
duplicated-fact-drift error class InnerLoop v1.2 named and could not gate.
This workplan converts the five worthwhile categories to classic compute
and **measures whether that actually recovered anything**. The predicted
recovery is $33–41 per pass, 25–30%.
**The control loop is the point, not a formality.** The named failure mode
is *relocation*: an agent that can no longer write a heredoc may simply
write more prose, and the pass costs the same. `make cost-mix` emits the
same categories that produced the baseline, so the claim is falsifiable by
the same instrument that made it. A saving that cannot be demonstrated in
that table did not happen.
Per InnerLoop v1.2, targets here are **provisional until the instrument
emits them**, and no target may be moved in the commit that measures it
unless the instrument disproved it (§Step 4, correction vs retarget).
## Phase A — The certain wins
## Task: Remove environment friction
```task
id: CB-WP-0004-T01
status: todo
priority: high
state_hub_task_id: ""
```
84 turns / $15.33, the largest category and the least interesting work in
the corpus. `cargo` is not on the default path, so every Rust-touching
command carried `export PATH="$HOME/.cargo/bin:$PATH"`, and the shell does
not persist `cd`.
Fix at the root, not the leaf: `tools/dep-weight.py` already special-cases
the missing `cargo` with a helpful error, which is evidence the friction
was noticed and patched in the wrong place.
Deliver: every `make` target runs from a clean shell with no prefix, from
any directory. Document the one-line environment requirement in
`README.md` if one remains. Remove the leaf workaround in `dep-weight.py`
only if it becomes unreachable — a positive control that never fires is
still cheaper than a regression.
**Predicted:** environment-setup turns → **< 10** (from 84),
**$12–15** recovered. Highest confidence in the review.
## Task: `make task-done` — one command for a task close
```task
id: CB-WP-0004-T02
status: todo
priority: high
state_hub_task_id: ""
```
Merges two categories: workplan status edits (21 turns, $4.06) and hub
task-status calls (25 turns, $7.46).
`make task-done T=CB-WP-0004-T02` must:
1. flip `status: todo` → `done` in the workplan file, **failing loudly on
an unknown or already-done task** — the heredocs it replaces silently
no-op'd on a typo;
2. read that task's **measured** cost and tokens from `cb-cost --by-task`;
3. push the hub event with the real numbers.
The third point is the one that matters beyond cost. Every
`update_task_status` in this project so far carried **hand-typed token
estimates**, in a repo whose central finding is that estimated token counts
are worthless. The hub currently holds fiction produced by the exact habit
CB-WP-0002 disproved.
**Predicted:** those 46 turns → **~6**, **$9–11** recovered, and the hub
stops holding estimates. Add `--self-test` per InnerLoop v1.1.
## Task: `make status` — one-shot orientation
```task
id: CB-WP-0004-T03
status: todo
priority: medium
state_hub_task_id: ""
```
49 turns / $6.87 of `grep`/`ls`/`wc` answering "what is the state of this
repo". Replace with one command printing: active workplan and task counts,
gate results, open spend since the last commit (CB-01), provisional item
ages, and any `loop-lint` findings.
Also shrinks cold-start context, which `specs/SessionShape.md` §3 measures
at ~51k for a fresh session — the artifacts a new session reads to orient
are exactly what this prints.
**Predicted:** ~10 orientation turns → **1 per session**, **$4** recovered.
Confidence medium: some inspection is genuinely exploratory and will not
disappear, and the review says so.
## Phase B — The one that also closes an error class
## Task: Fact registry and `make facts-check`
```task
id: CB-WP-0004-T04
status: todo
priority: high
state_hub_task_id: ""
```
75 turns / $13.86 of heredocs opening a markdown file, string-replacing a
number, and writing it back — the mechanism behind **duplicated-fact
drift**, the fourth error class, which InnerLoop v1.2 states as prose and
cannot currently gate.
Two instances on record: a price sheet inlined into a spec went stale
within an hour of the real sheet changing, and the acceptance figure
$92.21 → $92.87 → $93.32 → $93.15 had to be chased across a survey, a
workplan, and an evidence file on every move.
Deliver a registry where a number appearing in more than one artifact is
declared once — **generated by the instrument that measures it wherever
possible**, not hand-maintained — plus `make facts-check` failing when a
committed artifact disagrees with it.
**The trap to avoid, stated up front:** a hand-maintained registry moves
the problem rather than solving it, and would itself become a copy that
drifts. If generation from instruments proves impractical, deliver only
the *check* (detect the same number stated differently in two artifacts)
and say so — a gate with no generator still closes the class.
**Predicted:** **$6–9** recovered, plus DFD's first executable gate.
Confidence medium; this is the hardest task here and the most valuable.
## Phase C — Prove it, or withdraw the claim
## Task: Control loop — measure recovery and test for relocation
```task
id: CB-WP-0004-T05
status: todo
priority: high
state_hub_task_id: ""
```
The task this workplan exists for. Run `make cost-mix` over the sessions
that executed T01–T04 and commit
`evidence/CB-EV-0003-mechanical-work.md` comparing against the committed
baseline:
| category | baseline turns | baseline $ | predicted | measured | verdict |
|---|---|---|---|---|---|
| environment setup | 84 | $15.33 | <10 turns | | |
| ad-hoc text patching | 75 | $13.86 | $6–9 saved | | |
| hub task status + workplan edit | 46 | $11.52 | ~6 turns | | |
| orientation / inspect | 49 | $6.87 | $4 saved | | |
| **mechanical total** | **290** | **$51.26** | **$33–41 saved** | | |
**Three tests, all of which must be reported:**
1. **Did the mechanical turns disappear?** Per-category, against
prediction. An unmet prediction is reported unmet, not retargeted.
2. **Did they relocate?** Total pass cost and non-mechanical turn counts
must be compared too. If mechanical turns fell and prose turns rose by
as much, the saving is zero and this workplan failed — that is the
result to publish.
3. **Did quality hold?** `make all` green, and the same class of findings
still surfacing. A cheaper pass that catches fewer errors is worse, and
the loop has no metric for this yet — record the judgment explicitly
rather than implying the cost number settles it.
Normalize per unit of work, not per session: passes differ in size, so
report **mechanical share of pass cost** (baseline: 38%) alongside
absolute dollars.
## Task: Retrospective
```task
id: CB-WP-0004-T06
status: todo
priority: low
state_hub_task_id: ""
```
The question to answer honestly: **does converting agent work to
deterministic tooling actually recover capacity, or does the work
reappear elsewhere?**
This is the first pass in the project to make a *quantitative prediction
before acting*. Whether the prediction held is more informative than the
saving itself — a loop that can forecast its own economics can plan; one
that cannot is guessing with numbers attached.
Record the prediction error per candidate, and whether the review's stated
confidence levels (high/medium/low) tracked reality. If they did not, the
next review should stop stating confidence, or state it differently.