clay-borg/evidence/CB-EV-0003-mechanical-work.md

CB-EV-0003: did converting mechanical turns to tooling recover anything?

research: CB-RES-0003 workplan: CB-WP-0004 instrument: make cost-mix (tools/cb-cost.py --composition) commit: measured on the tree at CB-WP-0004 T05

Two windows over the same transcripts, cut at 578dcbe — the commit that published the baseline:

window	command	responses	pass cost
baseline (CB-WP-0001…0003 + the review)	cb-cost --pin 578dcbe	662	$135.60
this pass (CB-WP-0004 T01–T04)	cb-cost --since 578dcbe	83	$7.82

The windows differ 17× in size, so every verdict below is on share of pass, per the normalization this task required. Absolute dollars are shown but decide nothing.

---

Test 1 — did the mechanical turns disappear?

category	baseline $	baseline share	this pass $	this share	predicted	verdict
environment setup	$15.58 (85 turns)	11.5%	$0.05 (1 turn)	0.6%	<10 turns	met, decisively
hub task status + workplan edit	$11.52 (46 turns)	8.5%	$0.00 (0 turns)	0.0%	~6 turns	met
ad-hoc text patching	$14.11 (76 turns)	10.4%	$0.75 (8 turns)	9.6%	$6–9 saved	not met
orientation / inspect	$6.87 (49 turns)	5.1%	$1.66 (16 turns)	21.2%	~1 turn/session	not met — worse
ad-hoc transcript analysis	$4.56 (39 turns)	3.4%	$0.08 (1 turn)	1.0%	$2 saved	met
mechanical total	$51.76 (292)	38.2%	$2.53 (26)	32.4%	25–30% recovered	not met

Mechanical share fell 38.2% → 32.4%. That is 6 points, not the 25–30 points the review predicted. The prediction is reported unmet. Per InnerLoop §Step 4 no target was moved in the commit that measured it.

The two that worked

Environment setup went from the single largest category to effectively zero — 85 turns to 1, an 18× drop in share. make env-test runs in make all, so it cannot regress silently.

Hub and workplan closes went to exactly zero hand-written turns. Four tasks were closed this pass, each with one make task-done, and each hub event carries measured tokens rather than a typed estimate — the first time that has been true in this repo.

The two that did not

Text patching barely moved (10.4% → 9.6%). facts-check gates drifted copies but nothing removed the act of patching markdown: the Makefile edits and the fact-tagging in T04 were both done with the same heredocs the task was meant to retire. The gate closed the error class; it did not close the cost category. Those are different claims and the review conflated them.

Orientation got worse — 5.1% → 21.2% of pass. This is the largest single miss and it deserves the plain reading first: make status was built this pass and then barely used. 16 turns of grep/sed/ls still went to reading the repo.

The confound, stated before any defence of the numbers

This pass is the pass that built the tools. Writing make status requires inspecting exactly the artifacts make status summarizes; writing facts-check requires grepping for every duplicated number in the repo. The two categories that missed are precisely the two whose tools were under construction.

So this measurement understates the effect, and no part of it should be read as the settled answer. The clean test is the next pass, which uses the tools without building them. That test is not optional: a prediction that can only be confirmed by a differently-shaped future run is not yet confirmed, and CB-WP-0005 should carry it.

What this pass does establish is the half that is not confounded: environment setup and task closes are gone, they were 20% of baseline pass cost, and their tools have gates that keep them gone.

Test 2 — did the work relocate?

The named failure mode: an agent that can no longer write heredocs simply writes more prose, and the pass costs the same.

signal	baseline	this pass	reading
output tokens per response	896	681	fell 24% — no prose inflation
output share of cost	13.2%	18.1%	rose, but see below
cost per response	$0.205	$0.094	fell 54% — not attributable to this work
non-mechanical share	61.8%	67.6%	rose 6 pts, mirroring the mechanical fall

Relocation into prose is not supported. Output per response fell. The output share of cost rose only because cache-read share fell (64.6% → 62.5%) as context shrank — the denominator moved, not the numerator.

The halved cost per response is a compaction effect, not a T01–T04 effect, and claiming it would be the most tempting error available here. specs/SessionShape.md measures mean context at 232,982 tokens in the baseline window against 117,822 in this one, because this session was compacted twice. Attributing that to tooling would repeat CB-WP-0002's original mistake in a new direction.

The 6-point rise in non-mechanical share is arithmetic, not relocation: if mechanical work leaves and the same judgment work remains, judgment's share rises by construction.

Test 3 — did quality hold?

Recorded as explicit judgment, because the loop has no metric for this and a cost number does not settle it.

• make all green, now including two gates that did not exist: env-test and facts-check.
• Findings kept surfacing at the same rate. This pass produced four, each caught by a control rather than by review:
  1. loop-lint failed on repo.py — a new tool with a positive control but no --self-test entry point.
  2. task-done's self-test reported $12.10 for "T01" where the qualified figure is $2.33 — a bare T\d\d attribution key collided across three workplans. That 5.2× overstatement would have been pushed to the hub as a measured number. Fourth instance of trusted arithmetic.
  3. status.py's heading regex returned twenty paragraphs of the preceding task's prose as a "task heading".
  4. facts-check failed on a live fact tag inside its own documentation example in specs/InnerLoop.md.
• No scenario, benchmark or coverage number regressed. The pinned benchmark is $93.15, unchanged by the attribution-key fix, which is the evidence that historical attribution was not disturbed.

Verdict: quality held. Three of the four findings were caught by controls written this pass, which is the pattern the loop is trying to buy.

Verdict

claim	status
environment setup eliminated	confirmed
task closes eliminated, hub on measured numbers	confirmed
DFD has an executable gate	confirmed (falsified against a real artifact)
text patching reduced	not met
orientation reduced	not met — worse this pass
25–30% of pass recovered	not met — 6 points, on a confounded window
work did not relocate into prose	confirmed (output/response fell 24%)

The workplan said a saving that cannot be demonstrated in this table did not happen. Two of five categories are demonstrated; the aggregate prediction is not. Both are recorded as measured.