clay-borg/evidence/CB-EV-0003-mechanical-work.md

# CB-EV-0003: did converting mechanical turns to tooling recover anything?

research: [CB-RES-0003](../research/CB-RES-0003-agent-vs-deterministic.md)
workplan: [CB-WP-0004](../workplans/CB-WP-0004-mechanical-work.md)
instrument: `make cost-mix` (`tools/cb-cost.py --composition`)
commit: measured on the tree at CB-WP-0004 T05

**Two windows over the same transcripts**, cut at `578dcbe` — the commit
that published the baseline:

| window | command | responses | pass cost |
|---|---|---|---|
| **baseline** (CB-WP-0001…0003 + the review) | `cb-cost --pin 578dcbe` | 662 | **$135.60** |
| **this pass** (CB-WP-0004 T01–T04) | `cb-cost --since 578dcbe` | 83 | **$7.82** |

The windows differ 17× in size, so **every verdict below is on share of
pass**, per the normalization this task required. Absolute dollars are
shown but decide nothing.

---

## Test 1 — did the mechanical turns disappear?

| category | baseline $ | baseline share | this pass $ | this share | predicted | verdict |
|---|---|---|---|---|---|---|
| environment setup | $15.58 (85 turns) | **11.5%** | $0.05 (1 turn) | **0.6%** | <10 turns | **met, decisively** |
| hub task status + workplan edit | $11.52 (46 turns) | **8.5%** | $0.00 (0 turns) | **0.0%** | ~6 turns | **met** |
| ad-hoc text patching | $14.11 (76 turns) | **10.4%** | $0.75 (8 turns) | **9.6%** | $6–9 saved | **not met** |
| orientation / inspect | $6.87 (49 turns) | **5.1%** | $1.66 (16 turns) | **21.2%** | ~1 turn/session | **not met — worse** |
| ad-hoc transcript analysis | $4.56 (39 turns) | 3.4% | $0.08 (1 turn) | 1.0% | $2 saved | met |
| **mechanical total** | **$51.76 (292)** | **38.2%** | **$2.53 (26)** | **32.4%** | **25–30% recovered** | **not met** |

**Mechanical share fell 38.2% → 32.4%. That is 6 points, not the 25–30
points the review predicted.** The prediction is reported unmet. Per
InnerLoop §Step 4 no target was moved in the commit that measured it.

### The two that worked

Environment setup went from the single largest category to effectively
zero — 85 turns to 1, an 18× drop in share. `make env-test` runs in
`make all`, so it cannot regress silently.

Hub and workplan closes went to **exactly zero** hand-written turns. Four
tasks were closed this pass, each with one `make task-done`, and each hub
event carries measured tokens rather than a typed estimate — the first
time that has been true in this repo.

### The two that did not

**Text patching barely moved (10.4% → 9.6%).** `facts-check` gates
drifted copies but nothing removed the act of patching markdown: the
Makefile edits and the fact-tagging in T04 were both done with the same
heredocs the task was meant to retire. The gate closed the error class;
it did not close the cost category. Those are different claims and the
review conflated them.

**Orientation got worse — 5.1% → 21.2% of pass.** This is the largest
single miss and it deserves the plain reading first: `make status` was
built this pass and then barely used. 16 turns of `grep`/`sed`/`ls`
still went to reading the repo.

## The confound, stated before any defence of the numbers

**This pass is the pass that built the tools.** Writing `make status`
requires inspecting exactly the artifacts `make status` summarizes;
writing `facts-check` requires grepping for every duplicated number in
the repo. The two categories that missed are precisely the two whose
tools were under construction.

So this measurement **understates** the effect, and no part of it should
be read as the settled answer. The clean test is the *next* pass, which
uses the tools without building them. That test is not optional: a
prediction that can only be confirmed by a differently-shaped future run
is not yet confirmed, and CB-WP-0005 should carry it.

What this pass does establish is the half that is not confounded:
environment setup and task closes are gone, they were 20% of baseline
pass cost, and their tools have gates that keep them gone.

## Test 2 — did the work relocate?

The named failure mode: an agent that can no longer write heredocs simply
writes more prose, and the pass costs the same.

| signal | baseline | this pass | reading |
|---|---|---|---|
| output tokens per response | **896** | **681** | fell 24% — no prose inflation |
| output share of cost | 13.2% | 18.1% | rose, but see below |
| cost per response | $0.205 | $0.094 | fell 54% — **not attributable to this work** |
| non-mechanical share | 61.8% | 67.6% | rose 6 pts, mirroring the mechanical fall |

**Relocation into prose is not supported.** Output per response *fell*.
The output share of cost rose only because cache-read share fell
(64.6% → 62.5%) as context shrank — the denominator moved, not the
numerator.

**The halved cost per response is a compaction effect, not a T01–T04
effect,** and claiming it would be the most tempting error available
here. `specs/SessionShape.md` measures mean context at 232,982 tokens in
the baseline window against **117,822** in this one, because this session
was compacted twice. Attributing that to tooling would repeat CB-WP-0002's
original mistake in a new direction.

The 6-point rise in non-mechanical share is arithmetic, not relocation:
if mechanical work leaves and the same judgment work remains, judgment's
share rises by construction.

## Test 3 — did quality hold?

Recorded as explicit judgment, because the loop has no metric for this
and a cost number does not settle it.

- `make all` green, now including two gates that did not exist:
  `env-test` and `facts-check`.
- **Findings kept surfacing at the same rate.** This pass produced four,
  each caught by a control rather than by review:
  1. `loop-lint` failed on `repo.py` — a new tool with a positive control
     but no `--self-test` entry point.
  2. `task-done`'s self-test reported **$12.10** for "T01" where the
     qualified figure is **$2.33** — a bare `T\d\d` attribution key
     collided across three workplans. That 5.2× overstatement would have
     been pushed to the hub *as a measured number*. Fourth instance of
     trusted arithmetic.
  3. `status.py`'s heading regex returned twenty paragraphs of the
     preceding task's prose as a "task heading".
  4. `facts-check` failed on a live fact tag inside its own documentation
     example in `specs/InnerLoop.md`.
- No scenario, benchmark or coverage number regressed. The pinned
  benchmark is **$93.15**, unchanged by the attribution-key fix, which is
  the evidence that historical attribution was not disturbed.

**Verdict: quality held.** Three of the four findings were caught by
controls written this pass, which is the pattern the loop is trying to
buy.

## Verdict

| claim | status |
|---|---|
| environment setup eliminated | **confirmed** |
| task closes eliminated, hub on measured numbers | **confirmed** |
| DFD has an executable gate | **confirmed** (falsified against a real artifact) |
| text patching reduced | **not met** |
| orientation reduced | **not met — worse this pass** |
| 25–30% of pass recovered | **not met — 6 points, on a confounded window** |
| work did not relocate into prose | **confirmed** (output/response fell 24%) |

The workplan said a saving that cannot be demonstrated in this table did
not happen. Two of five categories are demonstrated; the aggregate
prediction is not. Both are recorded as measured.