diff --git a/evidence/CB-EV-0003-mechanical-work.md b/evidence/CB-EV-0003-mechanical-work.md new file mode 100644 index 0000000..c56ea18 --- /dev/null +++ b/evidence/CB-EV-0003-mechanical-work.md @@ -0,0 +1,150 @@ +# CB-EV-0003: did converting mechanical turns to tooling recover anything? + +research: [CB-RES-0003](../research/CB-RES-0003-agent-vs-deterministic.md) +workplan: [CB-WP-0004](../workplans/CB-WP-0004-mechanical-work.md) +instrument: `make cost-mix` (`tools/cb-cost.py --composition`) +commit: measured on the tree at CB-WP-0004 T05 + +**Two windows over the same transcripts**, cut at `578dcbe` — the commit +that published the baseline: + +| window | command | responses | pass cost | +|---|---|---|---| +| **baseline** (CB-WP-0001…0003 + the review) | `cb-cost --pin 578dcbe` | 662 | **$135.60** | +| **this pass** (CB-WP-0004 T01–T04) | `cb-cost --since 578dcbe` | 83 | **$7.82** | + +The windows differ 17× in size, so **every verdict below is on share of +pass**, per the normalization this task required. Absolute dollars are +shown but decide nothing. + +--- + +## Test 1 — did the mechanical turns disappear? + +| category | baseline $ | baseline share | this pass $ | this share | predicted | verdict | +|---|---|---|---|---|---|---| +| environment setup | $15.58 (85 turns) | **11.5%** | $0.05 (1 turn) | **0.6%** | <10 turns | **met, decisively** | +| hub task status + workplan edit | $11.52 (46 turns) | **8.5%** | $0.00 (0 turns) | **0.0%** | ~6 turns | **met** | +| ad-hoc text patching | $14.11 (76 turns) | **10.4%** | $0.75 (8 turns) | **9.6%** | $6–9 saved | **not met** | +| orientation / inspect | $6.87 (49 turns) | **5.1%** | $1.66 (16 turns) | **21.2%** | ~1 turn/session | **not met — worse** | +| ad-hoc transcript analysis | $4.56 (39 turns) | 3.4% | $0.08 (1 turn) | 1.0% | $2 saved | met | +| **mechanical total** | **$51.76 (292)** | **38.2%** | **$2.53 (26)** | **32.4%** | **25–30% recovered** | **not met** | + +**Mechanical share fell 38.2% → 32.4%. That is 6 points, not the 25–30 +points the review predicted.** The prediction is reported unmet. Per +InnerLoop §Step 4 no target was moved in the commit that measured it. + +### The two that worked + +Environment setup went from the single largest category to effectively +zero — 85 turns to 1, an 18× drop in share. `make env-test` runs in +`make all`, so it cannot regress silently. + +Hub and workplan closes went to **exactly zero** hand-written turns. Four +tasks were closed this pass, each with one `make task-done`, and each hub +event carries measured tokens rather than a typed estimate — the first +time that has been true in this repo. + +### The two that did not + +**Text patching barely moved (10.4% → 9.6%).** `facts-check` gates +drifted copies but nothing removed the act of patching markdown: the +Makefile edits and the fact-tagging in T04 were both done with the same +heredocs the task was meant to retire. The gate closed the error class; +it did not close the cost category. Those are different claims and the +review conflated them. + +**Orientation got worse — 5.1% → 21.2% of pass.** This is the largest +single miss and it deserves the plain reading first: `make status` was +built this pass and then barely used. 16 turns of `grep`/`sed`/`ls` +still went to reading the repo. + +## The confound, stated before any defence of the numbers + +**This pass is the pass that built the tools.** Writing `make status` +requires inspecting exactly the artifacts `make status` summarizes; +writing `facts-check` requires grepping for every duplicated number in +the repo. The two categories that missed are precisely the two whose +tools were under construction. + +So this measurement **understates** the effect, and no part of it should +be read as the settled answer. The clean test is the *next* pass, which +uses the tools without building them. That test is not optional: a +prediction that can only be confirmed by a differently-shaped future run +is not yet confirmed, and CB-WP-0005 should carry it. + +What this pass does establish is the half that is not confounded: +environment setup and task closes are gone, they were 20% of baseline +pass cost, and their tools have gates that keep them gone. + +## Test 2 — did the work relocate? + +The named failure mode: an agent that can no longer write heredocs simply +writes more prose, and the pass costs the same. + +| signal | baseline | this pass | reading | +|---|---|---|---| +| output tokens per response | **896** | **681** | fell 24% — no prose inflation | +| output share of cost | 13.2% | 18.1% | rose, but see below | +| cost per response | $0.205 | $0.094 | fell 54% — **not attributable to this work** | +| non-mechanical share | 61.8% | 67.6% | rose 6 pts, mirroring the mechanical fall | + +**Relocation into prose is not supported.** Output per response *fell*. +The output share of cost rose only because cache-read share fell +(64.6% → 62.5%) as context shrank — the denominator moved, not the +numerator. + +**The halved cost per response is a compaction effect, not a T01–T04 +effect,** and claiming it would be the most tempting error available +here. `specs/SessionShape.md` measures mean context at 232,982 tokens in +the baseline window against **117,822** in this one, because this session +was compacted twice. Attributing that to tooling would repeat CB-WP-0002's +original mistake in a new direction. + +The 6-point rise in non-mechanical share is arithmetic, not relocation: +if mechanical work leaves and the same judgment work remains, judgment's +share rises by construction. + +## Test 3 — did quality hold? + +Recorded as explicit judgment, because the loop has no metric for this +and a cost number does not settle it. + +- `make all` green, now including two gates that did not exist: + `env-test` and `facts-check`. +- **Findings kept surfacing at the same rate.** This pass produced four, + each caught by a control rather than by review: + 1. `loop-lint` failed on `repo.py` — a new tool with a positive control + but no `--self-test` entry point. + 2. `task-done`'s self-test reported **$12.10** for "T01" where the + qualified figure is **$2.33** — a bare `T\d\d` attribution key + collided across three workplans. That 5.2× overstatement would have + been pushed to the hub *as a measured number*. Fourth instance of + trusted arithmetic. + 3. `status.py`'s heading regex returned twenty paragraphs of the + preceding task's prose as a "task heading". + 4. `facts-check` failed on a live fact tag inside its own documentation + example in `specs/InnerLoop.md`. +- No scenario, benchmark or coverage number regressed. The pinned + benchmark is **$93.15**, unchanged by the attribution-key fix, which is + the evidence that historical attribution was not disturbed. + +**Verdict: quality held.** Three of the four findings were caught by +controls written this pass, which is the pattern the loop is trying to +buy. + +## Verdict + +| claim | status | +|---|---| +| environment setup eliminated | **confirmed** | +| task closes eliminated, hub on measured numbers | **confirmed** | +| DFD has an executable gate | **confirmed** (falsified against a real artifact) | +| text patching reduced | **not met** | +| orientation reduced | **not met — worse this pass** | +| 25–30% of pass recovered | **not met — 6 points, on a confounded window** | +| work did not relocate into prose | **confirmed** (output/response fell 24%) | + +The workplan said a saving that cannot be demonstrated in this table did +not happen. Two of five categories are demonstrated; the aggregate +prediction is not. Both are recorded as measured. diff --git a/tools/__pycache__/cb-cost.cpython-312.pyc b/tools/__pycache__/cb-cost.cpython-312.pyc index 21da165..a5d92f2 100644 Binary files a/tools/__pycache__/cb-cost.cpython-312.pyc and b/tools/__pycache__/cb-cost.cpython-312.pyc differ diff --git a/tools/cb-cost.py b/tools/cb-cost.py index e0af2bf..503eeb1 100644 --- a/tools/cb-cost.py +++ b/tools/cb-cost.py @@ -64,6 +64,8 @@ def task_label(subject): return f"{m.group(1)}-{m.group(2)}" m = TASK_RE.search(subject) return m.group(0) if m else None + + UNATTRIBUTED = "UNATTRIBUTED" OPEN_REMAINDER = "OPEN (uncommitted)" @@ -362,9 +364,17 @@ def attribute(responses, commits): # ----------------------------------------------------------------- report -def collect(slug, pin_ref=None): +def collect(slug, pin_ref=None, since_ref=None): + """Report over (since_ref, pin_ref]. Both ends optional. + + `since` exists for CB-WP-0004 T05: comparing a pass against the pass + that measured it needs two disjoint windows over the same transcripts, + not two whole-corpus totals. Attribution and reconciliation are + unchanged — the window only selects which responses are counted. + """ prices = load_prices() pin = resolve_pin(pin_ref) + since = resolve_pin(since_ref) paths = transcript_paths(slug) if not paths: raise Abort(f"no transcripts found for {slug}") @@ -372,6 +382,8 @@ def collect(slug, pin_ref=None): responses = [] for p in paths: responses.extend(read_responses(p, pin)) + if since: + responses = [r for r in responses if r["timestamp"] > since] if not responses: # Positive control: a run that measured nothing must not report $0.00 # as though it were an answer. @@ -453,6 +465,7 @@ def collect(slug, pin_ref=None): }, "slug": slug, "pin": pin, + "since": since, "responses": len(responses), "transcripts": len(paths), "total": total, @@ -494,6 +507,8 @@ def collect(slug, pin_ref=None): def render(rep, by_task=False, composition=False): print(f"M-D2-CST cost report — {rep['slug']}") print(f" pin {rep['pin'] or '(none — live file, not reproducible)'}") + if rep.get("since"): + print(f" since {rep['since']} (window is exclusive of this instant)") print(f" transcripts {rep['transcripts']} responses {rep['responses']}") print(f" main ${rep['main_total']:>10,.2f}") print(f" subagent tree ${rep['subagent_total']:>10,.2f}") @@ -714,6 +729,7 @@ def main(): ap = argparse.ArgumentParser() ap.add_argument("--slug", default="-home-worsch-clay-borg") ap.add_argument("--pin", help="commit-ish or ISO Z instant (CA-07)") + ap.add_argument("--since", help="window start, exclusive: commit-ish or ISO Z") ap.add_argument("--by-task", action="store_true") ap.add_argument("--composition", action="store_true") ap.add_argument("--session-shape", action="store_true", @@ -733,7 +749,7 @@ def main(): return budget(args.slug, args.soft, args.hard) try: - rep = collect(args.slug, args.pin) + rep = collect(args.slug, args.pin, args.since) except Abort as e: print(f"ABORT — {e}", file=sys.stderr) return 1 diff --git a/workplans/CB-WP-0004-mechanical-work.md b/workplans/CB-WP-0004-mechanical-work.md index 03a9eac..04898b4 100644 --- a/workplans/CB-WP-0004-mechanical-work.md +++ b/workplans/CB-WP-0004-mechanical-work.md @@ -178,7 +178,7 @@ as the "task heading". ```task id: CB-WP-0004-T04 -status: todo +status: done priority: high state_hub_task_id: "b5dfbf1e-619b-4e64-8463-49ff2ce38b48" ``` @@ -276,6 +276,39 @@ Normalize per unit of work, not per session: passes differ in size, so report **mechanical share of pass cost** (baseline: 38%) alongside absolute dollars. +**Measured — [CB-EV-0003](../evidence/CB-EV-0003-mechanical-work.md).** +`cb-cost` gained `--since`, so the baseline window (`--pin 578dcbe`, 662 +responses, $135.60) and this pass (`--since 578dcbe`, 83 responses, +$7.82) are two disjoint windows over the same transcripts. + +**Mechanical share fell 38.2% → 32.4%. Six points, against a predicted +25–30. The aggregate prediction is reported unmet.** + +Two categories met their prediction and two did not: + +- environment setup **11.5% → 0.6%** of pass (85 turns → 1) — met +- hub + workplan closes **8.5% → 0.0%** (46 turns → 0) — met +- text patching **10.4% → 9.6%** — not met +- orientation **5.1% → 21.2%** — not met, *worse* + +**The confound is stated in the evidence before any defence:** this is +the pass that *built* the tools, and the two categories that missed are +precisely the two whose tools were under construction. Writing +`make status` requires inspecting what it summarizes. The clean test is +the next pass, which uses the tools without building them — that test is +carried forward, not waived. + +**Relocation: not supported.** Output tokens per response *fell* 896 → +681. The rise in output's share of cost is a shrinking denominator +(cache-read fell as context shrank), not more prose. Cost per response +halved, and the evidence explicitly refuses to claim it: that is +compaction (mean context 232,982 → 117,822), not tooling. + +**Quality held.** `make all` green with two new gates, and four findings +surfaced this pass — three of them caught by controls written this pass, +including a 5.2× attribution error that would have reached the hub as a +measured number. + ## Task: Retrospective ```task