clay-borg/evidence/CB-EV-0005-instrument-the-table.md

CB-EV-0005: did instrumenting the acceptance table change what it enforces?

research: CB-RES-0004 adr: ADR-0005 workplan: CB-WP-0006 log: 260801-cb-wp-0006-log.md instruments: make mutation-check, make coverage, make cost-mix window: cb-cost --since dfd0d6d — 144 responses, $51.52

Numbering correction. CB-WP-0006 T08 says "commit evidence/CB-EV-0004". That number was already taken by CB-WP-0005's evidence. This is CB-EV-0005.

---

Test 1 — did the enforced count rise, per row?

M-D1-MUT: 4 of 14 → 8 of 14.

row	before	after	task
AM-1	red	red	—
AM-2	unmutatable	red	T02
AM-3	unmutatable	unmutatable — blocked, artifact never built	T02
AM-4a / AM-4b	red	red	—
AM-4c	unmutatable	unmutatable — withdrawn, retained in denominator	T04
AM-5	unmutatable	unmutatable — measured and met, spec declares it ungated	T03
AM-6	unmutatable	red	T01
AM-7	PARTIAL 1/3	PARTIAL 2/3 — hash-identical re-earned	T06
AM-8	PARTIAL 1/2	PARTIAL 1/2	—
AM-9	unmutatable	red	T03
AM-10	unmutatable	unmutatable — withdrawn	—
AM-11	unmutatable	red	T05
AM-12	red	red	—

Per-task predictions, reported against what happened:

task	predicted	measured
T01 AM-6	unmutatable → red	met
T02 AM-2	instrumented	met
T02 AM-3	instrumented or restated	restated — blocked on an artifact, not a metric
T03 AM-9	measured	met
T03 AM-5	measured or withdrawn	measured, and the first reading was wrong (below)
T04 AM-4c	targeted or dropped	dropped, with the argument
T05 AM-11	unmutatable → red	met
T06 K10	replay real, AM-7 re-earnable	met — hash clause re-earned
T07 K14/K18	implement or amend	one of each

The honest denominator, both ways. 8 of 14 is 57%. But four rows cannot be enforced — AM-3 is blocked on an artifact, AM-4c is withdrawn, AM-5 is declared ungated by the spec, AM-10 is withdrawn — and two are partial. Of the 10 enforceable rows, 8 are enforced.

The 14 stays the headline. AM-4c is deliberately retained in that denominator after withdrawal, and the tool says so on every run: a score improved by deleting the question is not an improvement.

Kernel spec→code link: 15/18 → 18/18. With the caveat the gate prints every run: that is names, not assertions.

Test 2 — did any row regress from red to SURVIVED?

Yes, once, and it was caught.

In T04, moving AM-6's gate from debug to release turned its mutation SURVIVED. The 4,000 black_box iterations were calibrated against debug's 3.4× headroom and are invisible against release's 20×. Raised to 100,000; back to red.

The generalizable finding: a weak mutation is not a fixed property of a row. It can become weak when the row's measurement conditions change — without the row, the mutation, or the code being touched. That is a maintenance burden M-D1-MUT carries and T09 should weigh.

No other row regressed. SURVIVED count across the final run: 0.

Test 3 — were the new mutations strong?

The workplan required every new mutation to be shown to fail for the stated reason. That is now mechanical, not asserted.

mutation-check gained an EXPECT-VACUOUS verdict: if a row's expect string appears in passing output, the FA guard would accept any failure at all, and the row is reported broken rather than red.

Final run: 0 vacuous expects across all 14 rows.

This control exists because the failure it detects happened. My first expect for AM-2 was "AM-2" — which appears in the passing report, so the guard would have accepted a compile error as proof the row was enforced. Caught by hand in T02; caught mechanically from T08 onward.

Every mutation added this pass is a property mutation, not a threshold tweak:

row	mutation	expect
AM-6	100,000 black_box iterations in GroundState::fold	AM-6 UNMET
AM-2	~800 filler lines into the impl	FAIL target <= 40
AM-9	a 300 MB allocation in the workload	FAIL target <= 64 MB
AM-11	NullRng::draw returns its bound	outside 0..

What the instruments caught that nothing else would

Six defects, all found by a gate rather than by review:

1. AM-6's gate measured contention, not throughput — 38,753 ev/s inside make all against 341,280 in isolation, because cargo test runs concurrently.
2. AM-5's first reading was wrong by 61% — 87.0 s under load against 37.3 s quiet. Reported as a 45% breach; it is met with 1.6× headroom. Same error class as (1), committed two tasks later by the same author in the row immediately after diagnosing it.
3. RSS over-reported 3× — getrusage(RUSAGE_CHILDREN) is a high-water mark across every reaped child, so cargo's memory was charged to the workload. Found only by cross-checking /usr/bin/time.
4. K10's first round trip did not reproduce — hashing a serde_json::Value is a different canonical form than hashing the typed aggregate. A round trip that recomputed its own comparison value would have passed this.
5. The bench workload existed twice — bench_shape kept passing while bench-test broke on the same edit.
6. facts-check caught two spec copies going stale within the hour the numbers moved.

Cost, and a regression that matters

	CB-WP-0004 window	CB-WP-0005	CB-WP-0006
pass cost	—	$16.03	$51.52
responses	—	83	144
mechanical share	26%	38%	50%
environment setup	3 turns	0	0
hub + workplan edits	1 turn	0	0
ad-hoc text patching	8 turns	11	45 turns, $13.79
orientation / inspect	41 turns	7	19 turns, $10.97
mean context	—	315,170	493,486

Mechanical share rose to 50% — the highest ever recorded, above the 38% baseline that motivated CB-WP-0004. This is not a tooling regression: environment setup and task closes are still at zero, two passes on. It is the other half of CB-WP-0004 T06's finding arriving in force — text patching and orientation never had their manual path removed, and a code-heavy pass is exactly where that spends.

Context is the sharper problem. Mean 493,486 against a 200,000 target, up from 315,170 last pass. specs/SessionShape.md has stated SS-01…SS-05 since CB-WP-0003 and none of them has ever been enforced — the only acceptance-adjacent numbers in this project with no gate at all, in a pass whose entire subject was ungated numbers.

Verdict

claim	status
enforced count rose	confirmed — 4 → 8 of 14, 8 of 10 enforceable
kernel link complete	confirmed — 18/18, names only
no row regressed to SURVIVED	confirmed — one near-miss, caught
new mutations are strong	confirmed mechanically — 0 vacuous expects
AM-3 instrumented	not met — blocked on an artifact, stated not hidden
AM-7, AM-8 fully enforced	not met — partial, reported as partial
the pass got cheaper	refuted — mechanical share rose to 50%