# CB-EV-0004: did widening the instruments change what they report? research: [CB-RES-0004](../research/CB-RES-0004-replay-and-kernel-coverage.md) adr: [ADR-0005](../decisions/ADR-0005-assertion-coverage-and-replay.md) workplan: [CB-WP-0005](../workplans/CB-WP-0005-assertion-coverage.md) instruments: `make coverage`, `make mutation-check`, `make cost-mix` window: `cb-cost --since bd4423a` — 70 responses, **$16.03** Phase C was deferred unstarted, so this closes T01–T03 only. That is itself the headline result: **the pass stopped because its own instrument contradicted the plan it was executing.** --- ## Test 1 — did the denominators widen? | | before | after | |---|---|---| | specs in scope | 1 (`GroundRules.md`) | **2** (+ `GameKernel.md`) | | numbered rules in scope | 58 | **76** | | source files searched for links | 1 | **10** | | denominators reported | 1 | **3** | ```text AM-1 rule coverage: 58/58 (100%) over 21 scenarios AM-1b spec->code link: 49/58 claimed rules named in the aggregate AM-1b kernel spec->code link: 15/18 (83%) across 10 source files unlinked: K10 K14 K18 ``` **Met.** The workplan's stated failure mode — *a widened denominator that reports the same percentage would prove the instruments still count names* — did not occur. The new denominator came in at **83%**, and named three rules that four workplans of green gates never mentioned. The gate reports without feeding the exit code **until 2026-08-31**, then binds. The date is in `tools/rule-coverage.py`, the days remaining print on every run, and the self-test asserts the arm returns 0 before that date and 2 after — so it cannot quietly become never, which is how AM-4's targets went unratified for four workplans. ## Test 2 — M-D1-MUT against the prediction **Prediction ≥10 of 14 (the ADR's 9-of-12, 75%). Measured 4. UNMET.** No target moved in the commit that measured it. ```text M-D1-MUT: 4/14 rows enforced PARTIAL 2 AM-7, AM-8 — some clauses live, some inert unmutatable 8 no property to invert SURVIVED 0 ``` **Two corrections to our own numbers**, both recorded rather than quietly absorbed: 1. **There are 14 acceptance rows, not 12.** ADR-0005 and CB-WP-0005 both said twelve; AM-4 splits into a/b/c. The prediction is evaluated as ≥10 of 14 on the same 75% basis. 2. **The first run reported two `SURVIVED` rows and both were my own no-op mutations.** `pub struct NullRng;` → `pub struct NullRng {}` is semantically identical; renaming `max_age_days` changes nothing because CA-17 reads it with `.get(..., 90)`. Replaced with real inversions, after which both go red. The eight `unmutatable` rows, each with the reason the harness records: | row | why nothing can be inverted | |---|---| | **AM-6** | **nothing compares any number to 100,000 events/s** — the headline throughput claim | | AM-2 | no instrument divides LOC by rule count or compares to 40 | | AM-3 | the synthetic workload's LOC is never measured | | AM-4c | `reported, not targeted` — no threshold, so nothing can fail | | AM-5 | `recorded not gated`, and not recorded either | | AM-9 | nothing measures resident memory | | AM-10 | population empty — no `cb-*-api` crate exists | | AM-11 | the conformance suite the metric is a bool over does not exist | **The prediction was wrong about *what*, not only *how much*.** CB-RES-0004 diagnosed three absent kernel rules. The measurement found that **more than half the acceptance table has no instrument at all** — which Phase C, scoped to five rules, would not have touched. ## Test 3 — did quality hold, and did widening surface anything new? `make all` green throughout, now including two gates that did not exist: the kernel coverage arm and `mutation-check --self-test`. **It surfaced substantially more than the seven defects CB-RES-0004 named.** Six further acceptance rows with no instrument (above), plus three new defect instances: - **HDN #7** — `rule-coverage.py`'s `--self-test` printed all-ok while every real `make coverage` died with `RecursionError`. A `print(` inside `say()` had become `say(`; the control only ever called the quiet path. *The control named the behaviour and did not assert it* — the pass's own thesis, inside the tool written to prove it. - **A new error class (below)** — the two no-op mutations. - **DFD** — `evidence/CB-EV-0001` still carried **$248.46** for AM-12, the figure CB-WP-0002 disproved four workplans ago. Untagged, and therefore invisible to `facts-check`. The gate built to catch duplicated facts only checks copies that opted in. ### The fifth error class: false accusation (FA) Every class on record under-reports: a harness that does nothing, trusted arithmetic, a blind sample, a stale copy. All four let a real problem pass. The weak mutation is the first that **over-reports**. A no-op mutation yields `SURVIVED`, which reads as *"this row asserts nothing"* — a published claim that working code is broken. It sends the next pass to fix something that is not broken, and it is *more* credible than the truth because it arrives with a measurement attached. | class | direction | caught by | |---|---|---| | HDN, TA, SSB, DFD | under-report — a real defect passes | assertions, re-derivation, all-data checks, reading the copy | | **FA** *(new)* | **over-report — working code is indicted** | checking that the mutation fails *for its stated reason* | Ten instances across four classes became **thirteen across five**. ## Cost, and the clean test CB-WP-0004 was owed CB-WP-0004 T05 said its measurement was confounded because the pass built the tools it measured, and that *"the clean test is the next pass, which uses the tools without building them."* T01–T03 are largely that pass. | category | baseline (662 resp) | CB-WP-0004 build | **this pass (70 resp)** | |---|---|---|---| | environment setup | 85 turns, $15.58 | 3 turns, $0.20 | **0 turns** | | hub task status + workplan edit | 46 turns, $11.52 | 1 turn, $0.13 | **0 turns** | | orientation / inspect | 49 turns, $6.87 | 41 turns, $4.10 | 7 turns, **$3.85** | | ad-hoc text patching | 76 turns, $14.11 | 8 turns, $0.75 | 11 turns, **$2.28** | | **mechanical share of pass** | **38.2%** | 26% | **38%** | **The verdict is clean, and it confirms CB-WP-0004 T06's mechanism exactly.** The two categories whose tools *removed the manual path* went to **zero and stayed there** with no further work. The two whose tools merely *offered a better option* — `make status` and `facts-check` — are now the entire mechanical cost of a pass. Mechanical share returned to 38% not because the wins reversed, but because the denominator shrank while the misses did not. **Absolute mechanical cost per pass fell from $51.76 to $6.13; mechanical cost per response rose slightly, $0.078 → $0.088.** Share is noisy at n=70 against a $16 denominator and should not be read as a trend. **A regression to report:** `SH-3` batching is **0.0%** this pass — 67 tool calls across 67 responses, none batched, against a 20% target. Mean context 315,170 against a 200,000 target. Both worse than the pass before. Nothing in this workplan addressed session shape, and nothing gates it. ## Verdict | claim | status | |---|---| | denominators widened, and the number moved | **confirmed** | | kernel rules became visible (K10, K14, K18) | **confirmed** | | the record was corrected, five rows | **confirmed** | | M-D1-MUT ≥10 of 14 | **unmet — 4 of 14** | | the diagnosis in CB-RES-0004 was complete | **refuted** — 6 more rows, 3 new defects, 1 new class | | CB-WP-0004's "remove the manual path" mechanism | **confirmed on a clean window** | | session shape | **regressed, ungated** | The pass did not finish what it planned. It stopped because the instrument it built contradicted the plan — which is the outcome the stop condition existed to produce, and the first time this loop has spent money to be told it was wrong and then acted on it.