--- id: CB-WP-0005 kind: meta title: "Make the instruments count assertions, then fix what they expose" status: done state_hub_workstream_id: "0b95a1e3-7780-43d0-81e9-072ef7978734" --- # Purpose `research/CB-RES-0004-replay-and-kernel-coverage.md` (v2, after an adversarial review that rejected v1 with four blocking findings): > **Every coverage instrument in this project counts names. None counts > assertions.** Four of the seven defects found this pass are named in the > source and inert, so no name-based check finds them. Two are mutation-proven: AM-7's `hash-identical` clause survives folding a 100k-event log from an unrelated genesis state, and K9's `through` field survives `Snapshot::take` discarding it entirely. `evidence/CB-EV-0001` carries **`AM-7 replay | met, 2,290×`** for the first of those. [ADR-0005](../decisions/ADR-0005-assertion-coverage-and-replay.md) decides the response. This workplan executes it in four phases: teach the instruments to count assertions, correct the record they falsified, implement what they expose, then measure whether any of it worked. **Headline numbers are expected to get worse before they get better.** A pass that widened every denominator and reported the same percentages would have proved the instruments still count names. Per InnerLoop §Step 4 no target moves in the commit that measures it. ## Phase A — teach the instruments to count assertions ## Task: spec→code link over every numbered spec and every crate ```task id: CB-WP-0005-T01 status: done priority: high state_hub_task_id: "d47338c6-eb2e-4443-aad2-0b08c2d91895" ``` `tools/rule-coverage.py` hardcodes `AGGREGATE = "games/ground/src/lib.rs"` and `RULE_RE = \*\*(GR-[A-Z]+\d+)` against `GroundRules.md` alone. The reviewer's finding: a ~10-line generalization would have surfaced K10, K14 and K18 the day AM-1b shipped. Deliver: `AGGREGATE` becomes a list of source roots; rule patterns become per-spec; the link runs over **every numbered spec × every crate**. Three constraints from ADR-0005 §5, all of which must be visible in the output: 1. The kernel denominator **reports and does not feed the exit code until 2026-08-31**, after which it binds. The date lives in the tool and the tool prints the days remaining — an open-ended "gate it later" is how AM-4's targets went unratified for four workplans. 2. **Replicate the zero-rules positive control** on every new denominator. The existing arm refuses to report over zero rules, a defect it was fixed for; a kernel regex matching nothing must abort, not print `0/0 (100%)`. 3. State the limit in the output, as the GR arm already does: this counts names. It is the cheap half. **Predicted:** K10, K14, K18 reported unlinked on the first run. **Refuted if** any numbered rule in any spec is still unnamed in source after the pass and the tool does not say so. **Delivered, and the prediction held on the first run:** ```text AM-1b kernel spec->code link: 15/18 (83%) K-rules named across 10 source files NOTE: link only — K-rules are kernel invariants with no scenario mechanism; and this counts names, not assertions gate: reporting only for 31 more day(s), binds 2026-08-31 (ADR-0005 §5) unlinked (declared in the spec, named nowhere in source): K10 K14 K18 ``` `AGGREGATE` is now a list of source roots, rule patterns are per-spec, and the link runs over every numbered spec × every crate. The binding date lives in the tool and the days remaining are printed every run. The kernel figures (`k_rules`, `k_linked`, `k_unlinked`) are registered facts, so they are under `make facts-check` from the day they first exist rather than after they drift. **The self-test passed while the tool was completely broken** — a `print(` inside `say()` became `say(`, so every real `make coverage` died with `RecursionError` while `--self-test` reported all-ok, because it only ever called the quiet path. The control named the behaviour and did not assert it: this task's own thesis in miniature, and HDN instance #7. Fixed by exercising the loud path, verified by re-breaking `say()`. **A limit of `facts-check`, recorded not patched:** it is line-based, so a tagged value that prose-wraps fails. Three edits to place two tags. ## Task: M-D1-MUT — one mutation per acceptance row ```task id: CB-WP-0005-T02 status: done priority: high state_hub_task_id: "c88f2696-ca12-44df-add3-0db3d2819c08" ``` The task this workplan exists for. Name-based checks catch 3 of the 7 defects; this catches the other 4. Deliver `make mutation-check`: for each of the twelve AM-* acceptance rows, a committed fixture that **inverts the row's stated property** and an assertion that the suite goes **red**. A row whose mutation leaves the suite green is a row backed by nothing. Per ADR-0005 §1, `adapted:mutation-testing` — the denominator is acceptance rows, not source lines, because the failure mode here is not an untested branch but a headline number backed by nothing. **The escape hatch is closed in advance:** a row for which no mutation can be written is recorded `unmutatable` **with the reason** and **counts against** the metric. A row nobody can invert asserts nothing. Carries `--self-test`. Its own positive control is the one that matters: a mutation harness that fails to apply its mutation reports every row as `unmutatable` and looks thorough — this project has six harness-does- nothing instances and one of them is being fixed in T03. **Predicted: 9 of 12 rows turn red.** Deliberately beatable in both directions — **12 of 12 refutes CB-RES-0004** (the instruments were better than claimed, the finding shrinks to three absent rules, and M-D1-MUT was not worth its CI cost); **3 of 12 means the pass is under-scoped and must stop and re-plan** rather than proceed to Phase C. **Delivered. Measured: 4 of 14 rows enforced. The prediction is badly unmet.** ```text M-D1-MUT: 4/14 rows enforced PARTIAL 2 (AM-7, AM-8 — some clauses live, some inert) unmutatable 8 (no property to invert, reason stated per row) SURVIVED 0 ``` Two corrections, detailed in [CB-EV-0004](../evidence/CB-EV-0004-assertion-coverage.md) Test 2. **There are 14 rows, not 12** — AM-4 splits into a/b/c, so the 9-of-12 prediction is evaluated as ≥10 of 14. And **the first run's two `SURVIVED` verdicts were my own no-op mutations**, either of which would have published a false accusation against working code; replaced with real inversions, after which both go red. T08's question is answered on the first attempt. **The finding is larger than the workplan assumed.** 8 of 14 rows are `unmutatable` — AM-2, AM-3, AM-4c, AM-5, AM-6, AM-9, AM-10, AM-11 have no instrument at all. AM-6 is the sharpest: **nothing compares any number to 100,000 events/s**, the headline throughput claim. The problem is not three unimplemented rules; it is that **more than half the acceptance table has nothing behind it.** Harness controls: a stale find-string reports `HARNESS-BROKEN` rather than scoring the baseline as the mutant; a red baseline reports `inconclusive`; the tree is restored in a `finally` and verified. Not in `make all` — it rebuilds the workspace once per row. **Stop condition: see the note in T07 and the decision recorded there.** ## Phase B — correct the record ## Task: correct three committed verdicts and restore the fourth ```task id: CB-WP-0005-T03 status: done priority: high state_hub_task_id: "87fa2245-5f27-48af-906e-18affa41efcf" ``` Per ADR-0005 §4. Corrections are made **in place with a dated note**, not silently edited — the correction trail is the artifact. | row | committed | corrected to | |---|---|---| | AM-7 replay | `met, 2,290×` | timing clause **met**; `hash-identical` clause **withdrawn**, mutation-proven unmeasured, and the fold spans multiple game seeds so it is not a replay | | AM-10 foreign types | `met` | **withdrawn as written** — no `cb-*-api` crate, so the population is empty; re-stated as a K6 determinism row, M-D4-LEAK marked **unmeasured** | | AM-11 impl pairs | `met, narrow` | **unmet** until a shared conformance suite exists; the pair exists, the suite does not, and the metric is a bool over the suite | | AM-1b | *absent from the scoreboard* | **added**, at its real value | The last one is the point of the phase. `make coverage` prints `49/58` two lines below the `100%` the scoreboard carried, and the scoreboard kept the flattering half. Update `specs/GameKernel.md` §5 and `specs/MetricsAndScenarios.md` §1 to match, and tag the new figures under `make facts-check` so they cannot drift. **Delivered.** `evidence/CB-EV-0001` §1 now carries a dated correction note and a new **Enforced** column carrying M-D1-MUT — because a row can be *measured* and still enforce nothing, and the scoreboard had no way to say so. §1a records what changed and how each was found. **A fifth correction surfaced that ADR-0005 did not list:** AM-12 still read **$248.46**, the figure CB-WP-0002 disproved and corrected to $93.15. It had been stale in the evidence file ever since — **untagged, and therefore invisible to `facts-check`**. It is now tagged `fact:pinned_total`. A DFD instance that survived the gate built to catch DFD, because that gate only checks copies that opted in. `specs/GameKernel.md` §5 carries the AM-10 withdrawal and the AM-11 downgrade inline, so a reader of the spec cannot reach the old claim. ## Phase C — DEFERRED before starting (see T02) > **Deferred 2026-07-31 by the stop condition in T02, maintainer decision.** > M-D1-MUT measured **4 of 14** (28.6%) against a 25% floor and a 71% > prediction. Phase C is scoped to K9/K10/K11/K14/K18 — five rules — but > the measurement says **8 of 14 acceptance rows have no instrument at > all**, which Phase C does not address. Building it as written would mean > proceeding on a diagnosis the instrument had just contradicted. > > T04, T05 and T06 stay in this file, unstarted, with their analysis > intact; they move to **CB-WP-0006**, which is scoped to the finding that > was actually measured rather than the one that was predicted. > > They carry `status: cancel` — the hub's status vocabulary is > `wait|todo|progress|done|cancel`, and of those `cancel` is the accurate > one: *these* task records are superseded, and equivalents live in > CB-WP-0006 T05–T07. The work is deferred, not abandoned. ## Phase C — implement what the instruments expose (cancelled here) The three tasks below were written in full, then cancelled unstarted. Their analysis now lives where the work will happen — **[CB-WP-0006](CB-WP-0006-instrument-the-table.md) Phase B** — rather than here, so there is one copy and it is next to the code it describes. What each was, and what it became: ## Task: K9 and K11 — durable log, LogStore port, conformance suite ```task id: CB-WP-0005-T04 status: cancel state_hub_task_id: "35876b28-97ac-4379-bb71-72e74c4116d4" ``` K9's acceptance property is mutation-proven unasserted; K11 has no durable format; AM-11's conformance suite does not exist. → **CB-WP-0006 T05**. ## Task: K10 — the replay bundle and `--replay` ```task id: CB-WP-0005-T05 status: cancel state_hub_task_id: "8592b6e7-8b83-4b47-b310-90e16a35eb6a" ``` INTENT design decision 8 of 10, unimplemented; four controls required verbatim from ADR-0005 §6. → **CB-WP-0006 T06**. ## Task: K14 and K18 — implement, or amend the spec ```task id: CB-WP-0005-T06 status: cancel state_hub_task_id: "774b1c8a-71c1-4e0a-91b1-0247a3e397cf" ``` `CommitWindow` has zero non-test users; the bench never touches `ScenarioFile`. Implement, or amend with a recorded argument. → **CB-WP-0006 T07**. ## Phase D — measure it, then say what the loop should change ## Task: control loop — did the numbers get worse, then better? ```task id: CB-WP-0005-T07 status: done priority: high state_hub_task_id: "a34baa46-fbfd-473b-9e15-811e17697791" ``` Commit `evidence/CB-EV-0004-assertion-coverage.md`. Three tests, all reported: 1. **Did the denominators widen?** Name coverage before and after, per spec. A widened denominator that reports the same percentage is the result to publish — it would mean the instruments still count names. 2. **M-D1-MUT against the 9-of-12 prediction.** Report unmet if unmet; no target moves in the commit that measures it. State which rows are `unmutatable` and why. 3. **Did quality hold?** `make all` green with the new gates; and the honest question this pass raises — did widening the instruments surface *new* defects, or only the seven already known? Finding none would be evidence the sweep was as complete as CB-RES-0004 claims, which is a stronger claim than it sounds and should be stated as such. Normalize per unit of work, as CB-WP-0004 T05 established: report share of pass alongside absolute figures, and use `cb-cost --since` to window this pass against the last. **Measured — [CB-EV-0004](../evidence/CB-EV-0004-assertion-coverage.md).** 70 responses, **$16.03**. 1. **Denominators widened, and the number moved.** 1 spec → 2, 58 rules → 76, 1 source file → 10, and the new denominator came in at **83%** naming K10/K14/K18. The stated failure mode — a widened denominator reporting the same percentage — did not occur. 2. **M-D1-MUT 4 of 14 against ≥10. Unmet**, and wrong about *what* as well as how much: the diagnosis was three absent kernel rules, the measurement found eight rows with no instrument at all. 3. **Quality held**, and widening surfaced far more than the seven known defects: six further uninstrumented rows, HDN #7 (`rule-coverage`'s self-test green while the tool was broken), a stale $248.46 that `facts-check` could not see because it was untagged, and **a fifth error class**. **The clean test CB-WP-0004 was owed, now run:** the two categories whose tools removed the manual path are at **0 turns** two passes on; the two that merely offered a better option are now the *entire* mechanical cost of a pass. Absolute mechanical cost fell $51.76 → $6.13. The mechanism holds. **A regression, reported because nothing else would:** SH-3 batching is **0.0%** (67 tool calls, 67 responses) against a 20% target, and mean context 315,170 against 200,000. SessionShape has stated these since CB-WP-0003 and none of them has ever been enforced. ## Task: retrospective and InnerLoop v1.4 ```task id: CB-WP-0005-T08 status: done priority: medium state_hub_task_id: "3add7e1e-2082-44ef-9287-742c90e5c958" ``` One loop change is already earned and should be written whatever else this pass finds. InnerLoop §Step 2's "numbers" row says the reviewer must *reproduce the number independently* — satisfiable by re-running the command that prints it, which is exactly what does **not** find this class. The reviewer found AM-7 by opening a test out of curiosity, and said so. > **v1.4:** the reviewer must read the assertion behind every quoted > acceptance number and **mutate it**, not re-run the command that prints > it. This is the **second** instance of a verification step inheriting the author's blindness — CB-WP-0002's dedup blind spot was the first. Both fixes replace re-derivation with adversarial execution. Record whether that generalizes. The question to answer honestly: **CB-WP-0004 concluded that tooling recovers capacity only where it removes the manual path. Does the same test predict which gates work?** `env-test` and `task-done` removed the manual path and held. Does a mutation gate remove the manual path — or is writing a weak mutation the new `grep`? **Answered — [260731-assertion-coverage-retrospective.md](../history/260731-assertion-coverage-retrospective.md). Yes, it is the new `grep`, and it is worse, because it fails in the opposite direction.** `mutation-check`'s first run produced two `SURVIVED` verdicts and both were the author's own no-op mutations — two of fourteen rows, first attempt, in a pass whose subject was inert assertions. Writing a weak mutation is exactly as easy as writing a strong one and the harness cannot tell them apart. **The fifth error class: false accusation (FA).** HDN, TA, SSB and DFD all under-report — a real problem passes. FA over-reports: it publishes the claim that working code is broken, sends the next pass to fix something that is not broken, and is *more* credible than the truth because it arrives with a measurement attached. Thirteen instances, five classes, six passes — and the newest class is one that hardening created. **What would bind:** each mutation records its expected failure message and the harness asserts the observed failure matches. A no-op fails that immediately, because it produces no failure at all. CB-WP-0006 T08 already requires it; T09 evaluates the stronger form — mutations written by someone other than the author of the assertion. **One correction to CB-WP-0004 T06**, stated rather than left implicit: "a gate only pays if it removes the manual path" is a **predictor of whether a gate saves money, not a criterion for whether it is worth having**. `mutation-check` fails that test and still produced the most valuable findings of the pass. T06 stated it too strongly; this pass is the counter-example. **Attribution note.** T07 and T08 shipped in one commit, and CA-08 keys on the first qualified task id in the subject — so T07's bucket ($3.06 over 13 responses) covers the substantive work of both, and T08's own figure covers only its close. `make task-done` refused to close T08 at all until a commit named it, which is the "no estimates" contract working as designed rather than a defect. The lesson is cheap: **one commit, one task id**. **InnerLoop v1.4** adopted: where a claim rests on numbers, the adversarial reviewer must read the assertion behind each quoted number and **mutate it**. Re-running the command that prints a number is not verification of that number. Second verification step to inherit the author's blindness; both fixes replace re-derivation with adversarial execution.