clay-borg/workplans/CB-WP-0005-assertion-coverage.md

id	kind	title	status	state_hub_workstream_id
CB-WP-0005	meta	Make the instruments count assertions, then fix what they expose	done	0b95a1e3-7780-43d0-81e9-072ef7978734

Purpose

research/CB-RES-0004-replay-and-kernel-coverage.md (v2, after an adversarial review that rejected v1 with four blocking findings):

Every coverage instrument in this project counts names. None counts assertions. Four of the seven defects found this pass are named in the source and inert, so no name-based check finds them.

Two are mutation-proven: AM-7's hash-identical clause survives folding a 100k-event log from an unrelated genesis state, and K9's through field survives Snapshot::take discarding it entirely. evidence/CB-EV-0001 carries AM-7 replay | met, 2,290× for the first of those.

ADR-0005 decides the response. This workplan executes it in four phases: teach the instruments to count assertions, correct the record they falsified, implement what they expose, then measure whether any of it worked.

Headline numbers are expected to get worse before they get better. A pass that widened every denominator and reported the same percentages would have proved the instruments still count names. Per InnerLoop §Step 4 no target moves in the commit that measures it.

Phase A — teach the instruments to count assertions

Task: spec→code link over every numbered spec and every crate

id: CB-WP-0005-T01
status: done
priority: high
state_hub_task_id: "d47338c6-eb2e-4443-aad2-0b08c2d91895"

tools/rule-coverage.py hardcodes AGGREGATE = "games/ground/src/lib.rs" and RULE_RE = \*\*(GR-[A-Z]+\d+) against GroundRules.md alone. The reviewer's finding: a ~10-line generalization would have surfaced K10, K14 and K18 the day AM-1b shipped.

Deliver: AGGREGATE becomes a list of source roots; rule patterns become per-spec; the link runs over every numbered spec × every crate.

Three constraints from ADR-0005 §5, all of which must be visible in the output:

1. The kernel denominator reports and does not feed the exit code until 2026-08-31, after which it binds. The date lives in the tool and the tool prints the days remaining — an open-ended "gate it later" is how AM-4's targets went unratified for four workplans.
2. Replicate the zero-rules positive control on every new denominator. The existing arm refuses to report over zero rules, a defect it was fixed for; a kernel regex matching nothing must abort, not print 0/0 (100%).
3. State the limit in the output, as the GR arm already does: this counts names. It is the cheap half.

Predicted: K10, K14, K18 reported unlinked on the first run. Refuted if any numbered rule in any spec is still unnamed in source after the pass and the tool does not say so.

Delivered, and the prediction held on the first run:

AM-1b kernel spec->code link: 15/18 (83%) K-rules named across 10 source files
  NOTE: link only — K-rules are kernel invariants with no scenario
        mechanism; and this counts names, not assertions
  gate: reporting only for 31 more day(s), binds 2026-08-31 (ADR-0005 §5)
  unlinked (declared in the spec, named nowhere in source):
    K10 K14 K18

AGGREGATE is now a list of source roots, rule patterns are per-spec, and the link runs over every numbered spec × every crate. The binding date lives in the tool and the days remaining are printed every run. The kernel figures (k_rules, k_linked, k_unlinked) are registered facts, so they are under make facts-check from the day they first exist rather than after they drift.

The self-test passed while the tool was completely broken — a print( inside say() became say(, so every real make coverage died with RecursionError while --self-test reported all-ok, because it only ever called the quiet path. The control named the behaviour and did not assert it: this task's own thesis in miniature, and HDN instance #7. Fixed by exercising the loud path, verified by re-breaking say().

A limit of facts-check, recorded not patched: it is line-based, so a tagged value that prose-wraps fails. Three edits to place two tags.

Task: M-D1-MUT — one mutation per acceptance row

id: CB-WP-0005-T02
status: done
priority: high
state_hub_task_id: "c88f2696-ca12-44df-add3-0db3d2819c08"

The task this workplan exists for. Name-based checks catch 3 of the 7 defects; this catches the other 4.

Deliver make mutation-check: for each of the twelve AM-* acceptance rows, a committed fixture that inverts the row's stated property and an assertion that the suite goes red. A row whose mutation leaves the suite green is a row backed by nothing.

Per ADR-0005 §1, adapted:mutation-testing — the denominator is acceptance rows, not source lines, because the failure mode here is not an untested branch but a headline number backed by nothing.

The escape hatch is closed in advance: a row for which no mutation can be written is recorded unmutatable with the reason and counts against the metric. A row nobody can invert asserts nothing.

Carries --self-test. Its own positive control is the one that matters: a mutation harness that fails to apply its mutation reports every row as unmutatable and looks thorough — this project has six harness-does- nothing instances and one of them is being fixed in T03.

Predicted: 9 of 12 rows turn red. Deliberately beatable in both directions — 12 of 12 refutes CB-RES-0004 (the instruments were better than claimed, the finding shrinks to three absent rules, and M-D1-MUT was not worth its CI cost); 3 of 12 means the pass is under-scoped and must stop and re-plan rather than proceed to Phase C.

Delivered. Measured: 4 of 14 rows enforced. The prediction is badly unmet.

  M-D1-MUT: 4/14 rows enforced
    PARTIAL       2      (AM-7, AM-8 — some clauses live, some inert)
    unmutatable   8      (no property to invert, reason stated per row)
    SURVIVED      0

Two corrections, detailed in CB-EV-0004 Test 2. There are 14 rows, not 12 — AM-4 splits into a/b/c, so the 9-of-12 prediction is evaluated as ≥10 of 14. And the first run's two SURVIVED verdicts were my own no-op mutations, either of which would have published a false accusation against working code; replaced with real inversions, after which both go red. T08's question is answered on the first attempt.

The finding is larger than the workplan assumed. 8 of 14 rows are unmutatable — AM-2, AM-3, AM-4c, AM-5, AM-6, AM-9, AM-10, AM-11 have no instrument at all. AM-6 is the sharpest: nothing compares any number to 100,000 events/s, the headline throughput claim. The problem is not three unimplemented rules; it is that more than half the acceptance table has nothing behind it.

Harness controls: a stale find-string reports HARNESS-BROKEN rather than scoring the baseline as the mutant; a red baseline reports inconclusive; the tree is restored in a finally and verified. Not in make all — it rebuilds the workspace once per row.

Stop condition: see the note in T07 and the decision recorded there.

Phase B — correct the record

Task: correct three committed verdicts and restore the fourth

id: CB-WP-0005-T03
status: done
priority: high
state_hub_task_id: "87fa2245-5f27-48af-906e-18affa41efcf"

Per ADR-0005 §4. Corrections are made in place with a dated note, not silently edited — the correction trail is the artifact.

row	committed	corrected to
AM-7 replay	met, 2,290×	timing clause met; hash-identical clause withdrawn, mutation-proven unmeasured, and the fold spans multiple game seeds so it is not a replay
AM-10 foreign types	met	withdrawn as written — no cb-*-api crate, so the population is empty; re-stated as a K6 determinism row, M-D4-LEAK marked unmeasured
AM-11 impl pairs	met, narrow	unmet until a shared conformance suite exists; the pair exists, the suite does not, and the metric is a bool over the suite
AM-1b	absent from the scoreboard	added, at its real value

The last one is the point of the phase. make coverage prints 49/58 two lines below the 100% the scoreboard carried, and the scoreboard kept the flattering half. Update specs/GameKernel.md §5 and specs/MetricsAndScenarios.md §1 to match, and tag the new figures under make facts-check so they cannot drift.

Delivered. evidence/CB-EV-0001 §1 now carries a dated correction note and a new Enforced column carrying M-D1-MUT — because a row can be measured and still enforce nothing, and the scoreboard had no way to say so. §1a records what changed and how each was found.

A fifth correction surfaced that ADR-0005 did not list: AM-12 still read $248.46, the figure CB-WP-0002 disproved and corrected to $93.15. It had been stale in the evidence file ever since — untagged, and therefore invisible to facts-check. It is now tagged fact:pinned_total. A DFD instance that survived the gate built to catch DFD, because that gate only checks copies that opted in.

specs/GameKernel.md §5 carries the AM-10 withdrawal and the AM-11 downgrade inline, so a reader of the spec cannot reach the old claim.

Phase C — DEFERRED before starting (see T02)

Deferred 2026-07-31 by the stop condition in T02, maintainer decision. M-D1-MUT measured 4 of 14 (28.6%) against a 25% floor and a 71% prediction. Phase C is scoped to K9/K10/K11/K14/K18 — five rules — but the measurement says 8 of 14 acceptance rows have no instrument at all, which Phase C does not address. Building it as written would mean proceeding on a diagnosis the instrument had just contradicted.

T04, T05 and T06 stay in this file, unstarted, with their analysis intact; they move to CB-WP-0006, which is scoped to the finding that was actually measured rather than the one that was predicted.

They carry status: cancel — the hub's status vocabulary is wait|todo|progress|done|cancel, and of those cancel is the accurate one: these task records are superseded, and equivalents live in CB-WP-0006 T05–T07. The work is deferred, not abandoned.

Phase C — implement what the instruments expose (cancelled here)

The three tasks below were written in full, then cancelled unstarted. Their analysis now lives where the work will happen — CB-WP-0006 Phase B — rather than here, so there is one copy and it is next to the code it describes. What each was, and what it became:

Task: K9 and K11 — durable log, LogStore port, conformance suite

id: CB-WP-0005-T04
status: cancel
state_hub_task_id: "35876b28-97ac-4379-bb71-72e74c4116d4"

K9's acceptance property is mutation-proven unasserted; K11 has no durable format; AM-11's conformance suite does not exist. → CB-WP-0006 T05.

Task: K10 — the replay bundle and --replay

id: CB-WP-0005-T05
status: cancel
state_hub_task_id: "8592b6e7-8b83-4b47-b310-90e16a35eb6a"

INTENT design decision 8 of 10, unimplemented; four controls required verbatim from ADR-0005 §6. → CB-WP-0006 T06.

Task: K14 and K18 — implement, or amend the spec

id: CB-WP-0005-T06
status: cancel
state_hub_task_id: "774b1c8a-71c1-4e0a-91b1-0247a3e397cf"

CommitWindow has zero non-test users; the bench never touches ScenarioFile. Implement, or amend with a recorded argument. → CB-WP-0006 T07.

Phase D — measure it, then say what the loop should change

Task: control loop — did the numbers get worse, then better?

id: CB-WP-0005-T07
status: done
priority: high
state_hub_task_id: "a34baa46-fbfd-473b-9e15-811e17697791"

Commit evidence/CB-EV-0004-assertion-coverage.md. Three tests, all reported:

1. Did the denominators widen? Name coverage before and after, per spec. A widened denominator that reports the same percentage is the result to publish — it would mean the instruments still count names.
2. M-D1-MUT against the 9-of-12 prediction. Report unmet if unmet; no target moves in the commit that measures it. State which rows are unmutatable and why.
3. Did quality hold? make all green with the new gates; and the honest question this pass raises — did widening the instruments surface new defects, or only the seven already known? Finding none would be evidence the sweep was as complete as CB-RES-0004 claims, which is a stronger claim than it sounds and should be stated as such.

Normalize per unit of work, as CB-WP-0004 T05 established: report share of pass alongside absolute figures, and use cb-cost --since to window this pass against the last.

Measured — CB-EV-0004. 70 responses, $16.03.

1. Denominators widened, and the number moved. 1 spec → 2, 58 rules → 76, 1 source file → 10, and the new denominator came in at 83% naming K10/K14/K18. The stated failure mode — a widened denominator reporting the same percentage — did not occur.
2. M-D1-MUT 4 of 14 against ≥10. Unmet, and wrong about what as well as how much: the diagnosis was three absent kernel rules, the measurement found eight rows with no instrument at all.
3. Quality held, and widening surfaced far more than the seven known defects: six further uninstrumented rows, HDN #7 (rule-coverage's self-test green while the tool was broken), a stale $248.46 that facts-check could not see because it was untagged, and a fifth error class.

The clean test CB-WP-0004 was owed, now run: the two categories whose tools removed the manual path are at 0 turns two passes on; the two that merely offered a better option are now the entire mechanical cost of a pass. Absolute mechanical cost fell $51.76 → $6.13. The mechanism holds.

A regression, reported because nothing else would: SH-3 batching is 0.0% (67 tool calls, 67 responses) against a 20% target, and mean context 315,170 against 200,000. SessionShape has stated these since CB-WP-0003 and none of them has ever been enforced.

Task: retrospective and InnerLoop v1.4

id: CB-WP-0005-T08
status: done
priority: medium
state_hub_task_id: "3add7e1e-2082-44ef-9287-742c90e5c958"

One loop change is already earned and should be written whatever else this pass finds. InnerLoop §Step 2's "numbers" row says the reviewer must reproduce the number independently — satisfiable by re-running the command that prints it, which is exactly what does not find this class. The reviewer found AM-7 by opening a test out of curiosity, and said so.

v1.4: the reviewer must read the assertion behind every quoted acceptance number and mutate it, not re-run the command that prints it.

This is the second instance of a verification step inheriting the author's blindness — CB-WP-0002's dedup blind spot was the first. Both fixes replace re-derivation with adversarial execution. Record whether that generalizes.

The question to answer honestly: CB-WP-0004 concluded that tooling recovers capacity only where it removes the manual path. Does the same test predict which gates work? env-test and task-done removed the manual path and held. Does a mutation gate remove the manual path — or is writing a weak mutation the new grep?

Answered — 260731-assertion-coverage-retrospective.md. Yes, it is the new grep, and it is worse, because it fails in the opposite direction.

mutation-check's first run produced two SURVIVED verdicts and both were the author's own no-op mutations — two of fourteen rows, first attempt, in a pass whose subject was inert assertions. Writing a weak mutation is exactly as easy as writing a strong one and the harness cannot tell them apart.

The fifth error class: false accusation (FA). HDN, TA, SSB and DFD all under-report — a real problem passes. FA over-reports: it publishes the claim that working code is broken, sends the next pass to fix something that is not broken, and is more credible than the truth because it arrives with a measurement attached. Thirteen instances, five classes, six passes — and the newest class is one that hardening created.

What would bind: each mutation records its expected failure message and the harness asserts the observed failure matches. A no-op fails that immediately, because it produces no failure at all. CB-WP-0006 T08 already requires it; T09 evaluates the stronger form — mutations written by someone other than the author of the assertion.

One correction to CB-WP-0004 T06, stated rather than left implicit: "a gate only pays if it removes the manual path" is a predictor of whether a gate saves money, not a criterion for whether it is worth having. mutation-check fails that test and still produced the most valuable findings of the pass. T06 stated it too strongly; this pass is the counter-example.

Attribution note. T07 and T08 shipped in one commit, and CA-08 keys on the first qualified task id in the subject — so T07's bucket ($3.06 over 13 responses) covers the substantive work of both, and T08's own figure covers only its close. make task-done refused to close T08 at all until a commit named it, which is the "no estimates" contract working as designed rather than a defect. The lesson is cheap: one commit, one task id.

InnerLoop v1.4 adopted: where a claim rests on numbers, the adversarial reviewer must read the assertion behind each quoted number and mutate it. Re-running the command that prints a number is not verification of that number. Second verification step to inherit the author's blindness; both fixes replace re-derivation with adversarial execution.