clay-borg/decisions/ADR-0005-assertion-coverage-and-replay.md

ADR-0005: count assertions, not names; reimplement the log; correct the record

status: accepted date: 2026-07-31 decided by: maintainer (Bernd Worsch), 2026-07-31 — ratified the adversarial review's findings for adoption tier: L (structural L — creates a capability port; changes what several headline acceptance numbers mean; chaos d4=1 → no override) supersedes: nothing. Corrects three verdicts in CB-EV-0001 references: CB-RES-0004, history/260731-replay-kernel-coverage-{research,challenge,response}.md, GameKernel.md §2.4 §5, MetricsAndScenarios.md §1, ADR-0004 (AM-4 budgets)

Context

CB-RES-0004 v1 was rejected by adversarial review with four blocking findings. v2 establishes that every coverage instrument in this project counts names, and none counts assertions — and that four of the seven defects found are named in the source and inert, so no name-based check finds them. Two are mutation-proven: AM-7's hash-identical clause and K9's through field both survive mutations that should turn them red.

This ADR decides four things: what to build, what to assimilate, which budget each half is charged to, and what to do about verdicts already committed as met.

---

Decision 1 — a third coverage metric, M-D1-MUT, measured by mutation

Adopted. Name-based coverage stays; it is cheap and catches absent rules. It is joined by:

ID	Dimension	Metric	Unit	Provenance
M-D1-MUT	D1	acceptance rows whose stated property, when inverted, turns a test red	count	adapted:mutation-testing — applied to acceptance rows only, not exhaustively over source

Why adapted rather than adopted. Mutation testing as practised (cargo-mutants, Stryker, PIT) mutates source exhaustively and scores a kill rate. That is the wrong shape here on two counts: it is slow enough to leave the inner loop, and its denominator is lines, which is not what this project claims. Our claims are the twelve AM-* acceptance rows, so those are the denominator. One mutation per acceptance row, committed as a named fixture, run in CI.

The trade is stated plainly: a hand-written mutation per row cannot find defects in rows nobody thought to mutate, whereas an exhaustive tool can. We accept that, because the failure mode this project actually has is not "an untested branch" — it is "a headline number backed by nothing", and that is exactly the population M-D1-MUT covers.

Escape hatch closed in advance: a mutation that cannot be written for a row is recorded as unmutatable with the reason, and counts against the metric. A row nobody can invert is a row that asserts nothing.

Decision 2 — reimplement the log framing; do not assimilate

Reimplement. K11's format is a length prefix, a version byte, per-record framing and a tail check — Kafka and EventStore converged on the same shape decades apart, and it is 80–120 lines.

This is the deliberate exception to "own the semantics; assimilate the implementation." The rule exists so we do not reimplement hard things: SHA-256 (K7) and ChaCha (K5) cost 12 crates between them and are kept, because an audited cryptographic implementation is worth more than a scoreboard number (ADR-0004). A length prefix is not in that category. Assimilating sled or redb here would buy nothing but a dependency.

Corrected budget attribution — v1 of the survey got this wrong and the review caught it:

component	shipped?	budget	headroom
durable log (K11) — cb-events, needed by stage-3 networked sessions	yes, shipped runtime	AM-4a	3,750 lines (1.5%)
bundle writer/reader (K10) — scenario runner, on failure	no, dev-only behind scenarios	AM-4b	32,979 lines (9.4%)

The survey wanted it both ways without saying so. With the split stated, the AM-4a headroom argument applies only to the log — where we are reimplementing anyway — and is withdrawn for the bundle, where it never applied.

Decision 3 — the durable log is a capability port with a real conformance suite

Adopted. This is what earns the tier.

M-D4-SWAP is defined as "null + reference impls passing the same conformance suite". grep -rn conformance over every .rs returns one doc comment describing future work. The RNG pair — the only pair we have — is exercised by two separate, non-shared tests. So AM-11 | 1 of 1 | met, narrow is unearned on the suite clause, and has been since it was written.

Deliver a LogStore port with a shared fn conformance<S: LogStore>(…) driven by both an in-memory impl and a file-backed impl, and retro-fit the same shape to KernelRng. AM-11 then means what it says.

Decision 4 — correct the committed record

Three verdicts in evidence/CB-EV-0001 are withdrawn or restated. They are corrected in place with a dated note, not silently edited — the correction trail is the artifact.

row	committed verdict	corrected to
AM-7 replay	100k events ≤5s | 2.18 ms | met, 2,290×	split. The timing clause is met. The hash-identical clause is unmeasured — mutation-proven — and the log is folded across multiple game seeds, so it is not a replay. Verdict on that clause: withdrawn.
AM-10 foreign types	zero HashMap/HashSet | 0 | met	withdrawn as written. There is no cb-*-api crate, so the stated population is empty; what is measured is a K6 determinism lint reported under a D4 leak row. Re-stated as a K6 row, with M-D4-LEAK marked unmeasured until a canonical-interface surface exists to measure.
AM-11 impl pairs	1 of 1 (KernelRng) | met, narrow	downgraded to unmet until a shared conformance suite exists. The pair exists; the suite does not, and the metric is a bool over the suite.

AM-1b is added to the evidence scoreboard. make coverage prints 49/58 and the scoreboard omitted it while carrying AM-1 … 100% … met two lines away. Reporting the flattering half of a gate and dropping the unflattering half is the mechanism this whole pass is about.

Decision 5 — widen the name check without breaking the build

Adopted, with a binding date. AGGREGATE becomes a list, RULE_RE becomes per-spec, and the spec→code link runs over every numbered spec × every crate. K10/K14/K18 surface immediately.

rule-coverage.py returns 2 when missing is non-empty and coverage is in make all, so widening the denominator would fail the build on day one. That is forbidden by the survey's own rule — a newly widened denominator is not a regression.

So: the kernel denominator reports and does not feed the exit code, until 2026-08-31, after which it binds like the GR denominator. The date is in the tool, not in prose, and the tool prints the remaining days. An open-ended "we'll gate it later" is how AM-4's targets went unratified for four workplans.

The zero-rules positive control is replicated on the new denominator. The existing arm refuses to report a percentage over zero rules — a defect it was fixed for. A kernel regex that matches nothing must abort, not print 0/0 (100%).

Decision 6 — replay is proven by re-execution in a fresh process

Adopted. K10 says "re-executes it bit-identically", so the acceptance test is a round-trip, with four controls the review specified and this ADR adopts verbatim:

1. a committed deliberately-failing fixture scenario, and an assertion that bundles produced > 0 and equals expected failures — all 21 scenarios pass today, so a corpus sweep would verify zero bundles and print ok;
2. the comparison hash read out of the bundle, written by the first process — never recomputed in the second, which is assert_eq!(h, h) and the AM-7 defect exactly;
3. negative controls for K11: truncate the log by one byte, and corrupt the length prefix; the reader must reject both. K11's operative clause is detection, and a round-trip only ever reads well-formed input;
4. a mutated-seed negative control: change the seed in the manifest and the replay must fail to reproduce. A round-trip that cannot fail proves nothing.

---

Expected advantage, per dimension — including where it is worse

D1 ease of specification — better, decisively. "This rule is enforced" becomes a claim with a test behind it. A .cbreplay bundle turns a failure report into executable work, which is INTENT design decision 8 and is currently unimplemented.

D2 efficiency of implementation — mixed, and honestly worse in one place. No new third-party dependency; the log is ~100 lines and the port seam is small. But own source grows — the bundle requires changing the runner's data flow (Pass extended, RunOutcome::Failed restructured, an initial snapshot captured, the log plumbed out of execute), and twelve mutation fixtures are twelve pieces of test code that exist only to be broken. AM-4c (own source per 100k third-party lines) is reported, not targeted, and will move against us. That is the right trade: the alternative is a smaller codebase whose acceptance numbers mean nothing.

D3 speed of execution — neutral in the hot path, worse in CI. Bundle writes happen only on failure. Log append gains a length-prefix write against an AM-7 budget already cleared by three orders of magnitude. But mutation checks rebuild and re-run the suite once per acceptance row, and CI gets measurably slower. Accepted: the loop's own cost instrument shows gate time is not where this project's money goes.

D4 optionality — better, and this is the tier-L justification. A LogStore port with a real shared conformance suite is the seam stage 3 (networked sessions, authoritative host, replay verification) and stage 4 (package validation) both need, and it converts AM-11 from a bool over a nonexistent suite into a measured claim.

Consequences

• Three committed met verdicts are corrected; one metric (M-D1-MUT) is added; AM-11 is downgraded to unmet until its suite exists.
• Headline numbers will get worse before they get better, and that is the intended outcome. A pass that widened the denominators and reported the same percentages would have proved the instruments still count names.
• InnerLoop.md §Step 2's "numbers" row changes: reproducing a number by re-running the command that prints it satisfies the current wording and finds nothing. The reviewer must read the assertion behind the number and mutate it. This is the second instance of a verification step inheriting the author's blindness — CB-WP-0002's dedup blind spot was the first — and both fixes replace re-derivation with adversarial execution.
• What would falsify this ADR: if 12 of 12 acceptance rows turn red under mutation, the instruments were better than CB-RES-0004 claims, the finding shrinks to three absent rules, and M-D1-MUT was not worth its CI cost. That outcome is named in the survey's prediction table and is reachable.