clay-borg/decisions/ADR-0005-assertion-coverage-and-replay.md

# ADR-0005: count assertions, not names; reimplement the log; correct the record

status: accepted
date: 2026-07-31
decided by: maintainer (Bernd Worsch), 2026-07-31 — ratified the adversarial
review's findings for adoption
tier: L (structural L — creates a capability port; changes what several
headline acceptance numbers mean; chaos d4=1 → no override)
supersedes: nothing. **Corrects** three verdicts in
[CB-EV-0001](../evidence/CB-EV-0001-game-kernel.md)
references: [CB-RES-0004](../research/CB-RES-0004-replay-and-kernel-coverage.md),
`history/260731-replay-kernel-coverage-{research,challenge,response}.md`,
[GameKernel.md](../specs/GameKernel.md) §2.4 §5,
[MetricsAndScenarios.md](../specs/MetricsAndScenarios.md) §1,
[ADR-0004](ADR-0004-am4-ratification.md) (AM-4 budgets)

## Context

CB-RES-0004 v1 was rejected by adversarial review with four blocking
findings. v2 establishes that **every coverage instrument in this project
counts names, and none counts assertions** — and that four of the seven
defects found are named in the source and inert, so no name-based check
finds them. Two are mutation-proven: AM-7's `hash-identical` clause and
K9's `through` field both survive mutations that should turn them red.

This ADR decides four things: what to build, what to assimilate, which
budget each half is charged to, and what to do about verdicts already
committed as `met`.

---

## Decision 1 — a third coverage metric, M-D1-MUT, measured by mutation

**Adopted.** Name-based coverage stays; it is cheap and catches absent
rules. It is joined by:

| ID | Dimension | Metric | Unit | Provenance |
|---|---|---|---|---|
| **M-D1-MUT** | D1 | acceptance rows whose stated property, when inverted, turns a test red | count | `adapted:mutation-testing` — applied to **acceptance rows only**, not exhaustively over source |

**Why adapted rather than adopted.** Mutation testing as practised
(`cargo-mutants`, Stryker, PIT) mutates source exhaustively and scores a
kill rate. That is the wrong shape here on two counts: it is slow enough
to leave the inner loop, and its denominator is *lines*, which is not what
this project claims. Our claims are the twelve AM-* acceptance rows, so
those are the denominator. **One mutation per acceptance row, committed as
a named fixture, run in CI.**

The trade is stated plainly: a hand-written mutation per row cannot find
defects in rows nobody thought to mutate, whereas an exhaustive tool can.
We accept that, because the failure mode this project actually has is not
"an untested branch" — it is "a headline number backed by nothing", and
that is exactly the population M-D1-MUT covers.

**Escape hatch closed in advance:** a mutation that cannot be written for
a row is recorded as `unmutatable` **with the reason**, and counts against
the metric. A row nobody can invert is a row that asserts nothing.

## Decision 2 — reimplement the log framing; do not assimilate

**Reimplement.** K11's format is a length prefix, a version byte,
per-record framing and a tail check — Kafka and EventStore converged on
the same shape decades apart, and it is 80–120 lines.

This is the deliberate exception to *"own the semantics; assimilate the
implementation."* The rule exists so we do not reimplement **hard**
things: SHA-256 (K7) and ChaCha (K5) cost 12 crates between them and are
kept, because an audited cryptographic implementation is worth more than a
scoreboard number (ADR-0004). A length prefix is not in that category.
Assimilating sled or redb here would buy nothing but a dependency.

**Corrected budget attribution** — v1 of the survey got this wrong and the
review caught it:

| component | shipped? | budget | headroom |
|---|---|---|---|
| **durable log (K11)** — `cb-events`, needed by stage-3 networked sessions | **yes**, shipped runtime | **AM-4a** | 3,750 lines (1.5%) |
| **bundle writer/reader (K10)** — scenario runner, on failure | **no**, dev-only behind `scenarios` | **AM-4b** | 32,979 lines (9.4%) |

The survey wanted it both ways without saying so. With the split stated,
the AM-4a headroom argument applies only to the log — where we are
reimplementing anyway — and is **withdrawn** for the bundle, where it never
applied.

## Decision 3 — the durable log is a capability port with a real conformance suite

**Adopted.** This is what earns the tier.

M-D4-SWAP is defined as "null + reference impls passing **the same
conformance suite**". `grep -rn conformance` over every `.rs` returns one
doc comment describing future work. The RNG pair — the only pair we have —
is exercised by two *separate, non-shared* tests. So `AM-11 | 1 of 1 | met,
narrow` is unearned on the suite clause, and has been since it was written.

Deliver a `LogStore` port with a shared `fn conformance<S: LogStore>(…)`
driven by **both** an in-memory impl and a file-backed impl, and
retro-fit the same shape to `KernelRng`. AM-11 then means what it says.

## Decision 4 — correct the committed record

Three verdicts in `evidence/CB-EV-0001` are withdrawn or restated. They are
**corrected in place with a dated note**, not silently edited — the
correction trail is the artifact.

| row | committed verdict | corrected to |
|---|---|---|
| **AM-7 replay** | `100k events ≤5s \| 2.18 ms \| met, 2,290×` | **split.** The timing clause is met. The `hash-identical` clause is **unmeasured** — mutation-proven — and the log is folded across multiple game seeds, so it is not a replay. Verdict on that clause: **withdrawn**. |
| **AM-10 foreign types** | `zero HashMap/HashSet \| 0 \| met` | **withdrawn as written.** There is no `cb-*-api` crate, so the stated population is empty; what is measured is a K6 determinism lint reported under a D4 leak row. Re-stated as a K6 row, with M-D4-LEAK marked **unmeasured** until a canonical-interface surface exists to measure. |
| **AM-11 impl pairs** | `1 of 1 (KernelRng) \| met, narrow` | **downgraded to unmet** until a shared conformance suite exists. The pair exists; the suite does not, and the metric is a bool over the suite. |

**AM-1b is added to the evidence scoreboard.** `make coverage` prints
`49/58` and the scoreboard omitted it while carrying `AM-1 … 100% … met`
two lines away. Reporting the flattering half of a gate and dropping the
unflattering half is the mechanism this whole pass is about.

## Decision 5 — widen the name check without breaking the build

**Adopted, with a binding date.** `AGGREGATE` becomes a list, `RULE_RE`
becomes per-spec, and the spec→code link runs over **every numbered spec ×
every crate**. K10/K14/K18 surface immediately.

`rule-coverage.py` returns `2` when `missing` is non-empty and `coverage`
is in `make all`, so widening the denominator would fail the build on day
one. That is forbidden by the survey's own rule — a newly widened
denominator is not a regression.

So: the kernel denominator **reports** and does not feed the exit code,
**until 2026-08-31**, after which it binds like the GR denominator. The
date is in the tool, not in prose, and the tool prints the remaining days.
An open-ended "we'll gate it later" is how AM-4's targets went unratified
for four workplans.

**The zero-rules positive control is replicated on the new denominator.**
The existing arm refuses to report a percentage over zero rules — a defect
it was fixed for. A kernel regex that matches nothing must abort, not
print `0/0 (100%)`.

## Decision 6 — replay is proven by re-execution in a fresh process

**Adopted.** K10 says "re-executes it bit-identically", so the acceptance
test is a round-trip, with four controls the review specified and this ADR
adopts verbatim:

1. a **committed deliberately-failing fixture** scenario, and an assertion
   that bundles produced `> 0` and equals expected failures — all 21
   scenarios pass today, so a corpus sweep would verify zero bundles and
   print `ok`;
2. the comparison hash **read out of the bundle**, written by the first
   process — never recomputed in the second, which is `assert_eq!(h, h)`
   and the AM-7 defect exactly;
3. **negative controls for K11**: truncate the log by one byte, and
   corrupt the length prefix; the reader must reject both. K11's operative
   clause is *detection*, and a round-trip only ever reads well-formed
   input;
4. a **mutated-seed negative control**: change the seed in the manifest and
   the replay must fail to reproduce. A round-trip that cannot fail proves
   nothing.

---

## Expected advantage, per dimension — including where it is worse

**D1 ease of specification — better, decisively.** "This rule is enforced"
becomes a claim with a test behind it. A `.cbreplay` bundle turns a failure
report into executable work, which is INTENT design decision 8 and is
currently unimplemented.

**D2 efficiency of implementation — mixed, and honestly worse in one
place.** No new third-party dependency; the log is ~100 lines and the port
seam is small. But **own source grows** — the bundle requires changing the
runner's data flow (`Pass` extended, `RunOutcome::Failed` restructured, an
initial snapshot captured, the log plumbed out of `execute`), and twelve
mutation fixtures are twelve pieces of test code that exist only to be
broken. AM-4c (own source per 100k third-party lines) is **reported, not
targeted**, and will move against us. That is the right trade: the
alternative is a smaller codebase whose acceptance numbers mean nothing.

**D3 speed of execution — neutral in the hot path, worse in CI.** Bundle
writes happen only on failure. Log append gains a length-prefix write
against an AM-7 budget already cleared by three orders of magnitude. But
mutation checks rebuild and re-run the suite once per acceptance row, and
CI gets measurably slower. Accepted: the loop's own cost instrument shows
gate time is not where this project's money goes.

**D4 optionality — better, and this is the tier-L justification.** A
`LogStore` port with a real shared conformance suite is the seam stage 3
(networked sessions, authoritative host, replay verification) and stage 4
(package validation) both need, and it converts AM-11 from a bool over a
nonexistent suite into a measured claim.

## Consequences

- Three committed `met` verdicts are corrected; one metric (M-D1-MUT) is
  added; AM-11 is downgraded to unmet until its suite exists.
- **Headline numbers will get worse before they get better**, and that is
  the intended outcome. A pass that widened the denominators and reported
  the same percentages would have proved the instruments still count names.
- `InnerLoop.md` §Step 2's "numbers" row changes: reproducing a number by
  re-running the command that prints it satisfies the current wording and
  finds nothing. The reviewer must **read the assertion behind the number
  and mutate it**. This is the second instance of a verification step
  inheriting the author's blindness — CB-WP-0002's dedup blind spot was the
  first — and both fixes replace re-derivation with adversarial execution.
- **What would falsify this ADR:** if 12 of 12 acceptance rows turn red
  under mutation, the instruments were better than CB-RES-0004 claims, the
  finding shrinks to three absent rules, and M-D1-MUT was not worth its CI
  cost. That outcome is named in the survey's prediction table and is
  reachable.