clay-borg/research/CB-RES-0004-replay-and-kernel-coverage.md

# CB-RES-0004: the instruments count names, not assertions

capability: meta.coverage.assertions + kernel.replay
status: approved (v2, after adversarial review — v1 was rejected with four
blocking findings)
tier: L (structural L — creates a new capability port (durable log
storage); changes what several headline acceptance numbers mean; chaos
d4=1 → no override)
instrument: extended `make coverage` (spec→code over every numbered spec)
plus a new `make mutation-check` and `make replay-test`
baseline: cited — boardgame.io, Rune, rr/Pernosco, Playwright trace,
event-sourcing log framing

Survey opened 2026-07-31. **v1 was rejected**; the review trail is
`history/260731-replay-kernel-coverage-{research,challenge,response}.md`.
The finding below is the reviewer's as much as mine.

---

## 1. The finding

> **Every coverage instrument in this project counts names. None counts
> assertions.** M-D1-COV counts `covers:` tags in scenario YAML. M-D1-LNK
> counts rule-ID strings in one source file. Both answer *"is this rule
> mentioned?"*; neither answers *"does anything fail if this rule is
> violated?"*

Seven defects were found this pass. Four of them are **named in the source
and unasserted** — invisible to any name-based check, including the
cheaper one the reviewer proposed:

| defect | named in code? | a name-based check finds it? |
|---|---|---|
| K10 replay bundle — no writer, reader, or `--replay` | no | **yes** |
| K14 commit window — `CommitWindow` has zero non-test users | no | **yes** |
| K18 benches from scenarios — bench hardcodes commands | no | **yes** |
| K9 snapshot `+ the EventId it includes` | yes | no |
| K11 durable format, truncated-tail detection | yes | no |
| AM-7 `hash-identical` | yes | no |
| AM-11 `passing the same conformance suite` | yes | no |

This subsumes the project's oldest error class. All five prior
harness-does-nothing instances have the same shape as AM-7 and K9 today:
**the name is there, the assertion is not.**

### 1.1 The two that are mutation-proven

**AM-7's `hash-identical` clause detects nothing.**
`games/ground/src/lib.rs` computes `state_hash_hex(&state)` and passes it
to a `println!`. The only assertion is on elapsed time. Mutating the
fold's genesis from `fresh(42)` to `fresh(999)` — replaying a 100k-event
log into an unrelated starting state — leaves the test green:

```text
test replay_probe::replay_100k_events_is_linear_and_fast ... ok
```

The hash could not be asserted as written anyway: the log is built across
games seeded 42, 43, 44… (`source = fresh(42 + games)`) and folded into a
state from `fresh(42)`. It is a fold-rate microbenchmark over a
semantically invalid stream, reported in `evidence/CB-EV-0001` as
**`AM-7 replay | met, 2,290×`**. That verdict is unearned on the
hash clause and needs a correction note. **HDN instance #6.**

**K9's `through` field is unverified.** The whole evidence for K9 is one
test round-tripping a `BTreeMap<String, u8>` with `EventSeq(17)` as a
literal — no game aggregate, no events applied, no from-genesis
comparison. `Snapshot` has no callers outside that test. Mutating
`Snapshot::take` to discard its `EventSeq` and store `EventSeq(0)` leaves
the test green.

On the acceptance criterion, **GameKernel §2.4 is zero of three, not two
of three.**

### 1.2 The three that are named nowhere

```text
spec:  K1..K18       code:  K1..K9 K11 K12 K13 K15 K16 K17
diff:  K10, K14, K18
```

- **K10** — no `.cbreplay` writer, reader, or `replays/` directory; the
  only `fs::` call in the workspace is `read_to_string` for scenario YAML.
  `cb-sim` has no flag parsing at all, so `--replay` has no place to go.
  `MetricsAndScenarios` §2 promises a failing run writes a bundle. It does
  not. INTENT names this design decision **8 of 10**.
- **K14** — "one commit window per Select". `CommitWindow` has zero
  non-test users and `games/ground` does not import it; GROUND collects
  selections in its own aggregate. A dead abstraction certified by a test
  that exercises only itself.
- **K18** — "Criterion benches driving the same scenario format". The
  bench hardcodes commands in Rust and never touches `ScenarioFile`.
  `benchmarks/` contains only `baselines/model-prices.toml`.

### 1.3 The scoped-denominator sweep

v1 claimed this problem was general and swept in one place. Swept properly,
in descending order of severity:

**AM-10 is vacuous — the population is empty.** The row reads
*M-D4-LEAK: foreign types in `cb-*-api`-visible signatures | 0*. There is
no `cb-*-api` crate. What is actually measured is a `clippy.toml` deny of
`HashMap`/`HashSet` whose stated reason cites **K6 (determinism)**,
reported under a **D4 leak** row. A determinism lint standing in for a
leak metric, over a crate that does not exist.

**AM-1's denominator is one spec, and the metric says otherwise.**
M-D1-COV is defined as *"numbered spec rules covered by ≥1 passing
scenario"*, and `covers` is documented as "numbered rules from the
capability spec" with a non-GR example. `RULE_RE` matches `GR-` only,
against `GroundRules.md` only. So `58/58 (100%)` is not merely narrowly
scoped — the instrument is **non-conformant with its own metric
definition**, and 100% is a wrong answer to the question M-D1-COV asks.

**AM-1b is measured and dropped from the scoreboard.** `make coverage`
prints `49/58` and `MetricsAndScenarios` records it **Unmet**;
`grep -c 'AM-1b' evidence/CB-EV-0001` returns **0**. The artifact carrying
the headline reports the flattering half of the gate and omits the
unflattering half the same command prints two lines below it.

**AM-11's suite does not exist.** M-D4-SWAP requires impls "passing **the
same conformance suite**". `grep -rn conformance` over all `.rs` returns
one doc comment describing future work. The RNG pair is exercised by two
separate, non-shared tests. `met, narrow` is unearned on the suite clause.

**AM-2, AM-3, AM-5, AM-9 are unreported** and were unmentioned by v1.

The history file's stated weakness — *"the evidence for this is n=1"* — is
retired. It is n≥5 in this repo, found without leaving the tree.

---

## 2. Baseline — what the state of the art does

Cited, not locally measured; per MetricsAndScenarios §3 this caps
comparison verdicts at `parity`.

| system | durable log | exportable failure bundle | machine re-executable |
|---|---|---|---|
| **boardgame.io** | in-memory, optional server persistence | debug panel, in-session time travel | no |
| **Rune** | rollback ring buffer, transient | no | no |
| **rr / Pernosco** | full process recording | yes — the gold standard | **yes**, bit-exact |
| **Playwright trace** | per-run trace zip | **yes**, one file | partly (UI, not logic) |
| **Kafka / EventStore** | append-only, length-prefixed, versioned, checksummed | n/a | yes, by re-consumption |

Two lessons:

1. **The valuable property is machine re-execution, not inspection.**
   boardgame.io's debug panel is more visible than a `.cbreplay` file and
   far less useful to an agent, because a human must interpret it. rr and
   Playwright are the precedents worth taking.
2. **The log format is solved engineering.** Length prefix, version byte,
   per-record framing, tail detection: Kafka and EventStore converged on
   the same shape decades apart. It is ~80–120 lines.

**Provenance.** Log framing: `adapted:event-sourcing-log-framing` —
length-prefixed versioned records with tail detection, dropping the
distributed concerns (partitions, offsets, compaction) a single-game log
does not have. Bundle: `adapted:playwright-trace` — one artifact per
failed run containing everything needed to reproduce, adapted so the
consumer is an agent, not a human viewer. Assertion coverage:
`adapted:mutation-testing` — mutate the property, expect a red test;
applied only to acceptance rows rather than exhaustively.

---

## 3. The four dimensions

**D1 — ease of specification.** Strongly positive. A `.cbreplay` bundle
turns "the scenario failed" into an artifact with a stated contract. More
importantly, mutation coverage makes "this rule is enforced" a *claim with
a test* rather than a tag.

**D2 — efficiency of implementation.** The log format adds no dependency
(serde is already shipped). **The bundle writer is not "a directory of
four files"** — v1 said that and it was wrong. `scenario.rs` creates an
`EventLog`, appends to it, and never reads it; it is dropped at the end of
`execute`. `Pass` carries the *end* state, not an initial snapshot, and
failures are a formatted `String`, not structured expected-vs-actual. The
bundle requires changing the runner's data flow: plumb the log out, capture
an initial snapshot, restructure `RunOutcome::Failed`.

**D3 — speed of execution.** Bundle writes happen only on failure, so the
hot path is untouched. Log append gains a length-prefix write. Mutation
checks run in CI, not in the loop.

**D4 — optionality.** Where the pass earns its L, on a corrected premise.
v1 argued AM-11 "claims" a storage port; it does not — that is the target
column, and the evidence honestly reports `1 of 1, met, narrow`. The real
argument is stronger: **M-D4-SWAP is a bool over a conformance suite that
does not exist.** Building a durable-log port with a genuine shared suite
discharges AM-11 properly and creates the seam stage 3 (networked
sessions) and stage 4 (game-creation framework) both need.

---

## 4. Options

**A — implement K10/K11, leave the instruments alone.** Rejected. Fixes
what was noticed; leaves the mechanism that hid it.

**B — extend coverage to K-rules, defer the rest.** Rejected as a final
state. Adopted as *ordering*.

**C — adopt an embedded store (sled, redb).** Rejected — and v1 rejected
it against the wrong budget. The **durable log (K11) is shipped-runtime**,
so AM-4a applies with 1.5% headroom; the **bundle writer (K10) is dev-only**
behind the `scenarios` feature, so AM-4b applies with 9.4%. v1 wanted it
both ways without saying so. With the split stated, C is rejected on
simplicity for the log — a length prefix is not a hard implementation, and
"assimilate the implementation" is for hard ones — and the headroom
argument is dropped for the bundle, where it never applied.

**E — extend AM-1b's spec→code link to every numbered spec and every
crate.** ~10 lines against an instrument that already exists. Catches
K10, K14, K18 immediately. **Adopted as step one — and demonstrably
insufficient**, since it catches 3 of the 7 defects and none of the four
mutation-class ones. `AGGREGATE` must become a list.

**F — E, plus mutation checks on acceptance rows, plus K10/K11.**
**Recommended.** The name-based check is cheap and catches the absent; the
mutation check is the only thing that catches the present-but-inert; the
capability work discharges the rules both instruments expose.

---

## 5. What this pass must not do

- **Do not tag K-rules as covered to make a number go up.** The overclaim
  CB-WP-0001 committed once and M-D1-LNK exists to catch. A K-rule is
  covered when breaking it breaks a test.
- **Do not let the widened coverage gate fail the build on day one.**
  `rule-coverage.py` returns `2` when `missing` is non-empty and
  `coverage` is in `all`, so 18 uncovered K-rules fail `make all`
  immediately. The kernel denominator must report outside the exit code
  until the pass closes the gap, with a stated date for when it binds.
- **Replicate the zero-rules positive control on the new denominator.**
  The existing arm refuses to report over zero rules — a defect it was
  fixed for. A kernel regex matching nothing must abort, not print
  `0/0 (100%)`.
- **`make replay-test` must not be able to pass vacuously.** It reports
  `ok` under at least four silent failures: no scenario fails so zero
  bundles are round-tripped; the writer emits nothing and identical error
  strings satisfy "same failure"; the replayed hash is compared to one
  recomputed in the same process (`assert_eq!(h, h)` — the AM-7 defect
  exactly); the truncation path is never exercised. **Required controls:**
  a committed deliberately-failing fixture plus `bundles > 0`; the
  comparison hash read *out of the bundle*, written by the first process;
  truncate-by-one-byte and corrupt-length-prefix negative controls; and a
  mutated-seed negative control so the round-trip is capable of failing.

---

## 6. Prediction

Stated as a mechanism per CB-WP-0004 T06, and with the refuting outcome
named — v1's prediction was rejected for being unfalsifiable *and* already
determined (any tag-based kernel count is **0 of 18** today, computable
before writing a line).

| claim | mechanism after the fix | refuted if |
|---|---|---|
| absent rules become visible | spec→code runs over every numbered spec × every crate; no manual path | any numbered rule in any spec is unnamed in source after the pass |
| **present-but-inert rules become visible** | mutation: each acceptance row's property is inverted and the suite must go red | **fewer than 9 of the 12 acceptance rows have a mutation that turns a test red** |
| replay is real | acceptance test re-executes a bundle in a **fresh process** against a hash read from the bundle | the replayed hash is recomputed in-process, or the round-trip cannot be made to fail |
| K9/K11 are enforced | snapshot-at-N + events N+1..M ≡ genesis fold on `GroundState`; truncation rejected | either passes when mutated |

The middle row is the one that carries information. **9 of 12 is a
guess, and it is meant to be beatable in both directions** — if 12 of 12
mutate red, the instruments were better than this survey claims and the
finding shrinks to the three absent rules; if 3 of 12 do, the problem is
larger than the pass is scoped for and the pass should stop and re-plan.