clay-borg/history/260731-replay-kernel-coverage-challenge.md

# Challenge — CB-RES-0004 (replay bundles, durable log, and the rules nobody measures)

Reviewer: adversarial pass, InnerLoop §Step 2, one round. Separate agent
session, given the survey **and** the harness and the evidence file per
the "numbers" row of the review-target table.
Tree reviewed: `59b73c3`.
Verdict: **not approvable as written.** Four BLOCKING, six MATERIAL, two
MINOR.

> The survey is right that something is unmeasured. It is wrong about
> what, wrong about how much, and its own baseline contains an instance of
> the exact error class it is diagnosing. The headline finding is
> under-scoped by at least three rules, and the instrument it proposes is
> heavier and less effective than a one-line extension of an instrument
> that already exists in this repo.

---

## 1. BLOCKING — the AM-7 replay row does not measure the half of AM-7 the survey defends as "honest". Instance #6 of the harness-does-nothing class is live in the tree right now.

The survey §1.3 calls the AM-7 row "honest but easily misread". It is not
honest. AM-7's target is "replay of 100k events ≤ 5 s, **hash-identical**".

`games/ground/src/lib.rs:2241-2256`:

```rust
let hash = state_hash_hex(&state);
let elapsed = start.elapsed();
println!("replay {} events in {:?} ({:.0} events/s), hash {}", ...);
assert!(elapsed.as_secs_f64() < 5.0, "AM-7: 100k replay under 5s");
```

`hash` reaches exactly one place: a `println!`. **There is no assertion on
it.** The Criterion bench is the same shape.

Worse, the hash *could not* be asserted as written. The log is built
across multiple games (`lib.rs:2231-2235`):

```rust
while log.len() < target {
    if source.outcome.is_some() { games += 1; source = fresh(42 + games); }
```

so the "replay" folds events from games seeded 42, 43, 44… into a single
state built from `fresh(42)`. It is not a replay of anything. It is a
fold-rate microbenchmark over a semantically invalid event stream,
reported as `AM-7 replay | met, 2,290×`.

**Must change:** §1.3 rewritten — AM-7's `hash-identical` clause is
unmeasured, the fold is over a multi-game log, and the evidence verdict is
unearned on that clause. This is a correction to `evidence/CB-EV-0001`,
and it puts AM-7 in the pass scope.

## 2. BLOCKING — the survey grants K9 as implemented. K9's acceptance property is asserted nowhere.

K9 requires "`snapshot + remaining events → state` must be hash-identical
to a from-genesis fold". The entire evidence is one unit test
(`cb-events/src/lib.rs:158-167`) that round-trips a
`BTreeMap<String, u8>` with `EventSeq(17)` as a literal. No game
aggregate, no events applied, no from-genesis comparison. And `Snapshot`
has no callers outside its own test.

The survey established which rules are implemented by reading `pub fn`
signatures — presence-of-API, which is exactly the tag-counting it
condemns in §5. **On the acceptance criterion §2.4 is not "two of three",
it is zero of three.**

## 3. BLOCKING — the "below 100%" prediction is arithmetic, not a prediction, and §6 is constructed so no outcome can refute it.

Below 100% → thesis confirmed. Near 100% → "the instrument measures tags",
thesis confirmed. No stated result counts against the survey — the
unfalsifiable framing the survey claims to guard against, dressed in the
vocabulary of falsifiability.

It is also not a prediction: every `covers:` id in the corpus is `GR-*`
(58 ids, zero `K*`), and `ls scenarios/` returns one directory. Any
tag-counting extension **must** report **0 of 18 (0%)** on first run.
Computable today, before writing a line.

**Must change:** a prediction that can fail, with the refuting outcome
named. Delete the §6 escape clause.

## 4. BLOCKING — K10 is not the only unimplemented kernel rule. K14 and K18 are too, and the one-line check that finds all three already exists in this repo.

```
spec:  K1..K18
code:  K1..K9 K11 K12 K13 K15 K16 K17
diff:  K10, K14, K18
```

**K14** ("one commit window per Select") — `CommitWindow` has zero
non-test users and `games/ground` does not import it. GROUND collects
selections in its own aggregate. A dead abstraction certified by a test
that exercises only itself.

**K18** ("Criterion benches driving the same scenario format") — the
bench hardcodes commands in Rust and never touches `ScenarioFile`. And
`ls -R benchmarks` contains only `baselines/model-prices.toml`.

**Why this is BLOCKING.** The check that found these is a two-line
generalization of **M-D1-LNK / AM-1b**, which this repo already runs every
`make coverage`. Pointing `parse_code_ids` at `specs/GameKernel.md` ×
`crates/**` would have surfaced all three the day AM-1b shipped. The
survey diagnoses a gap a cheap existing instrument covers and prescribes a
new expensive capability as the remedy — and, having chosen the expensive
framing, found 1 of the 3 gaps the cheap check finds in one line.

**Must change:** rescope to K10/K14/K18 (K9, K11 partial). Add **Option E:
extend AM-1b's spec→code link to every numbered spec and every crate**,
and argue why the tier-L work is warranted *on top of* it.

## 5. MATERIAL — K-rules cannot be covered by the mechanism that covers GR-rules. "A parameterization, not a rewrite" is false.

The M-D1-COV numerator comes from `covers:` lists consumed by a runner
requiring `ScenarioGame: Aggregate + Serialize + DeserializeOwned`, with
dispatch by game prefix and `ground` the only arm. There is no kernel
aggregate, setup preset, or command vocabulary. `scenarios/kernel/*.yaml`
would be a tag in a directory the runner cannot dispatch — the inflation
§5 forbids. Second blocker: `AGGREGATE = "games/ground/src/lib.rs"` is
one file, so every K-rule reports "unlinked" forever.

## 6. MATERIAL — the AM-11 argument, on which tier L rests, misreads a target as a claim and misses the actual overclaim next to it.

Nobody claims what the survey says. `GameKernel.md:175` is the **target**
column, and the evidence reports `AM-11 impl pairs | 1 of 1 (KernelRng) |
met, narrow`, adding "will mean more once storage has one". The evidence
already says what the survey accuses it of hiding.

The **real** overclaim is one clause away: M-D4-SWAP requires impls
"passing **the same conformance suite**", and no conformance suite exists
— one doc comment describing future work. The RNG pair is exercised by two
separate non-shared tests. The `met, narrow` verdict is unearned on the
suite clause.

## 7. MATERIAL — Option C is rejected against the wrong budget line.

AM-4a (1.5% headroom) is the **shipped-runtime** budget. The bundle is
written by the scenario runner on failure, and scenario tooling is
explicitly dev-only. A store behind the `scenarios` feature is charged to
**AM-4b**, headroom **32,979 lines (9.4%)** — nine times larger, and
comfortably above redb-class crates. The conclusion may survive on
simplicity, but ADR-0004 exists precisely to stop AM-4 numbers being used
as post-hoc justification, and this is that.

## 8. MATERIAL — what `make replay-test` prints if it does nothing: `ok`.

Four vacuous passes, none controlled for:

| silent failure | what it prints |
|---|---|
| no scenario fails (all 21 pass today) → zero bundles round-tripped | `0 bundles verified — ok` |
| writer emits nothing, reader tolerates; identical error strings satisfy "same failure" | `PASS — failure reproduced` |
| replayed hash compared to one recomputed in the same process — `assert_eq!(h, h)`, the AM-7 defect exactly | `PASS — bit-identical` |
| truncation path never exercised; K11's operative clause is dead code | `PASS` |

**Required controls:** a deliberately-failing fixture scenario plus
`bundles > 0`; the comparison hash read **out of the bundle**, not
recomputed; truncate-by-one-byte and corrupt-length-prefix negative
controls; and a mutated-seed negative control so the round-trip can fail.

## 9. MATERIAL — the extended `make coverage` contradicts §5 on day one.

`rule-coverage.py:195` returns `2` when `missing` is non-empty, and
`coverage` is in `all` with no `-` prefix. Adding 18 uncovered K-rules
fails `make all` immediately, which §5 bullet 2 forbids. Requires a
separate kernel denominator outside the exit code, plus a decision on when
it starts binding.

For the record: the *existing* arm cannot silently do nothing —
`rule-coverage.py:121-129` refuses to report over zero rules, and that
control is self-tested. **The new denominator must replicate it**, or a
kernel regex matching nothing reports `0/0 (100%)`.

## 10. MATERIAL — the survey says the scoped-denominator problem is general, then sweeps in one place. I swept.

**(a) AM-10 is vacuous.** `M-D4-LEAK: foreign types in cb-*-api-visible
signatures | 0`. There is no `cb-*-api` crate — true over an empty set.
What is measured is a `clippy.toml` deny of `HashMap`/`HashSet` citing
**K6 (determinism)**, reported under a **D4 leak** row. Stronger than
AM-1 as an instance of the survey's own thesis.

**(b) AM-1b is measured and dropped from the scoreboard.** `make coverage`
prints `49/58` and MetricsAndScenarios records it **Unmet**;
`grep -c 'AM-1b' evidence/CB-EV-0001` returns **0**. The artifact carrying
the headline reports the flattering half and omits the unflattering half
the same command prints two lines below.

**(c) M-D1-LNK's own denominator is one file.**

**(d) AM-2, AM-3, AM-5, AM-9 are unreported** and unmentioned by the
survey.

**Must change:** generalize §1.2 to a swept table, lead with (a). The
history file's "n=1" weakness can be retired: it is n≥4 in this repo.

## 11. MINOR — D2 understates the bundle writer.

`scenario.rs:139` creates `EventLog`, appends at :157, and **never reads
it** — not returned in `Pass`, not iterated, dropped at end of `execute`.
The one production instantiation of the K11 log is a write-only sink.
`Pass` carries end state, not an initial snapshot. Failures are a
formatted `String`, not structured expected-vs-actual. The bundle needs a
change to the runner's data flow, not "a directory of four files".

## 12. MINOR — "the number is not false" is itself false.

M-D1-COV is defined as "**numbered spec rules** covered by ≥1 passing
scenario", and `covers` is documented as "numbered rules from the
capability spec" with a non-GR example. The instrument is **non-conformant
with its own metric definition**; `58/58 (100%)` is a wrong answer to the
question M-D1-COV asks. The concession "the number is not false" gives
away the finding — it lets a future reader conclude only the reading was
careless.

---

# What I could not break

1. **No write path in the workspace.** Wider idiom set than the author's
   (`OpenOptions`, `create_dir`, `tempfile`, `to_writer`, `std::io::Write`)
   over the whole tree including `#[cfg(test)]` and benches. Every hit is a
   `Display::fmt` `write!` or `read_to_string`. Holds.
2. **No `--replay` flag and no flag parsing.** Read all 88 lines of
   `cb-sim/src/main.rs`. The survey's stronger phrasing is accurate.
3. **No `.cbreplay` artifact, reader, writer, or directory.** Confirmed
   across `.rs`, `.py`, `Makefile`, CI.
4. **18 K-rules, zero in coverage scope.** Verified two ways; `RULE_RE`
   and input read from source, not inferred from output.
5. **AM-4a headroom is 1.5%.** Number right; applicability disputed (7).
6. **`make coverage` reports 58/58 (100%) over 21 scenarios.** Reproduced.
7. **No counter-example to the absence claims.** Hunted for a K-rule in a
   scenario, a replay entry behind a feature flag, a bundle writer in CI,
   a replay round-trip test. None exists.

# Sampling limits of this review

1. **The "no write path" claim has one sample and my check does not fix
   that.** The workspace contains **ten** `.rs` files total; the author's
   sample *is* the population. My independence is in the idiom list and in
   reading whole files, not in file selection. This is the CB-WP-0002
   failure shape — two checks, one population — and I cannot make it
   otherwise from inside this repo.
2. **I did not execute the test suite.** `cargo test -- --list` enumerates
   19 tests; I ran none. Findings 1 and 2 come from reading source, not
   from mutation testing. A response disputing them should answer with a
   mutation test, not re-reading.
3. **Finding 4 rests on grep for rule IDs — the same tag-counting I
   criticize.** K14 and K18 have independent structural confirmation.
   "K10/K14/K18 are untracked" is solid; "they are the *only* further
   unimplemented K-rules" is **not** a claim I make. I did not audit
   K1–K9, K12–K17 against their acceptance text one by one. Given K9 and
   K14 both failed on close reading, a full audit should find more.
4. **I did not verify the §2 baseline table.** Still cited, by both of us.
5. **AM-12 / CostAccounting were out of scope.** I did not audit
   `cb-cost.py` or the price sheet.
6. **One process limit.** I found finding 1 only because a test had
   "replay" in its name and I opened it out of curiosity — **not** because
   any systematic step pointed there. "Reproduce the number independently"
   is satisfiable by re-running `make coverage`, which is exactly what does
   *not* find this class. **The procedure should require reading the
   assertion behind every quoted acceptance number, not re-running the
   command that prints it.** Offered as a loop change; n=1 evidence for a
   process claim, the same weakness I charged the survey with in 10.