CB-RES-0004: the instruments count names, not assertions

Tier-L survey plus the full review trail. v1 was REJECTED by adversarial review with four blocking findings; this is v2, and the finding below is the reviewer's as much as mine. The root cause is neither v1's story ("coverage scope is too narrow" — true but shallow) nor the reviewer's cheaper one ("the link check points at one file" — true but catches 3 of 7). It is: Every coverage instrument in this project counts NAMES. None counts ASSERTIONS. M-D1-COV counts covers: tags; M-D1-LNK counts rule-ID strings. Both answer "is this rule mentioned?" and neither answers "does anything fail if this rule is violated?" Four of seven defects found this pass are named in the source and inert, so no name-based check finds them. This subsumes the oldest error class: all five prior harness-does-nothing instances have the same shape. Two are mutation-proven, because the reviewer set that bar for findings they had only read: AM-7 — folding a 100k-event log from fresh(999) instead of fresh(42), an unrelated genesis state, leaves the test green. The hash reaches only a println!; the sole assertion is on elapsed time. And the log is built across games seeded 42,43,44... so it could not be asserted as written. evidence/CB-EV-0001 reports "AM-7 replay met, 2,290x" — unearned on the hash-identical clause. HDN instance #6. K9 — making Snapshot::take discard its EventSeq and store 0, which destroys the half of K9 that says "+ the EventId it includes", leaves the test green. On the acceptance criterion GameKernel §2.4 is zero of three, not two of three. Also found: K10, K14, K18 named nowhere in source; AM-10 vacuous (a K6 determinism lint reported under a D4 leak row, over a cb-*-api crate that does not exist); AM-11's conformance suite does not exist; AM-1b measured and omitted from the evidence scoreboard. All twelve challenge findings conceded, none defended on its facts. v1's prediction was rejected as unfalsifiable AND already determined; the replacement names its refuting outcome and is beatable in both directions. Chaos d4=1, no override. Declaration 1 of the 12-declaration window. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 14:19:30 +02:00 · 2026-07-31 14:19:30 +02:00 · 923c2038be
commit 923c2038be
parent 59b73c3e3d
4 changed files with 787 additions and 0 deletions
--- a/history/260731-replay-kernel-coverage-challenge.md
+++ b/history/260731-replay-kernel-coverage-challenge.md
@ -0,0 +1,263 @@
+# Challenge — CB-RES-0004 (replay bundles, durable log, and the rules nobody measures)
+
+Reviewer: adversarial pass, InnerLoop §Step 2, one round. Separate agent
+session, given the survey **and** the harness and the evidence file per
+the "numbers" row of the review-target table.
+Tree reviewed: `59b73c3`.
+Verdict: **not approvable as written.** Four BLOCKING, six MATERIAL, two
+MINOR.
+
+> The survey is right that something is unmeasured. It is wrong about
+> what, wrong about how much, and its own baseline contains an instance of
+> the exact error class it is diagnosing. The headline finding is
+> under-scoped by at least three rules, and the instrument it proposes is
+> heavier and less effective than a one-line extension of an instrument
+> that already exists in this repo.
+
+---
+
+## 1. BLOCKING — the AM-7 replay row does not measure the half of AM-7 the survey defends as "honest". Instance #6 of the harness-does-nothing class is live in the tree right now.
+
+The survey §1.3 calls the AM-7 row "honest but easily misread". It is not
+honest. AM-7's target is "replay of 100k events ≤ 5 s, **hash-identical**".
+
+`games/ground/src/lib.rs:2241-2256`:
+
+```rust
+let hash = state_hash_hex(&state);
+let elapsed = start.elapsed();
+println!("replay {} events in {:?} ({:.0} events/s), hash {}", ...);
+assert!(elapsed.as_secs_f64() < 5.0, "AM-7: 100k replay under 5s");
+```
+
+`hash` reaches exactly one place: a `println!`. **There is no assertion on
+it.** The Criterion bench is the same shape.
+
+Worse, the hash *could not* be asserted as written. The log is built
+across multiple games (`lib.rs:2231-2235`):
+
+```rust
+while log.len() < target {
+    if source.outcome.is_some() { games += 1; source = fresh(42 + games); }
+```
+
+so the "replay" folds events from games seeded 42, 43, 44… into a single
+state built from `fresh(42)`. It is not a replay of anything. It is a
+fold-rate microbenchmark over a semantically invalid event stream,
+reported as `AM-7 replay | met, 2,290×`.
+
+**Must change:** §1.3 rewritten — AM-7's `hash-identical` clause is
+unmeasured, the fold is over a multi-game log, and the evidence verdict is
+unearned on that clause. This is a correction to `evidence/CB-EV-0001`,
+and it puts AM-7 in the pass scope.
+
+## 2. BLOCKING — the survey grants K9 as implemented. K9's acceptance property is asserted nowhere.
+
+K9 requires "`snapshot + remaining events → state` must be hash-identical
+to a from-genesis fold". The entire evidence is one unit test
+(`cb-events/src/lib.rs:158-167`) that round-trips a
+`BTreeMap<String, u8>` with `EventSeq(17)` as a literal. No game
+aggregate, no events applied, no from-genesis comparison. And `Snapshot`
+has no callers outside its own test.
+
+The survey established which rules are implemented by reading `pub fn`
+signatures — presence-of-API, which is exactly the tag-counting it
+condemns in §5. **On the acceptance criterion §2.4 is not "two of three",
+it is zero of three.**
+
+## 3. BLOCKING — the "below 100%" prediction is arithmetic, not a prediction, and §6 is constructed so no outcome can refute it.
+
+Below 100% → thesis confirmed. Near 100% → "the instrument measures tags",
+thesis confirmed. No stated result counts against the survey — the
+unfalsifiable framing the survey claims to guard against, dressed in the
+vocabulary of falsifiability.
+
+It is also not a prediction: every `covers:` id in the corpus is `GR-*`
+(58 ids, zero `K*`), and `ls scenarios/` returns one directory. Any
+tag-counting extension **must** report **0 of 18 (0%)** on first run.
+Computable today, before writing a line.
+
+**Must change:** a prediction that can fail, with the refuting outcome
+named. Delete the §6 escape clause.
+
+## 4. BLOCKING — K10 is not the only unimplemented kernel rule. K14 and K18 are too, and the one-line check that finds all three already exists in this repo.
+
+```
+spec:  K1..K18
+code:  K1..K9 K11 K12 K13 K15 K16 K17
+diff:  K10, K14, K18
+```
+
+**K14** ("one commit window per Select") — `CommitWindow` has zero
+non-test users and `games/ground` does not import it. GROUND collects
+selections in its own aggregate. A dead abstraction certified by a test
+that exercises only itself.
+
+**K18** ("Criterion benches driving the same scenario format") — the
+bench hardcodes commands in Rust and never touches `ScenarioFile`. And
+`ls -R benchmarks` contains only `baselines/model-prices.toml`.
+
+**Why this is BLOCKING.** The check that found these is a two-line
+generalization of **M-D1-LNK / AM-1b**, which this repo already runs every
+`make coverage`. Pointing `parse_code_ids` at `specs/GameKernel.md` ×
+`crates/**` would have surfaced all three the day AM-1b shipped. The
+survey diagnoses a gap a cheap existing instrument covers and prescribes a
+new expensive capability as the remedy — and, having chosen the expensive
+framing, found 1 of the 3 gaps the cheap check finds in one line.
+
+**Must change:** rescope to K10/K14/K18 (K9, K11 partial). Add **Option E:
+extend AM-1b's spec→code link to every numbered spec and every crate**,
+and argue why the tier-L work is warranted *on top of* it.
+
+## 5. MATERIAL — K-rules cannot be covered by the mechanism that covers GR-rules. "A parameterization, not a rewrite" is false.
+
+The M-D1-COV numerator comes from `covers:` lists consumed by a runner
+requiring `ScenarioGame: Aggregate + Serialize + DeserializeOwned`, with
+dispatch by game prefix and `ground` the only arm. There is no kernel
+aggregate, setup preset, or command vocabulary. `scenarios/kernel/*.yaml`
+would be a tag in a directory the runner cannot dispatch — the inflation
+§5 forbids. Second blocker: `AGGREGATE = "games/ground/src/lib.rs"` is
+one file, so every K-rule reports "unlinked" forever.
+
+## 6. MATERIAL — the AM-11 argument, on which tier L rests, misreads a target as a claim and misses the actual overclaim next to it.
+
+Nobody claims what the survey says. `GameKernel.md:175` is the **target**
+column, and the evidence reports `AM-11 impl pairs | 1 of 1 (KernelRng) |
+met, narrow`, adding "will mean more once storage has one". The evidence
+already says what the survey accuses it of hiding.
+
+The **real** overclaim is one clause away: M-D4-SWAP requires impls
+"passing **the same conformance suite**", and no conformance suite exists
+— one doc comment describing future work. The RNG pair is exercised by two
+separate non-shared tests. The `met, narrow` verdict is unearned on the
+suite clause.
+
+## 7. MATERIAL — Option C is rejected against the wrong budget line.
+
+AM-4a (1.5% headroom) is the **shipped-runtime** budget. The bundle is
+written by the scenario runner on failure, and scenario tooling is
+explicitly dev-only. A store behind the `scenarios` feature is charged to
+**AM-4b**, headroom **32,979 lines (9.4%)** — nine times larger, and
+comfortably above redb-class crates. The conclusion may survive on
+simplicity, but ADR-0004 exists precisely to stop AM-4 numbers being used
+as post-hoc justification, and this is that.
+
+## 8. MATERIAL — what `make replay-test` prints if it does nothing: `ok`.
+
+Four vacuous passes, none controlled for:
+
+| silent failure | what it prints |
+|---|---|
+| no scenario fails (all 21 pass today) → zero bundles round-tripped | `0 bundles verified — ok` |
+| writer emits nothing, reader tolerates; identical error strings satisfy "same failure" | `PASS — failure reproduced` |
+| replayed hash compared to one recomputed in the same process — `assert_eq!(h, h)`, the AM-7 defect exactly | `PASS — bit-identical` |
+| truncation path never exercised; K11's operative clause is dead code | `PASS` |
+
+**Required controls:** a deliberately-failing fixture scenario plus
+`bundles > 0`; the comparison hash read **out of the bundle**, not
+recomputed; truncate-by-one-byte and corrupt-length-prefix negative
+controls; and a mutated-seed negative control so the round-trip can fail.
+
+## 9. MATERIAL — the extended `make coverage` contradicts §5 on day one.
+
+`rule-coverage.py:195` returns `2` when `missing` is non-empty, and
+`coverage` is in `all` with no `-` prefix. Adding 18 uncovered K-rules
+fails `make all` immediately, which §5 bullet 2 forbids. Requires a
+separate kernel denominator outside the exit code, plus a decision on when
+it starts binding.
+
+For the record: the *existing* arm cannot silently do nothing —
+`rule-coverage.py:121-129` refuses to report over zero rules, and that
+control is self-tested. **The new denominator must replicate it**, or a
+kernel regex matching nothing reports `0/0 (100%)`.
+
+## 10. MATERIAL — the survey says the scoped-denominator problem is general, then sweeps in one place. I swept.
+
+**(a) AM-10 is vacuous.** `M-D4-LEAK: foreign types in cb-*-api-visible
+signatures | 0`. There is no `cb-*-api` crate — true over an empty set.
+What is measured is a `clippy.toml` deny of `HashMap`/`HashSet` citing
+**K6 (determinism)**, reported under a **D4 leak** row. Stronger than
+AM-1 as an instance of the survey's own thesis.
+
+**(b) AM-1b is measured and dropped from the scoreboard.** `make coverage`
+prints `49/58` and MetricsAndScenarios records it **Unmet**;
+`grep -c 'AM-1b' evidence/CB-EV-0001` returns **0**. The artifact carrying
+the headline reports the flattering half and omits the unflattering half
+the same command prints two lines below.
+
+**(c) M-D1-LNK's own denominator is one file.**
+
+**(d) AM-2, AM-3, AM-5, AM-9 are unreported** and unmentioned by the
+survey.
+
+**Must change:** generalize §1.2 to a swept table, lead with (a). The
+history file's "n=1" weakness can be retired: it is n≥4 in this repo.
+
+## 11. MINOR — D2 understates the bundle writer.
+
+`scenario.rs:139` creates `EventLog`, appends at :157, and **never reads
+it** — not returned in `Pass`, not iterated, dropped at end of `execute`.
+The one production instantiation of the K11 log is a write-only sink.
+`Pass` carries end state, not an initial snapshot. Failures are a
+formatted `String`, not structured expected-vs-actual. The bundle needs a
+change to the runner's data flow, not "a directory of four files".
+
+## 12. MINOR — "the number is not false" is itself false.
+
+M-D1-COV is defined as "**numbered spec rules** covered by ≥1 passing
+scenario", and `covers` is documented as "numbered rules from the
+capability spec" with a non-GR example. The instrument is **non-conformant
+with its own metric definition**; `58/58 (100%)` is a wrong answer to the
+question M-D1-COV asks. The concession "the number is not false" gives
+away the finding — it lets a future reader conclude only the reading was
+careless.
+
+---
+
+# What I could not break
+
+1. **No write path in the workspace.** Wider idiom set than the author's
+   (`OpenOptions`, `create_dir`, `tempfile`, `to_writer`, `std::io::Write`)
+   over the whole tree including `#[cfg(test)]` and benches. Every hit is a
+   `Display::fmt` `write!` or `read_to_string`. Holds.
+2. **No `--replay` flag and no flag parsing.** Read all 88 lines of
+   `cb-sim/src/main.rs`. The survey's stronger phrasing is accurate.
+3. **No `.cbreplay` artifact, reader, writer, or directory.** Confirmed
+   across `.rs`, `.py`, `Makefile`, CI.
+4. **18 K-rules, zero in coverage scope.** Verified two ways; `RULE_RE`
+   and input read from source, not inferred from output.
+5. **AM-4a headroom is 1.5%.** Number right; applicability disputed (7).
+6. **`make coverage` reports 58/58 (100%) over 21 scenarios.** Reproduced.
+7. **No counter-example to the absence claims.** Hunted for a K-rule in a
+   scenario, a replay entry behind a feature flag, a bundle writer in CI,
+   a replay round-trip test. None exists.
+
+# Sampling limits of this review
+
+1. **The "no write path" claim has one sample and my check does not fix
+   that.** The workspace contains **ten** `.rs` files total; the author's
+   sample *is* the population. My independence is in the idiom list and in
+   reading whole files, not in file selection. This is the CB-WP-0002
+   failure shape — two checks, one population — and I cannot make it
+   otherwise from inside this repo.
+2. **I did not execute the test suite.** `cargo test -- --list` enumerates
+   19 tests; I ran none. Findings 1 and 2 come from reading source, not
+   from mutation testing. A response disputing them should answer with a
+   mutation test, not re-reading.
+3. **Finding 4 rests on grep for rule IDs — the same tag-counting I
+   criticize.** K14 and K18 have independent structural confirmation.
+   "K10/K14/K18 are untracked" is solid; "they are the *only* further
+   unimplemented K-rules" is **not** a claim I make. I did not audit
+   K1–K9, K12–K17 against their acceptance text one by one. Given K9 and
+   K14 both failed on close reading, a full audit should find more.
+4. **I did not verify the §2 baseline table.** Still cited, by both of us.
+5. **AM-12 / CostAccounting were out of scope.** I did not audit
+   `cb-cost.py` or the price sheet.
+6. **One process limit.** I found finding 1 only because a test had
+   "replay" in its name and I opened it out of curiosity — **not** because
+   any systematic step pointed there. "Reproduce the number independently"
+   is satisfiable by re-running `make coverage`, which is exactly what does
+   *not* find this class. **The procedure should require reading the
+   assertion behind every quoted acceptance number, not re-running the
+   command that prints it.** Offered as a loop change; n=1 evidence for a
+   process claim, the same weakness I charged the survey with in 10.
--- a/history/260731-replay-kernel-coverage-research.md
+++ b/history/260731-replay-kernel-coverage-research.md
@ -0,0 +1,92 @@
+# 2026-07-31 — how CB-RES-0004 was conducted
+
+The unpolished trail, so a later reader can judge how hard the survey was
+actually tested.
+
+## How the finding was reached
+
+Not by looking for it. CB-WP-0004 closed, `make status` reported every
+workplan done and every gate green, and the question was what the next
+pass should be. `INTENT.md` implementation order stage 0 lists "replay
+and scenario tests" among the headless-GROUND deliverables, so the first
+check was whether that was done.
+
+Sequence of commands, in order:
+
+```sh
+find crates games tools/cb-sim -name '*.rs' | xargs wc -l   # 3,443 lines total
+grep -n "replay\|cbreplay\|--replay" specs/GameKernel.md    # K10 found
+ls replays/                                                  # no such directory
+grep -rn "cbreplay\|--replay" crates tools --include=*.rs    # zero matches
+grep -n "pub fn\|pub struct" crates/cb-events/src/lib.rs     # K9 present, K10/K11 absent
+grep -rn "File::\|fs::\|write_all\|BufWriter" crates games tools/cb-sim --include=*.rs
+```
+
+The last one is the load-bearing check and it is worth stating what it
+found: **the only `fs::` call in the entire workspace is
+`read_to_string` for scenario YAML.** There is no write path anywhere. A
+"durable log format" rule (K11) is being satisfied by a `Vec`.
+
+## What was measured vs cited
+
+**Measured on the tree at `59b73c3`:**
+
+- absence of `.cbreplay` reader/writer — grep, zero matches
+- absence of a `--replay` flag — read `tools/cb-sim/src/main.rs` in full;
+  it collects `args` and treats every one as a scenario path. There is no
+  flag parsing at all, so the flag is not merely missing, there is no
+  place to put it
+- absence of any file-write path — grep for five different write idioms
+- 18 K-rules in `specs/GameKernel.md` (`grep -o "K[0-9]\+" | sort -u`)
+- `rule-coverage.py` `RULE_RE = \*\*(GR-[A-Z]+\d+)` against
+  `specs/GroundRules.md` only — read the source, not inferred from output
+
+**Cited, not measured:** every row of the state-of-the-art table in §2.
+boardgame.io, Rune, rr/Pernosco, Playwright and Kafka/EventStore are
+described from prior knowledge. None was run. Per MetricsAndScenarios §3
+this caps any comparison verdict at `parity`, and the survey says so.
+
+This is a real weakness of the survey and is not hidden: the *gap*
+finding is measured and the *design* recommendation rests on cited
+precedent.
+
+## Dead ends
+
+- **Looking for the gap in the scenario corpus first.** 21 scenarios, 58
+  GR-rules, 100% covered — the corpus looks complete and says nothing
+  about the kernel. Roughly ten minutes were spent reading scenario names
+  before it became clear the coverage instrument's *denominator* was the
+  question, not its numerator.
+- **Assuming AM-7 covered replay.** `evidence/CB-EV-0001` reports
+  `AM-7 replay | 100k events ≤5s | 2.18 ms | met, 2,290×`, which reads
+  like replay is done and fast. It measures an in-memory fold — the K9
+  path. The evidence is honest (it even documents that this benchmark was
+  the fourth harness-does-nothing instance) but the row's wording carried
+  me past the gap on first reading. Worth recording because a future
+  reader will hit the same row.
+- **Considering an embedded store (sled/redb) for K11.** Dropped once
+  ADR-0004's 1.5% AM-4a headroom was recalled: a new shipped-runtime
+  dependency now requires an argued decision, and the format is a
+  paragraph.
+
+## The chaos roll
+
+`shuf -i 1-4 -n 1` → **1**. Structural tier L (new capability port;
+changes the meaning of a headline metric). No override. Recorded per the
+rule that the roll is written down even when it changes nothing. This is
+declaration **1** of the 12-declaration d4 calibration window opened by
+CB-WP-0003 T06.
+
+## Known weaknesses of this survey, stated before review
+
+1. **All baseline rows are cited.** No competitor was run.
+2. **The author found the gap and also proposes the fix**, so the
+   framing ("coverage scope is the root cause") is the author's and has
+   not been independently attacked.
+3. **The predicted outcome — kernel coverage below 100% on first run —
+   is unfalsified.** The instrument that would test it has not been
+   written yet, which is exactly the shape of claim this project has
+   been wrong about before.
+4. **The single-sample problem applies.** One repo, one spec pair. The
+   claim "an instrument reports a percentage over the sample it was
+   pointed at" is general, but the evidence for it here is n=1.
--- a/history/260731-replay-kernel-coverage-response.md
+++ b/history/260731-replay-kernel-coverage-response.md
@ -0,0 +1,160 @@
+# 2026-07-31 — response to the CB-RES-0004 challenge
+
+**All twelve findings conceded.** Nothing was defended on its facts,
+because I re-derived every load-bearing claim myself and the reviewer was
+right on each one. Two findings the reviewer explicitly marked as
+*read-only, not mutation-tested* are now mutation-proven below — that was
+the right bar to set and it deserved to be met rather than argued around.
+
+The survey is withdrawn and rewritten. What follows is what changed and,
+more importantly, the one thing the round produced that neither the survey
+nor the challenge had.
+
+---
+
+## The mutation tests the challenge asked for
+
+> *"A response that disputes findings 1 or 2 should be answered with a
+> mutation test, not with re-reading."* — challenge, sampling limit 2
+
+I am not disputing them. I ran the mutations anyway, because a conceded
+finding backed by evidence is worth more to the next reader than a
+conceded finding backed by agreement.
+
+**Finding 1 — AM-7.** Changed the fold's genesis from `fresh(42)` to
+`fresh(999)`, so the 100k-event log is replayed into a completely
+different starting state:
+
+```
+MUTATION: fold starts from fresh(999) — a different genesis state.
+test replay_probe::replay_100k_events_is_linear_and_fast ... ok
+test result: ok. 1 passed; 0 failed
+```
+
+**The test passes.** AM-7's `hash-identical` clause detects nothing at
+all. The reviewer read this correctly from source; it is now demonstrated.
+
+**Finding 2 — K9.** Changed `Snapshot::take` to discard the `EventSeq` it
+is handed and store `EventSeq(0)`, destroying the half of K9 that says a
+snapshot is "the full aggregate **+ the EventId it includes**":
+
+```
+MUTATION: Snapshot::take DISCARDS its EventSeq and stores 0.
+test tests::snapshot_roundtrip ... ok
+test result: ok. 3 passed; 0 failed
+```
+
+**The test passes.** K9's `through` field is unverified.
+
+## What the round actually produced
+
+The challenge's finding 4 says the root cause is cheaper than the survey
+claimed: extend AM-1b's spec→code link to every numbered spec and every
+crate, and K10/K14/K18 fall out in one line. That is correct and it is
+adopted as Option E, first step of the pass.
+
+**But findings 1 and 2 prove Option E is not sufficient, and the reviewer
+did not connect their own findings.** Sort the seven defects this round
+surfaced by whether a name-based check would catch them:
+
+| defect | named in code? | Option E catches it? |
+|---|---|---|
+| K10 replay bundle | no | **yes** |
+| K14 commit window | no | **yes** |
+| K18 bench-from-scenarios | no | **yes** |
+| K9 snapshot `through` | **yes** | no |
+| K11 durable format | **yes** | no |
+| AM-7 hash-identical | **yes** | no |
+| AM-11 conformance suite | **yes** | no |
+
+Four of seven are named in the source and unasserted. So:
+
+> **Every coverage instrument in this project counts names. None counts
+> assertions.** M-D1-COV counts `covers:` tags. M-D1-LNK counts rule-ID
+> strings in source. Both answer "is this rule mentioned?" and neither
+> answers "does anything fail if this rule is violated?"
+
+That is the correct root cause. It is neither the survey's story
+("coverage scope is too narrow" — true but shallow) nor the challenge's
+("the link check is pointed at one file" — true but only covers 3 of 7).
+It also subsumes the project's oldest error class: all five prior
+harness-does-nothing instances, plus AM-7 and K9 today, are the same
+shape — **the name is there, the assertion is not.**
+
+The instrument this implies is **mutation**, not another grep. A rule is
+covered when breaking it breaks a test. That is expensive to do
+exhaustively and cheap to do on the acceptance rows, which is exactly
+where every serious error in this project has been.
+
+## The loop change this earns
+
+The challenge's sampling limit 6 is the sharpest paragraph in the
+document:
+
+> *"I found finding 1 only because a test had 'replay' in its name and I
+> opened it out of curiosity — not because any systematic step pointed
+> there. 'Reproduce the number independently' is satisfiable by re-running
+> `make coverage`, which is exactly what does not find this class."*
+
+Conceded and adopted. InnerLoop §Step 2's "numbers" row currently says the
+reviewer must *reproduce the number independently*. Re-running the command
+that prints it satisfies that wording and finds nothing. The row must
+require **reading the assertion behind the number, and mutating it** —
+state what the harness would report if the property were false, then make
+it false and check.
+
+The reviewer notes this is n=1 evidence for a process claim. It is n=2:
+CB-WP-0002's dedup blind spot had the same structure — a verification step
+that re-ran the author's method and inherited the author's blindness. The
+fix there was an assertion over all data at execution time; the fix here is
+mutation. Both replace *re-derivation* with *adversarial execution*.
+
+## Finding-by-finding disposition
+
+| # | severity | disposition |
+|---|---|---|
+| 1 | BLOCKING | conceded, **mutation-proven**. AM-7 correction enters the pass scope; `evidence/CB-EV-0001` needs a correction note. HDN instance #6. |
+| 2 | BLOCKING | conceded, **mutation-proven**. §2.4 is zero of three on the acceptance criterion, not two of three. |
+| 3 | BLOCKING | conceded fully. The prediction was unfalsifiable *and* already determined (0 of 18). Replaced with a prediction that names its refuting outcome. |
+| 4 | BLOCKING | conceded. Rescoped to K10/K14/K18; Option E adopted as step one — **and shown above to be insufficient**, which is the survey's real justification, not the one it gave. |
+| 5 | MATERIAL | conceded. "A parameterization, not a rewrite" withdrawn. The K-rule numerator must come from named executable assertions, not `covers:` tags. `AGGREGATE` becomes a list. |
+| 6 | MATERIAL | conceded. The AM-11 misreading is deleted. The real overclaim — M-D4-SWAP requires "the same conformance suite" and no suite exists — replaces it, and is a **stronger** basis for the port argument than what I wrote. |
+| 7 | MATERIAL | conceded with a clarification the survey owed: the **durable log (K11) is shipped-runtime** and AM-4a applies; the **bundle writer (K10) is dev-only** behind the `scenarios` feature and AM-4b applies. The survey wanted it both ways without saying so. Option C is now rejected on simplicity for the log, and the headroom claim is dropped for the bundle. |
+| 8 | MATERIAL | conceded. All four controls adopted verbatim, including the two negative controls I had not thought of (truncate-by-one-byte, mutated seed). |
+| 9 | MATERIAL | conceded. The exit-code path is named, and the new denominator gets the zero-rules positive control the existing arm already has. |
+| 10 | MATERIAL | conceded. The sweep is folded in with AM-10 as lead example. **AM-10 is the better instance than AM-1**: a K6 determinism lint reported under a D4 leak row, over a crate that does not exist. |
+| 11 | MINOR | conceded. D2's estimate corrected: `Pass` extended, `RunOutcome::Failed` restructured, initial snapshot captured, log plumbed out of `execute`. |
+| 12 | MINOR | conceded. "The number is not false" deleted. M-D1-COV is defined over "numbered spec rules"; the instrument is non-conformant with its own metric definition and `58/58 (100%)` is a wrong answer to the question the metric asks. |
+
+## One thing the challenge could not have found
+
+Running its mutation tests surfaced a limit in **CB-WP-0004 T01**, closed
+two hours earlier:
+
+```
+$ cargo test -p games-ground ...
+/bin/bash: line 22: cargo: command not found
+```
+
+`make env-test` proves every *tool* and every *make target* runs from any
+directory with no cargo on PATH. It does not put cargo on the agent's
+interactive PATH, so an ad-hoc `cargo` invocation still fails. T01's
+evidence claimed environment friction went from 85 turns to 1; that number
+stands for the measured window, but the fix is narrower than "no
+environment setup", and CB-EV-0003 should say so.
+
+Small, and recorded rather than quietly patched, because CB-WP-0004 T05's
+whole finding was that tooling only recovers capacity when the manual path
+stops existing — and here it did not stop existing.
+
+## Approval status
+
+**Not approved.** The survey is withdrawn and rewritten as
+`research/CB-RES-0004` v2 incorporating all twelve findings, the corrected
+root cause, and a rescoped pass that now includes AM-7, K9, K14, K18,
+AM-10 and AM-11 — none of which were in the original scope.
+
+The review cost is recorded in the pass evidence. On the two prior passes
+this step cost $0.66 and $1.11 and each time found an approval-blocking
+defect. This time it found four, one of which invalidates a committed
+`met` verdict in `evidence/CB-EV-0001`.
--- a/research/CB-RES-0004-replay-and-kernel-coverage.md
+++ b/research/CB-RES-0004-replay-and-kernel-coverage.md
@ -0,0 +1,272 @@
+# CB-RES-0004: the instruments count names, not assertions
+
+capability: meta.coverage.assertions + kernel.replay
+status: approved (v2, after adversarial review — v1 was rejected with four
+blocking findings)
+tier: L (structural L — creates a new capability port (durable log
+storage); changes what several headline acceptance numbers mean; chaos
+d4=1 → no override)
+instrument: extended `make coverage` (spec→code over every numbered spec)
+plus a new `make mutation-check` and `make replay-test`
+baseline: cited — boardgame.io, Rune, rr/Pernosco, Playwright trace,
+event-sourcing log framing
+
+Survey opened 2026-07-31. **v1 was rejected**; the review trail is
+`history/260731-replay-kernel-coverage-{research,challenge,response}.md`.
+The finding below is the reviewer's as much as mine.
+
+---
+
+## 1. The finding
+
+> **Every coverage instrument in this project counts names. None counts
+> assertions.** M-D1-COV counts `covers:` tags in scenario YAML. M-D1-LNK
+> counts rule-ID strings in one source file. Both answer *"is this rule
+> mentioned?"*; neither answers *"does anything fail if this rule is
+> violated?"*
+
+Seven defects were found this pass. Four of them are **named in the source
+and unasserted** — invisible to any name-based check, including the
+cheaper one the reviewer proposed:
+
+| defect | named in code? | a name-based check finds it? |
+|---|---|---|
+| K10 replay bundle — no writer, reader, or `--replay` | no | **yes** |
+| K14 commit window — `CommitWindow` has zero non-test users | no | **yes** |
+| K18 benches from scenarios — bench hardcodes commands | no | **yes** |
+| K9 snapshot `+ the EventId it includes` | yes | no |
+| K11 durable format, truncated-tail detection | yes | no |
+| AM-7 `hash-identical` | yes | no |
+| AM-11 `passing the same conformance suite` | yes | no |
+
+This subsumes the project's oldest error class. All five prior
+harness-does-nothing instances have the same shape as AM-7 and K9 today:
+**the name is there, the assertion is not.**
+
+### 1.1 The two that are mutation-proven
+
+**AM-7's `hash-identical` clause detects nothing.**
+`games/ground/src/lib.rs` computes `state_hash_hex(&state)` and passes it
+to a `println!`. The only assertion is on elapsed time. Mutating the
+fold's genesis from `fresh(42)` to `fresh(999)` — replaying a 100k-event
+log into an unrelated starting state — leaves the test green:
+
+```text
+test replay_probe::replay_100k_events_is_linear_and_fast ... ok
+```
+
+The hash could not be asserted as written anyway: the log is built across
+games seeded 42, 43, 44… (`source = fresh(42 + games)`) and folded into a
+state from `fresh(42)`. It is a fold-rate microbenchmark over a
+semantically invalid stream, reported in `evidence/CB-EV-0001` as
+**`AM-7 replay | met, 2,290×`**. That verdict is unearned on the
+hash clause and needs a correction note. **HDN instance #6.**
+
+**K9's `through` field is unverified.** The whole evidence for K9 is one
+test round-tripping a `BTreeMap<String, u8>` with `EventSeq(17)` as a
+literal — no game aggregate, no events applied, no from-genesis
+comparison. `Snapshot` has no callers outside that test. Mutating
+`Snapshot::take` to discard its `EventSeq` and store `EventSeq(0)` leaves
+the test green.
+
+On the acceptance criterion, **GameKernel §2.4 is zero of three, not two
+of three.**
+
+### 1.2 The three that are named nowhere
+
+```text
+spec:  K1..K18       code:  K1..K9 K11 K12 K13 K15 K16 K17
+diff:  K10, K14, K18
+```
+
+- **K10** — no `.cbreplay` writer, reader, or `replays/` directory; the
+  only `fs::` call in the workspace is `read_to_string` for scenario YAML.
+  `cb-sim` has no flag parsing at all, so `--replay` has no place to go.
+  `MetricsAndScenarios` §2 promises a failing run writes a bundle. It does
+  not. INTENT names this design decision **8 of 10**.
+- **K14** — "one commit window per Select". `CommitWindow` has zero
+  non-test users and `games/ground` does not import it; GROUND collects
+  selections in its own aggregate. A dead abstraction certified by a test
+  that exercises only itself.
+- **K18** — "Criterion benches driving the same scenario format". The
+  bench hardcodes commands in Rust and never touches `ScenarioFile`.
+  `benchmarks/` contains only `baselines/model-prices.toml`.
+
+### 1.3 The scoped-denominator sweep
+
+v1 claimed this problem was general and swept in one place. Swept properly,
+in descending order of severity:
+
+**AM-10 is vacuous — the population is empty.** The row reads
+*M-D4-LEAK: foreign types in `cb-*-api`-visible signatures | 0*. There is
+no `cb-*-api` crate. What is actually measured is a `clippy.toml` deny of
+`HashMap`/`HashSet` whose stated reason cites **K6 (determinism)**,
+reported under a **D4 leak** row. A determinism lint standing in for a
+leak metric, over a crate that does not exist.
+
+**AM-1's denominator is one spec, and the metric says otherwise.**
+M-D1-COV is defined as *"numbered spec rules covered by ≥1 passing
+scenario"*, and `covers` is documented as "numbered rules from the
+capability spec" with a non-GR example. `RULE_RE` matches `GR-` only,
+against `GroundRules.md` only. So `58/58 (100%)` is not merely narrowly
+scoped — the instrument is **non-conformant with its own metric
+definition**, and 100% is a wrong answer to the question M-D1-COV asks.
+
+**AM-1b is measured and dropped from the scoreboard.** `make coverage`
+prints `49/58` and `MetricsAndScenarios` records it **Unmet**;
+`grep -c 'AM-1b' evidence/CB-EV-0001` returns **0**. The artifact carrying
+the headline reports the flattering half of the gate and omits the
+unflattering half the same command prints two lines below it.
+
+**AM-11's suite does not exist.** M-D4-SWAP requires impls "passing **the
+same conformance suite**". `grep -rn conformance` over all `.rs` returns
+one doc comment describing future work. The RNG pair is exercised by two
+separate, non-shared tests. `met, narrow` is unearned on the suite clause.
+
+**AM-2, AM-3, AM-5, AM-9 are unreported** and were unmentioned by v1.
+
+The history file's stated weakness — *"the evidence for this is n=1"* — is
+retired. It is n≥5 in this repo, found without leaving the tree.
+
+---
+
+## 2. Baseline — what the state of the art does
+
+Cited, not locally measured; per MetricsAndScenarios §3 this caps
+comparison verdicts at `parity`.
+
+| system | durable log | exportable failure bundle | machine re-executable |
+|---|---|---|---|
+| **boardgame.io** | in-memory, optional server persistence | debug panel, in-session time travel | no |
+| **Rune** | rollback ring buffer, transient | no | no |
+| **rr / Pernosco** | full process recording | yes — the gold standard | **yes**, bit-exact |
+| **Playwright trace** | per-run trace zip | **yes**, one file | partly (UI, not logic) |
+| **Kafka / EventStore** | append-only, length-prefixed, versioned, checksummed | n/a | yes, by re-consumption |
+
+Two lessons:
+
+1. **The valuable property is machine re-execution, not inspection.**
+   boardgame.io's debug panel is more visible than a `.cbreplay` file and
+   far less useful to an agent, because a human must interpret it. rr and
+   Playwright are the precedents worth taking.
+2. **The log format is solved engineering.** Length prefix, version byte,
+   per-record framing, tail detection: Kafka and EventStore converged on
+   the same shape decades apart. It is ~80–120 lines.
+
+**Provenance.** Log framing: `adapted:event-sourcing-log-framing` —
+length-prefixed versioned records with tail detection, dropping the
+distributed concerns (partitions, offsets, compaction) a single-game log
+does not have. Bundle: `adapted:playwright-trace` — one artifact per
+failed run containing everything needed to reproduce, adapted so the
+consumer is an agent, not a human viewer. Assertion coverage:
+`adapted:mutation-testing` — mutate the property, expect a red test;
+applied only to acceptance rows rather than exhaustively.
+
+---
+
+## 3. The four dimensions
+
+**D1 — ease of specification.** Strongly positive. A `.cbreplay` bundle
+turns "the scenario failed" into an artifact with a stated contract. More
+importantly, mutation coverage makes "this rule is enforced" a *claim with
+a test* rather than a tag.
+
+**D2 — efficiency of implementation.** The log format adds no dependency
+(serde is already shipped). **The bundle writer is not "a directory of
+four files"** — v1 said that and it was wrong. `scenario.rs` creates an
+`EventLog`, appends to it, and never reads it; it is dropped at the end of
+`execute`. `Pass` carries the *end* state, not an initial snapshot, and
+failures are a formatted `String`, not structured expected-vs-actual. The
+bundle requires changing the runner's data flow: plumb the log out, capture
+an initial snapshot, restructure `RunOutcome::Failed`.
+
+**D3 — speed of execution.** Bundle writes happen only on failure, so the
+hot path is untouched. Log append gains a length-prefix write. Mutation
+checks run in CI, not in the loop.
+
+**D4 — optionality.** Where the pass earns its L, on a corrected premise.
+v1 argued AM-11 "claims" a storage port; it does not — that is the target
+column, and the evidence honestly reports `1 of 1, met, narrow`. The real
+argument is stronger: **M-D4-SWAP is a bool over a conformance suite that
+does not exist.** Building a durable-log port with a genuine shared suite
+discharges AM-11 properly and creates the seam stage 3 (networked
+sessions) and stage 4 (game-creation framework) both need.
+
+---
+
+## 4. Options
+
+**A — implement K10/K11, leave the instruments alone.** Rejected. Fixes
+what was noticed; leaves the mechanism that hid it.
+
+**B — extend coverage to K-rules, defer the rest.** Rejected as a final
+state. Adopted as *ordering*.
+
+**C — adopt an embedded store (sled, redb).** Rejected — and v1 rejected
+it against the wrong budget. The **durable log (K11) is shipped-runtime**,
+so AM-4a applies with 1.5% headroom; the **bundle writer (K10) is dev-only**
+behind the `scenarios` feature, so AM-4b applies with 9.4%. v1 wanted it
+both ways without saying so. With the split stated, C is rejected on
+simplicity for the log — a length prefix is not a hard implementation, and
+"assimilate the implementation" is for hard ones — and the headroom
+argument is dropped for the bundle, where it never applied.
+
+**E — extend AM-1b's spec→code link to every numbered spec and every
+crate.** ~10 lines against an instrument that already exists. Catches
+K10, K14, K18 immediately. **Adopted as step one — and demonstrably
+insufficient**, since it catches 3 of the 7 defects and none of the four
+mutation-class ones. `AGGREGATE` must become a list.
+
+**F — E, plus mutation checks on acceptance rows, plus K10/K11.**
+**Recommended.** The name-based check is cheap and catches the absent; the
+mutation check is the only thing that catches the present-but-inert; the
+capability work discharges the rules both instruments expose.
+
+---
+
+## 5. What this pass must not do
+
+- **Do not tag K-rules as covered to make a number go up.** The overclaim
+  CB-WP-0001 committed once and M-D1-LNK exists to catch. A K-rule is
+  covered when breaking it breaks a test.
+- **Do not let the widened coverage gate fail the build on day one.**
+  `rule-coverage.py` returns `2` when `missing` is non-empty and
+  `coverage` is in `all`, so 18 uncovered K-rules fail `make all`
+  immediately. The kernel denominator must report outside the exit code
+  until the pass closes the gap, with a stated date for when it binds.
+- **Replicate the zero-rules positive control on the new denominator.**
+  The existing arm refuses to report over zero rules — a defect it was
+  fixed for. A kernel regex matching nothing must abort, not print
+  `0/0 (100%)`.
+- **`make replay-test` must not be able to pass vacuously.** It reports
+  `ok` under at least four silent failures: no scenario fails so zero
+  bundles are round-tripped; the writer emits nothing and identical error
+  strings satisfy "same failure"; the replayed hash is compared to one
+  recomputed in the same process (`assert_eq!(h, h)` — the AM-7 defect
+  exactly); the truncation path is never exercised. **Required controls:**
+  a committed deliberately-failing fixture plus `bundles > 0`; the
+  comparison hash read *out of the bundle*, written by the first process;
+  truncate-by-one-byte and corrupt-length-prefix negative controls; and a
+  mutated-seed negative control so the round-trip is capable of failing.
+
+---
+
+## 6. Prediction
+
+Stated as a mechanism per CB-WP-0004 T06, and with the refuting outcome
+named — v1's prediction was rejected for being unfalsifiable *and* already
+determined (any tag-based kernel count is **0 of 18** today, computable
+before writing a line).
+
+| claim | mechanism after the fix | refuted if |
+|---|---|---|
+| absent rules become visible | spec→code runs over every numbered spec × every crate; no manual path | any numbered rule in any spec is unnamed in source after the pass |
+| **present-but-inert rules become visible** | mutation: each acceptance row's property is inverted and the suite must go red | **fewer than 9 of the 12 acceptance rows have a mutation that turns a test red** |
+| replay is real | acceptance test re-executes a bundle in a **fresh process** against a hash read from the bundle | the replayed hash is recomputed in-process, or the round-trip cannot be made to fail |
+| K9/K11 are enforced | snapshot-at-N + events N+1..M ≡ genesis fold on `GroundState`; truncation rejected | either passes when mutated |
+
+The middle row is the one that carries information. **9 of 12 is a
+guess, and it is meant to be beatable in both directions** — if 12 of 12
+mutate red, the instruments were better than this survey claims and the
+finding shrinks to the three absent rules; if 3 of 12 do, the problem is
+larger than the pass is scoped for and the pass should stop and re-plan.