diff --git a/history/260731-replay-kernel-coverage-challenge.md b/history/260731-replay-kernel-coverage-challenge.md new file mode 100644 index 0000000..5d5ee96 --- /dev/null +++ b/history/260731-replay-kernel-coverage-challenge.md @@ -0,0 +1,263 @@ +# Challenge — CB-RES-0004 (replay bundles, durable log, and the rules nobody measures) + +Reviewer: adversarial pass, InnerLoop §Step 2, one round. Separate agent +session, given the survey **and** the harness and the evidence file per +the "numbers" row of the review-target table. +Tree reviewed: `59b73c3`. +Verdict: **not approvable as written.** Four BLOCKING, six MATERIAL, two +MINOR. + +> The survey is right that something is unmeasured. It is wrong about +> what, wrong about how much, and its own baseline contains an instance of +> the exact error class it is diagnosing. The headline finding is +> under-scoped by at least three rules, and the instrument it proposes is +> heavier and less effective than a one-line extension of an instrument +> that already exists in this repo. + +--- + +## 1. BLOCKING — the AM-7 replay row does not measure the half of AM-7 the survey defends as "honest". Instance #6 of the harness-does-nothing class is live in the tree right now. + +The survey §1.3 calls the AM-7 row "honest but easily misread". It is not +honest. AM-7's target is "replay of 100k events ≤ 5 s, **hash-identical**". + +`games/ground/src/lib.rs:2241-2256`: + +```rust +let hash = state_hash_hex(&state); +let elapsed = start.elapsed(); +println!("replay {} events in {:?} ({:.0} events/s), hash {}", ...); +assert!(elapsed.as_secs_f64() < 5.0, "AM-7: 100k replay under 5s"); +``` + +`hash` reaches exactly one place: a `println!`. **There is no assertion on +it.** The Criterion bench is the same shape. + +Worse, the hash *could not* be asserted as written. The log is built +across multiple games (`lib.rs:2231-2235`): + +```rust +while log.len() < target { + if source.outcome.is_some() { games += 1; source = fresh(42 + games); } +``` + +so the "replay" folds events from games seeded 42, 43, 44… into a single +state built from `fresh(42)`. It is not a replay of anything. It is a +fold-rate microbenchmark over a semantically invalid event stream, +reported as `AM-7 replay | met, 2,290×`. + +**Must change:** §1.3 rewritten — AM-7's `hash-identical` clause is +unmeasured, the fold is over a multi-game log, and the evidence verdict is +unearned on that clause. This is a correction to `evidence/CB-EV-0001`, +and it puts AM-7 in the pass scope. + +## 2. BLOCKING — the survey grants K9 as implemented. K9's acceptance property is asserted nowhere. + +K9 requires "`snapshot + remaining events → state` must be hash-identical +to a from-genesis fold". The entire evidence is one unit test +(`cb-events/src/lib.rs:158-167`) that round-trips a +`BTreeMap` with `EventSeq(17)` as a literal. No game +aggregate, no events applied, no from-genesis comparison. And `Snapshot` +has no callers outside its own test. + +The survey established which rules are implemented by reading `pub fn` +signatures — presence-of-API, which is exactly the tag-counting it +condemns in §5. **On the acceptance criterion §2.4 is not "two of three", +it is zero of three.** + +## 3. BLOCKING — the "below 100%" prediction is arithmetic, not a prediction, and §6 is constructed so no outcome can refute it. + +Below 100% → thesis confirmed. Near 100% → "the instrument measures tags", +thesis confirmed. No stated result counts against the survey — the +unfalsifiable framing the survey claims to guard against, dressed in the +vocabulary of falsifiability. + +It is also not a prediction: every `covers:` id in the corpus is `GR-*` +(58 ids, zero `K*`), and `ls scenarios/` returns one directory. Any +tag-counting extension **must** report **0 of 18 (0%)** on first run. +Computable today, before writing a line. + +**Must change:** a prediction that can fail, with the refuting outcome +named. Delete the §6 escape clause. + +## 4. BLOCKING — K10 is not the only unimplemented kernel rule. K14 and K18 are too, and the one-line check that finds all three already exists in this repo. + +``` +spec: K1..K18 +code: K1..K9 K11 K12 K13 K15 K16 K17 +diff: K10, K14, K18 +``` + +**K14** ("one commit window per Select") — `CommitWindow` has zero +non-test users and `games/ground` does not import it. GROUND collects +selections in its own aggregate. A dead abstraction certified by a test +that exercises only itself. + +**K18** ("Criterion benches driving the same scenario format") — the +bench hardcodes commands in Rust and never touches `ScenarioFile`. And +`ls -R benchmarks` contains only `baselines/model-prices.toml`. + +**Why this is BLOCKING.** The check that found these is a two-line +generalization of **M-D1-LNK / AM-1b**, which this repo already runs every +`make coverage`. Pointing `parse_code_ids` at `specs/GameKernel.md` × +`crates/**` would have surfaced all three the day AM-1b shipped. The +survey diagnoses a gap a cheap existing instrument covers and prescribes a +new expensive capability as the remedy — and, having chosen the expensive +framing, found 1 of the 3 gaps the cheap check finds in one line. + +**Must change:** rescope to K10/K14/K18 (K9, K11 partial). Add **Option E: +extend AM-1b's spec→code link to every numbered spec and every crate**, +and argue why the tier-L work is warranted *on top of* it. + +## 5. MATERIAL — K-rules cannot be covered by the mechanism that covers GR-rules. "A parameterization, not a rewrite" is false. + +The M-D1-COV numerator comes from `covers:` lists consumed by a runner +requiring `ScenarioGame: Aggregate + Serialize + DeserializeOwned`, with +dispatch by game prefix and `ground` the only arm. There is no kernel +aggregate, setup preset, or command vocabulary. `scenarios/kernel/*.yaml` +would be a tag in a directory the runner cannot dispatch — the inflation +§5 forbids. Second blocker: `AGGREGATE = "games/ground/src/lib.rs"` is +one file, so every K-rule reports "unlinked" forever. + +## 6. MATERIAL — the AM-11 argument, on which tier L rests, misreads a target as a claim and misses the actual overclaim next to it. + +Nobody claims what the survey says. `GameKernel.md:175` is the **target** +column, and the evidence reports `AM-11 impl pairs | 1 of 1 (KernelRng) | +met, narrow`, adding "will mean more once storage has one". The evidence +already says what the survey accuses it of hiding. + +The **real** overclaim is one clause away: M-D4-SWAP requires impls +"passing **the same conformance suite**", and no conformance suite exists +— one doc comment describing future work. The RNG pair is exercised by two +separate non-shared tests. The `met, narrow` verdict is unearned on the +suite clause. + +## 7. MATERIAL — Option C is rejected against the wrong budget line. + +AM-4a (1.5% headroom) is the **shipped-runtime** budget. The bundle is +written by the scenario runner on failure, and scenario tooling is +explicitly dev-only. A store behind the `scenarios` feature is charged to +**AM-4b**, headroom **32,979 lines (9.4%)** — nine times larger, and +comfortably above redb-class crates. The conclusion may survive on +simplicity, but ADR-0004 exists precisely to stop AM-4 numbers being used +as post-hoc justification, and this is that. + +## 8. MATERIAL — what `make replay-test` prints if it does nothing: `ok`. + +Four vacuous passes, none controlled for: + +| silent failure | what it prints | +|---|---| +| no scenario fails (all 21 pass today) → zero bundles round-tripped | `0 bundles verified — ok` | +| writer emits nothing, reader tolerates; identical error strings satisfy "same failure" | `PASS — failure reproduced` | +| replayed hash compared to one recomputed in the same process — `assert_eq!(h, h)`, the AM-7 defect exactly | `PASS — bit-identical` | +| truncation path never exercised; K11's operative clause is dead code | `PASS` | + +**Required controls:** a deliberately-failing fixture scenario plus +`bundles > 0`; the comparison hash read **out of the bundle**, not +recomputed; truncate-by-one-byte and corrupt-length-prefix negative +controls; and a mutated-seed negative control so the round-trip can fail. + +## 9. MATERIAL — the extended `make coverage` contradicts §5 on day one. + +`rule-coverage.py:195` returns `2` when `missing` is non-empty, and +`coverage` is in `all` with no `-` prefix. Adding 18 uncovered K-rules +fails `make all` immediately, which §5 bullet 2 forbids. Requires a +separate kernel denominator outside the exit code, plus a decision on when +it starts binding. + +For the record: the *existing* arm cannot silently do nothing — +`rule-coverage.py:121-129` refuses to report over zero rules, and that +control is self-tested. **The new denominator must replicate it**, or a +kernel regex matching nothing reports `0/0 (100%)`. + +## 10. MATERIAL — the survey says the scoped-denominator problem is general, then sweeps in one place. I swept. + +**(a) AM-10 is vacuous.** `M-D4-LEAK: foreign types in cb-*-api-visible +signatures | 0`. There is no `cb-*-api` crate — true over an empty set. +What is measured is a `clippy.toml` deny of `HashMap`/`HashSet` citing +**K6 (determinism)**, reported under a **D4 leak** row. Stronger than +AM-1 as an instance of the survey's own thesis. + +**(b) AM-1b is measured and dropped from the scoreboard.** `make coverage` +prints `49/58` and MetricsAndScenarios records it **Unmet**; +`grep -c 'AM-1b' evidence/CB-EV-0001` returns **0**. The artifact carrying +the headline reports the flattering half and omits the unflattering half +the same command prints two lines below. + +**(c) M-D1-LNK's own denominator is one file.** + +**(d) AM-2, AM-3, AM-5, AM-9 are unreported** and unmentioned by the +survey. + +**Must change:** generalize §1.2 to a swept table, lead with (a). The +history file's "n=1" weakness can be retired: it is n≥4 in this repo. + +## 11. MINOR — D2 understates the bundle writer. + +`scenario.rs:139` creates `EventLog`, appends at :157, and **never reads +it** — not returned in `Pass`, not iterated, dropped at end of `execute`. +The one production instantiation of the K11 log is a write-only sink. +`Pass` carries end state, not an initial snapshot. Failures are a +formatted `String`, not structured expected-vs-actual. The bundle needs a +change to the runner's data flow, not "a directory of four files". + +## 12. MINOR — "the number is not false" is itself false. + +M-D1-COV is defined as "**numbered spec rules** covered by ≥1 passing +scenario", and `covers` is documented as "numbered rules from the +capability spec" with a non-GR example. The instrument is **non-conformant +with its own metric definition**; `58/58 (100%)` is a wrong answer to the +question M-D1-COV asks. The concession "the number is not false" gives +away the finding — it lets a future reader conclude only the reading was +careless. + +--- + +# What I could not break + +1. **No write path in the workspace.** Wider idiom set than the author's + (`OpenOptions`, `create_dir`, `tempfile`, `to_writer`, `std::io::Write`) + over the whole tree including `#[cfg(test)]` and benches. Every hit is a + `Display::fmt` `write!` or `read_to_string`. Holds. +2. **No `--replay` flag and no flag parsing.** Read all 88 lines of + `cb-sim/src/main.rs`. The survey's stronger phrasing is accurate. +3. **No `.cbreplay` artifact, reader, writer, or directory.** Confirmed + across `.rs`, `.py`, `Makefile`, CI. +4. **18 K-rules, zero in coverage scope.** Verified two ways; `RULE_RE` + and input read from source, not inferred from output. +5. **AM-4a headroom is 1.5%.** Number right; applicability disputed (7). +6. **`make coverage` reports 58/58 (100%) over 21 scenarios.** Reproduced. +7. **No counter-example to the absence claims.** Hunted for a K-rule in a + scenario, a replay entry behind a feature flag, a bundle writer in CI, + a replay round-trip test. None exists. + +# Sampling limits of this review + +1. **The "no write path" claim has one sample and my check does not fix + that.** The workspace contains **ten** `.rs` files total; the author's + sample *is* the population. My independence is in the idiom list and in + reading whole files, not in file selection. This is the CB-WP-0002 + failure shape — two checks, one population — and I cannot make it + otherwise from inside this repo. +2. **I did not execute the test suite.** `cargo test -- --list` enumerates + 19 tests; I ran none. Findings 1 and 2 come from reading source, not + from mutation testing. A response disputing them should answer with a + mutation test, not re-reading. +3. **Finding 4 rests on grep for rule IDs — the same tag-counting I + criticize.** K14 and K18 have independent structural confirmation. + "K10/K14/K18 are untracked" is solid; "they are the *only* further + unimplemented K-rules" is **not** a claim I make. I did not audit + K1–K9, K12–K17 against their acceptance text one by one. Given K9 and + K14 both failed on close reading, a full audit should find more. +4. **I did not verify the §2 baseline table.** Still cited, by both of us. +5. **AM-12 / CostAccounting were out of scope.** I did not audit + `cb-cost.py` or the price sheet. +6. **One process limit.** I found finding 1 only because a test had + "replay" in its name and I opened it out of curiosity — **not** because + any systematic step pointed there. "Reproduce the number independently" + is satisfiable by re-running `make coverage`, which is exactly what does + *not* find this class. **The procedure should require reading the + assertion behind every quoted acceptance number, not re-running the + command that prints it.** Offered as a loop change; n=1 evidence for a + process claim, the same weakness I charged the survey with in 10. diff --git a/history/260731-replay-kernel-coverage-research.md b/history/260731-replay-kernel-coverage-research.md new file mode 100644 index 0000000..d5ef5f1 --- /dev/null +++ b/history/260731-replay-kernel-coverage-research.md @@ -0,0 +1,92 @@ +# 2026-07-31 — how CB-RES-0004 was conducted + +The unpolished trail, so a later reader can judge how hard the survey was +actually tested. + +## How the finding was reached + +Not by looking for it. CB-WP-0004 closed, `make status` reported every +workplan done and every gate green, and the question was what the next +pass should be. `INTENT.md` implementation order stage 0 lists "replay +and scenario tests" among the headless-GROUND deliverables, so the first +check was whether that was done. + +Sequence of commands, in order: + +```sh +find crates games tools/cb-sim -name '*.rs' | xargs wc -l # 3,443 lines total +grep -n "replay\|cbreplay\|--replay" specs/GameKernel.md # K10 found +ls replays/ # no such directory +grep -rn "cbreplay\|--replay" crates tools --include=*.rs # zero matches +grep -n "pub fn\|pub struct" crates/cb-events/src/lib.rs # K9 present, K10/K11 absent +grep -rn "File::\|fs::\|write_all\|BufWriter" crates games tools/cb-sim --include=*.rs +``` + +The last one is the load-bearing check and it is worth stating what it +found: **the only `fs::` call in the entire workspace is +`read_to_string` for scenario YAML.** There is no write path anywhere. A +"durable log format" rule (K11) is being satisfied by a `Vec`. + +## What was measured vs cited + +**Measured on the tree at `59b73c3`:** + +- absence of `.cbreplay` reader/writer — grep, zero matches +- absence of a `--replay` flag — read `tools/cb-sim/src/main.rs` in full; + it collects `args` and treats every one as a scenario path. There is no + flag parsing at all, so the flag is not merely missing, there is no + place to put it +- absence of any file-write path — grep for five different write idioms +- 18 K-rules in `specs/GameKernel.md` (`grep -o "K[0-9]\+" | sort -u`) +- `rule-coverage.py` `RULE_RE = \*\*(GR-[A-Z]+\d+)` against + `specs/GroundRules.md` only — read the source, not inferred from output + +**Cited, not measured:** every row of the state-of-the-art table in §2. +boardgame.io, Rune, rr/Pernosco, Playwright and Kafka/EventStore are +described from prior knowledge. None was run. Per MetricsAndScenarios §3 +this caps any comparison verdict at `parity`, and the survey says so. + +This is a real weakness of the survey and is not hidden: the *gap* +finding is measured and the *design* recommendation rests on cited +precedent. + +## Dead ends + +- **Looking for the gap in the scenario corpus first.** 21 scenarios, 58 + GR-rules, 100% covered — the corpus looks complete and says nothing + about the kernel. Roughly ten minutes were spent reading scenario names + before it became clear the coverage instrument's *denominator* was the + question, not its numerator. +- **Assuming AM-7 covered replay.** `evidence/CB-EV-0001` reports + `AM-7 replay | 100k events ≤5s | 2.18 ms | met, 2,290×`, which reads + like replay is done and fast. It measures an in-memory fold — the K9 + path. The evidence is honest (it even documents that this benchmark was + the fourth harness-does-nothing instance) but the row's wording carried + me past the gap on first reading. Worth recording because a future + reader will hit the same row. +- **Considering an embedded store (sled/redb) for K11.** Dropped once + ADR-0004's 1.5% AM-4a headroom was recalled: a new shipped-runtime + dependency now requires an argued decision, and the format is a + paragraph. + +## The chaos roll + +`shuf -i 1-4 -n 1` → **1**. Structural tier L (new capability port; +changes the meaning of a headline metric). No override. Recorded per the +rule that the roll is written down even when it changes nothing. This is +declaration **1** of the 12-declaration d4 calibration window opened by +CB-WP-0003 T06. + +## Known weaknesses of this survey, stated before review + +1. **All baseline rows are cited.** No competitor was run. +2. **The author found the gap and also proposes the fix**, so the + framing ("coverage scope is the root cause") is the author's and has + not been independently attacked. +3. **The predicted outcome — kernel coverage below 100% on first run — + is unfalsified.** The instrument that would test it has not been + written yet, which is exactly the shape of claim this project has + been wrong about before. +4. **The single-sample problem applies.** One repo, one spec pair. The + claim "an instrument reports a percentage over the sample it was + pointed at" is general, but the evidence for it here is n=1. diff --git a/history/260731-replay-kernel-coverage-response.md b/history/260731-replay-kernel-coverage-response.md new file mode 100644 index 0000000..18710ab --- /dev/null +++ b/history/260731-replay-kernel-coverage-response.md @@ -0,0 +1,160 @@ +# 2026-07-31 — response to the CB-RES-0004 challenge + +**All twelve findings conceded.** Nothing was defended on its facts, +because I re-derived every load-bearing claim myself and the reviewer was +right on each one. Two findings the reviewer explicitly marked as +*read-only, not mutation-tested* are now mutation-proven below — that was +the right bar to set and it deserved to be met rather than argued around. + +The survey is withdrawn and rewritten. What follows is what changed and, +more importantly, the one thing the round produced that neither the survey +nor the challenge had. + +--- + +## The mutation tests the challenge asked for + +> *"A response that disputes findings 1 or 2 should be answered with a +> mutation test, not with re-reading."* — challenge, sampling limit 2 + +I am not disputing them. I ran the mutations anyway, because a conceded +finding backed by evidence is worth more to the next reader than a +conceded finding backed by agreement. + +**Finding 1 — AM-7.** Changed the fold's genesis from `fresh(42)` to +`fresh(999)`, so the 100k-event log is replayed into a completely +different starting state: + +``` +MUTATION: fold starts from fresh(999) — a different genesis state. +test replay_probe::replay_100k_events_is_linear_and_fast ... ok +test result: ok. 1 passed; 0 failed +``` + +**The test passes.** AM-7's `hash-identical` clause detects nothing at +all. The reviewer read this correctly from source; it is now demonstrated. + +**Finding 2 — K9.** Changed `Snapshot::take` to discard the `EventSeq` it +is handed and store `EventSeq(0)`, destroying the half of K9 that says a +snapshot is "the full aggregate **+ the EventId it includes**": + +``` +MUTATION: Snapshot::take DISCARDS its EventSeq and stores 0. +test tests::snapshot_roundtrip ... ok +test result: ok. 3 passed; 0 failed +``` + +**The test passes.** K9's `through` field is unverified. + +## What the round actually produced + +The challenge's finding 4 says the root cause is cheaper than the survey +claimed: extend AM-1b's spec→code link to every numbered spec and every +crate, and K10/K14/K18 fall out in one line. That is correct and it is +adopted as Option E, first step of the pass. + +**But findings 1 and 2 prove Option E is not sufficient, and the reviewer +did not connect their own findings.** Sort the seven defects this round +surfaced by whether a name-based check would catch them: + +| defect | named in code? | Option E catches it? | +|---|---|---| +| K10 replay bundle | no | **yes** | +| K14 commit window | no | **yes** | +| K18 bench-from-scenarios | no | **yes** | +| K9 snapshot `through` | **yes** | no | +| K11 durable format | **yes** | no | +| AM-7 hash-identical | **yes** | no | +| AM-11 conformance suite | **yes** | no | + +Four of seven are named in the source and unasserted. So: + +> **Every coverage instrument in this project counts names. None counts +> assertions.** M-D1-COV counts `covers:` tags. M-D1-LNK counts rule-ID +> strings in source. Both answer "is this rule mentioned?" and neither +> answers "does anything fail if this rule is violated?" + +That is the correct root cause. It is neither the survey's story +("coverage scope is too narrow" — true but shallow) nor the challenge's +("the link check is pointed at one file" — true but only covers 3 of 7). +It also subsumes the project's oldest error class: all five prior +harness-does-nothing instances, plus AM-7 and K9 today, are the same +shape — **the name is there, the assertion is not.** + +The instrument this implies is **mutation**, not another grep. A rule is +covered when breaking it breaks a test. That is expensive to do +exhaustively and cheap to do on the acceptance rows, which is exactly +where every serious error in this project has been. + +## The loop change this earns + +The challenge's sampling limit 6 is the sharpest paragraph in the +document: + +> *"I found finding 1 only because a test had 'replay' in its name and I +> opened it out of curiosity — not because any systematic step pointed +> there. 'Reproduce the number independently' is satisfiable by re-running +> `make coverage`, which is exactly what does not find this class."* + +Conceded and adopted. InnerLoop §Step 2's "numbers" row currently says the +reviewer must *reproduce the number independently*. Re-running the command +that prints it satisfies that wording and finds nothing. The row must +require **reading the assertion behind the number, and mutating it** — +state what the harness would report if the property were false, then make +it false and check. + +The reviewer notes this is n=1 evidence for a process claim. It is n=2: +CB-WP-0002's dedup blind spot had the same structure — a verification step +that re-ran the author's method and inherited the author's blindness. The +fix there was an assertion over all data at execution time; the fix here is +mutation. Both replace *re-derivation* with *adversarial execution*. + +## Finding-by-finding disposition + +| # | severity | disposition | +|---|---|---| +| 1 | BLOCKING | conceded, **mutation-proven**. AM-7 correction enters the pass scope; `evidence/CB-EV-0001` needs a correction note. HDN instance #6. | +| 2 | BLOCKING | conceded, **mutation-proven**. §2.4 is zero of three on the acceptance criterion, not two of three. | +| 3 | BLOCKING | conceded fully. The prediction was unfalsifiable *and* already determined (0 of 18). Replaced with a prediction that names its refuting outcome. | +| 4 | BLOCKING | conceded. Rescoped to K10/K14/K18; Option E adopted as step one — **and shown above to be insufficient**, which is the survey's real justification, not the one it gave. | +| 5 | MATERIAL | conceded. "A parameterization, not a rewrite" withdrawn. The K-rule numerator must come from named executable assertions, not `covers:` tags. `AGGREGATE` becomes a list. | +| 6 | MATERIAL | conceded. The AM-11 misreading is deleted. The real overclaim — M-D4-SWAP requires "the same conformance suite" and no suite exists — replaces it, and is a **stronger** basis for the port argument than what I wrote. | +| 7 | MATERIAL | conceded with a clarification the survey owed: the **durable log (K11) is shipped-runtime** and AM-4a applies; the **bundle writer (K10) is dev-only** behind the `scenarios` feature and AM-4b applies. The survey wanted it both ways without saying so. Option C is now rejected on simplicity for the log, and the headroom claim is dropped for the bundle. | +| 8 | MATERIAL | conceded. All four controls adopted verbatim, including the two negative controls I had not thought of (truncate-by-one-byte, mutated seed). | +| 9 | MATERIAL | conceded. The exit-code path is named, and the new denominator gets the zero-rules positive control the existing arm already has. | +| 10 | MATERIAL | conceded. The sweep is folded in with AM-10 as lead example. **AM-10 is the better instance than AM-1**: a K6 determinism lint reported under a D4 leak row, over a crate that does not exist. | +| 11 | MINOR | conceded. D2's estimate corrected: `Pass` extended, `RunOutcome::Failed` restructured, initial snapshot captured, log plumbed out of `execute`. | +| 12 | MINOR | conceded. "The number is not false" deleted. M-D1-COV is defined over "numbered spec rules"; the instrument is non-conformant with its own metric definition and `58/58 (100%)` is a wrong answer to the question the metric asks. | + +## One thing the challenge could not have found + +Running its mutation tests surfaced a limit in **CB-WP-0004 T01**, closed +two hours earlier: + +``` +$ cargo test -p games-ground ... +/bin/bash: line 22: cargo: command not found +``` + +`make env-test` proves every *tool* and every *make target* runs from any +directory with no cargo on PATH. It does not put cargo on the agent's +interactive PATH, so an ad-hoc `cargo` invocation still fails. T01's +evidence claimed environment friction went from 85 turns to 1; that number +stands for the measured window, but the fix is narrower than "no +environment setup", and CB-EV-0003 should say so. + +Small, and recorded rather than quietly patched, because CB-WP-0004 T05's +whole finding was that tooling only recovers capacity when the manual path +stops existing — and here it did not stop existing. + +## Approval status + +**Not approved.** The survey is withdrawn and rewritten as +`research/CB-RES-0004` v2 incorporating all twelve findings, the corrected +root cause, and a rescoped pass that now includes AM-7, K9, K14, K18, +AM-10 and AM-11 — none of which were in the original scope. + +The review cost is recorded in the pass evidence. On the two prior passes +this step cost $0.66 and $1.11 and each time found an approval-blocking +defect. This time it found four, one of which invalidates a committed +`met` verdict in `evidence/CB-EV-0001`. diff --git a/research/CB-RES-0004-replay-and-kernel-coverage.md b/research/CB-RES-0004-replay-and-kernel-coverage.md new file mode 100644 index 0000000..dd0a92b --- /dev/null +++ b/research/CB-RES-0004-replay-and-kernel-coverage.md @@ -0,0 +1,272 @@ +# CB-RES-0004: the instruments count names, not assertions + +capability: meta.coverage.assertions + kernel.replay +status: approved (v2, after adversarial review — v1 was rejected with four +blocking findings) +tier: L (structural L — creates a new capability port (durable log +storage); changes what several headline acceptance numbers mean; chaos +d4=1 → no override) +instrument: extended `make coverage` (spec→code over every numbered spec) +plus a new `make mutation-check` and `make replay-test` +baseline: cited — boardgame.io, Rune, rr/Pernosco, Playwright trace, +event-sourcing log framing + +Survey opened 2026-07-31. **v1 was rejected**; the review trail is +`history/260731-replay-kernel-coverage-{research,challenge,response}.md`. +The finding below is the reviewer's as much as mine. + +--- + +## 1. The finding + +> **Every coverage instrument in this project counts names. None counts +> assertions.** M-D1-COV counts `covers:` tags in scenario YAML. M-D1-LNK +> counts rule-ID strings in one source file. Both answer *"is this rule +> mentioned?"*; neither answers *"does anything fail if this rule is +> violated?"* + +Seven defects were found this pass. Four of them are **named in the source +and unasserted** — invisible to any name-based check, including the +cheaper one the reviewer proposed: + +| defect | named in code? | a name-based check finds it? | +|---|---|---| +| K10 replay bundle — no writer, reader, or `--replay` | no | **yes** | +| K14 commit window — `CommitWindow` has zero non-test users | no | **yes** | +| K18 benches from scenarios — bench hardcodes commands | no | **yes** | +| K9 snapshot `+ the EventId it includes` | yes | no | +| K11 durable format, truncated-tail detection | yes | no | +| AM-7 `hash-identical` | yes | no | +| AM-11 `passing the same conformance suite` | yes | no | + +This subsumes the project's oldest error class. All five prior +harness-does-nothing instances have the same shape as AM-7 and K9 today: +**the name is there, the assertion is not.** + +### 1.1 The two that are mutation-proven + +**AM-7's `hash-identical` clause detects nothing.** +`games/ground/src/lib.rs` computes `state_hash_hex(&state)` and passes it +to a `println!`. The only assertion is on elapsed time. Mutating the +fold's genesis from `fresh(42)` to `fresh(999)` — replaying a 100k-event +log into an unrelated starting state — leaves the test green: + +```text +test replay_probe::replay_100k_events_is_linear_and_fast ... ok +``` + +The hash could not be asserted as written anyway: the log is built across +games seeded 42, 43, 44… (`source = fresh(42 + games)`) and folded into a +state from `fresh(42)`. It is a fold-rate microbenchmark over a +semantically invalid stream, reported in `evidence/CB-EV-0001` as +**`AM-7 replay | met, 2,290×`**. That verdict is unearned on the +hash clause and needs a correction note. **HDN instance #6.** + +**K9's `through` field is unverified.** The whole evidence for K9 is one +test round-tripping a `BTreeMap` with `EventSeq(17)` as a +literal — no game aggregate, no events applied, no from-genesis +comparison. `Snapshot` has no callers outside that test. Mutating +`Snapshot::take` to discard its `EventSeq` and store `EventSeq(0)` leaves +the test green. + +On the acceptance criterion, **GameKernel §2.4 is zero of three, not two +of three.** + +### 1.2 The three that are named nowhere + +```text +spec: K1..K18 code: K1..K9 K11 K12 K13 K15 K16 K17 +diff: K10, K14, K18 +``` + +- **K10** — no `.cbreplay` writer, reader, or `replays/` directory; the + only `fs::` call in the workspace is `read_to_string` for scenario YAML. + `cb-sim` has no flag parsing at all, so `--replay` has no place to go. + `MetricsAndScenarios` §2 promises a failing run writes a bundle. It does + not. INTENT names this design decision **8 of 10**. +- **K14** — "one commit window per Select". `CommitWindow` has zero + non-test users and `games/ground` does not import it; GROUND collects + selections in its own aggregate. A dead abstraction certified by a test + that exercises only itself. +- **K18** — "Criterion benches driving the same scenario format". The + bench hardcodes commands in Rust and never touches `ScenarioFile`. + `benchmarks/` contains only `baselines/model-prices.toml`. + +### 1.3 The scoped-denominator sweep + +v1 claimed this problem was general and swept in one place. Swept properly, +in descending order of severity: + +**AM-10 is vacuous — the population is empty.** The row reads +*M-D4-LEAK: foreign types in `cb-*-api`-visible signatures | 0*. There is +no `cb-*-api` crate. What is actually measured is a `clippy.toml` deny of +`HashMap`/`HashSet` whose stated reason cites **K6 (determinism)**, +reported under a **D4 leak** row. A determinism lint standing in for a +leak metric, over a crate that does not exist. + +**AM-1's denominator is one spec, and the metric says otherwise.** +M-D1-COV is defined as *"numbered spec rules covered by ≥1 passing +scenario"*, and `covers` is documented as "numbered rules from the +capability spec" with a non-GR example. `RULE_RE` matches `GR-` only, +against `GroundRules.md` only. So `58/58 (100%)` is not merely narrowly +scoped — the instrument is **non-conformant with its own metric +definition**, and 100% is a wrong answer to the question M-D1-COV asks. + +**AM-1b is measured and dropped from the scoreboard.** `make coverage` +prints `49/58` and `MetricsAndScenarios` records it **Unmet**; +`grep -c 'AM-1b' evidence/CB-EV-0001` returns **0**. The artifact carrying +the headline reports the flattering half of the gate and omits the +unflattering half the same command prints two lines below it. + +**AM-11's suite does not exist.** M-D4-SWAP requires impls "passing **the +same conformance suite**". `grep -rn conformance` over all `.rs` returns +one doc comment describing future work. The RNG pair is exercised by two +separate, non-shared tests. `met, narrow` is unearned on the suite clause. + +**AM-2, AM-3, AM-5, AM-9 are unreported** and were unmentioned by v1. + +The history file's stated weakness — *"the evidence for this is n=1"* — is +retired. It is n≥5 in this repo, found without leaving the tree. + +--- + +## 2. Baseline — what the state of the art does + +Cited, not locally measured; per MetricsAndScenarios §3 this caps +comparison verdicts at `parity`. + +| system | durable log | exportable failure bundle | machine re-executable | +|---|---|---|---| +| **boardgame.io** | in-memory, optional server persistence | debug panel, in-session time travel | no | +| **Rune** | rollback ring buffer, transient | no | no | +| **rr / Pernosco** | full process recording | yes — the gold standard | **yes**, bit-exact | +| **Playwright trace** | per-run trace zip | **yes**, one file | partly (UI, not logic) | +| **Kafka / EventStore** | append-only, length-prefixed, versioned, checksummed | n/a | yes, by re-consumption | + +Two lessons: + +1. **The valuable property is machine re-execution, not inspection.** + boardgame.io's debug panel is more visible than a `.cbreplay` file and + far less useful to an agent, because a human must interpret it. rr and + Playwright are the precedents worth taking. +2. **The log format is solved engineering.** Length prefix, version byte, + per-record framing, tail detection: Kafka and EventStore converged on + the same shape decades apart. It is ~80–120 lines. + +**Provenance.** Log framing: `adapted:event-sourcing-log-framing` — +length-prefixed versioned records with tail detection, dropping the +distributed concerns (partitions, offsets, compaction) a single-game log +does not have. Bundle: `adapted:playwright-trace` — one artifact per +failed run containing everything needed to reproduce, adapted so the +consumer is an agent, not a human viewer. Assertion coverage: +`adapted:mutation-testing` — mutate the property, expect a red test; +applied only to acceptance rows rather than exhaustively. + +--- + +## 3. The four dimensions + +**D1 — ease of specification.** Strongly positive. A `.cbreplay` bundle +turns "the scenario failed" into an artifact with a stated contract. More +importantly, mutation coverage makes "this rule is enforced" a *claim with +a test* rather than a tag. + +**D2 — efficiency of implementation.** The log format adds no dependency +(serde is already shipped). **The bundle writer is not "a directory of +four files"** — v1 said that and it was wrong. `scenario.rs` creates an +`EventLog`, appends to it, and never reads it; it is dropped at the end of +`execute`. `Pass` carries the *end* state, not an initial snapshot, and +failures are a formatted `String`, not structured expected-vs-actual. The +bundle requires changing the runner's data flow: plumb the log out, capture +an initial snapshot, restructure `RunOutcome::Failed`. + +**D3 — speed of execution.** Bundle writes happen only on failure, so the +hot path is untouched. Log append gains a length-prefix write. Mutation +checks run in CI, not in the loop. + +**D4 — optionality.** Where the pass earns its L, on a corrected premise. +v1 argued AM-11 "claims" a storage port; it does not — that is the target +column, and the evidence honestly reports `1 of 1, met, narrow`. The real +argument is stronger: **M-D4-SWAP is a bool over a conformance suite that +does not exist.** Building a durable-log port with a genuine shared suite +discharges AM-11 properly and creates the seam stage 3 (networked +sessions) and stage 4 (game-creation framework) both need. + +--- + +## 4. Options + +**A — implement K10/K11, leave the instruments alone.** Rejected. Fixes +what was noticed; leaves the mechanism that hid it. + +**B — extend coverage to K-rules, defer the rest.** Rejected as a final +state. Adopted as *ordering*. + +**C — adopt an embedded store (sled, redb).** Rejected — and v1 rejected +it against the wrong budget. The **durable log (K11) is shipped-runtime**, +so AM-4a applies with 1.5% headroom; the **bundle writer (K10) is dev-only** +behind the `scenarios` feature, so AM-4b applies with 9.4%. v1 wanted it +both ways without saying so. With the split stated, C is rejected on +simplicity for the log — a length prefix is not a hard implementation, and +"assimilate the implementation" is for hard ones — and the headroom +argument is dropped for the bundle, where it never applied. + +**E — extend AM-1b's spec→code link to every numbered spec and every +crate.** ~10 lines against an instrument that already exists. Catches +K10, K14, K18 immediately. **Adopted as step one — and demonstrably +insufficient**, since it catches 3 of the 7 defects and none of the four +mutation-class ones. `AGGREGATE` must become a list. + +**F — E, plus mutation checks on acceptance rows, plus K10/K11.** +**Recommended.** The name-based check is cheap and catches the absent; the +mutation check is the only thing that catches the present-but-inert; the +capability work discharges the rules both instruments expose. + +--- + +## 5. What this pass must not do + +- **Do not tag K-rules as covered to make a number go up.** The overclaim + CB-WP-0001 committed once and M-D1-LNK exists to catch. A K-rule is + covered when breaking it breaks a test. +- **Do not let the widened coverage gate fail the build on day one.** + `rule-coverage.py` returns `2` when `missing` is non-empty and + `coverage` is in `all`, so 18 uncovered K-rules fail `make all` + immediately. The kernel denominator must report outside the exit code + until the pass closes the gap, with a stated date for when it binds. +- **Replicate the zero-rules positive control on the new denominator.** + The existing arm refuses to report over zero rules — a defect it was + fixed for. A kernel regex matching nothing must abort, not print + `0/0 (100%)`. +- **`make replay-test` must not be able to pass vacuously.** It reports + `ok` under at least four silent failures: no scenario fails so zero + bundles are round-tripped; the writer emits nothing and identical error + strings satisfy "same failure"; the replayed hash is compared to one + recomputed in the same process (`assert_eq!(h, h)` — the AM-7 defect + exactly); the truncation path is never exercised. **Required controls:** + a committed deliberately-failing fixture plus `bundles > 0`; the + comparison hash read *out of the bundle*, written by the first process; + truncate-by-one-byte and corrupt-length-prefix negative controls; and a + mutated-seed negative control so the round-trip is capable of failing. + +--- + +## 6. Prediction + +Stated as a mechanism per CB-WP-0004 T06, and with the refuting outcome +named — v1's prediction was rejected for being unfalsifiable *and* already +determined (any tag-based kernel count is **0 of 18** today, computable +before writing a line). + +| claim | mechanism after the fix | refuted if | +|---|---|---| +| absent rules become visible | spec→code runs over every numbered spec × every crate; no manual path | any numbered rule in any spec is unnamed in source after the pass | +| **present-but-inert rules become visible** | mutation: each acceptance row's property is inverted and the suite must go red | **fewer than 9 of the 12 acceptance rows have a mutation that turns a test red** | +| replay is real | acceptance test re-executes a bundle in a **fresh process** against a hash read from the bundle | the replayed hash is recomputed in-process, or the round-trip cannot be made to fail | +| K9/K11 are enforced | snapshot-at-N + events N+1..M ≡ genesis fold on `GroundState`; truncation rejected | either passes when mutated | + +The middle row is the one that carries information. **9 of 12 is a +guess, and it is meant to be beatable in both directions** — if 12 of 12 +mutate red, the instruments were better than this survey claims and the +finding shrinks to the three absent rules; if 3 of 12 do, the problem is +larger than the pass is scoped for and the pass should stop and re-plan.