clay-borg/history/260731-replay-kernel-coverage-response.md

2026-07-31 — response to the CB-RES-0004 challenge

All twelve findings conceded. Nothing was defended on its facts, because I re-derived every load-bearing claim myself and the reviewer was right on each one. Two findings the reviewer explicitly marked as read-only, not mutation-tested are now mutation-proven below — that was the right bar to set and it deserved to be met rather than argued around.

The survey is withdrawn and rewritten. What follows is what changed and, more importantly, the one thing the round produced that neither the survey nor the challenge had.

---

The mutation tests the challenge asked for

"A response that disputes findings 1 or 2 should be answered with a mutation test, not with re-reading." — challenge, sampling limit 2

I am not disputing them. I ran the mutations anyway, because a conceded finding backed by evidence is worth more to the next reader than a conceded finding backed by agreement.

Finding 1 — AM-7. Changed the fold's genesis from fresh(42) to fresh(999), so the 100k-event log is replayed into a completely different starting state:

MUTATION: fold starts from fresh(999) — a different genesis state.
test replay_probe::replay_100k_events_is_linear_and_fast ... ok
test result: ok. 1 passed; 0 failed

The test passes. AM-7's hash-identical clause detects nothing at all. The reviewer read this correctly from source; it is now demonstrated.

Finding 2 — K9. Changed Snapshot::take to discard the EventSeq it is handed and store EventSeq(0), destroying the half of K9 that says a snapshot is "the full aggregate + the EventId it includes":

MUTATION: Snapshot::take DISCARDS its EventSeq and stores 0.
test tests::snapshot_roundtrip ... ok
test result: ok. 3 passed; 0 failed

The test passes. K9's through field is unverified.

What the round actually produced

The challenge's finding 4 says the root cause is cheaper than the survey claimed: extend AM-1b's spec→code link to every numbered spec and every crate, and K10/K14/K18 fall out in one line. That is correct and it is adopted as Option E, first step of the pass.

But findings 1 and 2 prove Option E is not sufficient, and the reviewer did not connect their own findings. Sort the seven defects this round surfaced by whether a name-based check would catch them:

defect	named in code?	Option E catches it?
K10 replay bundle	no	yes
K14 commit window	no	yes
K18 bench-from-scenarios	no	yes
K9 snapshot through	yes	no
K11 durable format	yes	no
AM-7 hash-identical	yes	no
AM-11 conformance suite	yes	no

Four of seven are named in the source and unasserted. So:

Every coverage instrument in this project counts names. None counts assertions. M-D1-COV counts covers: tags. M-D1-LNK counts rule-ID strings in source. Both answer "is this rule mentioned?" and neither answers "does anything fail if this rule is violated?"

That is the correct root cause. It is neither the survey's story ("coverage scope is too narrow" — true but shallow) nor the challenge's ("the link check is pointed at one file" — true but only covers 3 of 7). It also subsumes the project's oldest error class: all five prior harness-does-nothing instances, plus AM-7 and K9 today, are the same shape — the name is there, the assertion is not.

The instrument this implies is mutation, not another grep. A rule is covered when breaking it breaks a test. That is expensive to do exhaustively and cheap to do on the acceptance rows, which is exactly where every serious error in this project has been.

The loop change this earns

The challenge's sampling limit 6 is the sharpest paragraph in the document:

"I found finding 1 only because a test had 'replay' in its name and I opened it out of curiosity — not because any systematic step pointed there. 'Reproduce the number independently' is satisfiable by re-running make coverage, which is exactly what does not find this class."

Conceded and adopted. InnerLoop §Step 2's "numbers" row currently says the reviewer must reproduce the number independently. Re-running the command that prints it satisfies that wording and finds nothing. The row must require reading the assertion behind the number, and mutating it — state what the harness would report if the property were false, then make it false and check.

The reviewer notes this is n=1 evidence for a process claim. It is n=2: CB-WP-0002's dedup blind spot had the same structure — a verification step that re-ran the author's method and inherited the author's blindness. The fix there was an assertion over all data at execution time; the fix here is mutation. Both replace re-derivation with adversarial execution.

Finding-by-finding disposition

#	severity	disposition
1	BLOCKING	conceded, mutation-proven. AM-7 correction enters the pass scope; evidence/CB-EV-0001 needs a correction note. HDN instance #6.
2	BLOCKING	conceded, mutation-proven. §2.4 is zero of three on the acceptance criterion, not two of three.
3	BLOCKING	conceded fully. The prediction was unfalsifiable and already determined (0 of 18). Replaced with a prediction that names its refuting outcome.
4	BLOCKING	conceded. Rescoped to K10/K14/K18; Option E adopted as step one — and shown above to be insufficient, which is the survey's real justification, not the one it gave.
5	MATERIAL	conceded. "A parameterization, not a rewrite" withdrawn. The K-rule numerator must come from named executable assertions, not covers: tags. AGGREGATE becomes a list.
6	MATERIAL	conceded. The AM-11 misreading is deleted. The real overclaim — M-D4-SWAP requires "the same conformance suite" and no suite exists — replaces it, and is a stronger basis for the port argument than what I wrote.
7	MATERIAL	conceded with a clarification the survey owed: the durable log (K11) is shipped-runtime and AM-4a applies; the bundle writer (K10) is dev-only behind the scenarios feature and AM-4b applies. The survey wanted it both ways without saying so. Option C is now rejected on simplicity for the log, and the headroom claim is dropped for the bundle.
8	MATERIAL	conceded. All four controls adopted verbatim, including the two negative controls I had not thought of (truncate-by-one-byte, mutated seed).
9	MATERIAL	conceded. The exit-code path is named, and the new denominator gets the zero-rules positive control the existing arm already has.
10	MATERIAL	conceded. The sweep is folded in with AM-10 as lead example. AM-10 is the better instance than AM-1: a K6 determinism lint reported under a D4 leak row, over a crate that does not exist.
11	MINOR	conceded. D2's estimate corrected: Pass extended, RunOutcome::Failed restructured, initial snapshot captured, log plumbed out of execute.
12	MINOR	conceded. "The number is not false" deleted. M-D1-COV is defined over "numbered spec rules"; the instrument is non-conformant with its own metric definition and 58/58 (100%) is a wrong answer to the question the metric asks.

One thing the challenge could not have found

Running its mutation tests surfaced a limit in CB-WP-0004 T01, closed two hours earlier:

$ cargo test -p games-ground ...
/bin/bash: line 22: cargo: command not found

make env-test proves every tool and every make target runs from any directory with no cargo on PATH. It does not put cargo on the agent's interactive PATH, so an ad-hoc cargo invocation still fails. T01's evidence claimed environment friction went from 85 turns to 1; that number stands for the measured window, but the fix is narrower than "no environment setup", and CB-EV-0003 should say so.

Small, and recorded rather than quietly patched, because CB-WP-0004 T05's whole finding was that tooling only recovers capacity when the manual path stops existing — and here it did not stop existing.

Approval status

Not approved. The survey is withdrawn and rewritten as research/CB-RES-0004 v2 incorporating all twelve findings, the corrected root cause, and a rescoped pass that now includes AM-7, K9, K14, K18, AM-10 and AM-11 — none of which were in the original scope.

The review cost is recorded in the pass evidence. On the two prior passes this step cost $0.66 and $1.11 and each time found an approval-blocking defect. This time it found four, one of which invalidates a committed met verdict in evidence/CB-EV-0001.