tegwick 923c2038be CB-RES-0004: the instruments count names, not assertions

Tier-L survey plus the full review trail. v1 was REJECTED by adversarial
review with four blocking findings; this is v2, and the finding below is
the reviewer's as much as mine.

The root cause is neither v1's story ("coverage scope is too narrow" —
true but shallow) nor the reviewer's cheaper one ("the link check points
at one file" — true but catches 3 of 7). It is:

  Every coverage instrument in this project counts NAMES. None counts
  ASSERTIONS. M-D1-COV counts covers: tags; M-D1-LNK counts rule-ID
  strings. Both answer "is this rule mentioned?" and neither answers
  "does anything fail if this rule is violated?"

Four of seven defects found this pass are named in the source and inert,
so no name-based check finds them. This subsumes the oldest error class:
all five prior harness-does-nothing instances have the same shape.

Two are mutation-proven, because the reviewer set that bar for findings
they had only read:

  AM-7 — folding a 100k-event log from fresh(999) instead of fresh(42),
  an unrelated genesis state, leaves the test green. The hash reaches
  only a println!; the sole assertion is on elapsed time. And the log is
  built across games seeded 42,43,44... so it could not be asserted as
  written. evidence/CB-EV-0001 reports "AM-7 replay met, 2,290x" —
  unearned on the hash-identical clause. HDN instance #6.

  K9 — making Snapshot::take discard its EventSeq and store 0, which
  destroys the half of K9 that says "+ the EventId it includes", leaves
  the test green. On the acceptance criterion GameKernel §2.4 is zero of
  three, not two of three.

Also found: K10, K14, K18 named nowhere in source; AM-10 vacuous (a K6
determinism lint reported under a D4 leak row, over a cb-*-api crate that
does not exist); AM-11's conformance suite does not exist; AM-1b measured
and omitted from the evidence scoreboard.

All twelve challenge findings conceded, none defended on its facts. v1's
prediction was rejected as unfalsifiable AND already determined; the
replacement names its refuting outcome and is beatable in both
directions.

Chaos d4=1, no override. Declaration 1 of the 12-declaration window.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-07-31 14:19:30 +02:00

4.3 KiB

Raw Blame History

2026-07-31 — how CB-RES-0004 was conducted

The unpolished trail, so a later reader can judge how hard the survey was actually tested.

How the finding was reached

Not by looking for it. CB-WP-0004 closed, make status reported every workplan done and every gate green, and the question was what the next pass should be. INTENT.md implementation order stage 0 lists "replay and scenario tests" among the headless-GROUND deliverables, so the first check was whether that was done.

Sequence of commands, in order:

find crates games tools/cb-sim -name '*.rs' | xargs wc -l   # 3,443 lines total
grep -n "replay\|cbreplay\|--replay" specs/GameKernel.md    # K10 found
ls replays/                                                  # no such directory
grep -rn "cbreplay\|--replay" crates tools --include=*.rs    # zero matches
grep -n "pub fn\|pub struct" crates/cb-events/src/lib.rs     # K9 present, K10/K11 absent
grep -rn "File::\|fs::\|write_all\|BufWriter" crates games tools/cb-sim --include=*.rs

The last one is the load-bearing check and it is worth stating what it found: the only fs:: call in the entire workspace is read_to_string for scenario YAML. There is no write path anywhere. A "durable log format" rule (K11) is being satisfied by a Vec.

What was measured vs cited

Measured on the tree at 59b73c3:

absence of .cbreplay reader/writer — grep, zero matches
absence of a --replay flag — read tools/cb-sim/src/main.rs in full; it collects args and treats every one as a scenario path. There is no flag parsing at all, so the flag is not merely missing, there is no place to put it
absence of any file-write path — grep for five different write idioms
18 K-rules in specs/GameKernel.md (grep -o "K[0-9]\+" | sort -u)
rule-coverage.py RULE_RE = \*\*(GR-[A-Z]+\d+) against specs/GroundRules.md only — read the source, not inferred from output

Cited, not measured: every row of the state-of-the-art table in §2. boardgame.io, Rune, rr/Pernosco, Playwright and Kafka/EventStore are described from prior knowledge. None was run. Per MetricsAndScenarios §3 this caps any comparison verdict at parity, and the survey says so.

This is a real weakness of the survey and is not hidden: the gap finding is measured and the design recommendation rests on cited precedent.

Dead ends

Looking for the gap in the scenario corpus first. 21 scenarios, 58 GR-rules, 100% covered — the corpus looks complete and says nothing about the kernel. Roughly ten minutes were spent reading scenario names before it became clear the coverage instrument's denominator was the question, not its numerator.
Assuming AM-7 covered replay. evidence/CB-EV-0001 reports AM-7 replay | 100k events ≤5s | 2.18 ms | met, 2,290×, which reads like replay is done and fast. It measures an in-memory fold — the K9 path. The evidence is honest (it even documents that this benchmark was the fourth harness-does-nothing instance) but the row's wording carried me past the gap on first reading. Worth recording because a future reader will hit the same row.
Considering an embedded store (sled/redb) for K11. Dropped once ADR-0004's 1.5% AM-4a headroom was recalled: a new shipped-runtime dependency now requires an argued decision, and the format is a paragraph.

The chaos roll

shuf -i 1-4 -n 1 → 1. Structural tier L (new capability port; changes the meaning of a headline metric). No override. Recorded per the rule that the roll is written down even when it changes nothing. This is declaration 1 of the 12-declaration d4 calibration window opened by CB-WP-0003 T06.

Known weaknesses of this survey, stated before review

All baseline rows are cited. No competitor was run.
The author found the gap and also proposes the fix, so the framing ("coverage scope is the root cause") is the author's and has not been independently attacked.
The predicted outcome — kernel coverage below 100% on first run — is unfalsified. The instrument that would test it has not been written yet, which is exactly the shape of claim this project has been wrong about before.
The single-sample problem applies. One repo, one spec pair. The claim "an instrument reports a percentage over the sample it was pointed at" is general, but the evidence for it here is n=1.

4.3 KiB Raw Blame History Unescape Escape