tegwick
923c2038be
Tier-L survey plus the full review trail. v1 was REJECTED by adversarial
review with four blocking findings; this is v2, and the finding below is
the reviewer's as much as mine.
The root cause is neither v1's story ("coverage scope is too narrow" —
true but shallow) nor the reviewer's cheaper one ("the link check points
at one file" — true but catches 3 of 7). It is:
Every coverage instrument in this project counts NAMES. None counts
ASSERTIONS. M-D1-COV counts covers: tags; M-D1-LNK counts rule-ID
strings. Both answer "is this rule mentioned?" and neither answers
"does anything fail if this rule is violated?"
Four of seven defects found this pass are named in the source and inert,
so no name-based check finds them. This subsumes the oldest error class:
all five prior harness-does-nothing instances have the same shape.
Two are mutation-proven, because the reviewer set that bar for findings
they had only read:
AM-7 — folding a 100k-event log from fresh(999) instead of fresh(42),
an unrelated genesis state, leaves the test green. The hash reaches
only a println!; the sole assertion is on elapsed time. And the log is
built across games seeded 42,43,44... so it could not be asserted as
written. evidence/CB-EV-0001 reports "AM-7 replay met, 2,290x" —
unearned on the hash-identical clause. HDN instance #6.
K9 — making Snapshot::take discard its EventSeq and store 0, which
destroys the half of K9 that says "+ the EventId it includes", leaves
the test green. On the acceptance criterion GameKernel §2.4 is zero of
three, not two of three.
Also found: K10, K14, K18 named nowhere in source; AM-10 vacuous (a K6
determinism lint reported under a D4 leak row, over a cb-*-api crate that
does not exist); AM-11's conformance suite does not exist; AM-1b measured
and omitted from the evidence scoreboard.
All twelve challenge findings conceded, none defended on its facts. v1's
prediction was rejected as unfalsifiable AND already determined; the
replacement names its refuting outcome and is beatable in both
directions.
Chaos d4=1, no override. Declaration 1 of the 12-declaration window.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
92 lines
4.3 KiB
Markdown
92 lines
4.3 KiB
Markdown
# 2026-07-31 — how CB-RES-0004 was conducted
|
||
|
||
The unpolished trail, so a later reader can judge how hard the survey was
|
||
actually tested.
|
||
|
||
## How the finding was reached
|
||
|
||
Not by looking for it. CB-WP-0004 closed, `make status` reported every
|
||
workplan done and every gate green, and the question was what the next
|
||
pass should be. `INTENT.md` implementation order stage 0 lists "replay
|
||
and scenario tests" among the headless-GROUND deliverables, so the first
|
||
check was whether that was done.
|
||
|
||
Sequence of commands, in order:
|
||
|
||
```sh
|
||
find crates games tools/cb-sim -name '*.rs' | xargs wc -l # 3,443 lines total
|
||
grep -n "replay\|cbreplay\|--replay" specs/GameKernel.md # K10 found
|
||
ls replays/ # no such directory
|
||
grep -rn "cbreplay\|--replay" crates tools --include=*.rs # zero matches
|
||
grep -n "pub fn\|pub struct" crates/cb-events/src/lib.rs # K9 present, K10/K11 absent
|
||
grep -rn "File::\|fs::\|write_all\|BufWriter" crates games tools/cb-sim --include=*.rs
|
||
```
|
||
|
||
The last one is the load-bearing check and it is worth stating what it
|
||
found: **the only `fs::` call in the entire workspace is
|
||
`read_to_string` for scenario YAML.** There is no write path anywhere. A
|
||
"durable log format" rule (K11) is being satisfied by a `Vec`.
|
||
|
||
## What was measured vs cited
|
||
|
||
**Measured on the tree at `59b73c3`:**
|
||
|
||
- absence of `.cbreplay` reader/writer — grep, zero matches
|
||
- absence of a `--replay` flag — read `tools/cb-sim/src/main.rs` in full;
|
||
it collects `args` and treats every one as a scenario path. There is no
|
||
flag parsing at all, so the flag is not merely missing, there is no
|
||
place to put it
|
||
- absence of any file-write path — grep for five different write idioms
|
||
- 18 K-rules in `specs/GameKernel.md` (`grep -o "K[0-9]\+" | sort -u`)
|
||
- `rule-coverage.py` `RULE_RE = \*\*(GR-[A-Z]+\d+)` against
|
||
`specs/GroundRules.md` only — read the source, not inferred from output
|
||
|
||
**Cited, not measured:** every row of the state-of-the-art table in §2.
|
||
boardgame.io, Rune, rr/Pernosco, Playwright and Kafka/EventStore are
|
||
described from prior knowledge. None was run. Per MetricsAndScenarios §3
|
||
this caps any comparison verdict at `parity`, and the survey says so.
|
||
|
||
This is a real weakness of the survey and is not hidden: the *gap*
|
||
finding is measured and the *design* recommendation rests on cited
|
||
precedent.
|
||
|
||
## Dead ends
|
||
|
||
- **Looking for the gap in the scenario corpus first.** 21 scenarios, 58
|
||
GR-rules, 100% covered — the corpus looks complete and says nothing
|
||
about the kernel. Roughly ten minutes were spent reading scenario names
|
||
before it became clear the coverage instrument's *denominator* was the
|
||
question, not its numerator.
|
||
- **Assuming AM-7 covered replay.** `evidence/CB-EV-0001` reports
|
||
`AM-7 replay | 100k events ≤5s | 2.18 ms | met, 2,290×`, which reads
|
||
like replay is done and fast. It measures an in-memory fold — the K9
|
||
path. The evidence is honest (it even documents that this benchmark was
|
||
the fourth harness-does-nothing instance) but the row's wording carried
|
||
me past the gap on first reading. Worth recording because a future
|
||
reader will hit the same row.
|
||
- **Considering an embedded store (sled/redb) for K11.** Dropped once
|
||
ADR-0004's 1.5% AM-4a headroom was recalled: a new shipped-runtime
|
||
dependency now requires an argued decision, and the format is a
|
||
paragraph.
|
||
|
||
## The chaos roll
|
||
|
||
`shuf -i 1-4 -n 1` → **1**. Structural tier L (new capability port;
|
||
changes the meaning of a headline metric). No override. Recorded per the
|
||
rule that the roll is written down even when it changes nothing. This is
|
||
declaration **1** of the 12-declaration d4 calibration window opened by
|
||
CB-WP-0003 T06.
|
||
|
||
## Known weaknesses of this survey, stated before review
|
||
|
||
1. **All baseline rows are cited.** No competitor was run.
|
||
2. **The author found the gap and also proposes the fix**, so the
|
||
framing ("coverage scope is the root cause") is the author's and has
|
||
not been independently attacked.
|
||
3. **The predicted outcome — kernel coverage below 100% on first run —
|
||
is unfalsified.** The instrument that would test it has not been
|
||
written yet, which is exactly the shape of claim this project has
|
||
been wrong about before.
|
||
4. **The single-sample problem applies.** One repo, one spec pair. The
|
||
claim "an instrument reports a percentage over the sample it was
|
||
pointed at" is general, but the evidence for it here is n=1.
|