tegwick
53c460c992
Duplicated-fact drift is the fourth error class and the only one with no executable rule. No positive control catches it (both copies are internally consistent) and re-derivation does not either (the copy reproduces whatever it was copied from). It is caught only by reading a copy against its source, which nothing in the loop required. facts.toml holds 15 facts and is GENERATED by `make facts-gen` from cb-cost, dep-weight and rule-coverage. The trap this task named — a hand-maintained registry that becomes another drifting copy — is closed by facts-check re-running the instruments and failing when the committed registry disagrees with them. A stale registry cannot certify stale artifacts. An artifact quoting a fact tags it: **$93.15** <!-- fact --> with the key. 17 occurrences across 5 artifacts are now checked. Falsified before being believed: changing CostAccounting.md line 158 from $93.15 to $92.87 — the exact historical drift — produced exit 1 naming the file, the line and the expected value. Tested against the class it exists to catch, on a real artifact, not only in its self-test. It then caught a live tag inside its own documentation example in InnerLoop.md within the hour. Third time a gate has failed on its own pass's work. What it does not close is stated rather than implied: 22 untagged literal copies remain and are reported, not failed. Tagging is opt-in, a number can legitimately recur, and a gate that fires on coincidence gets routed around. Naming the uncovered surface beats claiming the class is closed. InnerLoop single-source-of-fact moves from prose to executable — v1.3. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
274 lines
13 KiB
Markdown
274 lines
13 KiB
Markdown
# CB-EV-0001 — GROUND game kernel: acceptance evidence
|
||
|
||
Status: **T08 complete. AM-4 remediated and re-measured 2026-07-31.**
|
||
Recorded: 2026-07-31. Amended 2026-07-31 — §4 gains measured savings per
|
||
remediation option, and §5 corrects AM-12 from "uncomputable" to
|
||
measured-at-session-level; see CB-WP-0002.
|
||
Workplan: CB-WP-0001, task T08
|
||
Spec: `specs/GameKernel.md` §4 (AM-1..AM-12)
|
||
Baseline: `research/CB-RES-0001-game-kernel.md`, measurements in
|
||
`research/CB-RES-0001-harness/boardgame-io/results-260731.json`
|
||
|
||
Machine: WSL2, Linux 6.18.33.2-microsoft-standard-WSL2, rustc 1.97.1,
|
||
`--release`, Criterion 1s warm-up / 3s measurement.
|
||
|
||
## 1. Scoreboard
|
||
|
||
| Metric | Target | Measured | Verdict |
|
||
|---|---|---|---|
|
||
| AM-1 rule coverage | 100% of GR-rules | 58/58 (100%) | **met** |
|
||
| AM-4a dep weight, shipped runtime | ≤250,000 third-party lines | 246,250 (23 crates) | **met** | <!-- fact:am4a_loc -->
|
||
| AM-4b dep weight, dev toolchain | ≤350,000 third-party lines | 317,021 (29 crates) | **met** | <!-- fact:am4b_loc -->
|
||
| AM-6 throughput | ≥100,000 events/s | 1,651,400 events/s | **met, 16.5×** |
|
||
| AM-7 scaling | ≥0.9× at 20× workload | 1.08× | **met** |
|
||
| AM-7 replay | 100k events ≤5s | 2.18 ms (CI 2.14–2.23) | **met, 2,290×** |
|
||
| AM-8 determinism | zero divergence, 10 replays | 1 distinct hash / 10 runs | **met** |
|
||
| AM-8 lint | fmt + clippy clean | clean, `-D warnings` | **met** |
|
||
| AM-10 foreign types | zero `HashMap`/`HashSet` | 0 | **met** |
|
||
| AM-11 impl pairs | null + reference per port | 1 of 1 (`KernelRng`) | **met, narrow** |
|
||
| AM-12 cost | per-task USD | $248.46 session; per-task pending | **partial** |
|
||
|
||
AM-2, AM-3, AM-5 and AM-9 are not reported: see §6.
|
||
|
||
## 2. Throughput and scaling (AM-6, AM-7)
|
||
|
||
Workload: 3-player GROUND rounds, 7 commands and 13 events per round
|
||
(12 on a game's fifth round, where GR-R09 ends the game). Pinned by
|
||
`bench_shape` in `games/ground/src/lib.rs`, so a change to the workload
|
||
breaks the test rather than silently rescaling the metric.
|
||
|
||
| Rounds | Throughput (events/s) | vs 5k |
|
||
|---|---|---|
|
||
| 5,000 | 1,524,200 | 1.00× |
|
||
| 10,000 | 1,638,200 | 1.07× |
|
||
| 20,000 | 1,577,000 | 1.03× |
|
||
| 40,000 | 1,626,000 | 1.07× |
|
||
| 100,000 | 1,651,400 | 1.08× |
|
||
|
||
Replay — folding one growing event log back into state (Criterion
|
||
95% CI, low/median/high):
|
||
|
||
| Events | Time (median) | 95% CI | Rate |
|
||
|---|---|---|---|
|
||
| 10,010 | 184.8 µs | 180.9 – 189.3 µs | 54.2M events/s |
|
||
| 100,072 | 2.183 ms | 2.142 – 2.226 ms | 45.8M events/s |
|
||
|
||
**Correction (2026-07-31).** These originally read 465 µs / 4.13 ms and
|
||
were taken from the `replay_probe` **test**, not from the benchmark —
|
||
because the benchmark did not work. See §2a.
|
||
|
||
### The comparison against boardgame.io, stated carefully
|
||
|
||
boardgame.io measured **1,930 moves/s at 5,000 moves**, falling to
|
||
**870 moves/s at 20,000**, and **did not finish 100,000 moves in 300 s**.
|
||
Our figure in the same unit is ~129,000 rounds/s × 7 = **~903,000
|
||
commands/s**, and 100,000 rounds complete in 776 ms.
|
||
|
||
That is roughly a 400–500× ratio, and it is **not a like-for-like
|
||
measurement**. Four differences matter, all favouring us:
|
||
|
||
1. **Different language and process model.** Rust in-process against
|
||
Node.js with immutable state, patch generation and undo history.
|
||
2. **Different feature set.** boardgame.io's per-move cost includes
|
||
producing client patches and maintaining undo state; the run with
|
||
`--disable-undo` still degraded (0.66× at 40k). We do neither.
|
||
3. **Different workload shape.** Our 5-round games (GR-R09) bound state
|
||
size by construction. The boardgame.io harness ran one match with
|
||
unbounded history, which is exactly the axis it degraded on.
|
||
4. **No network or storage layer** on our side.
|
||
|
||
Point 3 is the important one and it is why the flat curve in the table
|
||
above is *weak evidence on its own* — a game that resets every five
|
||
rounds cannot exhibit history-growth degradation. The replay benchmark
|
||
is the honest test of that axis, because there the log grows without
|
||
bound, and it stays linear (21.5M → 24.2M events/s from 10k to 100k).
|
||
|
||
**Claim we are willing to defend:** the kernel meets AM-6 and AM-7 with
|
||
large margin, and does not degrade as event-log length grows.
|
||
**Claim we are not making:** that Clay-Borg is ~450× "faster than
|
||
boardgame.io" as a like-for-like engine comparison. Per the
|
||
InnerLoop parity-cap rule, a cross-runtime ratio this coarse is not a
|
||
verdict, it is a direction.
|
||
|
||
### A measurement error found and corrected
|
||
|
||
The first run of this benchmark reported **9.3M events/s with a
|
||
perfectly flat curve** — a number that would have been reported as a
|
||
93× beat of AM-6. It was wrong. The workload had P2 selecting SUPPORT
|
||
every round while being attacked; after three rounds P2 sat at Stress 4,
|
||
GR-R03 rejected the SUPPORT, the round never completed, and the loop
|
||
spun on rejected commands. Throughput was computed as
|
||
`rounds × 13 events` while most rounds produced 2.
|
||
|
||
Found by a probe test asserting that a round produces events at all.
|
||
The benchmark now asserts the per-round event count on every round and
|
||
panics rather than measuring a stalled loop. The corrected figure is
|
||
**5.6× lower** than the bogus one.
|
||
|
||
### 2a. A fourth measurement error, found by enforcing the rule
|
||
|
||
The replay benchmark committed alongside this evidence was **the broken
|
||
version**. A `python3` patch that was supposed to replace its
|
||
log-building loop never applied, leaving a sequence that omits `Resolve`
|
||
— so `EndRound` was rejected, every round produced no events, and the
|
||
`while log.len() < target` loop spun forever. It was never run to
|
||
completion; the AM-7 replay numbers were taken from a separate probe
|
||
test instead, and the dead benchmark was committed and left hanging.
|
||
|
||
Found by adding `cargo bench -- --test` to CI, which runs every
|
||
benchmark once. That is the fourth instance of one error class in this
|
||
project — a harness that appears to work while doing no work — and the
|
||
**first one caught by a gate rather than by noticing**.
|
||
|
||
The replay loop now carries the positive control the round loop already
|
||
had: it asserts each round appended events and fails rather than
|
||
spinning. Corrected figures are in the table above; both configurations
|
||
still clear the AM-7 budget by three orders of magnitude.
|
||
|
||
The lesson recorded for the loop: writing the positive-control rule into
|
||
`specs/InnerLoop.md` did **not** prevent the next instance. Making it a
|
||
CI step did. Prose rules do not enforce themselves.
|
||
|
||
## 3. Determinism (AM-8)
|
||
|
||
- Every scenario runs twice per invocation with the same seed and fails
|
||
on state-hash divergence (K8). 21/21 pass.
|
||
- Ten consecutive full runs of all 21 scenarios produced **one distinct
|
||
output hash**, i.e. zero divergence.
|
||
- `cargo fmt --check` and `cargo clippy --workspace --all-targets
|
||
-D warnings` are clean.
|
||
- `clippy.toml` denies `HashMap`/`HashSet` workspace-wide (K6); the
|
||
aggregate holds only ordered collections, so iteration order cannot
|
||
vary between runs.
|
||
|
||
## 4. AM-4 — remediated and re-measured
|
||
|
||
**Original result: NOT MET, 33 transitive crates against a ≤20 target.**
|
||
Resolved by adopting both remediations (maintainer decision,
|
||
2026-07-31): `serde_yaml` was made optional, and the metric was
|
||
retargeted onto third-party source under audit.
|
||
|
||
### Re-measurement (`make dep-weight`)
|
||
|
||
| Configuration | Crates | Third-party LOC | Target | Verdict |
|
||
|---|---|---|---|---|
|
||
| Shipped runtime (`--no-default-features`) | 23 | 246,250 | ≤250,000 | **met** | <!-- fact:am4a_loc --><!-- fact:am4a_target -->
|
||
| Dev toolchain (default features) | 29 | 317,021 | ≤350,000 | **met** | <!-- fact:am4b_loc --><!-- fact:am4b_target -->
|
||
| Our own source | — | 3,408 | — | — |
|
||
|
||
Scenario tooling costs **70,771 lines that a shipped game never
|
||
compiles**. That split is the substantive result: the single number
|
||
previously reported conflated a runtime concern with a test concern.
|
||
|
||
**What actually changed in the build.** `cb-game-runtime` gained a
|
||
`scenarios` feature carrying `serde_yaml`; the scenario module, the
|
||
`ScenarioGame` impl and the string parsers behind it are `#[cfg]`-gated.
|
||
Both configurations compile and lint clean under `-D warnings`.
|
||
|
||
One trap worth recording: setting `default-features = false` on a
|
||
*member* dependency is silently ignored when the workspace dependency
|
||
does not specify it, so the first attempt gated nothing while appearing
|
||
to work — `cargo tree` still showed all six YAML crates. The fix was
|
||
setting `default-features = false` on the workspace dependency itself,
|
||
with `cb-sim` opting into `scenarios` explicitly. This is exactly the
|
||
class of error InnerLoop v1.0's positive-control rule targets: the build
|
||
succeeded and the feature flag looked applied. It was caught by checking
|
||
the dependency graph rather than trusting that the edit had worked.
|
||
|
||
### Why the target moved, and why that is not moving the goalposts
|
||
|
||
The ≤20 crate target was retired for two measured reasons, both
|
||
recorded before the decision was taken:
|
||
|
||
Attribution:
|
||
|
||
| Group | Crates | Count |
|
||
|---|---|---|
|
||
| `sha2` (K7 state hashing) | sha2, digest, block-buffer, crypto-common, generic-array, typenum, cpufeatures, cfg-if | 8 |
|
||
| serde derive chain | serde_derive, proc-macro2, quote, syn, unicode-ident | 5 |
|
||
| serde runtime + json | serde, serde_core, serde_json, itoa, memchr, zmij | 6 |
|
||
| `serde_yaml` (scenario files only) | serde_yaml, unsafe-libyaml, indexmap, hashbrown, equivalent, ryu | 6 |
|
||
| `rand_chacha` (K5 seeded RNG) | rand_chacha, rand_core, ppv-lite86, zerocopy | 4 |
|
||
| Clay-Borg crates | cb-kernel, cb-events, cb-game-runtime, games-ground | 4 |
|
||
|
||
1. **It was unreachable without undoing the spec's own contracts.**
|
||
Measured ladder: `serde_yaml` optional −6 (→27), dropping
|
||
`serde_json` −4 (→23), inlining SHA-256 −8 (→19), inlining ChaCha12
|
||
−4 (→15). Nothing reaches 20 except reimplementing a primitive that
|
||
K5 or K7 requires — trading an audited implementation for a
|
||
scoreboard number.
|
||
2. **Crate count does not compare across ecosystems.** Rust splits
|
||
crates far more finely than npm. The same granularity difference made
|
||
"33 vs 120 npm packages" flatter us *and* made ≤20 punish us.
|
||
|
||
Third-party source under audit is what the count was proxying for, is
|
||
comparable across ecosystems, and cannot be gamed by granularity. The
|
||
new targets are set at roughly the current measurement plus headroom,
|
||
so they bind on future growth rather than retroactively passing
|
||
something that failed: adding another `serde_yaml`-sized dependency to
|
||
the shipped runtime would breach AM-4a.
|
||
|
||
What we did **not** do: hand-roll SHA-256 or ChaCha to win a count.
|
||
|
||
## 5. Cost log (AM-12)
|
||
|
||
Per `specs/MetricsAndScenarios.md` §1a. Model: Claude Fable 5, at
|
||
`benchmarks/baselines/model-prices.toml` rates ($10/$50 per MTok).
|
||
|
||
| Task | Model | Iterations | Notes |
|
||
|---|---|---|---|
|
||
| CB-WP-0001 (whole session, T01–T09) | claude-fable-5 | 6 T08 code iterations + benchmarks | $248.46 measured; per-task split pending CB-WP-0002 |
|
||
|
||
**Correction (2026-07-31).** This section originally recorded AM-12 as
|
||
*uncomputable*. That was wrong. The declining to estimate was right; the
|
||
conclusion that no instrument existed was not. Every session transcript
|
||
(`~/.claude/projects/<slug>/<session>.jsonl`) carries exact per-message
|
||
`usage` including the cache breakdown. Read for this session:
|
||
|
||
| Component | Tokens | Cost (Fable 5) |
|
||
|---|---|---|
|
||
| Output | 585,528 | $29.28 |
|
||
| Cache read | 131,863,164 | $131.86 |
|
||
| Cache write (1h) | 4,365,668 | $87.31 |
|
||
| Input | 1,090 | $0.01 |
|
||
| **Session total** | | **$248.46** (~$124 on Opus 5) |
|
||
|
||
**53% of the cost is cache reads**, not output. Cost in an agentic loop
|
||
is driven by context size × turn count, which no "tokens per task"
|
||
metric would have surfaced.
|
||
|
||
Still missing is *attribution*: this is a whole-session figure, not a
|
||
per-task one, because nothing marks task boundaries in the transcript.
|
||
That is what CB-WP-0002 is for. The AM-12 row above should be read as
|
||
"session-level cost measured; per-task attribution pending CB-WP-0002".
|
||
|
||
## 6. Metrics not reported
|
||
|
||
- **AM-2, AM-3, AM-5** — specification-quality metrics that need a
|
||
second capability to compare against; a single data point is not a
|
||
measurement.
|
||
- **AM-9 (≤64MB)** — not instrumented. The aggregate is a few KB and
|
||
the largest log measured here is 100k events, so the budget is very
|
||
unlikely to bind, but "unlikely" is not "measured" and it is left
|
||
unclaimed.
|
||
- **AM-11** — the `KernelRng` null/reference pair exists and is
|
||
exercised. It is the only port with a pair so far, so the metric is
|
||
met narrowly and will mean more once storage has one.
|
||
|
||
## 7. Rules implemented under a provisional default
|
||
|
||
Ten U-items in `specs/GroundRules.md` carry PROVISIONAL defaults. Those
|
||
realized here are U2 (clamp on every application), U3 (DENY with no
|
||
legal target is a no-op that still advances), U4 (deck reshuffle), U5
|
||
(REVERSE owner relief applies whether or not the Reverse was rejected)
|
||
and U8 (GROUND—OU cancellation precedes Protection).
|
||
|
||
One further ambiguity was found during T08 and is **not** in the U-list:
|
||
**GR-E02's "successes"** is undefined in dataset 0.1. It is implemented
|
||
as the count of claimed Problems. Both scoring scenarios are marked
|
||
`provisional: true`.
|
||
|
||
All provisional behaviour lives behind named functions and is covered by
|
||
scenarios tagged `provisional: true`, so a ground-game ruling flips a
|
||
scenario rather than the kernel (K16). **Action for ground-game:** rule
|
||
on the ten U-items and on GR-E02's "successes".
|