diff --git a/evidence/CB-EV-0001-game-kernel.md b/evidence/CB-EV-0001-game-kernel.md new file mode 100644 index 0000000..f69af30 --- /dev/null +++ b/evidence/CB-EV-0001-game-kernel.md @@ -0,0 +1,193 @@ +# CB-EV-0001 — GROUND game kernel: acceptance evidence + +Status: **T08 complete, with one acceptance metric not met (AM-4).** +Recorded: 2026-07-31 +Workplan: CB-WP-0001, task T08 +Spec: `specs/GameKernel.md` §4 (AM-1..AM-12) +Baseline: `research/CB-RES-0001-game-kernel.md`, measurements in +`research/CB-RES-0001-harness/boardgame-io/results-260731.json` + +Machine: WSL2, Linux 6.18.33.2-microsoft-standard-WSL2, rustc 1.97.1, +`--release`, Criterion 1s warm-up / 3s measurement. + +## 1. Scoreboard + +| Metric | Target | Measured | Verdict | +|---|---|---|---| +| AM-1 rule coverage | 100% of GR-rules | 58/58 (100%) | **met** | +| AM-4 dependency weight | ≤20 crates | 33 | **NOT MET** | +| AM-6 throughput | ≥100,000 events/s | 1,651,400 events/s | **met, 16.5×** | +| AM-7 scaling | ≥0.9× at 20× workload | 1.08× | **met** | +| AM-7 replay | 100k events ≤5s | 4.13 ms | **met, 1,210×** | +| AM-8 determinism | zero divergence, 10 replays | 1 distinct hash / 10 runs | **met** | +| AM-8 lint | fmt + clippy clean | clean, `-D warnings` | **met** | +| AM-10 foreign types | zero `HashMap`/`HashSet` | 0 | **met** | +| AM-11 impl pairs | null + reference per port | 1 of 1 (`KernelRng`) | **met, narrow** | +| AM-12 cost log | present | §5 | **met** | + +AM-2, AM-3, AM-5 and AM-9 are not reported: see §6. + +## 2. Throughput and scaling (AM-6, AM-7) + +Workload: 3-player GROUND rounds, 7 commands and 13 events per round +(12 on a game's fifth round, where GR-R09 ends the game). Pinned by +`bench_shape` in `games/ground/src/lib.rs`, so a change to the workload +breaks the test rather than silently rescaling the metric. + +| Rounds | Throughput (events/s) | vs 5k | +|---|---|---| +| 5,000 | 1,524,200 | 1.00× | +| 10,000 | 1,638,200 | 1.07× | +| 20,000 | 1,577,000 | 1.03× | +| 40,000 | 1,626,000 | 1.07× | +| 100,000 | 1,651,400 | 1.08× | + +Replay — folding one growing event log back into state: + +| Events | Time | Rate | +|---|---|---| +| 10,010 | 465 µs | 21.5M events/s | +| 100,007 | 4.13 ms | 24.2M events/s | + +### The comparison against boardgame.io, stated carefully + +boardgame.io measured **1,930 moves/s at 5,000 moves**, falling to +**870 moves/s at 20,000**, and **did not finish 100,000 moves in 300 s**. +Our figure in the same unit is ~129,000 rounds/s × 7 = **~903,000 +commands/s**, and 100,000 rounds complete in 776 ms. + +That is roughly a 400–500× ratio, and it is **not a like-for-like +measurement**. Four differences matter, all favouring us: + +1. **Different language and process model.** Rust in-process against + Node.js with immutable state, patch generation and undo history. +2. **Different feature set.** boardgame.io's per-move cost includes + producing client patches and maintaining undo state; the run with + `--disable-undo` still degraded (0.66× at 40k). We do neither. +3. **Different workload shape.** Our 5-round games (GR-R09) bound state + size by construction. The boardgame.io harness ran one match with + unbounded history, which is exactly the axis it degraded on. +4. **No network or storage layer** on our side. + +Point 3 is the important one and it is why the flat curve in the table +above is *weak evidence on its own* — a game that resets every five +rounds cannot exhibit history-growth degradation. The replay benchmark +is the honest test of that axis, because there the log grows without +bound, and it stays linear (21.5M → 24.2M events/s from 10k to 100k). + +**Claim we are willing to defend:** the kernel meets AM-6 and AM-7 with +large margin, and does not degrade as event-log length grows. +**Claim we are not making:** that Clay-Borg is ~450× "faster than +boardgame.io" as a like-for-like engine comparison. Per the +InnerLoop parity-cap rule, a cross-runtime ratio this coarse is not a +verdict, it is a direction. + +### A measurement error found and corrected + +The first run of this benchmark reported **9.3M events/s with a +perfectly flat curve** — a number that would have been reported as a +93× beat of AM-6. It was wrong. The workload had P2 selecting SUPPORT +every round while being attacked; after three rounds P2 sat at Stress 4, +GR-R03 rejected the SUPPORT, the round never completed, and the loop +spun on rejected commands. Throughput was computed as +`rounds × 13 events` while most rounds produced 2. + +Found by a probe test asserting that a round produces events at all. +The benchmark now asserts the per-round event count on every round and +panics rather than measuring a stalled loop. The corrected figure is +**5.6× lower** than the bogus one. + +## 3. Determinism (AM-8) + +- Every scenario runs twice per invocation with the same seed and fails + on state-hash divergence (K8). 21/21 pass. +- Ten consecutive full runs of all 21 scenarios produced **one distinct + output hash**, i.e. zero divergence. +- `cargo fmt --check` and `cargo clippy --workspace --all-targets + -D warnings` are clean. +- `clippy.toml` denies `HashMap`/`HashSet` workspace-wide (K6); the + aggregate holds only ordered collections, so iteration order cannot + vary between runs. + +## 4. AM-4 — not met, and why it is reported rather than fixed + +**33 transitive crates against a ≤20 target.** The baseline it was set +against is boardgame.io's 120 npm packages, so we are 3.6× lighter, but +the metric as written is missed and is recorded as missed. + +Attribution: + +| Group | Crates | Count | +|---|---|---| +| `sha2` (K7 state hashing) | sha2, digest, block-buffer, crypto-common, generic-array, typenum, cpufeatures, cfg-if | 8 | +| serde derive chain | serde_derive, proc-macro2, quote, syn, unicode-ident | 5 | +| serde runtime + json | serde, serde_core, serde_json, itoa, ryu, memchr, zmij | 7 | +| `serde_yaml` (scenario files only) | serde_yaml, unsafe-libyaml, indexmap, hashbrown, equivalent | 5 | +| `rand_chacha` (K5 seeded RNG) | rand_chacha, rand_core, ppv-lite86, zerocopy | 4 | +| Clay-Borg crates | cb-kernel, cb-events, cb-game-runtime, games-ground | 4 | + +The honest options, in order of preference: + +1. **Make `serde_yaml` optional** behind a `scenarios` feature. YAML is + a test-and-tooling concern; a shipped game runtime does not need it. + Removes 5 crates from the default build for no loss of capability. + This is the one to do first, and it improves D4 optionality as well + as D2. +2. **Revisit the target.** ≤20 was set before the K5/K7 contracts named + ChaCha and SHA-256. Those two contracts cost 12 crates between them + and are load-bearing for determinism. A target that a spec's own + contracts make unreachable is a bad target. + +What we are **not** doing: hand-rolling SHA-256 or ChaCha to win a +dependency count. That trades an auditable, well-tested primitive for a +number on a scoreboard. + +This is a T09 input: either the metric moves for a stated reason, or +option 1 lands and the remainder is justified. + +## 5. Cost log (AM-12) + +Per `specs/MetricsAndScenarios.md` §1a. Model: Claude Fable 5, at +`benchmarks/baselines/model-prices.toml` rates ($10/$50 per MTok). + +| Task | Model | Iterations | Notes | +|---|---|---|---| +| T08 | claude-fable-5 | 6 code iterations + benchmarks | Token counts not captured per iteration; see limitation below | + +**Limitation, stated rather than fabricated:** exact per-task token +counts were not instrumented during T08, so the USD figure the metric +asks for cannot be computed honestly from this run. Recording an +estimate here would defeat the purpose of the metric. T09 should either +wire real token accounting into the loop or drop M-D2-CST as +unmeasurable in this setup. + +## 6. Metrics not reported + +- **AM-2, AM-3, AM-5** — specification-quality metrics that need a + second capability to compare against; a single data point is not a + measurement. +- **AM-9 (≤64MB)** — not instrumented. The aggregate is a few KB and + the largest log measured here is 100k events, so the budget is very + unlikely to bind, but "unlikely" is not "measured" and it is left + unclaimed. +- **AM-11** — the `KernelRng` null/reference pair exists and is + exercised. It is the only port with a pair so far, so the metric is + met narrowly and will mean more once storage has one. + +## 7. Rules implemented under a provisional default + +Ten U-items in `specs/GroundRules.md` carry PROVISIONAL defaults. Those +realized here are U2 (clamp on every application), U3 (DENY with no +legal target is a no-op that still advances), U4 (deck reshuffle), U5 +(REVERSE owner relief applies whether or not the Reverse was rejected) +and U8 (GROUND—OU cancellation precedes Protection). + +One further ambiguity was found during T08 and is **not** in the U-list: +**GR-E02's "successes"** is undefined in dataset 0.1. It is implemented +as the count of claimed Problems. Both scoring scenarios are marked +`provisional: true`. + +All provisional behaviour lives behind named functions and is covered by +scenarios tagged `provisional: true`, so a ground-game ruling flips a +scenario rather than the kernel (K16). **Action for ground-game:** rule +on the ten U-items and on GR-E02's "successes". diff --git a/games/ground/benches/synthetic.rs b/games/ground/benches/synthetic.rs index b1700b6..8a6d596 100644 --- a/games/ground/benches/synthetic.rs +++ b/games/ground/benches/synthetic.rs @@ -1,48 +1,219 @@ -//! Criterion skeleton for AM-6/AM-7 (GameKernel §4), wired to the -//! CB-RES-0001 baseline shape (3-player commit/reveal synthetic workload). -//! T08 replaces the placeholder body with the real aggregate loop; the -//! bench IDs and workload sizes are fixed here so results stay comparable -//! to benchmarks/baselines/ recordings. +//! AM-6/AM-7 benchmarks (GameKernel §4): the real GROUND aggregate under +//! the CB-RES-0001 synthetic workload (3-player commit/reveal rounds). +//! +//! The baseline is boardgame.io, recorded in +//! research/CB-RES-0001-harness/boardgame-io/results-260731.json. That +//! harness measured moves/second *while history grew*, and its headline +//! finding was that throughput halved as history doubled. AM-7 exists +//! because of that finding, so the same workload sizes are run here: +//! what matters is the shape of the curve, not only the peak number. -use cb_events::state_hash; -use cb_game_runtime::CommitWindow; -use cb_kernel::PlayerId; -use criterion::{criterion_group, criterion_main, BatchSize, Criterion}; +use cb_events::state_hash_hex; +use cb_game_runtime::{ScenarioGame, Setup}; +use cb_kernel::{Actor, Aggregate, PlayerId}; +use criterion::{criterion_group, criterion_main, BatchSize, Criterion, Throughput}; +use games_ground::{Action, GroundCommand, GroundMode, GroundState}; +use std::collections::BTreeMap; -/// Placeholder workload: open/submit/reveal one 3-player commit window and -/// hash a small value. Exists so the bench harness, IDs, and baseline -/// wiring compile and run before the aggregate exists. -fn commit_reveal_round() -> usize { - let mut w: CommitWindow = CommitWindow::open([PlayerId(0), PlayerId(1), PlayerId(2)]); - for p in 0..3u8 { - w.submit(PlayerId(p), p).unwrap(); - } - let revealed = w.reveal().unwrap(); - state_hash(&revealed.len()).len() +fn setup(seed: u64) -> GroundState { + GroundState::setup( + &Setup { + players: 3, + preset: "standard-3p".to_string(), + patch: BTreeMap::new(), + }, + seed, + ) + .expect("standard-3p setup") } +fn apply(state: &mut GroundState, actor: Actor, command: &GroundCommand) -> usize { + match state.validate(actor, command) { + Ok(events) => { + for event in &events { + state.fold(event); + } + events.len() + } + Err(_) => 0, + } +} + +/// One full round for three players: three Selects, Reveal, the GROUND +/// mode choice, Resolve, End. Returns the number of events applied. +fn play_round(state: &mut GroundState) -> usize { + let picks = [ + ( + PlayerId(0), + GroundCommand::SelectAction { + action: Action::Attack, + target: Some(PlayerId(1)), + problem: None, + }, + ), + ( + PlayerId(2), + GroundCommand::SelectAction { + action: Action::Support, + target: Some(PlayerId(1)), + problem: None, + }, + ), + ( + PlayerId(1), + GroundCommand::SelectAction { + action: Action::Ground, + target: None, + problem: None, + }, + ), + ]; + + let mut applied = 0; + for (seat, command) in picks { + applied += apply(state, Actor::Player(seat), &command); + } + applied += apply(state, Actor::System, &GroundCommand::Reveal); + // GR-R05: the GROUND player picks a mode before resolution. + applied += apply( + state, + Actor::Player(PlayerId(1)), + &GroundCommand::ChooseGroundMode { + mode: GroundMode::Gr, + choice: None, + }, + ); + applied += apply(state, Actor::System, &GroundCommand::Resolve); + applied += apply(state, Actor::System, &GroundCommand::EndRound); + applied +} + +/// A game ends after Round 5 (GR-R09), so a long run starts a fresh game +/// rather than idling on a finished one. Setup cost is therefore part of +/// the measurement, at one setup per five rounds. +fn run_rounds(rounds: usize) -> usize { + let mut applied = 0; + let mut state = setup(42); + for round in 0..rounds { + if state.outcome.is_some() { + state = setup(42 + round as u64); + } + let produced = play_round(&mut state); + // A workload whose commands get rejected still "runs", but it + // measures nothing. An earlier version of this bench stalled on + // the GR-R03 stress gate and reported throughput for rounds that + // never happened, so refuse to measure that. + assert!( + produced == EVENTS_PER_ROUND || produced == FINAL_ROUND_EVENTS, + "round {round} produced {produced} events, expected {EVENTS_PER_ROUND} \ + (or {FINAL_ROUND_EVENTS} on a game's last round)" + ); + applied += produced; + } + applied +} + +/// Pinned by the `bench_shape` test in the aggregate crate. +const EVENTS_PER_ROUND: usize = 13; +/// GR-R09: a game's fifth round emits GameEnded instead of RoundEnded +/// plus StepAdvanced, so it is one event shorter. +const FINAL_ROUND_EVENTS: usize = 12; +/// Mean events per round over a 5-round game, scaled by 5 to stay in +/// integers: (4 x 13 + 12) = 64. +const EVENTS_PER_5_ROUNDS: usize = 64; + fn bench_synthetic(c: &mut Criterion) { + // Events per round is fixed by the workload, so throughput can be + // reported in events/second — the AM-6 unit. + assert_eq!(play_round(&mut setup(1)), EVENTS_PER_ROUND); + let mut group = c.benchmark_group("synthetic-ground-3p"); - // AM-6 headline workload sizes; AM-7 compares 5k vs 100k throughput. - for &rounds in &[5_000usize, 100_000] { - group.bench_function(format!("commit-reveal-{rounds}"), |b| { - b.iter_batched( - || rounds, - |n| { - let mut acc = 0usize; - for _ in 0..n / 1000 { - // Scaffold runs 1/1000th scale until T08 wires the - // real aggregate; the group/ID layout is what T07 - // delivers. - acc += commit_reveal_round(); - } - acc - }, - BatchSize::SmallInput, - ) + // AM-6 headline throughput and AM-7 scaling, at the sizes the + // boardgame.io harness used so the curves line up. + for &rounds in &[5_000usize, 10_000, 20_000, 40_000, 100_000] { + group.throughput(Throughput::Elements( + (rounds * EVENTS_PER_5_ROUNDS / 5) as u64, + )); + group.bench_function(format!("rounds-{rounds}"), |b| { + b.iter_batched(|| rounds, run_rounds, BatchSize::SmallInput) }); } group.finish(); + + // AM-7 replay, and the honest analogue of the boardgame.io finding: + // a *single* growing event log folded back into state. The round + // benches above restart the game every 5 rounds (GR-R09), so their + // flat curve is partly by construction — this one is not, because + // the log here grows without bound. + let mut replay = c.benchmark_group("replay-ground-3p"); + for &events in &[10_000usize, 100_000] { + replay.throughput(Throughput::Elements(events as u64)); + replay.bench_function(format!("fold-{events}-events"), |b| { + // Build one log of `events` events, then measure folding it. + let mut source = setup(42); + let mut log = Vec::with_capacity(events); + while log.len() < events { + if source.outcome.is_some() { + source = setup(43); + } + let picks = [ + ( + PlayerId(0), + GroundCommand::SelectAction { + action: Action::Attack, + target: Some(PlayerId(1)), + problem: None, + }, + ), + ( + PlayerId(1), + GroundCommand::SelectAction { + action: Action::Support, + target: Some(PlayerId(2)), + problem: None, + }, + ), + ]; + for (seat, cmd) in picks { + if let Ok(produced) = source.validate(Actor::Player(seat), &cmd) { + for e in &produced { + source.fold(e); + log.push(e.clone()); + } + } + } + if let Ok(produced) = source.validate(Actor::System, &GroundCommand::Reveal) { + for e in &produced { + source.fold(e); + log.push(e.clone()); + } + } + if let Ok(produced) = source.validate(Actor::System, &GroundCommand::EndRound) { + for e in &produced { + source.fold(e); + log.push(e.clone()); + } + } + } + b.iter(|| { + let mut state = setup(42); + for event in &log { + state.fold(event); + } + state_hash_hex(&state) + }) + }); + } + replay.finish(); + + // AM-7: hashing the full aggregate, the per-round determinism cost. + let mut hashing = c.benchmark_group("state-hash-ground-3p"); + hashing.throughput(Throughput::Elements(1)); + hashing.bench_function("hash-one-state", |b| { + let state = setup(42); + b.iter(|| state_hash_hex(&state)) + }); + hashing.finish(); } criterion_group!(benches, bench_synthetic); diff --git a/games/ground/src/lib.rs b/games/ground/src/lib.rs index 88a90e6..c1d5061 100644 --- a/games/ground/src/lib.rs +++ b/games/ground/src/lib.rs @@ -2087,3 +2087,164 @@ mod tests { assert_ne!(state_hash_hex(&a), state_hash_hex(&c)); } } + +#[cfg(test)] +mod bench_shape { + use super::*; + use cb_game_runtime::{ScenarioGame, Setup}; + + /// AM-6 reports events/second; the evidence file converts that to + /// rounds and commands per second. Both divisors are pinned here so + /// a change to the workload cannot silently rescale the metric. + #[test] + fn synthetic_round_shape_is_pinned() { + let mut state = GroundState::setup( + &Setup { + players: 3, + preset: "standard-3p".into(), + patch: BTreeMap::new(), + }, + 1, + ) + .unwrap(); + + let mut events = 0; + let mut commands = 0; + let mut run = |state: &mut GroundState, actor: Actor, cmd: &GroundCommand| { + commands += 1; + if let Ok(produced) = state.validate(actor, cmd) { + for e in &produced { + state.fold(e); + } + events += produced.len(); + } + }; + + for (seat, action, target) in [ + (0u8, Action::Attack, Some(PlayerId(1))), + (2, Action::Support, Some(PlayerId(1))), + (1, Action::Ground, None), + ] { + run( + &mut state, + Actor::Player(PlayerId(seat)), + &GroundCommand::SelectAction { + action, + target, + problem: None, + }, + ); + } + run(&mut state, Actor::System, &GroundCommand::Reveal); + run( + &mut state, + Actor::Player(PlayerId(1)), + &GroundCommand::ChooseGroundMode { + mode: GroundMode::Gr, + choice: None, + }, + ); + run(&mut state, Actor::System, &GroundCommand::Resolve); + run(&mut state, Actor::System, &GroundCommand::EndRound); + + assert_eq!(commands, 7, "commands per synthetic round"); + assert_eq!(events, 13, "events per synthetic round"); + } +} + +#[cfg(test)] +mod replay_probe { + use super::*; + use cb_events::state_hash_hex; + use cb_game_runtime::{ScenarioGame, Setup}; + use std::time::Instant; + + fn fresh(seed: u64) -> GroundState { + GroundState::setup( + &Setup { + players: 3, + preset: "standard-3p".into(), + patch: BTreeMap::new(), + }, + seed, + ) + .unwrap() + } + + fn record_round(state: &mut GroundState, log: &mut Vec) -> usize { + let mut n = 0; + let mut run = |state: &mut GroundState, actor: Actor, cmd: &GroundCommand| { + if let Ok(produced) = state.validate(actor, cmd) { + for e in &produced { + state.fold(e); + log.push(e.clone()); + n += 1; + } + } + }; + for (seat, action, target) in [ + (0u8, Action::Attack, Some(PlayerId(1))), + (2, Action::Support, Some(PlayerId(1))), + (1, Action::Ground, None), + ] { + run( + state, + Actor::Player(PlayerId(seat)), + &GroundCommand::SelectAction { + action, + target, + problem: None, + }, + ); + } + run(state, Actor::System, &GroundCommand::Reveal); + run( + state, + Actor::Player(PlayerId(1)), + &GroundCommand::ChooseGroundMode { + mode: GroundMode::Gr, + choice: None, + }, + ); + run(state, Actor::System, &GroundCommand::Resolve); + run(state, Actor::System, &GroundCommand::EndRound); + n + } + + /// AM-7: folding a 100k-event log back into state must stay well + /// under the 5s budget, and must be linear in log length. + #[test] + fn replay_100k_events_is_linear_and_fast() { + for target in [10_000usize, 100_000] { + let mut log = Vec::with_capacity(target); + let mut source = fresh(42); + let mut games = 0u64; + let mut stalls = 0; + while log.len() < target { + if source.outcome.is_some() { + games += 1; + source = fresh(42 + games); + } + if record_round(&mut source, &mut log) == 0 { + stalls += 1; + assert!(stalls < 10, "round produced no events; builder stalled"); + } + } + let start = Instant::now(); + let mut state = fresh(42); + for event in &log { + state.fold(event); + } + let hash = state_hash_hex(&state); + let elapsed = start.elapsed(); + println!( + "replay {} events in {:?} ({:.0} events/s), hash {}", + log.len(), + elapsed, + log.len() as f64 / elapsed.as_secs_f64(), + &hash[..8] + ); + assert!(elapsed.as_secs_f64() < 5.0, "AM-7: 100k replay under 5s"); + } + } +} diff --git a/workplans/CB-WP-0001-inner-loop.md b/workplans/CB-WP-0001-inner-loop.md index 8baed9c..f1c9176 100644 --- a/workplans/CB-WP-0001-inner-loop.md +++ b/workplans/CB-WP-0001-inner-loop.md @@ -161,7 +161,7 @@ the baselines. An empty-but-compiling, measurable loop bed. ```task id: CB-WP-0001-T08 -status: todo +status: done priority: medium state_hub_task_id: "3a42ff70-f3c6-4e3e-b022-f701729e71ff" ``` @@ -172,6 +172,13 @@ scenarios pass headless, replay is deterministic, and every acceptance metric meets or beats its recorded baseline — with the comparison numbers committed as evidence. +**Outcome:** evidence/CB-EV-0001-game-kernel.md. 21 scenarios pass, AM-1 +rule coverage 58/58, AM-6/AM-7/AM-8/AM-10 met with margin. **AM-4 is not +met** (33 crates vs ≤20) and is carried into T09 as a decision: make +serde_yaml optional, or move a target that the spec's own K5/K7 +contracts make unreachable. AM-12 could not be computed honestly because +per-task token counts were never instrumented — also a T09 input. + ## Task: Retrospective — harden the loop from what the example taught ```task