T08 complete: benchmarks, determinism evidence, and one missed metric
evidence/CB-EV-0001-game-kernel.md records the acceptance run against the CB-RES-0001 baseline. Met: AM-1 rule coverage 58/58; AM-6 throughput 1.65M events/s against a 100k target; AM-7 scaling 1.08x at 20x workload and a 100k-event replay in 4.13ms against a 5s budget; AM-8 zero divergence over 10 full runs with fmt and clippy clean; AM-10 zero foreign collection types. Not met and reported as such: AM-4 at 33 transitive crates against a <=20 target. Attribution is in the evidence file. The recommended fix is making serde_yaml optional (-5, a test-only concern), after which the remainder is sha2 and rand_chacha, which K5 and K7 require. We are not hand-rolling crypto primitives to win a dependency count. AM-12 is recorded as uncomputable: per-task token counts were never instrumented, and inventing a USD figure would defeat the metric. A measurement error was found and corrected before publication. The first benchmark reported 9.3M events/s on a flat curve. The workload had a player selecting SUPPORT while parked at Stress 4, so GR-R03 rejected it, rounds never completed, and throughput was computed for rounds that never happened. The bench now asserts the per-round event count and panics rather than measuring a stalled loop. The corrected figure is 5.6x lower. The evidence file states plainly what the boardgame.io comparison does and does not support: the ~450x command-rate ratio is cross-runtime and cross-feature-set, so it is a direction, not a verdict, per the InnerLoop parity-cap rule. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
85d93a9e3c
commit
eb1378e667
4 changed files with 569 additions and 37 deletions
|
|
@ -1,48 +1,219 @@
|
|||
//! Criterion skeleton for AM-6/AM-7 (GameKernel §4), wired to the
|
||||
//! CB-RES-0001 baseline shape (3-player commit/reveal synthetic workload).
|
||||
//! T08 replaces the placeholder body with the real aggregate loop; the
|
||||
//! bench IDs and workload sizes are fixed here so results stay comparable
|
||||
//! to benchmarks/baselines/ recordings.
|
||||
//! AM-6/AM-7 benchmarks (GameKernel §4): the real GROUND aggregate under
|
||||
//! the CB-RES-0001 synthetic workload (3-player commit/reveal rounds).
|
||||
//!
|
||||
//! The baseline is boardgame.io, recorded in
|
||||
//! research/CB-RES-0001-harness/boardgame-io/results-260731.json. That
|
||||
//! harness measured moves/second *while history grew*, and its headline
|
||||
//! finding was that throughput halved as history doubled. AM-7 exists
|
||||
//! because of that finding, so the same workload sizes are run here:
|
||||
//! what matters is the shape of the curve, not only the peak number.
|
||||
|
||||
use cb_events::state_hash;
|
||||
use cb_game_runtime::CommitWindow;
|
||||
use cb_kernel::PlayerId;
|
||||
use criterion::{criterion_group, criterion_main, BatchSize, Criterion};
|
||||
use cb_events::state_hash_hex;
|
||||
use cb_game_runtime::{ScenarioGame, Setup};
|
||||
use cb_kernel::{Actor, Aggregate, PlayerId};
|
||||
use criterion::{criterion_group, criterion_main, BatchSize, Criterion, Throughput};
|
||||
use games_ground::{Action, GroundCommand, GroundMode, GroundState};
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
/// Placeholder workload: open/submit/reveal one 3-player commit window and
|
||||
/// hash a small value. Exists so the bench harness, IDs, and baseline
|
||||
/// wiring compile and run before the aggregate exists.
|
||||
fn commit_reveal_round() -> usize {
|
||||
let mut w: CommitWindow<u8> = CommitWindow::open([PlayerId(0), PlayerId(1), PlayerId(2)]);
|
||||
for p in 0..3u8 {
|
||||
w.submit(PlayerId(p), p).unwrap();
|
||||
}
|
||||
let revealed = w.reveal().unwrap();
|
||||
state_hash(&revealed.len()).len()
|
||||
fn setup(seed: u64) -> GroundState {
|
||||
GroundState::setup(
|
||||
&Setup {
|
||||
players: 3,
|
||||
preset: "standard-3p".to_string(),
|
||||
patch: BTreeMap::new(),
|
||||
},
|
||||
seed,
|
||||
)
|
||||
.expect("standard-3p setup")
|
||||
}
|
||||
|
||||
fn apply(state: &mut GroundState, actor: Actor, command: &GroundCommand) -> usize {
|
||||
match state.validate(actor, command) {
|
||||
Ok(events) => {
|
||||
for event in &events {
|
||||
state.fold(event);
|
||||
}
|
||||
events.len()
|
||||
}
|
||||
Err(_) => 0,
|
||||
}
|
||||
}
|
||||
|
||||
/// One full round for three players: three Selects, Reveal, the GROUND
|
||||
/// mode choice, Resolve, End. Returns the number of events applied.
|
||||
fn play_round(state: &mut GroundState) -> usize {
|
||||
let picks = [
|
||||
(
|
||||
PlayerId(0),
|
||||
GroundCommand::SelectAction {
|
||||
action: Action::Attack,
|
||||
target: Some(PlayerId(1)),
|
||||
problem: None,
|
||||
},
|
||||
),
|
||||
(
|
||||
PlayerId(2),
|
||||
GroundCommand::SelectAction {
|
||||
action: Action::Support,
|
||||
target: Some(PlayerId(1)),
|
||||
problem: None,
|
||||
},
|
||||
),
|
||||
(
|
||||
PlayerId(1),
|
||||
GroundCommand::SelectAction {
|
||||
action: Action::Ground,
|
||||
target: None,
|
||||
problem: None,
|
||||
},
|
||||
),
|
||||
];
|
||||
|
||||
let mut applied = 0;
|
||||
for (seat, command) in picks {
|
||||
applied += apply(state, Actor::Player(seat), &command);
|
||||
}
|
||||
applied += apply(state, Actor::System, &GroundCommand::Reveal);
|
||||
// GR-R05: the GROUND player picks a mode before resolution.
|
||||
applied += apply(
|
||||
state,
|
||||
Actor::Player(PlayerId(1)),
|
||||
&GroundCommand::ChooseGroundMode {
|
||||
mode: GroundMode::Gr,
|
||||
choice: None,
|
||||
},
|
||||
);
|
||||
applied += apply(state, Actor::System, &GroundCommand::Resolve);
|
||||
applied += apply(state, Actor::System, &GroundCommand::EndRound);
|
||||
applied
|
||||
}
|
||||
|
||||
/// A game ends after Round 5 (GR-R09), so a long run starts a fresh game
|
||||
/// rather than idling on a finished one. Setup cost is therefore part of
|
||||
/// the measurement, at one setup per five rounds.
|
||||
fn run_rounds(rounds: usize) -> usize {
|
||||
let mut applied = 0;
|
||||
let mut state = setup(42);
|
||||
for round in 0..rounds {
|
||||
if state.outcome.is_some() {
|
||||
state = setup(42 + round as u64);
|
||||
}
|
||||
let produced = play_round(&mut state);
|
||||
// A workload whose commands get rejected still "runs", but it
|
||||
// measures nothing. An earlier version of this bench stalled on
|
||||
// the GR-R03 stress gate and reported throughput for rounds that
|
||||
// never happened, so refuse to measure that.
|
||||
assert!(
|
||||
produced == EVENTS_PER_ROUND || produced == FINAL_ROUND_EVENTS,
|
||||
"round {round} produced {produced} events, expected {EVENTS_PER_ROUND} \
|
||||
(or {FINAL_ROUND_EVENTS} on a game's last round)"
|
||||
);
|
||||
applied += produced;
|
||||
}
|
||||
applied
|
||||
}
|
||||
|
||||
/// Pinned by the `bench_shape` test in the aggregate crate.
|
||||
const EVENTS_PER_ROUND: usize = 13;
|
||||
/// GR-R09: a game's fifth round emits GameEnded instead of RoundEnded
|
||||
/// plus StepAdvanced, so it is one event shorter.
|
||||
const FINAL_ROUND_EVENTS: usize = 12;
|
||||
/// Mean events per round over a 5-round game, scaled by 5 to stay in
|
||||
/// integers: (4 x 13 + 12) = 64.
|
||||
const EVENTS_PER_5_ROUNDS: usize = 64;
|
||||
|
||||
fn bench_synthetic(c: &mut Criterion) {
|
||||
// Events per round is fixed by the workload, so throughput can be
|
||||
// reported in events/second — the AM-6 unit.
|
||||
assert_eq!(play_round(&mut setup(1)), EVENTS_PER_ROUND);
|
||||
|
||||
let mut group = c.benchmark_group("synthetic-ground-3p");
|
||||
// AM-6 headline workload sizes; AM-7 compares 5k vs 100k throughput.
|
||||
for &rounds in &[5_000usize, 100_000] {
|
||||
group.bench_function(format!("commit-reveal-{rounds}"), |b| {
|
||||
b.iter_batched(
|
||||
|| rounds,
|
||||
|n| {
|
||||
let mut acc = 0usize;
|
||||
for _ in 0..n / 1000 {
|
||||
// Scaffold runs 1/1000th scale until T08 wires the
|
||||
// real aggregate; the group/ID layout is what T07
|
||||
// delivers.
|
||||
acc += commit_reveal_round();
|
||||
}
|
||||
acc
|
||||
},
|
||||
BatchSize::SmallInput,
|
||||
)
|
||||
// AM-6 headline throughput and AM-7 scaling, at the sizes the
|
||||
// boardgame.io harness used so the curves line up.
|
||||
for &rounds in &[5_000usize, 10_000, 20_000, 40_000, 100_000] {
|
||||
group.throughput(Throughput::Elements(
|
||||
(rounds * EVENTS_PER_5_ROUNDS / 5) as u64,
|
||||
));
|
||||
group.bench_function(format!("rounds-{rounds}"), |b| {
|
||||
b.iter_batched(|| rounds, run_rounds, BatchSize::SmallInput)
|
||||
});
|
||||
}
|
||||
group.finish();
|
||||
|
||||
// AM-7 replay, and the honest analogue of the boardgame.io finding:
|
||||
// a *single* growing event log folded back into state. The round
|
||||
// benches above restart the game every 5 rounds (GR-R09), so their
|
||||
// flat curve is partly by construction — this one is not, because
|
||||
// the log here grows without bound.
|
||||
let mut replay = c.benchmark_group("replay-ground-3p");
|
||||
for &events in &[10_000usize, 100_000] {
|
||||
replay.throughput(Throughput::Elements(events as u64));
|
||||
replay.bench_function(format!("fold-{events}-events"), |b| {
|
||||
// Build one log of `events` events, then measure folding it.
|
||||
let mut source = setup(42);
|
||||
let mut log = Vec::with_capacity(events);
|
||||
while log.len() < events {
|
||||
if source.outcome.is_some() {
|
||||
source = setup(43);
|
||||
}
|
||||
let picks = [
|
||||
(
|
||||
PlayerId(0),
|
||||
GroundCommand::SelectAction {
|
||||
action: Action::Attack,
|
||||
target: Some(PlayerId(1)),
|
||||
problem: None,
|
||||
},
|
||||
),
|
||||
(
|
||||
PlayerId(1),
|
||||
GroundCommand::SelectAction {
|
||||
action: Action::Support,
|
||||
target: Some(PlayerId(2)),
|
||||
problem: None,
|
||||
},
|
||||
),
|
||||
];
|
||||
for (seat, cmd) in picks {
|
||||
if let Ok(produced) = source.validate(Actor::Player(seat), &cmd) {
|
||||
for e in &produced {
|
||||
source.fold(e);
|
||||
log.push(e.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
if let Ok(produced) = source.validate(Actor::System, &GroundCommand::Reveal) {
|
||||
for e in &produced {
|
||||
source.fold(e);
|
||||
log.push(e.clone());
|
||||
}
|
||||
}
|
||||
if let Ok(produced) = source.validate(Actor::System, &GroundCommand::EndRound) {
|
||||
for e in &produced {
|
||||
source.fold(e);
|
||||
log.push(e.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
b.iter(|| {
|
||||
let mut state = setup(42);
|
||||
for event in &log {
|
||||
state.fold(event);
|
||||
}
|
||||
state_hash_hex(&state)
|
||||
})
|
||||
});
|
||||
}
|
||||
replay.finish();
|
||||
|
||||
// AM-7: hashing the full aggregate, the per-round determinism cost.
|
||||
let mut hashing = c.benchmark_group("state-hash-ground-3p");
|
||||
hashing.throughput(Throughput::Elements(1));
|
||||
hashing.bench_function("hash-one-state", |b| {
|
||||
let state = setup(42);
|
||||
b.iter(|| state_hash_hex(&state))
|
||||
});
|
||||
hashing.finish();
|
||||
}
|
||||
|
||||
criterion_group!(benches, bench_synthetic);
|
||||
|
|
|
|||
|
|
@ -2087,3 +2087,164 @@ mod tests {
|
|||
assert_ne!(state_hash_hex(&a), state_hash_hex(&c));
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod bench_shape {
|
||||
use super::*;
|
||||
use cb_game_runtime::{ScenarioGame, Setup};
|
||||
|
||||
/// AM-6 reports events/second; the evidence file converts that to
|
||||
/// rounds and commands per second. Both divisors are pinned here so
|
||||
/// a change to the workload cannot silently rescale the metric.
|
||||
#[test]
|
||||
fn synthetic_round_shape_is_pinned() {
|
||||
let mut state = GroundState::setup(
|
||||
&Setup {
|
||||
players: 3,
|
||||
preset: "standard-3p".into(),
|
||||
patch: BTreeMap::new(),
|
||||
},
|
||||
1,
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
let mut events = 0;
|
||||
let mut commands = 0;
|
||||
let mut run = |state: &mut GroundState, actor: Actor, cmd: &GroundCommand| {
|
||||
commands += 1;
|
||||
if let Ok(produced) = state.validate(actor, cmd) {
|
||||
for e in &produced {
|
||||
state.fold(e);
|
||||
}
|
||||
events += produced.len();
|
||||
}
|
||||
};
|
||||
|
||||
for (seat, action, target) in [
|
||||
(0u8, Action::Attack, Some(PlayerId(1))),
|
||||
(2, Action::Support, Some(PlayerId(1))),
|
||||
(1, Action::Ground, None),
|
||||
] {
|
||||
run(
|
||||
&mut state,
|
||||
Actor::Player(PlayerId(seat)),
|
||||
&GroundCommand::SelectAction {
|
||||
action,
|
||||
target,
|
||||
problem: None,
|
||||
},
|
||||
);
|
||||
}
|
||||
run(&mut state, Actor::System, &GroundCommand::Reveal);
|
||||
run(
|
||||
&mut state,
|
||||
Actor::Player(PlayerId(1)),
|
||||
&GroundCommand::ChooseGroundMode {
|
||||
mode: GroundMode::Gr,
|
||||
choice: None,
|
||||
},
|
||||
);
|
||||
run(&mut state, Actor::System, &GroundCommand::Resolve);
|
||||
run(&mut state, Actor::System, &GroundCommand::EndRound);
|
||||
|
||||
assert_eq!(commands, 7, "commands per synthetic round");
|
||||
assert_eq!(events, 13, "events per synthetic round");
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod replay_probe {
|
||||
use super::*;
|
||||
use cb_events::state_hash_hex;
|
||||
use cb_game_runtime::{ScenarioGame, Setup};
|
||||
use std::time::Instant;
|
||||
|
||||
fn fresh(seed: u64) -> GroundState {
|
||||
GroundState::setup(
|
||||
&Setup {
|
||||
players: 3,
|
||||
preset: "standard-3p".into(),
|
||||
patch: BTreeMap::new(),
|
||||
},
|
||||
seed,
|
||||
)
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
fn record_round(state: &mut GroundState, log: &mut Vec<GroundEvent>) -> usize {
|
||||
let mut n = 0;
|
||||
let mut run = |state: &mut GroundState, actor: Actor, cmd: &GroundCommand| {
|
||||
if let Ok(produced) = state.validate(actor, cmd) {
|
||||
for e in &produced {
|
||||
state.fold(e);
|
||||
log.push(e.clone());
|
||||
n += 1;
|
||||
}
|
||||
}
|
||||
};
|
||||
for (seat, action, target) in [
|
||||
(0u8, Action::Attack, Some(PlayerId(1))),
|
||||
(2, Action::Support, Some(PlayerId(1))),
|
||||
(1, Action::Ground, None),
|
||||
] {
|
||||
run(
|
||||
state,
|
||||
Actor::Player(PlayerId(seat)),
|
||||
&GroundCommand::SelectAction {
|
||||
action,
|
||||
target,
|
||||
problem: None,
|
||||
},
|
||||
);
|
||||
}
|
||||
run(state, Actor::System, &GroundCommand::Reveal);
|
||||
run(
|
||||
state,
|
||||
Actor::Player(PlayerId(1)),
|
||||
&GroundCommand::ChooseGroundMode {
|
||||
mode: GroundMode::Gr,
|
||||
choice: None,
|
||||
},
|
||||
);
|
||||
run(state, Actor::System, &GroundCommand::Resolve);
|
||||
run(state, Actor::System, &GroundCommand::EndRound);
|
||||
n
|
||||
}
|
||||
|
||||
/// AM-7: folding a 100k-event log back into state must stay well
|
||||
/// under the 5s budget, and must be linear in log length.
|
||||
#[test]
|
||||
fn replay_100k_events_is_linear_and_fast() {
|
||||
for target in [10_000usize, 100_000] {
|
||||
let mut log = Vec::with_capacity(target);
|
||||
let mut source = fresh(42);
|
||||
let mut games = 0u64;
|
||||
let mut stalls = 0;
|
||||
while log.len() < target {
|
||||
if source.outcome.is_some() {
|
||||
games += 1;
|
||||
source = fresh(42 + games);
|
||||
}
|
||||
if record_round(&mut source, &mut log) == 0 {
|
||||
stalls += 1;
|
||||
assert!(stalls < 10, "round produced no events; builder stalled");
|
||||
}
|
||||
}
|
||||
let start = Instant::now();
|
||||
let mut state = fresh(42);
|
||||
for event in &log {
|
||||
state.fold(event);
|
||||
}
|
||||
let hash = state_hash_hex(&state);
|
||||
let elapsed = start.elapsed();
|
||||
println!(
|
||||
"replay {} events in {:?} ({:.0} events/s), hash {}",
|
||||
log.len(),
|
||||
elapsed,
|
||||
log.len() as f64 / elapsed.as_secs_f64(),
|
||||
&hash[..8]
|
||||
);
|
||||
assert!(elapsed.as_secs_f64() < 5.0, "AM-7: 100k replay under 5s");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue