coulomb/clay-borg
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

CI: enforce every gate; close the silent-skip holes

Browse source 
The gates existed; CI ran half of them and tolerated the failure case.

- cb-sim no longer has a "tolerable" non-zero exit. An unregistered game
  prefix is a failure, and a run in which nothing executed is a failure.
  Previously CI carried `|| test $? -eq 2`, so renaming a scenario prefix
  would have skipped every scenario while the pipeline stayed green.
  Verified with a negative control.
- CI now runs make coverage (AM-1) and make dep-weight (AM-4), both
  added after CI was written and neither enforced until now.
- dep-weight enforces its targets instead of only reporting them.
- CI lints the shipped-runtime configuration separately, so the feature
  split cannot rot unnoticed.
- Dropped the stale `make deps` target, which still measured the retired
  crate-count metric.

The positive-control rule is now executable: CI runs
`cargo bench -- --test`, which executes every benchmark once, so a
workload that stalls fails the build.

That step immediately found a fourth instance of the error class it was
written for. The committed replay benchmark was the broken version — an
earlier patch never applied, leaving a command sequence that omits
Resolve, so every round produced nothing and the log-building loop spun
forever. It had never run to completion; the reported AM-7 replay
numbers came from a probe test instead. Fixed, given the same positive
control as the round loop, and re-measured from the benchmark: 100k
events fold in 2.18ms (95% CI 2.14-2.23), against a 5s budget.

Evidence now reports confidence intervals rather than point estimates,
so the 3% regression rule in MetricsAndScenarios is enforceable.

The finding worth carrying: writing the positive-control rule into
InnerLoop v1.0 did not prevent the next instance. Making it a CI step
did.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 04:02:33 +02:00
parent 4be6e020ea
commit 72c594ee49
6 changed files with 169 additions and 70 deletions
Download patch file Download diff file Expand all files Collapse all files

109
games/ground/benches/synthetic.rs
View file

@ -122,6 +122,44 @@ const FINAL_ROUND_EVENTS: usize = 12;
/// integers: (4 x 13 + 12) = 64.
const EVENTS_PER_5_ROUNDS: usize = 64;
/// Play one round, appending every applied event to `log`.
fn record_round(state: &mut GroundState, log: &mut Vec<games_ground::GroundEvent>) {
let mut run = |state: &mut GroundState, actor: Actor, cmd: &GroundCommand| {
if let Ok(produced) = state.validate(actor, cmd) {
for event in &produced {
state.fold(event);
log.push(event.clone());
}
}
};
for (seat, action, target) in [
(0u8, Action::Attack, Some(PlayerId(1))),
(2, Action::Support, Some(PlayerId(1))),
(1, Action::Ground, None),
] {
run(
state,
Actor::Player(PlayerId(seat)),
&GroundCommand::SelectAction {
action,
target,
problem: None,
},
);
}
run(state, Actor::System, &GroundCommand::Reveal);
run(
state,
Actor::Player(PlayerId(1)),
&GroundCommand::ChooseGroundMode {
mode: GroundMode::Gr,
choice: None,
},
);
run(state, Actor::System, &GroundCommand::Resolve);
run(state, Actor::System, &GroundCommand::EndRound);
}
fn bench_synthetic(c: &mut Criterion) {
// Events per round is fixed by the workload, so throughput can be
// reported in events/second — the AM-6 unit.
@ -146,55 +184,30 @@ fn bench_synthetic(c: &mut Criterion) {
// flat curve is partly by construction — this one is not, because
// the log here grows without bound.
let mut replay = c.benchmark_group("replay-ground-3p");
for &events in &[10_000usize, 100_000] {
replay.throughput(Throughput::Elements(events as u64));
replay.bench_function(format!("fold-{events}-events"), |b| {
// Build one log of `events` events, then measure folding it.
let mut source = setup(42);
let mut log = Vec::with_capacity(events);
while log.len() < events {
if source.outcome.is_some() {
source = setup(43);
}
let picks = [
(
PlayerId(0),
GroundCommand::SelectAction {
action: Action::Attack,
target: Some(PlayerId(1)),
problem: None,
},
),
(
PlayerId(1),
GroundCommand::SelectAction {
action: Action::Support,
target: Some(PlayerId(2)),
problem: None,
},
),
];
for (seat, cmd) in picks {
if let Ok(produced) = source.validate(Actor::Player(seat), &cmd) {
for e in &produced {
source.fold(e);
log.push(e.clone());
}
}
}
if let Ok(produced) = source.validate(Actor::System, &GroundCommand::Reveal) {
for e in &produced {
source.fold(e);
log.push(e.clone());
}
}
if let Ok(produced) = source.validate(Actor::System, &GroundCommand::EndRound) {
for e in &produced {
source.fold(e);
log.push(e.clone());
}
}
for &target_events in &[10_000usize, 100_000] {
// Build one log by playing real rounds, then measure folding it
// back. Must use the full command sequence: a shortened one
// stalls, because Reveal needs every seat's selection and
// EndRound is gated on Resolve.
let mut log = Vec::with_capacity(target_events);
let mut source = setup(42);
let mut games = 0u64;
while log.len() < target_events {
if source.outcome.is_some() {
games += 1;
source = setup(42 + games);
}
let before = log.len();
record_round(&mut source, &mut log);
// Positive control: a round that yields nothing means the
// workload stalled, and the loop above would spin forever.
assert!(
log.len() > before,
"replay workload stalled: a round produced no events"
);
}
replay.throughput(Throughput::Elements(log.len() as u64));
replay.bench_function(format!("fold-{target_events}-events"), |b| {
b.iter(|| {
let mut state = setup(42);
for event in &log {