CI: enforce every gate; close the silent-skip holes

The gates existed; CI ran half of them and tolerated the failure case.

- cb-sim no longer has a "tolerable" non-zero exit. An unregistered game
  prefix is a failure, and a run in which nothing executed is a failure.
  Previously CI carried `|| test $? -eq 2`, so renaming a scenario prefix
  would have skipped every scenario while the pipeline stayed green.
  Verified with a negative control.
- CI now runs make coverage (AM-1) and make dep-weight (AM-4), both
  added after CI was written and neither enforced until now.
- dep-weight enforces its targets instead of only reporting them.
- CI lints the shipped-runtime configuration separately, so the feature
  split cannot rot unnoticed.
- Dropped the stale `make deps` target, which still measured the retired
  crate-count metric.

The positive-control rule is now executable: CI runs
`cargo bench -- --test`, which executes every benchmark once, so a
workload that stalls fails the build.

That step immediately found a fourth instance of the error class it was
written for. The committed replay benchmark was the broken version — an
earlier patch never applied, leaving a command sequence that omits
Resolve, so every round produced nothing and the log-building loop spun
forever. It had never run to completion; the reported AM-7 replay
numbers came from a probe test instead. Fixed, given the same positive
control as the round loop, and re-measured from the benchmark: 100k
events fold in 2.18ms (95% CI 2.14-2.23), against a 5s budget.

Evidence now reports confidence intervals rather than point estimates,
so the 3% regression rule in MetricsAndScenarios is enforceable.

The finding worth carrying: writing the positive-control rule into
InnerLoop v1.0 did not prevent the next instance. Making it a CI step
did.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 04:02:33 +02:00
parent 4be6e020ea
commit 72c594ee49
6 changed files with 169 additions and 70 deletions

View file

@ -1,7 +1,10 @@
//! cb-sim — scenario runner binary (GameKernel K17; precursor of `cb sim`).
//! Parses scenario files, dispatches each to its game by the `<game>/`
//! prefix of its `scenario` field, and executes it. Exit codes: 0 all
//! passed, 1 failures, 2 unknown game, 64 usage error.
//! passed, 1 any failure — including a scenario whose game prefix is not
//! registered, and a run in which nothing executed. There is deliberately
//! no "tolerable" non-zero exit: a silent skip is the failure mode this
//! binary exists to catch.
use cb_game_runtime::{scenario, RunOutcome, ScenarioFile};
use games_ground::GroundState;
@ -13,7 +16,6 @@ fn main() {
std::process::exit(64);
}
let mut unknown_game = false;
let mut failed = false;
let mut passed = 0usize;
let mut covered: Vec<String> = Vec::new();
@ -39,8 +41,14 @@ fn main() {
let outcome = match sc.scenario.split('/').next() {
Some("ground") => scenario::run::<GroundState>(&sc),
_ => {
println!("SKIP {} — no game registered for this prefix", sc.scenario);
unknown_game = true;
// A renamed or typo'd prefix would otherwise skip every
// scenario while the run still looked clean.
eprintln!(
"FAIL {} — no game registered for prefix {:?}",
sc.scenario,
sc.scenario.split('/').next().unwrap_or("")
);
failed = true;
continue;
}
};
@ -67,10 +75,14 @@ fn main() {
covered.dedup();
println!("{passed} passed, {} rules covered", covered.len());
// Positive control: a run that executed nothing must not pass. This
// is the same class of error as a benchmark timing rejected work.
if passed == 0 {
eprintln!("FAIL — no scenario executed; refusing to report success");
failed = true;
}
if failed {
std::process::exit(1);
}
if unknown_game {
std::process::exit(2);
}
}