CI: enforce every gate; close the silent-skip holes
The gates existed; CI ran half of them and tolerated the failure case. - cb-sim no longer has a "tolerable" non-zero exit. An unregistered game prefix is a failure, and a run in which nothing executed is a failure. Previously CI carried `|| test $? -eq 2`, so renaming a scenario prefix would have skipped every scenario while the pipeline stayed green. Verified with a negative control. - CI now runs make coverage (AM-1) and make dep-weight (AM-4), both added after CI was written and neither enforced until now. - dep-weight enforces its targets instead of only reporting them. - CI lints the shipped-runtime configuration separately, so the feature split cannot rot unnoticed. - Dropped the stale `make deps` target, which still measured the retired crate-count metric. The positive-control rule is now executable: CI runs `cargo bench -- --test`, which executes every benchmark once, so a workload that stalls fails the build. That step immediately found a fourth instance of the error class it was written for. The committed replay benchmark was the broken version — an earlier patch never applied, leaving a command sequence that omits Resolve, so every round produced nothing and the log-building loop spun forever. It had never run to completion; the reported AM-7 replay numbers came from a probe test instead. Fixed, given the same positive control as the round loop, and re-measured from the benchmark: 100k events fold in 2.18ms (95% CI 2.14-2.23), against a 5s budget. Evidence now reports confidence intervals rather than point estimates, so the 3% regression rule in MetricsAndScenarios is enforceable. The finding worth carrying: writing the positive-control rule into InnerLoop v1.0 did not prevent the next instance. Making it a CI step did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
4be6e020ea
commit
72c594ee49
6 changed files with 169 additions and 70 deletions
|
|
@ -1,7 +1,10 @@
|
|||
//! cb-sim — scenario runner binary (GameKernel K17; precursor of `cb sim`).
|
||||
//! Parses scenario files, dispatches each to its game by the `<game>/`
|
||||
//! prefix of its `scenario` field, and executes it. Exit codes: 0 all
|
||||
//! passed, 1 failures, 2 unknown game, 64 usage error.
|
||||
//! passed, 1 any failure — including a scenario whose game prefix is not
|
||||
//! registered, and a run in which nothing executed. There is deliberately
|
||||
//! no "tolerable" non-zero exit: a silent skip is the failure mode this
|
||||
//! binary exists to catch.
|
||||
|
||||
use cb_game_runtime::{scenario, RunOutcome, ScenarioFile};
|
||||
use games_ground::GroundState;
|
||||
|
|
@ -13,7 +16,6 @@ fn main() {
|
|||
std::process::exit(64);
|
||||
}
|
||||
|
||||
let mut unknown_game = false;
|
||||
let mut failed = false;
|
||||
let mut passed = 0usize;
|
||||
let mut covered: Vec<String> = Vec::new();
|
||||
|
|
@ -39,8 +41,14 @@ fn main() {
|
|||
let outcome = match sc.scenario.split('/').next() {
|
||||
Some("ground") => scenario::run::<GroundState>(&sc),
|
||||
_ => {
|
||||
println!("SKIP {} — no game registered for this prefix", sc.scenario);
|
||||
unknown_game = true;
|
||||
// A renamed or typo'd prefix would otherwise skip every
|
||||
// scenario while the run still looked clean.
|
||||
eprintln!(
|
||||
"FAIL {} — no game registered for prefix {:?}",
|
||||
sc.scenario,
|
||||
sc.scenario.split('/').next().unwrap_or("")
|
||||
);
|
||||
failed = true;
|
||||
continue;
|
||||
}
|
||||
};
|
||||
|
|
@ -67,10 +75,14 @@ fn main() {
|
|||
covered.dedup();
|
||||
println!("{passed} passed, {} rules covered", covered.len());
|
||||
|
||||
// Positive control: a run that executed nothing must not pass. This
|
||||
// is the same class of error as a benchmark timing rejected work.
|
||||
if passed == 0 {
|
||||
eprintln!("FAIL — no scenario executed; refusing to report success");
|
||||
failed = true;
|
||||
}
|
||||
|
||||
if failed {
|
||||
std::process::exit(1);
|
||||
}
|
||||
if unknown_game {
|
||||
std::process::exit(2);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -32,6 +32,13 @@ CONFIGS = {
|
|||
"dev-toolchain": [],
|
||||
}
|
||||
|
||||
# AM-4a / AM-4b targets from specs/GameKernel.md §4. Breaching one fails
|
||||
# the build: a gate that only reports is a suggestion.
|
||||
TARGETS = {
|
||||
"shipped-runtime": 250_000,
|
||||
"dev-toolchain": 350_000,
|
||||
}
|
||||
|
||||
|
||||
def crates(extra_args):
|
||||
"""Third-party crates in the normal (non-dev) dependency graph."""
|
||||
|
|
@ -116,9 +123,12 @@ def main():
|
|||
print(f" own source {own:>9,} lines")
|
||||
for label in CONFIGS:
|
||||
r = report[label]
|
||||
limit = TARGETS[label]
|
||||
mark = "ok " if r["third_party_loc"] <= limit else "FAIL"
|
||||
print(
|
||||
f" {label:<18}{r['crates']:>3} crates "
|
||||
f"{r['third_party_loc']:>9,} lines third-party"
|
||||
f"{r['third_party_loc']:>9,} lines third-party "
|
||||
f"[{mark} target {limit:,}]"
|
||||
)
|
||||
delta = (
|
||||
report["dev-toolchain"]["third_party_loc"]
|
||||
|
|
@ -131,7 +141,18 @@ def main():
|
|||
# shrink the total, so refuse to report rather than under-report.
|
||||
print("\nERROR — source not found for:", ", ".join(missing), file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
breached = [
|
||||
(label, report[label]["third_party_loc"], limit)
|
||||
for label, limit in TARGETS.items()
|
||||
if report[label]["third_party_loc"] > limit
|
||||
]
|
||||
for label, actual, limit in breached:
|
||||
print(
|
||||
f"\nFAIL AM-4 — {label}: {actual:,} lines exceeds target {limit:,}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1 if breached else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue