CI: enforce every gate; close the silent-skip holes

The gates existed; CI ran half of them and tolerated the failure case.

- cb-sim no longer has a "tolerable" non-zero exit. An unregistered game
  prefix is a failure, and a run in which nothing executed is a failure.
  Previously CI carried `|| test $? -eq 2`, so renaming a scenario prefix
  would have skipped every scenario while the pipeline stayed green.
  Verified with a negative control.
- CI now runs make coverage (AM-1) and make dep-weight (AM-4), both
  added after CI was written and neither enforced until now.
- dep-weight enforces its targets instead of only reporting them.
- CI lints the shipped-runtime configuration separately, so the feature
  split cannot rot unnoticed.
- Dropped the stale `make deps` target, which still measured the retired
  crate-count metric.

The positive-control rule is now executable: CI runs
`cargo bench -- --test`, which executes every benchmark once, so a
workload that stalls fails the build.

That step immediately found a fourth instance of the error class it was
written for. The committed replay benchmark was the broken version — an
earlier patch never applied, leaving a command sequence that omits
Resolve, so every round produced nothing and the log-building loop spun
forever. It had never run to completion; the reported AM-7 replay
numbers came from a probe test instead. Fixed, given the same positive
control as the round loop, and re-measured from the benchmark: 100k
events fold in 2.18ms (95% CI 2.14-2.23), against a 5s budget.

Evidence now reports confidence intervals rather than point estimates,
so the 3% regression rule in MetricsAndScenarios is enforceable.

The finding worth carrying: writing the positive-control rule into
InnerLoop v1.0 did not prevent the next instance. Making it a CI step
did.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 04:02:33 +02:00
parent 4be6e020ea
commit 72c594ee49
6 changed files with 169 additions and 70 deletions

View file

@ -21,7 +21,7 @@ Machine: WSL2, Linux 6.18.33.2-microsoft-standard-WSL2, rustc 1.97.1,
| AM-4b dep weight, dev toolchain | ≤350,000 third-party lines | 317,021 (29 crates) | **met** |
| AM-6 throughput | ≥100,000 events/s | 1,651,400 events/s | **met, 16.5×** |
| AM-7 scaling | ≥0.9× at 20× workload | 1.08× | **met** |
| AM-7 replay | 100k events ≤5s | 4.13 ms | **met, 1,210×** |
| AM-7 replay | 100k events ≤5s | 2.18 ms (CI 2.142.23) | **met, 2,290×** |
| AM-8 determinism | zero divergence, 10 replays | 1 distinct hash / 10 runs | **met** |
| AM-8 lint | fmt + clippy clean | clean, `-D warnings` | **met** |
| AM-10 foreign types | zero `HashMap`/`HashSet` | 0 | **met** |
@ -45,12 +45,17 @@ breaks the test rather than silently rescaling the metric.
| 40,000 | 1,626,000 | 1.07× |
| 100,000 | 1,651,400 | 1.08× |
Replay — folding one growing event log back into state:
Replay — folding one growing event log back into state (Criterion
95% CI, low/median/high):
| Events | Time | Rate |
|---|---|---|
| 10,010 | 465 µs | 21.5M events/s |
| 100,007 | 4.13 ms | 24.2M events/s |
| Events | Time (median) | 95% CI | Rate |
|---|---|---|---|
| 10,010 | 184.8 µs | 180.9 189.3 µs | 54.2M events/s |
| 100,072 | 2.183 ms | 2.142 2.226 ms | 45.8M events/s |
**Correction (2026-07-31).** These originally read 465 µs / 4.13 ms and
were taken from the `replay_probe` **test**, not from the benchmark —
because the benchmark did not work. See §2a.
### The comparison against boardgame.io, stated carefully
@ -100,6 +105,30 @@ The benchmark now asserts the per-round event count on every round and
panics rather than measuring a stalled loop. The corrected figure is
**5.6× lower** than the bogus one.
### 2a. A fourth measurement error, found by enforcing the rule
The replay benchmark committed alongside this evidence was **the broken
version**. A `python3` patch that was supposed to replace its
log-building loop never applied, leaving a sequence that omits `Resolve`
— so `EndRound` was rejected, every round produced no events, and the
`while log.len() < target` loop spun forever. It was never run to
completion; the AM-7 replay numbers were taken from a separate probe
test instead, and the dead benchmark was committed and left hanging.
Found by adding `cargo bench -- --test` to CI, which runs every
benchmark once. That is the fourth instance of one error class in this
project — a harness that appears to work while doing no work — and the
**first one caught by a gate rather than by noticing**.
The replay loop now carries the positive control the round loop already
had: it asserts each round appended events and fails rather than
spinning. Corrected figures are in the table above; both configurations
still clear the AM-7 budget by three orders of magnitude.
The lesson recorded for the loop: writing the positive-control rule into
`specs/InnerLoop.md` did **not** prevent the next instance. Making it a
CI step did. Prose rules do not enforce themselves.
## 3. Determinism (AM-8)
- Every scenario runs twice per invocation with the same seed and fails