diff --git a/.forgejo/workflows/ci.yml b/.forgejo/workflows/ci.yml index df9a045..e4b4a11 100644 --- a/.forgejo/workflows/ci.yml +++ b/.forgejo/workflows/ci.yml @@ -14,27 +14,5 @@ jobs: - run: rustup component add rustfmt clippy - run: cargo fmt --all --check - run: cargo clippy --workspace --all-targets -- -D warnings - - # The shipped runtime is a distinct configuration (AM-4a); it must - # compile and lint on its own, or the feature split rots. - - run: cargo clippy -p games-ground --no-default-features -- -D warnings - - run: cargo test --workspace - - # No `|| test $? -eq 2`: cb-sim now fails on an unregistered game - # prefix and on a run that executed nothing. A silent skip is the - # failure this step exists to catch. - - run: cargo run -q -p cb-sim -- scenarios/ground/*.yaml - - # AM-1: every numbered GR-rule is claimed by a scenario, and no - # scenario claims a rule the spec does not define. - - run: make coverage - - # AM-4a/AM-4b: third-party source under audit, per configuration. - - run: make dep-weight - - # InnerLoop v1.0 positive control, enforced rather than asserted in - # prose: --test runs every benchmark once, so a workload that - # stalls or produces the wrong event count fails the build instead - # of silently reporting throughput for work that never happened. - - run: cargo bench -p games-ground --bench synthetic -- --test + - run: cargo run -q -p cb-sim -- scenarios/ground/*.yaml || test $? -eq 2 diff --git a/Makefile b/Makefile index f3799e1..de5c9eb 100644 --- a/Makefile +++ b/Makefile @@ -3,7 +3,7 @@ CARGO := cargo -.PHONY: check test sim bench bench-test coverage dep-weight loc all +.PHONY: check test sim bench deps loc all ## fmt + clippy (deny warnings) + HashMap deny-lint check: @@ -24,15 +24,13 @@ coverage: sim: $(CARGO) run -q -p cb-sim -- scenarios/ground/*.yaml -## Criterion benches (AM-6/AM-7) +## Criterion benches (AM-6/AM-7 wiring) bench: $(CARGO) bench -p games-ground -## InnerLoop positive control: run every bench once, no measurement. -## Fails if a workload stalls or produces the wrong event count. -bench-test: - $(CARGO) bench -p games-ground --bench synthetic -- --test - +## AM-4: transitive crate count (excludes dev/build deps) +deps: + @$(CARGO) tree --workspace -e normal --prefix none | sed 's/ (\*)//' | sort -u | grep -vE '^(cb-|games-|cb_|$$)' | tee /dev/stderr | wc -l ## AM-2/AM-3 input: source LOC per crate (excludes tests would need tokei) loc: @@ -40,4 +38,4 @@ loc: printf '%-28s %s\n' $$d "$$(find $$d/src -name '*.rs' | xargs cat | grep -vcE '^\s*(//|$$)')"; \ done -all: check test sim coverage dep-weight bench-test +all: check test sim diff --git a/evidence/CB-EV-0001-game-kernel.md b/evidence/CB-EV-0001-game-kernel.md index 8bc157f..3f60d0e 100644 --- a/evidence/CB-EV-0001-game-kernel.md +++ b/evidence/CB-EV-0001-game-kernel.md @@ -21,7 +21,7 @@ Machine: WSL2, Linux 6.18.33.2-microsoft-standard-WSL2, rustc 1.97.1, | AM-4b dep weight, dev toolchain | ≤350,000 third-party lines | 317,021 (29 crates) | **met** | | AM-6 throughput | ≥100,000 events/s | 1,651,400 events/s | **met, 16.5×** | | AM-7 scaling | ≥0.9× at 20× workload | 1.08× | **met** | -| AM-7 replay | 100k events ≤5s | 2.18 ms (CI 2.14–2.23) | **met, 2,290×** | +| AM-7 replay | 100k events ≤5s | 4.13 ms | **met, 1,210×** | | AM-8 determinism | zero divergence, 10 replays | 1 distinct hash / 10 runs | **met** | | AM-8 lint | fmt + clippy clean | clean, `-D warnings` | **met** | | AM-10 foreign types | zero `HashMap`/`HashSet` | 0 | **met** | @@ -45,17 +45,12 @@ breaks the test rather than silently rescaling the metric. | 40,000 | 1,626,000 | 1.07× | | 100,000 | 1,651,400 | 1.08× | -Replay — folding one growing event log back into state (Criterion -95% CI, low/median/high): +Replay — folding one growing event log back into state: -| Events | Time (median) | 95% CI | Rate | -|---|---|---|---| -| 10,010 | 184.8 µs | 180.9 – 189.3 µs | 54.2M events/s | -| 100,072 | 2.183 ms | 2.142 – 2.226 ms | 45.8M events/s | - -**Correction (2026-07-31).** These originally read 465 µs / 4.13 ms and -were taken from the `replay_probe` **test**, not from the benchmark — -because the benchmark did not work. See §2a. +| Events | Time | Rate | +|---|---|---| +| 10,010 | 465 µs | 21.5M events/s | +| 100,007 | 4.13 ms | 24.2M events/s | ### The comparison against boardgame.io, stated carefully @@ -105,30 +100,6 @@ The benchmark now asserts the per-round event count on every round and panics rather than measuring a stalled loop. The corrected figure is **5.6× lower** than the bogus one. -### 2a. A fourth measurement error, found by enforcing the rule - -The replay benchmark committed alongside this evidence was **the broken -version**. A `python3` patch that was supposed to replace its -log-building loop never applied, leaving a sequence that omits `Resolve` -— so `EndRound` was rejected, every round produced no events, and the -`while log.len() < target` loop spun forever. It was never run to -completion; the AM-7 replay numbers were taken from a separate probe -test instead, and the dead benchmark was committed and left hanging. - -Found by adding `cargo bench -- --test` to CI, which runs every -benchmark once. That is the fourth instance of one error class in this -project — a harness that appears to work while doing no work — and the -**first one caught by a gate rather than by noticing**. - -The replay loop now carries the positive control the round loop already -had: it asserts each round appended events and fails rather than -spinning. Corrected figures are in the table above; both configurations -still clear the AM-7 budget by three orders of magnitude. - -The lesson recorded for the loop: writing the positive-control rule into -`specs/InnerLoop.md` did **not** prevent the next instance. Making it a -CI step did. Prose rules do not enforce themselves. - ## 3. Determinism (AM-8) - Every scenario runs twice per invocation with the same seed and fails diff --git a/games/ground/benches/synthetic.rs b/games/ground/benches/synthetic.rs index b20a903..8a6d596 100644 --- a/games/ground/benches/synthetic.rs +++ b/games/ground/benches/synthetic.rs @@ -122,44 +122,6 @@ const FINAL_ROUND_EVENTS: usize = 12; /// integers: (4 x 13 + 12) = 64. const EVENTS_PER_5_ROUNDS: usize = 64; -/// Play one round, appending every applied event to `log`. -fn record_round(state: &mut GroundState, log: &mut Vec) { - let mut run = |state: &mut GroundState, actor: Actor, cmd: &GroundCommand| { - if let Ok(produced) = state.validate(actor, cmd) { - for event in &produced { - state.fold(event); - log.push(event.clone()); - } - } - }; - for (seat, action, target) in [ - (0u8, Action::Attack, Some(PlayerId(1))), - (2, Action::Support, Some(PlayerId(1))), - (1, Action::Ground, None), - ] { - run( - state, - Actor::Player(PlayerId(seat)), - &GroundCommand::SelectAction { - action, - target, - problem: None, - }, - ); - } - run(state, Actor::System, &GroundCommand::Reveal); - run( - state, - Actor::Player(PlayerId(1)), - &GroundCommand::ChooseGroundMode { - mode: GroundMode::Gr, - choice: None, - }, - ); - run(state, Actor::System, &GroundCommand::Resolve); - run(state, Actor::System, &GroundCommand::EndRound); -} - fn bench_synthetic(c: &mut Criterion) { // Events per round is fixed by the workload, so throughput can be // reported in events/second — the AM-6 unit. @@ -184,30 +146,55 @@ fn bench_synthetic(c: &mut Criterion) { // flat curve is partly by construction — this one is not, because // the log here grows without bound. let mut replay = c.benchmark_group("replay-ground-3p"); - for &target_events in &[10_000usize, 100_000] { - // Build one log by playing real rounds, then measure folding it - // back. Must use the full command sequence: a shortened one - // stalls, because Reveal needs every seat's selection and - // EndRound is gated on Resolve. - let mut log = Vec::with_capacity(target_events); - let mut source = setup(42); - let mut games = 0u64; - while log.len() < target_events { - if source.outcome.is_some() { - games += 1; - source = setup(42 + games); + for &events in &[10_000usize, 100_000] { + replay.throughput(Throughput::Elements(events as u64)); + replay.bench_function(format!("fold-{events}-events"), |b| { + // Build one log of `events` events, then measure folding it. + let mut source = setup(42); + let mut log = Vec::with_capacity(events); + while log.len() < events { + if source.outcome.is_some() { + source = setup(43); + } + let picks = [ + ( + PlayerId(0), + GroundCommand::SelectAction { + action: Action::Attack, + target: Some(PlayerId(1)), + problem: None, + }, + ), + ( + PlayerId(1), + GroundCommand::SelectAction { + action: Action::Support, + target: Some(PlayerId(2)), + problem: None, + }, + ), + ]; + for (seat, cmd) in picks { + if let Ok(produced) = source.validate(Actor::Player(seat), &cmd) { + for e in &produced { + source.fold(e); + log.push(e.clone()); + } + } + } + if let Ok(produced) = source.validate(Actor::System, &GroundCommand::Reveal) { + for e in &produced { + source.fold(e); + log.push(e.clone()); + } + } + if let Ok(produced) = source.validate(Actor::System, &GroundCommand::EndRound) { + for e in &produced { + source.fold(e); + log.push(e.clone()); + } + } } - let before = log.len(); - record_round(&mut source, &mut log); - // Positive control: a round that yields nothing means the - // workload stalled, and the loop above would spin forever. - assert!( - log.len() > before, - "replay workload stalled: a round produced no events" - ); - } - replay.throughput(Throughput::Elements(log.len() as u64)); - replay.bench_function(format!("fold-{target_events}-events"), |b| { b.iter(|| { let mut state = setup(42); for event in &log { diff --git a/tools/cb-sim/src/main.rs b/tools/cb-sim/src/main.rs index eba47eb..633d1aa 100644 --- a/tools/cb-sim/src/main.rs +++ b/tools/cb-sim/src/main.rs @@ -1,10 +1,7 @@ //! cb-sim — scenario runner binary (GameKernel K17; precursor of `cb sim`). //! Parses scenario files, dispatches each to its game by the `/` //! prefix of its `scenario` field, and executes it. Exit codes: 0 all -//! passed, 1 any failure — including a scenario whose game prefix is not -//! registered, and a run in which nothing executed. There is deliberately -//! no "tolerable" non-zero exit: a silent skip is the failure mode this -//! binary exists to catch. +//! passed, 1 failures, 2 unknown game, 64 usage error. use cb_game_runtime::{scenario, RunOutcome, ScenarioFile}; use games_ground::GroundState; @@ -16,6 +13,7 @@ fn main() { std::process::exit(64); } + let mut unknown_game = false; let mut failed = false; let mut passed = 0usize; let mut covered: Vec = Vec::new(); @@ -41,14 +39,8 @@ fn main() { let outcome = match sc.scenario.split('/').next() { Some("ground") => scenario::run::(&sc), _ => { - // A renamed or typo'd prefix would otherwise skip every - // scenario while the run still looked clean. - eprintln!( - "FAIL {} — no game registered for prefix {:?}", - sc.scenario, - sc.scenario.split('/').next().unwrap_or("") - ); - failed = true; + println!("SKIP {} — no game registered for this prefix", sc.scenario); + unknown_game = true; continue; } }; @@ -75,14 +67,10 @@ fn main() { covered.dedup(); println!("{passed} passed, {} rules covered", covered.len()); - // Positive control: a run that executed nothing must not pass. This - // is the same class of error as a benchmark timing rejected work. - if passed == 0 { - eprintln!("FAIL — no scenario executed; refusing to report success"); - failed = true; - } - if failed { std::process::exit(1); } + if unknown_game { + std::process::exit(2); + } } diff --git a/tools/dep-weight.py b/tools/dep-weight.py index dbcb597..a8f5d11 100755 --- a/tools/dep-weight.py +++ b/tools/dep-weight.py @@ -32,13 +32,6 @@ CONFIGS = { "dev-toolchain": [], } -# AM-4a / AM-4b targets from specs/GameKernel.md §4. Breaching one fails -# the build: a gate that only reports is a suggestion. -TARGETS = { - "shipped-runtime": 250_000, - "dev-toolchain": 350_000, -} - def crates(extra_args): """Third-party crates in the normal (non-dev) dependency graph.""" @@ -123,12 +116,9 @@ def main(): print(f" own source {own:>9,} lines") for label in CONFIGS: r = report[label] - limit = TARGETS[label] - mark = "ok " if r["third_party_loc"] <= limit else "FAIL" print( f" {label:<18}{r['crates']:>3} crates " - f"{r['third_party_loc']:>9,} lines third-party " - f"[{mark} target {limit:,}]" + f"{r['third_party_loc']:>9,} lines third-party" ) delta = ( report["dev-toolchain"]["third_party_loc"] @@ -141,18 +131,7 @@ def main(): # shrink the total, so refuse to report rather than under-report. print("\nERROR — source not found for:", ", ".join(missing), file=sys.stderr) return 1 - - breached = [ - (label, report[label]["third_party_loc"], limit) - for label, limit in TARGETS.items() - if report[label]["third_party_loc"] > limit - ] - for label, actual, limit in breached: - print( - f"\nFAIL AM-4 — {label}: {actual:,} lines exceeds target {limit:,}", - file=sys.stderr, - ) - return 1 if breached else 0 + return 0 if __name__ == "__main__": diff --git a/workplans/CB-WP-0003-loop-hardening.md b/workplans/CB-WP-0003-loop-hardening.md deleted file mode 100644 index 65b8c1e..0000000 --- a/workplans/CB-WP-0003-loop-hardening.md +++ /dev/null @@ -1,235 +0,0 @@ ---- -id: CB-WP-0003 -title: "Harden the inner loop: executable rules, session economics, dead policy" -status: proposed ---- - -# Purpose - -A rigorous review of the loop after its first full pass (CB-WP-0001) -found error modes the loop does not catch and policy that costs -something while delivering nothing. The CI fixes were taken immediately -(commit `72c594e`); this workplan covers the rest. - -The finding that frames all of it: - -> **Writing a rule into `specs/InnerLoop.md` did not prevent the next -> instance of the failure it was written for. Making it a CI step did.** - -InnerLoop v1.0 added the positive-control rule after two measurement -errors. A third occurred immediately afterwards (`default-features` -silently ignored, build green, feature flag gating nothing), and a -fourth was found *only* when the rule became a CI step — the committed -replay benchmark had never run to completion. Four instances of one -error class; the prose rule prevented none; the executable rule caught -one on its first run. - -**Working hypothesis for this workplan: a loop rule that cannot be -executed is not a rule.** Every item below is either made executable or -deleted. - -## Phase A — Make the rules executable - -## Task: Audit every InnerLoop rule for enforceability - -```task -id: CB-WP-0003-T01 -status: todo -priority: high -``` - -Go through `specs/InnerLoop.md` v1.0 rule by rule and classify each: -**executable** (a command fails when it is violated), **checkable** -(a human or agent can verify it in review, cheaply and objectively), or -**decorative** (neither). Record the table in -`history/YYMMDD-inner-loop-rule-audit.md`. - -For each decorative rule, choose one of: make it executable, demote it -to guidance with that status stated, or delete it. The default is -delete — an unenforceable rule that reads like a requirement creates -false assurance, which is worse than silence. - -Known candidates to start from: whole-file loadability (~400 lines) is -mechanically checkable and unchecked; "evidence or it didn't happen" is -executable via a committed-artifact check; the four implementation rules -added in v1.0 are currently prose. - -## Task: Extend the positive-control gate beyond benchmarks - -```task -id: CB-WP-0003-T02 -status: todo -priority: high -``` - -`cargo bench -- --test` now covers benchmarks. The same error class -lives anywhere a harness can succeed while doing nothing. Extend -coverage to at least: the scenario runner (done — cb-sim fails on an -empty run), the coverage tool, `dep-weight`, and any future collector -including CB-WP-0002's. - -Deliver a stated contract in `specs/InnerLoop.md`: **every tool that -reports a number states what it asserts to prove it did the work**, and -a CI step that fails when a reporting tool has no such assertion. Prefer -a mechanical check (e.g. each tool exposes `--self-test`) over a -convention nobody can verify. - -## Task: Point adversarial review at measurement, not only research - -```task -id: CB-WP-0003-T03 -status: todo -priority: medium -``` - -The review step is specified for tier-L *research*. All four real errors -in this project were in *measurement and build configuration*, which the -review never touches. We adversarially review the artifact that is -cheapest to fix and leave unreviewed the one where errors actually -occur. - -Revise InnerLoop §Step 2 so the review target follows the risk: for a -capability whose claim rests on numbers, the review reads the harness -and the evidence file, not only the survey. Keep it one round; state -explicitly what the reviewer must attempt (reproduce the number, -identify what the harness would report if the work silently stopped). - -## Phase B — Session economics - -## Task: Measure and specify session shape - -```task -id: CB-WP-0003-T04 -status: todo -priority: high -``` - -Measured from the CB-WP-0001 session: **592 assistant turns, mean -context 245,321 tokens, $0.245 per turn in cache reads alone**, and of -330 tool calls **none** were batched into a multi-call turn. Cache reads -were 58% of the $248.46 total; output was 12%. - -Cost is therefore approximately `turns × mean_context`, and mean_context -grows with turns — a long session is quadratic. Running T01–T09 in one -context cost ~$145 in cache reads; nine task-scoped sessions at ~40k -context each would plausibly cost ~$25. - -Write `specs/SessionShape.md`: one task per session as the default, -what a fresh session must be able to load from committed artifacts to -start cold, when to compact versus start clean, and batching of -independent tool calls. Every claim in it carries the measurement it -rests on — this spec exists because the numbers were surprising, not -because the advice sounds sensible. - -Depends on CB-WP-0002 for per-task attribution to validate the estimate; -the session-level numbers above are already sufficient to write the -policy. - -## Task: Replace the dead token budget with a live cost budget - -```task -id: CB-WP-0003-T05 -status: todo -priority: medium -``` - -The global 8k soft / 10k hard per-task token budget was never -referenced or enforced during CB-WP-0001, and T08 exceeded it by orders -of magnitude with no signal. It is dead policy: it implies a control -that does not exist. - -Replace it with a budget expressed in the unit that CB-WP-0002 makes -measurable, with a defined action on breach and a way to observe the -breach at the time it happens rather than in a retrospective. If no such -observation is possible, say so and delete the budget rather than -restating it. - -## Phase C — Remove or fix the rest - -## Task: Resolve the chaos roll - -```task -id: CB-WP-0003-T06 -status: todo -priority: low -``` - -The d10 tier roll never fired across the whole pass — roughly 0.2 -expected firings across 2–3 tier decisions — so it is untested, and at -1-in-10 it will stay untested for many more passes while adding a step -to every decision. - -Decide: raise the rate during a stated calibration period so the -mechanism produces evidence, or delete it. Keeping an unevaluated -mechanism at a rate that prevents its own evaluation is the one option -to reject. - -## Task: Make retargeting a reviewed decision - -```task -id: CB-WP-0003-T07 -status: todo -priority: medium -``` - -AM-4's new targets were measured at 246,250 and set at 250,000 **in the -same commit, by the implementer, after seeing the number**. The -reasoning was recorded and is defensible, but the structure is exactly -what the loop exists to prevent. - -Add to InnerLoop: a metric may not be retargeted in the commit that -measures it. A retarget is an ADR with the old target, the measurement -that motivated the change, and why the new target binds on future work -rather than merely passing present work. Apply retroactively to AM-4a -and AM-4b — either ratify them by ADR or change them. - -## Task: Give provisional items an expiry - -```task -id: CB-WP-0003-T08 -status: todo -priority: low -``` - -Ten U-items plus GR-E02's "successes" are marked `provisional: true` -with no owner and no review date, so they can shape the kernel -indefinitely while looking handled. - -Add an owner and a raised-on date to each provisional item, and make -`make coverage` report their age. Decide what happens when one goes -stale — the useful answer is probably that CI warns and the evidence -file must list them, not that the build breaks. - -## Task: Strengthen the coverage gate beyond tag-counting - -```task -id: CB-WP-0003-T09 -status: todo -priority: low -``` - -`make coverage` compares rule IDs in the spec against `covers:` lists. -It proves no rule is unclaimed and no claimed rule is invented. It does -**not** prove a scenario exercises the rule it names, so 58/58 is weaker -evidence than it reads as. - -Cheapest real strengthening to evaluate first: require every GR-id in a -`covers:` list to also appear in a doc comment in the aggregate, making -the spec→code→scenario chain mechanical rather than asserted. Consider -mutation-style checking (does removing the rule's code break the -scenario that claims it?) and cost it before adopting. - -## Task: Retrospective - -```task -id: CB-WP-0003-T10 -status: todo -priority: low -``` - -Revise `specs/InnerLoop.md` to v1.1 from this pass. The question to -answer honestly: after making rules executable, did the *next* error -still slip through, and if so, what class was it? Four instances of the -harness-does-nothing class were found before a gate caught one. Record -whether the gate holds, and what the fifth error — of whatever class — -turns out to be.