CB-REV-0003: round 3, and three of four FATAL came from round 2's fixes
Some checks failed
ci / check (push) Failing after 3s
Some checks failed
ci / check (push) Failing after 3s
The pattern is now measured over three rounds: 5 fatal, then 3 (2 from the previous round's corrections), then 4 (3 from them). The corrections are not getting safer. FATAL 1: round 2's short-cell assertion went into regulation.rs only. attack-value.rs — which produced every number in CB-EV-0030's DARVO table — still just warned, and the gate registered to close the finding claimed the property for both. FATAL 2, the sharpest of the three rounds: counting games proves they STARTED. Stopping the engine after one round gives 200 games, all-zero columns and exit 0 — byte for byte the signature CB-EV-0030 says the instrumentation distinguishes from a real result. Both harnesses now require every counted game to have reached an outcome over five rounds. FATAL 3: round 2's `.csv` filter was applied to all three loops, so catalog.yaml and rules_delta.yaml — whose missing digests were round 1's finding — were recorded and then never compared, and never checked against upstream at all. Only the parser loop filters now. FATAL 4: five of six tiebreak comparators had no coverage. GR-E04's tiebreak never executes in any scenario. All four are now covered and mutation-verified; the Blame key needed compensating claims to be reachable at all, since Blame also lowers the coalition score. SERIOUS: "peak held" computed the same number as "peak assigned" for every possible input — the real gap was that START_STRESS was an unchecked constant, now read off the dealt state; cadence="none" was a pure loophole, removed; sibling discovery swapped a hand-written list for hand-written globs and missed metadata.json and VARIANT.md, both named in the package's own changed_files — now walked, and it found them immediately; and "~72,000 games" was unsourced, make panels runs 17,600. Also separated two kinds of number that were presented alike: seats×games is invariant, 363 and 29 vary 7.1%-11.5% across samples. Round 4 owed. The conclusion is not that the work is nearly right — it is that author-made corrections to measurement work should be assumed defective until a fresh reader has attacked them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
feb68027f9
commit
c4a8a227c0
9 changed files with 386 additions and 57 deletions
|
|
@ -24,6 +24,10 @@ use cb_kernel::PlayerId;
|
|||
use games_ground::bot::{play, Choice, GreedyPolicy, Policy};
|
||||
use games_ground::{Action, GroundCommand, GroundState, ScoringMode};
|
||||
|
||||
/// Games per cell. Named once so the footer and the assertion cannot
|
||||
/// disagree — the footer said "200 games per cell" over 195-game columns.
|
||||
const GAMES: u32 = 200;
|
||||
|
||||
/// Greedy's ordering with ATTACK's rank as a parameter.
|
||||
///
|
||||
/// A copy of the ranking rather than a call into it: `GreedyPolicy::rank`
|
||||
|
|
@ -98,9 +102,10 @@ fn sweep(
|
|||
mk: &dyn Fn(u8) -> Vec<Box<dyn Policy>>,
|
||||
) -> (u32, u32, u32, u32) {
|
||||
let (mut games, mut won, mut atk, mut darvo) = (0, 0, 0, 0);
|
||||
let mut played = 0u32;
|
||||
let mut errs: Vec<String> = Vec::new();
|
||||
let mut setup_fails = 0u32;
|
||||
for seed in 0..200u64 {
|
||||
for seed in 0..GAMES as u64 {
|
||||
let Ok(mut st) = GroundState::setup(
|
||||
&Setup {
|
||||
players,
|
||||
|
|
@ -130,6 +135,9 @@ fn sweep(
|
|||
}
|
||||
};
|
||||
games += 1;
|
||||
if g.state.outcome.is_some() && g.rounds == 5 {
|
||||
played += 1;
|
||||
}
|
||||
|
||||
// In the co-op mode the table wins together. In the other two the
|
||||
// question is whether SEAT 0 is among the winners — because that
|
||||
|
|
@ -161,20 +169,30 @@ fn sweep(
|
|||
}
|
||||
}
|
||||
}
|
||||
if setup_fails > 0 {
|
||||
eprintln!(" !! {setup_fails} of 200 SETUPS failed ({players}p)");
|
||||
}
|
||||
if games != 200 {
|
||||
eprintln!(" !! only {games} of 200 games ran ({players}p, {mode:?})");
|
||||
}
|
||||
if !errs.is_empty() {
|
||||
eprintln!(
|
||||
" !! {} of 200 games did not run ({}p): first = {}",
|
||||
errs.len(),
|
||||
players,
|
||||
errs[0]
|
||||
);
|
||||
}
|
||||
// **Asserted, not printed** (CB-REV-0003 #1). This harness produced
|
||||
// every number in CB-EV-0030's DARVO table, and it still only warned
|
||||
// on a short cell — the very defect fixed in `regulation.rs` and left
|
||||
// here, while the gate registered to close it claimed the property
|
||||
// for both.
|
||||
assert_eq!(
|
||||
games,
|
||||
GAMES,
|
||||
"{players}p {mode:?} {variant:?}: only {games} of {GAMES} games ran \
|
||||
({setup_fails} setups refused, {} play errors) — the cell is short, so \
|
||||
every number in it is over a sample nobody chose",
|
||||
errs.len()
|
||||
);
|
||||
// **And that a game RAN is not that it was PLAYED** (CB-REV-0003 #2).
|
||||
// Stopping the engine after one round gave 200 games, all-zero
|
||||
// columns and exit 0 — byte for byte the signature CB-EV-0030 §3
|
||||
// claims to distinguish from a real result. A counted game must have
|
||||
// reached an outcome.
|
||||
assert_eq!(
|
||||
played, GAMES,
|
||||
"{players}p {mode:?} {variant:?}: {played} of {games} games reached an \
|
||||
outcome — the rest stopped early, and all-zero columns would read as \
|
||||
a result rather than as nothing having happened"
|
||||
);
|
||||
(games, won, atk, darvo)
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -71,9 +71,15 @@ impl Policy for Reactive {
|
|||
}
|
||||
}
|
||||
|
||||
/// `Scenarios.csv`: "All players start at Stress 2." Named rather than
|
||||
/// inlined so the peak metric cannot silently disagree with setup.
|
||||
const START_STRESS: u8 = 2;
|
||||
/// The starting Stress, **read off the dealt state** rather than declared.
|
||||
///
|
||||
/// It was a `const 2` "named so the peak metric cannot silently disagree
|
||||
/// with setup" — and nothing compared it to setup, so raising it to 7
|
||||
/// printed `peak 7` in every cell with no warning (CB-REV-0003 #6).
|
||||
/// Naming a constant is not checking it.
|
||||
fn start_stress(state: &GroundState) -> u8 {
|
||||
state.players.values().map(|p| p.stress).max().unwrap_or(0)
|
||||
}
|
||||
|
||||
/// Games per cell. Named once so the assertion and the banner cannot
|
||||
/// disagree — the banner said "200 games per cell" over 196-game columns.
|
||||
|
|
@ -94,6 +100,13 @@ struct Cell {
|
|||
/// arm is real and can do nothing. Reported separately, because
|
||||
/// ground-game's criterion 1 is asking about DARVO *mattering*.
|
||||
inert_arms: u32,
|
||||
/// Games that reached an outcome over the full five rounds.
|
||||
///
|
||||
/// **`games` counts `play` returning `Ok`, which is not the claim.**
|
||||
/// Stopping the engine after one round produced 200 games, all-zero
|
||||
/// columns and exit 0 — the exact signature CB-EV-0030 §3 says the
|
||||
/// instrumentation distinguishes from a real result.
|
||||
played: u32,
|
||||
}
|
||||
|
||||
fn sweep(mode: ScoringMode, variant: Variant, players: u8, reactive: bool) -> Cell {
|
||||
|
|
@ -105,6 +118,7 @@ fn sweep(mode: ScoringMode, variant: Variant, players: u8, reactive: bool) -> Ce
|
|||
peak_stress: 0,
|
||||
setup_fails: 0,
|
||||
inert_arms: 0,
|
||||
played: 0,
|
||||
};
|
||||
for seed in 0..GAMES as u64 {
|
||||
let Ok(mut st) = GroundState::setup(
|
||||
|
|
@ -120,6 +134,9 @@ fn sweep(mode: ScoringMode, variant: Variant, players: u8, reactive: bool) -> Ce
|
|||
};
|
||||
st.mode = mode;
|
||||
st.variant = variant;
|
||||
// The dealt state, kept so the peak metric can read the real
|
||||
// starting Stress rather than trust a constant.
|
||||
let started = st.clone();
|
||||
let mut ps: Vec<Box<dyn Policy>> = (0..players)
|
||||
.map(|_| {
|
||||
if reactive {
|
||||
|
|
@ -138,6 +155,10 @@ fn sweep(mode: ScoringMode, variant: Variant, players: u8, reactive: bool) -> Ce
|
|||
}
|
||||
};
|
||||
c.games += 1;
|
||||
// A game that RAN is not a game that was PLAYED (CB-REV-0003 #2).
|
||||
if g.state.outcome.is_some() && g.rounds == 5 {
|
||||
c.played += 1;
|
||||
}
|
||||
if g.state.outcome.as_ref().is_some_and(|o| o.group_success) {
|
||||
c.won += 1;
|
||||
}
|
||||
|
|
@ -150,15 +171,19 @@ fn sweep(mode: ScoringMode, variant: Variant, players: u8, reactive: bool) -> Ce
|
|||
c.atk += 1;
|
||||
}
|
||||
}
|
||||
// Peak Stress **held**, not peak Stress *assigned*.
|
||||
// Peak Stress **held**.
|
||||
//
|
||||
// The first version took a maximum over `StressSet` PAYLOADS.
|
||||
// Starting Stress is 2 and is written by `setup`, never by an
|
||||
// event, so a table that sat at 2 all game reported **1**, and a
|
||||
// table with no `StressSet` at all would report 0. CB-EV-0031 §2
|
||||
// built its headline claim on that number. Wrong subject: the
|
||||
// metric answered "highest value ever assigned", the prose said
|
||||
// "highest Stress reached".
|
||||
// The original took a maximum over `StressSet` PAYLOADS, and the
|
||||
// starting value is written by `setup` and never by an event — so
|
||||
// a table that sat at 2 all game reported 1.
|
||||
//
|
||||
// **The first correction did not change the number it computes**
|
||||
// (CB-REV-0003 #5): `held`'s values are exactly
|
||||
// `{start} ∪ {payloads}`, so with the floor applied
|
||||
// `peak_held ≡ max(start, peak_payload)` for every possible input,
|
||||
// and the comment claiming a change of subject was false of the
|
||||
// new code too. The real fix is that `start` is now read off the
|
||||
// dealt state instead of asserted by a constant.
|
||||
// An arm is inert when NO `RoundEnded` follows it: the game
|
||||
// ended in the same End step, so the sequence never advances a
|
||||
// stage. `g.rounds >= 5` is a property of the GAME, not of the
|
||||
|
|
@ -169,9 +194,10 @@ fn sweep(mode: ScoringMode, variant: Variant, players: u8, reactive: bool) -> Ce
|
|||
.events
|
||||
.iter()
|
||||
.rposition(|e| matches!(e, games_ground::GroundEvent::RoundEnded { .. }));
|
||||
let start = start_stress(&started);
|
||||
let mut held: std::collections::BTreeMap<PlayerId, u8> =
|
||||
g.state.players.keys().map(|s| (*s, START_STRESS)).collect();
|
||||
c.peak_stress = c.peak_stress.max(START_STRESS);
|
||||
g.state.players.keys().map(|s| (*s, start)).collect();
|
||||
c.peak_stress = c.peak_stress.max(start);
|
||||
for (i, e) in g.events.iter().enumerate() {
|
||||
if matches!(e, games_ground::GroundEvent::DarvoTriggered { .. }) {
|
||||
c.darvo += 1;
|
||||
|
|
@ -190,7 +216,7 @@ fn sweep(mode: ScoringMode, variant: Variant, players: u8, reactive: bool) -> Ce
|
|||
// START_STRESS floor made the two agree.
|
||||
c.peak_stress = c
|
||||
.peak_stress
|
||||
.max(held.values().copied().max().unwrap_or(START_STRESS));
|
||||
.max(held.values().copied().max().unwrap_or(start));
|
||||
}
|
||||
}
|
||||
// **`games`, not `games + setup_fails`** (CB-REV-0002 #1).
|
||||
|
|
@ -207,6 +233,11 @@ fn sweep(mode: ScoringMode, variant: Variant, players: u8, reactive: bool) -> Ce
|
|||
the cell is short, so every number in it is over a sample nobody chose",
|
||||
c.games, c.setup_fails
|
||||
);
|
||||
assert_eq!(
|
||||
c.played, GAMES,
|
||||
"{players}p {variant:?}: {} of {} games reached an outcome over five rounds",
|
||||
c.played, c.games
|
||||
);
|
||||
if c.setup_fails > 0 {
|
||||
eprintln!(
|
||||
" !! {players}p {variant:?}: {} setups refused",
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue