CB-WP-0015: the two inert clauses, AM-7 scaling and AM-8 N=10
Some checks failed
ci / check (push) Failing after 3s
Some checks failed
ci / check (push) Failing after 3s
Provenance (tier S, one paragraph in lieu of survey and ADR): the two clauses mutation-check has reported inert since CB-WP-0005. AM-7's scaling ratio was held up by a test literally named replay_100k_events_is_linear_and_fast that computed both throughputs, printed both, and never divided one by the other. AM-8's N=10 was held up by a runner that does two. Both are now red. AM-7 3/3, AM-8 2/2, M-D1-MUT 10/14, and ADR-0005's >=10-of-14 prediction MET for the first time. Neither was closed by amending the question away, which was the live risk: the denominator is unchanged and the four unenforced rows are the four already unenforceable. AM-7 needed three estimators. Best-of-N per leg then divide (AM-6's, correct for a floor on one number) gave 0.581-1.085 on an unchanged binary; legs back-to-back gave medians 0.931-1.004; legs interleaved at fold granularity give 0.987/0.991/0.989, and 0.989 under 8-way CPU contention while absolute throughput fell 4x. The INDETERMINATE guard demanded unanimity and failed a good measurement over one sample 0.001 under the floor; it now requires a two-thirds majority. The control that matters: AM-6's constant-cost mutation halves throughput and leaves this ratio at 0.999x green, so AM-7 is not a second AM-6. AM-8 kept N=10 because the measurement said so. Perturbing the RNG only from its fourth construction on: --runs 2 PASSES, --runs 10 fails. A late-onset divergence is deterministic, not flaky, so it is a control rather than a coin flip. Ten runs live on one scenario (make am8, ~2s) rather than all 25 (47s a build). GameKernel 5b records it. The full run also found AM-4a's own mutation stale since ADR-0008 D3 moved the target 250,000 -> 161,000 in CB-WP-0013 -- reported HARNESS-BROKEN, no score published. The build-free half of that check is now a --self-test assertion, so make all catches the next one. mutation-check clauses may now carry their own verify and mutation, and then the enforced flag is measured rather than declared; a declaration disagreeing with its measurement is refused. make all exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
7e9ab221a7
commit
ee37b82675
9 changed files with 802 additions and 51 deletions
|
|
@ -205,6 +205,9 @@ where
|
|||
})
|
||||
}
|
||||
|
||||
/// K8's double-run: every scenario, every run, on `make sim`.
|
||||
pub const K8_RUNS: usize = 2;
|
||||
|
||||
/// Execute a scenario against game `G`: run twice with the same seed,
|
||||
/// compare hashes (K8), then check the `expect` block.
|
||||
pub fn run<G>(scenario: &ScenarioFile) -> RunOutcome
|
||||
|
|
@ -212,6 +215,33 @@ where
|
|||
G: ScenarioGame,
|
||||
G::Event: Serialize,
|
||||
{
|
||||
run_n::<G>(scenario, K8_RUNS)
|
||||
}
|
||||
|
||||
/// The same, with the replay count as a parameter (AM-8, CB-WP-0015 T02).
|
||||
///
|
||||
/// **What the extra runs are actually for.** A *deterministic* divergence
|
||||
/// — a seed threaded wrong, a fold that depends on insertion order —
|
||||
/// shows up on run 2 exactly as reliably as on run 10, so for that class
|
||||
/// `K8_RUNS` is sufficient and the other eight runs are repetitions of a
|
||||
/// check that already answered. The class that needs `N` is the
|
||||
/// *probabilistic* one: hash iteration order, address-dependent hashing,
|
||||
/// scheduling. There, `N` runs give `N-1` independent chances, and two
|
||||
/// runs catch a coin-flip divergence only half the time.
|
||||
///
|
||||
/// That class is meant to be structurally excluded here — `clippy.toml`
|
||||
/// denies `HashMap`/`HashSet` under K6 and `make check` runs `-D
|
||||
/// warnings`, which is AM-8's other clause. So `N` is defence in depth
|
||||
/// against the exclusion failing, not the primary control, and that is
|
||||
/// why it runs on one scenario in `make am8` rather than on all 25 in
|
||||
/// `make sim`: the latter costs 47 s per build to repeat an answered
|
||||
/// question 8 more times.
|
||||
pub fn run_n<G>(scenario: &ScenarioFile, runs: usize) -> RunOutcome
|
||||
where
|
||||
G: ScenarioGame,
|
||||
G::Event: Serialize,
|
||||
{
|
||||
assert!(runs >= 2, "a determinism check needs at least two runs");
|
||||
let first = match execute::<G>(scenario) {
|
||||
Ok(pass) => pass,
|
||||
Err(reason) => {
|
||||
|
|
@ -221,25 +251,30 @@ where
|
|||
}
|
||||
}
|
||||
};
|
||||
let second = match execute::<G>(scenario) {
|
||||
Ok(pass) => pass,
|
||||
Err(reason) => {
|
||||
return RunOutcome::Failed {
|
||||
reason,
|
||||
evidence: None,
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
if first.hash != second.hash {
|
||||
let reason = format!(
|
||||
"K8 divergence: run 1 hash {} != run 2 hash {}",
|
||||
first.hash, second.hash
|
||||
);
|
||||
return RunOutcome::Failed {
|
||||
evidence: Some(Box::new(evidence_of(&first))),
|
||||
reason,
|
||||
// Every later run is compared to the first, not to its predecessor: a
|
||||
// divergence that appears on run 5 and persists would otherwise be
|
||||
// invisible to a pairwise walk after run 6.
|
||||
for n in 2..=runs {
|
||||
let next = match execute::<G>(scenario) {
|
||||
Ok(pass) => pass,
|
||||
Err(reason) => {
|
||||
return RunOutcome::Failed {
|
||||
reason,
|
||||
evidence: None,
|
||||
}
|
||||
}
|
||||
};
|
||||
if first.hash != next.hash {
|
||||
let reason = format!(
|
||||
"K8 divergence: run 1 hash {} != run {n} hash {} (of {runs})",
|
||||
first.hash, next.hash
|
||||
);
|
||||
return RunOutcome::Failed {
|
||||
evidence: Some(Box::new(evidence_of(&first))),
|
||||
reason,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
if let Err(reason) = check(scenario, &first) {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue