CB-WP-0015: the two inert clauses, AM-7 scaling and AM-8 N=10
Some checks failed
ci / check (push) Failing after 3s
Some checks failed
ci / check (push) Failing after 3s
Provenance (tier S, one paragraph in lieu of survey and ADR): the two clauses mutation-check has reported inert since CB-WP-0005. AM-7's scaling ratio was held up by a test literally named replay_100k_events_is_linear_and_fast that computed both throughputs, printed both, and never divided one by the other. AM-8's N=10 was held up by a runner that does two. Both are now red. AM-7 3/3, AM-8 2/2, M-D1-MUT 10/14, and ADR-0005's >=10-of-14 prediction MET for the first time. Neither was closed by amending the question away, which was the live risk: the denominator is unchanged and the four unenforced rows are the four already unenforceable. AM-7 needed three estimators. Best-of-N per leg then divide (AM-6's, correct for a floor on one number) gave 0.581-1.085 on an unchanged binary; legs back-to-back gave medians 0.931-1.004; legs interleaved at fold granularity give 0.987/0.991/0.989, and 0.989 under 8-way CPU contention while absolute throughput fell 4x. The INDETERMINATE guard demanded unanimity and failed a good measurement over one sample 0.001 under the floor; it now requires a two-thirds majority. The control that matters: AM-6's constant-cost mutation halves throughput and leaves this ratio at 0.999x green, so AM-7 is not a second AM-6. AM-8 kept N=10 because the measurement said so. Perturbing the RNG only from its fourth construction on: --runs 2 PASSES, --runs 10 fails. A late-onset divergence is deterministic, not flaky, so it is a control rather than a coin flip. Ten runs live on one scenario (make am8, ~2s) rather than all 25 (47s a build). GameKernel 5b records it. The full run also found AM-4a's own mutation stale since ADR-0008 D3 moved the target 250,000 -> 161,000 in CB-WP-0013 -- reported HARNESS-BROKEN, no score published. The build-free half of that check is now a --self-test assertion, so make all catches the next one. mutation-check clauses may now carry their own verify and mutation, and then the enforced flag is measured rather than declared; a declaration disagreeing with its measurement is refused. make all exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
7e9ab221a7
commit
ee37b82675
9 changed files with 802 additions and 51 deletions
|
|
@ -2440,4 +2440,216 @@ mod replay_probe {
|
|||
assert!(elapsed.as_secs_f64() < 5.0, "AM-7: 100k replay under 5s");
|
||||
}
|
||||
}
|
||||
|
||||
/// AM-7 scaling floor from GameKernel §5: fold throughput at 100k
|
||||
/// events must be at least this fraction of throughput at 5k.
|
||||
///
|
||||
/// **Pinned, not tuned** — same rule as `AM6_EVENTS_PER_SEC`. The
|
||||
/// baseline it was written against is boardgame.io at 0.45–0.66×,
|
||||
/// degrading to DNF at 100k. Lowering it requires an ADR.
|
||||
const AM7_SCALING_FLOOR: f64 = 0.9;
|
||||
|
||||
/// The two sizes the spec names.
|
||||
const AM7_SMALL: usize = 5_000;
|
||||
const AM7_LARGE: usize = 100_000;
|
||||
|
||||
/// Events applied **per leg** per sample.
|
||||
///
|
||||
/// Sized against the machine's drift, not against timer resolution.
|
||||
/// At 2,000,000 a sample took ~50 ms, and this machine's throughput
|
||||
/// wanders by 2.5× over a few seconds (CB-EV-0013 §1) — so a 50 ms
|
||||
/// sample measures whatever the clock happened to be doing. At 10 M
|
||||
/// each leg runs ~0.35 s and averages over the drift instead of
|
||||
/// sampling a point on it. Measured: medians 0.987 / 0.991 / 0.989
|
||||
/// across three runs, and 0.989 under 8-way CPU contention. Doubling
|
||||
/// this to 20 M cost 9 s more per run and did not tighten them.
|
||||
const AM7_EVENTS_PER_LEG: usize = 10_000_000;
|
||||
|
||||
/// Paired samples per run. Nine rather than AM-6's three because the
|
||||
/// verdict is a **median**, not a best-of: a median needs enough
|
||||
/// samples that one excursion cannot move it.
|
||||
const AM7_SAMPLES: usize = 9;
|
||||
|
||||
/// Fraction of samples that must agree with the median's verdict for
|
||||
/// the run to be a measurement rather than noise.
|
||||
///
|
||||
/// **This replaced a unanimity guard that was measurably wrong.** The
|
||||
/// first version declared INDETERMINATE whenever any sample fell on
|
||||
/// the other side of the floor. Under deliberate 8-way CPU contention
|
||||
/// the ratio held at a median of 0.971 — the pairing works, and
|
||||
/// absolute throughput had dropped 4× — but one sample read 0.899,
|
||||
/// a thousandth under the floor, and the guard turned a good
|
||||
/// measurement into a failed build. It also fired intermittently
|
||||
/// inside `mutation-check`, where this test runs straight after a
|
||||
/// 50-second rebuild.
|
||||
///
|
||||
/// A gate that fails when the machine is busy is a flake, and a flake
|
||||
/// gets suppressed rather than fixed. Requiring a two-thirds majority
|
||||
/// keeps the guard's purpose — refusing to read a coin-flip as a
|
||||
/// verdict — without treating a single outlier as one.
|
||||
const AM7_AGREEMENT: f64 = 2.0 / 3.0;
|
||||
|
||||
/// Fold `log` once from a fresh state, returning only the time inside
|
||||
/// the fold loop.
|
||||
///
|
||||
/// **Setup is outside the clock, and that is the first trap here.**
|
||||
/// The small leg runs 20× more folds than the large one, so it pays
|
||||
/// 20× more `fresh()` calls. Timing those would penalise the
|
||||
/// denominator, inflate the ratio, and make the row pass for a reason
|
||||
/// that has nothing to do with scaling.
|
||||
fn fold_once(log: &[GroundEvent]) -> std::time::Duration {
|
||||
let mut state = fresh(42);
|
||||
let t = Instant::now();
|
||||
for event in log {
|
||||
state.fold(event);
|
||||
}
|
||||
let dt = t.elapsed();
|
||||
// Keep the fold from being optimised out without paying for a hash
|
||||
// inside the timed region.
|
||||
std::hint::black_box(&state);
|
||||
dt
|
||||
}
|
||||
|
||||
/// One paired sample: the two legs **interleaved**, ratio taken inside.
|
||||
///
|
||||
/// **Two estimators were wrong before this one, and both looked
|
||||
/// reasonable.**
|
||||
///
|
||||
/// The first took best-of-5 on each leg independently and divided —
|
||||
/// the estimator AM-6 uses, correct there because AM-6 is a floor on a
|
||||
/// single number and the question is "is this machine capable". For a
|
||||
/// *ratio* it is wrong: the legs are measured at different moments and
|
||||
/// the noise multiplies instead of cancelling. Five runs of an
|
||||
/// unchanged binary gave **0.581 to 1.085**.
|
||||
///
|
||||
/// The second ran the legs back to back inside one sample, expecting
|
||||
/// the load to be common-mode. It was not enough: this machine's
|
||||
/// absolute throughput wanders between **22 M and 53 M ev/s within a
|
||||
/// single run**, and a 50 ms leg samples a point on that wander rather
|
||||
/// than averaging it. Three runs gave medians 1.004 / 0.931 / 0.956 —
|
||||
/// clustered near the true value but still straddling the floor.
|
||||
///
|
||||
/// So: interleave at *fold* granularity, alternating one large fold
|
||||
/// against twenty small ones so both legs apply the same number of
|
||||
/// events, and run long enough that each leg spans the drift instead
|
||||
/// of sitting inside one excursion of it.
|
||||
fn paired_ratio(small: &[GroundEvent], large: &[GroundEvent]) -> (f64, f64, f64) {
|
||||
let per_round = large.len();
|
||||
let rounds = AM7_EVENTS_PER_LEG.div_ceil(per_round);
|
||||
let small_folds = per_round.div_ceil(small.len());
|
||||
|
||||
let (mut t_small, mut t_large) = (std::time::Duration::ZERO, std::time::Duration::ZERO);
|
||||
let (mut n_small, mut n_large) = (0usize, 0usize);
|
||||
for _ in 0..rounds {
|
||||
for _ in 0..small_folds {
|
||||
t_small += fold_once(small);
|
||||
n_small += small.len();
|
||||
}
|
||||
t_large += fold_once(large);
|
||||
n_large += large.len();
|
||||
}
|
||||
assert!(
|
||||
t_small.as_secs_f64() > 0.0 && t_large.as_secs_f64() > 0.0,
|
||||
"AM-7 measured zero elapsed time"
|
||||
);
|
||||
let tp_small = n_small as f64 / t_small.as_secs_f64();
|
||||
let tp_large = n_large as f64 / t_large.as_secs_f64();
|
||||
(tp_small, tp_large, tp_large / tp_small)
|
||||
}
|
||||
|
||||
/// Build one growing log of at least `target` events, the same way
|
||||
/// `replay_100k_events_is_linear_and_fast` does.
|
||||
fn growing_log(target: usize) -> Vec<GroundEvent> {
|
||||
let mut log = Vec::with_capacity(target);
|
||||
let mut seed = 42u64;
|
||||
let mut source = fresh(seed);
|
||||
let mut stalls = 0;
|
||||
while log.len() < target {
|
||||
if source.outcome.is_some() {
|
||||
seed += 1;
|
||||
source = fresh(seed);
|
||||
}
|
||||
if record_round(&mut source, &mut log) == 0 {
|
||||
stalls += 1;
|
||||
assert!(stalls < 10, "round produced no events; builder stalled");
|
||||
}
|
||||
}
|
||||
log
|
||||
}
|
||||
|
||||
/// AM-7's `scaling >= 0.9x` clause, which was inert for nine passes.
|
||||
///
|
||||
/// `mutation-check.py` said it plainly every run: *"no code computes
|
||||
/// the ratio of throughput @100k to @5k or compares it to 0.9;
|
||||
/// Criterion reports both and nothing relates them."* The sibling test
|
||||
/// above is even named `replay_100k_events_is_linear_and_fast` and
|
||||
/// checks the two sizes **independently** — it computes both numbers,
|
||||
/// prints both, and never divides one by the other.
|
||||
///
|
||||
/// **`#[ignore]` for the same reason AM-6 is** (CB-WP-0006 T04): a
|
||||
/// throughput assertion inside a parallel `cargo test` harness
|
||||
/// measures contention. A *ratio* of two such timings is worse, not
|
||||
/// better — the noise multiplies rather than cancels. Run `make am7`.
|
||||
#[test]
|
||||
#[ignore = "throughput ratio — invalid under a parallel harness; run `make am7`"]
|
||||
fn am7_scaling_holds_from_5k_to_100k_events() {
|
||||
// Positive control on the shape of the measurement. A harness that
|
||||
// measured the same size twice would report ~1.0 and look
|
||||
// excellent; one that swapped the legs would report the reciprocal
|
||||
// and look excellent for the opposite reason.
|
||||
let (small, large) = (growing_log(AM7_SMALL), growing_log(AM7_LARGE));
|
||||
assert!(
|
||||
large.len() >= 15 * small.len(),
|
||||
"AM-7 legs are not far enough apart: {} vs {}",
|
||||
small.len(),
|
||||
large.len()
|
||||
);
|
||||
|
||||
let mut ratios = Vec::with_capacity(AM7_SAMPLES);
|
||||
for _ in 0..AM7_SAMPLES {
|
||||
let (tp_small, tp_large, ratio) = paired_ratio(&small, &large);
|
||||
println!(
|
||||
" AM-7 sample: {tp_small:.0} ev/s @{AM7_SMALL} → \
|
||||
{tp_large:.0} ev/s @{AM7_LARGE} = {ratio:.3}x"
|
||||
);
|
||||
ratios.push(ratio);
|
||||
}
|
||||
ratios.sort_by(|a, b| a.partial_cmp(b).expect("no NaN ratios"));
|
||||
let (worst, median, best) = (
|
||||
ratios[0],
|
||||
ratios[ratios.len() / 2],
|
||||
ratios[ratios.len() - 1],
|
||||
);
|
||||
println!(
|
||||
"AM-7 scaling: {worst:.3}x / {median:.3}x / {best:.3}x \
|
||||
(worst/median/best of {AM7_SAMPLES}, floor {AM7_SCALING_FLOOR})"
|
||||
);
|
||||
|
||||
// The spread is reported, not hidden behind a best-of. The verdict
|
||||
// is the median, and it counts as a measurement only if a
|
||||
// two-thirds majority of samples agree with it — see
|
||||
// AM7_AGREEMENT for the unanimity guard this replaced and why.
|
||||
let agreeing = ratios
|
||||
.iter()
|
||||
.filter(|r| (**r >= AM7_SCALING_FLOOR) == (median >= AM7_SCALING_FLOOR))
|
||||
.count();
|
||||
let agreement = agreeing as f64 / ratios.len() as f64;
|
||||
assert!(
|
||||
agreement >= AM7_AGREEMENT,
|
||||
"AM-7 INDETERMINATE: only {agreeing} of {} samples agree with \
|
||||
the median ({worst:.3}x..{best:.3}x around \
|
||||
{AM7_SCALING_FLOOR}). The machine is too noisy for this to be \
|
||||
a measurement; do not read the best sample as a pass.",
|
||||
ratios.len()
|
||||
);
|
||||
assert!(
|
||||
median >= AM7_SCALING_FLOOR,
|
||||
"AM-7 UNMET: throughput at {AM7_LARGE} events is {median:.3}x \
|
||||
throughput at {AM7_SMALL} events (worst {worst:.3}x, best \
|
||||
{best:.3}x), below the {AM7_SCALING_FLOOR} floor. Baseline: \
|
||||
boardgame.io 0.45-0.66x, DNF at 100k. Do NOT lower the floor \
|
||||
to pass — GameKernel §5 AM-7 is a spec value and lowering it \
|
||||
needs an ADR."
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue