coulomb/clay-borg
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

CB-WP-0015: the two inert clauses, AM-7 scaling and AM-8 N=10
Some checks failed
ci / check (push) Failing after 3s
Details

Browse source 
Provenance (tier S, one paragraph in lieu of survey and ADR): the two
clauses mutation-check has reported inert since CB-WP-0005. AM-7's
scaling ratio was held up by a test literally named
replay_100k_events_is_linear_and_fast that computed both throughputs,
printed both, and never divided one by the other. AM-8's N=10 was held
up by a runner that does two.

Both are now red. AM-7 3/3, AM-8 2/2, M-D1-MUT 10/14, and ADR-0005's
>=10-of-14 prediction MET for the first time. Neither was closed by
amending the question away, which was the live risk: the denominator is
unchanged and the four unenforced rows are the four already
unenforceable.

AM-7 needed three estimators. Best-of-N per leg then divide (AM-6's,
correct for a floor on one number) gave 0.581-1.085 on an unchanged
binary; legs back-to-back gave medians 0.931-1.004; legs interleaved at
fold granularity give 0.987/0.991/0.989, and 0.989 under 8-way CPU
contention while absolute throughput fell 4x. The INDETERMINATE guard
demanded unanimity and failed a good measurement over one sample
0.001 under the floor; it now requires a two-thirds majority. The
control that matters: AM-6's constant-cost mutation halves throughput
and leaves this ratio at 0.999x green, so AM-7 is not a second AM-6.

AM-8 kept N=10 because the measurement said so. Perturbing the RNG only
from its fourth construction on: --runs 2 PASSES, --runs 10 fails. A
late-onset divergence is deterministic, not flaky, so it is a control
rather than a coin flip. Ten runs live on one scenario (make am8, ~2s)
rather than all 25 (47s a build). GameKernel 5b records it.

The full run also found AM-4a's own mutation stale since ADR-0008 D3
moved the target 250,000 -> 161,000 in CB-WP-0013 -- reported
HARNESS-BROKEN, no score published. The build-free half of that check
is now a --self-test assertion, so make all catches the next one.

mutation-check clauses may now carry their own verify and mutation, and
then the enforced flag is measured rather than declared; a declaration
disagreeing with its measurement is refused.

make all exits 0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-08-02 14:07:08 +02:00
parent 7e9ab221a7
commit ee37b82675
9 changed files with 802 additions and 51 deletions
Download patch file Download diff file Expand all files Collapse all files

185
tools/mutation-check.py
View file

@ -71,6 +71,16 @@ class Row:
self.expect = expect
# A row with several stated clauses is red only if EVERY clause has
# a mutation that goes red. AM-7 is the reason this exists.
#
# A clause is (name, enforced, why) or (name, enforced, why,
# (verify, mutate, expect)). The clause carries its own `verify`
# because a clause the row's command cannot reach is exactly the
# case this is for. With a mutation the `enforced` flag is
# MEASURED and cross-checked against the declaration; without one
# it is only the author's word, which is what it always was.
# CB-WP-0015 T01 added the fourth field because a hand-maintained
# boolean describing whether an assertion exists is the same shape
# of claim this whole tool was built to stop trusting.
self.clauses = clauses or []
@ -115,10 +125,15 @@ def rows():
"against M-D1-MUT (ADR-0005 §1) — a row that cannot "
"fail asserts nothing, however good the reason."),
Row("AM-4a", "third-party LOC, shipped runtime <= 250,000",
# The literal here must track ADR-0008 D3's corrected target. It did
# not: this row reported BROKEN on the first full run after
# CB-WP-0013 moved it 250,000 -> 161,000, because no full
# mutation-check had been run in between. Positive control 1 doing
# exactly its job — a stale find-string reported, not skipped.
Row("AM-4a", "third-party LOC, shipped runtime <= 161,000",
verify=py + ["tools/dep-weight.py"],
mutate=("tools/dep-weight.py",
'"shipped-runtime": 250_000,', '"shipped-runtime": 1_000,')),
'"shipped-runtime": 161_000,', '"shipped-runtime": 1_000,')),
Row("AM-4b", "third-party LOC, dev toolchain <= 350,000",
verify=py + ["tools/dep-weight.py"],
@ -188,10 +203,41 @@ def rows():
"RE-EARNED (CB-WP-0006 T06): the probe now replays each "
"per-game segment from its own genesis and asserts its "
"recorded hash. Folding a segment from the wrong seed fails."),
("scaling >= 0.9x", False,
"no code computes the ratio of throughput @100k to @5k or "
"compares it to 0.9; Criterion reports both and nothing "
"relates them."),
# The first clause on any row whose `enforced` flag is
# MEASURED rather than declared. It needs its own mutation
# because the row's — tightening the 5 s budget — proves
# the *timing* clause and cannot reach this one.
#
# The mutation makes fold cost grow with the number of
# events already folded, which is precisely boardgame.io's
# measured defect: throughput halving as history doubled.
# Nothing in `GroundState` grows with log length, so this
# is the only way to make the property false — see
# CB-EV-0013 §2.
#
# CONTROL, measured: AM-6's mutation adds a CONSTANT
# per-event cost. It halves throughput (28M -> 15M ev/s)
# and leaves this ratio at 0.999x, green. So AM-7 is not a
# second AM-6 — a constant slowdown is AM-6's to catch and
# a history-proportional one is AM-7's.
("scaling >= 0.9x", True,
"LIVE (CB-WP-0015 T01): `make am7` interleaves a 5k fold "
"against a 100k fold, takes the ratio inside each sample, "
"and gates the median at 0.9. Measured 0.956-1.068 over "
"three runs; the mutation drives it to 0.751.",
(CARGO + ["test", "--release", "-p", "games-ground",
"--all-features", "am7_scaling", "--",
"--ignored", "--test-threads=1"],
("games/ground/src/lib.rs",
" fn fold(&mut self, event: &Self::Event) {\n"
" match event {",
" fn fold(&mut self, event: &Self::Event) {\n"
" if let Some(c) = self.solution_deck.last().copied() "
"{ self.solution_deck.push(c); }\n"
" for c in self.solution_deck.iter().step_by(4096) "
"{ std::hint::black_box(c); }\n"
" match event {"),
"AM-7 UNMET")),
]),
Row("AM-8", "determinism: same-seed replays bit-identical",
@ -204,10 +250,31 @@ def rows():
"let n = N.fetch_add(1, std::sync::atomic::Ordering::Relaxed); "
"Self(rand_chacha::ChaCha12Rng::seed_from_u64(seed.0 + n)) }"),
clauses=[
("N=10 same-seed replays", False,
"the runner executes each scenario TWICE (K8), not ten "
"times; the spec says N=10. The double-run is live and the "
"count is not."),
# MEASURED, and the measurement is the argument for keeping
# N=10 rather than amending the spec down to K8's two.
#
# The row's own mutation perturbs the seed on EVERY RNG
# construction, so it diverges on run 2 and N=2 catches it.
# This one perturbs only from the fourth construction on:
# a late-onset divergence, deterministic rather than flaky.
# Measured on gr-r06 — `--runs 2` PASSES, `--runs 10` fails
# with "run 1 hash ... != run 4 hash ... (of 10)". That is
# a class the double-run structurally cannot see.
("N=10 same-seed replays", True,
"LIVE (CB-WP-0015 T02): `make am8` runs one scenario ten "
"times against the first hash. Not all 25 — see "
"scenario::run_n for why eight more runs of a deterministic "
"check is not worth 47 s a build.",
(CARGO + ["run", "-q", "-p", "cb-sim", "--", "--runs", "10",
"scenarios/ground/gr-r06-round-resolve.yaml"],
("crates/cb-kernel/src/rng.rs",
"Self(rand_chacha::ChaCha12Rng::seed_from_u64(seed.0))",
"{ static N: std::sync::atomic::AtomicU64 = "
"std::sync::atomic::AtomicU64::new(0); "
"let n = N.fetch_add(1, std::sync::atomic::Ordering::Relaxed); "
"Self(rand_chacha::ChaCha12Rng::seed_from_u64("
"seed.0 + u64::from(n >= 3))) }"),
"K8 divergence: run 1 hash")),
("HashMap deny lint clean", True,
"clippy.toml denies HashMap/HashSet and `make check` runs "
"with -D warnings"),
@ -269,20 +336,47 @@ def check_row(row):
if row.unmutatable:
return "unmutatable", row.unmutatable
path = os.path.join(ROOT, row.mutate[0])
verdict, detail = _run_mutation(row.verify, row.mutate, row.expect)
# Clause-level mutations, where a clause carries one. Each is measured
# the same way as the row's own mutation, and the measurement is
# cross-checked against the declared `enforced` flag — a declaration
# that disagrees with its own mutation is a DFD-class defect and must
# not be reported as either verdict.
for clause in row.clauses:
if len(clause) < 4 or clause[3] is None:
continue
name, declared, _why, (verify, mutate, expect) = clause
c_verdict, c_detail = _run_mutation(verify, mutate, expect)
if c_verdict in ("HARNESS-BROKEN", "EXPECT-VACUOUS"):
return c_verdict, f"clause {name!r}: {c_detail}"
measured = c_verdict == "red"
if measured != declared:
return "HARNESS-BROKEN", (
f"clause {name!r} is declared enforced={declared} but its "
f"mutation measured {c_verdict} — the declaration and the "
f"measurement disagree")
if not measured and verdict == "red":
verdict, detail = "PARTIAL", f"clause {name!r}: {c_detail}"
return verdict, detail
def _run_mutation(verify, mutate, expect):
"""(verdict, detail) for one mutation. Restores the tree regardless."""
path = os.path.join(ROOT, mutate[0])
original = open(path).read()
old, new = row.mutate[1], row.mutate[2]
old, new = mutate[1], mutate[2]
# Positive control 1: the mutation must be applicable at all. A
# find-string that no longer matches would otherwise mutate nothing
# and report the baseline result as the mutant result.
if original.count(old) < 1:
return "HARNESS-BROKEN", (
f"mutation target not found in {row.mutate[0]}: {old!r}")
f"mutation target not found in {mutate[0]}: {old!r}")
# Positive control 2: the baseline must be green, or "mutant red"
# proves nothing.
base_ok, base_tail, base_out = run(row.verify)
base_ok, base_tail, base_out = run(verify)
if not base_ok:
return "inconclusive", f"baseline already red: {base_tail}"
@ -291,9 +385,9 @@ def check_row(row):
# normal report and accept any failure at all — which is how the guard
# goes vacuous without anyone noticing. My first attempt on AM-2 did
# exactly that.
if row.expect and row.expect in base_out:
if expect and expect in base_out:
return "EXPECT-VACUOUS", (
f"expect string {row.expect!r} appears in PASSING output, so it "
f"expect string {expect!r} appears in PASSING output, so it "
f"would accept any failure — the FA guard is inert for this row")
try:
@ -305,24 +399,24 @@ def check_row(row):
if open(path).read() == original:
return "HARNESS-BROKEN", "write did not take effect"
mut_ok, mut_tail, mut_out = run(row.verify)
mut_ok, mut_tail, mut_out = run(verify)
finally:
open(path, "w").write(original)
# Positive control 4: restoration must have worked, or every later
# row runs against a corrupted tree.
if open(path).read() != original:
return "HARNESS-BROKEN", f"failed to restore {row.mutate[0]}"
return "HARNESS-BROKEN", f"failed to restore {mutate[0]}"
if mut_ok:
return "SURVIVED", "mutant is green — this row asserts nothing"
if row.expect and row.expect not in mut_out:
if expect and expect not in mut_out:
# The FA guard. The mutant went red, but not for the reason
# claimed — a compile error, a panic elsewhere, an unrelated
# assertion. Scoring that as `red` would credit the row with an
# assertion it does not have.
return "WRONG-REASON", (
f"mutant failed, but its output does not contain {row.expect!r} — "
f"mutant failed, but its output does not contain {expect!r} — "
f"this is not evidence the row is enforced")
return "red", mut_tail or "verifier failed as required"
@ -359,8 +453,13 @@ def report(only=None):
if detail:
for line in _wrap(detail, 66):
print(f" {line}")
for name, enforced, why in r.clauses:
print(f" - {'red ' if enforced else 'NONE'} "
for clause in r.clauses:
name, enforced, why = clause[0], clause[1], clause[2]
# `red*` marks a clause whose flag was measured by its own
# mutation this run, not asserted by the author.
measured = "*" if len(clause) > 3 and clause[3] else " "
print(f" - "
f"{('red' + measured) if enforced else 'NONE'} "
f"{name}: {why[:60]}")
if broken:
@ -424,6 +523,29 @@ def self_test():
check("every mutation target file exists",
all(os.path.isfile(os.path.join(ROOT, r.mutate[0]))
for r in rs if r.mutate))
# CB-WP-0015 T03. `check_row` already reports a stale find-string as
# HARNESS-BROKEN — but only on a full mutation-check, which is
# deliberately not in `make all` because it rebuilds per row. So a
# mutation could rot for passes at a time: AM-4a's did, from the moment
# ADR-0008 D3 moved the shipped-runtime target 250,000 -> 161,000 until
# the next full run. This asks the same question with no builds at all,
# which puts it in `make all` via `self-tests`.
stale_targets = []
for r in rs:
targets = [r.mutate] if r.mutate else []
# Clause mutations rot the same way and are checked the same way.
targets += [c[3][1] for c in r.clauses if len(c) > 3 and c[3]]
for relpath, find, _new in targets:
try:
if find not in open(os.path.join(ROOT, relpath)).read():
stale_targets.append(r.id)
except OSError:
stale_targets.append(r.id)
check("every mutation find-string still matches its source",
not stale_targets,
f"stale: {', '.join(sorted(set(stale_targets)))}" if stale_targets
else "checked without building — the cheap half of check_row")
# The control that matters: a mutation whose find-string no longer
# matches must be reported BROKEN, not silently skipped.
stale = Row("AM-X", "fixture", verify=[sys.executable, "-c", "pass"],
@ -455,6 +577,25 @@ def self_test():
check("a mutant failing for the wrong reason is not scored red",
v4 == "WRONG-REASON", v4)
# CB-WP-0015 T01: clause-level mutations. A clause carrying a mutation
# gets its `enforced` flag MEASURED, so the control that matters is
# that a declaration disagreeing with its own measurement is refused
# rather than reported as either verdict — otherwise the fourth field
# would just be decoration on the same hand-maintained boolean.
ok_verify = [sys.executable, "-c",
"import sys; sys.exit(0 if 'ZZC' not in "
"open('Makefile').read() else 5)"]
noop = ("Makefile", "PY := python3", "PY := python3 ")
lying = Row("AM-C", "fixture", verify=ok_verify,
mutate=("Makefile", "PY := python3", "PY := python3 # ZZC"),
clauses=[("a clause that claims more than it can show", True,
"declared enforced, but its mutation changes "
"nothing the verifier looks at",
(ok_verify, noop, None))])
v5, d5 = check_row(lying)
check("a clause whose declaration contradicts its mutation is refused",
v5 == "HARNESS-BROKEN" and "disagree" in d5, f"{v5}: {d5[:40]}")
# A verifier that is already red must not be scored.
dead = Row("AM-Z", "fixture", verify=[sys.executable, "-c", "raise SystemExit(3)"],
mutate=("Makefile", "PY := python3", "PY := python3 "))