tegwick
ee37b82675
Some checks failed
ci / check (push) Failing after 3s
Provenance (tier S, one paragraph in lieu of survey and ADR): the two clauses mutation-check has reported inert since CB-WP-0005. AM-7's scaling ratio was held up by a test literally named replay_100k_events_is_linear_and_fast that computed both throughputs, printed both, and never divided one by the other. AM-8's N=10 was held up by a runner that does two. Both are now red. AM-7 3/3, AM-8 2/2, M-D1-MUT 10/14, and ADR-0005's >=10-of-14 prediction MET for the first time. Neither was closed by amending the question away, which was the live risk: the denominator is unchanged and the four unenforced rows are the four already unenforceable. AM-7 needed three estimators. Best-of-N per leg then divide (AM-6's, correct for a floor on one number) gave 0.581-1.085 on an unchanged binary; legs back-to-back gave medians 0.931-1.004; legs interleaved at fold granularity give 0.987/0.991/0.989, and 0.989 under 8-way CPU contention while absolute throughput fell 4x. The INDETERMINATE guard demanded unanimity and failed a good measurement over one sample 0.001 under the floor; it now requires a two-thirds majority. The control that matters: AM-6's constant-cost mutation halves throughput and leaves this ratio at 0.999x green, so AM-7 is not a second AM-6. AM-8 kept N=10 because the measurement said so. Perturbing the RNG only from its fourth construction on: --runs 2 PASSES, --runs 10 fails. A late-onset divergence is deterministic, not flaky, so it is a control rather than a coin flip. Ten runs live on one scenario (make am8, ~2s) rather than all 25 (47s a build). GameKernel 5b records it. The full run also found AM-4a's own mutation stale since ADR-0008 D3 moved the target 250,000 -> 161,000 in CB-WP-0013 -- reported HARNESS-BROKEN, no score published. The build-free half of that check is now a --self-test assertion, so make all catches the next one. mutation-check clauses may now carry their own verify and mutation, and then the enforced flag is measured rather than declared; a declaration disagreeing with its measurement is refused. make all exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
626 lines
31 KiB
Python
626 lines
31 KiB
Python
#!/usr/bin/env python3
|
|
"""M-D1-MUT: does anything fail when an acceptance row's property is false?
|
|
|
|
CB-WP-0005 T02, per ADR-0005 §1. Every other instrument in this project
|
|
counts **names** — M-D1-COV counts `covers:` tags, M-D1-LNK counts rule-ID
|
|
strings. Both answer "is this rule mentioned?" Neither answers "does
|
|
anything fail if it is violated?"
|
|
|
|
Four of the seven defects found by CB-RES-0004 were named in the source
|
|
and inert. Two were mutation-proven: AM-7's `hash-identical` clause
|
|
survives folding a 100k-event log from an unrelated genesis state, and
|
|
K9's `through` field survives `Snapshot::take` discarding it.
|
|
|
|
So: for each acceptance row in `specs/GameKernel.md` §5, invert the
|
|
property and require the verifying command to go **red**.
|
|
|
|
adapted:mutation-testing — the denominator is acceptance rows, not
|
|
source lines. Exhaustive tools (cargo-mutants, Stryker, PIT) score a
|
|
kill rate over lines; that is slow and measures something we do not
|
|
claim. Our claims are the AM-* rows, so those are the population.
|
|
|
|
## Verdicts
|
|
|
|
red baseline green, mutant red — the row is enforced
|
|
SURVIVED baseline green, mutant ALSO green — the row asserts
|
|
nothing; this is the defect being hunted
|
|
unmutatable no property to invert, with a stated reason — **counts
|
|
against** the metric (ADR-0005 §1). A row nobody can
|
|
invert asserts nothing.
|
|
inconclusive baseline already red, so "mutant red" would prove nothing
|
|
|
|
## Positive controls
|
|
|
|
A mutation harness that silently fails to apply its mutation reports every
|
|
row as green-baseline/green-mutant and looks thorough. So each run
|
|
asserts, per row: the file content actually changed; the baseline command
|
|
passed **before** mutating; and the file was restored afterwards. This
|
|
project has seven harness-does-nothing instances and one was found in
|
|
`rule-coverage.py` an hour ago.
|
|
|
|
Usage:
|
|
python3 tools/mutation-check.py # run every row
|
|
python3 tools/mutation-check.py --row AM-1
|
|
python3 tools/mutation-check.py --self-test
|
|
"""
|
|
import argparse
|
|
import os
|
|
import subprocess
|
|
import sys
|
|
|
|
from repo import ROOT, cargo_env, enter_root
|
|
|
|
CARGO = ["cargo"]
|
|
|
|
|
|
class Row:
|
|
"""One acceptance row and the mutation that must break it."""
|
|
|
|
def __init__(self, id, claim, verify=None, mutate=None, unmutatable=None,
|
|
clauses=None, expect=None):
|
|
self.id = id
|
|
self.claim = claim
|
|
self.verify = verify # command that must be green, then red
|
|
self.mutate = mutate # (relpath, old, new)
|
|
self.unmutatable = unmutatable
|
|
# CB-WP-0005 T08 / the FA class: a mutation is only evidence if it
|
|
# fails FOR ITS STATED REASON. A no-op mutation produces no failure
|
|
# at all, and would otherwise be scored as a genuine SURVIVED —
|
|
# publishing the claim that working code is broken. When `expect`
|
|
# is set, the mutant's output must contain it.
|
|
self.expect = expect
|
|
# A row with several stated clauses is red only if EVERY clause has
|
|
# a mutation that goes red. AM-7 is the reason this exists.
|
|
#
|
|
# A clause is (name, enforced, why) or (name, enforced, why,
|
|
# (verify, mutate, expect)). The clause carries its own `verify`
|
|
# because a clause the row's command cannot reach is exactly the
|
|
# case this is for. With a mutation the `enforced` flag is
|
|
# MEASURED and cross-checked against the declaration; without one
|
|
# it is only the author's word, which is what it always was.
|
|
# CB-WP-0015 T01 added the fourth field because a hand-maintained
|
|
# boolean describing whether an assertion exists is the same shape
|
|
# of claim this whole tool was built to stop trusting.
|
|
self.clauses = clauses or []
|
|
|
|
|
|
def rows():
|
|
"""The acceptance table, with a mutation or a stated reason for none.
|
|
|
|
Ordered as in specs/GameKernel.md §5. AM-4 splits into a/b/c, so there
|
|
are **14** rows, not the twelve ADR-0005 asserted — see the
|
|
correction note in the report.
|
|
"""
|
|
py = [sys.executable]
|
|
return [
|
|
Row("AM-1", "100% of GR-rules covered by >=1 passing scenario",
|
|
verify=py + ["tools/rule-coverage.py"],
|
|
mutate=("scenarios/ground/gr-r06-round-resolve.yaml",
|
|
"GR-R06, ", "")),
|
|
|
|
# A PROPERTY mutation: inflate the implementation past the ratio.
|
|
# 58 rules x 40 = 2,320; the impl is 1,575, so ~800 lines of filler
|
|
# crosses it. Raising the threshold instead would only prove the
|
|
# comparison runs.
|
|
Row("AM-2", "<= 40 spec lines per rule in games/ground",
|
|
verify=py + ["tools/size-metrics.py"],
|
|
mutate=("games/ground/src/lib.rs",
|
|
"impl Aggregate for GroundState {",
|
|
"fn _am2_filler() {\n" + " let _x = 0;\n" * 800
|
|
+ "}\n\nimpl Aggregate for GroundState {"),
|
|
expect="FAIL target <= 40"),
|
|
|
|
Row("AM-3", "synthetic workload definition <= 50 LOC",
|
|
unmutatable="BLOCKED, not uninstrumented — the distinction "
|
|
"matters. `make size-metrics` ships the measurement "
|
|
"(a marker-delimited region) and reports the row "
|
|
"blocked, because the artifact it measures has never "
|
|
"been built: games/ contains only ground, and "
|
|
"benches/synthetic.rs drives GROUND rather than "
|
|
"defining a synthetic game. The baseline is a "
|
|
"declarative 3p commit/reveal game object (~36 LOC, "
|
|
"boardgame.io); measuring GROUND's 1,575 impl lines "
|
|
"against it would compare two different games and "
|
|
"call the difference a D1 result. Still counts "
|
|
"against M-D1-MUT (ADR-0005 §1) — a row that cannot "
|
|
"fail asserts nothing, however good the reason."),
|
|
|
|
# The literal here must track ADR-0008 D3's corrected target. It did
|
|
# not: this row reported BROKEN on the first full run after
|
|
# CB-WP-0013 moved it 250,000 -> 161,000, because no full
|
|
# mutation-check had been run in between. Positive control 1 doing
|
|
# exactly its job — a stale find-string reported, not skipped.
|
|
Row("AM-4a", "third-party LOC, shipped runtime <= 161,000",
|
|
verify=py + ["tools/dep-weight.py"],
|
|
mutate=("tools/dep-weight.py",
|
|
'"shipped-runtime": 161_000,', '"shipped-runtime": 1_000,')),
|
|
|
|
Row("AM-4b", "third-party LOC, dev toolchain <= 350,000",
|
|
verify=py + ["tools/dep-weight.py"],
|
|
mutate=("tools/dep-weight.py",
|
|
'"dev-toolchain": 350_000,', '"dev-toolchain": 1_000,')),
|
|
|
|
# Deliberately RETAINED in this denominator after its withdrawal
|
|
# from the acceptance table. Dropping it would move M-D1-MUT from
|
|
# 7/14 to 7/13 without enforcing anything — a score improved by
|
|
# deleting the question.
|
|
Row("AM-4c", "own source per third-party 100k lines",
|
|
unmutatable="WITHDRAWN from the acceptance table 2026-08-01 "
|
|
"(CB-WP-0006 T04, GameKernel §5a) and retained here "
|
|
"as a diagnostic. The ratio has no monotone better "
|
|
"direction — rising can mean owning semantics or "
|
|
"reimplementing what should be assimilated; falling "
|
|
"can mean leverage or dependency bloat — so it "
|
|
"cannot carry a threshold. It is also redundant: "
|
|
"AM-4a/AM-4b bound the denominator and AM-2 bounds "
|
|
"own-source density. Kept in this denominator on "
|
|
"purpose, so the metric cannot be improved by "
|
|
"deleting rows."),
|
|
|
|
Row("AM-5", "clean release build <= 60 s",
|
|
unmutatable="now RECORDED (`make build-time`) and **met**: "
|
|
"37.3 s dev toolchain, 41.2 s shipped runtime, best "
|
|
"of 3 on a quiet machine, against a 60 s target on "
|
|
"bnt-lap001 (1.6x headroom). An earlier reading of "
|
|
"87.0 s was taken under contention and was wrong. "
|
|
"Still counts "
|
|
"against M-D1-MUT because GameKernel §5 declares the "
|
|
"row `recorded not gated` — a row that cannot fail "
|
|
"asserts nothing, and promoting it is a spec change "
|
|
"needing an ADR. The breach is a maintainer decision, "
|
|
"not something this tool should resolve by gating "
|
|
"itself."),
|
|
|
|
Row("AM-6", ">= 100,000 applied events/s",
|
|
# Release: the gate runs where headroom is ~24x. In debug the
|
|
# test only asserts a sanity floor, so a debug mutation run
|
|
# would need a far larger slowdown to register.
|
|
verify=CARGO + ["test", "--release", "-p", "games-ground",
|
|
"--all-features", "am6_throughput", "--",
|
|
"--ignored", "--test-threads=1"],
|
|
# A PROPERTY mutation, not a threshold tweak: slow the fold hot
|
|
# path and require the gate to notice. Raising the target
|
|
# instead would only prove the comparison runs.
|
|
mutate=("games/ground/src/lib.rs",
|
|
" fn fold(&mut self, event: &Self::Event) {\n"
|
|
" match event {",
|
|
" fn fold(&mut self, event: &Self::Event) {\n"
|
|
" for _ in 0..100_000 { std::hint::black_box(0u8); }\n"
|
|
" match event {"),
|
|
expect="AM-6 UNMET"),
|
|
|
|
Row("AM-7", "scaling >= 0.9x, and replay of 100k events <= 5 s, "
|
|
"hash-identical",
|
|
verify=CARGO + ["test", "-p", "games-ground", "--all-features",
|
|
"replay_100k"],
|
|
mutate=("games/ground/src/lib.rs",
|
|
'assert!(elapsed.as_secs_f64() < 5.0,',
|
|
'assert!(elapsed.as_secs_f64() < 0.0,'),
|
|
clauses=[
|
|
("timing <= 5 s", True,
|
|
"the elapsed assert is live; tightening it to 0.0 s goes red"),
|
|
("hash-identical", True,
|
|
"RE-EARNED (CB-WP-0006 T06): the probe now replays each "
|
|
"per-game segment from its own genesis and asserts its "
|
|
"recorded hash. Folding a segment from the wrong seed fails."),
|
|
# The first clause on any row whose `enforced` flag is
|
|
# MEASURED rather than declared. It needs its own mutation
|
|
# because the row's — tightening the 5 s budget — proves
|
|
# the *timing* clause and cannot reach this one.
|
|
#
|
|
# The mutation makes fold cost grow with the number of
|
|
# events already folded, which is precisely boardgame.io's
|
|
# measured defect: throughput halving as history doubled.
|
|
# Nothing in `GroundState` grows with log length, so this
|
|
# is the only way to make the property false — see
|
|
# CB-EV-0013 §2.
|
|
#
|
|
# CONTROL, measured: AM-6's mutation adds a CONSTANT
|
|
# per-event cost. It halves throughput (28M -> 15M ev/s)
|
|
# and leaves this ratio at 0.999x, green. So AM-7 is not a
|
|
# second AM-6 — a constant slowdown is AM-6's to catch and
|
|
# a history-proportional one is AM-7's.
|
|
("scaling >= 0.9x", True,
|
|
"LIVE (CB-WP-0015 T01): `make am7` interleaves a 5k fold "
|
|
"against a 100k fold, takes the ratio inside each sample, "
|
|
"and gates the median at 0.9. Measured 0.956-1.068 over "
|
|
"three runs; the mutation drives it to 0.751.",
|
|
(CARGO + ["test", "--release", "-p", "games-ground",
|
|
"--all-features", "am7_scaling", "--",
|
|
"--ignored", "--test-threads=1"],
|
|
("games/ground/src/lib.rs",
|
|
" fn fold(&mut self, event: &Self::Event) {\n"
|
|
" match event {",
|
|
" fn fold(&mut self, event: &Self::Event) {\n"
|
|
" if let Some(c) = self.solution_deck.last().copied() "
|
|
"{ self.solution_deck.push(c); }\n"
|
|
" for c in self.solution_deck.iter().step_by(4096) "
|
|
"{ std::hint::black_box(c); }\n"
|
|
" match event {"),
|
|
"AM-7 UNMET")),
|
|
]),
|
|
|
|
Row("AM-8", "determinism: same-seed replays bit-identical",
|
|
verify=CARGO + ["run", "-q", "-p", "cb-sim", "--",
|
|
"scenarios/ground/gr-r06-round-resolve.yaml"],
|
|
mutate=("crates/cb-kernel/src/rng.rs",
|
|
"Self(rand_chacha::ChaCha12Rng::seed_from_u64(seed.0))",
|
|
"{ static N: std::sync::atomic::AtomicU64 = "
|
|
"std::sync::atomic::AtomicU64::new(0); "
|
|
"let n = N.fetch_add(1, std::sync::atomic::Ordering::Relaxed); "
|
|
"Self(rand_chacha::ChaCha12Rng::seed_from_u64(seed.0 + n)) }"),
|
|
clauses=[
|
|
# MEASURED, and the measurement is the argument for keeping
|
|
# N=10 rather than amending the spec down to K8's two.
|
|
#
|
|
# The row's own mutation perturbs the seed on EVERY RNG
|
|
# construction, so it diverges on run 2 and N=2 catches it.
|
|
# This one perturbs only from the fourth construction on:
|
|
# a late-onset divergence, deterministic rather than flaky.
|
|
# Measured on gr-r06 — `--runs 2` PASSES, `--runs 10` fails
|
|
# with "run 1 hash ... != run 4 hash ... (of 10)". That is
|
|
# a class the double-run structurally cannot see.
|
|
("N=10 same-seed replays", True,
|
|
"LIVE (CB-WP-0015 T02): `make am8` runs one scenario ten "
|
|
"times against the first hash. Not all 25 — see "
|
|
"scenario::run_n for why eight more runs of a deterministic "
|
|
"check is not worth 47 s a build.",
|
|
(CARGO + ["run", "-q", "-p", "cb-sim", "--", "--runs", "10",
|
|
"scenarios/ground/gr-r06-round-resolve.yaml"],
|
|
("crates/cb-kernel/src/rng.rs",
|
|
"Self(rand_chacha::ChaCha12Rng::seed_from_u64(seed.0))",
|
|
"{ static N: std::sync::atomic::AtomicU64 = "
|
|
"std::sync::atomic::AtomicU64::new(0); "
|
|
"let n = N.fetch_add(1, std::sync::atomic::Ordering::Relaxed); "
|
|
"Self(rand_chacha::ChaCha12Rng::seed_from_u64("
|
|
"seed.0 + u64::from(n >= 3))) }"),
|
|
"K8 divergence: run 1 hash")),
|
|
("HashMap deny lint clean", True,
|
|
"clippy.toml denies HashMap/HashSet and `make check` runs "
|
|
"with -D warnings"),
|
|
]),
|
|
|
|
# A PROPERTY mutation: make the workload actually use memory.
|
|
Row("AM-9", "peak RSS <= 64 MB",
|
|
verify=py + ["tools/runtime-metrics.py", "--fast"],
|
|
mutate=("games/ground/src/lib.rs",
|
|
" fn replay_100k_events_is_linear_and_fast() {",
|
|
" fn replay_100k_events_is_linear_and_fast() {\n"
|
|
" let hog: Vec<u8> = vec![7u8; 300_000_000];\n"
|
|
" std::hint::black_box(&hog);"),
|
|
expect="FAIL target <= 64 MB"),
|
|
|
|
Row("AM-10", "0 foreign types in cb-*-api-visible signatures",
|
|
unmutatable="the population is empty — there is no `cb-*-api` "
|
|
"crate, so the claim is true over nothing. What is "
|
|
"measured instead is a clippy deny of "
|
|
"HashMap/HashSet whose stated reason cites K6 "
|
|
"(determinism), reported under a D4 leak row. "
|
|
"Withdrawn by ADR-0005 §4."),
|
|
|
|
# A PROPERTY mutation: break ONE impl and require the SHARED suite
|
|
# to fail. That is what M-D4-SWAP claims — that either impl can be
|
|
# substituted for the other — and it is exactly what two separate
|
|
# per-impl tests could never demonstrate.
|
|
Row("AM-11", "null + reference impls passing ONE conformance suite",
|
|
verify=CARGO + ["test", "-p", "cb-kernel", "-p", "cb-events",
|
|
"conformance"],
|
|
mutate=("crates/cb-kernel/src/rng.rs",
|
|
" fn draw(&mut self, _bound: u32) -> u32 {\n 0\n }",
|
|
" fn draw(&mut self, _bound: u32) -> u32 {\n"
|
|
" _bound\n }"),
|
|
expect="outside 0.."),
|
|
|
|
Row("AM-12", "tokens and USD recorded per task",
|
|
verify=py + ["tools/cb-cost.py", "--self-test"],
|
|
mutate=("tools/cb-cost.py",
|
|
'toks["output"] = max(t["output"] for t in per_row)',
|
|
'toks["output"] = per_row[0]["output"]')),
|
|
]
|
|
|
|
|
|
def run(cmd, timeout=900):
|
|
"""(ok, tail) for one verifying command, run at the repo root."""
|
|
try:
|
|
r = subprocess.run(cmd, cwd=ROOT, env=cargo_env(),
|
|
capture_output=True, text=True, timeout=timeout)
|
|
except subprocess.TimeoutExpired:
|
|
return False, "TIMEOUT", "TIMEOUT"
|
|
out = (r.stdout + r.stderr).strip()
|
|
tail = out.splitlines()
|
|
return r.returncode == 0, (tail[-1][:70] if tail else ""), out
|
|
|
|
|
|
def check_row(row):
|
|
"""(verdict, detail). Restores the tree even when the command fails."""
|
|
if row.unmutatable:
|
|
return "unmutatable", row.unmutatable
|
|
|
|
verdict, detail = _run_mutation(row.verify, row.mutate, row.expect)
|
|
|
|
# Clause-level mutations, where a clause carries one. Each is measured
|
|
# the same way as the row's own mutation, and the measurement is
|
|
# cross-checked against the declared `enforced` flag — a declaration
|
|
# that disagrees with its own mutation is a DFD-class defect and must
|
|
# not be reported as either verdict.
|
|
for clause in row.clauses:
|
|
if len(clause) < 4 or clause[3] is None:
|
|
continue
|
|
name, declared, _why, (verify, mutate, expect) = clause
|
|
c_verdict, c_detail = _run_mutation(verify, mutate, expect)
|
|
if c_verdict in ("HARNESS-BROKEN", "EXPECT-VACUOUS"):
|
|
return c_verdict, f"clause {name!r}: {c_detail}"
|
|
measured = c_verdict == "red"
|
|
if measured != declared:
|
|
return "HARNESS-BROKEN", (
|
|
f"clause {name!r} is declared enforced={declared} but its "
|
|
f"mutation measured {c_verdict} — the declaration and the "
|
|
f"measurement disagree")
|
|
if not measured and verdict == "red":
|
|
verdict, detail = "PARTIAL", f"clause {name!r}: {c_detail}"
|
|
return verdict, detail
|
|
|
|
|
|
def _run_mutation(verify, mutate, expect):
|
|
"""(verdict, detail) for one mutation. Restores the tree regardless."""
|
|
path = os.path.join(ROOT, mutate[0])
|
|
original = open(path).read()
|
|
old, new = mutate[1], mutate[2]
|
|
|
|
# Positive control 1: the mutation must be applicable at all. A
|
|
# find-string that no longer matches would otherwise mutate nothing
|
|
# and report the baseline result as the mutant result.
|
|
if original.count(old) < 1:
|
|
return "HARNESS-BROKEN", (
|
|
f"mutation target not found in {mutate[0]}: {old!r}")
|
|
|
|
# Positive control 2: the baseline must be green, or "mutant red"
|
|
# proves nothing.
|
|
base_ok, base_tail, base_out = run(verify)
|
|
if not base_ok:
|
|
return "inconclusive", f"baseline already red: {base_tail}"
|
|
|
|
# CB-WP-0006 T08: the FA guard is only a guard if its `expect` string
|
|
# cannot appear in PASSING output. An expect of "AM-2" would match the
|
|
# normal report and accept any failure at all — which is how the guard
|
|
# goes vacuous without anyone noticing. My first attempt on AM-2 did
|
|
# exactly that.
|
|
if expect and expect in base_out:
|
|
return "EXPECT-VACUOUS", (
|
|
f"expect string {expect!r} appears in PASSING output, so it "
|
|
f"would accept any failure — the FA guard is inert for this row")
|
|
|
|
try:
|
|
mutated = original.replace(old, new, 1)
|
|
# Positive control 3: the file content must actually differ.
|
|
if mutated == original:
|
|
return "HARNESS-BROKEN", "replace produced no change"
|
|
open(path, "w").write(mutated)
|
|
if open(path).read() == original:
|
|
return "HARNESS-BROKEN", "write did not take effect"
|
|
|
|
mut_ok, mut_tail, mut_out = run(verify)
|
|
finally:
|
|
open(path, "w").write(original)
|
|
|
|
# Positive control 4: restoration must have worked, or every later
|
|
# row runs against a corrupted tree.
|
|
if open(path).read() != original:
|
|
return "HARNESS-BROKEN", f"failed to restore {mutate[0]}"
|
|
|
|
if mut_ok:
|
|
return "SURVIVED", "mutant is green — this row asserts nothing"
|
|
if expect and expect not in mut_out:
|
|
# The FA guard. The mutant went red, but not for the reason
|
|
# claimed — a compile error, a panic elsewhere, an unrelated
|
|
# assertion. Scoring that as `red` would credit the row with an
|
|
# assertion it does not have.
|
|
return "WRONG-REASON", (
|
|
f"mutant failed, but its output does not contain {expect!r} — "
|
|
f"this is not evidence the row is enforced")
|
|
return "red", mut_tail or "verifier failed as required"
|
|
|
|
|
|
def report(only=None):
|
|
rs = [r for r in rows() if not only or r.id == only]
|
|
print("M-D1-MUT — does anything fail when the property is false?")
|
|
print(f" {len(rows())} acceptance rows in specs/GameKernel.md §5")
|
|
print(" NOTE: ADR-0005 and CB-WP-0005 both say 'twelve'. There are "
|
|
"**14** — AM-4\n splits into a/b/c. The 9-of-12 prediction "
|
|
"(75%) is evaluated below\n on the same basis: >=10 of 14.\n")
|
|
|
|
tally = {}
|
|
broken = []
|
|
for r in rs:
|
|
verdict, detail = check_row(r)
|
|
# A multi-clause row is red only if every clause is enforced.
|
|
if verdict == "red" and r.clauses:
|
|
unmet = [c for c in r.clauses if not c[1]]
|
|
if unmet:
|
|
verdict = "PARTIAL"
|
|
detail = (f"{len(r.clauses) - len(unmet)}/{len(r.clauses)} "
|
|
f"clauses enforced")
|
|
tally[verdict] = tally.get(verdict, 0) + 1
|
|
if verdict in ("HARNESS-BROKEN", "EXPECT-VACUOUS"):
|
|
broken.append(r.id)
|
|
mark = {"red": "red ", "SURVIVED": "SURVIVED ",
|
|
"unmutatable": "unmutatable", "inconclusive": "inconclusive",
|
|
"PARTIAL": "PARTIAL ",
|
|
"HARNESS-BROKEN": "BROKEN ",
|
|
"EXPECT-VACUOUS": "EXPECT-VOID",
|
|
"WRONG-REASON": "WRONG-REASON"}[verdict]
|
|
print(f" [{mark}] {r.id:<6} {r.claim[:52]}")
|
|
if detail:
|
|
for line in _wrap(detail, 66):
|
|
print(f" {line}")
|
|
for clause in r.clauses:
|
|
name, enforced, why = clause[0], clause[1], clause[2]
|
|
# `red*` marks a clause whose flag was measured by its own
|
|
# mutation this run, not asserted by the author.
|
|
measured = "*" if len(clause) > 3 and clause[3] else " "
|
|
print(f" - "
|
|
f"{('red' + measured) if enforced else 'NONE'} "
|
|
f"{name}: {why[:60]}")
|
|
|
|
if broken:
|
|
# The harness asserting it did the work it reports.
|
|
print(f"\nERROR — harness broken on {', '.join(broken)}; "
|
|
f"no verdict is valid", file=sys.stderr)
|
|
return 1
|
|
|
|
red = tally.get("red", 0)
|
|
total = len(rs)
|
|
withdrawn = sum(1 for r in rs if "WITHDRAWN" in (r.unmutatable or ""))
|
|
print(f"\n M-D1-MUT: {red}/{total} rows enforced")
|
|
if withdrawn:
|
|
print(f" ({withdrawn} withdrawn row(s) retained in the denominator "
|
|
f"on purpose — a score\n improved by deleting the question "
|
|
f"is not an improvement)")
|
|
for k in ("PARTIAL", "SURVIVED", "WRONG-REASON", "EXPECT-VACUOUS",
|
|
"unmutatable", "inconclusive"):
|
|
if tally.get(k):
|
|
print(f" {k:<13} {tally[k]}")
|
|
if only:
|
|
return 0
|
|
print(f"\n prediction was >=10 of 14 (the ADR's 9-of-12, 75%). "
|
|
f"Measured {red}: "
|
|
f"{'MET' if red >= 10 else 'UNMET'}")
|
|
if red <= 3:
|
|
print(" CB-WP-0005 stop condition: at or below 3 of 12-equivalent, "
|
|
"the pass is\n under-scoped and must re-plan before Phase C.")
|
|
return 0
|
|
|
|
|
|
def _wrap(text, width):
|
|
words, line, out = text.split(), "", []
|
|
for w in words:
|
|
if len(line) + len(w) + 1 > width:
|
|
out.append(line)
|
|
line = w
|
|
else:
|
|
line = f"{line} {w}".strip()
|
|
if line:
|
|
out.append(line)
|
|
return out
|
|
|
|
|
|
def self_test():
|
|
"""Each assertion pins a failure this harness must detect."""
|
|
results = []
|
|
|
|
def check(name, ok, detail=""):
|
|
results.append((name, ok, detail))
|
|
|
|
rs = rows()
|
|
check("the acceptance table is enumerated", len(rs) == 14, f"{len(rs)} rows")
|
|
check("every row has a mutation or a stated reason",
|
|
all(r.mutate or r.unmutatable for r in rs))
|
|
unmut = [r for r in rs if r.unmutatable]
|
|
check("no unmutatable row has a token reason",
|
|
bool(unmut) and all(len(r.unmutatable) > 40 for r in unmut),
|
|
f"{len(unmut)} unmutatable; a bare 'unmutatable' is how a row "
|
|
f"escapes the metric")
|
|
check("every mutation target file exists",
|
|
all(os.path.isfile(os.path.join(ROOT, r.mutate[0]))
|
|
for r in rs if r.mutate))
|
|
|
|
# CB-WP-0015 T03. `check_row` already reports a stale find-string as
|
|
# HARNESS-BROKEN — but only on a full mutation-check, which is
|
|
# deliberately not in `make all` because it rebuilds per row. So a
|
|
# mutation could rot for passes at a time: AM-4a's did, from the moment
|
|
# ADR-0008 D3 moved the shipped-runtime target 250,000 -> 161,000 until
|
|
# the next full run. This asks the same question with no builds at all,
|
|
# which puts it in `make all` via `self-tests`.
|
|
stale_targets = []
|
|
for r in rs:
|
|
targets = [r.mutate] if r.mutate else []
|
|
# Clause mutations rot the same way and are checked the same way.
|
|
targets += [c[3][1] for c in r.clauses if len(c) > 3 and c[3]]
|
|
for relpath, find, _new in targets:
|
|
try:
|
|
if find not in open(os.path.join(ROOT, relpath)).read():
|
|
stale_targets.append(r.id)
|
|
except OSError:
|
|
stale_targets.append(r.id)
|
|
check("every mutation find-string still matches its source",
|
|
not stale_targets,
|
|
f"stale: {', '.join(sorted(set(stale_targets)))}" if stale_targets
|
|
else "checked without building — the cheap half of check_row")
|
|
# The control that matters: a mutation whose find-string no longer
|
|
# matches must be reported BROKEN, not silently skipped.
|
|
stale = Row("AM-X", "fixture", verify=[sys.executable, "-c", "pass"],
|
|
mutate=("Makefile", "this-string-does-not-exist", "x"))
|
|
v, _ = check_row(stale)
|
|
check("a stale mutation target is reported broken, not skipped",
|
|
v == "HARNESS-BROKEN", v)
|
|
# And a mutation that does apply must be detected as applying.
|
|
live = Row("AM-Y", "fixture",
|
|
verify=[sys.executable, "-c",
|
|
"import sys,os; sys.exit(0 if 'ZZMARKER' "
|
|
"not in open('Makefile').read() else 1)"],
|
|
mutate=("Makefile", "PY := python3", "PY := python3 # ZZMARKER"))
|
|
v2, _ = check_row(live)
|
|
check("an applied mutation that breaks the verifier reports red",
|
|
v2 == "red", v2)
|
|
check("the tree is restored after a mutation run",
|
|
"ZZMARKER" not in open(os.path.join(ROOT, "Makefile")).read())
|
|
# The FA guard: a mutant that fails for the WRONG reason must not be
|
|
# scored as red. Without this, a mutation that merely fails to compile
|
|
# would credit its row with an assertion it does not have.
|
|
wrong = Row("AM-W", "fixture",
|
|
verify=[sys.executable, "-c",
|
|
"import sys; sys.exit(0 if 'ZZW' not in "
|
|
"open('Makefile').read() else 7)"],
|
|
mutate=("Makefile", "PY := python3", "PY := python3 # ZZW"),
|
|
expect="a message the verifier never prints")
|
|
v4, _ = check_row(wrong)
|
|
check("a mutant failing for the wrong reason is not scored red",
|
|
v4 == "WRONG-REASON", v4)
|
|
|
|
# CB-WP-0015 T01: clause-level mutations. A clause carrying a mutation
|
|
# gets its `enforced` flag MEASURED, so the control that matters is
|
|
# that a declaration disagreeing with its own measurement is refused
|
|
# rather than reported as either verdict — otherwise the fourth field
|
|
# would just be decoration on the same hand-maintained boolean.
|
|
ok_verify = [sys.executable, "-c",
|
|
"import sys; sys.exit(0 if 'ZZC' not in "
|
|
"open('Makefile').read() else 5)"]
|
|
noop = ("Makefile", "PY := python3", "PY := python3 ")
|
|
lying = Row("AM-C", "fixture", verify=ok_verify,
|
|
mutate=("Makefile", "PY := python3", "PY := python3 # ZZC"),
|
|
clauses=[("a clause that claims more than it can show", True,
|
|
"declared enforced, but its mutation changes "
|
|
"nothing the verifier looks at",
|
|
(ok_verify, noop, None))])
|
|
v5, d5 = check_row(lying)
|
|
check("a clause whose declaration contradicts its mutation is refused",
|
|
v5 == "HARNESS-BROKEN" and "disagree" in d5, f"{v5}: {d5[:40]}")
|
|
|
|
# A verifier that is already red must not be scored.
|
|
dead = Row("AM-Z", "fixture", verify=[sys.executable, "-c", "raise SystemExit(3)"],
|
|
mutate=("Makefile", "PY := python3", "PY := python3 "))
|
|
v3, _ = check_row(dead)
|
|
check("a red baseline is inconclusive, not red", v3 == "inconclusive", v3)
|
|
|
|
print("mutation-check self-test (positive control)")
|
|
ok = True
|
|
for name, passed, det in results:
|
|
print(f" [{'ok ' if passed else 'FAIL'}] {name}"
|
|
+ (f" — {det}" if det else ""))
|
|
ok &= passed
|
|
return 0 if ok else 1
|
|
|
|
|
|
def main():
|
|
enter_root()
|
|
ap = argparse.ArgumentParser()
|
|
ap.add_argument("--row")
|
|
ap.add_argument("--self-test", action="store_true")
|
|
args = ap.parse_args()
|
|
if args.self_test:
|
|
return self_test()
|
|
return report(args.row)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|