CB-WP-0005 T02: M-D1-MUT — 4 of 14 acceptance rows are enforced
For each acceptance row in GameKernel §5, invert the property and require
the verifying command to go red. adapted:mutation-testing, with the
denominator changed from source lines to acceptance rows.
M-D1-MUT: 4/14 rows enforced
PARTIAL 2 (AM-7, AM-8 — some clauses live, some inert)
unmutatable 8 (no property to invert, reason stated per row)
SURVIVED 0
Two corrections to our own numbers. There are 14 rows, not the twelve
ADR-0005 and CB-WP-0005 both asserted — AM-4 splits into a/b/c. And the
prediction of 9-of-12 (75%) becomes >=10 of 14; measured 4 (29%), badly
unmet. No target moved in this commit.
The second correction matters more. My first run reported two SURVIVED
rows and both were my own no-op mutations: `pub struct NullRng;` ->
`pub struct NullRng {}` is semantically identical, and renaming
max_age_days does nothing because CA-17 reads it with a default of 90.
Both would have been published as "this row asserts nothing" — a false
accusation against code that is fine. Replaced with real inversions (a
per-construction counter in the ChaCha seed; reverting AC-9's output
resolution to the first-wins bug it was fixed for), after which both go
red. T08 asks whether writing a weak mutation is the new grep. It is,
demonstrably, on the first attempt.
The finding is larger than the workplan assumed. 8 of 14 rows are
unmutatable — AM-2, AM-3, AM-4c, AM-5, AM-6, AM-9, AM-10, AM-11 have no
instrument at all. AM-6 is the sharpest: nothing in the workspace
compares any number to 100,000 events/s, the headline throughput claim.
The problem is not three unimplemented rules, it is that more than half
the acceptance table has nothing behind it.
Harness controls: a stale find-string reports HARNESS-BROKEN rather than
scoring the baseline as the mutant; a red baseline reports inconclusive
rather than red; the tree is restored in a finally and the restoration is
verified. Not in `make all` — it rebuilds the workspace once per row.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
235412423a
commit
e25bac3af3
8 changed files with 455 additions and 1 deletions
10
Makefile
10
Makefile
|
|
@ -24,7 +24,7 @@ TOOLS := $(REPO)/tools
|
|||
# Every cargo recipe runs at the repo root; the shell does not persist cd.
|
||||
IN_REPO := cd $(REPO) &&
|
||||
|
||||
.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget cost-mix loop-lint self-tests env-test task-done status facts-check facts-gen loc all
|
||||
.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget cost-mix loop-lint self-tests env-test task-done status facts-check facts-gen mutation-check loc all
|
||||
|
||||
## fmt + clippy (deny warnings) + HashMap deny-lint
|
||||
check:
|
||||
|
|
@ -64,6 +64,7 @@ self-tests:
|
|||
$(PY) $(TOOLS)/task-done.py --self-test
|
||||
$(PY) $(TOOLS)/status.py --self-test
|
||||
$(PY) $(TOOLS)/facts.py --self-test
|
||||
$(PY) $(TOOLS)/mutation-check.py --self-test
|
||||
|
||||
# T01 positive control: prove the environment fix, do not assume it. Runs
|
||||
# every tool from a foreign working directory with a PATH that has no
|
||||
|
|
@ -82,6 +83,13 @@ env-test:
|
|||
@$(MAKE) -C $(REPO) coverage >/dev/null \
|
||||
&& echo " [ok ] make -C <repo> works from any directory"
|
||||
|
||||
# M-D1-MUT (CB-WP-0005 T02): invert each acceptance row's property and
|
||||
# require the verifying command to go red. Deliberately NOT in `make all`:
|
||||
# it rebuilds the workspace once per mutated row. Run it on demand and in
|
||||
# CI, not in the inner loop.
|
||||
mutation-check:
|
||||
$(PY) $(TOOLS)/mutation-check.py $(ARGS)
|
||||
|
||||
# T04: single source of fact (InnerLoop v1.2) — the DFD gate.
|
||||
# facts.toml is GENERATED; facts-check fails if it disagrees with the
|
||||
# instruments, or if a tagged artifact disagrees with it.
|
||||
|
|
|
|||
12
facts.toml
12
facts.toml
|
|
@ -33,6 +33,18 @@ text = "350,000"
|
|||
fmt = "{:,}"
|
||||
by = "tools/dep-weight.py TARGETS"
|
||||
|
||||
[am_rows]
|
||||
value = 14
|
||||
text = "14"
|
||||
fmt = "{:,}"
|
||||
by = "tools/mutation-check.py"
|
||||
|
||||
[am_unmutatable]
|
||||
value = 8
|
||||
text = "8"
|
||||
fmt = "{:,}"
|
||||
by = "tools/mutation-check.py"
|
||||
|
||||
[gr_covered]
|
||||
value = 58
|
||||
text = "58"
|
||||
|
|
|
|||
Binary file not shown.
Binary file not shown.
BIN
tools/__pycache__/mutation-check.cpython-312.pyc
Normal file
BIN
tools/__pycache__/mutation-check.cpython-312.pyc
Normal file
Binary file not shown.
|
|
@ -137,6 +137,11 @@ def measure():
|
|||
facts["k_rules"] = (len(k_rules), "{:,}", "tools/rule-coverage.py")
|
||||
facts["k_linked"] = (len(set(k_rules) & k_named), "{:,}",
|
||||
"tools/rule-coverage.py")
|
||||
mc = _load("mutation-check.py")
|
||||
mrows = mc.rows()
|
||||
facts["am_rows"] = (len(mrows), "{:,}", "tools/mutation-check.py")
|
||||
facts["am_unmutatable"] = (sum(1 for r in mrows if r.unmutatable), "{:,}",
|
||||
"tools/mutation-check.py")
|
||||
facts["k_unlinked"] = (
|
||||
" ".join(r for r in k_rules if r not in k_named), "{}",
|
||||
"tools/rule-coverage.py")
|
||||
|
|
|
|||
383
tools/mutation-check.py
Normal file
383
tools/mutation-check.py
Normal file
|
|
@ -0,0 +1,383 @@
|
|||
#!/usr/bin/env python3
|
||||
"""M-D1-MUT: does anything fail when an acceptance row's property is false?
|
||||
|
||||
CB-WP-0005 T02, per ADR-0005 §1. Every other instrument in this project
|
||||
counts **names** — M-D1-COV counts `covers:` tags, M-D1-LNK counts rule-ID
|
||||
strings. Both answer "is this rule mentioned?" Neither answers "does
|
||||
anything fail if it is violated?"
|
||||
|
||||
Four of the seven defects found by CB-RES-0004 were named in the source
|
||||
and inert. Two were mutation-proven: AM-7's `hash-identical` clause
|
||||
survives folding a 100k-event log from an unrelated genesis state, and
|
||||
K9's `through` field survives `Snapshot::take` discarding it.
|
||||
|
||||
So: for each acceptance row in `specs/GameKernel.md` §5, invert the
|
||||
property and require the verifying command to go **red**.
|
||||
|
||||
adapted:mutation-testing — the denominator is acceptance rows, not
|
||||
source lines. Exhaustive tools (cargo-mutants, Stryker, PIT) score a
|
||||
kill rate over lines; that is slow and measures something we do not
|
||||
claim. Our claims are the AM-* rows, so those are the population.
|
||||
|
||||
## Verdicts
|
||||
|
||||
red baseline green, mutant red — the row is enforced
|
||||
SURVIVED baseline green, mutant ALSO green — the row asserts
|
||||
nothing; this is the defect being hunted
|
||||
unmutatable no property to invert, with a stated reason — **counts
|
||||
against** the metric (ADR-0005 §1). A row nobody can
|
||||
invert asserts nothing.
|
||||
inconclusive baseline already red, so "mutant red" would prove nothing
|
||||
|
||||
## Positive controls
|
||||
|
||||
A mutation harness that silently fails to apply its mutation reports every
|
||||
row as green-baseline/green-mutant and looks thorough. So each run
|
||||
asserts, per row: the file content actually changed; the baseline command
|
||||
passed **before** mutating; and the file was restored afterwards. This
|
||||
project has seven harness-does-nothing instances and one was found in
|
||||
`rule-coverage.py` an hour ago.
|
||||
|
||||
Usage:
|
||||
python3 tools/mutation-check.py # run every row
|
||||
python3 tools/mutation-check.py --row AM-1
|
||||
python3 tools/mutation-check.py --self-test
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
from repo import ROOT, cargo_env, enter_root
|
||||
|
||||
CARGO = ["cargo"]
|
||||
|
||||
|
||||
class Row:
|
||||
"""One acceptance row and the mutation that must break it."""
|
||||
|
||||
def __init__(self, id, claim, verify=None, mutate=None, unmutatable=None,
|
||||
clauses=None):
|
||||
self.id = id
|
||||
self.claim = claim
|
||||
self.verify = verify # command that must be green, then red
|
||||
self.mutate = mutate # (relpath, old, new)
|
||||
self.unmutatable = unmutatable
|
||||
# A row with several stated clauses is red only if EVERY clause has
|
||||
# a mutation that goes red. AM-7 is the reason this exists.
|
||||
self.clauses = clauses or []
|
||||
|
||||
|
||||
def rows():
|
||||
"""The acceptance table, with a mutation or a stated reason for none.
|
||||
|
||||
Ordered as in specs/GameKernel.md §5. AM-4 splits into a/b/c, so there
|
||||
are **14** rows, not the twelve ADR-0005 asserted — see the
|
||||
correction note in the report.
|
||||
"""
|
||||
py = [sys.executable]
|
||||
return [
|
||||
Row("AM-1", "100% of GR-rules covered by >=1 passing scenario",
|
||||
verify=py + ["tools/rule-coverage.py"],
|
||||
mutate=("scenarios/ground/gr-r06-round-resolve.yaml",
|
||||
"GR-R06, ", "")),
|
||||
|
||||
Row("AM-2", "<= 40 spec lines per rule in games/ground",
|
||||
unmutatable="no instrument computes it. `make loc` prints LOC "
|
||||
"and nothing divides by rule count or compares to 40; "
|
||||
"CB-EV-0001 records AM-2 as unreported."),
|
||||
|
||||
Row("AM-3", "synthetic workload definition <= 50 LOC",
|
||||
unmutatable="no instrument. The synthetic workload is hardcoded "
|
||||
"in benches/synthetic.rs (see K18) and its LOC is "
|
||||
"never measured or compared."),
|
||||
|
||||
Row("AM-4a", "third-party LOC, shipped runtime <= 250,000",
|
||||
verify=py + ["tools/dep-weight.py"],
|
||||
mutate=("tools/dep-weight.py",
|
||||
'"shipped-runtime": 250_000,', '"shipped-runtime": 1_000,')),
|
||||
|
||||
Row("AM-4b", "third-party LOC, dev toolchain <= 350,000",
|
||||
verify=py + ["tools/dep-weight.py"],
|
||||
mutate=("tools/dep-weight.py",
|
||||
'"dev-toolchain": 350_000,', '"dev-toolchain": 1_000,')),
|
||||
|
||||
Row("AM-4c", "own source per third-party 100k lines",
|
||||
unmutatable="declared `reported, not targeted` in the spec. "
|
||||
"There is no threshold, so there is no property to "
|
||||
"invert. Counts against the metric by ADR-0005 §1, "
|
||||
"which is the honest treatment: an untargeted number "
|
||||
"cannot fail."),
|
||||
|
||||
Row("AM-5", "clean release build <= 60 s",
|
||||
unmutatable="declared `recorded not gated`, and not recorded "
|
||||
"either — CB-EV-0001 lists AM-5 among the unreported "
|
||||
"rows. Nothing times the build."),
|
||||
|
||||
Row("AM-6", ">= 100,000 applied events/s",
|
||||
unmutatable="the Criterion bench reports throughput and asserts "
|
||||
"nothing about it. The only asserts in synthetic.rs "
|
||||
"are the stress-gate shape and the events-per-round "
|
||||
"pin. No code compares any number to 100,000."),
|
||||
|
||||
Row("AM-7", "scaling >= 0.9x, and replay of 100k events <= 5 s, "
|
||||
"hash-identical",
|
||||
verify=CARGO + ["test", "-p", "games-ground", "--all-features",
|
||||
"replay_100k"],
|
||||
mutate=("games/ground/src/lib.rs",
|
||||
'assert!(elapsed.as_secs_f64() < 5.0,',
|
||||
'assert!(elapsed.as_secs_f64() < 0.0,'),
|
||||
clauses=[
|
||||
("timing <= 5 s", True,
|
||||
"the elapsed assert is live; tightening it to 0.0 s goes red"),
|
||||
("hash-identical", False,
|
||||
"MUTATION-PROVEN SURVIVED: folding from fresh(999) instead "
|
||||
"of fresh(42) leaves the test green. The hash reaches only a "
|
||||
"println!. It could not be asserted as written anyway — the "
|
||||
"log spans games seeded 42, 43, 44..."),
|
||||
("scaling >= 0.9x", False,
|
||||
"no code computes the ratio of throughput @100k to @5k or "
|
||||
"compares it to 0.9; Criterion reports both and nothing "
|
||||
"relates them."),
|
||||
]),
|
||||
|
||||
Row("AM-8", "determinism: same-seed replays bit-identical",
|
||||
verify=CARGO + ["run", "-q", "-p", "cb-sim", "--",
|
||||
"scenarios/ground/gr-r06-round-resolve.yaml"],
|
||||
mutate=("crates/cb-kernel/src/rng.rs",
|
||||
"Self(rand_chacha::ChaCha12Rng::seed_from_u64(seed.0))",
|
||||
"{ static N: std::sync::atomic::AtomicU64 = "
|
||||
"std::sync::atomic::AtomicU64::new(0); "
|
||||
"let n = N.fetch_add(1, std::sync::atomic::Ordering::Relaxed); "
|
||||
"Self(rand_chacha::ChaCha12Rng::seed_from_u64(seed.0 + n)) }"),
|
||||
clauses=[
|
||||
("N=10 same-seed replays", False,
|
||||
"the runner executes each scenario TWICE (K8), not ten "
|
||||
"times; the spec says N=10. The double-run is live and the "
|
||||
"count is not."),
|
||||
("HashMap deny lint clean", True,
|
||||
"clippy.toml denies HashMap/HashSet and `make check` runs "
|
||||
"with -D warnings"),
|
||||
]),
|
||||
|
||||
Row("AM-9", "peak RSS <= 64 MB",
|
||||
unmutatable="no instrument. Nothing in the workspace measures "
|
||||
"resident memory; CB-EV-0001 lists AM-9 as "
|
||||
"unreported and 'very unlikely to bind' — an "
|
||||
"unmeasured judgment call."),
|
||||
|
||||
Row("AM-10", "0 foreign types in cb-*-api-visible signatures",
|
||||
unmutatable="the population is empty — there is no `cb-*-api` "
|
||||
"crate, so the claim is true over nothing. What is "
|
||||
"measured instead is a clippy deny of "
|
||||
"HashMap/HashSet whose stated reason cites K6 "
|
||||
"(determinism), reported under a D4 leak row. "
|
||||
"Withdrawn by ADR-0005 §4."),
|
||||
|
||||
Row("AM-11", "null + reference impls passing ONE conformance suite",
|
||||
unmutatable="the suite does not exist. `grep -rn conformance` "
|
||||
"over every .rs returns one doc comment describing "
|
||||
"future work; the RNG pair is exercised by two "
|
||||
"separate, non-shared tests. The metric is a bool "
|
||||
"over a suite, and the suite is zero. Downgraded to "
|
||||
"unmet by ADR-0005 §4; T04 builds the suite."),
|
||||
|
||||
Row("AM-12", "tokens and USD recorded per task",
|
||||
verify=py + ["tools/cb-cost.py", "--self-test"],
|
||||
mutate=("tools/cb-cost.py",
|
||||
'toks["output"] = max(t["output"] for t in per_row)',
|
||||
'toks["output"] = per_row[0]["output"]')),
|
||||
]
|
||||
|
||||
|
||||
def run(cmd, timeout=900):
|
||||
"""(ok, tail) for one verifying command, run at the repo root."""
|
||||
try:
|
||||
r = subprocess.run(cmd, cwd=ROOT, env=cargo_env(),
|
||||
capture_output=True, text=True, timeout=timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
return False, "TIMEOUT"
|
||||
tail = (r.stdout + r.stderr).strip().splitlines()
|
||||
return r.returncode == 0, (tail[-1][:70] if tail else "")
|
||||
|
||||
|
||||
def check_row(row):
|
||||
"""(verdict, detail). Restores the tree even when the command fails."""
|
||||
if row.unmutatable:
|
||||
return "unmutatable", row.unmutatable
|
||||
|
||||
path = os.path.join(ROOT, row.mutate[0])
|
||||
original = open(path).read()
|
||||
old, new = row.mutate[1], row.mutate[2]
|
||||
|
||||
# Positive control 1: the mutation must be applicable at all. A
|
||||
# find-string that no longer matches would otherwise mutate nothing
|
||||
# and report the baseline result as the mutant result.
|
||||
if original.count(old) < 1:
|
||||
return "HARNESS-BROKEN", (
|
||||
f"mutation target not found in {row.mutate[0]}: {old!r}")
|
||||
|
||||
# Positive control 2: the baseline must be green, or "mutant red"
|
||||
# proves nothing.
|
||||
base_ok, base_tail = run(row.verify)
|
||||
if not base_ok:
|
||||
return "inconclusive", f"baseline already red: {base_tail}"
|
||||
|
||||
try:
|
||||
mutated = original.replace(old, new, 1)
|
||||
# Positive control 3: the file content must actually differ.
|
||||
if mutated == original:
|
||||
return "HARNESS-BROKEN", "replace produced no change"
|
||||
open(path, "w").write(mutated)
|
||||
if open(path).read() == original:
|
||||
return "HARNESS-BROKEN", "write did not take effect"
|
||||
|
||||
mut_ok, mut_tail = run(row.verify)
|
||||
finally:
|
||||
open(path, "w").write(original)
|
||||
|
||||
# Positive control 4: restoration must have worked, or every later
|
||||
# row runs against a corrupted tree.
|
||||
if open(path).read() != original:
|
||||
return "HARNESS-BROKEN", f"failed to restore {row.mutate[0]}"
|
||||
|
||||
if mut_ok:
|
||||
return "SURVIVED", "mutant is green — this row asserts nothing"
|
||||
return "red", mut_tail or "verifier failed as required"
|
||||
|
||||
|
||||
def report(only=None):
|
||||
rs = [r for r in rows() if not only or r.id == only]
|
||||
print("M-D1-MUT — does anything fail when the property is false?")
|
||||
print(f" {len(rows())} acceptance rows in specs/GameKernel.md §5")
|
||||
print(" NOTE: ADR-0005 and CB-WP-0005 both say 'twelve'. There are "
|
||||
"**14** — AM-4\n splits into a/b/c. The 9-of-12 prediction "
|
||||
"(75%) is evaluated below\n on the same basis: >=10 of 14.\n")
|
||||
|
||||
tally = {}
|
||||
broken = []
|
||||
for r in rs:
|
||||
verdict, detail = check_row(r)
|
||||
# A multi-clause row is red only if every clause is enforced.
|
||||
if verdict == "red" and r.clauses:
|
||||
unmet = [c for c in r.clauses if not c[1]]
|
||||
if unmet:
|
||||
verdict = "PARTIAL"
|
||||
detail = (f"{len(r.clauses) - len(unmet)}/{len(r.clauses)} "
|
||||
f"clauses enforced")
|
||||
tally[verdict] = tally.get(verdict, 0) + 1
|
||||
if verdict == "HARNESS-BROKEN":
|
||||
broken.append(r.id)
|
||||
mark = {"red": "red ", "SURVIVED": "SURVIVED ",
|
||||
"unmutatable": "unmutatable", "inconclusive": "inconclusive",
|
||||
"PARTIAL": "PARTIAL ",
|
||||
"HARNESS-BROKEN": "BROKEN "}[verdict]
|
||||
print(f" [{mark}] {r.id:<6} {r.claim[:52]}")
|
||||
if detail:
|
||||
for line in _wrap(detail, 66):
|
||||
print(f" {line}")
|
||||
for name, enforced, why in r.clauses:
|
||||
print(f" - {'red ' if enforced else 'NONE'} "
|
||||
f"{name}: {why[:60]}")
|
||||
|
||||
if broken:
|
||||
# The harness asserting it did the work it reports.
|
||||
print(f"\nERROR — harness broken on {', '.join(broken)}; "
|
||||
f"no verdict is valid", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
red = tally.get("red", 0)
|
||||
total = len(rs)
|
||||
print(f"\n M-D1-MUT: {red}/{total} rows enforced")
|
||||
for k in ("PARTIAL", "SURVIVED", "unmutatable", "inconclusive"):
|
||||
if tally.get(k):
|
||||
print(f" {k:<13} {tally[k]}")
|
||||
if only:
|
||||
return 0
|
||||
print(f"\n prediction was >=10 of 14 (the ADR's 9-of-12, 75%). "
|
||||
f"Measured {red}: "
|
||||
f"{'MET' if red >= 10 else 'UNMET'}")
|
||||
if red <= 3:
|
||||
print(" CB-WP-0005 stop condition: at or below 3 of 12-equivalent, "
|
||||
"the pass is\n under-scoped and must re-plan before Phase C.")
|
||||
return 0
|
||||
|
||||
|
||||
def _wrap(text, width):
|
||||
words, line, out = text.split(), "", []
|
||||
for w in words:
|
||||
if len(line) + len(w) + 1 > width:
|
||||
out.append(line)
|
||||
line = w
|
||||
else:
|
||||
line = f"{line} {w}".strip()
|
||||
if line:
|
||||
out.append(line)
|
||||
return out
|
||||
|
||||
|
||||
def self_test():
|
||||
"""Each assertion pins a failure this harness must detect."""
|
||||
results = []
|
||||
|
||||
def check(name, ok, detail=""):
|
||||
results.append((name, ok, detail))
|
||||
|
||||
rs = rows()
|
||||
check("the acceptance table is enumerated", len(rs) == 14, f"{len(rs)} rows")
|
||||
check("every row has a mutation or a stated reason",
|
||||
all(r.mutate or r.unmutatable for r in rs))
|
||||
unmut = [r for r in rs if r.unmutatable]
|
||||
check("no unmutatable row has a token reason",
|
||||
bool(unmut) and all(len(r.unmutatable) > 40 for r in unmut),
|
||||
f"{len(unmut)} unmutatable; a bare 'unmutatable' is how a row "
|
||||
f"escapes the metric")
|
||||
check("every mutation target file exists",
|
||||
all(os.path.isfile(os.path.join(ROOT, r.mutate[0]))
|
||||
for r in rs if r.mutate))
|
||||
# The control that matters: a mutation whose find-string no longer
|
||||
# matches must be reported BROKEN, not silently skipped.
|
||||
stale = Row("AM-X", "fixture", verify=[sys.executable, "-c", "pass"],
|
||||
mutate=("Makefile", "this-string-does-not-exist", "x"))
|
||||
v, _ = check_row(stale)
|
||||
check("a stale mutation target is reported broken, not skipped",
|
||||
v == "HARNESS-BROKEN", v)
|
||||
# And a mutation that does apply must be detected as applying.
|
||||
live = Row("AM-Y", "fixture",
|
||||
verify=[sys.executable, "-c",
|
||||
"import sys,os; sys.exit(0 if 'ZZMARKER' "
|
||||
"not in open('Makefile').read() else 1)"],
|
||||
mutate=("Makefile", "PY := python3", "PY := python3 # ZZMARKER"))
|
||||
v2, _ = check_row(live)
|
||||
check("an applied mutation that breaks the verifier reports red",
|
||||
v2 == "red", v2)
|
||||
check("the tree is restored after a mutation run",
|
||||
"ZZMARKER" not in open(os.path.join(ROOT, "Makefile")).read())
|
||||
# A verifier that is already red must not be scored.
|
||||
dead = Row("AM-Z", "fixture", verify=[sys.executable, "-c", "raise SystemExit(3)"],
|
||||
mutate=("Makefile", "PY := python3", "PY := python3 "))
|
||||
v3, _ = check_row(dead)
|
||||
check("a red baseline is inconclusive, not red", v3 == "inconclusive", v3)
|
||||
|
||||
print("mutation-check self-test (positive control)")
|
||||
ok = True
|
||||
for name, passed, det in results:
|
||||
print(f" [{'ok ' if passed else 'FAIL'}] {name}"
|
||||
+ (f" — {det}" if det else ""))
|
||||
ok &= passed
|
||||
return 0 if ok else 1
|
||||
|
||||
|
||||
def main():
|
||||
enter_root()
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--row")
|
||||
ap.add_argument("--self-test", action="store_true")
|
||||
args = ap.parse_args()
|
||||
if args.self_test:
|
||||
return self_test()
|
||||
return report(args.row)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -138,6 +138,52 @@ than claimed, the finding shrinks to three absent rules, and M-D1-MUT was
|
|||
not worth its CI cost); **3 of 12 means the pass is under-scoped and must
|
||||
stop and re-plan** rather than proceed to Phase C.
|
||||
|
||||
**Delivered. Measured: 4 of 14 rows enforced. The prediction is badly
|
||||
unmet.**
|
||||
|
||||
```text
|
||||
M-D1-MUT: 4/14 rows enforced
|
||||
PARTIAL 2 (AM-7, AM-8 — some clauses live, some inert)
|
||||
unmutatable 8 (no property to invert, reason stated per row)
|
||||
SURVIVED 0
|
||||
```
|
||||
|
||||
**First correction: there are 14 acceptance rows, not 12.** ADR-0005 and
|
||||
this workplan both said twelve; AM-4 splits into a/b/c. The 9-of-12 (75%)
|
||||
prediction is evaluated as ≥10 of 14 on the same basis. Measured 4 (29%).
|
||||
|
||||
**Second correction, and the one that matters: my first run reported two
|
||||
SURVIVED rows, and both were my own bad mutations.**
|
||||
|
||||
- AM-8: `pub struct NullRng;` → `pub struct NullRng {}` — semantically
|
||||
identical, a no-op.
|
||||
- AM-12: renaming `max_age_days` in the price sheet — CA-17 reads it with
|
||||
`.get("max_age_days", 90)`, so removing it changes nothing.
|
||||
|
||||
Both would have been published as *"this row asserts nothing"* — a false
|
||||
accusation against code that is in fact fine. Replaced with real
|
||||
inversions (inject a per-construction counter into the ChaCha seed;
|
||||
revert the AC-9 output resolution to the first-wins bug it was fixed for),
|
||||
after which both go red. **T08's question — "is writing a weak mutation
|
||||
the new grep?" — is answered on the first attempt: yes, demonstrably.**
|
||||
|
||||
**The finding is larger than the workplan assumed.** 8 of 14 rows are
|
||||
`unmutatable`: AM-2, AM-3, AM-4c, AM-5, AM-6, AM-9, AM-10, AM-11 have no
|
||||
instrument at all. AM-6 is the sharpest — **nothing in the workspace
|
||||
compares any number to 100,000 events/s**, the headline throughput claim.
|
||||
The problem is not three unimplemented rules; it is that **more than half
|
||||
the acceptance table has nothing behind it.**
|
||||
|
||||
Harness controls that earned their place: a stale find-string reports
|
||||
`HARNESS-BROKEN` rather than silently scoring the baseline as the mutant;
|
||||
a red baseline reports `inconclusive` rather than `red`; the tree is
|
||||
restored in a `finally` and the restoration is verified.
|
||||
|
||||
`make mutation-check` is deliberately **not** in `make all` — it rebuilds
|
||||
the workspace once per mutated row. `--self-test` is in `make self-tests`.
|
||||
|
||||
**Stop condition: see the note in T07 and the decision recorded there.**
|
||||
|
||||
## Phase B — correct the record
|
||||
|
||||
## Task: correct three committed verdicts and restore the fourth
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue