coulomb/clay-borg
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

CB-WP-0006 T03: measure AM-5 and AM-9; AM-5 breaches

Browse source 
AM-9 is met and gated: 13.4 MB peak RSS against a 64 MB target, 4.8x
headroom, in `make all` via --fast. CB-EV-0001's "very unlikely to bind"
was right, but it is now measured rather than assumed, and verified red by
a property mutation (a 300 MB allocation in the workload).

AM-5 is BREACHED on both readings, on the machine the spec names:

  dev toolchain (default features)        87.0 s  [FAIL target <= 60 s]
  shipped runtime (--no-default-features) 61.3 s  [FAIL target <= 60 s]

bnt-lap001, 8 cores — a direct comparison, not a directional one. A row
declared "recorded not gated" and never recorded fails its own target by
45% on first measurement.

The tool reports and exits 0 because the spec says the row is ungated.
Gating it is a spec change needing an ADR; a tool that promotes itself is
how a target starts binding without anyone deciding it should. So AM-5
stays unmutatable — for the accurate reason now — and the breach is raised
as a maintainer decision: speed the build, move the target by ADR (arguing
why 60 s was wrong rather than why 87 s is convenient), or withdraw the
row.

The measurement itself had a real bug, found only by cross-validation.
getrusage(RUSAGE_CHILDREN) is a high-water mark across every reaped child,
so it attributed cargo's memory to the workload and reported 38.2 MB for a
run that used 12.3 MB — a 3x over-report that was plausible, passed its
target, and would have been published. Fixed with os.wait4, which returns
that specific child's rusage, and the self-test now cross-checks against
/usr/bin/time -v.

That is the false-accusation shape in the measurement layer rather than
the mutation layer: an instrument confidently reporting a number it had
not earned.

Also: the clean build measures into a throwaway CARGO_TARGET_DIR rather
than running `cargo clean`, so measuring the metric does not cost several
minutes of rebuild afterwards. A metric that punishes its own measurement
gets measured once and never again.

M-D1-MUT: 6 -> 7 of 14.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 18:59:32 +02:00
parent fd67813876
commit e7312b3b8a
8 changed files with 301 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

24
tools/mutation-check.py
View file

@ -133,9 +133,15 @@ def rows():
"cannot fail."),
Row("AM-5", "clean release build <= 60 s",
unmutatable="declared `recorded not gated`, and not recorded "
"either — CB-EV-0001 lists AM-5 among the unreported "
"rows. Nothing times the build."),
unmutatable="now RECORDED (`make build-time`) and **breaching**: "
"87.0 s dev toolchain, 61.3 s shipped runtime, "
"against a 60 s target on bnt-lap001. Still counts "
"against M-D1-MUT because GameKernel §5 declares the "
"row `recorded not gated` — a row that cannot fail "
"asserts nothing, and promoting it is a spec change "
"needing an ADR. The breach is a maintainer decision, "
"not something this tool should resolve by gating "
"itself."),
Row("AM-6", ">= 100,000 applied events/s",
verify=CARGO + ["test", "-p", "games-ground", "--all-features",
@ -191,11 +197,15 @@ def rows():
"with -D warnings"),
]),
# A PROPERTY mutation: make the workload actually use memory.
Row("AM-9", "peak RSS <= 64 MB",
unmutatable="no instrument. Nothing in the workspace measures "
"resident memory; CB-EV-0001 lists AM-9 as "
"unreported and 'very unlikely to bind' — an "
"unmeasured judgment call."),
verify=py + ["tools/runtime-metrics.py", "--fast"],
mutate=("games/ground/src/lib.rs",
" fn replay_100k_events_is_linear_and_fast() {",
" fn replay_100k_events_is_linear_and_fast() {\n"
" let hog: Vec<u8> = vec![7u8; 300_000_000];\n"
" std::hint::black_box(&hog);"),
expect="FAIL target <= 64 MB"),
Row("AM-10", "0 foreign types in cb-*-api-visible signatures",
unmutatable="the population is empty — there is no `cb-*-api` "