CB-WP-0006 T03: measure AM-5 and AM-9; AM-5 breaches
AM-9 is met and gated: 13.4 MB peak RSS against a 64 MB target, 4.8x headroom, in `make all` via --fast. CB-EV-0001's "very unlikely to bind" was right, but it is now measured rather than assumed, and verified red by a property mutation (a 300 MB allocation in the workload). AM-5 is BREACHED on both readings, on the machine the spec names: dev toolchain (default features) 87.0 s [FAIL target <= 60 s] shipped runtime (--no-default-features) 61.3 s [FAIL target <= 60 s] bnt-lap001, 8 cores — a direct comparison, not a directional one. A row declared "recorded not gated" and never recorded fails its own target by 45% on first measurement. The tool reports and exits 0 because the spec says the row is ungated. Gating it is a spec change needing an ADR; a tool that promotes itself is how a target starts binding without anyone deciding it should. So AM-5 stays unmutatable — for the accurate reason now — and the breach is raised as a maintainer decision: speed the build, move the target by ADR (arguing why 60 s was wrong rather than why 87 s is convenient), or withdraw the row. The measurement itself had a real bug, found only by cross-validation. getrusage(RUSAGE_CHILDREN) is a high-water mark across every reaped child, so it attributed cargo's memory to the workload and reported 38.2 MB for a run that used 12.3 MB — a 3x over-report that was plausible, passed its target, and would have been published. Fixed with os.wait4, which returns that specific child's rusage, and the self-test now cross-checks against /usr/bin/time -v. That is the false-accusation shape in the measurement layer rather than the mutation layer: an instrument confidently reporting a number it had not earned. Also: the clean build measures into a throwaway CARGO_TARGET_DIR rather than running `cargo clean`, so measuring the metric does not cost several minutes of rebuild afterwards. A metric that punishes its own measurement gets measured once and never again. M-D1-MUT: 6 -> 7 of 14. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
fd67813876
commit
e7312b3b8a
8 changed files with 301 additions and 11 deletions
|
|
@ -182,6 +182,48 @@ argument**, which is a legitimate outcome and possibly the right one for
|
|||
AM-5 on a machine-dependent number. What is not legitimate is a third pass
|
||||
leaving them blank while the scoreboard reports elsewhere.
|
||||
|
||||
**Delivered — `tools/runtime-metrics.py`. One passes, one breaches.**
|
||||
|
||||
**AM-9: met, and comfortably.** 13.4 MB peak RSS against a ≤64 MB target,
|
||||
**4.8× headroom**. Gated and in `make all` (`--fast`, ~1 s). The row that
|
||||
CB-EV-0001 called "very unlikely to bind" was right — but it is now
|
||||
*measured* rather than assumed, and verified red by a property mutation
|
||||
(a 300 MB allocation in the workload).
|
||||
|
||||
**AM-5: BREACHED, on both readings, on the machine the spec names.**
|
||||
|
||||
```text
|
||||
dev toolchain (default features) 87.0 s [FAIL target <= 60 s]
|
||||
shipped runtime (--no-default-features) 61.3 s [FAIL target <= 60 s]
|
||||
```
|
||||
|
||||
Measured on `bnt-lap001`, 8 cores — the machine `specs/GameKernel.md` §5
|
||||
names, so this is a direct comparison, not a directional one. A row
|
||||
declared `recorded not gated` and never recorded turns out, on first
|
||||
measurement, to **fail its own target by 45%**.
|
||||
|
||||
The tool **reports and exits 0**, because the spec says the row is not
|
||||
gated. Gating it is a spec change and needs an ADR; a tool that promotes
|
||||
itself is how a target starts binding without anyone deciding it should.
|
||||
So AM-5 stays `unmutatable` — for the accurate reason now — and the breach
|
||||
is **raised as a maintainer decision**, not resolved here. Three honest
|
||||
options: speed the build, move the target by ADR (arguing why 60 s was
|
||||
wrong rather than why 87 s is convenient), or withdraw the row.
|
||||
|
||||
**The measurement had a real bug, found by cross-validation.**
|
||||
`getrusage(RUSAGE_CHILDREN)` is a high-water mark across *every* reaped
|
||||
child, so it attributed `cargo`'s memory to the workload and reported
|
||||
**38.2 MB for a run that used 12.3 MB** — a 3× over-report. Fixed with
|
||||
`os.wait4`, which returns that specific child's rusage. The self-test now
|
||||
**cross-checks against `/usr/bin/time -v`** (13.4 vs 12.4 MB), which is
|
||||
the only reason the bug was visible at all: the wrong number was
|
||||
plausible, passed its target, and would have been published.
|
||||
|
||||
That is FA in the measurement layer rather than the mutation layer — an
|
||||
instrument confidently reporting a number it had not earned.
|
||||
|
||||
**M-D1-MUT: 6 → 7 of 14.**
|
||||
|
||||
## Task: AM-4c — target it or drop it
|
||||
|
||||
```task
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue