Fix the AM-5 instrument to measure quietly; the breach was not real
Some checks failed
ci / check (push) Failing after 3s
Some checks failed
ci / check (push) Failing after 3s
T03 reported AM-5 at 87.0 s / 61.3 s and called it a 45% breach of the
60 s target. Re-measured with the fixed instrument on a quiet machine:
load before measuring: 0.14 per CPU over 8 CPUs — quiet
dev toolchain (default features) 37.3 s [ok target <= 60 s]
best of 3: 37.3, 42.9, 46.2 (spread 1.24x)
shipped runtime (--no-default-features) 41.2 s [ok target <= 60 s]
best of 3: 41.2, 50.8, 54.2 (spread 1.32x)
AM-5 is MET with 1.6x headroom. The 87.0 s was measured while the machine
was busy with mutation-check and cargo builds — a timing measurement under
contention measures the contention.
That is the same error class as AM-6's, committed two tasks later in the
same session by the same author, in the row immediately after the one
where it was diagnosed. Knowing the failure mode did not prevent it; only
building the guard did. That is the InnerLoop v1.2 design-goal argument
holding up under a third instance: optimize for cheap correction, because
prevention keeps not converging.
The instrument now refuses to measure above 0.5 load per CPU, takes the
best of 3, and warns when the spread exceeds 1.25x. Best, not worst: a
build-time ceiling asks whether the machine can do it in 60 s, the mirror
of AM-6's best-of-N for a throughput floor. The spread warning fired on
the shipped-runtime samples — consecutive clean builds degrade 37.3 ->
46.2 — so a quiet machine is not a uniform one either.
The escalation to a maintainer decision is withdrawn: there is no breach.
The build profiling done while the breach was believed real is recorded in
the log rather than acted on — 174 s of CPU work at only 3.2x parallelism
on 8 cores, a ~22 s serial proc-macro chain, lto=thin worth ~6 s, and
pinning ppv-lite86 to drop zerocopy making it worse (23 -> 25 crates).
With 1.6x headroom there is nothing to buy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
ca832329e7
commit
5f7d9015d9
4 changed files with 112 additions and 20 deletions
|
|
@ -98,25 +98,58 @@ CB-EV-0001 called "very unlikely to bind" was right — but it is now
|
|||
*measured* rather than assumed, and verified red by a property mutation
|
||||
(a 300 MB allocation in the workload).
|
||||
|
||||
**AM-5: BREACHED, on both readings, on the machine the spec names.**
|
||||
**AM-5: ~~BREACHED~~ MET — the first reading was wrong.**
|
||||
|
||||
> **Corrected 2026-08-01 (during T04).** T03 reported 87.0 s / 61.3 s and
|
||||
> called AM-5 a 45% breach. **It is not.** Re-measured on a quiet machine
|
||||
> with the fixed instrument:
|
||||
>
|
||||
> ```text
|
||||
> load before measuring: 0.14 per CPU over 8 CPUs — quiet
|
||||
> dev toolchain (default features) 37.3 s [ok target <= 60 s]
|
||||
> best of 3: 37.3, 42.9, 46.2 (spread 1.24x)
|
||||
> shipped runtime (--no-default-features) 41.2 s [ok target <= 60 s]
|
||||
> best of 3: 41.2, 50.8, 54.2 (spread 1.32x)
|
||||
> ```
|
||||
>
|
||||
> **1.6× headroom, comfortably met.** The 87.0 s was measured while the
|
||||
> machine was busy running `mutation-check` and cargo builds — a timing
|
||||
> measurement under contention measures the contention. That is the *same
|
||||
> error class as AM-6's*, committed two tasks later in the same session,
|
||||
> by the same author, in the row immediately after the one where it was
|
||||
> diagnosed.
|
||||
>
|
||||
> The instrument now (a) reads `/proc` load average and **refuses to
|
||||
> measure** above 0.5 per CPU, and (b) takes the **best of 3** — best, not
|
||||
> worst, because a build-time *ceiling* asks "can this machine do it in
|
||||
> 60 s", the mirror of AM-6's best-of-N for a throughput *floor*. It also
|
||||
> warns when the spread exceeds 1.25×, which fired on the shipped-runtime
|
||||
> samples: consecutive clean builds degrade (37.3 → 46.2), so even a quiet
|
||||
> machine is not a uniform one.
|
||||
|
||||
The original, now-withdrawn finding follows for the record:
|
||||
|
||||
```text
|
||||
dev toolchain (default features) 87.0 s [FAIL target <= 60 s]
|
||||
shipped runtime (--no-default-features) 61.3 s [FAIL target <= 60 s]
|
||||
```
|
||||
|
||||
Measured on `bnt-lap001`, 8 cores — the machine `specs/GameKernel.md` §5
|
||||
names, so this is a direct comparison, not a directional one. A row
|
||||
declared `recorded not gated` and never recorded turns out, on first
|
||||
measurement, to **fail its own target by 45%**.
|
||||
|
||||
The tool **reports and exits 0**, because the spec says the row is not
|
||||
gated. Gating it is a spec change and needs an ADR; a tool that promotes
|
||||
gated (and on the corrected number there is nothing to escalate anyway). Gating it is a spec change and needs an ADR; a tool that promotes
|
||||
itself is how a target starts binding without anyone deciding it should.
|
||||
So AM-5 stays `unmutatable` — for the accurate reason now — and the breach
|
||||
is **raised as a maintainer decision**, not resolved here. Three honest
|
||||
options: speed the build, move the target by ADR (arguing why 60 s was
|
||||
wrong rather than why 87 s is convenient), or withdraw the row.
|
||||
So AM-5 stays `unmutatable` — it cannot fail while the spec declares it
|
||||
ungated, which is the accurate reason. **The escalation is withdrawn**:
|
||||
there is no breach to decide about.
|
||||
|
||||
A build profile was investigated anyway while the breach was believed
|
||||
real, and the findings stand on their own: 51 compile units, 174 s of CPU
|
||||
work compressed into ~54 s wall, **parallelism only 3.2× on 8 cores**
|
||||
because `proc-macro2 → syn → serde_derive → serde_core` is a ~22 s serial
|
||||
chain. `lto = "thin"` costs ~6 s (55.8 → 49.4 s mean) and is the only real
|
||||
lever; pinning `ppv-lite86` down to drop `zerocopy` (the largest single
|
||||
unit, 21.8 s) makes it **worse** — 23 → 25 crates, because the older
|
||||
version pulls `zerocopy-derive`. Recorded here rather than acted on: with
|
||||
1.6× headroom there is nothing to buy.
|
||||
|
||||
**The measurement had a real bug, found by cross-validation.**
|
||||
`getrusage(RUSAGE_CHILDREN)` is a high-water mark across *every* reaped
|
||||
|
|
|
|||
Binary file not shown.
|
|
@ -143,9 +143,12 @@ def rows():
|
|||
"deleting rows."),
|
||||
|
||||
Row("AM-5", "clean release build <= 60 s",
|
||||
unmutatable="now RECORDED (`make build-time`) and **breaching**: "
|
||||
"87.0 s dev toolchain, 61.3 s shipped runtime, "
|
||||
"against a 60 s target on bnt-lap001. Still counts "
|
||||
unmutatable="now RECORDED (`make build-time`) and **met**: "
|
||||
"37.3 s dev toolchain, 41.2 s shipped runtime, best "
|
||||
"of 3 on a quiet machine, against a 60 s target on "
|
||||
"bnt-lap001 (1.6x headroom). An earlier reading of "
|
||||
"87.0 s was taken under contention and was wrong. "
|
||||
"Still counts "
|
||||
"against M-D1-MUT because GameKernel §5 declares the "
|
||||
"row `recorded not gated` — a row that cannot fail "
|
||||
"asserts nothing, and promoting it is a spec change "
|
||||
|
|
|
|||
|
|
@ -39,10 +39,31 @@ from repo import ROOT, cargo_bin, cargo_env, enter_root
|
|||
|
||||
AM5_MAX_SECONDS = 60.0
|
||||
AM5_MACHINE = "bnt-lap001" # the spec names the machine; so do we
|
||||
# CB-WP-0006 T04 correction. The first AM-5 measurement read 87.0 s and was
|
||||
# reported as a 45% breach. Quiet re-runs read 54.1 s and 57.6 s — the row
|
||||
# passes. The 87 s was taken while the machine was busy with mutation-check
|
||||
# and cargo builds: a timing measurement under contention measures the
|
||||
# contention, which is the same error AM-6 had two tasks earlier.
|
||||
#
|
||||
# So AM-5 now (a) refuses to measure on a loaded machine and (b) takes the
|
||||
# BEST of N. Best, not worst: a build-time *ceiling* asks "can this machine
|
||||
# do it in 60 s", so the least-contended sample is the honest one — the
|
||||
# mirror of AM-6's best-of-N for a throughput *floor*.
|
||||
AM5_SAMPLES = 3
|
||||
AM5_MAX_LOAD_PER_CPU = 0.5 # refuse above this; the machine is busy
|
||||
AM5_SPREAD_WARN = 1.25 # max/min above this = not a quiet run
|
||||
AM9_MAX_RSS_MB = 64.0
|
||||
AM9_TEST = "replay_probe::replay_100k_events_is_linear_and_fast"
|
||||
|
||||
|
||||
def load_per_cpu():
|
||||
"""1-minute load average per CPU. > ~0.5 means real competing work."""
|
||||
try:
|
||||
return os.getloadavg()[0] / (os.cpu_count() or 1)
|
||||
except OSError:
|
||||
return 0.0
|
||||
|
||||
|
||||
def clean_build_seconds(extra=()):
|
||||
"""Wall seconds for a clean release build, into a throwaway target dir."""
|
||||
cargo = cargo_bin()
|
||||
|
|
@ -143,16 +164,41 @@ def report(fast=False):
|
|||
if host != AM5_MACHINE:
|
||||
print(f" NOTE: running on {host!r}, not {AM5_MACHINE!r} — the spec "
|
||||
f"target is machine-specific, so this is directional only.")
|
||||
# Positive control: refuse to measure on a busy machine rather than
|
||||
# publish the contention as a build time. This is the guard T03 did
|
||||
# not have, and its absence produced a reported 45% breach that was
|
||||
# not real.
|
||||
load = load_per_cpu()
|
||||
if load > AM5_MAX_LOAD_PER_CPU:
|
||||
print(f" ABORT — load average is {load:.2f} per CPU (limit "
|
||||
f"{AM5_MAX_LOAD_PER_CPU}); a build timed under contention "
|
||||
f"measures the contention. Re-run on a quiet machine.",
|
||||
file=sys.stderr)
|
||||
return 1
|
||||
print(f" load before measuring: {load:.2f} per CPU over "
|
||||
f"{os.cpu_count()} CPUs — quiet")
|
||||
|
||||
for label, extra in (("dev toolchain (default features)", ()),
|
||||
("shipped runtime (--no-default-features)",
|
||||
("--no-default-features",))):
|
||||
secs = clean_build_seconds(extra)
|
||||
if secs is None:
|
||||
print(f" ERROR — clean build failed: {label}", file=sys.stderr)
|
||||
return 1
|
||||
mark = "ok " if secs <= AM5_MAX_SECONDS else "FAIL"
|
||||
print(f" {label:<42} {secs:6.1f} s [{mark} target "
|
||||
samples = []
|
||||
for _ in range(AM5_SAMPLES):
|
||||
secs = clean_build_seconds(extra)
|
||||
if secs is None:
|
||||
print(f" ERROR — clean build failed: {label}", file=sys.stderr)
|
||||
return 1
|
||||
samples.append(secs)
|
||||
best, worst = min(samples), max(samples)
|
||||
mark = "ok " if best <= AM5_MAX_SECONDS else "FAIL"
|
||||
spread = worst / best if best else 1.0
|
||||
print(f" {label:<42} {best:6.1f} s [{mark} target "
|
||||
f"<= {AM5_MAX_SECONDS:.0f} s]")
|
||||
print(f" best of {AM5_SAMPLES}: "
|
||||
+ ", ".join(f"{x:.1f}" for x in sorted(samples))
|
||||
+ f" (spread {spread:.2f}x)")
|
||||
if spread > AM5_SPREAD_WARN:
|
||||
print(f" WARN — spread exceeds {AM5_SPREAD_WARN}x; the machine "
|
||||
f"was not quiet and this number is directional only")
|
||||
print(" NOTE: reported, not gated — specs/GameKernel.md §5 declares AM-5")
|
||||
print(" `recorded not gated`. Promoting it is a spec change and")
|
||||
print(" needs an ADR; this tool does not promote itself.")
|
||||
|
|
@ -203,6 +249,16 @@ def self_test():
|
|||
check("RSS agrees with an independent measurement",
|
||||
near, f"ours {rss:.1f} MB vs /usr/bin/time {indep:.1f} MB")
|
||||
|
||||
# The control for the defect this instrument actually had.
|
||||
check("AM-5 refuses to measure under load",
|
||||
AM5_MAX_LOAD_PER_CPU > 0 and "ABORT — load average" in open(__file__).read(),
|
||||
f"limit {AM5_MAX_LOAD_PER_CPU} per CPU; T03 reported 87.0 s under "
|
||||
f"contention against 54.1 s quiet")
|
||||
check("AM-5 takes the best of several samples",
|
||||
AM5_SAMPLES >= 3, f"{AM5_SAMPLES} samples")
|
||||
check("load is measured per CPU, not raw",
|
||||
0.0 <= load_per_cpu() < 100.0, f"{load_per_cpu():.2f} per CPU now")
|
||||
|
||||
check("clean build measures into a throwaway target dir, not target/",
|
||||
"CARGO_TARGET_DIR" in open(__file__).read()
|
||||
and "shutil.rmtree" in open(__file__).read(),
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue