181 lines
8.4 KiB
Markdown
181 lines
8.4 KiB
Markdown
|
|
# CB-WP-0006 — delivery log
|
|||
|
|
|
|||
|
|
Per-task delivery notes, moved out of
|
|||
|
|
[the workplan](../workplans/CB-WP-0006-instrument-the-table.md) when it
|
|||
|
|
crossed the ~400-line loadability limit for the third time in this
|
|||
|
|
project. The workplan holds the plan; this holds what happened. T08
|
|||
|
|
draws on both.
|
|||
|
|
|
|||
|
|
## CB-WP-0006-T01
|
|||
|
|
|
|||
|
|
**Delivered.** `am6_throughput_clears_the_spec_target` in
|
|||
|
|
`games/ground/src/lib.rs` — a test, not a bench. Best of 3 samples of
|
|||
|
|
50,000 applied events each.
|
|||
|
|
|
|||
|
|
Measured on bnt-lap001 2026-07-31: **341,280 ev/s in debug (3.4× the
|
|||
|
|
target)**, ~2.4–3.1M in release (~24–30×). **The spec target holds even in
|
|||
|
|
an unoptimized build**, so the gate needed no `cfg` split and runs in the
|
|||
|
|
ordinary `make test`.
|
|||
|
|
|
|||
|
|
**The trap was avoided by construction, not by intention.** The threshold
|
|||
|
|
is the spec value `100_000`, untouched; the constant carries a comment
|
|||
|
|
saying lowering it requires an ADR; and the failure message repeats that,
|
|||
|
|
states the measured headroom, and names the reference figures — so a
|
|||
|
|
future agent hitting a red AM-6 is told not to tune it, in the place they
|
|||
|
|
will actually be reading. Robustness comes from **best-of-N**, not from a
|
|||
|
|
lower bar: a throughput *floor* asks "is this machine capable", so
|
|||
|
|
transient load should not fail the build.
|
|||
|
|
|
|||
|
|
Two positive controls in the test itself: a run that applied fewer than
|
|||
|
|
50,000 events, or measured zero elapsed time, fails rather than scoring as
|
|||
|
|
infinite throughput.
|
|||
|
|
|
|||
|
|
**Verified:** `make mutation-check --row AM-6` → **red**, via a *property*
|
|||
|
|
mutation (4,000 `black_box` iterations injected into `GroundState::fold`,
|
|||
|
|
the hot path) rather than a threshold tweak — raising the target would only
|
|||
|
|
prove the comparison runs.
|
|||
|
|
|
|||
|
|
**And the FA class is now gated.** `mutation-check` rows gained an
|
|||
|
|
`expect` field: the mutant's output must contain the row's stated failure
|
|||
|
|
string, or the verdict is **`WRONG-REASON`**, not `red`. Without it, a
|
|||
|
|
mutation that merely failed to compile would credit its row with an
|
|||
|
|
assertion it does not have. Verified by pointing `expect` at a string the
|
|||
|
|
verifier never prints and confirming the verdict flips. This is
|
|||
|
|
remedy (2) from the CB-WP-0005 retrospective, built one task earlier than
|
|||
|
|
T08 planned because the class it guards is the newest and the most
|
|||
|
|
dangerous.
|
|||
|
|
|
|||
|
|
**M-D1-MUT: 4 → 5 of 14.**
|
|||
|
|
|
|||
|
|
## CB-WP-0006-T02
|
|||
|
|
|
|||
|
|
**Delivered — but the two rows resolved differently, and the difference is
|
|||
|
|
the point.**
|
|||
|
|
|
|||
|
|
**AM-2 is instrumented and enforced.** `tools/size-metrics.py` +
|
|||
|
|
`make size-metrics`, in `make all`:
|
|||
|
|
|
|||
|
|
```text
|
|||
|
|
AM-2: 27.2 LOC/rule [ok target <= 40] (1.47x headroom)
|
|||
|
|
1,575 code lines before the first #[cfg(test)] / 58 numbered rules
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Tests are excluded because AM-2 asks what a rule *costs*, not how much it
|
|||
|
|
is exercised — `lib.rs` is ~18% test code and including it would have
|
|||
|
|
flattered the number. Verified red by a **property** mutation: ~800 lines
|
|||
|
|
of filler injected into the impl, pushing the ratio past 40. `expect` is
|
|||
|
|
the precise failure signature `FAIL target <= 40`, not the row name, which
|
|||
|
|
would have matched passing output too.
|
|||
|
|
|
|||
|
|
**AM-3 is BLOCKED, not uninstrumented — and this is a finding, not a
|
|||
|
|
deferral.** The row measures "LOC to express the CB-RES-0001 synthetic
|
|||
|
|
game on our kernel" against a boardgame.io baseline of ~36 LOC for a
|
|||
|
|
declarative **3p commit/reveal game object**. That artifact has never been
|
|||
|
|
built: `games/` contains only `ground`, and `benches/synthetic.rs`
|
|||
|
|
*drives* GROUND rather than *defining* a synthetic game.
|
|||
|
|
|
|||
|
|
Measuring GROUND's 1,575 impl lines against a 36-line synthetic game
|
|||
|
|
object would compare **two different games** and call the difference a D1
|
|||
|
|
result. So the tool ships the measurement mechanism — a marker-delimited
|
|||
|
|
`// AM-3:BEGIN` / `// AM-3:END` region, self-tested — and **reports the row
|
|||
|
|
blocked, naming the missing artifact**. A number here would have been
|
|||
|
|
worse than a blank.
|
|||
|
|
|
|||
|
|
It therefore stays `unmutatable` and **still counts against M-D1-MUT**, per
|
|||
|
|
ADR-0005 §1: a row that cannot fail asserts nothing, however good the
|
|||
|
|
reason. Resolving it needs an artifact, not a metric tweak — carried
|
|||
|
|
forward, not silently dropped.
|
|||
|
|
|
|||
|
|
**M-D1-MUT: 5 → 6 of 14.**
|
|||
|
|
|
|||
|
|
## CB-WP-0006-T03
|
|||
|
|
|
|||
|
|
**Delivered — `tools/runtime-metrics.py`. One passes, one breaches.**
|
|||
|
|
|
|||
|
|
**AM-9: met, and comfortably.** 13.4 MB peak RSS against a ≤64 MB target,
|
|||
|
|
**4.8× headroom**. Gated and in `make all` (`--fast`, ~1 s). The row that
|
|||
|
|
CB-EV-0001 called "very unlikely to bind" was right — but it is now
|
|||
|
|
*measured* rather than assumed, and verified red by a property mutation
|
|||
|
|
(a 300 MB allocation in the workload).
|
|||
|
|
|
|||
|
|
**AM-5: BREACHED, on both readings, on the machine the spec names.**
|
|||
|
|
|
|||
|
|
```text
|
|||
|
|
dev toolchain (default features) 87.0 s [FAIL target <= 60 s]
|
|||
|
|
shipped runtime (--no-default-features) 61.3 s [FAIL target <= 60 s]
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Measured on `bnt-lap001`, 8 cores — the machine `specs/GameKernel.md` §5
|
|||
|
|
names, so this is a direct comparison, not a directional one. A row
|
|||
|
|
declared `recorded not gated` and never recorded turns out, on first
|
|||
|
|
measurement, to **fail its own target by 45%**.
|
|||
|
|
|
|||
|
|
The tool **reports and exits 0**, because the spec says the row is not
|
|||
|
|
gated. Gating it is a spec change and needs an ADR; a tool that promotes
|
|||
|
|
itself is how a target starts binding without anyone deciding it should.
|
|||
|
|
So AM-5 stays `unmutatable` — for the accurate reason now — and the breach
|
|||
|
|
is **raised as a maintainer decision**, not resolved here. Three honest
|
|||
|
|
options: speed the build, move the target by ADR (arguing why 60 s was
|
|||
|
|
wrong rather than why 87 s is convenient), or withdraw the row.
|
|||
|
|
|
|||
|
|
**The measurement had a real bug, found by cross-validation.**
|
|||
|
|
`getrusage(RUSAGE_CHILDREN)` is a high-water mark across *every* reaped
|
|||
|
|
child, so it attributed `cargo`'s memory to the workload and reported
|
|||
|
|
**38.2 MB for a run that used 12.3 MB** — a 3× over-report. Fixed with
|
|||
|
|
`os.wait4`, which returns that specific child's rusage. The self-test now
|
|||
|
|
**cross-checks against `/usr/bin/time -v`** (13.4 vs 12.4 MB), which is
|
|||
|
|
the only reason the bug was visible at all: the wrong number was
|
|||
|
|
plausible, passed its target, and would have been published.
|
|||
|
|
|
|||
|
|
That is FA in the measurement layer rather than the mutation layer — an
|
|||
|
|
instrument confidently reporting a number it had not earned.
|
|||
|
|
|
|||
|
|
**M-D1-MUT: 6 → 7 of 14.**
|
|||
|
|
|
|||
|
|
## CB-WP-0006-T04
|
|||
|
|
|
|||
|
|
**Delivered: withdrawn from the acceptance table, retained as a
|
|||
|
|
diagnostic.** `specs/GameKernel.md` §5a carries the argument.
|
|||
|
|
|
|||
|
|
**The ratio has no monotone better direction.** INTENT's rule is *own the
|
|||
|
|
semantics; assimilate the implementation*. Rising can mean owning
|
|||
|
|
semantics properly, or reimplementing what should have been assimilated.
|
|||
|
|
Falling can mean good leverage, or dependency bloat. A target requires
|
|||
|
|
knowing which way is better, and this metric does not.
|
|||
|
|
|
|||
|
|
**It is also redundant:** AM-4a/AM-4b bound the denominator and AM-2 bounds
|
|||
|
|
own-source density per rule. AM-4c is a ratio of two already-targeted
|
|||
|
|
quantities.
|
|||
|
|
|
|||
|
|
Measured at withdrawal: **1,426** own lines per 100k third-party (shipped),
|
|||
|
|
**1,107** (dev). `make dep-weight` now prints both, labelled
|
|||
|
|
`diagnostic, not targeted` — it was never actually reported before.
|
|||
|
|
|
|||
|
|
**M-D1-MUT keeps AM-4c in its denominator, deliberately**, and says so in
|
|||
|
|
the output. Dropping it would move the score 7/14 → 7/13 without enforcing
|
|||
|
|
anything: a score improved by deleting the question.
|
|||
|
|
|
|||
|
|
### A T01 correction found here
|
|||
|
|
|
|||
|
|
**The AM-6 gate failed inside `make all` at 38,753 ev/s against 341,280
|
|||
|
|
measured in isolation — a 9× drop.** `cargo test` runs test binaries and
|
|||
|
|
threads concurrently, so **a throughput assertion inside a parallel
|
|||
|
|
harness measures contention, not throughput.** T01's measurement was valid
|
|||
|
|
and its gate placement was not.
|
|||
|
|
|
|||
|
|
Fixed by running the measurement only where it is valid — `#[ignore]`,
|
|||
|
|
plus `make am6` in release with `--test-threads=1`, now **2.0M ev/s,
|
|||
|
|
20.2× headroom** — and *not* by lowering the target, which T01 explicitly
|
|||
|
|
forbade and which would have reproduced the defect being fixed. My first
|
|||
|
|
attempt did drift that way (a debug "sanity floor" of 50,000) and was
|
|||
|
|
backed out: a second threshold is still a second chance to tune.
|
|||
|
|
|
|||
|
|
**And the mutation went `SURVIVED` on the first run after the move** —
|
|||
|
|
4,000 `black_box` iterations were calibrated against debug's 3.4×
|
|||
|
|
headroom and are invisible against release's 20×. Raised to 100,000; back
|
|||
|
|
to `red`. A weak mutation is not a fixed property of a row: **it can
|
|||
|
|
become weak when the row's measurement conditions change.**
|
|||
|
|
|
|||
|
|
**M-D1-MUT: 7 of 14** (unchanged — AM-4c was always going to stay
|
|||
|
|
uncounted; what changed is that the reason is now correct and recorded).
|