Two rules, two different answers, which is the point of a task phrased "implement, or amend and say why". K18 is implemented. "Criterion benches driving the same scenario format at scale" was false — the bench hardcoded its commands and never touched ScenarioFile, while MetricsAndScenarios §3 pointed at a benchmarks/ directory containing only baselines/. benchmarks/synthetic-3p.yaml now holds the workload and both the bench and bench_shape read it: the workload is data, not code. A second defect surfaced while fixing the first. After the bench switched to the file, bench_shape still hardcoded the same sequence, so the workload existed twice — deleting end_round from the YAML broke bench-test while bench_shape kept passing. Duplicated-fact drift in executable form. Both now read the same include_str! and deleting a command breaks both. Explicitly not claimed: this does not unblock AM-3. AM-3's baseline is a declarative game object — moves, turn order, rules. synthetic-3p.yaml is a command list; the rules live in games/ground. Marking it as AM-3's subject would compare a script to a game definition, which is the category error AM-3 is blocked on. The file says so in its own header, where the next person will be tempted. K14 is amended. CommitWindow had zero non-test users and GROUND enforces the same contract inline. Wiring GROUND through it was rejected: it would change the serialized shape of `selections`, which four scenario files assert by dot-path and every state hash depends on, for the sole benefit of making a sentence literally true. The deciding argument is INTENT's, not convenience: abstractions are extracted from working games rather than invented in isolation, and no concept becomes canonical until it survives a second concrete use. CommitWindow was invented before any game needed it and has survived none. Imposing it on GROUND would manufacture the first use rather than discover it. So K14 states what is actually guaranteed, CommitWindow is marked provisional in the source, and it carries a delete-by date of 2026-12-31. Kernel spec->code link 16/18 -> 18/18, stated with the caveat the gate prints every run: that is about names, not assertions. Two self-tests broke and both broke correctly. rule-coverage's gate test hardcoded "unlinked rules exist today" and failed when the last one was linked; it now computes that and asserts the gate fails iff rules are unlinked. facts' text check rejected k_unlinked once it became legitimately empty; empty now renders as "(none)" and the check distinguishes absent from empty. M-D1-MUT: 8 of 14, unchanged — K14 and K18 are kernel rules, not acceptance rows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
413 lines
19 KiB
Markdown
413 lines
19 KiB
Markdown
# CB-WP-0006 — delivery log
|
||
|
||
Per-task delivery notes, moved out of
|
||
[the workplan](../workplans/CB-WP-0006-instrument-the-table.md) when it
|
||
crossed the ~400-line loadability limit for the third time in this
|
||
project. The workplan holds the plan; this holds what happened. T08
|
||
draws on both.
|
||
|
||
## CB-WP-0006-T01
|
||
|
||
**Delivered.** `am6_throughput_clears_the_spec_target` in
|
||
`games/ground/src/lib.rs` — a test, not a bench. Best of 3 samples of
|
||
50,000 applied events each.
|
||
|
||
Measured on bnt-lap001 2026-07-31: **341,280 ev/s in debug (3.4× the
|
||
target)**, ~2.4–3.1M in release (~24–30×). **The spec target holds even in
|
||
an unoptimized build**, so the gate needed no `cfg` split and runs in the
|
||
ordinary `make test`.
|
||
|
||
**The trap was avoided by construction, not by intention.** The threshold
|
||
is the spec value `100_000`, untouched; the constant carries a comment
|
||
saying lowering it requires an ADR; and the failure message repeats that,
|
||
states the measured headroom, and names the reference figures — so a
|
||
future agent hitting a red AM-6 is told not to tune it, in the place they
|
||
will actually be reading. Robustness comes from **best-of-N**, not from a
|
||
lower bar: a throughput *floor* asks "is this machine capable", so
|
||
transient load should not fail the build.
|
||
|
||
Two positive controls in the test itself: a run that applied fewer than
|
||
50,000 events, or measured zero elapsed time, fails rather than scoring as
|
||
infinite throughput.
|
||
|
||
**Verified:** `make mutation-check --row AM-6` → **red**, via a *property*
|
||
mutation (4,000 `black_box` iterations injected into `GroundState::fold`,
|
||
the hot path) rather than a threshold tweak — raising the target would only
|
||
prove the comparison runs.
|
||
|
||
**And the FA class is now gated.** `mutation-check` rows gained an
|
||
`expect` field: the mutant's output must contain the row's stated failure
|
||
string, or the verdict is **`WRONG-REASON`**, not `red`. Without it, a
|
||
mutation that merely failed to compile would credit its row with an
|
||
assertion it does not have. Verified by pointing `expect` at a string the
|
||
verifier never prints and confirming the verdict flips. This is
|
||
remedy (2) from the CB-WP-0005 retrospective, built one task earlier than
|
||
T08 planned because the class it guards is the newest and the most
|
||
dangerous.
|
||
|
||
**M-D1-MUT: 4 → 5 of 14.**
|
||
|
||
## CB-WP-0006-T02
|
||
|
||
**Delivered — but the two rows resolved differently, and the difference is
|
||
the point.**
|
||
|
||
**AM-2 is instrumented and enforced.** `tools/size-metrics.py` +
|
||
`make size-metrics`, in `make all`:
|
||
|
||
```text
|
||
AM-2: 27.2 LOC/rule [ok target <= 40] (1.47x headroom)
|
||
1,575 code lines before the first #[cfg(test)] / 58 numbered rules
|
||
```
|
||
|
||
Tests are excluded because AM-2 asks what a rule *costs*, not how much it
|
||
is exercised — `lib.rs` is ~18% test code and including it would have
|
||
flattered the number. Verified red by a **property** mutation: ~800 lines
|
||
of filler injected into the impl, pushing the ratio past 40. `expect` is
|
||
the precise failure signature `FAIL target <= 40`, not the row name, which
|
||
would have matched passing output too.
|
||
|
||
**AM-3 is BLOCKED, not uninstrumented — and this is a finding, not a
|
||
deferral.** The row measures "LOC to express the CB-RES-0001 synthetic
|
||
game on our kernel" against a boardgame.io baseline of ~36 LOC for a
|
||
declarative **3p commit/reveal game object**. That artifact has never been
|
||
built: `games/` contains only `ground`, and `benches/synthetic.rs`
|
||
*drives* GROUND rather than *defining* a synthetic game.
|
||
|
||
Measuring GROUND's 1,575 impl lines against a 36-line synthetic game
|
||
object would compare **two different games** and call the difference a D1
|
||
result. So the tool ships the measurement mechanism — a marker-delimited
|
||
`// AM-3:BEGIN` / `// AM-3:END` region, self-tested — and **reports the row
|
||
blocked, naming the missing artifact**. A number here would have been
|
||
worse than a blank.
|
||
|
||
It therefore stays `unmutatable` and **still counts against M-D1-MUT**, per
|
||
ADR-0005 §1: a row that cannot fail asserts nothing, however good the
|
||
reason. Resolving it needs an artifact, not a metric tweak — carried
|
||
forward, not silently dropped.
|
||
|
||
**M-D1-MUT: 5 → 6 of 14.**
|
||
|
||
## CB-WP-0006-T03
|
||
|
||
**Delivered — `tools/runtime-metrics.py`. One passes, one breaches.**
|
||
|
||
**AM-9: met, and comfortably.** 13.4 MB peak RSS against a ≤64 MB target,
|
||
**4.8× headroom**. Gated and in `make all` (`--fast`, ~1 s). The row that
|
||
CB-EV-0001 called "very unlikely to bind" was right — but it is now
|
||
*measured* rather than assumed, and verified red by a property mutation
|
||
(a 300 MB allocation in the workload).
|
||
|
||
**AM-5: ~~BREACHED~~ MET — the first reading was wrong.**
|
||
|
||
> **Corrected 2026-08-01 (during T04).** T03 reported 87.0 s / 61.3 s and
|
||
> called AM-5 a 45% breach. **It is not.** Re-measured on a quiet machine
|
||
> with the fixed instrument:
|
||
>
|
||
> ```text
|
||
> load before measuring: 0.14 per CPU over 8 CPUs — quiet
|
||
> dev toolchain (default features) 37.3 s [ok target <= 60 s]
|
||
> best of 3: 37.3, 42.9, 46.2 (spread 1.24x)
|
||
> shipped runtime (--no-default-features) 41.2 s [ok target <= 60 s]
|
||
> best of 3: 41.2, 50.8, 54.2 (spread 1.32x)
|
||
> ```
|
||
>
|
||
> **1.6× headroom, comfortably met.** The 87.0 s was measured while the
|
||
> machine was busy running `mutation-check` and cargo builds — a timing
|
||
> measurement under contention measures the contention. That is the *same
|
||
> error class as AM-6's*, committed two tasks later in the same session,
|
||
> by the same author, in the row immediately after the one where it was
|
||
> diagnosed.
|
||
>
|
||
> The instrument now (a) reads `/proc` load average and **refuses to
|
||
> measure** above 0.5 per CPU, and (b) takes the **best of 3** — best, not
|
||
> worst, because a build-time *ceiling* asks "can this machine do it in
|
||
> 60 s", the mirror of AM-6's best-of-N for a throughput *floor*. It also
|
||
> warns when the spread exceeds 1.25×, which fired on the shipped-runtime
|
||
> samples: consecutive clean builds degrade (37.3 → 46.2), so even a quiet
|
||
> machine is not a uniform one.
|
||
|
||
The original, now-withdrawn finding follows for the record:
|
||
|
||
```text
|
||
dev toolchain (default features) 87.0 s [FAIL target <= 60 s]
|
||
shipped runtime (--no-default-features) 61.3 s [FAIL target <= 60 s]
|
||
```
|
||
|
||
The tool **reports and exits 0**, because the spec says the row is not
|
||
gated (and on the corrected number there is nothing to escalate anyway). Gating it is a spec change and needs an ADR; a tool that promotes
|
||
itself is how a target starts binding without anyone deciding it should.
|
||
So AM-5 stays `unmutatable` — it cannot fail while the spec declares it
|
||
ungated, which is the accurate reason. **The escalation is withdrawn**:
|
||
there is no breach to decide about.
|
||
|
||
A build profile was investigated anyway while the breach was believed
|
||
real, and the findings stand on their own: 51 compile units, 174 s of CPU
|
||
work compressed into ~54 s wall, **parallelism only 3.2× on 8 cores**
|
||
because `proc-macro2 → syn → serde_derive → serde_core` is a ~22 s serial
|
||
chain. `lto = "thin"` costs ~6 s (55.8 → 49.4 s mean) and is the only real
|
||
lever; pinning `ppv-lite86` down to drop `zerocopy` (the largest single
|
||
unit, 21.8 s) makes it **worse** — 23 → 25 crates, because the older
|
||
version pulls `zerocopy-derive`. Recorded here rather than acted on: with
|
||
1.6× headroom there is nothing to buy.
|
||
|
||
**The measurement had a real bug, found by cross-validation.**
|
||
`getrusage(RUSAGE_CHILDREN)` is a high-water mark across *every* reaped
|
||
child, so it attributed `cargo`'s memory to the workload and reported
|
||
**38.2 MB for a run that used 12.3 MB** — a 3× over-report. Fixed with
|
||
`os.wait4`, which returns that specific child's rusage. The self-test now
|
||
**cross-checks against `/usr/bin/time -v`** (13.4 vs 12.4 MB), which is
|
||
the only reason the bug was visible at all: the wrong number was
|
||
plausible, passed its target, and would have been published.
|
||
|
||
That is FA in the measurement layer rather than the mutation layer — an
|
||
instrument confidently reporting a number it had not earned.
|
||
|
||
**M-D1-MUT: 6 → 7 of 14.**
|
||
|
||
## CB-WP-0006-T04
|
||
|
||
**Delivered: withdrawn from the acceptance table, retained as a
|
||
diagnostic.** `specs/GameKernel.md` §5a carries the argument.
|
||
|
||
**The ratio has no monotone better direction.** INTENT's rule is *own the
|
||
semantics; assimilate the implementation*. Rising can mean owning
|
||
semantics properly, or reimplementing what should have been assimilated.
|
||
Falling can mean good leverage, or dependency bloat. A target requires
|
||
knowing which way is better, and this metric does not.
|
||
|
||
**It is also redundant:** AM-4a/AM-4b bound the denominator and AM-2 bounds
|
||
own-source density per rule. AM-4c is a ratio of two already-targeted
|
||
quantities.
|
||
|
||
Measured at withdrawal: **1,426** own lines per 100k third-party (shipped),
|
||
**1,107** (dev). `make dep-weight` now prints both, labelled
|
||
`diagnostic, not targeted` — it was never actually reported before.
|
||
|
||
**M-D1-MUT keeps AM-4c in its denominator, deliberately**, and says so in
|
||
the output. Dropping it would move the score 7/14 → 7/13 without enforcing
|
||
anything: a score improved by deleting the question.
|
||
|
||
### A T01 correction found here
|
||
|
||
**The AM-6 gate failed inside `make all` at 38,753 ev/s against 341,280
|
||
measured in isolation — a 9× drop.** `cargo test` runs test binaries and
|
||
threads concurrently, so **a throughput assertion inside a parallel
|
||
harness measures contention, not throughput.** T01's measurement was valid
|
||
and its gate placement was not.
|
||
|
||
Fixed by running the measurement only where it is valid — `#[ignore]`,
|
||
plus `make am6` in release with `--test-threads=1`, now **2.0M ev/s,
|
||
20.2× headroom** — and *not* by lowering the target, which T01 explicitly
|
||
forbade and which would have reproduced the defect being fixed. My first
|
||
attempt did drift that way (a debug "sanity floor" of 50,000) and was
|
||
backed out: a second threshold is still a second chance to tune.
|
||
|
||
**And the mutation went `SURVIVED` on the first run after the move** —
|
||
4,000 `black_box` iterations were calibrated against debug's 3.4×
|
||
headroom and are invisible against release's 20×. Raised to 100,000; back
|
||
to `red`. A weak mutation is not a fixed property of a row: **it can
|
||
become weak when the row's measurement conditions change.**
|
||
|
||
**M-D1-MUT: 7 of 14** (unchanged — AM-4c was always going to stay
|
||
uncounted; what changed is that the reason is now correct and recorded).
|
||
|
||
## CB-WP-0006-T05
|
||
|
||
**Delivered: K9's real assertion, K11's durable format, the `LogStore`
|
||
port, and the shared conformance suites that finally earn AM-11.**
|
||
|
||
**K11 — `crates/cb-events/src/store.rs`.** Magic + version header, 4-byte
|
||
little-endian length prefix per record, append-only. Reimplemented, not
|
||
assimilated (ADR-0005 §2): no new dependency, charged to AM-4a, and
|
||
AM-4a/AM-4b are unchanged at 246,250 / 317,021 because nothing was added
|
||
to the graph.
|
||
|
||
The operative clause is **detected**, so corruption is tested, not assumed:
|
||
a tail short by one byte, a half-written length prefix, a length prefix
|
||
corrupted to claim more than the file holds, foreign magic, and a future
|
||
format version are each rejected with a distinct error. A reader that
|
||
accepts a truncated tail is worse than no format, because it silently
|
||
returns a short history that looks complete.
|
||
|
||
**The port and the suite (AM-11).** `LogStore` has two impls —
|
||
`MemLogStore` and `FileLogStore` — and **one** `conformance()` that both
|
||
are driven through. The trait carries `raw`/`set_raw` specifically so the
|
||
corruption controls live in the *shared* suite: a format contract only one
|
||
impl enforces is not a contract.
|
||
|
||
The same shape was retro-fitted to `KernelRng`, which is what AM-11
|
||
actually names. `ChaChaRng` and `NullRng` now pass one
|
||
`conformance()` asserting the properties true of both — bounds,
|
||
`draw(1) == 0`, determinism across fresh instances, and shuffle preserving
|
||
the multiset. Previously they were exercised by two *separate* tests,
|
||
which is precisely why `AM-11 | met, narrow` was never earned and ADR-0005
|
||
§4 downgraded it.
|
||
|
||
**K9 — the assertion it did not have.** *Snapshot at seq N + events
|
||
N+1..M ≡ genesis fold*, hash-compared, on `GroundState`, single-seed on
|
||
purpose (AM-7's probe folds a multi-seed log, which is not a replay of
|
||
anything; that defect is not repeated). Two positive controls: the log
|
||
must exceed 50 events, and the mid-log snapshot must **differ** from the
|
||
end state, or "apply the remainder" would be vacuous.
|
||
|
||
**Proof it works:** the exact mutation that *survived* in CB-WP-0005 —
|
||
making `Snapshot::take` discard its `EventSeq` — now fails:
|
||
|
||
```text
|
||
assertion `left == right` failed: K9: the snapshot must carry the EventSeq it includes
|
||
test result: FAILED. 0 passed; 1 failed
|
||
```
|
||
|
||
**AM-11 mutation:** break `NullRng::draw` to return its bound and the
|
||
*shared* suite fails. That is what M-D4-SWAP claims — that either impl can
|
||
be substituted — and it is exactly what two separate per-impl tests could
|
||
never demonstrate.
|
||
|
||
**M-D1-MUT: 7 → 8 of 14.** `evidence/CB-EV-0001`'s scoreboard is refreshed
|
||
with AM-2, AM-5 and AM-9 added, AM-6 moved to enforced, and the headline
|
||
total corrected from 4 to 8 — it had gone stale within the same workplan
|
||
that produced it.
|
||
|
||
## CB-WP-0006-T06
|
||
|
||
**Delivered: K10 replay bundles, `cb-sim --replay`, and AM-7's withdrawn
|
||
hash clause re-earned.**
|
||
|
||
INTENT design decision 8 of 10, unimplemented for six passes. `cb-sim` had
|
||
no flag parsing at all, so `--replay` had nowhere to go.
|
||
|
||
**The bundle** (`crates/cb-game-runtime/src/replay.rs`, dev-only behind
|
||
`scenarios`, charged to AM-4b): manifest, `commands.log`,
|
||
`initial.snapshot`, `expected.yaml`. The command stream goes through the
|
||
**K11 framing built in T05**, so a truncated bundle is detected rather
|
||
than replayed short — the two tasks compose rather than duplicating.
|
||
|
||
**The D2 correction the reviewer forced was real.** This was not "a
|
||
directory of four files": `Pass` carried only the *end* state, `RunOutcome::Failed`
|
||
was a formatted `String`, and `scenario.rs` created an `EventLog`,
|
||
appended to it and never read it. All three had to change.
|
||
|
||
### The bug that proves the round trip is load-bearing
|
||
|
||
The first round trip **failed to reproduce**. Cause: `state_hash_hex` over
|
||
a `serde_json::Value` is a **different canonical form** than over the
|
||
typed aggregate — `Value`'s map is key-sorted, a struct serializes in
|
||
declaration order. The bundle was written with one basis and verified with
|
||
the other.
|
||
|
||
Worth stating plainly: a round trip that had been written to recompute its
|
||
own comparison value would have **passed** this bug. It failed precisely
|
||
because the recorded hash came from the producing process — control 2,
|
||
which exists for exactly this.
|
||
|
||
### The four controls, all executable
|
||
|
||
`make replay-test`, **14/14**:
|
||
|
||
| control | how it is proven |
|
||
|---|---|
|
||
| something actually fails | a committed fixture in `scenarios/fixtures/` (outside the corpus, `covers: []`, so it neither fails `make sim` nor inflates AM-1); exactly one bundle produced; all four files present **and non-empty** |
|
||
| hash read from the bundle | tamper only the recorded hash → replay fails. A recomputed comparison would still pass |
|
||
| truncation detected | log short by one byte → rejected; length prefix corrupted → rejected |
|
||
| replay can fail | mutate the manifest seed → rejected. This bites only because replay re-derives the initial state from seed+setup and checks it against the recorded snapshot; restoring from the snapshot alone would leave the seed inert |
|
||
|
||
Plus a control on the controls: the bundle must still replay after every
|
||
mutation is reverted, so the suite cannot leave the tree broken.
|
||
|
||
### AM-7 re-earned
|
||
|
||
The probe now records a hash per **per-game segment** and replays each
|
||
from its own genesis. Folding a segment from the wrong seed fails:
|
||
|
||
```text
|
||
AM-7 hash-identical UNMET: replaying segment seeded 42 (0..64) did not
|
||
reproduce its recorded state hash
|
||
```
|
||
|
||
That is the clause ADR-0005 §4 withdrew as mutation-proven inert. The
|
||
`scaling >= 0.9x` clause is still unenforced, so AM-7 stays **PARTIAL** —
|
||
reported, not rounded up.
|
||
|
||
**Kernel coverage 15/18 → 16/18**; K10 is no longer unlinked. `facts-check`
|
||
immediately caught the spec's copy of that number going stale — the DFD
|
||
gate earning its place on a number that moved the same hour.
|
||
|
||
**M-D1-MUT: 8 of 14** (unchanged; AM-7 remains partial).
|
||
|
||
## CB-WP-0006-T07
|
||
|
||
**Two rules, two different answers — which is the point of a task phrased
|
||
"implement, or amend and say why".**
|
||
|
||
### K18: implemented
|
||
|
||
"Criterion benches driving the same scenario format at scale" was false —
|
||
the bench hardcoded its command sequence in Rust and never touched
|
||
`ScenarioFile`, while MetricsAndScenarios §3 pointed at a `benchmarks/`
|
||
directory containing only `baselines/`.
|
||
|
||
`benchmarks/synthetic-3p.yaml` now holds the workload, and both the bench
|
||
and `bench_shape` read it. **The workload is data, not code.**
|
||
|
||
**A second defect surfaced while fixing the first.** After the bench
|
||
switched to the file, `bench_shape` still hardcoded the same sequence — so
|
||
the workload existed **twice**, and deleting `end_round` from the YAML
|
||
broke `bench-test` while `bench_shape` kept passing. That is
|
||
duplicated-fact drift in executable form. Both now read the same
|
||
`include_str!`, and deleting a command breaks both:
|
||
|
||
```text
|
||
assertion `left == right` failed: commands per synthetic round
|
||
```
|
||
|
||
**Explicitly not claimed:** this does *not* unblock AM-3. AM-3's baseline
|
||
is a declarative **game object** (~36 LOC, boardgame.io) — moves, turn
|
||
order, rules. `synthetic-3p.yaml` is a command list; the rules live in
|
||
`games/ground` (1,575 lines). Marking it as AM-3's subject would compare a
|
||
script to a game definition, which is the category error AM-3 is blocked
|
||
on. The file says so in its own header, where the next person will be
|
||
tempted.
|
||
|
||
### K14: amended, and the reasoning is recorded because it could have gone
|
||
the other way
|
||
|
||
`CommitWindow` had **zero non-test users**; GROUND enforces the same
|
||
contract inline. Wiring GROUND through it was rejected: it would change
|
||
the serialized shape of `selections`, which four scenario files assert by
|
||
dot-path and every state hash depends on — a large, risky refactor whose
|
||
only benefit is making a sentence literally true.
|
||
|
||
The deciding argument is INTENT's, not convenience: *"abstractions are
|
||
extracted from working games... rather than invented in isolation"*, and
|
||
*"No concept becomes canonical merely because it looks general. It becomes
|
||
canonical after surviving a second concrete use."* `CommitWindow` was
|
||
invented before any game needed it and has survived **zero** uses.
|
||
Imposing it on GROUND would **manufacture** the first use rather than
|
||
discover it.
|
||
|
||
So K14 now states what is actually guaranteed, `CommitWindow` is marked
|
||
**provisional in the source**, and it carries a **delete-by date of
|
||
2026-12-31** if no second game uses it. A primitive with one hypothetical
|
||
user and a test that exercises only itself is the AM-11 shape, and this
|
||
project has now paid for that shape twice.
|
||
|
||
### Consequences
|
||
|
||
**Kernel spec→code link: 16/18 → 18/18 (100%).** Stated with the caveat
|
||
the gate prints on every run: that is a statement about **names**, not
|
||
assertions.
|
||
|
||
Two self-tests broke and both broke *correctly*:
|
||
|
||
* `rule-coverage`'s gate test hardcoded "unlinked rules exist today" and
|
||
failed the moment the last one was linked. It now computes whether
|
||
anything is unlinked and asserts the gate fails **iff** so — the
|
||
assertion was right, its premise was transient.
|
||
* `facts`' "every fact records a value and its rendered text" rejected
|
||
`k_unlinked` once it became legitimately empty. Empty now renders as
|
||
`(none)`, and the check distinguishes *absent* from *empty*.
|
||
|
||
**M-D1-MUT: 8 of 14, unchanged.** K14 and K18 are kernel rules, not
|
||
acceptance rows, so neither moves that number — worth stating, because the
|
||
temptation is to imply progress on a metric this task never touched.
|