11 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| 0b6f7c5bc8 |
CB-WP-0019 T01/T02: AM-4b asks what a contributor acquires
Some checks failed
ci / check (push) Failing after 4s
The two AM-4 budgets had the SAME scope -- one package, no dev edges -- while claiming to bound different things. AM-4b now measures the workspace with dev edges: 57 crates / 725,258 lines where it read 29 / 317,021, having been blind to 28 crates and 408,237 lines, more source than its own target. Target 745,000, ~2.7% of room -- the same margin ADR-0008 D3 gave AM-4a, applied to a number that grew because the instrument was repaired, not because anything was added. The target moved to fit the measurement. T02: proc-macros are COUNTED here and excluded from AM-4a, on purpose. AM-4a asks what ships and a proc-macro never ships. AM-4b asks what is acquired, and ADR-0007 D3's acquisition rule counts what the build fetches -- 'it does not ship' is no answer to 'we downloaded it'. When the rules disagree, the question each budget asks decides. Measured share 109,585 lines / 15.1% against AM-4a's 36.2%, so ADR-0008 D2's refusal to borrow the ratio was right by more than a factor of two. Caught by this project's own earlier work twice: the mutation find-string went stale and --self-test reported it BUILD-FREE (the check CB-WP-0015 added after AM-4a's rotted for two passes), then the DFD gate caught facts.toml carrying the old numbers. CB-EV-0001 and ADR-0004 carried live fact: tags on historical readings. A dated record asserting a CURRENT value is a category error, so those occurrences are marked as-measured instead of retro-edited, and ADR-0004 gains a supersession note. make all exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
|||
| ee37b82675 |
CB-WP-0015: the two inert clauses, AM-7 scaling and AM-8 N=10
Some checks failed
ci / check (push) Failing after 3s
Provenance (tier S, one paragraph in lieu of survey and ADR): the two clauses mutation-check has reported inert since CB-WP-0005. AM-7's scaling ratio was held up by a test literally named replay_100k_events_is_linear_and_fast that computed both throughputs, printed both, and never divided one by the other. AM-8's N=10 was held up by a runner that does two. Both are now red. AM-7 3/3, AM-8 2/2, M-D1-MUT 10/14, and ADR-0005's >=10-of-14 prediction MET for the first time. Neither was closed by amending the question away, which was the live risk: the denominator is unchanged and the four unenforced rows are the four already unenforceable. AM-7 needed three estimators. Best-of-N per leg then divide (AM-6's, correct for a floor on one number) gave 0.581-1.085 on an unchanged binary; legs back-to-back gave medians 0.931-1.004; legs interleaved at fold granularity give 0.987/0.991/0.989, and 0.989 under 8-way CPU contention while absolute throughput fell 4x. The INDETERMINATE guard demanded unanimity and failed a good measurement over one sample 0.001 under the floor; it now requires a two-thirds majority. The control that matters: AM-6's constant-cost mutation halves throughput and leaves this ratio at 0.999x green, so AM-7 is not a second AM-6. AM-8 kept N=10 because the measurement said so. Perturbing the RNG only from its fourth construction on: --runs 2 PASSES, --runs 10 fails. A late-onset divergence is deterministic, not flaky, so it is a control rather than a coin flip. Ten runs live on one scenario (make am8, ~2s) rather than all 25 (47s a build). GameKernel 5b records it. The full run also found AM-4a's own mutation stale since ADR-0008 D3 moved the target 250,000 -> 161,000 in CB-WP-0013 -- reported HARNESS-BROKEN, no score published. The build-free half of that check is now a --self-test assertion, so make all catches the next one. mutation-check clauses may now carry their own verify and mutation, and then the enforced flag is measured rather than declared; a declaration disagreeing with its measurement is refused. make all exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
|||
| ce353adde8 |
CB-WP-0006 T08: control loop — 4 of 14 to 8 of 14, and a cost regression
Test 1: the enforced count rose. AM-2, AM-6, AM-9 and AM-11 moved from unmutatable to red; AM-7's hash clause was re-earned so it is 2/3 rather than 1/3. Kernel spec->code link 15/18 -> 18/18, names only. Both denominators are stated. 8 of 14 is 57%, but four rows cannot be enforced — AM-3 blocked on an artifact, AM-4c withdrawn, AM-5 declared ungated by the spec, AM-10 withdrawn — so it is 8 of 10 enforceable. The 14 stays the headline and AM-4c stays in it on purpose: a score improved by deleting the question is not an improvement. Test 2: one row regressed and was caught. Moving AM-6's gate from debug to release turned its mutation SURVIVED, because 4,000 black_box iterations were calibrated against debug's 3.4x headroom and are invisible against release's 20x. The generalizable finding is that a weak mutation is not a fixed property of a row — it can become weak when the row's measurement conditions change, without the row, the mutation or the code being touched. Final SURVIVED count: 0. Test 3 is now mechanical rather than asserted. mutation-check gained an EXPECT-VACUOUS verdict: if a row's expect string appears in PASSING output, the FA guard would accept any failure at all, so the row is reported broken rather than red. Final run: 0 vacuous expects across 14 rows. The control exists because the failure happened — my first expect for AM-2 was "AM-2", which appears in the passing report and would have accepted a compile error as proof of enforcement. The cost result is a refutation, not a win. Mechanical share rose to 50%, the highest ever recorded and above the 38% baseline that motivated CB-WP-0004. That is not a tooling regression: environment setup and task closes are still at zero two passes on. It is the other half of CB-WP-0004 T06's finding arriving in force — text patching (45 turns, $13.79) and orientation (19 turns, $10.97) never had their manual path removed, and a code-heavy pass is exactly where that spends. Mean context 493,486 against a 200,000 target, up from 315,170. SessionShape has stated SS-01..SS-05 since CB-WP-0003 and none has ever been enforced — the only acceptance-adjacent numbers in this project with no gate at all, in a pass whose entire subject was ungated numbers. Numbering corrected: the workplan said CB-EV-0004, which CB-WP-0005 already used. This is CB-EV-0005. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
|||
| 1edadac9a2 |
CB-WP-0006 T06: K10 replay bundles, --replay, and AM-7 re-earned
INTENT design decision 8 of 10, unimplemented for six passes. cb-sim had no flag parsing at all, so --replay had nowhere to go. The bundle is manifest + commands.log + initial.snapshot + expected.yaml, dev-only behind the scenarios feature and charged to AM-4b. The command stream goes through the K11 framing built in T05, so a truncated bundle is detected rather than replayed short — the two tasks compose rather than duplicating. The reviewer's D2 correction was real: this was not "a directory of four files". Pass carried only the end state, RunOutcome::Failed was a formatted String, and scenario.rs created an EventLog, appended to it and never read it. All three had to change. The first round trip failed to reproduce, and the cause is worth keeping: state_hash_hex over a serde_json::Value is a different canonical form than over the typed aggregate — Value's map is key-sorted, a struct serializes in declaration order. The bundle was written with one basis and verified with the other. A round trip written to recompute its own comparison value would have PASSED this bug; it failed because the recorded hash came from the producing process, which is control 2's entire purpose. make replay-test implements ADR-0005 §6's four controls, 14/14: a committed deliberately-failing fixture outside the corpus with covers: [] so it neither fails `make sim` nor inflates AM-1; a tampered recorded hash must fail; a log short by one byte and a corrupted length prefix must be rejected; and a mutated manifest seed must fail — which bites only because replay re-derives the initial state from seed+setup and checks it against the recorded snapshot, since restoring from the snapshot alone would leave the seed inert. Plus a control on the controls: the bundle must still replay after every mutation is reverted. AM-7's hash-identical clause is re-earned. The probe records a hash per per-game segment and replays each from its own genesis; folding from the wrong seed now fails. That is the clause ADR-0005 §4 withdrew as mutation-proven inert. The scaling >= 0.9x clause is still unenforced, so AM-7 stays PARTIAL — reported, not rounded up. Kernel coverage 15/18 -> 16/18. facts-check immediately caught the spec's copy of that number going stale, on a number that moved the same hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
|||
| 98c6cd24c3 |
CB-WP-0006 T05: K9's assertion, K11's format, and the AM-11 suites
K11 is implemented: crates/cb-events/src/store.rs, magic + version header, 4-byte little-endian length prefix, append-only. Reimplemented not assimilated per ADR-0005 §2 — no new dependency, and AM-4a/AM-4b are unchanged at 246,250 / 317,021 because nothing entered the graph. The operative clause is "detected", so corruption is tested rather than assumed: a tail short by one byte, a half-written length prefix, a length prefix corrupted to claim more than the file holds, foreign magic, and a future format version are each rejected with a distinct error. A reader that accepts a truncated tail is worse than no format, because it silently returns a short history that looks complete. AM-11 is earned. LogStore has two impls — MemLogStore and FileLogStore — driven through ONE conformance(). The trait carries raw/set_raw precisely so the corruption controls live in the shared suite: a format contract that only one impl enforces is not a contract. The same shape is retro-fitted to KernelRng, which is what AM-11 actually names: ChaChaRng and NullRng now pass one suite asserting bounds, draw(1) == 0, determinism across fresh instances, and shuffle preserving the multiset. They were previously exercised by two separate tests, which is why "met, narrow" was never earned and ADR-0005 §4 downgraded it. K9 gets the assertion it did not have: snapshot at seq N + events N+1..M must equal the from-genesis fold, hash-compared, on GroundState, single-seed on purpose — AM-7's probe folds a multi-seed log, which is not a replay of anything, and that defect is not repeated. Two positive controls: the log must exceed 50 events, and the mid-log snapshot must differ from the end state or "apply the remainder" is vacuous. Proof it works: the exact mutation that SURVIVED in CB-WP-0005 — making Snapshot::take discard its EventSeq — now fails on the K9 assertion. AM-11's mutation breaks NullRng::draw to return its bound and the shared suite fails. That is what M-D4-SWAP claims — either impl substitutable — and exactly what two separate per-impl tests could never demonstrate. M-D1-MUT: 7 -> 8 of 14. CB-EV-0001's scoreboard is refreshed: AM-2, AM-5 and AM-9 added, AM-6 moved to enforced, and the headline total corrected from 4 to 8 — it had gone stale inside the same workplan that produced it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
|||
| 5f7d9015d9 |
Fix the AM-5 instrument to measure quietly; the breach was not real
Some checks failed
ci / check (push) Failing after 3s
T03 reported AM-5 at 87.0 s / 61.3 s and called it a 45% breach of the
60 s target. Re-measured with the fixed instrument on a quiet machine:
load before measuring: 0.14 per CPU over 8 CPUs — quiet
dev toolchain (default features) 37.3 s [ok target <= 60 s]
best of 3: 37.3, 42.9, 46.2 (spread 1.24x)
shipped runtime (--no-default-features) 41.2 s [ok target <= 60 s]
best of 3: 41.2, 50.8, 54.2 (spread 1.32x)
AM-5 is MET with 1.6x headroom. The 87.0 s was measured while the machine
was busy with mutation-check and cargo builds — a timing measurement under
contention measures the contention.
That is the same error class as AM-6's, committed two tasks later in the
same session by the same author, in the row immediately after the one
where it was diagnosed. Knowing the failure mode did not prevent it; only
building the guard did. That is the InnerLoop v1.2 design-goal argument
holding up under a third instance: optimize for cheap correction, because
prevention keeps not converging.
The instrument now refuses to measure above 0.5 load per CPU, takes the
best of 3, and warns when the spread exceeds 1.25x. Best, not worst: a
build-time ceiling asks whether the machine can do it in 60 s, the mirror
of AM-6's best-of-N for a throughput floor. The spread warning fired on
the shipped-runtime samples — consecutive clean builds degrade 37.3 ->
46.2 — so a quiet machine is not a uniform one either.
The escalation to a maintainer decision is withdrawn: there is no breach.
The build profiling done while the breach was believed real is recorded in
the log rather than acted on — 174 s of CPU work at only 3.2x parallelism
on 8 cores, a ~22 s serial proc-macro chain, lto=thin worth ~6 s, and
pinning ppv-lite86 to drop zerocopy making it worse (23 -> 25 crates).
With 1.6x headroom there is nothing to buy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|||
| db6445ae37 |
CB-WP-0006 T04: withdraw AM-4c; and fix where AM-6 is measured
AM-4c is withdrawn from the acceptance table and retained as a reported diagnostic. GameKernel §5a carries the argument. The ratio has no monotone better direction. INTENT's rule is "own the semantics, assimilate the implementation": rising can mean owning semantics properly or reimplementing what should have been assimilated; falling can mean leverage or dependency bloat. A target requires knowing which way is better. It is also redundant — AM-4a/AM-4b bound the denominator and AM-2 bounds own-source density, so AM-4c is a ratio of two already-targeted quantities. Measured at withdrawal: 1,426 own lines per 100k third-party (shipped), 1,107 (dev). make dep-weight now prints both, labelled diagnostic — the row was never actually reported before. M-D1-MUT keeps AM-4c in its denominator on purpose and says so in the output. Dropping it would move the score 7/14 -> 7/13 without enforcing anything: a score improved by deleting the question. Decided before Phase B deliberately, since ADR-0005 predicts own-source growth that will move this ratio; deciding after would be the retarget §Step 4 forbids. A T01 correction found here. The AM-6 gate failed inside `make all` at 38,753 ev/s against 341,280 in isolation — a 9x drop, because cargo test runs binaries and threads concurrently. A throughput assertion inside a parallel harness measures contention, not throughput. T01's measurement was valid; its gate placement was not. Fixed by running it only where valid — #[ignore] plus `make am6` in release with --test-threads=1, now 2.0M ev/s at 20.2x headroom — and not by lowering the target, which T01 forbade. My first attempt did drift that way, adding a debug "sanity floor" of 50,000, and was backed out: a second threshold is still a second chance to tune. The mutation then went SURVIVED on the first run after the move. 4,000 black_box iterations were calibrated against debug's 3.4x headroom and are invisible against release's 20x. Raised to 100,000; back to red. A weak mutation is not a fixed property of a row — it can become weak when the row's measurement conditions change. Tier S (amends one row, creates no capability), chaos d4=2, no override. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
|||
| e7312b3b8a |
CB-WP-0006 T03: measure AM-5 and AM-9; AM-5 breaches
AM-9 is met and gated: 13.4 MB peak RSS against a 64 MB target, 4.8x headroom, in `make all` via --fast. CB-EV-0001's "very unlikely to bind" was right, but it is now measured rather than assumed, and verified red by a property mutation (a 300 MB allocation in the workload). AM-5 is BREACHED on both readings, on the machine the spec names: dev toolchain (default features) 87.0 s [FAIL target <= 60 s] shipped runtime (--no-default-features) 61.3 s [FAIL target <= 60 s] bnt-lap001, 8 cores — a direct comparison, not a directional one. A row declared "recorded not gated" and never recorded fails its own target by 45% on first measurement. The tool reports and exits 0 because the spec says the row is ungated. Gating it is a spec change needing an ADR; a tool that promotes itself is how a target starts binding without anyone deciding it should. So AM-5 stays unmutatable — for the accurate reason now — and the breach is raised as a maintainer decision: speed the build, move the target by ADR (arguing why 60 s was wrong rather than why 87 s is convenient), or withdraw the row. The measurement itself had a real bug, found only by cross-validation. getrusage(RUSAGE_CHILDREN) is a high-water mark across every reaped child, so it attributed cargo's memory to the workload and reported 38.2 MB for a run that used 12.3 MB — a 3x over-report that was plausible, passed its target, and would have been published. Fixed with os.wait4, which returns that specific child's rusage, and the self-test now cross-checks against /usr/bin/time -v. That is the false-accusation shape in the measurement layer rather than the mutation layer: an instrument confidently reporting a number it had not earned. Also: the clean build measures into a throwaway CARGO_TARGET_DIR rather than running `cargo clean`, so measuring the metric does not cost several minutes of rebuild afterwards. A metric that punishes its own measurement gets measured once and never again. M-D1-MUT: 6 -> 7 of 14. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
|||
| 1c3019c7e8 |
CB-WP-0006 T02: instrument AM-2; report AM-3 blocked, with the argument
Both rows were unmutatable for the same stated reason. They resolved
differently, and the difference is the point.
AM-2 is instrumented and enforced — tools/size-metrics.py, in `make all`:
AM-2: 27.2 LOC/rule [ok target <= 40] (1.47x headroom)
1,575 impl lines / 58 rules
Tests are excluded because AM-2 asks what a rule costs, not how much it is
exercised; lib.rs is ~18% test code and including it would have flattered
the number. This matters because AM-2 is AM-1's anti-gaming pair: 100%
rule coverage means nothing if the rules are trivially small, and AM-1 has
been reported met since CB-WP-0001 with its pair uninstrumented.
Verified red by a property mutation — ~800 lines of filler injected into
the impl, pushing the ratio past 40 — not a threshold tweak. The expect
string is the precise failure signature "FAIL target <= 40"; my first
attempt used "AM-2", which also matches passing output and would have
made the FA guard vacuous.
AM-3 is BLOCKED, not uninstrumented, and that is a finding rather than a
deferral. It measures LOC to express the CB-RES-0001 synthetic game on our
kernel, against a boardgame.io baseline of ~36 LOC for a declarative 3p
commit/reveal game object. That artifact has never been built: games/
contains only ground, and benches/synthetic.rs drives GROUND rather than
defining a synthetic game. Measuring GROUND's 1,575 impl lines against a
36-line synthetic game object would compare two different games and call
the difference a D1 result.
So the tool ships the measurement — a marker-delimited region, self-tested
— and reports the row blocked, naming the missing artifact. A number would
have been worse than a blank. It stays unmutatable and still counts
against M-D1-MUT per ADR-0005 §1: a row that cannot fail asserts nothing,
however good the reason.
M-D1-MUT: 5 -> 6 of 14.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|||
| c43754f0fe |
CB-WP-0006 T01: assert the AM-6 throughput target
Nothing in the workspace compared any number to 100,000 events/s while the evidence file reported "AM-6 | met, 16.5x". Now a test does — a test, not a bench, because Criterion reports throughput and asserts nothing, which is why this row measured nothing for six passes. Measured on bnt-lap001: 341,280 ev/s in debug (3.4x the target), ~2.4-3.1M in release. The spec target holds even in an unoptimized build, so the gate needs no cfg split and runs in the ordinary `make test`. The trap this task named — loosening a flaky timing assertion until it never fires — is avoided by construction. The threshold is the spec value, untouched; the constant says lowering it requires an ADR; and the failure message repeats that, states measured headroom, and names reference figures, so an agent hitting a red AM-6 is told not to tune it in the place they are actually reading. Robustness comes from best-of-N, not from a lower bar: a throughput floor asks whether the machine is capable, so transient load should not fail the build. Two positive controls in the test: a run that applied fewer than 50,000 events, or measured zero elapsed time, fails rather than scoring as infinite throughput. Verified by a PROPERTY mutation — 4,000 black_box iterations injected into GroundState::fold, the hot path — not a threshold tweak, which would only prove the comparison runs. And the FA class found last pass is now gated. mutation-check rows gained an `expect` field: the mutant's output must contain the row's stated failure string or the verdict is WRONG-REASON, not red. Without it a mutation that merely failed to compile would credit its row with an assertion it does not have. Verified by pointing expect at a string the verifier never prints and watching the verdict flip. This is remedy (2) from the CB-WP-0005 retrospective, built a task earlier than planned because the class it guards is the newest and most dangerous. M-D1-MUT: 4 -> 5 of 14. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
|||
| e25bac3af3 |
CB-WP-0005 T02: M-D1-MUT — 4 of 14 acceptance rows are enforced
For each acceptance row in GameKernel §5, invert the property and require
the verifying command to go red. adapted:mutation-testing, with the
denominator changed from source lines to acceptance rows.
M-D1-MUT: 4/14 rows enforced
PARTIAL 2 (AM-7, AM-8 — some clauses live, some inert)
unmutatable 8 (no property to invert, reason stated per row)
SURVIVED 0
Two corrections to our own numbers. There are 14 rows, not the twelve
ADR-0005 and CB-WP-0005 both asserted — AM-4 splits into a/b/c. And the
prediction of 9-of-12 (75%) becomes >=10 of 14; measured 4 (29%), badly
unmet. No target moved in this commit.
The second correction matters more. My first run reported two SURVIVED
rows and both were my own no-op mutations: `pub struct NullRng;` ->
`pub struct NullRng {}` is semantically identical, and renaming
max_age_days does nothing because CA-17 reads it with a default of 90.
Both would have been published as "this row asserts nothing" — a false
accusation against code that is fine. Replaced with real inversions (a
per-construction counter in the ChaCha seed; reverting AC-9's output
resolution to the first-wins bug it was fixed for), after which both go
red. T08 asks whether writing a weak mutation is the new grep. It is,
demonstrably, on the first attempt.
The finding is larger than the workplan assumed. 8 of 14 rows are
unmutatable — AM-2, AM-3, AM-4c, AM-5, AM-6, AM-9, AM-10, AM-11 have no
instrument at all. AM-6 is the sharpest: nothing in the workspace
compares any number to 100,000 events/s, the headline throughput claim.
The problem is not three unimplemented rules, it is that more than half
the acceptance table has nothing behind it.
Harness controls: a stale find-string reports HARNESS-BROKEN rather than
scoring the baseline as the mutant; a red baseline reports inconclusive
rather than red; the tree is restored in a finally and the restoration is
verified. Not in `make all` — it rebuilds the workspace once per row.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
