From 1bbbfc846fea94173b7e369124867832508ffca4 Mon Sep 17 00:00:00 2001 From: tegwick Date: Fri, 31 Jul 2026 14:25:26 +0200 Subject: [PATCH] ADR-0005 + CB-WP-0005: count assertions, not names MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Decides the response to CB-RES-0004 and its adversarial review. Decision 1 — a third coverage metric, M-D1-MUT: one mutation per acceptance row, and the suite must go red. adapted:mutation-testing, with the denominator deliberately changed from source lines to acceptance rows, because the failure mode here is not an untested branch but a headline number backed by nothing. A row for which no mutation can be written is recorded unmutatable WITH THE REASON and counts against the metric — a row nobody can invert asserts nothing. Decision 2 — reimplement the log framing rather than assimilate it. This is the deliberate exception to "own the semantics, assimilate the implementation": that rule exists so we do not reimplement HARD things. SHA-256 and ChaCha cost 12 crates and are kept. A length prefix is not in that category. Budget attribution corrected per the review: the durable log is shipped runtime (AM-4a, 1.5% headroom), the bundle writer is dev-only behind the scenarios feature (AM-4b, 9.4%). The survey wanted it both ways without saying so. Decision 3 — a LogStore port with a real shared conformance suite, and the same shape retro-fitted to KernelRng. M-D4-SWAP is a bool over "the same conformance suite" and no suite exists, so AM-11 was never earned. Decision 4 — correct three committed verdicts in evidence/CB-EV-0001: AM-7 split (timing met, hash-identical withdrawn), AM-10 withdrawn as written (no cb-*-api crate; a K6 determinism lint was reported under a D4 leak row), AM-11 downgraded to unmet. AM-1b is added to the scoreboard it was missing from while the flattering half of the same command's output was carried. Decision 5 — widen the name check over every numbered spec and every crate, reporting outside the exit code until 2026-08-31, with the date in the tool rather than in prose. Open-ended "gate it later" is how AM-4's targets went unratified for four workplans. Decision 6 — replay is proven by re-execution in a fresh process, with the reviewer's four controls adopted verbatim, including the two I had not thought of: truncate-by-one-byte and mutated-seed. Stated worse, not only better: own source grows and AM-4c moves against us; CI gets slower by one rebuild per acceptance row. Both accepted — the alternative is a smaller, faster codebase whose acceptance numbers mean nothing. CB-WP-0005 executes it in 8 tasks across four phases. The M-D1-MUT prediction is 9 of 12 and is beatable in both directions: 12 of 12 refutes CB-RES-0004 outright, 3 of 12 means the pass is under-scoped and must stop and re-plan rather than proceed to Phase C. Co-Authored-By: Claude Opus 5 --- .../ADR-0005-assertion-coverage-and-replay.md | 208 +++++++++++++ workplans/CB-WP-0005-assertion-coverage.md | 277 ++++++++++++++++++ 2 files changed, 485 insertions(+) create mode 100644 decisions/ADR-0005-assertion-coverage-and-replay.md create mode 100644 workplans/CB-WP-0005-assertion-coverage.md diff --git a/decisions/ADR-0005-assertion-coverage-and-replay.md b/decisions/ADR-0005-assertion-coverage-and-replay.md new file mode 100644 index 0000000..62936de --- /dev/null +++ b/decisions/ADR-0005-assertion-coverage-and-replay.md @@ -0,0 +1,208 @@ +# ADR-0005: count assertions, not names; reimplement the log; correct the record + +status: accepted +date: 2026-07-31 +decided by: maintainer (Bernd Worsch), 2026-07-31 — ratified the adversarial +review's findings for adoption +tier: L (structural L — creates a capability port; changes what several +headline acceptance numbers mean; chaos d4=1 → no override) +supersedes: nothing. **Corrects** three verdicts in +[CB-EV-0001](../evidence/CB-EV-0001-game-kernel.md) +references: [CB-RES-0004](../research/CB-RES-0004-replay-and-kernel-coverage.md), +`history/260731-replay-kernel-coverage-{research,challenge,response}.md`, +[GameKernel.md](../specs/GameKernel.md) §2.4 §5, +[MetricsAndScenarios.md](../specs/MetricsAndScenarios.md) §1, +[ADR-0004](ADR-0004-am4-ratification.md) (AM-4 budgets) + +## Context + +CB-RES-0004 v1 was rejected by adversarial review with four blocking +findings. v2 establishes that **every coverage instrument in this project +counts names, and none counts assertions** — and that four of the seven +defects found are named in the source and inert, so no name-based check +finds them. Two are mutation-proven: AM-7's `hash-identical` clause and +K9's `through` field both survive mutations that should turn them red. + +This ADR decides four things: what to build, what to assimilate, which +budget each half is charged to, and what to do about verdicts already +committed as `met`. + +--- + +## Decision 1 — a third coverage metric, M-D1-MUT, measured by mutation + +**Adopted.** Name-based coverage stays; it is cheap and catches absent +rules. It is joined by: + +| ID | Dimension | Metric | Unit | Provenance | +|---|---|---|---|---| +| **M-D1-MUT** | D1 | acceptance rows whose stated property, when inverted, turns a test red | count | `adapted:mutation-testing` — applied to **acceptance rows only**, not exhaustively over source | + +**Why adapted rather than adopted.** Mutation testing as practised +(`cargo-mutants`, Stryker, PIT) mutates source exhaustively and scores a +kill rate. That is the wrong shape here on two counts: it is slow enough +to leave the inner loop, and its denominator is *lines*, which is not what +this project claims. Our claims are the twelve AM-* acceptance rows, so +those are the denominator. **One mutation per acceptance row, committed as +a named fixture, run in CI.** + +The trade is stated plainly: a hand-written mutation per row cannot find +defects in rows nobody thought to mutate, whereas an exhaustive tool can. +We accept that, because the failure mode this project actually has is not +"an untested branch" — it is "a headline number backed by nothing", and +that is exactly the population M-D1-MUT covers. + +**Escape hatch closed in advance:** a mutation that cannot be written for +a row is recorded as `unmutatable` **with the reason**, and counts against +the metric. A row nobody can invert is a row that asserts nothing. + +## Decision 2 — reimplement the log framing; do not assimilate + +**Reimplement.** K11's format is a length prefix, a version byte, +per-record framing and a tail check — Kafka and EventStore converged on +the same shape decades apart, and it is 80–120 lines. + +This is the deliberate exception to *"own the semantics; assimilate the +implementation."* The rule exists so we do not reimplement **hard** +things: SHA-256 (K7) and ChaCha (K5) cost 12 crates between them and are +kept, because an audited cryptographic implementation is worth more than a +scoreboard number (ADR-0004). A length prefix is not in that category. +Assimilating sled or redb here would buy nothing but a dependency. + +**Corrected budget attribution** — v1 of the survey got this wrong and the +review caught it: + +| component | shipped? | budget | headroom | +|---|---|---|---| +| **durable log (K11)** — `cb-events`, needed by stage-3 networked sessions | **yes**, shipped runtime | **AM-4a** | 3,750 lines (1.5%) | +| **bundle writer/reader (K10)** — scenario runner, on failure | **no**, dev-only behind `scenarios` | **AM-4b** | 32,979 lines (9.4%) | + +The survey wanted it both ways without saying so. With the split stated, +the AM-4a headroom argument applies only to the log — where we are +reimplementing anyway — and is **withdrawn** for the bundle, where it never +applied. + +## Decision 3 — the durable log is a capability port with a real conformance suite + +**Adopted.** This is what earns the tier. + +M-D4-SWAP is defined as "null + reference impls passing **the same +conformance suite**". `grep -rn conformance` over every `.rs` returns one +doc comment describing future work. The RNG pair — the only pair we have — +is exercised by two *separate, non-shared* tests. So `AM-11 | 1 of 1 | met, +narrow` is unearned on the suite clause, and has been since it was written. + +Deliver a `LogStore` port with a shared `fn conformance(…)` +driven by **both** an in-memory impl and a file-backed impl, and +retro-fit the same shape to `KernelRng`. AM-11 then means what it says. + +## Decision 4 — correct the committed record + +Three verdicts in `evidence/CB-EV-0001` are withdrawn or restated. They are +**corrected in place with a dated note**, not silently edited — the +correction trail is the artifact. + +| row | committed verdict | corrected to | +|---|---|---| +| **AM-7 replay** | `100k events ≤5s \| 2.18 ms \| met, 2,290×` | **split.** The timing clause is met. The `hash-identical` clause is **unmeasured** — mutation-proven — and the log is folded across multiple game seeds, so it is not a replay. Verdict on that clause: **withdrawn**. | +| **AM-10 foreign types** | `zero HashMap/HashSet \| 0 \| met` | **withdrawn as written.** There is no `cb-*-api` crate, so the stated population is empty; what is measured is a K6 determinism lint reported under a D4 leak row. Re-stated as a K6 row, with M-D4-LEAK marked **unmeasured** until a canonical-interface surface exists to measure. | +| **AM-11 impl pairs** | `1 of 1 (KernelRng) \| met, narrow` | **downgraded to unmet** until a shared conformance suite exists. The pair exists; the suite does not, and the metric is a bool over the suite. | + +**AM-1b is added to the evidence scoreboard.** `make coverage` prints +`49/58` and the scoreboard omitted it while carrying `AM-1 … 100% … met` +two lines away. Reporting the flattering half of a gate and dropping the +unflattering half is the mechanism this whole pass is about. + +## Decision 5 — widen the name check without breaking the build + +**Adopted, with a binding date.** `AGGREGATE` becomes a list, `RULE_RE` +becomes per-spec, and the spec→code link runs over **every numbered spec × +every crate**. K10/K14/K18 surface immediately. + +`rule-coverage.py` returns `2` when `missing` is non-empty and `coverage` +is in `make all`, so widening the denominator would fail the build on day +one. That is forbidden by the survey's own rule — a newly widened +denominator is not a regression. + +So: the kernel denominator **reports** and does not feed the exit code, +**until 2026-08-31**, after which it binds like the GR denominator. The +date is in the tool, not in prose, and the tool prints the remaining days. +An open-ended "we'll gate it later" is how AM-4's targets went unratified +for four workplans. + +**The zero-rules positive control is replicated on the new denominator.** +The existing arm refuses to report a percentage over zero rules — a defect +it was fixed for. A kernel regex that matches nothing must abort, not +print `0/0 (100%)`. + +## Decision 6 — replay is proven by re-execution in a fresh process + +**Adopted.** K10 says "re-executes it bit-identically", so the acceptance +test is a round-trip, with four controls the review specified and this ADR +adopts verbatim: + +1. a **committed deliberately-failing fixture** scenario, and an assertion + that bundles produced `> 0` and equals expected failures — all 21 + scenarios pass today, so a corpus sweep would verify zero bundles and + print `ok`; +2. the comparison hash **read out of the bundle**, written by the first + process — never recomputed in the second, which is `assert_eq!(h, h)` + and the AM-7 defect exactly; +3. **negative controls for K11**: truncate the log by one byte, and + corrupt the length prefix; the reader must reject both. K11's operative + clause is *detection*, and a round-trip only ever reads well-formed + input; +4. a **mutated-seed negative control**: change the seed in the manifest and + the replay must fail to reproduce. A round-trip that cannot fail proves + nothing. + +--- + +## Expected advantage, per dimension — including where it is worse + +**D1 ease of specification — better, decisively.** "This rule is enforced" +becomes a claim with a test behind it. A `.cbreplay` bundle turns a failure +report into executable work, which is INTENT design decision 8 and is +currently unimplemented. + +**D2 efficiency of implementation — mixed, and honestly worse in one +place.** No new third-party dependency; the log is ~100 lines and the port +seam is small. But **own source grows** — the bundle requires changing the +runner's data flow (`Pass` extended, `RunOutcome::Failed` restructured, an +initial snapshot captured, the log plumbed out of `execute`), and twelve +mutation fixtures are twelve pieces of test code that exist only to be +broken. AM-4c (own source per 100k third-party lines) is **reported, not +targeted**, and will move against us. That is the right trade: the +alternative is a smaller codebase whose acceptance numbers mean nothing. + +**D3 speed of execution — neutral in the hot path, worse in CI.** Bundle +writes happen only on failure. Log append gains a length-prefix write +against an AM-7 budget already cleared by three orders of magnitude. But +mutation checks rebuild and re-run the suite once per acceptance row, and +CI gets measurably slower. Accepted: the loop's own cost instrument shows +gate time is not where this project's money goes. + +**D4 optionality — better, and this is the tier-L justification.** A +`LogStore` port with a real shared conformance suite is the seam stage 3 +(networked sessions, authoritative host, replay verification) and stage 4 +(package validation) both need, and it converts AM-11 from a bool over a +nonexistent suite into a measured claim. + +## Consequences + +- Three committed `met` verdicts are corrected; one metric (M-D1-MUT) is + added; AM-11 is downgraded to unmet until its suite exists. +- **Headline numbers will get worse before they get better**, and that is + the intended outcome. A pass that widened the denominators and reported + the same percentages would have proved the instruments still count names. +- `InnerLoop.md` §Step 2's "numbers" row changes: reproducing a number by + re-running the command that prints it satisfies the current wording and + finds nothing. The reviewer must **read the assertion behind the number + and mutate it**. This is the second instance of a verification step + inheriting the author's blindness — CB-WP-0002's dedup blind spot was the + first — and both fixes replace re-derivation with adversarial execution. +- **What would falsify this ADR:** if 12 of 12 acceptance rows turn red + under mutation, the instruments were better than CB-RES-0004 claims, the + finding shrinks to three absent rules, and M-D1-MUT was not worth its CI + cost. That outcome is named in the survey's prediction table and is + reachable. diff --git a/workplans/CB-WP-0005-assertion-coverage.md b/workplans/CB-WP-0005-assertion-coverage.md new file mode 100644 index 0000000..e1717f2 --- /dev/null +++ b/workplans/CB-WP-0005-assertion-coverage.md @@ -0,0 +1,277 @@ +--- +id: CB-WP-0005 +title: "Make the instruments count assertions, then fix what they expose" +status: proposed +--- + +# Purpose + +`research/CB-RES-0004-replay-and-kernel-coverage.md` (v2, after an +adversarial review that rejected v1 with four blocking findings): + +> **Every coverage instrument in this project counts names. None counts +> assertions.** Four of the seven defects found this pass are named in the +> source and inert, so no name-based check finds them. + +Two are mutation-proven: AM-7's `hash-identical` clause survives folding a +100k-event log from an unrelated genesis state, and K9's `through` field +survives `Snapshot::take` discarding it entirely. `evidence/CB-EV-0001` +carries **`AM-7 replay | met, 2,290×`** for the first of those. + +[ADR-0005](../decisions/ADR-0005-assertion-coverage-and-replay.md) decides +the response. This workplan executes it in four phases: teach the +instruments to count assertions, correct the record they falsified, +implement what they expose, then measure whether any of it worked. + +**Headline numbers are expected to get worse before they get better.** A +pass that widened every denominator and reported the same percentages +would have proved the instruments still count names. Per InnerLoop §Step 4 +no target moves in the commit that measures it. + +## Phase A — teach the instruments to count assertions + +## Task: spec→code link over every numbered spec and every crate + +```task +id: CB-WP-0005-T01 +status: todo +priority: high +``` + +`tools/rule-coverage.py` hardcodes `AGGREGATE = "games/ground/src/lib.rs"` +and `RULE_RE = \*\*(GR-[A-Z]+\d+)` against `GroundRules.md` alone. The +reviewer's finding: a ~10-line generalization would have surfaced K10, K14 +and K18 the day AM-1b shipped. + +Deliver: `AGGREGATE` becomes a list of source roots; rule patterns become +per-spec; the link runs over **every numbered spec × every crate**. + +Three constraints from ADR-0005 §5, all of which must be visible in the +output: + +1. The kernel denominator **reports and does not feed the exit code until + 2026-08-31**, after which it binds. The date lives in the tool and the + tool prints the days remaining — an open-ended "gate it later" is how + AM-4's targets went unratified for four workplans. +2. **Replicate the zero-rules positive control** on every new denominator. + The existing arm refuses to report over zero rules, a defect it was + fixed for; a kernel regex matching nothing must abort, not print + `0/0 (100%)`. +3. State the limit in the output, as the GR arm already does: this counts + names. It is the cheap half. + +**Predicted:** K10, K14, K18 reported unlinked on the first run. +**Refuted if** any numbered rule in any spec is still unnamed in source +after the pass and the tool does not say so. + +## Task: M-D1-MUT — one mutation per acceptance row + +```task +id: CB-WP-0005-T02 +status: todo +priority: high +``` + +The task this workplan exists for. Name-based checks catch 3 of the 7 +defects; this catches the other 4. + +Deliver `make mutation-check`: for each of the twelve AM-* acceptance +rows, a committed fixture that **inverts the row's stated property** and +an assertion that the suite goes **red**. A row whose mutation leaves the +suite green is a row backed by nothing. + +Per ADR-0005 §1, `adapted:mutation-testing` — the denominator is +acceptance rows, not source lines, because the failure mode here is not an +untested branch but a headline number backed by nothing. + +**The escape hatch is closed in advance:** a row for which no mutation can +be written is recorded `unmutatable` **with the reason** and **counts +against** the metric. A row nobody can invert asserts nothing. + +Carries `--self-test`. Its own positive control is the one that matters: +a mutation harness that fails to apply its mutation reports every row as +`unmutatable` and looks thorough — this project has six harness-does- +nothing instances and one of them is being fixed in T03. + +**Predicted: 9 of 12 rows turn red.** Deliberately beatable in both +directions — **12 of 12 refutes CB-RES-0004** (the instruments were better +than claimed, the finding shrinks to three absent rules, and M-D1-MUT was +not worth its CI cost); **3 of 12 means the pass is under-scoped and must +stop and re-plan** rather than proceed to Phase C. + +## Phase B — correct the record + +## Task: correct three committed verdicts and restore the fourth + +```task +id: CB-WP-0005-T03 +status: todo +priority: high +``` + +Per ADR-0005 §4. Corrections are made **in place with a dated note**, not +silently edited — the correction trail is the artifact. + +| row | committed | corrected to | +|---|---|---| +| AM-7 replay | `met, 2,290×` | timing clause **met**; `hash-identical` clause **withdrawn**, mutation-proven unmeasured, and the fold spans multiple game seeds so it is not a replay | +| AM-10 foreign types | `met` | **withdrawn as written** — no `cb-*-api` crate, so the population is empty; re-stated as a K6 determinism row, M-D4-LEAK marked **unmeasured** | +| AM-11 impl pairs | `met, narrow` | **unmet** until a shared conformance suite exists; the pair exists, the suite does not, and the metric is a bool over the suite | +| AM-1b | *absent from the scoreboard* | **added**, at its real value | + +The last one is the point of the phase. `make coverage` prints `49/58` two +lines below the `100%` the scoreboard carried, and the scoreboard kept the +flattering half. Update `specs/GameKernel.md` §5 and +`specs/MetricsAndScenarios.md` §1 to match, and tag the new figures under +`make facts-check` so they cannot drift. + +## Phase C — implement what the instruments expose + +## Task: K9 and K11 — durable log, LogStore port, real conformance suite + +```task +id: CB-WP-0005-T04 +status: todo +priority: high +``` + +**K9** currently round-trips a `BTreeMap` with `EventSeq(17)` +as a literal. Deliver the real property: *snapshot at seq N + events +N+1..M ≡ genesis fold*, on `GroundState`, hash-compared. + +**K11** — append-only, length-prefixed, versioned framing, with a +truncated tail **detected**. Reimplemented, not assimilated (ADR-0005 §2): +~100 lines against a format Kafka and EventStore converged on +independently. Charged to **AM-4a** (shipped runtime, 1.5% headroom) and +adds no dependency. + +**The port** (ADR-0005 §3): a `LogStore` trait with a shared +`fn conformance(…)` driven by an in-memory impl **and** a +file-backed impl. Retro-fit the same shape to `KernelRng`, whose pair is +currently exercised by two separate non-shared tests — which is why AM-11 +was never earned. + +Negative controls, not optional: truncate by one byte → reject; corrupt +the length prefix → reject. K11's operative clause is *detection*. + +## Task: K10 — the replay bundle and `--replay` + +```task +id: CB-WP-0005-T05 +status: todo +priority: high +``` + +INTENT design decision **8 of 10**, unimplemented. `cb-sim` has no flag +parsing at all, so `--replay` has nowhere to go yet. + +D2 note carried from the review: this is **not** "a directory of four +files". `scenario.rs` creates an `EventLog`, appends to it and never reads +it — the one production instantiation of the K11 log is a write-only sink. +`Pass` carries the *end* state, not an initial snapshot, and failures are a +formatted `String`, not structured expected-vs-actual. Plumb the log out of +`execute`, capture an initial snapshot, restructure `RunOutcome::Failed`. + +The bundle writer is **dev-only** behind the `scenarios` feature and is +charged to **AM-4b** (9.4% headroom), not AM-4a. + +**Four controls, from ADR-0005 §6, verbatim and all required:** + +1. a committed **deliberately-failing fixture** scenario, plus an + assertion that bundles produced `> 0` and equals expected failures — + all 21 scenarios pass today, so a corpus sweep verifies zero bundles + and prints `ok`; +2. the comparison hash **read out of the bundle**, written by the first + process — never recomputed in the second, which is `assert_eq!(h, h)` + and the AM-7 defect exactly; +3. truncate-by-one-byte and corrupt-length-prefix rejection; +4. a **mutated-seed** control: change the manifest seed and the replay + must fail to reproduce. A round-trip that cannot fail proves nothing. + +## Task: K14 and K18 — implement, or amend the spec and say why + +```task +id: CB-WP-0005-T06 +status: todo +priority: medium +``` + +Both are named nowhere in source, and both have an honest second option +that must be considered rather than assumed away. + +**K14** — "one commit window per Select". `CommitWindow` exists, has zero +non-test users, and `games/ground` does not import it; GROUND collects +selections in its own aggregate. Either wire GROUND through the runtime +primitive as K14 requires, **or** amend K14 to describe what the kernel +actually guarantees and record why the primitive stays. A dead abstraction +certified by a test that exercises only itself is worse than no +abstraction — it is an AM-11-shaped claim waiting to be quoted. + +**K18** — "Criterion benches driving the same scenario format". The bench +hardcodes commands in Rust and never touches `ScenarioFile`, and +`benchmarks/` contains only `baselines/model-prices.toml` against a spec +that says benches live there. Either drive the bench from scenario files, +**or** amend K18 and MetricsAndScenarios §3 to match reality. + +Whichever way each goes, the outcome is a rule and an implementation that +agree. Deleting a rule to make a gate green is forbidden; deleting a rule +*with an argument*, recorded, is a legitimate outcome and the more likely +one for K18. + +## Phase D — measure it, then say what the loop should change + +## Task: control loop — did the numbers get worse, then better? + +```task +id: CB-WP-0005-T07 +status: todo +priority: high +``` + +Commit `evidence/CB-EV-0004-assertion-coverage.md`. Three tests, all +reported: + +1. **Did the denominators widen?** Name coverage before and after, per + spec. A widened denominator that reports the same percentage is the + result to publish — it would mean the instruments still count names. +2. **M-D1-MUT against the 9-of-12 prediction.** Report unmet if unmet; no + target moves in the commit that measures it. State which rows are + `unmutatable` and why. +3. **Did quality hold?** `make all` green with the new gates; and the + honest question this pass raises — did widening the instruments surface + *new* defects, or only the seven already known? Finding none would be + evidence the sweep was as complete as CB-RES-0004 claims, which is a + stronger claim than it sounds and should be stated as such. + +Normalize per unit of work, as CB-WP-0004 T05 established: report share of +pass alongside absolute figures, and use `cb-cost --since` to window this +pass against the last. + +## Task: retrospective and InnerLoop v1.4 + +```task +id: CB-WP-0005-T08 +status: todo +priority: medium +``` + +One loop change is already earned and should be written whatever else this +pass finds. InnerLoop §Step 2's "numbers" row says the reviewer must +*reproduce the number independently* — satisfiable by re-running the +command that prints it, which is exactly what does **not** find this class. +The reviewer found AM-7 by opening a test out of curiosity, and said so. + +> **v1.4:** the reviewer must read the assertion behind every quoted +> acceptance number and **mutate it**, not re-run the command that prints +> it. + +This is the **second** instance of a verification step inheriting the +author's blindness — CB-WP-0002's dedup blind spot was the first. Both +fixes replace re-derivation with adversarial execution. Record whether +that generalizes. + +The question to answer honestly: **CB-WP-0004 concluded that tooling +recovers capacity only where it removes the manual path. Does the same +test predict which gates work?** `env-test` and `task-done` removed the +manual path and held. Does a mutation gate remove the manual path — or is +writing a weak mutation the new `grep`?