coulomb/clay-borg
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

ADR-0005 + CB-WP-0005: count assertions, not names
Some checks failed
ci / check (push) Failing after 3s
Details

Browse source 
Decides the response to CB-RES-0004 and its adversarial review.

Decision 1 — a third coverage metric, M-D1-MUT: one mutation per
acceptance row, and the suite must go red. adapted:mutation-testing, with
the denominator deliberately changed from source lines to acceptance
rows, because the failure mode here is not an untested branch but a
headline number backed by nothing. A row for which no mutation can be
written is recorded unmutatable WITH THE REASON and counts against the
metric — a row nobody can invert asserts nothing.

Decision 2 — reimplement the log framing rather than assimilate it. This
is the deliberate exception to "own the semantics, assimilate the
implementation": that rule exists so we do not reimplement HARD things.
SHA-256 and ChaCha cost 12 crates and are kept. A length prefix is not in
that category. Budget attribution corrected per the review: the durable
log is shipped runtime (AM-4a, 1.5% headroom), the bundle writer is
dev-only behind the scenarios feature (AM-4b, 9.4%). The survey wanted it
both ways without saying so.

Decision 3 — a LogStore port with a real shared conformance suite, and
the same shape retro-fitted to KernelRng. M-D4-SWAP is a bool over "the
same conformance suite" and no suite exists, so AM-11 was never earned.

Decision 4 — correct three committed verdicts in evidence/CB-EV-0001:
AM-7 split (timing met, hash-identical withdrawn), AM-10 withdrawn as
written (no cb-*-api crate; a K6 determinism lint was reported under a D4
leak row), AM-11 downgraded to unmet. AM-1b is added to the scoreboard it
was missing from while the flattering half of the same command's output
was carried.

Decision 5 — widen the name check over every numbered spec and every
crate, reporting outside the exit code until 2026-08-31, with the date in
the tool rather than in prose. Open-ended "gate it later" is how AM-4's
targets went unratified for four workplans.

Decision 6 — replay is proven by re-execution in a fresh process, with
the reviewer's four controls adopted verbatim, including the two I had
not thought of: truncate-by-one-byte and mutated-seed.

Stated worse, not only better: own source grows and AM-4c moves against
us; CI gets slower by one rebuild per acceptance row. Both accepted — the
alternative is a smaller, faster codebase whose acceptance numbers mean
nothing.

CB-WP-0005 executes it in 8 tasks across four phases. The M-D1-MUT
prediction is 9 of 12 and is beatable in both directions: 12 of 12
refutes CB-RES-0004 outright, 3 of 12 means the pass is under-scoped and
must stop and re-plan rather than proceed to Phase C.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 14:25:26 +02:00
parent 923c2038be
commit 1bbbfc846f
2 changed files with 485 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

208
decisions/ADR-0005-assertion-coverage-and-replay.md Normal file
View file

@ -0,0 +1,208 @@
# ADR-0005: count assertions, not names; reimplement the log; correct the record
status: accepted
date: 2026-07-31
decided by: maintainer (Bernd Worsch), 2026-07-31 — ratified the adversarial
review's findings for adoption
tier: L (structural L — creates a capability port; changes what several
headline acceptance numbers mean; chaos d4=1 → no override)
supersedes: nothing. **Corrects** three verdicts in
[CB-EV-0001](../evidence/CB-EV-0001-game-kernel.md)
references: [CB-RES-0004](../research/CB-RES-0004-replay-and-kernel-coverage.md),
`history/260731-replay-kernel-coverage-{research,challenge,response}.md`,
[GameKernel.md](../specs/GameKernel.md) §2.4 §5,
[MetricsAndScenarios.md](../specs/MetricsAndScenarios.md) §1,
[ADR-0004](ADR-0004-am4-ratification.md) (AM-4 budgets)
## Context
CB-RES-0004 v1 was rejected by adversarial review with four blocking
findings. v2 establishes that **every coverage instrument in this project
counts names, and none counts assertions** — and that four of the seven
defects found are named in the source and inert, so no name-based check
finds them. Two are mutation-proven: AM-7's `hash-identical` clause and
K9's `through` field both survive mutations that should turn them red.
This ADR decides four things: what to build, what to assimilate, which
budget each half is charged to, and what to do about verdicts already
committed as `met`.
---
## Decision 1 — a third coverage metric, M-D1-MUT, measured by mutation
**Adopted.** Name-based coverage stays; it is cheap and catches absent
rules. It is joined by:
| ID | Dimension | Metric | Unit | Provenance |
|---|---|---|---|---|
| **M-D1-MUT** | D1 | acceptance rows whose stated property, when inverted, turns a test red | count | `adapted:mutation-testing` — applied to **acceptance rows only**, not exhaustively over source |
**Why adapted rather than adopted.** Mutation testing as practised
(`cargo-mutants`, Stryker, PIT) mutates source exhaustively and scores a
kill rate. That is the wrong shape here on two counts: it is slow enough
to leave the inner loop, and its denominator is *lines*, which is not what
this project claims. Our claims are the twelve AM-* acceptance rows, so
those are the denominator. **One mutation per acceptance row, committed as
a named fixture, run in CI.**
The trade is stated plainly: a hand-written mutation per row cannot find
defects in rows nobody thought to mutate, whereas an exhaustive tool can.
We accept that, because the failure mode this project actually has is not
"an untested branch" — it is "a headline number backed by nothing", and
that is exactly the population M-D1-MUT covers.
**Escape hatch closed in advance:** a mutation that cannot be written for
a row is recorded as `unmutatable` **with the reason**, and counts against
the metric. A row nobody can invert is a row that asserts nothing.
## Decision 2 — reimplement the log framing; do not assimilate
**Reimplement.** K11's format is a length prefix, a version byte,
per-record framing and a tail check — Kafka and EventStore converged on
the same shape decades apart, and it is 80–120 lines.
This is the deliberate exception to *"own the semantics; assimilate the
implementation."* The rule exists so we do not reimplement **hard**
things: SHA-256 (K7) and ChaCha (K5) cost 12 crates between them and are
kept, because an audited cryptographic implementation is worth more than a
scoreboard number (ADR-0004). A length prefix is not in that category.
Assimilating sled or redb here would buy nothing but a dependency.
**Corrected budget attribution** — v1 of the survey got this wrong and the
review caught it:
| component | shipped? | budget | headroom |
|---|---|---|---|
| **durable log (K11)** — `cb-events`, needed by stage-3 networked sessions | **yes**, shipped runtime | **AM-4a** | 3,750 lines (1.5%) |
| **bundle writer/reader (K10)** — scenario runner, on failure | **no**, dev-only behind `scenarios` | **AM-4b** | 32,979 lines (9.4%) |
The survey wanted it both ways without saying so. With the split stated,
the AM-4a headroom argument applies only to the log — where we are
reimplementing anyway — and is **withdrawn** for the bundle, where it never
applied.
## Decision 3 — the durable log is a capability port with a real conformance suite
**Adopted.** This is what earns the tier.
M-D4-SWAP is defined as "null + reference impls passing **the same
conformance suite**". `grep -rn conformance` over every `.rs` returns one
doc comment describing future work. The RNG pair — the only pair we have —
is exercised by two *separate, non-shared* tests. So `AM-11 | 1 of 1 | met,
narrow` is unearned on the suite clause, and has been since it was written.
Deliver a `LogStore` port with a shared `fn conformance<S: LogStore>(…)`
driven by **both** an in-memory impl and a file-backed impl, and
retro-fit the same shape to `KernelRng`. AM-11 then means what it says.
## Decision 4 — correct the committed record
Three verdicts in `evidence/CB-EV-0001` are withdrawn or restated. They are
**corrected in place with a dated note**, not silently edited — the
correction trail is the artifact.
| row | committed verdict | corrected to |
|---|---|---|
| **AM-7 replay** | `100k events ≤5s \| 2.18 ms \| met, 2,290×` | **split.** The timing clause is met. The `hash-identical` clause is **unmeasured** — mutation-proven — and the log is folded across multiple game seeds, so it is not a replay. Verdict on that clause: **withdrawn**. |
| **AM-10 foreign types** | `zero HashMap/HashSet \| 0 \| met` | **withdrawn as written.** There is no `cb-*-api` crate, so the stated population is empty; what is measured is a K6 determinism lint reported under a D4 leak row. Re-stated as a K6 row, with M-D4-LEAK marked **unmeasured** until a canonical-interface surface exists to measure. |
| **AM-11 impl pairs** | `1 of 1 (KernelRng) \| met, narrow` | **downgraded to unmet** until a shared conformance suite exists. The pair exists; the suite does not, and the metric is a bool over the suite. |
**AM-1b is added to the evidence scoreboard.** `make coverage` prints
`49/58` and the scoreboard omitted it while carrying `AM-1 … 100% … met`
two lines away. Reporting the flattering half of a gate and dropping the
unflattering half is the mechanism this whole pass is about.
## Decision 5 — widen the name check without breaking the build
**Adopted, with a binding date.** `AGGREGATE` becomes a list, `RULE_RE`
becomes per-spec, and the spec→code link runs over **every numbered spec ×
every crate**. K10/K14/K18 surface immediately.
`rule-coverage.py` returns `2` when `missing` is non-empty and `coverage`
is in `make all`, so widening the denominator would fail the build on day
one. That is forbidden by the survey's own rule — a newly widened
denominator is not a regression.
So: the kernel denominator **reports** and does not feed the exit code,
**until 2026-08-31**, after which it binds like the GR denominator. The
date is in the tool, not in prose, and the tool prints the remaining days.
An open-ended "we'll gate it later" is how AM-4's targets went unratified
for four workplans.
**The zero-rules positive control is replicated on the new denominator.**
The existing arm refuses to report a percentage over zero rules — a defect
it was fixed for. A kernel regex that matches nothing must abort, not
print `0/0 (100%)`.
## Decision 6 — replay is proven by re-execution in a fresh process
**Adopted.** K10 says "re-executes it bit-identically", so the acceptance
test is a round-trip, with four controls the review specified and this ADR
adopts verbatim:
1. a **committed deliberately-failing fixture** scenario, and an assertion
that bundles produced `> 0` and equals expected failures — all 21
scenarios pass today, so a corpus sweep would verify zero bundles and
print `ok`;
2. the comparison hash **read out of the bundle**, written by the first
process — never recomputed in the second, which is `assert_eq!(h, h)`
and the AM-7 defect exactly;
3. **negative controls for K11**: truncate the log by one byte, and
corrupt the length prefix; the reader must reject both. K11's operative
clause is *detection*, and a round-trip only ever reads well-formed
input;
4. a **mutated-seed negative control**: change the seed in the manifest and
the replay must fail to reproduce. A round-trip that cannot fail proves
nothing.
---
## Expected advantage, per dimension — including where it is worse
**D1 ease of specification — better, decisively.** "This rule is enforced"
becomes a claim with a test behind it. A `.cbreplay` bundle turns a failure
report into executable work, which is INTENT design decision 8 and is
currently unimplemented.
**D2 efficiency of implementation — mixed, and honestly worse in one
place.** No new third-party dependency; the log is ~100 lines and the port
seam is small. But **own source grows** — the bundle requires changing the
runner's data flow (`Pass` extended, `RunOutcome::Failed` restructured, an
initial snapshot captured, the log plumbed out of `execute`), and twelve
mutation fixtures are twelve pieces of test code that exist only to be
broken. AM-4c (own source per 100k third-party lines) is **reported, not
targeted**, and will move against us. That is the right trade: the
alternative is a smaller codebase whose acceptance numbers mean nothing.
**D3 speed of execution — neutral in the hot path, worse in CI.** Bundle
writes happen only on failure. Log append gains a length-prefix write
against an AM-7 budget already cleared by three orders of magnitude. But
mutation checks rebuild and re-run the suite once per acceptance row, and
CI gets measurably slower. Accepted: the loop's own cost instrument shows
gate time is not where this project's money goes.
**D4 optionality — better, and this is the tier-L justification.** A
`LogStore` port with a real shared conformance suite is the seam stage 3
(networked sessions, authoritative host, replay verification) and stage 4
(package validation) both need, and it converts AM-11 from a bool over a
nonexistent suite into a measured claim.
## Consequences
- Three committed `met` verdicts are corrected; one metric (M-D1-MUT) is
added; AM-11 is downgraded to unmet until its suite exists.
- **Headline numbers will get worse before they get better**, and that is
the intended outcome. A pass that widened the denominators and reported
the same percentages would have proved the instruments still count names.
- `InnerLoop.md` §Step 2's "numbers" row changes: reproducing a number by
re-running the command that prints it satisfies the current wording and
finds nothing. The reviewer must **read the assertion behind the number
and mutate it**. This is the second instance of a verification step
inheriting the author's blindness — CB-WP-0002's dedup blind spot was the
first — and both fixes replace re-derivation with adversarial execution.
- **What would falsify this ADR:** if 12 of 12 acceptance rows turn red
under mutation, the instruments were better than CB-RES-0004 claims, the
finding shrinks to three absent rules, and M-D1-MUT was not worth its CI
cost. That outcome is named in the survey's prediction table and is
reachable.

277
workplans/CB-WP-0005-assertion-coverage.md Normal file
View file

@ -0,0 +1,277 @@
---
id: CB-WP-0005
title: "Make the instruments count assertions, then fix what they expose"
status: proposed
---
# Purpose
`research/CB-RES-0004-replay-and-kernel-coverage.md` (v2, after an
adversarial review that rejected v1 with four blocking findings):
> **Every coverage instrument in this project counts names. None counts
> assertions.** Four of the seven defects found this pass are named in the
> source and inert, so no name-based check finds them.
Two are mutation-proven: AM-7's `hash-identical` clause survives folding a
100k-event log from an unrelated genesis state, and K9's `through` field
survives `Snapshot::take` discarding it entirely. `evidence/CB-EV-0001`
carries **`AM-7 replay | met, 2,290×`** for the first of those.
[ADR-0005](../decisions/ADR-0005-assertion-coverage-and-replay.md) decides
the response. This workplan executes it in four phases: teach the
instruments to count assertions, correct the record they falsified,
implement what they expose, then measure whether any of it worked.
**Headline numbers are expected to get worse before they get better.** A
pass that widened every denominator and reported the same percentages
would have proved the instruments still count names. Per InnerLoop §Step 4
no target moves in the commit that measures it.
## Phase A — teach the instruments to count assertions
## Task: spec→code link over every numbered spec and every crate
```task
id: CB-WP-0005-T01
status: todo
priority: high
```
`tools/rule-coverage.py` hardcodes `AGGREGATE = "games/ground/src/lib.rs"`
and `RULE_RE = \*\*(GR-[A-Z]+\d+)` against `GroundRules.md` alone. The
reviewer's finding: a ~10-line generalization would have surfaced K10, K14
and K18 the day AM-1b shipped.
Deliver: `AGGREGATE` becomes a list of source roots; rule patterns become
per-spec; the link runs over **every numbered spec × every crate**.
Three constraints from ADR-0005 §5, all of which must be visible in the
output:
1. The kernel denominator **reports and does not feed the exit code until
2026-08-31**, after which it binds. The date lives in the tool and the
tool prints the days remaining — an open-ended "gate it later" is how
AM-4's targets went unratified for four workplans.
2. **Replicate the zero-rules positive control** on every new denominator.
The existing arm refuses to report over zero rules, a defect it was
fixed for; a kernel regex matching nothing must abort, not print
`0/0 (100%)`.
3. State the limit in the output, as the GR arm already does: this counts
names. It is the cheap half.
**Predicted:** K10, K14, K18 reported unlinked on the first run.
**Refuted if** any numbered rule in any spec is still unnamed in source
after the pass and the tool does not say so.
## Task: M-D1-MUT — one mutation per acceptance row
```task
id: CB-WP-0005-T02
status: todo
priority: high
```
The task this workplan exists for. Name-based checks catch 3 of the 7
defects; this catches the other 4.
Deliver `make mutation-check`: for each of the twelve AM-* acceptance
rows, a committed fixture that **inverts the row's stated property** and
an assertion that the suite goes **red**. A row whose mutation leaves the
suite green is a row backed by nothing.
Per ADR-0005 §1, `adapted:mutation-testing` — the denominator is
acceptance rows, not source lines, because the failure mode here is not an
untested branch but a headline number backed by nothing.
**The escape hatch is closed in advance:** a row for which no mutation can
be written is recorded `unmutatable` **with the reason** and **counts
against** the metric. A row nobody can invert asserts nothing.
Carries `--self-test`. Its own positive control is the one that matters:
a mutation harness that fails to apply its mutation reports every row as
`unmutatable` and looks thorough — this project has six harness-does-
nothing instances and one of them is being fixed in T03.
**Predicted: 9 of 12 rows turn red.** Deliberately beatable in both
directions — **12 of 12 refutes CB-RES-0004** (the instruments were better
than claimed, the finding shrinks to three absent rules, and M-D1-MUT was
not worth its CI cost); **3 of 12 means the pass is under-scoped and must
stop and re-plan** rather than proceed to Phase C.
## Phase B — correct the record
## Task: correct three committed verdicts and restore the fourth
```task
id: CB-WP-0005-T03
status: todo
priority: high
```
Per ADR-0005 §4. Corrections are made **in place with a dated note**, not
silently edited — the correction trail is the artifact.
| row | committed | corrected to |
|---|---|---|
| AM-7 replay | `met, 2,290×` | timing clause **met**; `hash-identical` clause **withdrawn**, mutation-proven unmeasured, and the fold spans multiple game seeds so it is not a replay |
| AM-10 foreign types | `met` | **withdrawn as written** — no `cb-*-api` crate, so the population is empty; re-stated as a K6 determinism row, M-D4-LEAK marked **unmeasured** |
| AM-11 impl pairs | `met, narrow` | **unmet** until a shared conformance suite exists; the pair exists, the suite does not, and the metric is a bool over the suite |
| AM-1b | *absent from the scoreboard* | **added**, at its real value |
The last one is the point of the phase. `make coverage` prints `49/58` two
lines below the `100%` the scoreboard carried, and the scoreboard kept the
flattering half. Update `specs/GameKernel.md` §5 and
`specs/MetricsAndScenarios.md` §1 to match, and tag the new figures under
`make facts-check` so they cannot drift.
## Phase C — implement what the instruments expose
## Task: K9 and K11 — durable log, LogStore port, real conformance suite
```task
id: CB-WP-0005-T04
status: todo
priority: high
```
**K9** currently round-trips a `BTreeMap<String, u8>` with `EventSeq(17)`
as a literal. Deliver the real property: *snapshot at seq N + events
N+1..M ≡ genesis fold*, on `GroundState`, hash-compared.
**K11** — append-only, length-prefixed, versioned framing, with a
truncated tail **detected**. Reimplemented, not assimilated (ADR-0005 §2):
~100 lines against a format Kafka and EventStore converged on
independently. Charged to **AM-4a** (shipped runtime, 1.5% headroom) and
adds no dependency.
**The port** (ADR-0005 §3): a `LogStore` trait with a shared
`fn conformance<S: LogStore>(…)` driven by an in-memory impl **and** a
file-backed impl. Retro-fit the same shape to `KernelRng`, whose pair is
currently exercised by two separate non-shared tests — which is why AM-11
was never earned.
Negative controls, not optional: truncate by one byte → reject; corrupt
the length prefix → reject. K11's operative clause is *detection*.
## Task: K10 — the replay bundle and `--replay`
```task
id: CB-WP-0005-T05
status: todo
priority: high
```
INTENT design decision **8 of 10**, unimplemented. `cb-sim` has no flag
parsing at all, so `--replay` has nowhere to go yet.
D2 note carried from the review: this is **not** "a directory of four
files". `scenario.rs` creates an `EventLog`, appends to it and never reads
it — the one production instantiation of the K11 log is a write-only sink.
`Pass` carries the *end* state, not an initial snapshot, and failures are a
formatted `String`, not structured expected-vs-actual. Plumb the log out of
`execute`, capture an initial snapshot, restructure `RunOutcome::Failed`.
The bundle writer is **dev-only** behind the `scenarios` feature and is
charged to **AM-4b** (9.4% headroom), not AM-4a.
**Four controls, from ADR-0005 §6, verbatim and all required:**
1. a committed **deliberately-failing fixture** scenario, plus an
assertion that bundles produced `> 0` and equals expected failures —
all 21 scenarios pass today, so a corpus sweep verifies zero bundles
and prints `ok`;
2. the comparison hash **read out of the bundle**, written by the first
process — never recomputed in the second, which is `assert_eq!(h, h)`
and the AM-7 defect exactly;
3. truncate-by-one-byte and corrupt-length-prefix rejection;
4. a **mutated-seed** control: change the manifest seed and the replay
must fail to reproduce. A round-trip that cannot fail proves nothing.
## Task: K14 and K18 — implement, or amend the spec and say why
```task
id: CB-WP-0005-T06
status: todo
priority: medium
```
Both are named nowhere in source, and both have an honest second option
that must be considered rather than assumed away.
**K14** — "one commit window per Select". `CommitWindow` exists, has zero
non-test users, and `games/ground` does not import it; GROUND collects
selections in its own aggregate. Either wire GROUND through the runtime
primitive as K14 requires, **or** amend K14 to describe what the kernel
actually guarantees and record why the primitive stays. A dead abstraction
certified by a test that exercises only itself is worse than no
abstraction — it is an AM-11-shaped claim waiting to be quoted.
**K18** — "Criterion benches driving the same scenario format". The bench
hardcodes commands in Rust and never touches `ScenarioFile`, and
`benchmarks/` contains only `baselines/model-prices.toml` against a spec
that says benches live there. Either drive the bench from scenario files,
**or** amend K18 and MetricsAndScenarios §3 to match reality.
Whichever way each goes, the outcome is a rule and an implementation that
agree. Deleting a rule to make a gate green is forbidden; deleting a rule
*with an argument*, recorded, is a legitimate outcome and the more likely
one for K18.
## Phase D — measure it, then say what the loop should change
## Task: control loop — did the numbers get worse, then better?
```task
id: CB-WP-0005-T07
status: todo
priority: high
```
Commit `evidence/CB-EV-0004-assertion-coverage.md`. Three tests, all
reported:
1. **Did the denominators widen?** Name coverage before and after, per
spec. A widened denominator that reports the same percentage is the
result to publish — it would mean the instruments still count names.
2. **M-D1-MUT against the 9-of-12 prediction.** Report unmet if unmet; no
target moves in the commit that measures it. State which rows are
`unmutatable` and why.
3. **Did quality hold?** `make all` green with the new gates; and the
honest question this pass raises — did widening the instruments surface
*new* defects, or only the seven already known? Finding none would be
evidence the sweep was as complete as CB-RES-0004 claims, which is a
stronger claim than it sounds and should be stated as such.
Normalize per unit of work, as CB-WP-0004 T05 established: report share of
pass alongside absolute figures, and use `cb-cost --since` to window this
pass against the last.
## Task: retrospective and InnerLoop v1.4
```task
id: CB-WP-0005-T08
status: todo
priority: medium
```
One loop change is already earned and should be written whatever else this
pass finds. InnerLoop §Step 2's "numbers" row says the reviewer must
*reproduce the number independently* — satisfiable by re-running the
command that prints it, which is exactly what does **not** find this class.
The reviewer found AM-7 by opening a test out of curiosity, and said so.
> **v1.4:** the reviewer must read the assertion behind every quoted
> acceptance number and **mutate it**, not re-run the command that prints
> it.
This is the **second** instance of a verification step inheriting the
author's blindness — CB-WP-0002's dedup blind spot was the first. Both
fixes replace re-derivation with adversarial execution. Record whether
that generalizes.
The question to answer honestly: **CB-WP-0004 concluded that tooling
recovers capacity only where it removes the manual path. Does the same
test predict which gates work?** `env-test` and `task-done` removed the
manual path and held. Does a mutation gate remove the manual path — or is
writing a weak mutation the new `grep`?