232 lines
8.3 KiB
Markdown
232 lines
8.3 KiB
Markdown
|
|
---
|
|||
|
|
id: CB-WP-0006
|
|||
|
|
title: "Instrument the acceptance table, then implement what it exposes"
|
|||
|
|
status: proposed
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Purpose
|
|||
|
|
|
|||
|
|
CB-WP-0005 T02 measured M-D1-MUT for the first time:
|
|||
|
|
|
|||
|
|
```text
|
|||
|
|
M-D1-MUT: 4/14 rows enforced
|
|||
|
|
PARTIAL 2 (AM-7, AM-8 — some clauses live, some inert)
|
|||
|
|
unmutatable 8 (no property to invert)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**4 of 14, against a predicted 10 of 14.** The prediction was wrong by
|
|||
|
|
2.5×, and wrong about *what* as well as *how much*: CB-RES-0004 diagnosed
|
|||
|
|
three absent kernel rules, and the measurement found that **more than half
|
|||
|
|
the acceptance table has no instrument behind it at all**.
|
|||
|
|
|
|||
|
|
CB-WP-0005 Phase C was deferred before starting on that basis (T02's stop
|
|||
|
|
condition, maintainer decision 2026-07-31). It was scoped to five rules —
|
|||
|
|
K9, K10, K11, K14, K18 — which does not address the eight.
|
|||
|
|
|
|||
|
|
This workplan is scoped to what was measured rather than what was
|
|||
|
|
predicted. **Phase A instruments the unmeasured rows; Phase B implements
|
|||
|
|
the kernel work inherited from CB-WP-0005 T04–T06.** That order is the
|
|||
|
|
whole point: the previous pass built a fix before its instrument existed,
|
|||
|
|
and the instrument then contradicted the fix's premise.
|
|||
|
|
|
|||
|
|
Per InnerLoop §Step 4, no target moves in the commit that measures it.
|
|||
|
|
|
|||
|
|
## Phase A — instrument the rows that measure nothing
|
|||
|
|
|
|||
|
|
## Task: AM-6 — assert the throughput target
|
|||
|
|
|
|||
|
|
```task
|
|||
|
|
id: CB-WP-0006-T01
|
|||
|
|
status: todo
|
|||
|
|
priority: high
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The sharpest single finding of CB-WP-0005. **Nothing in the workspace
|
|||
|
|
compares any number to 100,000 events/s** — the project's headline
|
|||
|
|
throughput claim, reported as `met, 16.5×`. The Criterion bench emits
|
|||
|
|
throughput; the only asserts in `synthetic.rs` are the stress-gate shape
|
|||
|
|
and the events-per-round pin.
|
|||
|
|
|
|||
|
|
Deliver a standing assertion — a test, not a bench — that measures applied
|
|||
|
|
events/s on the synthetic workload and fails below the AM-6 target.
|
|||
|
|
|
|||
|
|
**The trap, stated up front:** a timing assertion in CI is flaky by
|
|||
|
|
nature, and the usual response is to loosen it until it never fires, which
|
|||
|
|
would reproduce exactly the defect being fixed. So the threshold is
|
|||
|
|
**pinned well below the measured 1.65M/s** (AM-6 asks for 100k), the
|
|||
|
|
assertion states its own headroom in the failure message, and a loosening
|
|||
|
|
requires an ADR. A gate tuned until it cannot fire is a gate that counts
|
|||
|
|
names.
|
|||
|
|
|
|||
|
|
**Verified by:** `make mutation-check --row AM-6` goes from `unmutatable`
|
|||
|
|
to `red`.
|
|||
|
|
|
|||
|
|
## Task: AM-2, AM-3 — instrument the size metrics
|
|||
|
|
|
|||
|
|
```task
|
|||
|
|
id: CB-WP-0006-T02
|
|||
|
|
status: todo
|
|||
|
|
priority: medium
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
AM-2 (≤40 spec lines per rule) and AM-3 (≤50 LOC for the synthetic
|
|||
|
|
workload) are both `unmutatable` for the same reason: `make loc` prints
|
|||
|
|
line counts and nothing divides, compares, or gates.
|
|||
|
|
|
|||
|
|
AM-2 is the **anti-gaming pair** for AM-1 — 100% rule coverage means
|
|||
|
|
nothing if the rules are trivially small — so an uninstrumented AM-2
|
|||
|
|
leaves AM-1 gameable, which is the specific risk MetricsAndScenarios §1
|
|||
|
|
names when it introduces M-D1-SPL.
|
|||
|
|
|
|||
|
|
AM-3 depends on K18 (benches driven from scenario files); if K18 resolves
|
|||
|
|
toward amending the spec rather than implementing it, AM-3 must be
|
|||
|
|
restated or withdrawn with an argument rather than left unmeasured.
|
|||
|
|
|
|||
|
|
## Task: AM-5, AM-9 — measure or withdraw, but stop leaving them blank
|
|||
|
|
|
|||
|
|
```task
|
|||
|
|
id: CB-WP-0006-T03
|
|||
|
|
status: todo
|
|||
|
|
priority: medium
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
AM-5 (clean build ≤60 s) is `recorded not gated` — and not recorded
|
|||
|
|
either. AM-9 (peak RSS ≤64 MB) is declared "very unlikely to bind", which
|
|||
|
|
CB-EV-0001 itself calls out as an unmeasured judgment call.
|
|||
|
|
|
|||
|
|
Both are cheap to measure: one `time` around a clean build, one
|
|||
|
|
`/usr/bin/time -v` around the 100k-event run. **Or withdraw them with an
|
|||
|
|
argument**, which is a legitimate outcome and possibly the right one for
|
|||
|
|
AM-5 on a machine-dependent number. What is not legitimate is a third pass
|
|||
|
|
leaving them blank while the scoreboard reports elsewhere.
|
|||
|
|
|
|||
|
|
## Task: AM-4c — target it or drop it
|
|||
|
|
|
|||
|
|
```task
|
|||
|
|
id: CB-WP-0006-T04
|
|||
|
|
status: todo
|
|||
|
|
priority: low
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
`reported, not targeted`, so there is no property to invert and it counts
|
|||
|
|
against M-D1-MUT by ADR-0005 §1.
|
|||
|
|
|
|||
|
|
Two honest options, and the task must pick one with an argument: give it a
|
|||
|
|
threshold (making it enforceable), or **remove it from the acceptance
|
|||
|
|
table** and keep it as a reported diagnostic. A row in an acceptance table
|
|||
|
|
that cannot fail is decoration, and this pass has learned to say so.
|
|||
|
|
|
|||
|
|
Note the incentive: ADR-0005 already predicts own-source growth from the
|
|||
|
|
kernel work in Phase B, so AM-4c will move against us. Setting its
|
|||
|
|
threshold *after* seeing that movement would be the retarget InnerLoop
|
|||
|
|
§Step 4 forbids — so it is set here, before Phase B, or not at all.
|
|||
|
|
|
|||
|
|
## Phase B — the kernel work inherited from CB-WP-0005
|
|||
|
|
|
|||
|
|
> These three carry their full analysis in
|
|||
|
|
> [CB-WP-0005](CB-WP-0005-assertion-coverage.md) Phase C, where they were
|
|||
|
|
> written and then deferred unstarted. They are summarized here; read that
|
|||
|
|
> file for the detail, the four replay controls, and the D2 correction on
|
|||
|
|
> what the bundle writer actually requires.
|
|||
|
|
|
|||
|
|
## Task: K9 and K11 — durable log, LogStore port, real conformance suite
|
|||
|
|
|
|||
|
|
```task
|
|||
|
|
id: CB-WP-0006-T05
|
|||
|
|
status: todo
|
|||
|
|
priority: high
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Inherited from CB-WP-0005 T04. K9's acceptance property (snapshot at N +
|
|||
|
|
events N+1..M ≡ genesis fold) is mutation-proven unasserted; K11 has no
|
|||
|
|
durable format at all. The `LogStore` port with a shared
|
|||
|
|
`fn conformance<S: LogStore>(…)`, retro-fitted to `KernelRng`, is what
|
|||
|
|
finally earns **AM-11** — currently downgraded to unmet because the pair
|
|||
|
|
exists and the suite does not.
|
|||
|
|
|
|||
|
|
**Verified by:** `make mutation-check --row AM-11` goes from
|
|||
|
|
`unmutatable` to `red`.
|
|||
|
|
|
|||
|
|
## Task: K10 — the replay bundle and `--replay`
|
|||
|
|
|
|||
|
|
```task
|
|||
|
|
id: CB-WP-0006-T06
|
|||
|
|
status: todo
|
|||
|
|
priority: high
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Inherited from CB-WP-0005 T05. INTENT design decision 8 of 10,
|
|||
|
|
unimplemented. All four controls from ADR-0005 §6 are required verbatim,
|
|||
|
|
including the two the adversarial reviewer supplied: truncate-by-one-byte
|
|||
|
|
rejection, and a mutated-seed control so the round-trip is capable of
|
|||
|
|
failing.
|
|||
|
|
|
|||
|
|
Fixing this is also what lets **AM-7's withdrawn `hash-identical`
|
|||
|
|
clause** be re-earned, since a real replay is what the clause was always
|
|||
|
|
about.
|
|||
|
|
|
|||
|
|
## Task: K14 and K18 — implement, or amend the spec and say why
|
|||
|
|
|
|||
|
|
```task
|
|||
|
|
id: CB-WP-0006-T07
|
|||
|
|
status: todo
|
|||
|
|
priority: medium
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Inherited from CB-WP-0005 T06. `CommitWindow` has zero non-test users;
|
|||
|
|
the bench never touches `ScenarioFile`. Either wire them up or amend the
|
|||
|
|
rules with a recorded argument. Deleting a rule to make a gate green is
|
|||
|
|
forbidden; deleting one *with an argument* is legitimate and probably
|
|||
|
|
right for K18.
|
|||
|
|
|
|||
|
|
## Phase C — measure, and answer the question this pass inherited
|
|||
|
|
|
|||
|
|
## Task: control loop — M-D1-MUT before and after
|
|||
|
|
|
|||
|
|
```task
|
|||
|
|
id: CB-WP-0006-T08
|
|||
|
|
status: todo
|
|||
|
|
priority: high
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Commit `evidence/CB-EV-0004`. The baseline is **4 of 14**, committed and
|
|||
|
|
reproducible via `make mutation-check`.
|
|||
|
|
|
|||
|
|
Three tests, all reported:
|
|||
|
|
|
|||
|
|
1. **Did the enforced count rise, per row?** Against the per-task
|
|||
|
|
predictions above, unmet reported as unmet.
|
|||
|
|
2. **Did any row regress from `red` to `SURVIVED`?** The four currently
|
|||
|
|
enforced rows are the ones with something to lose.
|
|||
|
|
3. **Were the new mutations strong?** CB-WP-0005 T02 wrote two no-op
|
|||
|
|
mutations on its first attempt, either of which would have published a
|
|||
|
|
false accusation against working code. Every mutation added by this
|
|||
|
|
pass must be shown to fail for the *stated* reason — record how that
|
|||
|
|
was checked.
|
|||
|
|
|
|||
|
|
## Task: retrospective — is a weak mutation the new grep?
|
|||
|
|
|
|||
|
|
```task
|
|||
|
|
id: CB-WP-0006-T09
|
|||
|
|
status: todo
|
|||
|
|
priority: medium
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The question CB-WP-0005 T08 raised and this pass is positioned to answer.
|
|||
|
|
|
|||
|
|
CB-WP-0004 concluded that tooling recovers capacity only where it removes
|
|||
|
|
the manual path. M-D1-MUT does **not** remove a manual path — writing a
|
|||
|
|
weak mutation is exactly as easy as writing a strong one, and the harness
|
|||
|
|
cannot tell the difference. CB-WP-0005 produced two no-op mutations on its
|
|||
|
|
first attempt, so the failure mode is demonstrated, not hypothetical.
|
|||
|
|
|
|||
|
|
**And it fails in the more dangerous direction:** a weak mutation does not
|
|||
|
|
merely miss a defect, it *indicts working code*. A false `SURVIVED` sends
|
|||
|
|
the next pass to fix something that is not broken.
|
|||
|
|
|
|||
|
|
So: is M-D1-MUT a real instrument, or a name-counter with extra steps? If
|
|||
|
|
the latter, say so and propose what would actually bind — the candidate
|
|||
|
|
being that the mutation must be written by someone other than the author
|
|||
|
|
of the assertion, which is the adversarial-review principle applied one
|
|||
|
|
level down.
|