CB-WP-0006: instrument the acceptance table, then implement
Some checks failed
ci / check (push) Failing after 3s
Some checks failed
ci / check (push) Failing after 3s
Scoped to what CB-WP-0005 T02 measured rather than what CB-RES-0004 predicted. The survey diagnosed three absent kernel rules; M-D1-MUT found that 8 of 14 acceptance rows have no instrument behind them at all. Phase A instruments the unmeasured rows, starting with AM-6 — nothing in the workspace compares any number to 100,000 events/s, the project's headline throughput claim, reported as "met, 16.5x". Phase B carries the kernel work deferred unstarted from CB-WP-0005 Phase C. That order is the point: the previous pass built a fix before its instrument existed and the instrument then contradicted the fix's premise. T01 names its own trap up front: a timing assertion in CI is flaky by nature and the usual response is to loosen it until it never fires, which would reproduce exactly the defect being fixed. The threshold is pinned far below measured, states its headroom in the failure message, and loosening requires an ADR. T04 notes an incentive worth blocking early: ADR-0005 predicts own-source growth from Phase B, so AM-4c will move against us. Setting its threshold after seeing that movement is the retarget InnerLoop §Step 4 forbids, so it is set before Phase B or not at all. T09 asks the question this pass is positioned to answer: M-D1-MUT does not remove a manual path — writing a weak mutation is exactly as easy as writing a strong one and the harness cannot tell them apart. CB-WP-0005 produced two no-op mutations on its first attempt, so the failure mode is demonstrated. And it fails in the dangerous direction: a weak mutation does not miss a defect, it indicts working code. CB-WP-0005 T04-T06 are marked deferred rather than deleted, with their analysis intact, so the trail from prediction to contradiction to re-plan stays readable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
ed91ec198e
commit
74d9b8ef73
2 changed files with 234 additions and 3 deletions
|
|
@ -245,7 +245,7 @@ downgrade inline, so a reader of the spec cannot reach the old claim.
|
|||
|
||||
```task
|
||||
id: CB-WP-0005-T04
|
||||
status: todo
|
||||
status: deferred
|
||||
priority: high
|
||||
state_hub_task_id: "35876b28-97ac-4379-bb71-72e74c4116d4"
|
||||
```
|
||||
|
|
@ -273,7 +273,7 @@ the length prefix → reject. K11's operative clause is *detection*.
|
|||
|
||||
```task
|
||||
id: CB-WP-0005-T05
|
||||
status: todo
|
||||
status: deferred
|
||||
priority: high
|
||||
state_hub_task_id: "8592b6e7-8b83-4b47-b310-90e16a35eb6a"
|
||||
```
|
||||
|
|
@ -308,7 +308,7 @@ charged to **AM-4b** (9.4% headroom), not AM-4a.
|
|||
|
||||
```task
|
||||
id: CB-WP-0005-T06
|
||||
status: todo
|
||||
status: deferred
|
||||
priority: medium
|
||||
state_hub_task_id: "774b1c8a-71c1-4e0a-91b1-0247a3e397cf"
|
||||
```
|
||||
|
|
|
|||
231
workplans/CB-WP-0006-instrument-the-table.md
Normal file
231
workplans/CB-WP-0006-instrument-the-table.md
Normal file
|
|
@ -0,0 +1,231 @@
|
|||
---
|
||||
id: CB-WP-0006
|
||||
title: "Instrument the acceptance table, then implement what it exposes"
|
||||
status: proposed
|
||||
---
|
||||
|
||||
# Purpose
|
||||
|
||||
CB-WP-0005 T02 measured M-D1-MUT for the first time:
|
||||
|
||||
```text
|
||||
M-D1-MUT: 4/14 rows enforced
|
||||
PARTIAL 2 (AM-7, AM-8 — some clauses live, some inert)
|
||||
unmutatable 8 (no property to invert)
|
||||
```
|
||||
|
||||
**4 of 14, against a predicted 10 of 14.** The prediction was wrong by
|
||||
2.5×, and wrong about *what* as well as *how much*: CB-RES-0004 diagnosed
|
||||
three absent kernel rules, and the measurement found that **more than half
|
||||
the acceptance table has no instrument behind it at all**.
|
||||
|
||||
CB-WP-0005 Phase C was deferred before starting on that basis (T02's stop
|
||||
condition, maintainer decision 2026-07-31). It was scoped to five rules —
|
||||
K9, K10, K11, K14, K18 — which does not address the eight.
|
||||
|
||||
This workplan is scoped to what was measured rather than what was
|
||||
predicted. **Phase A instruments the unmeasured rows; Phase B implements
|
||||
the kernel work inherited from CB-WP-0005 T04–T06.** That order is the
|
||||
whole point: the previous pass built a fix before its instrument existed,
|
||||
and the instrument then contradicted the fix's premise.
|
||||
|
||||
Per InnerLoop §Step 4, no target moves in the commit that measures it.
|
||||
|
||||
## Phase A — instrument the rows that measure nothing
|
||||
|
||||
## Task: AM-6 — assert the throughput target
|
||||
|
||||
```task
|
||||
id: CB-WP-0006-T01
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
The sharpest single finding of CB-WP-0005. **Nothing in the workspace
|
||||
compares any number to 100,000 events/s** — the project's headline
|
||||
throughput claim, reported as `met, 16.5×`. The Criterion bench emits
|
||||
throughput; the only asserts in `synthetic.rs` are the stress-gate shape
|
||||
and the events-per-round pin.
|
||||
|
||||
Deliver a standing assertion — a test, not a bench — that measures applied
|
||||
events/s on the synthetic workload and fails below the AM-6 target.
|
||||
|
||||
**The trap, stated up front:** a timing assertion in CI is flaky by
|
||||
nature, and the usual response is to loosen it until it never fires, which
|
||||
would reproduce exactly the defect being fixed. So the threshold is
|
||||
**pinned well below the measured 1.65M/s** (AM-6 asks for 100k), the
|
||||
assertion states its own headroom in the failure message, and a loosening
|
||||
requires an ADR. A gate tuned until it cannot fire is a gate that counts
|
||||
names.
|
||||
|
||||
**Verified by:** `make mutation-check --row AM-6` goes from `unmutatable`
|
||||
to `red`.
|
||||
|
||||
## Task: AM-2, AM-3 — instrument the size metrics
|
||||
|
||||
```task
|
||||
id: CB-WP-0006-T02
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
AM-2 (≤40 spec lines per rule) and AM-3 (≤50 LOC for the synthetic
|
||||
workload) are both `unmutatable` for the same reason: `make loc` prints
|
||||
line counts and nothing divides, compares, or gates.
|
||||
|
||||
AM-2 is the **anti-gaming pair** for AM-1 — 100% rule coverage means
|
||||
nothing if the rules are trivially small — so an uninstrumented AM-2
|
||||
leaves AM-1 gameable, which is the specific risk MetricsAndScenarios §1
|
||||
names when it introduces M-D1-SPL.
|
||||
|
||||
AM-3 depends on K18 (benches driven from scenario files); if K18 resolves
|
||||
toward amending the spec rather than implementing it, AM-3 must be
|
||||
restated or withdrawn with an argument rather than left unmeasured.
|
||||
|
||||
## Task: AM-5, AM-9 — measure or withdraw, but stop leaving them blank
|
||||
|
||||
```task
|
||||
id: CB-WP-0006-T03
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
AM-5 (clean build ≤60 s) is `recorded not gated` — and not recorded
|
||||
either. AM-9 (peak RSS ≤64 MB) is declared "very unlikely to bind", which
|
||||
CB-EV-0001 itself calls out as an unmeasured judgment call.
|
||||
|
||||
Both are cheap to measure: one `time` around a clean build, one
|
||||
`/usr/bin/time -v` around the 100k-event run. **Or withdraw them with an
|
||||
argument**, which is a legitimate outcome and possibly the right one for
|
||||
AM-5 on a machine-dependent number. What is not legitimate is a third pass
|
||||
leaving them blank while the scoreboard reports elsewhere.
|
||||
|
||||
## Task: AM-4c — target it or drop it
|
||||
|
||||
```task
|
||||
id: CB-WP-0006-T04
|
||||
status: todo
|
||||
priority: low
|
||||
```
|
||||
|
||||
`reported, not targeted`, so there is no property to invert and it counts
|
||||
against M-D1-MUT by ADR-0005 §1.
|
||||
|
||||
Two honest options, and the task must pick one with an argument: give it a
|
||||
threshold (making it enforceable), or **remove it from the acceptance
|
||||
table** and keep it as a reported diagnostic. A row in an acceptance table
|
||||
that cannot fail is decoration, and this pass has learned to say so.
|
||||
|
||||
Note the incentive: ADR-0005 already predicts own-source growth from the
|
||||
kernel work in Phase B, so AM-4c will move against us. Setting its
|
||||
threshold *after* seeing that movement would be the retarget InnerLoop
|
||||
§Step 4 forbids — so it is set here, before Phase B, or not at all.
|
||||
|
||||
## Phase B — the kernel work inherited from CB-WP-0005
|
||||
|
||||
> These three carry their full analysis in
|
||||
> [CB-WP-0005](CB-WP-0005-assertion-coverage.md) Phase C, where they were
|
||||
> written and then deferred unstarted. They are summarized here; read that
|
||||
> file for the detail, the four replay controls, and the D2 correction on
|
||||
> what the bundle writer actually requires.
|
||||
|
||||
## Task: K9 and K11 — durable log, LogStore port, real conformance suite
|
||||
|
||||
```task
|
||||
id: CB-WP-0006-T05
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Inherited from CB-WP-0005 T04. K9's acceptance property (snapshot at N +
|
||||
events N+1..M ≡ genesis fold) is mutation-proven unasserted; K11 has no
|
||||
durable format at all. The `LogStore` port with a shared
|
||||
`fn conformance<S: LogStore>(…)`, retro-fitted to `KernelRng`, is what
|
||||
finally earns **AM-11** — currently downgraded to unmet because the pair
|
||||
exists and the suite does not.
|
||||
|
||||
**Verified by:** `make mutation-check --row AM-11` goes from
|
||||
`unmutatable` to `red`.
|
||||
|
||||
## Task: K10 — the replay bundle and `--replay`
|
||||
|
||||
```task
|
||||
id: CB-WP-0006-T06
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Inherited from CB-WP-0005 T05. INTENT design decision 8 of 10,
|
||||
unimplemented. All four controls from ADR-0005 §6 are required verbatim,
|
||||
including the two the adversarial reviewer supplied: truncate-by-one-byte
|
||||
rejection, and a mutated-seed control so the round-trip is capable of
|
||||
failing.
|
||||
|
||||
Fixing this is also what lets **AM-7's withdrawn `hash-identical`
|
||||
clause** be re-earned, since a real replay is what the clause was always
|
||||
about.
|
||||
|
||||
## Task: K14 and K18 — implement, or amend the spec and say why
|
||||
|
||||
```task
|
||||
id: CB-WP-0006-T07
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Inherited from CB-WP-0005 T06. `CommitWindow` has zero non-test users;
|
||||
the bench never touches `ScenarioFile`. Either wire them up or amend the
|
||||
rules with a recorded argument. Deleting a rule to make a gate green is
|
||||
forbidden; deleting one *with an argument* is legitimate and probably
|
||||
right for K18.
|
||||
|
||||
## Phase C — measure, and answer the question this pass inherited
|
||||
|
||||
## Task: control loop — M-D1-MUT before and after
|
||||
|
||||
```task
|
||||
id: CB-WP-0006-T08
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Commit `evidence/CB-EV-0004`. The baseline is **4 of 14**, committed and
|
||||
reproducible via `make mutation-check`.
|
||||
|
||||
Three tests, all reported:
|
||||
|
||||
1. **Did the enforced count rise, per row?** Against the per-task
|
||||
predictions above, unmet reported as unmet.
|
||||
2. **Did any row regress from `red` to `SURVIVED`?** The four currently
|
||||
enforced rows are the ones with something to lose.
|
||||
3. **Were the new mutations strong?** CB-WP-0005 T02 wrote two no-op
|
||||
mutations on its first attempt, either of which would have published a
|
||||
false accusation against working code. Every mutation added by this
|
||||
pass must be shown to fail for the *stated* reason — record how that
|
||||
was checked.
|
||||
|
||||
## Task: retrospective — is a weak mutation the new grep?
|
||||
|
||||
```task
|
||||
id: CB-WP-0006-T09
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
The question CB-WP-0005 T08 raised and this pass is positioned to answer.
|
||||
|
||||
CB-WP-0004 concluded that tooling recovers capacity only where it removes
|
||||
the manual path. M-D1-MUT does **not** remove a manual path — writing a
|
||||
weak mutation is exactly as easy as writing a strong one, and the harness
|
||||
cannot tell the difference. CB-WP-0005 produced two no-op mutations on its
|
||||
first attempt, so the failure mode is demonstrated, not hypothetical.
|
||||
|
||||
**And it fails in the more dangerous direction:** a weak mutation does not
|
||||
merely miss a defect, it *indicts working code*. A false `SURVIVED` sends
|
||||
the next pass to fix something that is not broken.
|
||||
|
||||
So: is M-D1-MUT a real instrument, or a name-counter with extra steps? If
|
||||
the latter, say so and propose what would actually bind — the candidate
|
||||
being that the mutation must be written by someone other than the author
|
||||
of the assertion, which is the adversarial-review principle applied one
|
||||
level down.
|
||||
Loading…
Add table
Add a link
Reference in a new issue