clay-borg/history/260731-assertion-coverage-retrospective.md

# 2026-07-31 — retrospective: is a weak mutation the new grep?

CB-WP-0005 T08. The pass produced the widened coverage arm,
`make mutation-check`, five corrections to committed evidence, a deferred
Phase C, and CB-WP-0006. It measured itself in
[CB-EV-0004](../evidence/CB-EV-0004-assertion-coverage.md).

## The question this task was given

> **CB-WP-0004 concluded that tooling recovers capacity only where it
> removes the manual path. Does the same test predict which gates work?
> Does a mutation gate remove the manual path — or is writing a weak
> mutation the new `grep`?**

**Yes, it is the new `grep` — and it is worse than `grep`, because it
fails in the opposite direction.**

The evidence is not hypothetical. `mutation-check`'s first run produced
two `SURVIVED` verdicts, and **both were my own no-op mutations**:

- `pub struct NullRng;` → `pub struct NullRng {}` — semantically identical
- renaming `max_age_days` — CA-17 reads it with `.get(..., 90)`

Two of fourteen rows, on the first attempt, by the author of the tool,
in a pass whose entire subject was inert assertions.

## Why this is worse than the failure it replaces

Apply CB-WP-0004 T06's test — *after this fix, is the manual path
impossible, absurd, or merely inferior?*

| instrument | manual path after the fix | holds? |
|---|---|---|
| `make env-test` | impossible — the build fails if friction returns | **yes** (0 turns, two passes running) |
| `make task-done` | absurd — the old way means knowingly writing fiction | **yes** (0 turns) |
| `make status` | `grep` is one keystroke away | no |
| `facts-check` | patching markdown by hand still works | no |
| **`make mutation-check`** | **writing a weak mutation is exactly as easy as writing a strong one** | **no** |

The mechanism predicted this correctly, again, for the fifth and sixth
instrument. But mutation-check is a new category of failure. `make status`
failing means someone greps instead — the loop is no worse off. A weak
mutation **publishes a false claim that working code is broken**:

> **The fifth error class: false accusation (FA).** Every prior class
> under-reports — HDN, TA, SSB, DFD all let a real problem pass. FA
> over-reports. It sends the next pass to fix something that is not
> broken, and it is *more* credible than the truth because it arrives with
> a measurement attached.

Thirteen instances, five classes. **Six passes, and every one has produced
an instance of a class the previous pass had not seen.** CB-WP-0003's
retrospective concluded that class-by-class hardening does not converge.
Three passes later that conclusion has only strengthened — and the newest
class is one that hardening *created*.

## What would actually bind

Stated as a mechanism, per the discipline CB-WP-0004 T06 established.

A mutation is only evidence if it **fails for its stated reason**. The
harness already asserts the mutation applied, that the baseline was green,
and that the tree was restored. It cannot assert that the mutation
*means* what its author thought. Three candidates, in order of strength:

1. **Mutations are written by someone other than the author of the
   assertion.** The adversarial-review principle applied one level down.
   This is the only one that structurally removes the manual path, and
   CB-WP-0006 T09 should evaluate it.
2. **Each mutation records its expected failure message, and the harness
   asserts the observed failure matches.** A no-op mutation fails this
   immediately: it produces no failure at all. Cheap, mechanical, and
   catches exactly the two defects this pass produced.
3. **A `SURVIVED` verdict is provisional until a second, differently
   shaped mutation of the same property also survives.** Expensive, and it
   only reduces the false-accusation rate rather than removing it.

**(2) is the one to build first** and CB-WP-0006 T08 already requires that
every new mutation be *shown to fail for the stated reason*.

## Was the prediction discipline worth it?

CB-WP-0004 T06 introduced quantitative prediction before acting. Second
outing:

| pass | predicted | measured | error |
|---|---|---|---|
| CB-WP-0004 | 25–30 points recovered | 6 points | 4–5× |
| CB-WP-0005 | ≥10 of 14 rows enforced | 4 of 14 | 2.5× |

Both badly wrong, both wrong in the same direction — **optimism about how
much of the system was already working**. That consistency is itself
information, and it is the first thing this loop has learned about its own
forecasting rather than about its code.

But the value was never the accuracy. **CB-WP-0005's prediction was wrong
in a way that changed what the pass did**: it named a stop condition, the
condition fired, Phase C was deferred unstarted, and CB-WP-0006 is scoped
to the eight rows the measurement found rather than the three rules the
survey predicted. Without the number there was nothing to be wrong about
and the pass would have implemented K10/K11 into a table that mostly
measures nothing.

**That is the first time this project has paid to be told it was wrong and
then acted on it.** Worth more than the $16.03 the pass cost.

## What the loop should change — InnerLoop v1.4

**Adopted (was already earned before this pass began).** §Step 2's
"numbers" row says the reviewer must *reproduce the number
independently* — satisfiable by re-running the command that prints it,
which finds nothing of this class. The reviewer found AM-7 by opening a
test out of curiosity and said so.

> **v1.4 §Step 2:** where a claim rests on numbers, the reviewer must
> **read the assertion behind each quoted number and mutate it**. Re-running
> the command that prints a number is not verification of that number.

Second instance of a verification step inheriting the author's blindness —
CB-WP-0002's dedup blind spot was the first. Both fixes replace
re-derivation with **adversarial execution**, and that now looks like the
general shape rather than two coincidences.

**Not adopted, and stated so it is not quietly forgotten:** a rule that
every gate must remove its manual path. It is true of the four instruments
measured so far, but `mutation-check` shows a gate can be worth building
while failing that test — the eight `unmutatable` rows are real findings
that nothing else would have produced. The rule is a *predictor of
whether a gate saves money*, not a criterion for whether it is worth
having. CB-WP-0004 T06 stated it slightly too strongly and this pass is
the counter-example.

## Open, not closed

- **8 of 14 acceptance rows have no instrument.** CB-WP-0006 Phase A.
- **Phase C never started.** K9, K10, K11, K14, K18 unimplemented;
  AM-7's hash clause and AM-11 remain withdrawn/unmet. CB-WP-0006 Phase B.
- **`facts-check` only checks copies that opted in** — AM-12's stale
  $248.46 survived it for four workplans. Untagged copies are reported,
  not failed, and that gap is now measured at 21 copies.
- **Session shape regressed and nothing gates it.** Batching **0.0%**
  this pass against a 20% target, mean context 315,170 against 200,000.
  `specs/SessionShape.md` has stated SS-01…SS-05 since CB-WP-0003 and
  none of them has ever been enforced.
- **The chaos roll is at declaration 2 of 12** (d4=1, no override).