clay-borg/history/260731-assertion-coverage-retrospective.md

2026-07-31 — retrospective: is a weak mutation the new grep?

CB-WP-0005 T08. The pass produced the widened coverage arm, make mutation-check, five corrections to committed evidence, a deferred Phase C, and CB-WP-0006. It measured itself in CB-EV-0004.

The question this task was given

CB-WP-0004 concluded that tooling recovers capacity only where it removes the manual path. Does the same test predict which gates work? Does a mutation gate remove the manual path — or is writing a weak mutation the new grep?

Yes, it is the new grep — and it is worse than grep, because it fails in the opposite direction.

The evidence is not hypothetical. mutation-check's first run produced two SURVIVED verdicts, and both were my own no-op mutations:

• pub struct NullRng; → pub struct NullRng {} — semantically identical
• renaming max_age_days — CA-17 reads it with .get(..., 90)

Two of fourteen rows, on the first attempt, by the author of the tool, in a pass whose entire subject was inert assertions.

Why this is worse than the failure it replaces

Apply CB-WP-0004 T06's test — after this fix, is the manual path impossible, absurd, or merely inferior?

instrument	manual path after the fix	holds?
make env-test	impossible — the build fails if friction returns	yes (0 turns, two passes running)
make task-done	absurd — the old way means knowingly writing fiction	yes (0 turns)
make status	grep is one keystroke away	no
facts-check	patching markdown by hand still works	no
make mutation-check	writing a weak mutation is exactly as easy as writing a strong one	no

The mechanism predicted this correctly, again, for the fifth and sixth instrument. But mutation-check is a new category of failure. make status failing means someone greps instead — the loop is no worse off. A weak mutation publishes a false claim that working code is broken:

The fifth error class: false accusation (FA). Every prior class under-reports — HDN, TA, SSB, DFD all let a real problem pass. FA over-reports. It sends the next pass to fix something that is not broken, and it is more credible than the truth because it arrives with a measurement attached.

Thirteen instances, five classes. Six passes, and every one has produced an instance of a class the previous pass had not seen. CB-WP-0003's retrospective concluded that class-by-class hardening does not converge. Three passes later that conclusion has only strengthened — and the newest class is one that hardening created.

What would actually bind

Stated as a mechanism, per the discipline CB-WP-0004 T06 established.

A mutation is only evidence if it fails for its stated reason. The harness already asserts the mutation applied, that the baseline was green, and that the tree was restored. It cannot assert that the mutation means what its author thought. Three candidates, in order of strength:

1. Mutations are written by someone other than the author of the assertion. The adversarial-review principle applied one level down. This is the only one that structurally removes the manual path, and CB-WP-0006 T09 should evaluate it.
2. Each mutation records its expected failure message, and the harness asserts the observed failure matches. A no-op mutation fails this immediately: it produces no failure at all. Cheap, mechanical, and catches exactly the two defects this pass produced.
3. A SURVIVED verdict is provisional until a second, differently shaped mutation of the same property also survives. Expensive, and it only reduces the false-accusation rate rather than removing it.

(2) is the one to build first and CB-WP-0006 T08 already requires that every new mutation be shown to fail for the stated reason.

Was the prediction discipline worth it?

CB-WP-0004 T06 introduced quantitative prediction before acting. Second outing:

pass	predicted	measured	error
CB-WP-0004	25–30 points recovered	6 points	4–5×
CB-WP-0005	≥10 of 14 rows enforced	4 of 14	2.5×

Both badly wrong, both wrong in the same direction — optimism about how much of the system was already working. That consistency is itself information, and it is the first thing this loop has learned about its own forecasting rather than about its code.

But the value was never the accuracy. CB-WP-0005's prediction was wrong in a way that changed what the pass did: it named a stop condition, the condition fired, Phase C was deferred unstarted, and CB-WP-0006 is scoped to the eight rows the measurement found rather than the three rules the survey predicted. Without the number there was nothing to be wrong about and the pass would have implemented K10/K11 into a table that mostly measures nothing.

That is the first time this project has paid to be told it was wrong and then acted on it. Worth more than the $16.03 the pass cost.

What the loop should change — InnerLoop v1.4

Adopted (was already earned before this pass began). §Step 2's "numbers" row says the reviewer must reproduce the number independently — satisfiable by re-running the command that prints it, which finds nothing of this class. The reviewer found AM-7 by opening a test out of curiosity and said so.

v1.4 §Step 2: where a claim rests on numbers, the reviewer must read the assertion behind each quoted number and mutate it. Re-running the command that prints a number is not verification of that number.

Second instance of a verification step inheriting the author's blindness — CB-WP-0002's dedup blind spot was the first. Both fixes replace re-derivation with adversarial execution, and that now looks like the general shape rather than two coincidences.

Not adopted, and stated so it is not quietly forgotten: a rule that every gate must remove its manual path. It is true of the four instruments measured so far, but mutation-check shows a gate can be worth building while failing that test — the eight unmutatable rows are real findings that nothing else would have produced. The rule is a predictor of whether a gate saves money, not a criterion for whether it is worth having. CB-WP-0004 T06 stated it slightly too strongly and this pass is the counter-example.

Open, not closed

• 8 of 14 acceptance rows have no instrument. CB-WP-0006 Phase A.
• Phase C never started. K9, K10, K11, K14, K18 unimplemented; AM-7's hash clause and AM-11 remain withdrawn/unmet. CB-WP-0006 Phase B.
• facts-check only checks copies that opted in — AM-12's stale $248.46 survived it for four workplans. Untagged copies are reported, not failed, and that gap is now measured at 21 copies.
• Session shape regressed and nothing gates it. Batching 0.0% this pass against a 20% target, mean context 315,170 against 200,000. specs/SessionShape.md has stated SS-01…SS-05 since CB-WP-0003 and none of them has ever been enforced.
• The chaos roll is at declaration 2 of 12 (d4=1, no override).