Add CB-WP-0003: harden the loop from its own review

Covers what the immediate CI fixes did not. Framing finding: writing a
rule into InnerLoop did not prevent the next instance of the failure it
was written for; making it a CI step caught one on its first run. Four
instances of the harness-does-nothing class, zero prevented by prose.

Working hypothesis: a loop rule that cannot be executed is not a rule.
Each item is made executable or deleted, default delete.

Phase A makes rules executable: audit every InnerLoop rule as
executable/checkable/decorative, extend the positive-control gate beyond
benchmarks, and point adversarial review at measurement rather than only
research — all four real errors were in measurement and build config,
which the review never reads.

Phase B is session economics, from measurement: 592 turns, mean context
245k tokens, $0.245/turn in cache reads, zero batched tool calls, cache
reads 58% of cost against output at 12%. Cost is turns x context and
context grows with turns, so a long session is quadratic. Also replaces
the dead 8k/10k token budget, which was never enforced and which T08
exceeded by orders of magnitude with no signal.

Phase C removes or fixes the rest: resolve the chaos roll (never fired,
and at 1-in-10 will stay unevaluated), require retargeting to be an ADR
rather than something the implementer does in the commit that measures
the metric, give provisional items an owner and expiry, and strengthen
the coverage gate beyond tag-counting.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

workplans/CB-WP-0003-loop-hardening.md

@@ -0,0 +1,235 @@
+---
+id: CB-WP-0003
+title: "Harden the inner loop: executable rules, session economics, dead policy"
+status: proposed
+---
+
+# Purpose
+
+A rigorous review of the loop after its first full pass (CB-WP-0001)
+found error modes the loop does not catch and policy that costs
+something while delivering nothing. The CI fixes were taken immediately
+(commit `72c594e`); this workplan covers the rest.
+
+The finding that frames all of it:
+
+> **Writing a rule into `specs/InnerLoop.md` did not prevent the next
+> instance of the failure it was written for. Making it a CI step did.**
+
+InnerLoop v1.0 added the positive-control rule after two measurement
+errors. A third occurred immediately afterwards (`default-features`
+silently ignored, build green, feature flag gating nothing), and a
+fourth was found *only* when the rule became a CI step — the committed
+replay benchmark had never run to completion. Four instances of one
+error class; the prose rule prevented none; the executable rule caught
+one on its first run.
+
+**Working hypothesis for this workplan: a loop rule that cannot be
+executed is not a rule.** Every item below is either made executable or
+deleted.
+
+## Phase A — Make the rules executable
+
+## Task: Audit every InnerLoop rule for enforceability
+
+```task
+id: CB-WP-0003-T01
+status: todo
+priority: high
+```
+
+Go through `specs/InnerLoop.md` v1.0 rule by rule and classify each:
+**executable** (a command fails when it is violated), **checkable**
+(a human or agent can verify it in review, cheaply and objectively), or
+**decorative** (neither). Record the table in
+`history/YYMMDD-inner-loop-rule-audit.md`.
+
+For each decorative rule, choose one of: make it executable, demote it
+to guidance with that status stated, or delete it. The default is
+delete — an unenforceable rule that reads like a requirement creates
+false assurance, which is worse than silence.
+
+Known candidates to start from: whole-file loadability (~400 lines) is
+mechanically checkable and unchecked; "evidence or it didn't happen" is
+executable via a committed-artifact check; the four implementation rules
+added in v1.0 are currently prose.
+
+## Task: Extend the positive-control gate beyond benchmarks
+
+```task
+id: CB-WP-0003-T02
+status: todo
+priority: high
+```
+
+`cargo bench -- --test` now covers benchmarks. The same error class
+lives anywhere a harness can succeed while doing nothing. Extend
+coverage to at least: the scenario runner (done — cb-sim fails on an
+empty run), the coverage tool, `dep-weight`, and any future collector
+including CB-WP-0002's.
+
+Deliver a stated contract in `specs/InnerLoop.md`: **every tool that
+reports a number states what it asserts to prove it did the work**, and
+a CI step that fails when a reporting tool has no such assertion. Prefer
+a mechanical check (e.g. each tool exposes `--self-test`) over a
+convention nobody can verify.
+
+## Task: Point adversarial review at measurement, not only research
+
+```task
+id: CB-WP-0003-T03
+status: todo
+priority: medium
+```
+
+The review step is specified for tier-L *research*. All four real errors
+in this project were in *measurement and build configuration*, which the
+review never touches. We adversarially review the artifact that is
+cheapest to fix and leave unreviewed the one where errors actually
+occur.
+
+Revise InnerLoop §Step 2 so the review target follows the risk: for a
+capability whose claim rests on numbers, the review reads the harness
+and the evidence file, not only the survey. Keep it one round; state
+explicitly what the reviewer must attempt (reproduce the number,
+identify what the harness would report if the work silently stopped).
+
+## Phase B — Session economics
+
+## Task: Measure and specify session shape
+
+```task
+id: CB-WP-0003-T04
+status: todo
+priority: high
+```
+
+Measured from the CB-WP-0001 session: **592 assistant turns, mean
+context 245,321 tokens, $0.245 per turn in cache reads alone**, and of
+330 tool calls **none** were batched into a multi-call turn. Cache reads
+were 58% of the $248.46 total; output was 12%.
+
+Cost is therefore approximately `turns × mean_context`, and mean_context
+grows with turns — a long session is quadratic. Running T01–T09 in one
+context cost ~$145 in cache reads; nine task-scoped sessions at ~40k
+context each would plausibly cost ~$25.
+
+Write `specs/SessionShape.md`: one task per session as the default,
+what a fresh session must be able to load from committed artifacts to
+start cold, when to compact versus start clean, and batching of
+independent tool calls. Every claim in it carries the measurement it
+rests on — this spec exists because the numbers were surprising, not
+because the advice sounds sensible.
+
+Depends on CB-WP-0002 for per-task attribution to validate the estimate;
+the session-level numbers above are already sufficient to write the
+policy.
+
+## Task: Replace the dead token budget with a live cost budget
+
+```task
+id: CB-WP-0003-T05
+status: todo
+priority: medium
+```
+
+The global 8k soft / 10k hard per-task token budget was never
+referenced or enforced during CB-WP-0001, and T08 exceeded it by orders
+of magnitude with no signal. It is dead policy: it implies a control
+that does not exist.
+
+Replace it with a budget expressed in the unit that CB-WP-0002 makes
+measurable, with a defined action on breach and a way to observe the
+breach at the time it happens rather than in a retrospective. If no such
+observation is possible, say so and delete the budget rather than
+restating it.
+
+## Phase C — Remove or fix the rest
+
+## Task: Resolve the chaos roll
+
+```task
+id: CB-WP-0003-T06
+status: todo
+priority: low
+```
+
+The d10 tier roll never fired across the whole pass — roughly 0.2
+expected firings across 2–3 tier decisions — so it is untested, and at
+1-in-10 it will stay untested for many more passes while adding a step
+to every decision.
+
+Decide: raise the rate during a stated calibration period so the
+mechanism produces evidence, or delete it. Keeping an unevaluated
+mechanism at a rate that prevents its own evaluation is the one option
+to reject.
+
+## Task: Make retargeting a reviewed decision
+
+```task
+id: CB-WP-0003-T07
+status: todo
+priority: medium
+```
+
+AM-4's new targets were measured at 246,250 and set at 250,000 **in the
+same commit, by the implementer, after seeing the number**. The
+reasoning was recorded and is defensible, but the structure is exactly
+what the loop exists to prevent.
+
+Add to InnerLoop: a metric may not be retargeted in the commit that
+measures it. A retarget is an ADR with the old target, the measurement
+that motivated the change, and why the new target binds on future work
+rather than merely passing present work. Apply retroactively to AM-4a
+and AM-4b — either ratify them by ADR or change them.
+
+## Task: Give provisional items an expiry
+
+```task
+id: CB-WP-0003-T08
+status: todo
+priority: low
+```
+
+Ten U-items plus GR-E02's "successes" are marked `provisional: true`
+with no owner and no review date, so they can shape the kernel
+indefinitely while looking handled.
+
+Add an owner and a raised-on date to each provisional item, and make
+`make coverage` report their age. Decide what happens when one goes
+stale — the useful answer is probably that CI warns and the evidence
+file must list them, not that the build breaks.
+
+## Task: Strengthen the coverage gate beyond tag-counting
+
+```task
+id: CB-WP-0003-T09
+status: todo
+priority: low
+```
+
+`make coverage` compares rule IDs in the spec against `covers:` lists.
+It proves no rule is unclaimed and no claimed rule is invented. It does
+**not** prove a scenario exercises the rule it names, so 58/58 is weaker
+evidence than it reads as.
+
+Cheapest real strengthening to evaluate first: require every GR-id in a
+`covers:` list to also appear in a doc comment in the aggregate, making
+the spec→code→scenario chain mechanical rather than asserted. Consider
+mutation-style checking (does removing the rule's code break the
+scenario that claims it?) and cost it before adopting.
+
+## Task: Retrospective
+
+```task
+id: CB-WP-0003-T10
+status: todo
+priority: low
+```
+
+Revise `specs/InnerLoop.md` to v1.1 from this pass. The question to
+answer honestly: after making rules executable, did the *next* error
+still slip through, and if so, what class was it? Four instances of the
+harness-does-nothing class were found before a gate caught one. Record
+whether the gate holds, and what the fifth error — of whatever class —
+turns out to be.