diff --git a/workplans/CB-WP-0003-loop-hardening.md b/workplans/CB-WP-0003-loop-hardening.md new file mode 100644 index 0000000..65b8c1e --- /dev/null +++ b/workplans/CB-WP-0003-loop-hardening.md @@ -0,0 +1,235 @@ +--- +id: CB-WP-0003 +title: "Harden the inner loop: executable rules, session economics, dead policy" +status: proposed +--- + +# Purpose + +A rigorous review of the loop after its first full pass (CB-WP-0001) +found error modes the loop does not catch and policy that costs +something while delivering nothing. The CI fixes were taken immediately +(commit `72c594e`); this workplan covers the rest. + +The finding that frames all of it: + +> **Writing a rule into `specs/InnerLoop.md` did not prevent the next +> instance of the failure it was written for. Making it a CI step did.** + +InnerLoop v1.0 added the positive-control rule after two measurement +errors. A third occurred immediately afterwards (`default-features` +silently ignored, build green, feature flag gating nothing), and a +fourth was found *only* when the rule became a CI step — the committed +replay benchmark had never run to completion. Four instances of one +error class; the prose rule prevented none; the executable rule caught +one on its first run. + +**Working hypothesis for this workplan: a loop rule that cannot be +executed is not a rule.** Every item below is either made executable or +deleted. + +## Phase A — Make the rules executable + +## Task: Audit every InnerLoop rule for enforceability + +```task +id: CB-WP-0003-T01 +status: todo +priority: high +``` + +Go through `specs/InnerLoop.md` v1.0 rule by rule and classify each: +**executable** (a command fails when it is violated), **checkable** +(a human or agent can verify it in review, cheaply and objectively), or +**decorative** (neither). Record the table in +`history/YYMMDD-inner-loop-rule-audit.md`. + +For each decorative rule, choose one of: make it executable, demote it +to guidance with that status stated, or delete it. The default is +delete — an unenforceable rule that reads like a requirement creates +false assurance, which is worse than silence. + +Known candidates to start from: whole-file loadability (~400 lines) is +mechanically checkable and unchecked; "evidence or it didn't happen" is +executable via a committed-artifact check; the four implementation rules +added in v1.0 are currently prose. + +## Task: Extend the positive-control gate beyond benchmarks + +```task +id: CB-WP-0003-T02 +status: todo +priority: high +``` + +`cargo bench -- --test` now covers benchmarks. The same error class +lives anywhere a harness can succeed while doing nothing. Extend +coverage to at least: the scenario runner (done — cb-sim fails on an +empty run), the coverage tool, `dep-weight`, and any future collector +including CB-WP-0002's. + +Deliver a stated contract in `specs/InnerLoop.md`: **every tool that +reports a number states what it asserts to prove it did the work**, and +a CI step that fails when a reporting tool has no such assertion. Prefer +a mechanical check (e.g. each tool exposes `--self-test`) over a +convention nobody can verify. + +## Task: Point adversarial review at measurement, not only research + +```task +id: CB-WP-0003-T03 +status: todo +priority: medium +``` + +The review step is specified for tier-L *research*. All four real errors +in this project were in *measurement and build configuration*, which the +review never touches. We adversarially review the artifact that is +cheapest to fix and leave unreviewed the one where errors actually +occur. + +Revise InnerLoop §Step 2 so the review target follows the risk: for a +capability whose claim rests on numbers, the review reads the harness +and the evidence file, not only the survey. Keep it one round; state +explicitly what the reviewer must attempt (reproduce the number, +identify what the harness would report if the work silently stopped). + +## Phase B — Session economics + +## Task: Measure and specify session shape + +```task +id: CB-WP-0003-T04 +status: todo +priority: high +``` + +Measured from the CB-WP-0001 session: **592 assistant turns, mean +context 245,321 tokens, $0.245 per turn in cache reads alone**, and of +330 tool calls **none** were batched into a multi-call turn. Cache reads +were 58% of the $248.46 total; output was 12%. + +Cost is therefore approximately `turns × mean_context`, and mean_context +grows with turns — a long session is quadratic. Running T01–T09 in one +context cost ~$145 in cache reads; nine task-scoped sessions at ~40k +context each would plausibly cost ~$25. + +Write `specs/SessionShape.md`: one task per session as the default, +what a fresh session must be able to load from committed artifacts to +start cold, when to compact versus start clean, and batching of +independent tool calls. Every claim in it carries the measurement it +rests on — this spec exists because the numbers were surprising, not +because the advice sounds sensible. + +Depends on CB-WP-0002 for per-task attribution to validate the estimate; +the session-level numbers above are already sufficient to write the +policy. + +## Task: Replace the dead token budget with a live cost budget + +```task +id: CB-WP-0003-T05 +status: todo +priority: medium +``` + +The global 8k soft / 10k hard per-task token budget was never +referenced or enforced during CB-WP-0001, and T08 exceeded it by orders +of magnitude with no signal. It is dead policy: it implies a control +that does not exist. + +Replace it with a budget expressed in the unit that CB-WP-0002 makes +measurable, with a defined action on breach and a way to observe the +breach at the time it happens rather than in a retrospective. If no such +observation is possible, say so and delete the budget rather than +restating it. + +## Phase C — Remove or fix the rest + +## Task: Resolve the chaos roll + +```task +id: CB-WP-0003-T06 +status: todo +priority: low +``` + +The d10 tier roll never fired across the whole pass — roughly 0.2 +expected firings across 2–3 tier decisions — so it is untested, and at +1-in-10 it will stay untested for many more passes while adding a step +to every decision. + +Decide: raise the rate during a stated calibration period so the +mechanism produces evidence, or delete it. Keeping an unevaluated +mechanism at a rate that prevents its own evaluation is the one option +to reject. + +## Task: Make retargeting a reviewed decision + +```task +id: CB-WP-0003-T07 +status: todo +priority: medium +``` + +AM-4's new targets were measured at 246,250 and set at 250,000 **in the +same commit, by the implementer, after seeing the number**. The +reasoning was recorded and is defensible, but the structure is exactly +what the loop exists to prevent. + +Add to InnerLoop: a metric may not be retargeted in the commit that +measures it. A retarget is an ADR with the old target, the measurement +that motivated the change, and why the new target binds on future work +rather than merely passing present work. Apply retroactively to AM-4a +and AM-4b — either ratify them by ADR or change them. + +## Task: Give provisional items an expiry + +```task +id: CB-WP-0003-T08 +status: todo +priority: low +``` + +Ten U-items plus GR-E02's "successes" are marked `provisional: true` +with no owner and no review date, so they can shape the kernel +indefinitely while looking handled. + +Add an owner and a raised-on date to each provisional item, and make +`make coverage` report their age. Decide what happens when one goes +stale — the useful answer is probably that CI warns and the evidence +file must list them, not that the build breaks. + +## Task: Strengthen the coverage gate beyond tag-counting + +```task +id: CB-WP-0003-T09 +status: todo +priority: low +``` + +`make coverage` compares rule IDs in the spec against `covers:` lists. +It proves no rule is unclaimed and no claimed rule is invented. It does +**not** prove a scenario exercises the rule it names, so 58/58 is weaker +evidence than it reads as. + +Cheapest real strengthening to evaluate first: require every GR-id in a +`covers:` list to also appear in a doc comment in the aggregate, making +the spec→code→scenario chain mechanical rather than asserted. Consider +mutation-style checking (does removing the rule's code break the +scenario that claims it?) and cost it before adopting. + +## Task: Retrospective + +```task +id: CB-WP-0003-T10 +status: todo +priority: low +``` + +Revise `specs/InnerLoop.md` to v1.1 from this pass. The question to +answer honestly: after making rules executable, did the *next* error +still slip through, and if so, what class was it? Four instances of the +harness-does-nothing class were found before a gate caught one. Record +whether the gate holds, and what the fifth error — of whatever class — +turns out to be.