clay-borg/workplans/CB-WP-0003-loop-hardening.md

id	title	status	state_hub_workstream_id
CB-WP-0003	Harden the inner loop: executable rules, session economics, dead policy	proposed	39d61dc0-870d-45c1-a595-bcf91f289dce

Purpose

A rigorous review of the loop after its first full pass (CB-WP-0001) found error modes the loop does not catch and policy that costs something while delivering nothing. The CI fixes were taken immediately (commit 72c594e); this workplan covers the rest.

The finding that frames all of it:

Writing a rule into specs/InnerLoop.md did not prevent the next instance of the failure it was written for. Making it a CI step did.

InnerLoop v1.0 added the positive-control rule after two measurement errors. A third occurred immediately afterwards (default-features silently ignored, build green, feature flag gating nothing), and a fourth was found only when the rule became a CI step — the committed replay benchmark had never run to completion. Four instances of one error class; the prose rule prevented none; the executable rule caught one on its first run.

Working hypothesis for this workplan: a loop rule that cannot be executed is not a rule. Every item below is either made executable or deleted.

Phase A — Make the rules executable

Task: Audit every InnerLoop rule for enforceability

id: CB-WP-0003-T01
status: todo
priority: high
state_hub_task_id: "3d5d45fb-f931-407a-be71-d2d727279d5e"

Go through specs/InnerLoop.md v1.0 rule by rule and classify each: executable (a command fails when it is violated), checkable (a human or agent can verify it in review, cheaply and objectively), or decorative (neither). Record the table in history/YYMMDD-inner-loop-rule-audit.md.

For each decorative rule, choose one of: make it executable, demote it to guidance with that status stated, or delete it. The default is delete — an unenforceable rule that reads like a requirement creates false assurance, which is worse than silence.

Known candidates to start from: whole-file loadability (~400 lines) is mechanically checkable and unchecked; "evidence or it didn't happen" is executable via a committed-artifact check; the four implementation rules added in v1.0 are currently prose.

Task: Extend the positive-control gate beyond benchmarks

id: CB-WP-0003-T02
status: todo
priority: high
state_hub_task_id: "f861e67f-ed09-489d-963d-06697da9c08e"

cargo bench -- --test now covers benchmarks. The same error class lives anywhere a harness can succeed while doing nothing. Extend coverage to at least: the scenario runner (done — cb-sim fails on an empty run), the coverage tool, dep-weight, and any future collector including CB-WP-0002's.

Deliver a stated contract in specs/InnerLoop.md: every tool that reports a number states what it asserts to prove it did the work, and a CI step that fails when a reporting tool has no such assertion. Prefer a mechanical check (e.g. each tool exposes --self-test) over a convention nobody can verify.

Task: Point adversarial review at measurement, not only research

id: CB-WP-0003-T03
status: todo
priority: medium
state_hub_task_id: "1365b35d-4539-489a-beff-c072221e4702"

The review step is specified for tier-L research. All four real errors in this project were in measurement and build configuration, which the review never touches. We adversarially review the artifact that is cheapest to fix and leave unreviewed the one where errors actually occur.

Revise InnerLoop §Step 2 so the review target follows the risk: for a capability whose claim rests on numbers, the review reads the harness and the evidence file, not only the survey. Keep it one round; state explicitly what the reviewer must attempt (reproduce the number, identify what the harness would report if the work silently stopped).

Phase B — Session economics

Task: Measure and specify session shape

id: CB-WP-0003-T04
status: todo
priority: high
state_hub_task_id: "382724e6-efd5-4ef8-a438-9d31a65dacdb"

Measured from the CB-WP-0001 session: 592 assistant turns, mean context 245,321 tokens, $0.245 per turn in cache reads alone, and of 330 tool calls none were batched into a multi-call turn. Cache reads were 58% of the $248.46 total; output was 12%.

Cost is therefore approximately turns × mean_context, and mean_context grows with turns — a long session is quadratic. Running T01–T09 in one context cost ~$145 in cache reads; nine task-scoped sessions at ~40k context each would plausibly cost ~$25.

Write specs/SessionShape.md: one task per session as the default, what a fresh session must be able to load from committed artifacts to start cold, when to compact versus start clean, and batching of independent tool calls. Every claim in it carries the measurement it rests on — this spec exists because the numbers were surprising, not because the advice sounds sensible.

Depends on CB-WP-0002 for per-task attribution to validate the estimate; the session-level numbers above are already sufficient to write the policy.

Task: Replace the dead token budget with a live cost budget

id: CB-WP-0003-T05
status: todo
priority: medium
state_hub_task_id: "26c920ee-2c09-4b45-a908-4d343532db09"

The global 8k soft / 10k hard per-task token budget was never referenced or enforced during CB-WP-0001, and T08 exceeded it by orders of magnitude with no signal. It is dead policy: it implies a control that does not exist.

Replace it with a budget expressed in the unit that CB-WP-0002 makes measurable, with a defined action on breach and a way to observe the breach at the time it happens rather than in a retrospective. If no such observation is possible, say so and delete the budget rather than restating it.

Phase C — Remove or fix the rest

Task: Resolve the chaos roll

id: CB-WP-0003-T06
status: todo
priority: low
state_hub_task_id: "8f3a6147-54e8-48f9-8543-c13b0e0b0278"

The d10 tier roll never fired across the whole pass — roughly 0.2 expected firings across 2–3 tier decisions — so it is untested, and at 1-in-10 it will stay untested for many more passes while adding a step to every decision.

Decide: raise the rate during a stated calibration period so the mechanism produces evidence, or delete it. Keeping an unevaluated mechanism at a rate that prevents its own evaluation is the one option to reject.

Task: Make retargeting a reviewed decision

id: CB-WP-0003-T07
status: todo
priority: medium
state_hub_task_id: "2f702821-7e64-4981-9885-82fa07f638aa"

AM-4's new targets were measured at 246,250 and set at 250,000 in the same commit, by the implementer, after seeing the number. The reasoning was recorded and is defensible, but the structure is exactly what the loop exists to prevent.

Add to InnerLoop: a metric may not be retargeted in the commit that measures it. A retarget is an ADR with the old target, the measurement that motivated the change, and why the new target binds on future work rather than merely passing present work. Apply retroactively to AM-4a and AM-4b — either ratify them by ADR or change them.

Task: Give provisional items an expiry

id: CB-WP-0003-T08
status: todo
priority: low
state_hub_task_id: "c2ee91ee-e551-48a7-b1a9-477c34c0690c"

Ten U-items plus GR-E02's "successes" are marked provisional: true with no owner and no review date, so they can shape the kernel indefinitely while looking handled.

Add an owner and a raised-on date to each provisional item, and make make coverage report their age. Decide what happens when one goes stale — the useful answer is probably that CI warns and the evidence file must list them, not that the build breaks.

Task: Strengthen the coverage gate beyond tag-counting

id: CB-WP-0003-T09
status: todo
priority: low
state_hub_task_id: "69ccc9ec-dc35-481c-8065-cef040f07f50"

make coverage compares rule IDs in the spec against covers: lists. It proves no rule is unclaimed and no claimed rule is invented. It does not prove a scenario exercises the rule it names, so 58/58 is weaker evidence than it reads as.

Cheapest real strengthening to evaluate first: require every GR-id in a covers: list to also appear in a doc comment in the aggregate, making the spec→code→scenario chain mechanical rather than asserted. Consider mutation-style checking (does removing the rule's code break the scenario that claims it?) and cost it before adopting.

Task: Retrospective

id: CB-WP-0003-T10
status: todo
priority: low
state_hub_task_id: "f43de208-92b1-4f0e-9236-a9d27f3ec451"

Revise specs/InnerLoop.md to v1.1 from this pass. The question to answer honestly: after making rules executable, did the next error still slip through, and if so, what class was it? Four instances of the harness-does-nothing class were found before a gate caught one. Record whether the gate holds, and what the fifth error — of whatever class — turns out to be.