clay-borg/workplans/CB-WP-0003-loop-hardening.md

---
id: CB-WP-0003
title: "Harden the inner loop: executable rules, session economics, dead policy"
status: proposed
---

# Purpose

A rigorous review of the loop after its first full pass (CB-WP-0001)
found error modes the loop does not catch and policy that costs
something while delivering nothing. The CI fixes were taken immediately
(commit `72c594e`); this workplan covers the rest.

The finding that frames all of it:

> **Writing a rule into `specs/InnerLoop.md` did not prevent the next
> instance of the failure it was written for. Making it a CI step did.**

InnerLoop v1.0 added the positive-control rule after two measurement
errors. A third occurred immediately afterwards (`default-features`
silently ignored, build green, feature flag gating nothing), and a
fourth was found *only* when the rule became a CI step — the committed
replay benchmark had never run to completion. Four instances of one
error class; the prose rule prevented none; the executable rule caught
one on its first run.

**Working hypothesis for this workplan: a loop rule that cannot be
executed is not a rule.** Every item below is either made executable or
deleted.

## Phase A — Make the rules executable

## Task: Audit every InnerLoop rule for enforceability

```task
id: CB-WP-0003-T01
status: todo
priority: high
```

Go through `specs/InnerLoop.md` v1.0 rule by rule and classify each:
**executable** (a command fails when it is violated), **checkable**
(a human or agent can verify it in review, cheaply and objectively), or
**decorative** (neither). Record the table in
`history/YYMMDD-inner-loop-rule-audit.md`.

For each decorative rule, choose one of: make it executable, demote it
to guidance with that status stated, or delete it. The default is
delete — an unenforceable rule that reads like a requirement creates
false assurance, which is worse than silence.

Known candidates to start from: whole-file loadability (~400 lines) is
mechanically checkable and unchecked; "evidence or it didn't happen" is
executable via a committed-artifact check; the four implementation rules
added in v1.0 are currently prose.

## Task: Extend the positive-control gate beyond benchmarks

```task
id: CB-WP-0003-T02
status: todo
priority: high
```

`cargo bench -- --test` now covers benchmarks. The same error class
lives anywhere a harness can succeed while doing nothing. Extend
coverage to at least: the scenario runner (done — cb-sim fails on an
empty run), the coverage tool, `dep-weight`, and any future collector
including CB-WP-0002's.

Deliver a stated contract in `specs/InnerLoop.md`: **every tool that
reports a number states what it asserts to prove it did the work**, and
a CI step that fails when a reporting tool has no such assertion. Prefer
a mechanical check (e.g. each tool exposes `--self-test`) over a
convention nobody can verify.

## Task: Point adversarial review at measurement, not only research

```task
id: CB-WP-0003-T03
status: todo
priority: medium
```

The review step is specified for tier-L *research*. All four real errors
in this project were in *measurement and build configuration*, which the
review never touches. We adversarially review the artifact that is
cheapest to fix and leave unreviewed the one where errors actually
occur.

Revise InnerLoop §Step 2 so the review target follows the risk: for a
capability whose claim rests on numbers, the review reads the harness
and the evidence file, not only the survey. Keep it one round; state
explicitly what the reviewer must attempt (reproduce the number,
identify what the harness would report if the work silently stopped).

## Phase B — Session economics

## Task: Measure and specify session shape

```task
id: CB-WP-0003-T04
status: todo
priority: high
```

Measured from the CB-WP-0001 session: **592 assistant turns, mean
context 245,321 tokens, $0.245 per turn in cache reads alone**, and of
330 tool calls **none** were batched into a multi-call turn. Cache reads
were 58% of the $248.46 total; output was 12%.

Cost is therefore approximately `turns × mean_context`, and mean_context
grows with turns — a long session is quadratic. Running T01–T09 in one
context cost ~$145 in cache reads; nine task-scoped sessions at ~40k
context each would plausibly cost ~$25.

Write `specs/SessionShape.md`: one task per session as the default,
what a fresh session must be able to load from committed artifacts to
start cold, when to compact versus start clean, and batching of
independent tool calls. Every claim in it carries the measurement it
rests on — this spec exists because the numbers were surprising, not
because the advice sounds sensible.

Depends on CB-WP-0002 for per-task attribution to validate the estimate;
the session-level numbers above are already sufficient to write the
policy.

## Task: Replace the dead token budget with a live cost budget

```task
id: CB-WP-0003-T05
status: todo
priority: medium
```

The global 8k soft / 10k hard per-task token budget was never
referenced or enforced during CB-WP-0001, and T08 exceeded it by orders
of magnitude with no signal. It is dead policy: it implies a control
that does not exist.

Replace it with a budget expressed in the unit that CB-WP-0002 makes
measurable, with a defined action on breach and a way to observe the
breach at the time it happens rather than in a retrospective. If no such
observation is possible, say so and delete the budget rather than
restating it.

## Phase C — Remove or fix the rest

## Task: Resolve the chaos roll

```task
id: CB-WP-0003-T06
status: todo
priority: low
```

The d10 tier roll never fired across the whole pass — roughly 0.2
expected firings across 2–3 tier decisions — so it is untested, and at
1-in-10 it will stay untested for many more passes while adding a step
to every decision.

Decide: raise the rate during a stated calibration period so the
mechanism produces evidence, or delete it. Keeping an unevaluated
mechanism at a rate that prevents its own evaluation is the one option
to reject.

## Task: Make retargeting a reviewed decision

```task
id: CB-WP-0003-T07
status: todo
priority: medium
```

AM-4's new targets were measured at 246,250 and set at 250,000 **in the
same commit, by the implementer, after seeing the number**. The
reasoning was recorded and is defensible, but the structure is exactly
what the loop exists to prevent.

Add to InnerLoop: a metric may not be retargeted in the commit that
measures it. A retarget is an ADR with the old target, the measurement
that motivated the change, and why the new target binds on future work
rather than merely passing present work. Apply retroactively to AM-4a
and AM-4b — either ratify them by ADR or change them.

## Task: Give provisional items an expiry

```task
id: CB-WP-0003-T08
status: todo
priority: low
```

Ten U-items plus GR-E02's "successes" are marked `provisional: true`
with no owner and no review date, so they can shape the kernel
indefinitely while looking handled.

Add an owner and a raised-on date to each provisional item, and make
`make coverage` report their age. Decide what happens when one goes
stale — the useful answer is probably that CI warns and the evidence
file must list them, not that the build breaks.

## Task: Strengthen the coverage gate beyond tag-counting

```task
id: CB-WP-0003-T09
status: todo
priority: low
```

`make coverage` compares rule IDs in the spec against `covers:` lists.
It proves no rule is unclaimed and no claimed rule is invented. It does
**not** prove a scenario exercises the rule it names, so 58/58 is weaker
evidence than it reads as.

Cheapest real strengthening to evaluate first: require every GR-id in a
`covers:` list to also appear in a doc comment in the aggregate, making
the spec→code→scenario chain mechanical rather than asserted. Consider
mutation-style checking (does removing the rule's code break the
scenario that claims it?) and cost it before adopting.

## Task: Retrospective

```task
id: CB-WP-0003-T10
status: todo
priority: low
```

Revise `specs/InnerLoop.md` to v1.1 from this pass. The question to
answer honestly: after making rules executable, did the *next* error
still slip through, and if so, what class was it? Four instances of the
harness-does-nothing class were found before a gate caught one. Record
whether the gate holds, and what the fifth error — of whatever class —
turns out to be.