tegwick
fc76445aec
Some checks failed
ci / check (push) Failing after 4s
Written back by fix-consistency: workstream and task UUIDs for the ten CB-WP-0003 tasks, plus the regenerated WORK-RECORDS.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
246 lines
8.8 KiB
Markdown
246 lines
8.8 KiB
Markdown
---
|
||
id: CB-WP-0003
|
||
title: "Harden the inner loop: executable rules, session economics, dead policy"
|
||
status: proposed
|
||
state_hub_workstream_id: "39d61dc0-870d-45c1-a595-bcf91f289dce"
|
||
---
|
||
|
||
# Purpose
|
||
|
||
A rigorous review of the loop after its first full pass (CB-WP-0001)
|
||
found error modes the loop does not catch and policy that costs
|
||
something while delivering nothing. The CI fixes were taken immediately
|
||
(commit `72c594e`); this workplan covers the rest.
|
||
|
||
The finding that frames all of it:
|
||
|
||
> **Writing a rule into `specs/InnerLoop.md` did not prevent the next
|
||
> instance of the failure it was written for. Making it a CI step did.**
|
||
|
||
InnerLoop v1.0 added the positive-control rule after two measurement
|
||
errors. A third occurred immediately afterwards (`default-features`
|
||
silently ignored, build green, feature flag gating nothing), and a
|
||
fourth was found *only* when the rule became a CI step — the committed
|
||
replay benchmark had never run to completion. Four instances of one
|
||
error class; the prose rule prevented none; the executable rule caught
|
||
one on its first run.
|
||
|
||
**Working hypothesis for this workplan: a loop rule that cannot be
|
||
executed is not a rule.** Every item below is either made executable or
|
||
deleted.
|
||
|
||
## Phase A — Make the rules executable
|
||
|
||
## Task: Audit every InnerLoop rule for enforceability
|
||
|
||
```task
|
||
id: CB-WP-0003-T01
|
||
status: todo
|
||
priority: high
|
||
state_hub_task_id: "3d5d45fb-f931-407a-be71-d2d727279d5e"
|
||
```
|
||
|
||
Go through `specs/InnerLoop.md` v1.0 rule by rule and classify each:
|
||
**executable** (a command fails when it is violated), **checkable**
|
||
(a human or agent can verify it in review, cheaply and objectively), or
|
||
**decorative** (neither). Record the table in
|
||
`history/YYMMDD-inner-loop-rule-audit.md`.
|
||
|
||
For each decorative rule, choose one of: make it executable, demote it
|
||
to guidance with that status stated, or delete it. The default is
|
||
delete — an unenforceable rule that reads like a requirement creates
|
||
false assurance, which is worse than silence.
|
||
|
||
Known candidates to start from: whole-file loadability (~400 lines) is
|
||
mechanically checkable and unchecked; "evidence or it didn't happen" is
|
||
executable via a committed-artifact check; the four implementation rules
|
||
added in v1.0 are currently prose.
|
||
|
||
## Task: Extend the positive-control gate beyond benchmarks
|
||
|
||
```task
|
||
id: CB-WP-0003-T02
|
||
status: todo
|
||
priority: high
|
||
state_hub_task_id: "f861e67f-ed09-489d-963d-06697da9c08e"
|
||
```
|
||
|
||
`cargo bench -- --test` now covers benchmarks. The same error class
|
||
lives anywhere a harness can succeed while doing nothing. Extend
|
||
coverage to at least: the scenario runner (done — cb-sim fails on an
|
||
empty run), the coverage tool, `dep-weight`, and any future collector
|
||
including CB-WP-0002's.
|
||
|
||
Deliver a stated contract in `specs/InnerLoop.md`: **every tool that
|
||
reports a number states what it asserts to prove it did the work**, and
|
||
a CI step that fails when a reporting tool has no such assertion. Prefer
|
||
a mechanical check (e.g. each tool exposes `--self-test`) over a
|
||
convention nobody can verify.
|
||
|
||
## Task: Point adversarial review at measurement, not only research
|
||
|
||
```task
|
||
id: CB-WP-0003-T03
|
||
status: todo
|
||
priority: medium
|
||
state_hub_task_id: "1365b35d-4539-489a-beff-c072221e4702"
|
||
```
|
||
|
||
The review step is specified for tier-L *research*. All four real errors
|
||
in this project were in *measurement and build configuration*, which the
|
||
review never touches. We adversarially review the artifact that is
|
||
cheapest to fix and leave unreviewed the one where errors actually
|
||
occur.
|
||
|
||
Revise InnerLoop §Step 2 so the review target follows the risk: for a
|
||
capability whose claim rests on numbers, the review reads the harness
|
||
and the evidence file, not only the survey. Keep it one round; state
|
||
explicitly what the reviewer must attempt (reproduce the number,
|
||
identify what the harness would report if the work silently stopped).
|
||
|
||
## Phase B — Session economics
|
||
|
||
## Task: Measure and specify session shape
|
||
|
||
```task
|
||
id: CB-WP-0003-T04
|
||
status: todo
|
||
priority: high
|
||
state_hub_task_id: "382724e6-efd5-4ef8-a438-9d31a65dacdb"
|
||
```
|
||
|
||
Measured from the CB-WP-0001 session: **592 assistant turns, mean
|
||
context 245,321 tokens, $0.245 per turn in cache reads alone**, and of
|
||
330 tool calls **none** were batched into a multi-call turn. Cache reads
|
||
were 58% of the $248.46 total; output was 12%.
|
||
|
||
Cost is therefore approximately `turns × mean_context`, and mean_context
|
||
grows with turns — a long session is quadratic. Running T01–T09 in one
|
||
context cost ~$145 in cache reads; nine task-scoped sessions at ~40k
|
||
context each would plausibly cost ~$25.
|
||
|
||
Write `specs/SessionShape.md`: one task per session as the default,
|
||
what a fresh session must be able to load from committed artifacts to
|
||
start cold, when to compact versus start clean, and batching of
|
||
independent tool calls. Every claim in it carries the measurement it
|
||
rests on — this spec exists because the numbers were surprising, not
|
||
because the advice sounds sensible.
|
||
|
||
Depends on CB-WP-0002 for per-task attribution to validate the estimate;
|
||
the session-level numbers above are already sufficient to write the
|
||
policy.
|
||
|
||
## Task: Replace the dead token budget with a live cost budget
|
||
|
||
```task
|
||
id: CB-WP-0003-T05
|
||
status: todo
|
||
priority: medium
|
||
state_hub_task_id: "26c920ee-2c09-4b45-a908-4d343532db09"
|
||
```
|
||
|
||
The global 8k soft / 10k hard per-task token budget was never
|
||
referenced or enforced during CB-WP-0001, and T08 exceeded it by orders
|
||
of magnitude with no signal. It is dead policy: it implies a control
|
||
that does not exist.
|
||
|
||
Replace it with a budget expressed in the unit that CB-WP-0002 makes
|
||
measurable, with a defined action on breach and a way to observe the
|
||
breach at the time it happens rather than in a retrospective. If no such
|
||
observation is possible, say so and delete the budget rather than
|
||
restating it.
|
||
|
||
## Phase C — Remove or fix the rest
|
||
|
||
## Task: Resolve the chaos roll
|
||
|
||
```task
|
||
id: CB-WP-0003-T06
|
||
status: todo
|
||
priority: low
|
||
state_hub_task_id: "8f3a6147-54e8-48f9-8543-c13b0e0b0278"
|
||
```
|
||
|
||
The d10 tier roll never fired across the whole pass — roughly 0.2
|
||
expected firings across 2–3 tier decisions — so it is untested, and at
|
||
1-in-10 it will stay untested for many more passes while adding a step
|
||
to every decision.
|
||
|
||
Decide: raise the rate during a stated calibration period so the
|
||
mechanism produces evidence, or delete it. Keeping an unevaluated
|
||
mechanism at a rate that prevents its own evaluation is the one option
|
||
to reject.
|
||
|
||
## Task: Make retargeting a reviewed decision
|
||
|
||
```task
|
||
id: CB-WP-0003-T07
|
||
status: todo
|
||
priority: medium
|
||
state_hub_task_id: "2f702821-7e64-4981-9885-82fa07f638aa"
|
||
```
|
||
|
||
AM-4's new targets were measured at 246,250 and set at 250,000 **in the
|
||
same commit, by the implementer, after seeing the number**. The
|
||
reasoning was recorded and is defensible, but the structure is exactly
|
||
what the loop exists to prevent.
|
||
|
||
Add to InnerLoop: a metric may not be retargeted in the commit that
|
||
measures it. A retarget is an ADR with the old target, the measurement
|
||
that motivated the change, and why the new target binds on future work
|
||
rather than merely passing present work. Apply retroactively to AM-4a
|
||
and AM-4b — either ratify them by ADR or change them.
|
||
|
||
## Task: Give provisional items an expiry
|
||
|
||
```task
|
||
id: CB-WP-0003-T08
|
||
status: todo
|
||
priority: low
|
||
state_hub_task_id: "c2ee91ee-e551-48a7-b1a9-477c34c0690c"
|
||
```
|
||
|
||
Ten U-items plus GR-E02's "successes" are marked `provisional: true`
|
||
with no owner and no review date, so they can shape the kernel
|
||
indefinitely while looking handled.
|
||
|
||
Add an owner and a raised-on date to each provisional item, and make
|
||
`make coverage` report their age. Decide what happens when one goes
|
||
stale — the useful answer is probably that CI warns and the evidence
|
||
file must list them, not that the build breaks.
|
||
|
||
## Task: Strengthen the coverage gate beyond tag-counting
|
||
|
||
```task
|
||
id: CB-WP-0003-T09
|
||
status: todo
|
||
priority: low
|
||
state_hub_task_id: "69ccc9ec-dc35-481c-8065-cef040f07f50"
|
||
```
|
||
|
||
`make coverage` compares rule IDs in the spec against `covers:` lists.
|
||
It proves no rule is unclaimed and no claimed rule is invented. It does
|
||
**not** prove a scenario exercises the rule it names, so 58/58 is weaker
|
||
evidence than it reads as.
|
||
|
||
Cheapest real strengthening to evaluate first: require every GR-id in a
|
||
`covers:` list to also appear in a doc comment in the aggregate, making
|
||
the spec→code→scenario chain mechanical rather than asserted. Consider
|
||
mutation-style checking (does removing the rule's code break the
|
||
scenario that claims it?) and cost it before adopting.
|
||
|
||
## Task: Retrospective
|
||
|
||
```task
|
||
id: CB-WP-0003-T10
|
||
status: todo
|
||
priority: low
|
||
state_hub_task_id: "f43de208-92b1-4f0e-9236-a9d27f3ec451"
|
||
```
|
||
|
||
Revise `specs/InnerLoop.md` to v1.1 from this pass. The question to
|
||
answer honestly: after making rules executable, did the *next* error
|
||
still slip through, and if so, what class was it? Four instances of the
|
||
harness-does-nothing class were found before a gate caught one. Record
|
||
whether the gate holds, and what the fifth error — of whatever class —
|
||
turns out to be.
|