coulomb/clay-borg
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Revise CB-WP-0003 from CB-WP-0002's findings
Some checks failed
ci / check (push) Failing after 4s
Details

Browse source 
The framing hypothesis -- a rule that cannot be executed is not a rule
-- survives but was too coarse. CB-WP-0002 corrected one number four
times and each correction came from a DIFFERENT mechanism, none finding
more than one. Two limits are now measured rather than assumed:

  - a positive control cannot catch trusted arithmetic (both errors in
    $248.46 were sums over data that really existed)
  - adversarial review cannot catch a same-sample blind spot: survey and
    reviewer both verified dedup on the main transcript, and it is false
    in the subagent tree neither examined

Task-level revisions:

  T01  also classify each rule by WHICH failure class it catches; seven
       instances across three classes are now on record
  T02  scope reduced -- the --self-test contract, cb-cost's five
       assertions, make cost-test in CI are delivered; remaining work is
       rule-coverage/dep-weight plus a check that new tools carry one
  T03  now backed by two measurements ($0.66, $1.11 at ~1% of a pass),
       and must also state what review CANNOT do
  T04  premise overturned -- sessions are bounded-quadratic, not
       quadratic. /compact cut context 27x and cost/turn 3.1x. Measure
       compaction against a fresh session before prescribing either
  T05  unblocked, with calibration data ($1.29-$21.02/task) and the
       constraint that a budget binds only on the 67.5% attribution
       reaches, only after the closing commit
  T07  sharpened -- CB-WP-0002 retargeted three times, correctly, so a
       blanket prohibition is wrong; separate instrument-disproved-target
       from implementation-missed-target
  T10  its prediction is already falsified; the successor question is
       whether class-by-class hardening converges at all
  T11  NEW -- the price sheet cannot express a dated rate and becomes
       wrong on 2026-08-31

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 09:11:08 +02:00
parent 85b86e3805
commit 06b3df6eca
1 changed files with 199 additions and 72 deletions
Download patch file Download diff file Expand all files Collapse all files

271
workplans/CB-WP-0003-loop-hardening.md
View file

@ -12,22 +12,42 @@ found error modes the loop does not catch and policy that costs
something while delivering nothing. The CI fixes were taken immediately
(commit `72c594e`); this workplan covers the rest.
The finding that frames all of it:
The finding that framed it:
> **Writing a rule into `specs/InnerLoop.md` did not prevent the next
> instance of the failure it was written for. Making it a CI step did.**
InnerLoop v1.0 added the positive-control rule after two measurement
errors. A third occurred immediately afterwards (`default-features`
silently ignored, build green, feature flag gating nothing), and a
fourth was found *only* when the rule became a CI step — the committed
replay benchmark had never run to completion. Four instances of one
error class; the prose rule prevented none; the executable rule caught
one on its first run.
**Revised 2026-07-31 from CB-WP-0002**, which was run first and changed
the premise. The original hypothesis — *a loop rule that cannot be
executed is not a rule* — survives but is now too coarse. CB-WP-0002
corrected one number four times ($248.46 → $92.21 → $92.87 → $93.32),
and **each correction came from a different mechanism**:
**Working hypothesis for this workplan: a loop rule that cannot be
executed is not a rule.** Every item below is either made executable or
deleted.
| mechanism | caught | cost |
|---|---|---|
| re-deriving instead of quoting | inherited double-count + single-model pricing | ~0 |
| adversarial review | target contradicted the survey's own finding | $1.11 |
| executable assertion (`--self-test`) | dedup invariant false in the subagent tree | ~0 |
**No mechanism found more than one.** So the goal is not to convert every
rule into a CI step; it is to know which class each mechanism catches and
to stop expecting one to cover another. Two specific limits are now
measured rather than assumed:
1. **A positive control cannot catch trusted arithmetic.** Both errors in
$248.46 were sums over data that genuinely existed. An assertion of the
form "did this harness do work?" answers yes, correctly, and the number
is still wrong.
2. **Adversarial review cannot catch a same-sample blind spot.** The dedup
invariant was verified on the main transcript by the survey *and*
independently by the reviewer; it is false in the 8-response subagent
tree neither examined. A reviewer's job is to re-derive the author's
claims, and re-deriving on the same sample reproduces the same gap.
CB-WP-0002 also falsified this workplan's own prediction (old T10): the
fifth error was **not** of the harness-does-nothing class. Two more
instances arrived, of two new classes. The retrospective task is rewritten
accordingly.
## Phase A — Make the rules executable
@ -40,7 +60,7 @@ priority: high
state_hub_task_id: "3d5d45fb-f931-407a-be71-d2d727279d5e"
```
Go through `specs/InnerLoop.md` v1.0 rule by rule and classify each:
Go through `specs/InnerLoop.md` **v1.1** rule by rule and classify each:
**executable** (a command fails when it is violated), **checkable**
(a human or agent can verify it in review, cheaply and objectively), or
**decorative** (neither). Record the table in
@ -51,10 +71,19 @@ to guidance with that status stated, or delete it. The default is
delete — an unenforceable rule that reads like a requirement creates
false assurance, which is worse than silence.
**Revised:** classification is no longer enough. For each rule also record
**which failure class it catches**, drawn from the seven instances now on
record (four harness-does-nothing, two trusted-arithmetic, one
same-sample-blind-spot). A rule that catches a class no observed error
belongs to is a candidate for deletion even if it is perfectly executable.
Conversely, a class with no rule covering it is the gap worth the next
rule.
Known candidates to start from: whole-file loadability (~400 lines) is
mechanically checkable and unchecked; "evidence or it didn't happen" is
executable via a committed-artifact check; the four implementation rules
added in v1.0 are currently prose.
mechanically checkable and unchecked — `specs/InnerLoop.md` is now near
that limit and should be measured first; "evidence or it didn't happen" is
executable via a committed-artifact check; the four v1.0 implementation
rules and the three v1.1 rules are currently prose.
## Task: Extend the positive-control gate beyond benchmarks
@ -65,19 +94,28 @@ priority: high
state_hub_task_id: "f861e67f-ed09-489d-963d-06697da9c08e"
```
`cargo bench -- --test` now covers benchmarks. The same error class
lives anywhere a harness can succeed while doing nothing. Extend
coverage to at least: the scenario runner (done — cb-sim fails on an
empty run), the coverage tool, `dep-weight`, and any future collector
including CB-WP-0002's.
**Partly delivered by CB-WP-0002.** Already done: the `--self-test`
contract is stated in `specs/InnerLoop.md` v1.1 §Step 5; `cb-cost` exposes
one with five assertions (AC-5..AC-9); `make cost` depends on
`make cost-test`; `cost-test` runs in CI and in `make all`. `cargo bench
-- --test` and cb-sim's empty-run check were already in place.
Deliver a stated contract in `specs/InnerLoop.md`: **every tool that
reports a number states what it asserts to prove it did the work**, and
a CI step that fails when a reporting tool has no such assertion. Prefer
a mechanical check (e.g. each tool exposes `--self-test`) over a
convention nobody can verify.
**Remaining scope:**
## Task: Point adversarial review at measurement, not only research
- `tools/rule-coverage.py` and `tools/dep-weight.py` have positive-control
logic but no `--self-test` entry point, so nothing verifies the control
itself still works.
- No CI step fails when a *new* reporting tool ships without `--self-test`.
Prefer a mechanical check (enumerate `tools/*.py` + `tools/cb-*`, require
the flag) over a convention nobody can verify.
Carry forward from CB-WP-0002: a self-test that only exercises the happy
path is decorative. Each assertion must name a failure it detects, and the
strongest ones are regression pins on defects that actually occurred —
`cb-cost`'s AC-9 pins the exact `5, 5, 195` case that fooled first-wins
dedup.
## Task: Point adversarial review at measurement, and state its limits
```task
id: CB-WP-0003-T03
@ -86,21 +124,32 @@ priority: medium
state_hub_task_id: "1365b35d-4539-489a-beff-c072221e4702"
```
The review step is specified for tier-L *research*. All four real errors
in this project were in *measurement and build configuration*, which the
review never touches. We adversarially review the artifact that is
cheapest to fix and leave unreviewed the one where errors actually
occur.
The review step is specified for tier-L *research*. The errors in this
project are in *measurement and build configuration*, which the review
never formally targets.
**Now supported by two measurements, not an argument.** Reviews cost
$0.66 (CB-WP-0001) and $1.11 (CB-WP-0002) — ~1% of the pass each — and
each found approval-blocking defects. In CB-WP-0002 the review caught a
target that would have made the evidence file certify a broken collector.
Revise InnerLoop §Step 2 so the review target follows the risk: for a
capability whose claim rests on numbers, the review reads the harness
and the evidence file, not only the survey. Keep it one round; state
explicitly what the reviewer must attempt (reproduce the number,
identify what the harness would report if the work silently stopped).
capability whose claim rests on numbers, the review reads the harness and
the evidence file, not only the survey. Keep it one round; state
explicitly what the reviewer must attempt (reproduce the number, identify
what the harness would report if the work silently stopped).
**Added:** state what review *cannot* do, with the case attached. A
reviewer re-derives the author's claims and therefore inherits the
author's sampling. CB-WP-0002's dedup invariant was checked by both
parties on the main transcript and is false in the subagent tree. The
instruction that follows: **a reviewer must re-derive on a different
sample than the author used**, and where only one sample exists, say so
rather than reporting a clean verify.
## Phase B — Session economics
## Task: Measure and specify session shape
## Task: Measure session shape before prescribing it
```task
id: CB-WP-0003-T04
@ -109,26 +158,41 @@ priority: high
state_hub_task_id: "382724e6-efd5-4ef8-a438-9d31a65dacdb"
```
Measured from the CB-WP-0001 session: **592 assistant turns, mean
context 245,321 tokens, $0.245 per turn in cache reads alone**, and of
330 tool calls **none** were batched into a multi-call turn. Cache reads
were 58% of the $248.46 total; output was 12%.
**Premise revised — the original conclusion was wrong.** This task was
written to prescribe one task per session on the theory that cost ≈
turns × mean_context and mean_context grows with turns, making long
sessions quadratic. CB-WP-0002 measured it:
Cost is therefore approximately `turns × mean_context`, and mean_context
grows with turns — a long session is quadratic. Running T01–T09 in one
context cost ~$145 in cache reads; nine task-scoped sessions at ~40k
context each would plausibly cost ~$25.
| segment | turns | mean context | total | $/turn |
|---|---|---|---|---|
| start → compact 1 | 136 | 304,178 | $62.19 | **$0.457** |
| compact 1 → compact 2 | 202 | 193,493 | $30.01 | **$0.149** |
Write `specs/SessionShape.md`: one task per session as the default,
what a fresh session must be able to load from committed artifacts to
start cold, when to compact versus start clean, and batching of
independent tool calls. Every claim in it carries the measurement it
rests on — this spec exists because the numbers were surprising, not
because the advice sounds sensible.
`/compact` cut context 542,991 → 19,974 tokens (**27×**), and the 202
turns after it cost less than half the 136 before. Sessions are
**bounded**-quadratic: cost grows with context between compactions and
resets at each one. The failure mode is a long *uncompacted* session, not
a long one.
Depends on CB-WP-0002 for per-task attribution to validate the estimate;
the session-level numbers above are already sufficient to write the
policy.
So the deliverable is no longer a policy derived from a theory. **Measure
the two remedies against each other first**, using `make cost`:
- cost of a compaction (the summarization call) plus the post-compact
ramp, versus
- cost of a fresh session, including the cold-start re-read of committed
artifacts that a fresh session must pay.
Then write `specs/SessionShape.md` with whichever wins, and the number.
Also carry over what is already measured and independent of that
question: 0 of 330 tool calls in CB-WP-0001 were batched into a
multi-call turn, and batching is free.
Every claim in the spec carries the measurement it rests on — this spec
exists because the numbers were surprising, and one of them has already
overturned the advice this task was created to give.
Unblocked: `tools/cb-cost.py` exists and `make cost` reports per-task
attribution.
## Task: Replace the dead token budget with a live cost budget
@ -139,16 +203,25 @@ priority: medium
state_hub_task_id: "26c920ee-2c09-4b45-a908-4d343532db09"
```
The global 8k soft / 10k hard per-task token budget was never
referenced or enforced during CB-WP-0001, and T08 exceeded it by orders
of magnitude with no signal. It is dead policy: it implies a control
that does not exist.
The global 8k soft / 10k hard per-task token budget was never referenced
or enforced during CB-WP-0001, and T08 exceeded it by orders of magnitude
with no signal. It is dead policy: it implies a control that does not
exist.
Replace it with a budget expressed in the unit that CB-WP-0002 makes
Replace it with a budget in USD, the unit `make cost` now makes
measurable, with a defined action on breach and a way to observe the
breach at the time it happens rather than in a retrospective. If no such
observation is possible, say so and delete the budget rather than
restating it.
breach **at the time it happens** rather than in a retrospective. If no
such in-flight observation is possible, say so and delete the budget
rather than restating it.
**Calibration data now exists.** Per-task cost on CB-WP-0001 ranged
$1.29 (T09) to $21.02 (T08), with 32.5% of spend unattributed to any
task. Note the constraint this creates: a budget can only bind on the
67.5% that attribution reaches, and attribution is only computable
*after* the commit that closes a task. Any budget claiming to fire
mid-task must explain what it reads.
Unblocked: depends on CB-WP-0002, now complete.
## Phase C — Remove or fix the rest
@ -161,10 +234,10 @@ priority: low
state_hub_task_id: "8f3a6147-54e8-48f9-8543-c13b0e0b0278"
```
The d10 tier roll never fired across the whole pass — roughly 0.2
expected firings across 2–3 tier decisions — so it is untested, and at
1-in-10 it will stay untested for many more passes while adding a step
to every decision.
The d10 tier roll has now been rolled across two workplans (CB-WP-0001:
9; CB-WP-0002: 2) and has **never** triggered the override, as expected
at 1-in-10. It remains untested while adding a step to every tier
decision.
Decide: raise the rate during a stated calibration period so the
mechanism produces evidence, or delete it. Keeping an unevaluated
@ -191,6 +264,19 @@ that motivated the change, and why the new target binds on future work
rather than merely passing present work. Apply retroactively to AM-4a
and AM-4b — either ratify them by ADR or change them.
**Sharpened by CB-WP-0002, which did this three times in one pass.**
AC-1 moved $92.21 → $92.87 → $93.32, each time in the commit that
discovered the discrepancy. Those retargets were *correct* — the target
was wrong and the instrument was right — which is precisely why a blanket
prohibition is the wrong rule. Distinguish the two cases:
- **target corrected because the instrument disproved it** — legitimate,
requires the instrument's output in the commit; and
- **target moved because the implementation missed it** — requires an ADR.
The rule must separate these without relying on the implementer's
self-report of which one it was.
## Task: Give provisional items an expiry
```task
@ -227,7 +313,33 @@ Cheapest real strengthening to evaluate first: require every GR-id in a
`covers:` list to also appear in a doc comment in the aggregate, making
the spec→code→scenario chain mechanical rather than asserted. Consider
mutation-style checking (does removing the rule's code break the
scenario that claims it?) and cost it before adopting.
scenario that claims it?) and cost it before adopting — `make cost` can
now price the evaluation itself.
## Task: Fix the price sheet's time-boxed rate before 2026-08-31
```task
id: CB-WP-0003-T11
status: todo
priority: medium
state_hub_task_id: ""
```
**New, from CB-WP-0002.** `benchmarks/baselines/model-prices.toml`
encodes Sonnet at 3.00/15.00 with the intro rate 2.00/10.00 as a **TOML
comment**. A collector reading the sheet — which `cb-cost` does — silently
uses the wrong number. Today this costs $0.17 on a $93.32 pass (0.19%).
The deadline is real: **on 2026-08-31 the intro rate expires**, and the
comment and the data disagree in the opposite direction. Whoever reads the
sheet after that date gets a defensible number by accident rather than by
construction.
Give the schema a representation for a dated rate, and make the staleness
rule (§1a: refresh on price change or after 90 days) executable rather
than prose — it currently has no check at all, and every M-D2-CST verdict
inherits it. This is the same defect class the cost survey levelled at the
State Hub: a schema that cannot hold the fact it needs.
## Task: Retrospective
@ -238,9 +350,24 @@ priority: low
state_hub_task_id: "f43de208-92b1-4f0e-9236-a9d27f3ec451"
```
Revise `specs/InnerLoop.md` to v1.1 from this pass. The question to
answer honestly: after making rules executable, did the *next* error
still slip through, and if so, what class was it? Four instances of the
harness-does-nothing class were found before a gate caught one. Record
whether the gate holds, and what the fifth error — of whatever class —
turns out to be.
Revise `specs/InnerLoop.md` to v1.2 from this pass.
**The original question is already answered, and the answer was no.**
This task asked whether, after making rules executable, the next error
would still slip through and of what class. CB-WP-0002 ran first and
supplied two data points: the next errors were **not** harness-does-nothing.
They were trusted arithmetic over real data (which a positive control
passes) and a property verified on the large sample and assumed on the
small one (which adversarial review reproduces rather than catches).
So the question for this retrospective is the harder successor:
**is the set of mechanisms now complete, or is each new pass still finding
a new class?** Seven error instances are on record across three classes.
If this pass produces an eighth in a fourth class, the honest conclusion
is that class-by-class hardening does not converge, and the loop should
optimize for *cheap detection and correction* rather than for prevention —
which is a different design.
Record the count either way. A retrospective that reports only what was
fixed, and not whether the fixing is converging, is the same shape of
false assurance this workplan exists to remove.