T07: InnerLoop v1.1 — the instrument must emit its own target
Some checks failed
ci / check (push) Failing after 3s

Answers the question the workplan posed: did v1.0's "every metric names
its instrument" rule stop a metric being written without a working
instrument? No. AC-1 named `cb-cost --pin` before that tool existed and
set a hand-computed target of $92.87; the tool returned $93.32. The rule
was satisfied completely and the metric was still wrong.

v1.1 adds:
  - the instrument must exist and the target must come out of it;
    targets are provisional until the tool emits them
  - a number inherited from earlier work is re-derived before use as a
    target, or cited as unverified
  - every reporting tool exposes --self-test, run before the number
  - cost is in the definition of done; M-D2-CST may not be uncomputable

The cost of CB-WP-0001 was stated four times before it was right --
$248.46, $92.21, $92.87, $93.32 -- and each correction came from a
different mechanism: re-derivation, adversarial review, and the positive
control. None found more than one. That is the case for keeping all
three.

CB-WP-0003 T10 predicted the next error would be harness-does-nothing.
It was not, twice. Trusted arithmetic over real data would pass a
positive control; and a property verified on 206/206 groups of the main
transcript is false in the 8-response subagent tree that neither the
survey nor the reviewer examined separately. Review structurally cannot
catch the second -- re-deriving on the same sample reproduces the same
blind spot.

Workplan status: done.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 08:53:52 +02:00
parent 2f086d26b6
commit 42180fbc57
3 changed files with 150 additions and 9 deletions

View file

@ -0,0 +1,112 @@
# 2026-07-31 — retrospective: what CB-WP-0002 taught the loop
Pass: CB-WP-0002, cost accounting. Produced `tools/cb-cost.py`,
`specs/CostAccounting.md`, ADR-0003, CB-EV-0002, and InnerLoop **v1.1**.
## The question the workplan asked
> CB-WP-0001 produced a fully specified metric that could not be computed,
> and v1.0's "every metric names its instrument" rule was written to stop
> that. Did it?
**No.** The rule was satisfied completely and the metric was still wrong.
`specs/CostAccounting.md` AC-1 named `cb-cost --pin fc76445` as its
instrument — a command that did not exist when the row was written — and
set the target to **$92.87**, computed by hand. When the tool was built one
task later it returned **$93.32**. The hand computation carried a dedup bug
that the tool's own positive control caught on first contact.
So the v1.0 rule stops "a metric with no instrument named" and does not stop
"a metric whose target the instrument never produced". The distinction did
not exist when the rule was written because the failure it was written for
was the coarser one.
**v1.1's answer:** the instrument must exist and the target must come out of
it. Where the tool is built later in the pass, the target is `provisional:`
until the tool emits it, and the spec is amended to whatever it returns —
not the reverse.
## The number, and its four corrections
The cost of CB-WP-0001 was stated four times before it was right:
| value | method | what was wrong |
|---|---|---|
| $248.46 | inherited, quoted | per-line summation (~1.9×) + single-model pricing |
| $92.21 | re-derived, deduped, per-model | omitted the subagent tree |
| $92.87 | + subagent at first-wins dedup | `output_tokens` is a running count |
| **$93.32** | emitted by `cb-cost` | — |
Each correction was found by a different mechanism, and this is the useful
part:
1. **$248.46 → $92.21** by *re-deriving instead of quoting*. Now a v1.1
rule.
2. **$92.21 → $92.87** by *adversarial review*, which noticed the target
contradicted the survey's own blind-spot finding. Cost: $1.11.
3. **$92.87 → $93.32** by the *positive control*, which fired on real data
and refused to print a number.
No single mechanism found more than one. That is the argument for keeping
all three rather than consolidating.
## The error class the last pass predicted, and what actually happened
CB-WP-0003 T10 predicted the next error would be the harness-does-nothing
class, since four instances had already been seen. It was not — twice.
- The two errors in $248.46 were **trusted arithmetic over real data**. Both
sums ran over data that genuinely existed. A positive control asserting
"did this harness do work?" would have answered yes, correctly, and
reported a wrong number.
- The error in $92.87 was **a property verified on the large sample and
assumed on the small one**. The dedup invariant was checked on the main
transcript (206/206 groups) by the survey, and independently re-checked by
the adversarial reviewer, who also used the main transcript. It is false
in the 8-response subagent tree that neither examined separately.
The second is the one worth carrying forward, because **review structurally
cannot catch it**: the reviewer's job is to re-derive the author's claims,
and re-deriving on the same sample reproduces the same blind spot. Only an
assertion running over *all* the data at execution time catches it. That is
now a v1.1 rule (`--self-test` on every reporting tool, run before the
number).
## What the pass bought, in its own units
The capability measures itself, which is the first time this project has
been able to say what a pass cost while the pass was running:
- **CB-WP-0001 (measured retroactively):** $93.32 pinned at `fc76445`.
88.0% cache, 11.9% output, 249:1 context re-read to text written.
- **Adversarial review, this pass:** $1.11 — ~1% of the pass it reviewed —
and it found three approval-blocking defects. Second consecutive pass
where that trade was decisive. Two data points now support CB-WP-0003 T03.
- **`/compact`, measured for the first time:** 542,991 → 19,974 tokens, a
27× context reduction. Cost per turn fell 3.1×, from $0.457 across the
136 turns before it to $0.149 across the 202 after.
That last number changes advice this project was about to give itself.
CB-WP-0003 T04 was going to prescribe one task per session on the theory
that long sessions are quadratic. They are **bounded**-quadratic: cost grows
with context between compactions and resets at each one. The failure mode is
a long *uncompacted* session, and whether a fresh session beats a compaction
is now a measurable question rather than a matter of taste. T04 should
measure it before prescribing.
## Raised, not resolved
- **The price sheet cannot express a time-boxed rate.** Sonnet's intro price
is a TOML comment. Costs $0.17 today (0.19%); becomes a real error on
**2026-08-31** when the intro rate expires and the comment and the data
disagree in the other direction.
- **32.5% of spend has no task**, because 19 of 33 commits carry no `T##`
tag. Reported as its own line rather than hidden, but the underlying fix
is commit hygiene, not tooling.
- **Outside this repo:** the `ralph-workplan` skill instructs agents to read
token counts from the Claude Code status bar. That is not readable from a
tool call, so an agent asked for it estimates instead — the proximate
cause of the hub holding 401,100 tokens for a workplan that actually
consumed 80.9M. Flagged for the maintainer; `make cost` is the authority
in the meantime.

View file

@ -1,11 +1,17 @@
# The Inner Loop — Assimilate and Surpass
Status: **v1.0** — survived its first full pass (CB-WP-0001, the GROUND
game kernel) and was corrected from it on 2026-07-31. Changes from v0.2:
measurement validity (the positive control), metric feasibility and
instrument naming, four implementation rules the pass earned, and the
requirement that evidence state what it does not support. Rationale and
the failures behind each: `history/260731-inner-loop-retrospective.md`.
Status: **v1.1** — corrected from CB-WP-0002 (cost accounting) on
2026-07-31. Changes from v1.0: the instrument must exist and emit its own
target; inherited numbers are re-derived before use; every reporting tool
exposes `--self-test`; cost is in the definition of done. Rationale:
`history/260731-cost-accounting-retrospective.md`.
v1.0 — survived its first full pass (CB-WP-0001, the GROUND game kernel)
and was corrected from it on 2026-07-31. Changes from v0.2: measurement
validity (the positive control), metric feasibility and instrument naming,
four implementation rules the pass earned, and the requirement that
evidence state what it does not support. Rationale and the failures behind
each: `history/260731-inner-loop-retrospective.md`.
Normative process for building every Clay-Borg capability. Referenced by
all workplans. The loop's own optimization target is **agentic efficiency**:
@ -122,7 +128,30 @@ research step (metric provenance).
**Every metric names its instrument, and is checked reachable.** A row
in the acceptance table carries the command that produces its number.
A metric with no named instrument is a wish, not a metric. A metric must
A metric with no named instrument is a wish, not a metric.
**The instrument must exist, and the target must come out of it.**
Naming a command is not the same as running one. A target computed by
hand and merely *labelled* with a command is the same defect the rule
was written to stop, one level down. Where the instrument is built later
in the pass, the target is marked `provisional:` until the instrument
emits it, and the spec is amended to whatever the instrument returns.
*(v1.1, from CB-WP-0002: `specs/CostAccounting.md` AC-1 named
`cb-cost --pin fc76445` before that tool existed, and set the target to
a hand-computed $92.87. When the tool was built it returned $93.32 —
the hand computation carried a dedup bug the tool's own positive control
caught. The metric satisfied v1.0's rule completely and was still
wrong.)*
**A number inherited from earlier work is re-derived before it is used
as a target, or it is cited as unverified.** Quoting is not measuring.
*(v1.1, from CB-WP-0002: the workplan opened with $248.46, inherited
from a prior pass. Re-derivation put it at $92.21 — the quoted figure
double-counted transcript lines and priced a three-model session at one
model's rate. Neither error was of the harness-does-nothing class; both
sums ran over real data, and a positive control would have passed them.)* A metric must
also be checked against the contracts in the *same spec*: if a contract
makes a target unreachable, one of the two is wrong and the conflict is
resolved when it is noticed, not at the acceptance run. Re-check the

View file

@ -1,7 +1,7 @@
---
id: CB-WP-0002
title: "Make agentic cost measurable, so D2 claims are falsifiable"
status: proposed
status: done
state_hub_workstream_id: "b7c22f69-fbe9-48df-9619-007db79ae338"
---
@ -207,7 +207,7 @@ the one command surface.
```task
id: CB-WP-0002-T07
status: todo
status: done
priority: low
state_hub_task_id: "ebe58d91-be5f-4d5b-ba40-b03275b4eefc"
```