T07: InnerLoop v1.1 — the instrument must emit its own target
Some checks failed
ci / check (push) Failing after 3s
Some checks failed
ci / check (push) Failing after 3s
Answers the question the workplan posed: did v1.0's "every metric names
its instrument" rule stop a metric being written without a working
instrument? No. AC-1 named `cb-cost --pin` before that tool existed and
set a hand-computed target of $92.87; the tool returned $93.32. The rule
was satisfied completely and the metric was still wrong.
v1.1 adds:
- the instrument must exist and the target must come out of it;
targets are provisional until the tool emits them
- a number inherited from earlier work is re-derived before use as a
target, or cited as unverified
- every reporting tool exposes --self-test, run before the number
- cost is in the definition of done; M-D2-CST may not be uncomputable
The cost of CB-WP-0001 was stated four times before it was right --
$248.46, $92.21, $92.87, $93.32 -- and each correction came from a
different mechanism: re-derivation, adversarial review, and the positive
control. None found more than one. That is the case for keeping all
three.
CB-WP-0003 T10 predicted the next error would be harness-does-nothing.
It was not, twice. Trusted arithmetic over real data would pass a
positive control; and a property verified on 206/206 groups of the main
transcript is false in the 8-response subagent tree that neither the
survey nor the reviewer examined separately. Review structurally cannot
catch the second -- re-deriving on the same sample reproduces the same
blind spot.
Workplan status: done.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
2f086d26b6
commit
42180fbc57
3 changed files with 150 additions and 9 deletions
112
history/260731-cost-accounting-retrospective.md
Normal file
112
history/260731-cost-accounting-retrospective.md
Normal file
|
|
@ -0,0 +1,112 @@
|
|||
# 2026-07-31 — retrospective: what CB-WP-0002 taught the loop
|
||||
|
||||
Pass: CB-WP-0002, cost accounting. Produced `tools/cb-cost.py`,
|
||||
`specs/CostAccounting.md`, ADR-0003, CB-EV-0002, and InnerLoop **v1.1**.
|
||||
|
||||
## The question the workplan asked
|
||||
|
||||
> CB-WP-0001 produced a fully specified metric that could not be computed,
|
||||
> and v1.0's "every metric names its instrument" rule was written to stop
|
||||
> that. Did it?
|
||||
|
||||
**No.** The rule was satisfied completely and the metric was still wrong.
|
||||
|
||||
`specs/CostAccounting.md` AC-1 named `cb-cost --pin fc76445` as its
|
||||
instrument — a command that did not exist when the row was written — and
|
||||
set the target to **$92.87**, computed by hand. When the tool was built one
|
||||
task later it returned **$93.32**. The hand computation carried a dedup bug
|
||||
that the tool's own positive control caught on first contact.
|
||||
|
||||
So the v1.0 rule stops "a metric with no instrument named" and does not stop
|
||||
"a metric whose target the instrument never produced". The distinction did
|
||||
not exist when the rule was written because the failure it was written for
|
||||
was the coarser one.
|
||||
|
||||
**v1.1's answer:** the instrument must exist and the target must come out of
|
||||
it. Where the tool is built later in the pass, the target is `provisional:`
|
||||
until the tool emits it, and the spec is amended to whatever it returns —
|
||||
not the reverse.
|
||||
|
||||
## The number, and its four corrections
|
||||
|
||||
The cost of CB-WP-0001 was stated four times before it was right:
|
||||
|
||||
| value | method | what was wrong |
|
||||
|---|---|---|
|
||||
| $248.46 | inherited, quoted | per-line summation (~1.9×) + single-model pricing |
|
||||
| $92.21 | re-derived, deduped, per-model | omitted the subagent tree |
|
||||
| $92.87 | + subagent at first-wins dedup | `output_tokens` is a running count |
|
||||
| **$93.32** | emitted by `cb-cost` | — |
|
||||
|
||||
Each correction was found by a different mechanism, and this is the useful
|
||||
part:
|
||||
|
||||
1. **$248.46 → $92.21** by *re-deriving instead of quoting*. Now a v1.1
|
||||
rule.
|
||||
2. **$92.21 → $92.87** by *adversarial review*, which noticed the target
|
||||
contradicted the survey's own blind-spot finding. Cost: $1.11.
|
||||
3. **$92.87 → $93.32** by the *positive control*, which fired on real data
|
||||
and refused to print a number.
|
||||
|
||||
No single mechanism found more than one. That is the argument for keeping
|
||||
all three rather than consolidating.
|
||||
|
||||
## The error class the last pass predicted, and what actually happened
|
||||
|
||||
CB-WP-0003 T10 predicted the next error would be the harness-does-nothing
|
||||
class, since four instances had already been seen. It was not — twice.
|
||||
|
||||
- The two errors in $248.46 were **trusted arithmetic over real data**. Both
|
||||
sums ran over data that genuinely existed. A positive control asserting
|
||||
"did this harness do work?" would have answered yes, correctly, and
|
||||
reported a wrong number.
|
||||
- The error in $92.87 was **a property verified on the large sample and
|
||||
assumed on the small one**. The dedup invariant was checked on the main
|
||||
transcript (206/206 groups) by the survey, and independently re-checked by
|
||||
the adversarial reviewer, who also used the main transcript. It is false
|
||||
in the 8-response subagent tree that neither examined separately.
|
||||
|
||||
The second is the one worth carrying forward, because **review structurally
|
||||
cannot catch it**: the reviewer's job is to re-derive the author's claims,
|
||||
and re-deriving on the same sample reproduces the same blind spot. Only an
|
||||
assertion running over *all* the data at execution time catches it. That is
|
||||
now a v1.1 rule (`--self-test` on every reporting tool, run before the
|
||||
number).
|
||||
|
||||
## What the pass bought, in its own units
|
||||
|
||||
The capability measures itself, which is the first time this project has
|
||||
been able to say what a pass cost while the pass was running:
|
||||
|
||||
- **CB-WP-0001 (measured retroactively):** $93.32 pinned at `fc76445`.
|
||||
88.0% cache, 11.9% output, 249:1 context re-read to text written.
|
||||
- **Adversarial review, this pass:** $1.11 — ~1% of the pass it reviewed —
|
||||
and it found three approval-blocking defects. Second consecutive pass
|
||||
where that trade was decisive. Two data points now support CB-WP-0003 T03.
|
||||
- **`/compact`, measured for the first time:** 542,991 → 19,974 tokens, a
|
||||
27× context reduction. Cost per turn fell 3.1×, from $0.457 across the
|
||||
136 turns before it to $0.149 across the 202 after.
|
||||
|
||||
That last number changes advice this project was about to give itself.
|
||||
CB-WP-0003 T04 was going to prescribe one task per session on the theory
|
||||
that long sessions are quadratic. They are **bounded**-quadratic: cost grows
|
||||
with context between compactions and resets at each one. The failure mode is
|
||||
a long *uncompacted* session, and whether a fresh session beats a compaction
|
||||
is now a measurable question rather than a matter of taste. T04 should
|
||||
measure it before prescribing.
|
||||
|
||||
## Raised, not resolved
|
||||
|
||||
- **The price sheet cannot express a time-boxed rate.** Sonnet's intro price
|
||||
is a TOML comment. Costs $0.17 today (0.19%); becomes a real error on
|
||||
**2026-08-31** when the intro rate expires and the comment and the data
|
||||
disagree in the other direction.
|
||||
- **32.5% of spend has no task**, because 19 of 33 commits carry no `T##`
|
||||
tag. Reported as its own line rather than hidden, but the underlying fix
|
||||
is commit hygiene, not tooling.
|
||||
- **Outside this repo:** the `ralph-workplan` skill instructs agents to read
|
||||
token counts from the Claude Code status bar. That is not readable from a
|
||||
tool call, so an agent asked for it estimates instead — the proximate
|
||||
cause of the hub holding 401,100 tokens for a workplan that actually
|
||||
consumed 80.9M. Flagged for the maintainer; `make cost` is the authority
|
||||
in the meantime.
|
||||
|
|
@ -1,11 +1,17 @@
|
|||
# The Inner Loop — Assimilate and Surpass
|
||||
|
||||
Status: **v1.0** — survived its first full pass (CB-WP-0001, the GROUND
|
||||
game kernel) and was corrected from it on 2026-07-31. Changes from v0.2:
|
||||
measurement validity (the positive control), metric feasibility and
|
||||
instrument naming, four implementation rules the pass earned, and the
|
||||
requirement that evidence state what it does not support. Rationale and
|
||||
the failures behind each: `history/260731-inner-loop-retrospective.md`.
|
||||
Status: **v1.1** — corrected from CB-WP-0002 (cost accounting) on
|
||||
2026-07-31. Changes from v1.0: the instrument must exist and emit its own
|
||||
target; inherited numbers are re-derived before use; every reporting tool
|
||||
exposes `--self-test`; cost is in the definition of done. Rationale:
|
||||
`history/260731-cost-accounting-retrospective.md`.
|
||||
|
||||
v1.0 — survived its first full pass (CB-WP-0001, the GROUND game kernel)
|
||||
and was corrected from it on 2026-07-31. Changes from v0.2: measurement
|
||||
validity (the positive control), metric feasibility and instrument naming,
|
||||
four implementation rules the pass earned, and the requirement that
|
||||
evidence state what it does not support. Rationale and the failures behind
|
||||
each: `history/260731-inner-loop-retrospective.md`.
|
||||
|
||||
Normative process for building every Clay-Borg capability. Referenced by
|
||||
all workplans. The loop's own optimization target is **agentic efficiency**:
|
||||
|
|
@ -122,7 +128,30 @@ research step (metric provenance).
|
|||
|
||||
**Every metric names its instrument, and is checked reachable.** A row
|
||||
in the acceptance table carries the command that produces its number.
|
||||
A metric with no named instrument is a wish, not a metric. A metric must
|
||||
A metric with no named instrument is a wish, not a metric.
|
||||
|
||||
**The instrument must exist, and the target must come out of it.**
|
||||
Naming a command is not the same as running one. A target computed by
|
||||
hand and merely *labelled* with a command is the same defect the rule
|
||||
was written to stop, one level down. Where the instrument is built later
|
||||
in the pass, the target is marked `provisional:` until the instrument
|
||||
emits it, and the spec is amended to whatever the instrument returns.
|
||||
|
||||
*(v1.1, from CB-WP-0002: `specs/CostAccounting.md` AC-1 named
|
||||
`cb-cost --pin fc76445` before that tool existed, and set the target to
|
||||
a hand-computed $92.87. When the tool was built it returned $93.32 —
|
||||
the hand computation carried a dedup bug the tool's own positive control
|
||||
caught. The metric satisfied v1.0's rule completely and was still
|
||||
wrong.)*
|
||||
|
||||
**A number inherited from earlier work is re-derived before it is used
|
||||
as a target, or it is cited as unverified.** Quoting is not measuring.
|
||||
|
||||
*(v1.1, from CB-WP-0002: the workplan opened with $248.46, inherited
|
||||
from a prior pass. Re-derivation put it at $92.21 — the quoted figure
|
||||
double-counted transcript lines and priced a three-model session at one
|
||||
model's rate. Neither error was of the harness-does-nothing class; both
|
||||
sums ran over real data, and a positive control would have passed them.)* A metric must
|
||||
also be checked against the contracts in the *same spec*: if a contract
|
||||
makes a target unreachable, one of the two is wrong and the conflict is
|
||||
resolved when it is noticed, not at the acceptance run. Re-check the
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
id: CB-WP-0002
|
||||
title: "Make agentic cost measurable, so D2 claims are falsifiable"
|
||||
status: proposed
|
||||
status: done
|
||||
state_hub_workstream_id: "b7c22f69-fbe9-48df-9619-007db79ae338"
|
||||
---
|
||||
|
||||
|
|
@ -207,7 +207,7 @@ the one command surface.
|
|||
|
||||
```task
|
||||
id: CB-WP-0002-T07
|
||||
status: todo
|
||||
status: done
|
||||
priority: low
|
||||
state_hub_task_id: "ebe58d91-be5f-4d5b-ba40-b03275b4eefc"
|
||||
```
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue