clay-borg/history/260731-cost-accounting-retrospective.md

# 2026-07-31 — retrospective: what CB-WP-0002 taught the loop

Pass: CB-WP-0002, cost accounting. Produced `tools/cb-cost.py`,
`specs/CostAccounting.md`, ADR-0003, CB-EV-0002, and InnerLoop **v1.1**.

## The question the workplan asked

> CB-WP-0001 produced a fully specified metric that could not be computed,
> and v1.0's "every metric names its instrument" rule was written to stop
> that. Did it?

**No.** The rule was satisfied completely and the metric was still wrong.

`specs/CostAccounting.md` AC-1 named `cb-cost --pin fc76445` as its
instrument — a command that did not exist when the row was written — and
set the target to **$92.87**, computed by hand. When the tool was built one
task later it returned **$93.32**. The hand computation carried a dedup bug
that the tool's own positive control caught on first contact.

So the v1.0 rule stops "a metric with no instrument named" and does not stop
"a metric whose target the instrument never produced". The distinction did
not exist when the rule was written because the failure it was written for
was the coarser one.

**v1.1's answer:** the instrument must exist and the target must come out of
it. Where the tool is built later in the pass, the target is `provisional:`
until the tool emits it, and the spec is amended to whatever it returns —
not the reverse.

## The number, and its four corrections

The cost of CB-WP-0001 was stated four times before it was right:

| value | method | what was wrong |
|---|---|---|
| $248.46 | inherited, quoted | per-line summation (~1.9×) + single-model pricing |
| $92.21 | re-derived, deduped, per-model | omitted the subagent tree |
| $92.87 | + subagent at first-wins dedup | `output_tokens` is a running count |
| **$93.32** | emitted by `cb-cost` | — |

Each correction was found by a different mechanism, and this is the useful
part:

1. **$248.46 → $92.21** by *re-deriving instead of quoting*. Now a v1.1
   rule.
2. **$92.21 → $92.87** by *adversarial review*, which noticed the target
   contradicted the survey's own blind-spot finding. Cost: $1.11.
3. **$92.87 → $93.32** by the *positive control*, which fired on real data
   and refused to print a number.

No single mechanism found more than one. That is the argument for keeping
all three rather than consolidating.

## The error class the last pass predicted, and what actually happened

CB-WP-0003 T10 predicted the next error would be the harness-does-nothing
class, since four instances had already been seen. It was not — twice.

- The two errors in $248.46 were **trusted arithmetic over real data**. Both
  sums ran over data that genuinely existed. A positive control asserting
  "did this harness do work?" would have answered yes, correctly, and
  reported a wrong number.
- The error in $92.87 was **a property verified on the large sample and
  assumed on the small one**. The dedup invariant was checked on the main
  transcript (206/206 groups) by the survey, and independently re-checked by
  the adversarial reviewer, who also used the main transcript. It is false
  in the 8-response subagent tree that neither examined separately.

The second is the one worth carrying forward, because **review structurally
cannot catch it**: the reviewer's job is to re-derive the author's claims,
and re-deriving on the same sample reproduces the same blind spot. Only an
assertion running over *all* the data at execution time catches it. That is
now a v1.1 rule (`--self-test` on every reporting tool, run before the
number).

## What the pass bought, in its own units

The capability measures itself, which is the first time this project has
been able to say what a pass cost while the pass was running:

- **CB-WP-0001 (measured retroactively):** $93.32 pinned at `fc76445`.
  88.0% cache, 11.9% output, 249:1 context re-read to text written.
- **Adversarial review, this pass:** $1.11 — ~1% of the pass it reviewed —
  and it found three approval-blocking defects. Second consecutive pass
  where that trade was decisive. Two data points now support CB-WP-0003 T03.
- **`/compact`, measured for the first time:** 542,991 → 19,974 tokens, a
  27× context reduction. Cost per turn fell 3.1×, from $0.457 across the
  136 turns before it to $0.149 across the 202 after.

That last number changes advice this project was about to give itself.
CB-WP-0003 T04 was going to prescribe one task per session on the theory
that long sessions are quadratic. They are **bounded**-quadratic: cost grows
with context between compactions and resets at each one. The failure mode is
a long *uncompacted* session, and whether a fresh session beats a compaction
is now a measurable question rather than a matter of taste. T04 should
measure it before prescribing.

## Raised, not resolved

- **The price sheet cannot express a time-boxed rate.** Sonnet's intro price
  is a TOML comment. Costs $0.17 today (0.19%); becomes a real error on
  **2026-08-31** when the intro rate expires and the comment and the data
  disagree in the other direction.
- **32.5% of spend has no task**, because 19 of 33 commits carry no `T##`
  tag. Reported as its own line rather than hidden, but the underlying fix
  is commit hygiene, not tooling.
- **Outside this repo:** the `ralph-workplan` skill instructs agents to read
  token counts from the Claude Code status bar. That is not readable from a
  tool call, so an agent asked for it estimates instead — the proximate
  cause of the hub holding 401,100 tokens for a workplan that actually
  consumed 80.9M. Flagged for the maintainer; `make cost` is the authority
  in the meantime.