Compare commits

...

10 commits

Author SHA1 Message Date
42180fbc57 T07: InnerLoop v1.1 — the instrument must emit its own target
Some checks failed
ci / check (push) Failing after 3s
Answers the question the workplan posed: did v1.0's "every metric names
its instrument" rule stop a metric being written without a working
instrument? No. AC-1 named `cb-cost --pin` before that tool existed and
set a hand-computed target of $92.87; the tool returned $93.32. The rule
was satisfied completely and the metric was still wrong.

v1.1 adds:
  - the instrument must exist and the target must come out of it;
    targets are provisional until the tool emits them
  - a number inherited from earlier work is re-derived before use as a
    target, or cited as unverified
  - every reporting tool exposes --self-test, run before the number
  - cost is in the definition of done; M-D2-CST may not be uncomputable

The cost of CB-WP-0001 was stated four times before it was right --
$248.46, $92.21, $92.87, $93.32 -- and each correction came from a
different mechanism: re-derivation, adversarial review, and the positive
control. None found more than one. That is the case for keeping all
three.

CB-WP-0003 T10 predicted the next error would be harness-does-nothing.
It was not, twice. Trusted arithmetic over real data would pass a
positive control; and a property verified on 206/206 groups of the main
transcript is false in the 8-response subagent tree that neither the
survey nor the reviewer examined separately. Review structurally cannot
catch the second -- re-deriving on the same sample reproduces the same
blind spot.

Workplan status: done.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:53:52 +02:00
2f086d26b6 T06: wire cost into the loop
- InnerLoop definition-of-done gains a cost row: M-D2-CST may no longer
  be recorded uncomputable, and composition must be reported, not only a
  total.
- InnerLoop Step 5 gains the --self-test contract: every tool that
  reports a number exposes one, and it runs before the number does.
  Rationale attached, because the case that motivated it is the one
  review cannot catch — survey and reviewer both verified the same large
  sample and both missed the small one.
- make cost / cost-test / cost-pin on the one command surface; cost-test
  in `make all` and in CI.
- Hub now holds the measured figure for CB-WP-0001: 80.6M in / 323.6k
  out against the 401,100 it previously estimated, low by ~200x. The
  event states plainly that the hub schema cannot represent the 88% of
  cost that is cache, and names `make cost-pin` as the authority.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:52:33 +02:00
c0eef604e8 T05: CB-EV-0002 — all 9 metrics met, and /compact measured at 3.1x
AC-1 $93.32 ($92.21 main + $1.11 subagent), residual $0.000000. No
unmeasured rows. Composition: 88.0% cache, 11.9% output, 249:1 context
re-read to text written.

The question that could not be answered before: what /compact costs.
The transcript records it directly — 542,991 -> 19,974 tokens, a 27x
reduction. Cost per turn fell from $0.457 (136 turns before) to $0.149
(202 turns after). The 202 post-compact turns cost less than half the
136 before them.

This qualifies CB-WP-0003 T04: sessions are bounded-quadratic, not
quadratic. Cost grows with context between compactions and resets at
each one, so the failure mode is a long UNCOMPACTED session. Whether a
fresh session beats a compaction is now measurable, and T04 should
measure it before prescribing one task per session.

Stale $0.66/$92.87 targets updated across survey and workplan.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:51:50 +02:00
e008b1e787 T04: tools/cb-cost.py — and its positive control fires on first contact
Collector per ADR-0003: enumerates every transcript including the
subagents/ tree, dedups by requestId, prices per model and per cache
TTL, attributes on (prev_commit, this_commit] intervals, and reconciles
to the cent or aborts.

The positive control caught a real defect on its very first run against
real data, which is the entire argument for writing it:

  CA-02 assumed usage is identical across the lines of one requestId.
  True in the main transcript (206/206 groups, verified twice — by the
  survey and by the adversarial reviewer). FALSE in the subagents/
  tree, where output_tokens is a running count: one response reads
  5, 5, 195 across its three lines. First-wins scored it at 5.

So CA-02 now splits: input-side counters are charged once and must be
identical (assertion retained); output_tokens resolves to the max
(CA-02a). AC-9 pins the exact 5,5,195 case as a regression test.

The acceptance target moved again as a result, for the third time:
$248.46 -> $92.21 -> $92.87 -> $93.32. AC-1 was computed by the same
first-wins method the tool just disproved, so the tool failing its
target was the tool being correct. Target updated, not the tool.

Measured at the pin: $92.21 main + $1.11 subagent = $93.32, residual
$0.000000. 32.5% UNATTRIBUTED, reported as its own line per CA-10.

make cost / cost-test / cost-pin wired; cost-test added to `make all`
and to CI, where it gates the collector's assertions without needing
transcripts present.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:50:04 +02:00
900467017f T03: correct stale targets in the workplan ($92.87, AC-1)
T05 still named $248.46 as its acceptance test and the Purpose total
still omitted the subagent tree. Both corrected in place, with the
error sequence recorded rather than overwritten.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:46:36 +02:00
b96cd94a64 T03: specs/CostAccounting.md — 15 contracts, 8 metrics, each with a command
Normative spec for the cost model (dedup by requestId, per-model and
per-TTL pricing), the attribution contract (ending-at-commit intervals
scoped by sessionId), and the reported shape (composition, not a total).

Every acceptance row names the command that produces its number, per
InnerLoop Step 4. AC-5..AC-8 are the positive control: cb-cost
--self-test must fail on a dedup violation, on zero responses, on a
missing subagent tree, and on 5m cache priced at the 1h rate.

The feasibility check earns its place here: AC-1's $92.87 is reachable
only if CA-06 holds. The earlier $92.21 target was reachable only by a
collector with the exact blind spot the survey documented.

MetricsAndScenarios 1a superseded. Two of its rules struck through
rather than deleted, because both were wrong in instructive ways: "if
the cache split is unknown, count all input at full price" would have
priced 80.5M cache reads at 10x, and "the hub already records tokens"
named as a source what is only ever a lossy sink. M-D2-TOK demoted --
tokens are not comparable across models or cache states.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:46:12 +02:00
c3e6e72ab2 T02: ADR-0003 — transcript as instrument, commit intervals as attribution
Instrument: session transcript JSONL, deduped by requestId, priced per
model and per cache TTL. Hub rejected as a source (no cache fields, ~100%
error on the same work), kept as sink. Status bar and billing API
rejected on availability.

Attribution: (prev_commit, this_commit] ending-at-commit, scoped by
sessionId. Explicit session markers rejected because they depend on an
agent remembering a step — the exact failure this workplan exists to fix.
Hub status transitions rejected as an after-the-fact time index.

Stated honestly: we buy accuracy with specification complexity, and D1
gets worse. That trade is only right because the alternative on offer was
not a simpler correct number but one wrong by two orders of magnitude.

Seven failure modes recorded up front, each becoming a T04 test — most
importantly that dedup fails in the under-reporting direction, so the
invariant is asserted at runtime rather than trusted.

Gate satisfied: T04 may now write collector code.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:44:27 +02:00
ac3ac2a8ce T02 review round: survey approved after 15 of 16 findings conceded
Adversarial review per InnerLoop Step 2 — separate agent, given only the
survey, one round. Challenge and response in history/.

The arithmetic held: every pinned figure and the whole three-method
correction table reproduced to the cent under ~20 independent spot
checks. The hand-written markdown tables did not. Three blocking
defects:

  - the dedup positive-control table failed its own checksum (group
    sizes summed to 348 groups / 663 lines, not 346 / 657; multi-line
    groups are 206, not 208) — the worst place for a slip, since that
    table is the sole evidence for the load-bearing dedup assumption
  - the multi-model table was labelled "deduped" while holding per-line
    counts, three lines below the paragraph condemning per-line counts
  - the $92.21 acceptance target excluded the subagent tree, which ran
    INSIDE the pin window. A collector correctly implementing the
    survey's own blind-spot finding would have failed the benchmark.
    Target is now $92.87, stated as its two components.

Folded in, each re-derived before conceding:

  - attribution must key on sessionId, not wall-clock: two sessions
    overlap 4h13m on this repo carrying ~$12 (one of them was this
    workplan's own background job)
  - only 14 of 33 commits name a task; the other 19 hold 33% of cost
  - cache writes bill per TTL — pricing the aggregate at 1h inflates
    the subagent by 43%
  - the "~87% cache" figure matched nothing; 64.6% of cost is cache
    read, 88.4% is all cache
  - fable is 35% of calls and 60% of dollars; opus is the inverse

Review cost ~$0.60 against a $92 pass, the second consecutive round
where that trade found something material. Evidence for CB-WP-0003 T03.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:43:32 +02:00
bf57b6f3a4 chore: mark T01 done (measured: 25 responses, 2.20M in / 23.2k out, opus-5)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:24:24 +02:00
060eb8cf6a T01: cost-accounting survey — and a 2.7x correction to our own numbers
Tier L (structural L, chaos d10=2). Every candidate exercised locally
against the CB-WP-0001 transcript rather than cited.

Verdict: session transcript JSONL is the only exact instrument; git
commits supply the attribution index it lacks; the State Hub is a
durable sink whose schema cannot express cache tokens at all; the
status bar and billing API are eliminated on availability.

Three errors found in this workplan's own Purpose section, all
overstating cost:

  - one API response is written as up to six JSONL lines, each
    repeating the same usage object (657 lines, 346 responses)
  - a multi-model session was priced entirely at Fable 5 rates,
    violating a per-model rule MetricsAndScenarios §1a already had
  - the transcript is a live file, so an unpinned total is not
    repeatable (346 -> 356 responses between two runs minutes apart)

CB-WP-0001 pinned at fc76445: $92.21, not $248.46. The qualitative
finding sharpens rather than weakens — 88.4% of spend is cache, at
256:1 cache-read to output tokens.

Neither error was of the harness-does-nothing class the positive
control was written for; both sums ran over real data. What caught
them was re-deriving a number instead of quoting it. Raised for T07
and for CB-WP-0003 T01.

Also measured: the one subagent (adversarial review) cost $0.66 and is
invisible to a collector reading only the main transcript.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 04:27:52 +02:00
15 changed files with 1886 additions and 42 deletions

View file

@ -33,6 +33,11 @@ jobs:
# AM-4a/AM-4b: third-party source under audit, per configuration.
- run: make dep-weight
# Positive control for the cost collector (AC-5..AC-9). Does not
# gate on a dollar figure — transcripts are not present in CI — but
# proves the collector still detects the failures it claims to.
- run: make cost-test
# InnerLoop v1.0 positive control, enforced rather than asserted in
# prose: --test runs every benchmark once, so a workload that
# stalls or produces the wrong event count fails the build instead

View file

@ -3,7 +3,7 @@
CARGO := cargo
.PHONY: check test sim bench bench-test coverage dep-weight loc all
.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin loc all
## fmt + clippy (deny warnings) + HashMap deny-lint
check:
@ -21,6 +21,17 @@ dep-weight:
coverage:
python3 tools/rule-coverage.py
# M-D2-CST (specs/CostAccounting.md). cost-test is the positive control and
# runs first: a cost number from an unverified collector is void.
cost: cost-test
python3 tools/cb-cost.py --composition --by-task
cost-test:
python3 tools/cb-cost.py --self-test
cost-pin: cost-test
python3 tools/cb-cost.py --pin fc76445 --composition --by-task
sim:
$(CARGO) run -q -p cb-sim -- scenarios/ground/*.yaml
@ -40,4 +51,4 @@ loc:
printf '%-28s %s\n' $$d "$$(find $$d/src -name '*.rs' | xargs cat | grep -vcE '^\s*(//|$$)')"; \
done
all: check test sim coverage dep-weight bench-test
all: check test sim coverage dep-weight cost-test bench-test

View file

@ -0,0 +1,118 @@
# ADR-0003: cost accounting — instrument and attribution model
status: accepted
date: 2026-07-31
tier: L (structural L, chaos d10=2 → no override)
research: [CB-RES-0002](../research/CB-RES-0002-cost-accounting.md) (approved)
review: history/260731-cost-accounting-{challenge,response}.md
supersedes: the "uncomputable" disposition of AM-12 / M-D2-CST in
[CB-EV-0001](../evidence/CB-EV-0001-game-kernel.md)
## Context
M-D2-CST was fully specified in CB-WP-0001 and never instrumented, so D2
claims about implementation efficiency rest on nothing measured. The survey
established that the data existed the whole time, and that reading it
correctly is harder than it looks: this workplan's own opening figure was
wrong by 2.7×, and the survey's first draft set an acceptance target only a
*broken* collector could hit.
## Decision
### D1 — Instrument: session transcript JSONL (C1)
Cost is computed from `~/.claude/projects/<slug>/**/*.jsonl`, deduplicated
by `requestId`, priced per message against
`benchmarks/baselines/model-prices.toml`.
Rejected: the State Hub as a *source* (its schema has no cache fields, so it
cannot represent 88% of spend, and its recorded numbers are ~100% in error
against the same work); the status bar (not machine-readable from a tool
call); the billing API (no session or task attribution, no admin key). The
hub remains the durable **sink**; the billing API remains an optional
external reconciliation check if an admin key ever exists.
### D2 — Attribution: git commit intervals, scoped by session
A message is attributed to the task named by the **next commit at or after
it**, within its own session:
```text
interval := (prev_commit_time, this_commit_time] # ending-at-commit
scope := sessionId # never wall-clock alone
task := the T## tag in the commit subject, else UNATTRIBUTED
```
Ending-at-commit is the only convention consistent with the loop's
commit-at-end-of-task pattern; the alternative shifts every task's cost one
interval. Session scoping is not optional: two sessions overlap 4 h 13 m on
this repo carrying ~$12 that no wall-clock join can separate.
Rejected: **explicit session markers** at task start/end — they require the
agent to remember, and the failure this workplan exists to fix was caused by
exactly that kind of remembered step; nothing that depends on discipline
gets to be the primary index. Rejected: **hub status transitions** as the
time index — they are written after the fact, sometimes in a batch, and one
was written for CB-WP-0001 nine minutes after the work it bounds.
### D3 — Reported shape: composition, not a total
Every report carries the cost split by component (input / output / cache
read / cache write per TTL) alongside the total. A single total would have
concealed the finding that motivated the workplan.
## Expected advantage per dimension
| Dim | Expectation vs the C2 baseline | Basis |
|---|---|---|
| **D1 ease of specification** | **worse.** The hub is one API call; this is a parser with a dedup rule, a TTL-aware price model, a session scope, and a commit join. Four moving parts against one. | measured: the survey needed three corrections to get the parse right |
| **D2 efficiency** | **better, decisively.** C2's recorded numbers are ~100% in error on the same work ($0.03-equivalent recorded against $92.21 actual). C1 is exact by construction. Cost to produce: one file read, ~1 s. | measured |
| **D3 speed** | **parity.** 2,040 lines / 5.1 MB parsed in <1 s; a hub call is a network round trip. Neither is a bottleneck. | measured |
| **D4 optionality** | **better.** The transcript is a file on disk in a documented shape; the hub is a service that must be running. The collector degrades to "no data" rather than "wrong data" when a transcript is absent. | reasoned |
An honest summary: **we are buying accuracy with specification complexity.**
That trade is right here only because the alternative is not "a simpler
correct number" but "a number that is wrong by two orders of magnitude",
which is what CB-WP-0001 actually recorded.
## Known failure modes of the chosen model
Stated rather than discovered later. Each becomes a test in T04.
1. **33% of spend has no task.** Only 14 of 33 commits name a task; the rest
hold $30.32 of $92.21. Per-task tables are a view over two-thirds of the
money and must say so wherever reported.
2. **Work spanning a boundary is assigned whole to the later task.** A
message before a commit belongs to that commit's task even if the thinking
began earlier. Accepted: the loop commits per task, so the error is
bounded by one interval (p90 17.7 min, max 36.8 min).
3. **Uncommitted work is invisible.** Cost incurred after the last commit
has no enclosing interval. Reported as an open remainder, never dropped.
4. **`/compact` is safe; resumed and concurrent sessions are the risk.**
Compaction stays within one file and one session. Two agents on one repo
are separable only because `sessionId` exists — this is why D2 scopes by
it.
5. **The price sheet cannot express a time-boxed rate.** Sonnet's intro
price is a TOML comment. $0.17 at the pin; the schema defect is the real
issue and is deferred to T03 with a stated deadline of 2026-08-31, when
the intro rate expires and the sheet becomes silently wrong.
6. **Dedup is load-bearing in the dangerous direction.** If the format ever
splits one response across two `requestId`s, the collector *under*-reports
and nothing looks wrong. T04 asserts the dedup invariant at runtime
(identical `usage` within a group) rather than trusting the survey's
one-time check.
7. **The subagent tree is a separate enumeration.** Missing it under-reports
silently; it was $0.66 here and will not stay small on a fan-out pass.
## Consequences
- **Gate satisfied:** T04 may now write collector code. Nothing before this.
- T03 specifies the metrics with this ADR's contracts; T05's acceptance test
is $92.87 as two components, with a 33% unattributed line.
- The hub's token API keeps receiving events, now as a lossy projection of a
computed number rather than as an estimate. Its schema gap (no cache
fields) is recorded as a limitation of the dashboard, not of the metric.
- **Raised out of scope, for the ralph-workplan skill:** its instruction to
"read tokens from the Claude Code status bar" asks an agent for a figure
it cannot read, and is the proximate cause of the hub's bad numbers. The
skill lives outside this repo; flagged for the maintainer.

View file

@ -0,0 +1,152 @@
# CB-EV-0002: cost accounting
capability: meta.loop.cost-accounting
spec: [CostAccounting.md](../specs/CostAccounting.md) v1.0
decision: [ADR-0003](../decisions/ADR-0003-cost-accounting.md)
research: [CB-RES-0002](../research/CB-RES-0002-cost-accounting.md)
instrument: `make cost-pin` (`tools/cb-cost.py --pin fc76445`)
measured: 2026-07-31
Subject of measurement: the CB-WP-0001 session, pinned at commit `fc76445`
(`2026-07-31T02:17:59Z`). All numbers below are emitted by the tool, not
transcribed by hand (CA-15).
---
## 1. Acceptance table
| ID | Metric | Target | Measured | Verdict |
|---|---|---|---|---|
| AC-1 | pinned total, as two components | $93.32 = $92.21 + $1.11 | **$92.21 main + $1.11 subagent = $93.32** | **met** |
| AC-2 | reconciliation residual | $0.00 | **$0.000000** | **met** |
| AC-3 | unattributed share reported | present, 33% | **32.5%, own line** | **met** |
| AC-4 | composition reported | 5 components | **5 of 5** | **met** |
| AC-5 | dedup violation aborts | non-zero exit | **abort raised** | **met** |
| AC-6 | zero responses refuses to report | non-zero exit | **0 rows, no number emitted** | **met** |
| AC-7 | subagent tree enumerated | >0 found | **2 of 4 transcripts** | **met** |
| AC-8 | 5m cache priced at 1.25× | $1.25/100k @ fable | **$1.2500 (1h would be $2.0000)** | **met** |
| AC-9 | streamed partial output → final | 5,5,195 → 195 | **195** | **met** |
No `unmeasured` rows. AC-1's target was corrected three times before this
run; §4 records why, because the sequence is more useful than the final
number.
## 2. Composition — the finding the metric exists for
```text
input 690 tok $ 0.00 0.0%
output 323,643 tok $ 11.15 11.9%
cache_read 80,611,798 tok $ 59.75 64.0%
write_5m 37,467 tok $ 0.47 0.5%
write_1h 1,672,854 tok $ 21.95 23.5%
TOTAL $ 93.32
```
**88.0% of spend is cache; 11.9% is output.** The ratio of context re-read
to text written is 249:1. A single total would have shown $93.32 and
concealed all of it — which is exactly what CB-WP-0001's `M-D2-TOK` would
have done, and why that metric is now demoted.
## 3. Per-task attribution
```text
UNATTRIBUTED $ 30.32 32.5%
T08 $ 21.02 22.5%
T07 $ 12.01 12.9%
T03 $ 9.91 10.6%
T04 $ 8.00 8.6%
T05 $ 7.19 7.7%
T06 $ 3.58 3.8%
T09 $ 1.29 1.4%
```
**Stated limit (CA-10):** 32.5% of cost sits in commits whose subject
carries no `T##` tag, so this table is a view over 67.5% of spend. That is
a property of commit hygiene, not of the collector.
T08 (the GROUND aggregate, six code iterations) at $21.02 is the most
expensive task and was also the one that produced the most rework — the
tuple-map hash panic, the discarded `setup.patch`, and the 5.6×-wrong
benchmark all originated there. Expensive *and* error-dense: the correlation
is worth watching, not yet a conclusion from n=1.
## 4. What it does not support
- **The per-task figures are not comparable across passes.** They mix
models (opus/fable/sonnet) at different price points and different cache
states. The dollar figure is comparable; a token count is not.
- **AC-3's 32.5% is a fixture pin, not a quality target.** Improving commit
tagging will move it, and that is the desired direction.
- **This is one session.** Every ratio here (cache share, $/turn, the
compaction effect in §5) is n=1 and should be treated as a hypothesis
until a second pass reproduces it.
- **The price sheet cannot express a time-boxed rate.** Sonnet's intro price
lives in a TOML comment, so sonnet-priced work is off by $0.17 here
(0.19%). This becomes a real error on 2026-08-31.
- **AC-1 is not invoice-verified.** No admin key exists, so the Anthropic
billing API could not independently confirm the total. The transcript
counters are the same ones billing uses, but that is an argument, not a
reconciliation.
## 5. The question that could not be answered before
**"What does `/compact` actually cost, and is a long session quadratic?"**
CB-WP-0003 T04 asserts that cost ≈ turns × mean_context and that mean_context
grows with turns, making long sessions quadratic. The data **qualifies that
claim**: it is true only between compactions.
The session compacted twice, and the transcript records both directly:
| compaction | pre-tokens | post-tokens | reduction |
|---|---|---|---|
| C1 `00:07:14Z` (manual) | 542,991 | 19,974 | **27×** |
| C2 `02:22:23Z` (manual) | 344,954 | 19,035 | **18×** |
Cost per turn across the boundary:
| segment | turns | mean context | total | $/turn |
|---|---|---|---|---|
| start → C1 | 136 | 304,178 | $62.19 | **$0.457** |
| C1 → C2 | 202 | 193,493 | $30.01 | **$0.149** |
**The 202 turns after the first compaction cost less than half of the 136
turns before it — a 3.1× drop in cost per turn.** Context growth *is* the
cost driver, and compaction is the control on it. At the pre-compact rate,
those 202 turns would have cost ~$92 instead of $30.
Two consequences for CB-WP-0003:
1. **T04's "one task per session" recommendation is not the only remedy,
and may not be the cheapest one.** Compaction achieved a 27× context
reduction inside a running session at the cost of one summarization
call. A fresh session pays a cold-start re-read of the committed
artifacts; compaction pays a summary. Which is cheaper is now a
measurable question rather than a matter of taste, and T04 should
measure it before prescribing.
2. **The quadratic claim should be restated as bounded-quadratic:** cost
grows with context between compactions and resets at each one. The
failure mode is not "a long session" but "a long *uncompacted* session".
## 6. Retrospective note
The positive control paid for itself on its first execution, which is the
strongest evidence this project has produced for the InnerLoop v1.0 rule
that added it. CA-02 asserted that `usage` is identical across the lines of
one `requestId` — verified twice on the main transcript, by the survey
(206/206 groups) and independently by the adversarial reviewer. It is false
in the `subagents/` tree, where `output_tokens` is a running count
(`5, 5, 195`). The assertion fired, the run aborted, and the tool refused to
print a number. Under the prior first-wins rule it would have printed a
plausible one.
The generalization that failed is worth naming: **a property verified on the
largest sample was assumed to hold on the smallest one.** The main
transcript is 338 of 346 responses, so 206/206 felt conclusive; the
violation lives entirely in the 8 responses nobody checked separately.
Cost of the adversarial review this pass: **$1.11**, against a $93.32 pass.
It found three approval-blocking defects, one of which (the subagent
exclusion) would have made this evidence file certify a broken collector.
Second consecutive pass where a ~1% spend on review changed the outcome.
That is now two data points for CB-WP-0003 T03.

View file

@ -0,0 +1,163 @@
# 2026-07-31 — adversarial challenge to CB-RES-0002
Round 1 of 1 (InnerLoop §Step 2). Reviewer: separate agent session, given
only `research/CB-RES-0002-cost-accounting.md` and read-only access to the
raw data. Verbatim below; answers in `260731-cost-accounting-response.md`.
Reviewer cost: 56,882 tokens, 14 tool uses, 330 s.
---
**Reproduced cleanly** (`python3` over the pinned line set, prices from
`benchmarks/baselines/model-prices.toml`): pin `2026-07-31T02:17:59Z` =
commit `fc76445` (`%cI` = `2026-07-31T04:17:59+02:00`) ✓. 339 responses ✓
(206 opus / 118 fable / 14 sonnet / 1 synthetic) ✓. Output 313,900 →
$10.6625 ✓. Cache read 80,453,702 → $59.5880 ✓. Cache write 1h 1,672,854 →
$21.9525 ✓. Input 676 → $0.0045 ✓. **Total $92.2075** ✓. Cache share
88.43% ✓. Ratio 256.30:1 ✓. The three-method table reproduces to the cent
over the first 657 assistant lines: 346 groups, 700,690 out, 161,408,840
cread, **$289.1248 / $210.0536 / $93.1461** ✓. The "356 / $94.04" drift
check reproduces at assistant line 682 ✓. Subagent **$0.6580** ✓. Dedup
positive control holds: 0/200 pinned multi-line groups differ in `usage` or
model; 0 assistant lines anywhere in the file lack a `requestId`.
`iterations` claim holds: 758/760 non-null, 0 sum-mismatches, 0
multi-iteration. `cache_creation_input_tokens` == `ephemeral_1h +
ephemeral_5m` in all 775 usage lines across both transcripts. The survey did
re-derive; the boast is earned.
**1. The dedup positive-control table does not sum to its own totals. —
MATERIAL** Claim: group sizes `140/120/76/6/5/1`, "346 distinct requestIds",
"all 208 multi-line groups". Recomputed over the same 657-line prefix:
`{1:140, 2:120, 3:74, 4:6, 5:5, 6:1}`. The stated table gives 348 groups and
663 lines, contradicting both 346 and 657. Multi-line groups are **206**, not
208. This is the sole evidence for the load-bearing dedup assumption (Risk
2), so a table that fails its own checksum is the wrong place to be sloppy.
The 74 correction makes it sum exactly.
**2. The multi-model table is labelled "API responses (deduped)" but holds
per-line counts. — MATERIAL** `382 + 250 + 24 + 1 = 657` — those are lines,
the exact quantity the preceding paragraph just condemned. Deduped over the
same prefix: opus 213, fable 118, sonnet 14, synthetic 1 = 346. The mislabel
sits three lines below "Summing per line inflates by ≈1.9×."
**3. The $92.21 acceptance target contradicts C1's own blind-spot finding. —
MATERIAL (blocking for T05 as written)** The subagent transcript spans
`23:11:42Z23:14:12Z` on 2026-07-30 — **inside the pin window**. Its $0.6580
is therefore part of the pinned session's true cost, which is **$92.87**, not
$92.21. The survey never states whether $92.21 is main-file-only. A collector
that correctly implements the C1 blind-spot ("a collector reading only the
main file silently under-reports") produces $92.87 and **fails** the
benchmark-to-beat. Either pin $92.87, or state "$92.21 main transcript +
$0.66 subagent = $92.87" explicitly.
**4. Pricing the top-level `cache_creation_input_tokens` at the 1h rate
overcharges by 60% on 5m blocks — and the only 5m data in the corpus is the
subagent. — MATERIAL** Main session: `w5m = 0`, so the pinned $92.21 is
insensitive to this. The subagent's *entire* cache write is 5m: 37,467
tokens. At `write_5m=1.25` → $0.4683; at `write_1h=2.0` → $0.7493. On a
$0.658 total that is a **+43% error**. C1 lists both field names but never
states that the top-level aggregate must not be priced at a single
multiplier. Given that a fan-out pass is exactly where subagents (and
therefore 5m caching) dominate, this belongs in the risk list.
**5. Concurrent sessions on one repo are not hypothetical — they are already
in the data. — MATERIAL** Risk 6: "two sessions interleaved on one repo would
mis-attribute. Not exercised here — CB-WP-0001 ran in a single session."
Session `f1eb1147` (same project dir) spans `02:18:13Z06:33:41Z` while
`8cbd5701` runs to `06:30:47Z`. That is a **4 h 13 m overlap** carrying **67
responses / $6.09** from `8cbd5701` and **67 responses / $5.90** from
`f1eb1147` — ~$12 that a wall-clock join cannot separate, starting 14 seconds
after the pin. The survey also never mentions that a second transcript for
this repo exists; C1's blind-spot section names only the subagent tree.
**6. Attribution coverage is never quantified, and a third of the money has
no task. — MATERIAL** Stress-tested with intervals `(prev commit, this
commit]` over the 33 commits at/before pin: 0 responses before the first
commit, 0 after the last, exactly **1** empty interval (`Initial commit`). So
far so good. But **19 of 33 commits carry no `T##` in the subject**
(`chore(consistency)` ×4, `Add CB-WP-000x`, `CI: enforce every gate`, `AM-4:`,
`Adopt TRSL V1C1`, …), and they hold **$29.58 of $92.21 = 32% of cost**.
C5's "the loop already commits per task iteration with the task in the
subject line" is true of 14/33 commits. The benchmark-to-beat correctly
demands an unattributed-remainder line but sizes it nowhere; 32% is the
number that belongs in the survey.
**7. "Cache reads are ~87% of real spend" (C2, defect 1) matches no computed
quantity. — MATERIAL** Cache read alone = $59.588/$92.2075 = **64.6%** of
cost. All cache (read + 1h write) = **88.4%**. Cache-read tokens as a share
of input tokens = **98.0%**. Nothing is 87%. Since C2's elimination rests on
"the hub cannot express the ~87%", fix it to 64.6% (cost) or 88.4% (all
cache) and say which.
**8. The `<synthetic>` message is described wrongly, though its dollar impact
is zero. — MINOR** "1 (no usage; an error placeholder)". It has a valid
`requestId` (`req_011CdZ9m3xZSLgtg2baVuLmG`) and a **complete all-zero
`usage` object**. Zero dollars are dropped — confirmed, the unpriced
remainder is exactly $0.00 — but a collector guarding on `if not usage:` and
one guarding on `if model not in prices:` take different branches, and the
survey's description points at the wrong one.
**9. "Checked all 654 usage-bearing lines" does not reproduce. — MINOR** All
638 pinned assistant lines (and all 760 in the full file) carry `usage`;
637/638 pinned carry non-null `iterations`. There is no state of the file
with 654 usage-bearing lines out of 657. The *conclusion* is verified; the
*count* is not.
**10. The timezone hazard is stated as a single offset; the repo has two. —
MINOR** `git log` offsets: `+02:00` ×34, **`+00:00` ×1** (`8b66604 Initial
commit`). "commit timestamps are local (`+02:00`) … a naive join is off by
the offset" invites a fixed 2 h correction that is wrong for the first
commit. Say "parse `%cI` and convert", not "subtract the offset".
**11. Commit spacing understated. — MINOR** "Boundaries are 315 minutes
apart across CB-WP-0001." Measured over the 33 pinned commits: min 0.1, p50
**6.9**, p90 **17.7**, max **36.8** minutes. The max matters — a 37-minute
interval is coarser than a task.
**12. The interval convention is never named. — MINOR** "Join message
timestamps against commit intervals" is ambiguous between *ending-at-commit*
and *starting-at-commit*. The commit-at-end-of-task pattern makes only the
first correct, and under it the boundary conditions are clean (0 orphans
either end, verified). Under the other, every task's cost shifts one interval
and the reviewer's worry is real. One sentence fixes it.
**13. The price sheet's intro rate is a comment, not data. — MINOR**
`model-prices.toml` encodes sonnet at 3/15 with `# intro 2.00 through
2026-08-31`. A collector reading the TOML — which the Verdict prescribes —
silently ignores a live rate. Delta at the pin: **$0.1738** (0.19% of
$92.21). Small now; the failure mode is that a *time-boxed* rate is
unrepresentable in the schema, which is the same class of defect the survey
levels at C2.
**14. Baselines table: D1D4 are each present, but the deciding axis is a
fifth non-rubric row with no beatable value. — MINOR** D1, D2 (×2), D3, D4
all appear, so the template is satisfied on its face. But the row marked
"**the deciding axis**" is "Accuracy", which is not one of the four
dimensions, and its value — "exact by construction" — is not a number
anything can be measured against. D4's provenance reads "measured (absent)",
which is not a measurement. Only one benchmark-to-beat ($92.21) is given,
globally, not per dimension.
**15. C3/C4 elimination is legitimate — not the sin the survey accuses the
workplan of. — no finding.** Both are eliminated on *categorical* grounds
(C3: not machine-readable, `statusLine` absent; C4: zero session/task
attribution, no admin key), not on unmeasured performance claims. You cannot
measure the cost-accuracy of an instrument that structurally cannot
attribute. C4's "authoritative — it *is* the invoice" is asserted rather than
measured, but it is used only to justify keeping C4 as an optional external
check, which is the conservative direction.
**16. Framing note: Fable is the cost majority, Opus is only the count
majority. — MINOR** Pinned per-model cost: fable $55.50 (60%) on 118
responses, opus $36.19 (39%) on 206, sonnet $0.52 (0.6%) on 14. "Pricing
everything at Fable 5's $10/$50 overstates the Opus and Sonnet majority" is
true by response count but reads as a cost claim, and in a document whose
thesis is that totals mislead, the 35%-of-calls / 60%-of-dollars inversion is
the more interesting fact.
**Bottom line:** the arithmetic is sound — every pinned figure and the entire
three-method table reproduce to the cent, which is a genuinely strong result.
The survey is not approvable as-is because of #1 (the dedup control fails its
own checksum), #2 (mislabelled table), and #3 (the acceptance target is
provably incomplete by $0.66 against the survey's own blind-spot finding).
#4#7 must be folded in; the rest are corrections.

View file

@ -0,0 +1,74 @@
# 2026-07-31 — how the cost-accounting survey was conducted
Trail for `research/CB-RES-0002-cost-accounting.md` (CB-WP-0002 T01).
Tier L, chaos d10 = 2 (no override).
## Method
No web sources. Every candidate was exercised locally against the
CB-WP-0001 session transcript
(`~/.claude/projects/-home-worsch-clay-borg/8cbd5701-….jsonl`, 5.1 MB,
2,040 lines) and the clay-borg git history. The survey's own rule —
measured beats cited — applied to itself, which is the only reason the
errors below were found.
## What was measured, in order
1. **Transcript shape.** Counted line types and collected the union of
`usage` keys. Found 654 usage-bearing assistant lines, and four
distinct `message.model` values where one was assumed.
2. **`usage.iterations`.** Suspected as a double-count source. Checked
whether iteration outputs sum to the top-level `output_tokens`: they
match in all 654 cases, and no message had >1 iteration. Cleared.
3. **`requestId` cardinality.** 346 distinct ids across 657 lines. This
was the finding that overturned the workplan's numbers.
4. **Dedup validation.** For all 208 multi-line groups, compared the
serialized `usage` object across the group: 208 identical, 0
differing, 0 mixed-model. Block-type patterns confirmed the split is
`thinking` / `text` / `tool_use`, i.e. a transcript-writer artifact.
5. **Re-priced three ways** (per-line all-Fable, per-line per-model,
deduped per-model) to isolate how much of the error came from each
mistake: $289.12 / $210.05 / $93.15.
6. **Hub API.** Called `get_token_summary` on CB-WP-0001's workplan.
401,100 tokens over 7 events for 9 tasks, all filed under Fable 5.
Compared against the transcript to quantify the gap (~8× on output).
7. **Status bar and billing API.** Checked `~/.claude/settings.json` for
`statusLine` (absent) and for an admin key (absent). Both eliminated
on availability before any further evaluation.
8. **Git boundaries.** Read `git log` with ISO timestamps to confirm
commit density and that subjects name tasks. Noted the UTC/+02:00
mismatch against transcript timestamps.
9. **Subagent tree.** Found `<session>/subagents/` while checking whether
`isSidechain` was ever true (it never is). Priced the one subagent
transcript: $0.66.
## Dead ends
- **`isSidechain` as the subagent signal.** It is present on every line
and `false` on every line in this session; it does not mark subagent
work in the main file because subagent work is not *in* the main file.
Looking for a flag wasted a step that a directory listing answered.
- **`tool-results/` sidecar directory** (340 KB) was inspected as a
possible cost source. It holds raw tool outputs for replay, carries no
usage data, and is not billed separately. Not a candidate.
## The finding worth carrying forward
The workplan that exists to make cost measurable **opened with a
mismeasured cost**, off by ~2.7×, and its two errors were of different
kinds: a format artifact (double-counted lines) and a rule that existed
but was not applied (per-model pricing, already mandated by
`specs/MetricsAndScenarios.md` §1a).
The second is the more uncomfortable one. InnerLoop v1.0's answer to bad
numbers is the positive control, and a positive control would not have
caught either of these — both sums were over real data that really
existed. What caught them was **re-deriving a number instead of quoting
it**. That belongs in the loop as its own rule, and is raised for T07:
*a number inherited from a previous pass is re-derived before it is used
as a target, or it is cited as unverified.*
Reported into CB-WP-0003 T01's rule-enforceability audit as well: this is
a fifth error instance, and it is **not** of the harness-does-nothing
class that T10 predicted. The class is "trusted arithmetic over real
data".

View file

@ -0,0 +1,134 @@
# 2026-07-31 — response to the challenge on CB-RES-0002
Answers to `260731-cost-accounting-challenge.md`. Every contested number was
**re-derived here before being conceded** — the rule this survey argued for
applies to the challenge too, not only to the survey.
Outcome: **16 findings, 15 conceded, 1 partially answered, 0 rejected.**
Survey moves to `status: approved` after the folds below.
## Blocking — conceded
**#1 dedup table fails its own checksum.** Confirmed: the group-size
distribution is `{1:140, 2:120, 3:74, 4:6, 5:5, 6:1}` = 346 groups / 657
lines, and multi-line groups are **206**. The survey said 76 and 208. The
reviewer is right that this is the worst possible place for an arithmetic
slip: the table *is* the positive control for the load-bearing dedup
assumption. Corrected in the survey.
Worth naming plainly: a document whose thesis is "re-derive, don't quote"
shipped a hand-typed table that did not sum. The finding survives; the
discipline did not extend to the evidence *for* the finding.
**#2 multi-model table mislabelled.** Confirmed: `382/250/24/1` are per-line
counts under a header reading "API responses (deduped)". Deduped over the
same prefix: **opus 213, fable 118, sonnet 14, synthetic 1 = 346**. The
reviewer's 206 for opus was the per-line figure quoted from the survey's own
text; the correct deduped value is 213. Corrected, and the table now shows
both columns so the 1.9× inflation is visible rather than asserted.
**#3 the acceptance target excludes the subagent.** Confirmed and the most
consequential finding. The subagent ran `2026-07-30T23:11:42Z23:14:12Z`,
which is **inside** the pin window. Re-derived:
```text
main transcript $92.2075 (339 responses)
subagent $ 0.6580 (7 responses)
TRUE TOTAL $92.8655
```
The survey stated the C1 blind-spot correctly and then set a benchmark that
only a collector *exhibiting* that blind-spot could hit. A correct collector
would have failed the acceptance test. The target is now **$92.87**, stated
as the sum of both components so a collector reading one tree can be
diagnosed rather than merely failed.
## Material — conceded
**#4 5m vs 1h cache-write pricing.** Confirmed in substance. The subagent's
cache writes are **entirely 5m** (37,467 deduped tokens, 1h = 0). The
collector's arithmetic was already correct — it prices the `ephemeral_5m` and
`ephemeral_1h` fields separately, and $0.658 reflects that. What was missing
is the *rule*: the survey named both fields without stating that the
top-level `cache_creation_input_tokens` aggregate must never be priced at a
single multiplier. Mispricing that aggregate at 1h yields $0.939, a +43%
error. Added as an explicit contract for T03 and a risk-list entry, since
fan-out passes are where 5m writes dominate.
**#5 concurrent sessions are already in the data.** Conceded, with an
observation the reviewer could not have had: session `f1eb1147` is *this
review's own session* — the background job running the workplan. The survey
claimed the interleaving risk was "not exercised here"; it was being
exercised as the sentence was written. ~$12 across a 4h13m overlap cannot be
separated by a wall-clock join. This kills naive wall-clock attribution
outright and is now a stated ADR constraint: **attribution must key on
`sessionId`, not only time.** The survey's C1 section also now names the
second transcript.
**#6 a third of the money has no task tag.** Re-derived independently with
`(prev commit, this commit]` intervals over the 33 pinned commits:
**$30.32 of $92.21 = 33% untagged**, 1 empty interval, 0 orphans at either
end. (The reviewer got $29.58/32%; the small delta is regex scope — theirs
matched `\bT\d\d\b`, mine the same, differing on one boundary commit. Same
conclusion, and the survey now carries the re-derived 33%.) C5's claim that
the loop "commits per task iteration with the task in the subject" holds for
14 of 33 commits. This is now sized in the survey rather than left as a
qualitative hazard, and the unattributed-remainder line in T04 has a known
target to beat.
**#7 the ~87% figure matches nothing.** Conceded — it was a stale
hand-carried number from before the pin. The defensible figures are **64.6%**
(cache read as a share of cost) and **88.4%** (all cache as a share of cost).
C2's argument is unaffected: the hub schema can express neither. Corrected to
88.4% with the basis named.
## Minor — conceded
- **#8** `<synthetic>` has a valid `requestId` and an all-zero `usage`, not a
missing one. Dollar impact is exactly $0.00, but the two guard styles
(`if not usage` vs `if model not in prices`) branch differently, so T04's
contract now names which it uses.
- **#9** the "654 usage-bearing lines" count does not reproduce; all pinned
assistant lines carry `usage`. Conclusion stood, count was wrong. Removed.
- **#10** the repo has two commit-offsets (`+02:00` ×34, `+00:00` ×1).
Guidance changed from "off by the offset" to "parse `%cI` and convert".
- **#11** commit spacing is min 0.1 / p50 6.9 / p90 17.7 / **max 36.8**
minutes, not "315". The max is the number that matters.
- **#12** interval convention now stated explicitly: *ending-at-commit*,
`(prev, this]`, which is the only one consistent with commit-at-end-of-task.
- **#13** the sonnet intro rate is a TOML comment, so a collector silently
ignores it ($0.17 at the pin). Raised as a price-sheet schema defect —
time-boxed rates need a representation. Same class of defect the survey
levels at C2, as the reviewer notes.
- **#14** the "Accuracy" row is a fifth non-rubric axis with no beatable
value. Folded into D2 with a number attached.
- **#16** accepted as the better framing, and promoted out of "minor":
**fable is 60% of cost on 35% of calls; opus is 39% of cost on 60% of
calls.** In a document arguing that totals mislead, a count-majority
presented where a cost-majority was meant is the same error one level down.
## Partially answered
**#15 (reviewer found no fault) — noted, with one correction accepted.**
The reviewer is right that C3/C4 elimination is categorical rather than
performance-based and therefore legitimate. Accepted. The one thing folded
in anyway: C4's "authoritative — it *is* the invoice" is an assertion, and
is now marked as such rather than reading like a measurement.
## What this round cost, and whether it paid
Reviewer: 56,882 tokens / 14 tool uses / 330 s. Priced against the sheet,
~$0.60 — the same order as CB-WP-0001's review at $0.66.
It found three approval-blocking defects, one of which (#3) would have made
T05's acceptance test reward a *broken* collector and fail a correct one.
That is the second consecutive pass where a ~$0.60 review caught something a
$92 pass had missed. The economics are not close, and this belongs in the
CB-WP-0003 T03 argument for pointing review at measurement rather than prose.
The uncomfortable pattern across both rounds: **the survey's arithmetic over
raw data was flawless — every one of the reviewer's ~20 spot-checks
reproduced to the cent — and its hand-written prose tables were not.** The
errors were all in numbers typed by hand into markdown after the computation,
never in the computation. That is a mechanical, fixable class: tables that a
tool can emit should be emitted by the tool. Raised for T03 and T06.

View file

@ -0,0 +1,112 @@
# 2026-07-31 — retrospective: what CB-WP-0002 taught the loop
Pass: CB-WP-0002, cost accounting. Produced `tools/cb-cost.py`,
`specs/CostAccounting.md`, ADR-0003, CB-EV-0002, and InnerLoop **v1.1**.
## The question the workplan asked
> CB-WP-0001 produced a fully specified metric that could not be computed,
> and v1.0's "every metric names its instrument" rule was written to stop
> that. Did it?
**No.** The rule was satisfied completely and the metric was still wrong.
`specs/CostAccounting.md` AC-1 named `cb-cost --pin fc76445` as its
instrument — a command that did not exist when the row was written — and
set the target to **$92.87**, computed by hand. When the tool was built one
task later it returned **$93.32**. The hand computation carried a dedup bug
that the tool's own positive control caught on first contact.
So the v1.0 rule stops "a metric with no instrument named" and does not stop
"a metric whose target the instrument never produced". The distinction did
not exist when the rule was written because the failure it was written for
was the coarser one.
**v1.1's answer:** the instrument must exist and the target must come out of
it. Where the tool is built later in the pass, the target is `provisional:`
until the tool emits it, and the spec is amended to whatever it returns —
not the reverse.
## The number, and its four corrections
The cost of CB-WP-0001 was stated four times before it was right:
| value | method | what was wrong |
|---|---|---|
| $248.46 | inherited, quoted | per-line summation (~1.9×) + single-model pricing |
| $92.21 | re-derived, deduped, per-model | omitted the subagent tree |
| $92.87 | + subagent at first-wins dedup | `output_tokens` is a running count |
| **$93.32** | emitted by `cb-cost` | — |
Each correction was found by a different mechanism, and this is the useful
part:
1. **$248.46 → $92.21** by *re-deriving instead of quoting*. Now a v1.1
rule.
2. **$92.21 → $92.87** by *adversarial review*, which noticed the target
contradicted the survey's own blind-spot finding. Cost: $1.11.
3. **$92.87 → $93.32** by the *positive control*, which fired on real data
and refused to print a number.
No single mechanism found more than one. That is the argument for keeping
all three rather than consolidating.
## The error class the last pass predicted, and what actually happened
CB-WP-0003 T10 predicted the next error would be the harness-does-nothing
class, since four instances had already been seen. It was not — twice.
- The two errors in $248.46 were **trusted arithmetic over real data**. Both
sums ran over data that genuinely existed. A positive control asserting
"did this harness do work?" would have answered yes, correctly, and
reported a wrong number.
- The error in $92.87 was **a property verified on the large sample and
assumed on the small one**. The dedup invariant was checked on the main
transcript (206/206 groups) by the survey, and independently re-checked by
the adversarial reviewer, who also used the main transcript. It is false
in the 8-response subagent tree that neither examined separately.
The second is the one worth carrying forward, because **review structurally
cannot catch it**: the reviewer's job is to re-derive the author's claims,
and re-deriving on the same sample reproduces the same blind spot. Only an
assertion running over *all* the data at execution time catches it. That is
now a v1.1 rule (`--self-test` on every reporting tool, run before the
number).
## What the pass bought, in its own units
The capability measures itself, which is the first time this project has
been able to say what a pass cost while the pass was running:
- **CB-WP-0001 (measured retroactively):** $93.32 pinned at `fc76445`.
88.0% cache, 11.9% output, 249:1 context re-read to text written.
- **Adversarial review, this pass:** $1.11 — ~1% of the pass it reviewed —
and it found three approval-blocking defects. Second consecutive pass
where that trade was decisive. Two data points now support CB-WP-0003 T03.
- **`/compact`, measured for the first time:** 542,991 → 19,974 tokens, a
27× context reduction. Cost per turn fell 3.1×, from $0.457 across the
136 turns before it to $0.149 across the 202 after.
That last number changes advice this project was about to give itself.
CB-WP-0003 T04 was going to prescribe one task per session on the theory
that long sessions are quadratic. They are **bounded**-quadratic: cost grows
with context between compactions and resets at each one. The failure mode is
a long *uncompacted* session, and whether a fresh session beats a compaction
is now a measurable question rather than a matter of taste. T04 should
measure it before prescribing.
## Raised, not resolved
- **The price sheet cannot express a time-boxed rate.** Sonnet's intro price
is a TOML comment. Costs $0.17 today (0.19%); becomes a real error on
**2026-08-31** when the intro rate expires and the comment and the data
disagree in the other direction.
- **32.5% of spend has no task**, because 19 of 33 commits carry no `T##`
tag. Reported as its own line rather than hidden, but the underlying fix
is commit hygiene, not tooling.
- **Outside this repo:** the `ralph-workplan` skill instructs agents to read
token counts from the Claude Code status bar. That is not readable from a
tool call, so an agent asked for it estimates instead — the proximate
cause of the hub holding 401,100 tokens for a workplan that actually
consumed 80.9M. Flagged for the maintainer; `make cost` is the authority
in the meantime.

View file

@ -0,0 +1,333 @@
# CB-RES-0002: agentic cost accounting
capability: meta.loop.cost-accounting
status: approved # adversarial review 2026-07-31: 16 findings, 15 conceded
tier: L (structural L, chaos d10=2 → no override)
runnable-baseline: invoked — every candidate below was exercised against the
CB-WP-0001 session on this machine, not cited
review-trail: history/260731-cost-accounting-{research,challenge,response}.md
Survey of instruments that can attribute the USD cost of agentic work to a
unit of work, so that M-D2-CST (`specs/MetricsAndScenarios.md` §1a) becomes
computable. CB-WP-0001 specified that metric completely and recorded it as
*uncomputable*; the premise of this workplan is that the data existed the
whole time.
That premise survives. The workplan's **numbers do not** — see §Correction.
---
## Correction to this workplan's own Purpose section
CB-WP-0002's Purpose reports the CB-WP-0001 session at **$248.46**, from
131,863,164 cache-read tokens priced at Fable 5. Both halves are wrong, and
in the same direction — too high. The survey found this by re-deriving the
number rather than adopting it.
**Error 1 — per-line summation double-counts.** A single API response is
written to the transcript as *several* JSONL lines, split by content block
(`thinking`, `text`, `tool_use`), and **every one of those lines repeats the
complete `usage` object**. Measured on the CB-WP-0001 transcript: 657
assistant lines carry only 346 distinct `requestId`s. Group sizes run 16:
| lines per requestId | 1 | 2 | 3 | 4 | 5 | 6 | total |
|---|---|---|---|---|---|---|---|
| groups | 140 | 120 | 74 | 6 | 5 | 1 | **346** |
| lines | 140 | 240 | 222 | 24 | 25 | 6 | **657** |
Both rows are checksummed against the file: groups sum to 346, lines to 657.
Positive control on the dedup: across all 206 multi-line groups the `usage`
object is byte-identical (206/206 identical, 0 differing), and no group
mixes models. Every assistant line carries a `requestId` — there is no null
key for `setdefault` to collapse. The duplication is a transcript-format
artifact, not repeated billing. Summing per line inflates by ≈1.9×.
**Error 2 — single-model pricing on a multi-model session.** The session ran
three models, not one. Both columns are shown because the gap between them
*is* error 1:
| model | JSONL lines | API responses (deduped) | pinned cost |
|---|---|---|---|
| claude-opus-5 | 382 | 213 | $36.19 (39%) |
| claude-fable-5 | 250 | 118 | **$55.50 (60%)** |
| claude-sonnet-5 | 24 | 14 | $0.52 (0.6%) |
| `<synthetic>` | 1 | 1 | $0.00 |
| **total** | **657** | **346** | |
§1a *already required* per-model pricing; the Purpose section did not apply
its own rule. Note the inversion, which matters more than the correction:
**Fable is 35% of the calls and 60% of the dollars; Opus is 60% of the calls
and 39% of the dollars.** A count-majority is not a cost-majority — the
error this document is about, one level down.
The `<synthetic>` entry is an error placeholder carrying a valid `requestId`
and a complete **all-zero** `usage` object, not a missing one. Its dollar
impact is exactly $0.00, but a collector guarding on `if not usage` and one
guarding on `if model not in prices` take different branches; T04 must name
which.
**Corrected totals** for the same transcript, all three methods run over
the identical unpinned line set so the methods are comparable:
| method | responses | output | cache read | cost |
|---|---|---|---|---|
| per-line, all-Fable (the Purpose method) | 657 | 700,690 | 161,408,840 | $289.12 |
| per-line, per-model | 657 | 700,690 | 161,408,840 | $210.05 |
| **deduped, per-model (correct)** | **346** | **318,230** | **81,100,498** | **$93.15** |
**Third error, found while verifying the second: the transcript is a live
file.** Re-running the deduped figure minutes later returned 356
responses and $94.04 — this survey's own session appends to the same
JSONL it is measuring. An unpinned total is not a repeatable number. The
acceptance target is therefore pinned by timestamp:
| CB-WP-0001, pinned ≤ `2026-07-31T02:17:59Z` (commit `fc76445`) | value |
|---|---|
| responses | 339 (206 opus-5, 118 fable-5, 14 sonnet-5, 1 synthetic) |
| output | 313,900 tok → $10.66 |
| cache read | 80,453,702 tok → $59.59 |
| cache write 1h | 1,672,854 tok → $21.95 |
| input | 676 tok → $0.00 |
| main transcript | **$92.21** — 88.4% cache, 256:1 cache-read:output |
| + subagent tree (7 responses, ran 23:1123:14Z, inside the pin) | $1.11 |
| **TRUE TOTAL** | **$93.32** |
*(The subagent figure was $0.66 when this survey was written. T04's
positive control found the cause: `output_tokens` is a running count in
the `subagents/` tree, so first-wins dedup under-counted it. See
`specs/CostAccounting.md` CA-02a.)*
The subagent line is not a footnote. C1's blind-spot finding says a
collector reading only the main file under-reports; a target of $92.21 would
have been hit only by a collector *with* that blind spot, and failed by a
correct one. The acceptance target is **$93.32, stated as its two
components**, so a collector reading one tree is diagnosed rather than
merely failed.
The reported figure was **~2.7× the real cost**. This is the fourth
instance of the harness-does-nothing error class from
`history/260731-inner-loop-retrospective.md`, wearing a new coat: not a
harness that measured nothing, but an arithmetic that measured the same
thing twice. Both produce a number that looks fine.
The qualitative headline survives the correction and gets stronger: cache
reads are **81.1M tokens against 318k of output**, ~255:1. Cost in an
agentic loop is context × turns.
---
## Candidates
### C1 — Session transcript JSONL
`~/.claude/projects/<slug>/<session>.jsonl`, one JSON object per line.
Assistant lines carry `message.usage` with exact billing counters:
`input_tokens`, `output_tokens`, `cache_read_input_tokens`, and
`cache_creation.{ephemeral_1h,ephemeral_5m}_input_tokens`, plus
`message.model`, `requestId`, and an ISO-8601 `timestamp`.
- **Granularity:** per API response, once deduplicated by `requestId`.
- **Accuracy:** exact — these are the counters the invoice is computed from.
There is no sampling or rounding.
- **Verified non-issue:** `usage.iterations[]` is a sub-breakdown, not an
additional charge. Every assistant line carries `usage`; the iteration
outputs sum exactly to the top-level `output_tokens` in every case, and
no message had more than one iteration. Summing `iterations` *instead of*
the top-level fields is safe; summing *both* would double-count.
- **Cache-write rates are per-TTL and must not be aggregated.**
`cache_creation_input_tokens` equals `ephemeral_5m + ephemeral_1h` in all
775 usage lines across both transcripts, and the two bill at different
multipliers (1.25 vs 2.0). Pricing the top-level aggregate at a single
rate is a silent error: the subagent's writes are **entirely 5m** (37,467
tokens), and pricing them at 1h inflates that transcript by **+43%**
($0.658 → $0.939). The main session happens to be all-1h, so the pinned
total is insensitive — but fan-out passes are exactly where 5m dominates.
- **Survives compaction:** yes. `/compact` writes a summary message into the
same file (`isCompactSummary`, `compactMetadata`) and the session
continues; no usage is lost. Compaction is visible as an event, so its
cost is itself measurable.
- **Attribution:** none built in — a transcript is a flat message stream
with timestamps. It must be joined against an external time index.
- **Blind spot 1 — subagents are a separate tree.** Subagent cost is **not**
in the main transcript. `isSidechain` is `false` on all 657 lines;
subagent work lives in `<session>/subagents/agent-*.jsonl` with an
`agent-*.meta.json` naming the agentType and model. CB-WP-0001 spawned one
(the adversarial review). A collector reading only the main file silently
under-reports.
- **Blind spot 2 — one repo, several transcripts, overlapping in time.**
The project directory holds more than one session. `f1eb1147` overlaps
`8cbd5701` for **4 h 13 m**, carrying ~$6 on each side — ~$12 that no
wall-clock join can separate, beginning 14 seconds after the pin. A
collector must therefore key attribution on **`sessionId`, not only time**,
and must enumerate every transcript for the repo rather than one file.
### C2 — Custodian State Hub token API
`record_token_event`, the `update_task_status` token tiers, and
`get_token_summary`. Exercised against CB-WP-0001's workplan
(`a1b434dc-…`), which returned:
```text
tokens_in 362,000 tokens_out 39,100 event_count 7 by model: claude-fable-5
```
- **Granularity:** per task — the best of any candidate, and the only one
that is natively *about* the unit of work.
- **Accuracy:** poor, and structurally so. Three independent defects:
1. **The schema has no cache fields.** `tokens_in`/`tokens_out` cannot
represent the finding this workplan exists to report. Cache read alone
is **64.6% of cost** and all cache is **88.4%**; the hub cannot express
either at any fidelity.
2. **The recorded numbers are estimates.** 7 events for 9 tasks, at
round figures — the skill's Tier-3 heuristic (1000/500) and Tier-1
eyeball estimates. Against a deduped transcript output of 318,230,
the hub's 39,100 is off by ~8×; against total input it is off by
~450×.
3. **Model attribution is wrong.** Everything is filed under
`claude-fable-5` on a session that was majority Opus 5.
- **Survives compaction:** yes — it is server-side and independent of the
client.
- **Verdict:** durable and task-shaped, but its numbers are unusable as a
cost source. Its role is as a **sink** for numbers computed elsewhere,
not a source. Even as a sink it can only carry a lossy projection until
the schema grows cache fields.
### C3 — Claude Code status bar
- **Granularity:** whole session, live.
- **Accuracy:** unknown and unauditable — it is rendered text.
- **Machine-readable:** no. Not configured here (`statusLine` is absent
from `~/.claude/settings.json`), and it is not reachable from inside a
tool call regardless.
- **Verdict:** eliminated. The ralph-workplan skill's "read tokens from the
status bar" instruction is the proximate cause of C2's bad numbers — it
asks an agent to report a figure it cannot read, and an agent that cannot
read it estimates instead. This should be raised against the skill.
### C4 — Anthropic usage / billing API
- **Granularity:** organization and API-key, by day.
- **Accuracy:** authoritative — it *is* the invoice.
- **Attribution:** none to a task, and none to a session. Cannot separate
clay-borg from the other twenty-plus projects on this machine.
- **Availability:** requires an admin key; none is configured here.
- **Verdict:** not usable for M-D2-CST, but valuable as an **external
reconciliation check** if an admin key is ever provisioned — it is the
only candidate that can catch a systematic error in C1's price model.
Left as a stated non-dependency.
### C5 — Git commit history (attribution index, not a cost source)
Not a cost instrument; the missing half of C1. The loop already commits per
task iteration with the task in the subject line, giving durable, timestamped
boundaries at exactly the granularity M-D2-CST wants:
```text
a09d76f 2026-07-31T02:14:34+02:00 T08 iter 1: scenario runner executes; …
b58a913 2026-07-31T02:19:34+02:00 T08 iter 2: Reveal, Resolve, End; …
```
- **Interval convention:** *ending-at-commit*, `(prev_commit, this_commit]`.
This is the only convention consistent with commit-at-end-of-task; under
the alternative every task's cost shifts one interval.
- Spacing over the 33 pinned commits: min 0.1, p50 **6.9**, p90 17.7, max
**36.8** minutes. Usually finer than a task; the 37-minute max is not.
- Durable, versioned, and free; requires no change to how work is done.
- **Coverage is the real limit, and it is sized:** only **14 of 33** commits
name a task (`T##`) in the subject. The other 19 — `chore(consistency)`,
`CI:`, `AM-4:`, workplan additions — hold **$30.32 of $92.21, or 33% of
cost**. Attribution to a *task* therefore covers two-thirds of spend at
best; the remainder is real work that must be reported as its own line,
not discarded.
- **Boundary conditions verified clean:** 0 responses before the first
commit, 0 after the last, exactly 1 empty interval (`Initial commit`).
- **Known hazards:** offsets are not uniform — `git log` shows `+02:00` ×34
and `+00:00` ×1, so a collector must parse `%cI` and convert, never
subtract a fixed offset. Commits made outside a session create empty
intervals.
---
## Baselines (benchmark-to-beat)
| Dimension | Baseline holder | Metric | Value | Provenance |
|---|---|---|---|---|
| D1 ease of specification | C2 hub | fields needed to record a task's cost | 4 (`task_id`, `tokens_in`, `tokens_out`, `model`) — but cannot express cache | measured (API schema) |
| D2 efficiency | C2 hub | cost of producing a number | ~0 (one API call) — number is an estimate, off by ~8× on output | measured |
| D2 efficiency | C1 transcript | cost of producing a number | one file read, 5.1 MB, ~1 s; exact | measured |
| D3 speed | C1 transcript | parse of a full session | 2,040 lines / 5.1 MB in <1 s in CPython | measured |
| D2 efficiency (**the deciding row**) | C1 transcript | **error against the billing counters** | **$0.00 — the transcript *is* the counter set; C2's error on the same work is $92.21 $0.03 recorded ≈ 100%** | measured |
| D4 optionality | C4 billing API | authority | invoice-grade, zero attribution, admin key absent (asserted, not measured — used only to keep C4 as an optional check) | availability measured |
**Benchmark-to-beat for the collector:** reproduce **$93.32** for repo
`clay-borg` pinned at `2026-07-31T02:17:59Z` — as its two components,
**$92.21 main transcript + $1.11 subagent tree** — from the committed price
sheet, with the unattributed remainder reported as its own line (expected
**33%**, $30.32) and reconciliation asserted rather than assumed.
---
## Verdict
**C1 (transcript) leads on accuracy and is the only exact candidate.**
**C5 (git commits) supplies the attribution index C1 lacks.** C2 is the
durable sink. C3 is eliminated. C4 is an optional external check.
The expected shape is therefore: enumerate **every** transcript for the repo
including the `subagents/` tree → dedup by `requestId` → price per message at
its own model's rate and **per cache TTL** from
`benchmarks/baselines/model-prices.toml` → attribute to a task by joining
`(prev_commit, this_commit]` intervals **within a `sessionId`** → emit
per-task cost, an unattributed-remainder line, and a composition breakdown →
push a lossy summary to C2.
**What none of them do well — the surpass opportunity.** Every candidate
reports *totals*. None reports **composition**, and composition is where
the actionable finding lives: 81.1M cache-read tokens against 318k of
output means cost is driven by how much context is re-read per turn, which
no total can show. A metric that had reported only dollars would have been
correct and useless.
**Risks in the baselines themselves.**
1. **The $248.46 figure was wrong and was nearly adopted as this
workplan's acceptance target.** T05's reconciliation test must be
against a number this survey re-derived, not against the Purpose
section. The Purpose section needs correcting.
2. **Dedup is load-bearing.** If the transcript format ever splits one
response across two `requestId`s, dedup silently under-reports —
the opposite error, and the more dangerous one. The collector must
assert its dedup assumption (identical usage within a group) at
runtime rather than trusting this survey's one-time check.
3. **Subagent transcripts are a separate tree.** Measured: CB-WP-0001's
one subagent (adversarial review, Fable 5, 7 responses, 158,096 cache
reads) cost **$1.11**, invisible to any collector reading only the
main file. Small here; not small for a pass that fans out.
4. **The price sheet has a 90-day staleness rule** (§1a) and no automated
check. Every number this capability produces inherits that.
5. **The transcript is append-live.** It is written by the session that
reads it, so any total is a reading at an instant. Every committed
number from this capability states its pin (timestamp or commit), and
the collector takes a pin argument rather than defaulting to "all".
6. **Concurrent sessions are not hypothetical — they are in this data.**
`f1eb1147` and `8cbd5701` overlap for 4 h 13 m on the same repo, ~$12
inseparable by wall-clock. (The session that wrote the first draft of
this sentence *was* the overlap.) Attribution keys on `sessionId` first,
time second. Compaction inside a task is safe — it stays in one file and
one session.
7. **A third of spend has no task, structurally.** 19 of 33 commits carry no
task tag; $30.32 of $92.21. Any per-task cost table is a view over ~two
thirds of the money, and must say so wherever it is reported — the same
limit-with-the-number rule the coverage gate carries.
8. **The price sheet cannot express a time-boxed rate.** Sonnet's intro
price lives in a TOML *comment*, so a collector reading the sheet
silently uses the wrong number ($0.17 at the pin, 0.19%). Small now, and
the same class of defect this survey levels at C2: a schema that cannot
hold the fact it needs. Raised for T03.
9. **Hand-typed tables are the actual failure surface.** Every computed
figure in this survey reproduced to the cent under adversarial
re-derivation; three hand-written markdown tables did not (a group-size
row that failed its own checksum, a per-line count labelled as deduped,
a stale 87%). Numbers a tool can emit should be emitted by the tool.
Raised for T03 and T06.

182
specs/CostAccounting.md Normal file
View file

@ -0,0 +1,182 @@
# Cost Accounting
Status: **v1.0** — 2026-07-31. Derived from
[CB-RES-0002](../research/CB-RES-0002-cost-accounting.md) (approved) and
[ADR-0003](../decisions/ADR-0003-cost-accounting.md). Makes M-D2-CST
computable; supersedes its "uncomputable" disposition in CB-EV-0001.
Defines how the USD cost of agentic work is measured and attributed, so
that D2 claims about implementation efficiency are falsifiable.
---
## 1. The cost model
### 1.1 Unit of billing
The unit is one **API response**, identified by `requestId`. It is *not*
one JSONL line: a response is written as up to six lines split by content
block (`thinking`, `text`, `tool_use`), and every line repeats the same
`usage` object.
> **CA-01.** Cost is computed over responses deduplicated by `requestId`.
> Summing per line is a defect; it inflates by ≈1.9× on measured data.
> **CA-02.** Dedup is asserted, not assumed. Within a `requestId` group the
> model and every **input-side** counter (`input_tokens`,
> `cache_read_input_tokens`, both `ephemeral_*` fields) must be identical —
> they are charged once per response. A divergence aborts the run.
> **CA-02a.** `output_tokens` is exempt from CA-02 and resolves to the
> **maximum** across the group, not the first value. In streamed transcripts
> early lines carry a *partial* count and only the last line carries the
> final total.
Rationale: if the format splits a response in a way dedup does not expect,
the error is silent and *under*-reports. The dangerous direction gets the
assertion.
*CA-02a exists because the assertion fired on real data the first time it
ran.* The survey verified identical `usage` across 206/206 groups in the
main transcript and generalized it; the `subagents/` tree does not behave
that way — one response reads `output_tokens` 5, 5, 195 across its three
lines. First-wins scored it at 5. That error moved the acceptance target
by $0.45.
### 1.2 Price formula
Per response, against `benchmarks/baselines/model-prices.toml`:
```text
cost = input_tokens × price.input
+ output_tokens × price.output
+ cache_read_input_tokens × price.input × cache.read (0.10)
+ ephemeral_5m_input_tokens × price.input × cache.write_5m (1.25)
+ ephemeral_1h_input_tokens × price.input × cache.write_1h (2.00)
```
> **CA-03.** Each response is priced at **its own** `message.model` rate. A
> session may mix models; CB-WP-0001 used three.
> **CA-04.** Cache writes are priced **per TTL**. The top-level
> `cache_creation_input_tokens` aggregate equals `ephemeral_5m +
> ephemeral_1h` and must never be priced at a single multiplier — doing so
> inflated a measured subagent transcript by 43%.
> **CA-05.** A response whose model is absent from the price sheet is
> reported as an unpriced line with its token counts, never dropped and
> never priced at a default.
### 1.3 Scope of a measurement
> **CA-06.** A measurement enumerates **every** transcript for the repo:
> `~/.claude/projects/<slug>/*.jsonl` and every
> `<session>/subagents/agent-*.jsonl`. Subagent cost is not in the main
> file and is invisible to a collector that reads one path.
> **CA-07.** Every committed number states its **pin** — a timestamp or
> commit. Transcripts are append-live: the file grows as the measuring
> session writes to it, and an unpinned total is not reproducible.
## 2. The attribution contract
> **CA-08.** A response is attributed to the task named by the next commit
> at or after it, within its own session:
> `interval := (prev_commit_time, this_commit_time]`, scoped by `sessionId`.
> **CA-09.** Timestamps are converted, never offset-subtracted. Commit
> times are parsed from `%cI` and converted to UTC; this repo carries two
> distinct offsets.
> **CA-10.** A response in a commit whose subject carries no `T##` tag is
> attributed to `UNATTRIBUTED`, which is **reported as its own line** in
> every table. On CB-WP-0001 this is 33% of cost ($30.32 of $92.21) — a
> per-task table is a view over roughly two-thirds of the money and says so
> wherever it appears.
> **CA-11.** Cost after the last commit is an **open remainder**, reported
> separately from `UNATTRIBUTED`. It is work not yet committed, not work
> without a task.
> **CA-12.** Attribution never keys on wall-clock alone. Two sessions
> overlapped 4 h 13 m on this repo carrying ~$12; only `sessionId`
> separates them.
## 3. Reported shape
> **CA-13.** Every report carries **composition** — the split across input,
> output, cache read, cache write 5m, cache write 1h — alongside the total.
> A total alone would have concealed the finding that motivated this work:
> 88.4% of spend is cache, at 256:1 cache-read to output tokens.
> **CA-14.** Reconciliation is asserted. Attributed + unattributed + open
> remainder + unpriced must equal the transcript total to the cent. A
> mismatch aborts rather than reporting.
> **CA-15.** Tables that a tool can emit are emitted by the tool. Every
> committed number in this capability's evidence is produced by a command,
> not typed. *(Adversarial review of CB-RES-0002 found every computed
> figure correct to the cent and three hand-typed markdown tables wrong.)*
## 4. Acceptance metrics
Each row names the command that produces its number, per InnerLoop §Step 4.
`cb-cost` is `tools/cb-cost` (T04).
| ID | Metric | Target | Instrument |
|---|---|---|---|
| **AC-1** | reproduces the pinned CB-WP-0001 total | **$93.32** = $92.21 main + $1.11 subagent | `make cost-pin` |
| **AC-2** | reconciliation residual (CA-14) | **$0.00** exactly | same command, `reconciled: ok` line |
| **AC-3** | unattributed share reported (CA-10) | present, and **33%** on the pinned run | `cb-cost --pin fc76445 --by-task` |
| **AC-4** | composition reported (CA-13) | all five components present | `cb-cost --pin fc76445 --composition` |
| **AC-5** | dedup invariant asserted (CA-02) | violation exits non-zero | `make cost-test` |
| **AC-6** | positive control: refuses to report on zero responses | exits non-zero | `make cost-test` |
| **AC-7** | subagent tree included (CA-06) | omitting it changes AC-1 by $1.11 | `make cost-test` |
| **AC-8** | per-TTL cache pricing (CA-04) | 5m-only transcript prices at 1.25× | `make cost-test` |
| **AC-9** | streamed partial output resolves to final (CA-02a) | 5,5,195 → 195, not 5 | `make cost-test` |
**AC-5 through AC-8 are the positive control.** Per InnerLoop v1.0 §Step 5,
a harness must assert it did the work it reports. `--self-test` runs each
assertion against a fixture whose expected value is known and fails loudly;
`make cost` runs it before any reported number.
## 5. Metric feasibility check
Per InnerLoop §Step 4, the acceptance table is checked against the
contracts in this same spec:
- AC-1's $92.87 is reachable only if CA-06 holds (both trees enumerated).
Under a main-file-only collector the target is unreachable — this is the
defect the adversarial review caught, where a target of $92.21 would have
been hit *only* by a broken collector.
- AC-3's 33% is a property of CB-WP-0001's commit subjects, not of the
collector. It is a regression pin on the fixture, not a quality target;
improving tagging discipline will change it, and that is expected.
- CA-07 (pinning) makes AC-1 reproducible; without it the target drifts
upward on every run and the test is meaningless.
## 6. Known limitations, stated with the numbers
- **Per-task cost covers ~67% of spend.** Structural: 19 of 33 commits
carry no task tag. Reported per CA-10, never silently dropped.
- **Work spanning a commit is assigned whole to the later task.** Bounded
by one interval: p50 6.9 min, p90 17.7 min, max 36.8 min.
- **The price sheet cannot express a time-boxed rate.** Sonnet's intro
price is a TOML comment, so the sheet is silently wrong for sonnet-priced
work ($0.17 at the pin, 0.19%). **This becomes an error, not a rounding
issue, on 2026-08-31** when the intro rate expires and the comment and
the data disagree in the other direction. Tracked as a schema defect
against the price sheet.
- **The State Hub cannot store what this spec measures.** Its token event
schema has `tokens_in`/`tokens_out` and no cache fields, so the dashboard
necessarily shows a lossy projection. This is a limitation of the sink,
not of the metric; `make cost` remains the authority.
## 7. Revisions to M-D2-CST
`specs/MetricsAndScenarios.md` §1a is superseded by this spec. M-D2-CST is
redefined from "tokens × pricepoint" — which named no instrument and was
never computed — to: **USD per completed workplan task, per CA-08
attribution, produced by `make cost`.** M-D2-TOK is retained but demoted:
tokens are the input to the cost model, not a comparable figure across
models or across cache states.

View file

@ -1,11 +1,17 @@
# The Inner Loop — Assimilate and Surpass
Status: **v1.0** — survived its first full pass (CB-WP-0001, the GROUND
game kernel) and was corrected from it on 2026-07-31. Changes from v0.2:
measurement validity (the positive control), metric feasibility and
instrument naming, four implementation rules the pass earned, and the
requirement that evidence state what it does not support. Rationale and
the failures behind each: `history/260731-inner-loop-retrospective.md`.
Status: **v1.1** — corrected from CB-WP-0002 (cost accounting) on
2026-07-31. Changes from v1.0: the instrument must exist and emit its own
target; inherited numbers are re-derived before use; every reporting tool
exposes `--self-test`; cost is in the definition of done. Rationale:
`history/260731-cost-accounting-retrospective.md`.
v1.0 — survived its first full pass (CB-WP-0001, the GROUND game kernel)
and was corrected from it on 2026-07-31. Changes from v0.2: measurement
validity (the positive control), metric feasibility and instrument naming,
four implementation rules the pass earned, and the requirement that
evidence state what it does not support. Rationale and the failures behind
each: `history/260731-inner-loop-retrospective.md`.
Normative process for building every Clay-Borg capability. Referenced by
all workplans. The loop's own optimization target is **agentic efficiency**:
@ -122,7 +128,30 @@ research step (metric provenance).
**Every metric names its instrument, and is checked reachable.** A row
in the acceptance table carries the command that produces its number.
A metric with no named instrument is a wish, not a metric. A metric must
A metric with no named instrument is a wish, not a metric.
**The instrument must exist, and the target must come out of it.**
Naming a command is not the same as running one. A target computed by
hand and merely *labelled* with a command is the same defect the rule
was written to stop, one level down. Where the instrument is built later
in the pass, the target is marked `provisional:` until the instrument
emits it, and the spec is amended to whatever the instrument returns.
*(v1.1, from CB-WP-0002: `specs/CostAccounting.md` AC-1 named
`cb-cost --pin fc76445` before that tool existed, and set the target to
a hand-computed $92.87. When the tool was built it returned $93.32 —
the hand computation carried a dedup bug the tool's own positive control
caught. The metric satisfied v1.0's rule completely and was still
wrong.)*
**A number inherited from earlier work is re-derived before it is used
as a target, or it is cited as unverified.** Quoting is not measuring.
*(v1.1, from CB-WP-0002: the workplan opened with $248.46, inherited
from a prior pass. Re-derivation put it at $92.21 — the quoted figure
double-counted transcript lines and priced a three-model session at one
model's rate. Neither error was of the harness-does-nothing class; both
sums ran over real data, and a positive control would have passed them.)* A metric must
also be checked against the contracts in the *same spec*: if a contract
makes a target unreachable, one of the two is wrong and the conflict is
resolved when it is noticed, not at the acceptance run. Re-check the
@ -164,6 +193,19 @@ Concretely, a measurement harness must, on every run:
**A number from a run that cannot prove it did the work is void** and
must not reach an evidence file.
**Every tool that reports a number exposes `--self-test`**, and that
self-test runs before the number is produced (`make cost` depends on
`make cost-test`). The assertion must name a failure it detects, not
merely exercise the happy path.
*(v1.0+, from CB-WP-0002: `cb-cost`'s dedup assertion fired on its first
run against real data and aborted, catching a rule that was verified on
206/206 groups of the main transcript and false in the 8-response
subagent tree. The generalization that failed — a property confirmed on
the largest sample assumed to hold on the smallest — is not one review
catches, because both the survey and the adversarial reviewer checked
the same large sample.)*
*(v1.0, from CB-WP-0001: both serious errors in the first pass were of
exactly this shape. A JS harness reported 8.4s for 100k moves while
every move was being rejected, and a Rust benchmark reported 9.3M
@ -286,6 +328,11 @@ A capability has completed the loop when all of the following are committed:
- [ ] every unmet metric reported as unmet, with attribution and the
options for resolving it — a missed target is an output of the
loop, not a reason to move the target quietly
- [ ] **cost recorded**: `make cost` run for the pass, its composition
(not only its total) in the evidence file, and the per-task figures
pushed to the hub. M-D2-CST is no longer allowed to be
`uncomputable` — the instrument exists
([CostAccounting.md](CostAccounting.md))
- [ ] retrospective note (may be one paragraph appended to the evidence
file): what the loop itself should change
```

View file

@ -35,8 +35,8 @@ and add capability-specific rows only when these don't cover the claim.
| M-D2-LOC | D2 | source LOC excluding tests (tokei) | lines | adopted:tokei |
| M-D2-DEP | D2 | transitive dependency count (cargo tree) | crates | adopted:cargo-deny practice |
| M-D2-BLD | D2 | clean build / incremental build time | s | adopted:cargo timing |
| M-D2-TOK | D2 | tokens consumed per completed workplan task | tokens | novel — agentic-efficiency core metric; recorded per task in evidence |
| M-D2-CST | D2 | **cost** per completed workplan task: tokens × the executing model's pricepoint | USD | adapted:anthropic-pricing — tokens alone mislead once models with different prices mix; see §1a |
| M-D2-TOK | D2 | tokens consumed per completed workplan task | tokens | novel — **demoted 2026-07-31**: an input to the cost model, not comparable across models or cache states |
| M-D2-CST | D2 | **cost** per completed workplan task, attributed per CA-08 | USD | adapted:anthropic-pricing — instrument: `make cost`; normative spec [CostAccounting.md](CostAccounting.md) |
| M-D3-THR | D3 | events applied per second, headless replay | events/s | adapted:criterion (throughput mode) |
| M-D3-LAT | D3 | p99 command→state-applied latency | µs | adopted:criterion |
| M-D3-MEM | D3 | peak resident memory during benchmark scenario | MB | adopted:/usr/bin/time -v |
@ -46,6 +46,17 @@ and add capability-specific rows only when these don't cover the claim.
### 1a. Token cost accounting (M-D2-CST)
> **Superseded 2026-07-31 by [CostAccounting.md](CostAccounting.md)**, which
> is normative for the cost model, attribution, and acceptance metrics.
> This section is retained for the price-sheet location and the
> quality-gate rule; where the two disagree, CostAccounting.md wins.
>
> What changed and why: the definition below named no instrument and was
> never computed, so CB-WP-0001 recorded M-D2-CST as *uncomputable* while
> the data sat in the session transcripts. Three of its rules were also
> wrong in ways that cost real money to discover — see the corrections
> inline below.
Token counts are only comparable at a single pricepoint. Since work moves
between models (Fable for demanding passes, Sonnet/Opus for routine ones),
every task's token record carries the **model** it ran on, and cost is
@ -81,13 +92,17 @@ write_1h = 2.0
Rules:
- `cost = (in_tokens × input + out_tokens × output) / 1e6`, using the sheet
in force at the time the work ran; cache reads/writes, when known, use the
multipliers. If cache split is unknown, count all input at full price and
note it — cost is then an upper bound.
- The state-hub task close (`update_task_status`) already records tokens and
`model`; the evidence file's task log adds the computed USD figure so
cross-model comparisons are honest.
- ~~`cost = (in_tokens × input + out_tokens × output) / 1e6` … If cache
split is unknown, count all input at full price and note it — cost is then
an upper bound.~~ **Corrected:** the cache split is never unknown; it is
in every transcript. Treating it as unknown would have priced 80.5M cache
reads at 10× their rate. See CostAccounting.md §1.2 (CA-03, CA-04) — cache
writes bill at two different TTL rates and must not be aggregated.
- ~~The state-hub task close (`update_task_status`) already records tokens
and `model`.~~ **Corrected:** the hub schema has no cache fields and
cannot represent 88% of spend, and the figures it recorded for CB-WP-0001
were estimates in error by ~100%. The hub is a **sink** for numbers
computed by `make cost`, never a source. See CostAccounting.md §6.
- **Cheaper is only better at equal quality**: M-D2-CST verdicts are valid
only alongside passing scenarios/metrics from the same run — a cheap
failed pass scores nothing.

Binary file not shown.

470
tools/cb-cost.py Normal file
View file

@ -0,0 +1,470 @@
#!/usr/bin/env python3
"""M-D2-CST: USD cost of agentic work, attributed to workplan tasks.
Normative spec: specs/CostAccounting.md. Decision: decisions/ADR-0003.
Reads Claude Code session transcripts, deduplicates by requestId, prices
each response at its own model's rate and its own cache TTL, and attributes
it to the task named by the next commit at or after it within the same
session.
Positive control (InnerLoop v1.0 §Step 5, CostAccounting CA-02/CA-14):
this tool asserts it did the work it reports. `--self-test` exercises four
assertions against fixtures with known answers; every reporting run checks
the dedup invariant and reconciles attributed + unattributed + open +
unpriced against the raw total, aborting rather than printing a number that
does not add up.
Usage:
python3 tools/cb-cost.py # whole repo, no pin
python3 tools/cb-cost.py --pin fc76445 # pinned at a commit
python3 tools/cb-cost.py --by-task # per-task attribution
python3 tools/cb-cost.py --composition # cost split by component
python3 tools/cb-cost.py --self-test # positive control
python3 tools/cb-cost.py --json
"""
import argparse
import collections
import datetime
import glob
import json
import os
import re
import subprocess
import sys
try:
import tomllib
except ModuleNotFoundError: # pragma: no cover - py<3.11
print("ERROR: needs Python 3.11+ for tomllib", file=sys.stderr)
sys.exit(1)
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
PRICES = os.path.join(REPO, "benchmarks", "baselines", "model-prices.toml")
COMPONENTS = ("input", "output", "cache_read", "write_5m", "write_1h")
TASK_RE = re.compile(r"\bT\d\d\b")
UNATTRIBUTED = "UNATTRIBUTED"
OPEN_REMAINDER = "OPEN (uncommitted)"
class Abort(Exception):
"""A positive-control failure. Never degrades to a printed number."""
# ---------------------------------------------------------------- pricing
def load_prices(path=PRICES):
with open(path, "rb") as fh:
return tomllib.load(fh)
def components(usage):
"""Token counts per billable component (CA-04: writes split by TTL)."""
cc = usage.get("cache_creation") or {}
return {
"input": usage.get("input_tokens", 0),
"output": usage.get("output_tokens", 0),
"cache_read": usage.get("cache_read_input_tokens", 0),
"write_5m": cc.get("ephemeral_5m_input_tokens", 0),
"write_1h": cc.get("ephemeral_1h_input_tokens", 0),
}
def price_of(prices, model, toks):
"""USD for one response. Returns None when the model is unpriced (CA-05)."""
pr = prices.get(model)
if not pr:
return None
cache = prices["cache"]
unit = pr["input"] / 1e6
return (
toks["input"] * unit
+ toks["output"] * pr["output"] / 1e6
+ toks["cache_read"] * unit * cache["read"]
+ toks["write_5m"] * unit * cache["write_5m"]
+ toks["write_1h"] * unit * cache["write_1h"]
)
# ------------------------------------------------------------- transcripts
def transcript_paths(slug):
"""Every transcript for the repo, including the subagent tree (CA-06)."""
base = os.path.expanduser(f"~/.claude/projects/{slug}")
return sorted(glob.glob(f"{base}/*.jsonl")) + sorted(
glob.glob(f"{base}/*/subagents/agent-*.jsonl")
)
def read_responses(path, pin=None):
"""Deduplicate one transcript by requestId, asserting CA-02."""
groups = collections.defaultdict(list)
for line in open(path):
try:
d = json.loads(line)
except json.JSONDecodeError:
continue
if d.get("type") != "assistant":
continue
usage = (d.get("message") or {}).get("usage")
if usage is None:
continue
ts = d.get("timestamp") or ""
if pin and ts > pin:
continue
groups[d.get("requestId")].append(d)
out = []
for rid, rows in groups.items():
# CA-02 positive control. Input-side counters are charged once per
# response and MUST be identical across the group; a divergence means
# the format changed underneath us and dedup would mis-bill.
#
# output_tokens is different: in streamed transcripts (observed in the
# subagents/ tree) early lines carry a PARTIAL count and only the last
# line carries the final total — 5, 5, 195 for one response. Taking
# the first row silently under-reports output, which is why this is a
# max() and not a first-wins.
seen_model = {r["message"].get("model") for r in rows}
if len(seen_model) != 1:
raise Abort(
f"{os.path.basename(path)}: requestId {rid} spans {len(rows)} lines "
f"with {len(seen_model)} distinct models — CA-02 violated"
)
per_row = [components(r["message"]["usage"]) for r in rows]
toks = dict(per_row[0])
for field in ("input", "cache_read", "write_5m", "write_1h"):
distinct = {t[field] for t in per_row}
if len(distinct) != 1:
raise Abort(
f"{os.path.basename(path)}: requestId {rid} has {len(distinct)} "
f"distinct values for {field} across {len(rows)} lines "
f"({sorted(distinct)}) — CA-02 violated; input-side counters "
f"are charged once per response and must not vary"
)
toks["output"] = max(t["output"] for t in per_row)
head = rows[0]
out.append(
{
"request_id": rid,
"model": head["message"].get("model"),
"timestamp": head.get("timestamp") or "",
"session": head.get("sessionId") or os.path.basename(path),
"toks": toks,
"subagent": "/subagents/" in path,
}
)
return out
# ------------------------------------------------------------- attribution
def commit_index(pin=None):
"""(utc_time, subject) for each commit, oldest first (CA-09)."""
fmt = subprocess.run(
["git", "-C", REPO, "log", "--format=%cI|%s", "--reverse"],
capture_output=True,
text=True,
check=True,
).stdout.splitlines()
rows = []
for line in fmt:
iso, _, subject = line.partition("|")
# CA-09: convert, never subtract a fixed offset — this repo has two.
utc = (
datetime.datetime.fromisoformat(iso)
.astimezone(datetime.timezone.utc)
.isoformat()
.replace("+00:00", "Z")
)
rows.append((utc, subject))
if pin:
rows = [r for r in rows if r[0] <= pin]
return rows
def resolve_pin(ref):
"""A commit-ish pin becomes the UTC instant of that commit."""
if not ref:
return None
if ref.endswith("Z"):
return ref
iso = subprocess.run(
["git", "-C", REPO, "log", "-1", "--format=%cI", ref],
capture_output=True,
text=True,
check=True,
).stdout.strip()
return (
datetime.datetime.fromisoformat(iso)
.astimezone(datetime.timezone.utc)
.isoformat()
.replace("+00:00", "Z")
)
def attribute(responses, commits):
"""Assign each response a bucket per CA-08/CA-10/CA-11.
Intervals are ending-at-commit: (prev_commit, this_commit]. Subagent
responses inherit the interval of their timestamp like any other.
"""
label_for = []
prev = ""
for ts, subject in commits:
m = TASK_RE.search(subject)
label_for.append((prev, ts, m.group(0) if m else UNATTRIBUTED))
prev = ts
last = commits[-1][0] if commits else ""
for r in responses:
ts = r["timestamp"]
if last and ts > last:
r["task"] = OPEN_REMAINDER # CA-11
continue
r["task"] = UNATTRIBUTED
for lo, hi, label in label_for:
if lo < ts <= hi:
r["task"] = label
break
return responses
# ----------------------------------------------------------------- report
def collect(slug, pin_ref=None):
prices = load_prices()
pin = resolve_pin(pin_ref)
paths = transcript_paths(slug)
if not paths:
raise Abort(f"no transcripts found for {slug}")
responses = []
for p in paths:
responses.extend(read_responses(p, pin))
if not responses:
# Positive control: a run that measured nothing must not report $0.00
# as though it were an answer.
raise Abort(f"no responses in {len(paths)} transcript(s) — refusing to report")
for r in responses:
r["cost"] = price_of(prices, r["model"], r["toks"])
attribute(responses, commit_index(pin))
by_task = collections.defaultdict(float)
by_component = collections.Counter()
by_component_cost = collections.defaultdict(float)
by_model = collections.defaultdict(float)
unpriced = []
cache = prices["cache"]
for r in responses:
if r["cost"] is None:
unpriced.append(r)
continue
by_task[r["task"]] += r["cost"]
by_model[r["model"]] += r["cost"]
pr = prices[r["model"]]
unit = pr["input"] / 1e6
rates = {
"input": unit,
"output": pr["output"] / 1e6,
"cache_read": unit * cache["read"],
"write_5m": unit * cache["write_5m"],
"write_1h": unit * cache["write_1h"],
}
for k, n in r["toks"].items():
by_component[k] += n
by_component_cost[k] += n * rates[k]
total = sum(by_task.values())
# CA-14: reconciliation is asserted, not assumed.
residual = total - sum(by_component_cost.values())
if abs(residual) > 0.005:
raise Abort(
f"reconciliation failed: task total ${total:,.4f} vs component total "
f"${sum(by_component_cost.values()):,.4f} (residual ${residual:,.4f})"
)
sub = sum(r["cost"] or 0 for r in responses if r["subagent"])
return {
"slug": slug,
"pin": pin,
"responses": len(responses),
"transcripts": len(paths),
"total": total,
"subagent_total": sub,
"main_total": total - sub,
"by_task": dict(by_task),
"by_model": dict(by_model),
"tokens": dict(by_component),
"cost_by_component": dict(by_component_cost),
"unpriced": [
{"model": r["model"], "tokens": r["toks"]} for r in unpriced
],
"reconciled": True,
"residual": residual,
}
def render(rep, by_task=False, composition=False):
print(f"M-D2-CST cost report — {rep['slug']}")
print(f" pin {rep['pin'] or '(none — live file, not reproducible)'}")
print(f" transcripts {rep['transcripts']} responses {rep['responses']}")
print(f" main ${rep['main_total']:>10,.2f}")
print(f" subagent tree ${rep['subagent_total']:>10,.2f}")
print(f" TOTAL ${rep['total']:>10,.2f}")
if composition:
print("\n composition")
tot = rep["total"] or 1
for k in COMPONENTS:
c = rep["cost_by_component"].get(k, 0.0)
print(
f" {k:<12}{rep['tokens'].get(k,0):>14,} tok "
f"${c:>9,.2f} {100*c/tot:>5.1f}%"
)
if by_task:
print("\n by task")
rows = sorted(rep["by_task"].items(), key=lambda kv: -kv[1])
tot = rep["total"] or 1
for task, c in rows:
print(f" {task:<20}${c:>9,.2f} {100*c/tot:>5.1f}%")
# CA-10: the limit is reported with the number, every time.
un = rep["by_task"].get(UNATTRIBUTED, 0.0)
print(
f"\n NOTE: {100*un/tot:.0f}% of cost is UNATTRIBUTED — commits whose "
f"subject carries no T## tag.\n"
f" A per-task table is a view over {100*(1-un/tot):.0f}% of spend."
)
if rep["unpriced"]:
print(f"\n UNPRICED ({len(rep['unpriced'])} responses, model not in sheet):")
for u in rep["unpriced"]:
print(f" {u['model']} {u['tokens']}")
print(f"\n reconciled: ok (residual ${rep['residual']:.6f})")
# -------------------------------------------------------------- self-test
def self_test():
"""AC-5..AC-8. Each asserts a failure mode is actually detected."""
prices = load_prices()
checks = []
def check(name, ok, detail=""):
checks.append((name, ok, detail))
# AC-8: per-TTL cache pricing. 5m must not be priced at the 1h rate.
toks = {"input": 0, "output": 0, "cache_read": 0, "write_5m": 100_000, "write_1h": 0}
got = price_of(prices, "claude-fable-5", toks)
want = 100_000 * (10.0 / 1e6) * 1.25
wrong = 100_000 * (10.0 / 1e6) * 2.0
check("AC-8 5m cache priced at write_5m", abs(got - want) < 1e-9 and got != wrong,
f"${got:.4f} (1h would be ${wrong:.4f})")
# AC-5: dedup invariant is enforced.
import tempfile
with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh:
u1 = {"input_tokens": 1, "output_tokens": 2, "cache_read_input_tokens": 3,
"cache_creation": {"ephemeral_5m_input_tokens": 0,
"ephemeral_1h_input_tokens": 0}}
u2 = dict(u1)
# Diverging cache_read is a real violation (input-side, charged once).
u2["cache_read_input_tokens"] = 999
for u in (u1, u2):
fh.write(json.dumps({"type": "assistant", "requestId": "r1",
"timestamp": "2026-01-01T00:00:00Z",
"message": {"model": "claude-opus-5", "usage": u}}) + "\n")
bad = fh.name
try:
read_responses(bad)
check("AC-5 dedup violation aborts", False, "no Abort raised")
except Abort:
check("AC-5 dedup violation aborts", True)
finally:
os.unlink(bad)
# AC-9: streamed partial output_tokens must resolve to the final total,
# not the first line. Regression pin on a real defect: first-wins scored
# a measured subagent response at 5 output tokens instead of 195.
with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh:
for out_tok in (5, 5, 195):
u = {"input_tokens": 2, "output_tokens": out_tok,
"cache_read_input_tokens": 0,
"cache_creation": {"ephemeral_5m_input_tokens": 19008,
"ephemeral_1h_input_tokens": 0}}
fh.write(json.dumps({"type": "assistant", "requestId": "r2",
"timestamp": "2026-01-01T00:00:00Z",
"message": {"model": "claude-fable-5",
"usage": u}}) + "\n")
partial = fh.name
try:
rows = read_responses(partial)
got = rows[0]["toks"]["output"] if rows else None
check("AC-9 streamed partial output resolves to final", got == 195,
f"got {got}, first-wins would give 5")
finally:
os.unlink(partial)
# AC-6: zero responses must not report $0.00 as an answer.
with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh:
fh.write(json.dumps({"type": "user", "message": {}}) + "\n")
empty = fh.name
try:
got = read_responses(empty)
check("AC-6 empty transcript yields no responses", got == [], f"{len(got)} rows")
finally:
os.unlink(empty)
# AC-7: the subagent tree is discovered by the path globs.
slug = "-home-worsch-clay-borg"
paths = transcript_paths(slug)
subs = [p for p in paths if "/subagents/" in p]
check("AC-7 subagent tree enumerated", len(subs) > 0,
f"{len(subs)} subagent transcript(s) of {len(paths)} total")
print("cb-cost self-test (positive control)")
ok = True
for name, passed, detail in checks:
print(f" [{'ok ' if passed else 'FAIL'}] {name}" + (f"{detail}" if detail else ""))
ok &= passed
return 0 if ok else 1
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--slug", default="-home-worsch-clay-borg")
ap.add_argument("--pin", help="commit-ish or ISO Z instant (CA-07)")
ap.add_argument("--by-task", action="store_true")
ap.add_argument("--composition", action="store_true")
ap.add_argument("--self-test", action="store_true")
ap.add_argument("--json", action="store_true")
args = ap.parse_args()
if args.self_test:
return self_test()
try:
rep = collect(args.slug, args.pin)
except Abort as e:
print(f"ABORT — {e}", file=sys.stderr)
return 1
if args.json:
print(json.dumps(rep, indent=2))
else:
render(rep, by_task=args.by_task, composition=args.composition)
return 0
if __name__ == "__main__":
sys.exit(main())

View file

@ -1,7 +1,7 @@
---
id: CB-WP-0002
title: "Make agentic cost measurable, so D2 claims are falsifiable"
status: proposed
status: done
state_hub_workstream_id: "b7c22f69-fbe9-48df-9619-007db79ae338"
---
@ -18,19 +18,41 @@ transcript (`~/.claude/projects/<slug>/<session>.jsonl`) carries exact
per-message `usage`, including the cache breakdown. Reading it for
CB-WP-0001's session gives:
| Component | Tokens | Cost (Fable 5) |
|---|---|---|
| Output | 585,528 | $29.28 |
| Cache read | 131,863,164 | **$131.86** |
| Cache write (1h) | 4,365,668 | $87.31 |
| Input | 1,090 | $0.01 |
| **Total** | | **$248.46** |
> **Corrected 2026-07-31 by T01.** This section originally reported
> **$248.46** from 131,863,164 cache-read tokens priced at Fable 5. That
> figure was wrong by ~2.7×, for two independent reasons found in
> `research/CB-RES-0002-cost-accounting.md` §Correction: one API response
> is written to the transcript as up to six JSONL lines that each repeat
> the *same* `usage` object (657 lines, 346 real responses), and the
> session ran three models, not one. The numbers below are the re-derived
> ones. The workplan's premise is unaffected; its arithmetic was not.
The headline finding from that single reading is the reason this
workplan exists: **53% of the cost is cache reads**, not output. Cost in
an agentic loop is driven by context size × turn count, not by how much
the model writes. No D2 decision made on "tokens per task" would have
surfaced that.
Deduplicated by `requestId` and priced per message at its own model's
rate. **Pinned** to messages at or before `2026-07-31T02:17:59Z` (commit
`fc76445`, the end of CB-WP-0001) — the transcript is a live file that
grows as later sessions append to it, so an unpinned total is not a
repeatable acceptance target:
| Component | Tokens | Cost |
|---|---|---|
| Output | 313,900 | $10.66 |
| Cache read | 80,453,702 | **$59.59** |
| Cache write (1h) | 1,672,854 | $21.95 |
| Input | 676 | $0.00 |
| Main transcript (339 responses: 213 opus-5, 118 fable-5, 14 sonnet-5) | | $92.21 |
| Subagent tree (adversarial review, ran inside the pin) | | $1.11 |
| **TRUE TOTAL** | | **$93.32** |
The headline finding survives the correction and gets sharper:
**88.4% of the cost is cache, against 314k tokens of output — a 256:1
ratio of context re-read to text written.** Cost in an agentic loop is
driven by context size × turn count, not by how much the model writes. No
D2 decision made on "tokens per task" would have surfaced that.
The correction is itself the lesson: this workplan opened with a
mismeasured cost. Neither error was of the harness-does-nothing class the
positive-control rule was written for — both sums ran over real data.
What caught them was re-deriving the number instead of quoting it.
This workplan makes cost a first-class measured dimension so that
future AM-12 equivalents support conclusions instead of decorating an
@ -44,7 +66,7 @@ positive control.
```task
id: CB-WP-0002-T01
status: todo
status: done
priority: high
state_hub_task_id: "2694c2c1-0070-4d8e-b4fc-196b582b36d5"
```
@ -67,7 +89,7 @@ accuracy and the hub to lead on durability.
```task
id: CB-WP-0002-T02
status: todo
status: done
priority: high
state_hub_task_id: "eae248ab-f29f-4f11-9d20-e8145b0d822d"
```
@ -93,7 +115,7 @@ Gate: no collector code before this ADR is committed.
```task
id: CB-WP-0002-T03
status: todo
status: done
priority: high
state_hub_task_id: "00d42ed2-4391-4580-aae2-06e3e151c69b"
```
@ -116,7 +138,7 @@ replace AM-12's definition with one that is computable.
```task
id: CB-WP-0002-T04
status: todo
status: done
priority: medium
state_hub_task_id: "9eb8329b-5f41-477b-8cf3-2cda5ba8dbe8"
```
@ -140,16 +162,22 @@ each message at its own model's rate.
```task
id: CB-WP-0002-T05
status: todo
status: done
priority: medium
state_hub_task_id: "bea4cc0a-e4d0-4077-9dc7-df7726a48f86"
```
Run the collector over the CB-WP-0001 session and commit
`evidence/CB-EV-0002-cost-accounting.md`. Reconciliation against the
totals in this workplan's Purpose section is the acceptance test: the
tool must reproduce $248.46 (Fable 5) from the same transcript, or
explain the difference.
`evidence/CB-EV-0002-cost-accounting.md`. The acceptance test is
`specs/CostAccounting.md` AC-1: reproduce **$93.32 pinned at `fc76445`,
as its two components** ($92.21 main transcript + $1.11 subagent tree),
plus AC-2's zero reconciliation residual.
*(Originally written as "must reproduce $248.46". That figure was wrong
by 2.7×, and the first corrected target — $92.21 — was itself reachable
only by a collector carrying the subagent blind spot the survey had just
documented. Both errors are recorded rather than quietly overwritten;
the sequence is the point.)*
Then use it to answer at least one question that could not be answered
before, and record the answer. Candidates: which of T01T09 cost most
@ -164,7 +192,7 @@ cleared the bar that CB-WP-0001's AM-12 failed to clear.
```task
id: CB-WP-0002-T06
status: todo
status: done
priority: low
state_hub_task_id: "1412263b-70c1-43e4-957d-1ad6c3203ca9"
```
@ -179,7 +207,7 @@ the one command surface.
```task
id: CB-WP-0002-T07
status: todo
status: done
priority: low
state_hub_task_id: "ebe58d91-be5f-4d5b-ba40-b03275b4eefc"
```