tegwick
2f086d26b6
- InnerLoop definition-of-done gains a cost row: M-D2-CST may no longer be recorded uncomputable, and composition must be reported, not only a total. - InnerLoop Step 5 gains the --self-test contract: every tool that reports a number exposes one, and it runs before the number does. Rationale attached, because the case that motivated it is the one review cannot catch — survey and reviewer both verified the same large sample and both missed the small one. - make cost / cost-test / cost-pin on the one command surface; cost-test in `make all` and in CI. - Hub now holds the measured figure for CB-WP-0001: 80.6M in / 323.6k out against the 401,100 it previously estimated, low by ~200x. The event states plainly that the hub schema cannot represent the 88% of cost that is cache, and names `make cost-pin` as the authority. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
221 lines
8.7 KiB
Markdown
221 lines
8.7 KiB
Markdown
---
|
||
id: CB-WP-0002
|
||
title: "Make agentic cost measurable, so D2 claims are falsifiable"
|
||
status: proposed
|
||
state_hub_workstream_id: "b7c22f69-fbe9-48df-9619-007db79ae338"
|
||
---
|
||
|
||
# Purpose
|
||
|
||
CB-WP-0001 specified a cost metric (AM-12, `M-D2-CST`) in full — price
|
||
sheet, formula, staleness rule — and then could not report a number,
|
||
because nothing was ever instrumented. It was recorded as
|
||
"uncomputable".
|
||
|
||
That was wrong, and the retrospective for this workplan should say so
|
||
plainly: **the data existed the whole time.** Every Claude Code session
|
||
transcript (`~/.claude/projects/<slug>/<session>.jsonl`) carries exact
|
||
per-message `usage`, including the cache breakdown. Reading it for
|
||
CB-WP-0001's session gives:
|
||
|
||
> **Corrected 2026-07-31 by T01.** This section originally reported
|
||
> **$248.46** from 131,863,164 cache-read tokens priced at Fable 5. That
|
||
> figure was wrong by ~2.7×, for two independent reasons found in
|
||
> `research/CB-RES-0002-cost-accounting.md` §Correction: one API response
|
||
> is written to the transcript as up to six JSONL lines that each repeat
|
||
> the *same* `usage` object (657 lines, 346 real responses), and the
|
||
> session ran three models, not one. The numbers below are the re-derived
|
||
> ones. The workplan's premise is unaffected; its arithmetic was not.
|
||
|
||
Deduplicated by `requestId` and priced per message at its own model's
|
||
rate. **Pinned** to messages at or before `2026-07-31T02:17:59Z` (commit
|
||
`fc76445`, the end of CB-WP-0001) — the transcript is a live file that
|
||
grows as later sessions append to it, so an unpinned total is not a
|
||
repeatable acceptance target:
|
||
|
||
| Component | Tokens | Cost |
|
||
|---|---|---|
|
||
| Output | 313,900 | $10.66 |
|
||
| Cache read | 80,453,702 | **$59.59** |
|
||
| Cache write (1h) | 1,672,854 | $21.95 |
|
||
| Input | 676 | $0.00 |
|
||
| Main transcript (339 responses: 213 opus-5, 118 fable-5, 14 sonnet-5) | | $92.21 |
|
||
| Subagent tree (adversarial review, ran inside the pin) | | $1.11 |
|
||
| **TRUE TOTAL** | | **$93.32** |
|
||
|
||
The headline finding survives the correction and gets sharper:
|
||
**88.4% of the cost is cache, against 314k tokens of output — a 256:1
|
||
ratio of context re-read to text written.** Cost in an agentic loop is
|
||
driven by context size × turn count, not by how much the model writes. No
|
||
D2 decision made on "tokens per task" would have surfaced that.
|
||
|
||
The correction is itself the lesson: this workplan opened with a
|
||
mismeasured cost. Neither error was of the harness-does-nothing class the
|
||
positive-control rule was written for — both sums ran over real data.
|
||
What caught them was re-deriving the number instead of quoting it.
|
||
|
||
This workplan makes cost a first-class measured dimension so that
|
||
future AM-12 equivalents support conclusions instead of decorating an
|
||
evidence file. It follows InnerLoop v1.0, including the rules that pass
|
||
earned: every metric names its instrument, and every harness carries a
|
||
positive control.
|
||
|
||
## Phase A — Instrument
|
||
|
||
## Task: Survey the available cost instruments
|
||
|
||
```task
|
||
id: CB-WP-0002-T01
|
||
status: done
|
||
priority: high
|
||
state_hub_task_id: "2694c2c1-0070-4d8e-b4fc-196b582b36d5"
|
||
```
|
||
|
||
Produce `research/CB-RES-0002-cost-accounting.md` per the InnerLoop
|
||
survey template. Candidates at minimum: session transcript JSONL
|
||
(`usage` per assistant message), the Custodian State Hub token-event
|
||
API (`record_token_event`, `update_task_status` token tiers,
|
||
`get_token_summary`), the Claude Code status bar, and the Anthropic
|
||
usage/billing API. Per candidate: what it reports, granularity,
|
||
accuracy, whether it survives compaction, and whether it can attribute
|
||
cost to a unit of work.
|
||
|
||
Note explicitly which are **exact** and which are **estimates** — the
|
||
prior failure was tolerating an estimate-shaped metric. Name the
|
||
benchmark-to-beat per dimension; expect the transcript to lead on
|
||
accuracy and the hub to lead on durability.
|
||
|
||
## Task: Decide the instrument and the attribution model (ADR)
|
||
|
||
```task
|
||
id: CB-WP-0002-T02
|
||
status: done
|
||
priority: high
|
||
state_hub_task_id: "eae248ab-f29f-4f11-9d20-e8145b0d822d"
|
||
```
|
||
|
||
Adversarial review of T01 first (InnerLoop §Step 2), committed as
|
||
`history/YYMMDD-cost-accounting-{challenge,response}.md`.
|
||
|
||
Then `decisions/ADR-0003-cost-accounting.md`. The genuinely hard part is
|
||
**attribution**: a transcript is a flat message stream, and a "task" is
|
||
a workplan concept. Options to weigh, not assume:
|
||
|
||
- git commit timestamps as task boundaries (the loop commits per
|
||
iteration, so boundaries already exist and are durable)
|
||
- explicit session markers emitted at task start/end
|
||
- hub task status transitions as the time index
|
||
|
||
State the expected advantage per dimension and the known failure modes
|
||
of the chosen model — in particular, what happens to attribution across
|
||
`/compact`, across resumed sessions, and for work spanning a boundary.
|
||
Gate: no collector code before this ADR is committed.
|
||
|
||
## Task: Specify the cost metrics with named instruments
|
||
|
||
```task
|
||
id: CB-WP-0002-T03
|
||
status: done
|
||
priority: high
|
||
state_hub_task_id: "00d42ed2-4391-4580-aae2-06e3e151c69b"
|
||
```
|
||
|
||
Write `specs/CostAccounting.md`: the cost model (input, output, cache
|
||
read, cache write at 5m and 1h, per the price sheet), the attribution
|
||
contract, and the acceptance metrics — each naming the **command that
|
||
produces its number**, per InnerLoop v1.0 §Step 4.
|
||
|
||
Must include a metric for the finding that motivated this workplan:
|
||
**cost composition** (what fraction is cache read vs write vs output),
|
||
not only a total. A single total would have hidden the 53%.
|
||
|
||
Revise `specs/MetricsAndScenarios.md` §1a to point at this spec, and
|
||
replace AM-12's definition with one that is computable.
|
||
|
||
## Phase B — Build and prove
|
||
|
||
## Task: Implement the cost collector
|
||
|
||
```task
|
||
id: CB-WP-0002-T04
|
||
status: done
|
||
priority: medium
|
||
state_hub_task_id: "9eb8329b-5f41-477b-8cf3-2cda5ba8dbe8"
|
||
```
|
||
|
||
Implement the tool chosen in T02 (expected: `tools/cb-cost`). It reads
|
||
transcripts, applies the price sheet at
|
||
`benchmarks/baselines/model-prices.toml`, attributes cost per the T02
|
||
model, and emits both a per-task table and a composition breakdown in
|
||
the evidence-row format from T03.
|
||
|
||
**Positive control is mandatory** (InnerLoop v1.0 §Step 5): the tool
|
||
asserts that attributed tokens sum to the transcript total, and refuses
|
||
to emit numbers when they do not reconcile. An unattributed remainder is
|
||
reported as its own line, never silently dropped — the failure this
|
||
whole workplan exists to prevent was a number that looked fine.
|
||
|
||
Handle multi-model sessions: a session that switches models must price
|
||
each message at its own model's rate.
|
||
|
||
## Task: Validate against CB-WP-0001 and answer a real question
|
||
|
||
```task
|
||
id: CB-WP-0002-T05
|
||
status: done
|
||
priority: medium
|
||
state_hub_task_id: "bea4cc0a-e4d0-4077-9dc7-df7726a48f86"
|
||
```
|
||
|
||
Run the collector over the CB-WP-0001 session and commit
|
||
`evidence/CB-EV-0002-cost-accounting.md`. The acceptance test is
|
||
`specs/CostAccounting.md` AC-1: reproduce **$93.32 pinned at `fc76445`,
|
||
as its two components** ($92.21 main transcript + $1.11 subagent tree),
|
||
plus AC-2's zero reconciliation residual.
|
||
|
||
*(Originally written as "must reproduce $248.46". That figure was wrong
|
||
by 2.7×, and the first corrected target — $92.21 — was itself reachable
|
||
only by a collector carrying the subagent blind spot the survey had just
|
||
documented. Both errors are recorded rather than quietly overwritten;
|
||
the sequence is the point.)*
|
||
|
||
Then use it to answer at least one question that could not be answered
|
||
before, and record the answer. Candidates: which of T01–T09 cost most
|
||
and whether that matched its value; what a `/compact` costs; whether the
|
||
adversarial review paid for itself; how much the six T08 code iterations
|
||
cost relative to the research phase.
|
||
|
||
A tool that produces numbers nobody draws a conclusion from has not
|
||
cleared the bar that CB-WP-0001's AM-12 failed to clear.
|
||
|
||
## Task: Wire cost into the loop
|
||
|
||
```task
|
||
id: CB-WP-0002-T06
|
||
status: done
|
||
priority: low
|
||
state_hub_task_id: "1412263b-70c1-43e4-957d-1ad6c3203ca9"
|
||
```
|
||
|
||
Make cost collection automatic rather than remembered: add the
|
||
collector to the evidence checklist in `specs/InnerLoop.md`, emit hub
|
||
token events so the Token Cost dashboard reflects real numbers rather
|
||
than the 1000/500 heuristic fallback, and add a `make cost` target to
|
||
the one command surface.
|
||
|
||
## Task: Retrospective
|
||
|
||
```task
|
||
id: CB-WP-0002-T07
|
||
status: todo
|
||
priority: low
|
||
state_hub_task_id: "ebe58d91-be5f-4d5b-ba40-b03275b4eefc"
|
||
```
|
||
|
||
Revise `specs/InnerLoop.md` and `specs/MetricsAndScenarios.md` from what
|
||
this pass teaches about metric design. The specific question to answer:
|
||
CB-WP-0001 produced a fully specified metric that could not be computed,
|
||
and v1.0's "every metric names its instrument" rule was written to stop
|
||
that. Did it? If a metric can still be written without a working
|
||
instrument, the rule needs teeth — for example, requiring that the
|
||
instrument be demonstrated on real data before the metric is accepted.
|