clay-borg/workplans/CB-WP-0002-cost-accounting.md

id	title	status	state_hub_workstream_id
CB-WP-0002	Make agentic cost measurable, so D2 claims are falsifiable	done	b7c22f69-fbe9-48df-9619-007db79ae338

Purpose

CB-WP-0001 specified a cost metric (AM-12, M-D2-CST) in full — price sheet, formula, staleness rule — and then could not report a number, because nothing was ever instrumented. It was recorded as "uncomputable".

That was wrong, and the retrospective for this workplan should say so plainly: the data existed the whole time. Every Claude Code session transcript (~/.claude/projects/<slug>/<session>.jsonl) carries exact per-message usage, including the cache breakdown. Reading it for CB-WP-0001's session gives:

Corrected 2026-07-31 by T01. This section originally reported $248.46 from 131,863,164 cache-read tokens priced at Fable 5. That figure was wrong by ~2.7×, for two independent reasons found in research/CB-RES-0002-cost-accounting.md §Correction: one API response is written to the transcript as up to six JSONL lines that each repeat the same usage object (657 lines, 346 real responses), and the session ran three models, not one. The numbers below are the re-derived ones. The workplan's premise is unaffected; its arithmetic was not.

Deduplicated by requestId and priced per message at its own model's rate. Pinned to messages at or before 2026-07-31T02:17:59Z (commit fc76445, the end of CB-WP-0001) — the transcript is a live file that grows as later sessions append to it, so an unpinned total is not a repeatable acceptance target:

Component	Tokens	Cost
Output	313,900	$10.66
Cache read	80,453,702	$59.59
Cache write (1h)	1,672,854	$21.95
Input	676	$0.00
Main transcript (339 responses: 213 opus-5, 118 fable-5, 14 sonnet-5)		$92.21
Subagent tree (adversarial review, ran inside the pin)		$1.11
TRUE TOTAL		$93.32

The headline finding survives the correction and gets sharper: 88.4% of the cost is cache, against 314k tokens of output — a 256:1 ratio of context re-read to text written. Cost in an agentic loop is driven by context size × turn count, not by how much the model writes. No D2 decision made on "tokens per task" would have surfaced that.

The correction is itself the lesson: this workplan opened with a mismeasured cost. Neither error was of the harness-does-nothing class the positive-control rule was written for — both sums ran over real data. What caught them was re-deriving the number instead of quoting it.

This workplan makes cost a first-class measured dimension so that future AM-12 equivalents support conclusions instead of decorating an evidence file. It follows InnerLoop v1.0, including the rules that pass earned: every metric names its instrument, and every harness carries a positive control.

Phase A — Instrument

Task: Survey the available cost instruments

id: CB-WP-0002-T01
status: done
priority: high
state_hub_task_id: "2694c2c1-0070-4d8e-b4fc-196b582b36d5"

Produce research/CB-RES-0002-cost-accounting.md per the InnerLoop survey template. Candidates at minimum: session transcript JSONL (usage per assistant message), the Custodian State Hub token-event API (record_token_event, update_task_status token tiers, get_token_summary), the Claude Code status bar, and the Anthropic usage/billing API. Per candidate: what it reports, granularity, accuracy, whether it survives compaction, and whether it can attribute cost to a unit of work.

Note explicitly which are exact and which are estimates — the prior failure was tolerating an estimate-shaped metric. Name the benchmark-to-beat per dimension; expect the transcript to lead on accuracy and the hub to lead on durability.

Task: Decide the instrument and the attribution model (ADR)

id: CB-WP-0002-T02
status: done
priority: high
state_hub_task_id: "eae248ab-f29f-4f11-9d20-e8145b0d822d"

Adversarial review of T01 first (InnerLoop §Step 2), committed as history/YYMMDD-cost-accounting-{challenge,response}.md.

Then decisions/ADR-0003-cost-accounting.md. The genuinely hard part is attribution: a transcript is a flat message stream, and a "task" is a workplan concept. Options to weigh, not assume:

• git commit timestamps as task boundaries (the loop commits per iteration, so boundaries already exist and are durable)
• explicit session markers emitted at task start/end
• hub task status transitions as the time index

State the expected advantage per dimension and the known failure modes of the chosen model — in particular, what happens to attribution across /compact, across resumed sessions, and for work spanning a boundary. Gate: no collector code before this ADR is committed.

Task: Specify the cost metrics with named instruments

id: CB-WP-0002-T03
status: done
priority: high
state_hub_task_id: "00d42ed2-4391-4580-aae2-06e3e151c69b"

Write specs/CostAccounting.md: the cost model (input, output, cache read, cache write at 5m and 1h, per the price sheet), the attribution contract, and the acceptance metrics — each naming the command that produces its number, per InnerLoop v1.0 §Step 4.

Must include a metric for the finding that motivated this workplan: cost composition (what fraction is cache read vs write vs output), not only a total. A single total would have hidden the 53%.

Revise specs/MetricsAndScenarios.md §1a to point at this spec, and replace AM-12's definition with one that is computable.

Phase B — Build and prove

Task: Implement the cost collector

id: CB-WP-0002-T04
status: done
priority: medium
state_hub_task_id: "9eb8329b-5f41-477b-8cf3-2cda5ba8dbe8"

Implement the tool chosen in T02 (expected: tools/cb-cost). It reads transcripts, applies the price sheet at benchmarks/baselines/model-prices.toml, attributes cost per the T02 model, and emits both a per-task table and a composition breakdown in the evidence-row format from T03.

Positive control is mandatory (InnerLoop v1.0 §Step 5): the tool asserts that attributed tokens sum to the transcript total, and refuses to emit numbers when they do not reconcile. An unattributed remainder is reported as its own line, never silently dropped — the failure this whole workplan exists to prevent was a number that looked fine.

Handle multi-model sessions: a session that switches models must price each message at its own model's rate.

Task: Validate against CB-WP-0001 and answer a real question

id: CB-WP-0002-T05
status: done
priority: medium
state_hub_task_id: "bea4cc0a-e4d0-4077-9dc7-df7726a48f86"

Run the collector over the CB-WP-0001 session and commit evidence/CB-EV-0002-cost-accounting.md. The acceptance test is specs/CostAccounting.md AC-1: reproduce $93.32 pinned at fc76445, as its two components ($92.21 main transcript + $1.11 subagent tree), plus AC-2's zero reconciliation residual.

(Originally written as "must reproduce $248.46". That figure was wrong by 2.7×, and the first corrected target — $92.21 — was itself reachable only by a collector carrying the subagent blind spot the survey had just documented. Both errors are recorded rather than quietly overwritten; the sequence is the point.)

Then use it to answer at least one question that could not be answered before, and record the answer. Candidates: which of T01–T09 cost most and whether that matched its value; what a /compact costs; whether the adversarial review paid for itself; how much the six T08 code iterations cost relative to the research phase.

A tool that produces numbers nobody draws a conclusion from has not cleared the bar that CB-WP-0001's AM-12 failed to clear.

Task: Wire cost into the loop

id: CB-WP-0002-T06
status: done
priority: low
state_hub_task_id: "1412263b-70c1-43e4-957d-1ad6c3203ca9"

Make cost collection automatic rather than remembered: add the collector to the evidence checklist in specs/InnerLoop.md, emit hub token events so the Token Cost dashboard reflects real numbers rather than the 1000/500 heuristic fallback, and add a make cost target to the one command surface.

Task: Retrospective

id: CB-WP-0002-T07
status: done
priority: low
state_hub_task_id: "ebe58d91-be5f-4d5b-ba40-b03275b4eefc"

Revise specs/InnerLoop.md and specs/MetricsAndScenarios.md from what this pass teaches about metric design. The specific question to answer: CB-WP-0001 produced a fully specified metric that could not be computed, and v1.0's "every metric names its instrument" rule was written to stop that. Did it? If a metric can still be written without a working instrument, the rule needs teeth — for example, requiring that the instrument be demonstrated on real data before the metric is accepted.