clay-borg/workplans/CB-WP-0002-cost-accounting.md

id	title	status
CB-WP-0002	Make agentic cost measurable, so D2 claims are falsifiable	proposed

Purpose

CB-WP-0001 specified a cost metric (AM-12, M-D2-CST) in full — price sheet, formula, staleness rule — and then could not report a number, because nothing was ever instrumented. It was recorded as "uncomputable".

That was wrong, and the retrospective for this workplan should say so plainly: the data existed the whole time. Every Claude Code session transcript (~/.claude/projects/<slug>/<session>.jsonl) carries exact per-message usage, including the cache breakdown. Reading it for CB-WP-0001's session gives:

Component	Tokens	Cost (Fable 5)
Output	585,528	$29.28
Cache read	131,863,164	$131.86
Cache write (1h)	4,365,668	$87.31
Input	1,090	$0.01
Total		$248.46

The headline finding from that single reading is the reason this workplan exists: 53% of the cost is cache reads, not output. Cost in an agentic loop is driven by context size × turn count, not by how much the model writes. No D2 decision made on "tokens per task" would have surfaced that.

This workplan makes cost a first-class measured dimension so that future AM-12 equivalents support conclusions instead of decorating an evidence file. It follows InnerLoop v1.0, including the rules that pass earned: every metric names its instrument, and every harness carries a positive control.

Phase A — Instrument

Task: Survey the available cost instruments

id: CB-WP-0002-T01
status: todo
priority: high

Produce research/CB-RES-0002-cost-accounting.md per the InnerLoop survey template. Candidates at minimum: session transcript JSONL (usage per assistant message), the Custodian State Hub token-event API (record_token_event, update_task_status token tiers, get_token_summary), the Claude Code status bar, and the Anthropic usage/billing API. Per candidate: what it reports, granularity, accuracy, whether it survives compaction, and whether it can attribute cost to a unit of work.

Note explicitly which are exact and which are estimates — the prior failure was tolerating an estimate-shaped metric. Name the benchmark-to-beat per dimension; expect the transcript to lead on accuracy and the hub to lead on durability.

Task: Decide the instrument and the attribution model (ADR)

id: CB-WP-0002-T02
status: todo
priority: high

Adversarial review of T01 first (InnerLoop §Step 2), committed as history/YYMMDD-cost-accounting-{challenge,response}.md.

Then decisions/ADR-0003-cost-accounting.md. The genuinely hard part is attribution: a transcript is a flat message stream, and a "task" is a workplan concept. Options to weigh, not assume:

• git commit timestamps as task boundaries (the loop commits per iteration, so boundaries already exist and are durable)
• explicit session markers emitted at task start/end
• hub task status transitions as the time index

State the expected advantage per dimension and the known failure modes of the chosen model — in particular, what happens to attribution across /compact, across resumed sessions, and for work spanning a boundary. Gate: no collector code before this ADR is committed.

Task: Specify the cost metrics with named instruments

id: CB-WP-0002-T03
status: todo
priority: high

Write specs/CostAccounting.md: the cost model (input, output, cache read, cache write at 5m and 1h, per the price sheet), the attribution contract, and the acceptance metrics — each naming the command that produces its number, per InnerLoop v1.0 §Step 4.

Must include a metric for the finding that motivated this workplan: cost composition (what fraction is cache read vs write vs output), not only a total. A single total would have hidden the 53%.

Revise specs/MetricsAndScenarios.md §1a to point at this spec, and replace AM-12's definition with one that is computable.

Phase B — Build and prove

Task: Implement the cost collector

id: CB-WP-0002-T04
status: todo
priority: medium

Implement the tool chosen in T02 (expected: tools/cb-cost). It reads transcripts, applies the price sheet at benchmarks/baselines/model-prices.toml, attributes cost per the T02 model, and emits both a per-task table and a composition breakdown in the evidence-row format from T03.

Positive control is mandatory (InnerLoop v1.0 §Step 5): the tool asserts that attributed tokens sum to the transcript total, and refuses to emit numbers when they do not reconcile. An unattributed remainder is reported as its own line, never silently dropped — the failure this whole workplan exists to prevent was a number that looked fine.

Handle multi-model sessions: a session that switches models must price each message at its own model's rate.

Task: Validate against CB-WP-0001 and answer a real question

id: CB-WP-0002-T05
status: todo
priority: medium

Run the collector over the CB-WP-0001 session and commit evidence/CB-EV-0002-cost-accounting.md. Reconciliation against the totals in this workplan's Purpose section is the acceptance test: the tool must reproduce $248.46 (Fable 5) from the same transcript, or explain the difference.

Then use it to answer at least one question that could not be answered before, and record the answer. Candidates: which of T01–T09 cost most and whether that matched its value; what a /compact costs; whether the adversarial review paid for itself; how much the six T08 code iterations cost relative to the research phase.

A tool that produces numbers nobody draws a conclusion from has not cleared the bar that CB-WP-0001's AM-12 failed to clear.

Task: Wire cost into the loop

id: CB-WP-0002-T06
status: todo
priority: low

Make cost collection automatic rather than remembered: add the collector to the evidence checklist in specs/InnerLoop.md, emit hub token events so the Token Cost dashboard reflects real numbers rather than the 1000/500 heuristic fallback, and add a make cost target to the one command surface.

Task: Retrospective

id: CB-WP-0002-T07
status: todo
priority: low

Revise specs/InnerLoop.md and specs/MetricsAndScenarios.md from what this pass teaches about metric design. The specific question to answer: CB-WP-0001 produced a fully specified metric that could not be computed, and v1.0's "every metric names its instrument" rule was written to stop that. Did it? If a metric can still be written without a working instrument, the rule needs teeth — for example, requiring that the instrument be demonstrated on real data before the metric is accepted.