clay-borg/workplans/CB-WP-0002-cost-accounting.md

---
id: CB-WP-0002
title: "Make agentic cost measurable, so D2 claims are falsifiable"
status: proposed
state_hub_workstream_id: "b7c22f69-fbe9-48df-9619-007db79ae338"
---

# Purpose

CB-WP-0001 specified a cost metric (AM-12, `M-D2-CST`) in full — price
sheet, formula, staleness rule — and then could not report a number,
because nothing was ever instrumented. It was recorded as
"uncomputable".

That was wrong, and the retrospective for this workplan should say so
plainly: **the data existed the whole time.** Every Claude Code session
transcript (`~/.claude/projects/<slug>/<session>.jsonl`) carries exact
per-message `usage`, including the cache breakdown. Reading it for
CB-WP-0001's session gives:

> **Corrected 2026-07-31 by T01.** This section originally reported
> **$248.46** from 131,863,164 cache-read tokens priced at Fable 5. That
> figure was wrong by ~2.7×, for two independent reasons found in
> `research/CB-RES-0002-cost-accounting.md` §Correction: one API response
> is written to the transcript as up to six JSONL lines that each repeat
> the *same* `usage` object (657 lines, 346 real responses), and the
> session ran three models, not one. The numbers below are the re-derived
> ones. The workplan's premise is unaffected; its arithmetic was not.

Deduplicated by `requestId` and priced per message at its own model's
rate. **Pinned** to messages at or before `2026-07-31T02:17:59Z` (commit
`fc76445`, the end of CB-WP-0001) — the transcript is a live file that
grows as later sessions append to it, so an unpinned total is not a
repeatable acceptance target:

| Component | Tokens | Cost |
|---|---|---|
| Output | 313,900 | $10.66 |
| Cache read | 80,453,702 | **$59.59** |
| Cache write (1h) | 1,672,854 | $21.95 |
| Input | 676 | $0.00 |
| **Total** (339 responses: 206 opus-5, 118 fable-5, 14 sonnet-5) | | **$92.21** |

The headline finding survives the correction and gets sharper:
**88.4% of the cost is cache, against 314k tokens of output — a 256:1
ratio of context re-read to text written.** Cost in an agentic loop is
driven by context size × turn count, not by how much the model writes. No
D2 decision made on "tokens per task" would have surfaced that.

The correction is itself the lesson: this workplan opened with a
mismeasured cost. Neither error was of the harness-does-nothing class the
positive-control rule was written for — both sums ran over real data.
What caught them was re-deriving the number instead of quoting it.

This workplan makes cost a first-class measured dimension so that
future AM-12 equivalents support conclusions instead of decorating an
evidence file. It follows InnerLoop v1.0, including the rules that pass
earned: every metric names its instrument, and every harness carries a
positive control.

## Phase A — Instrument

## Task: Survey the available cost instruments

```task
id: CB-WP-0002-T01
status: done
priority: high
state_hub_task_id: "2694c2c1-0070-4d8e-b4fc-196b582b36d5"
```

Produce `research/CB-RES-0002-cost-accounting.md` per the InnerLoop
survey template. Candidates at minimum: session transcript JSONL
(`usage` per assistant message), the Custodian State Hub token-event
API (`record_token_event`, `update_task_status` token tiers,
`get_token_summary`), the Claude Code status bar, and the Anthropic
usage/billing API. Per candidate: what it reports, granularity,
accuracy, whether it survives compaction, and whether it can attribute
cost to a unit of work.

Note explicitly which are **exact** and which are **estimates** — the
prior failure was tolerating an estimate-shaped metric. Name the
benchmark-to-beat per dimension; expect the transcript to lead on
accuracy and the hub to lead on durability.

## Task: Decide the instrument and the attribution model (ADR)

```task
id: CB-WP-0002-T02
status: todo
priority: high
state_hub_task_id: "eae248ab-f29f-4f11-9d20-e8145b0d822d"
```

Adversarial review of T01 first (InnerLoop §Step 2), committed as
`history/YYMMDD-cost-accounting-{challenge,response}.md`.

Then `decisions/ADR-0003-cost-accounting.md`. The genuinely hard part is
**attribution**: a transcript is a flat message stream, and a "task" is
a workplan concept. Options to weigh, not assume:

- git commit timestamps as task boundaries (the loop commits per
  iteration, so boundaries already exist and are durable)
- explicit session markers emitted at task start/end
- hub task status transitions as the time index

State the expected advantage per dimension and the known failure modes
of the chosen model — in particular, what happens to attribution across
`/compact`, across resumed sessions, and for work spanning a boundary.
Gate: no collector code before this ADR is committed.

## Task: Specify the cost metrics with named instruments

```task
id: CB-WP-0002-T03
status: todo
priority: high
state_hub_task_id: "00d42ed2-4391-4580-aae2-06e3e151c69b"
```

Write `specs/CostAccounting.md`: the cost model (input, output, cache
read, cache write at 5m and 1h, per the price sheet), the attribution
contract, and the acceptance metrics — each naming the **command that
produces its number**, per InnerLoop v1.0 §Step 4.

Must include a metric for the finding that motivated this workplan:
**cost composition** (what fraction is cache read vs write vs output),
not only a total. A single total would have hidden the 53%.

Revise `specs/MetricsAndScenarios.md` §1a to point at this spec, and
replace AM-12's definition with one that is computable.

## Phase B — Build and prove

## Task: Implement the cost collector

```task
id: CB-WP-0002-T04
status: todo
priority: medium
state_hub_task_id: "9eb8329b-5f41-477b-8cf3-2cda5ba8dbe8"
```

Implement the tool chosen in T02 (expected: `tools/cb-cost`). It reads
transcripts, applies the price sheet at
`benchmarks/baselines/model-prices.toml`, attributes cost per the T02
model, and emits both a per-task table and a composition breakdown in
the evidence-row format from T03.

**Positive control is mandatory** (InnerLoop v1.0 §Step 5): the tool
asserts that attributed tokens sum to the transcript total, and refuses
to emit numbers when they do not reconcile. An unattributed remainder is
reported as its own line, never silently dropped — the failure this
whole workplan exists to prevent was a number that looked fine.

Handle multi-model sessions: a session that switches models must price
each message at its own model's rate.

## Task: Validate against CB-WP-0001 and answer a real question

```task
id: CB-WP-0002-T05
status: todo
priority: medium
state_hub_task_id: "bea4cc0a-e4d0-4077-9dc7-df7726a48f86"
```

Run the collector over the CB-WP-0001 session and commit
`evidence/CB-EV-0002-cost-accounting.md`. Reconciliation against the
totals in this workplan's Purpose section is the acceptance test: the
tool must reproduce $248.46 (Fable 5) from the same transcript, or
explain the difference.

Then use it to answer at least one question that could not be answered
before, and record the answer. Candidates: which of T01–T09 cost most
and whether that matched its value; what a `/compact` costs; whether the
adversarial review paid for itself; how much the six T08 code iterations
cost relative to the research phase.

A tool that produces numbers nobody draws a conclusion from has not
cleared the bar that CB-WP-0001's AM-12 failed to clear.

## Task: Wire cost into the loop

```task
id: CB-WP-0002-T06
status: todo
priority: low
state_hub_task_id: "1412263b-70c1-43e4-957d-1ad6c3203ca9"
```

Make cost collection automatic rather than remembered: add the
collector to the evidence checklist in `specs/InnerLoop.md`, emit hub
token events so the Token Cost dashboard reflects real numbers rather
than the 1000/500 heuristic fallback, and add a `make cost` target to
the one command surface.

## Task: Retrospective

```task
id: CB-WP-0002-T07
status: todo
priority: low
state_hub_task_id: "ebe58d91-be5f-4d5b-ba40-b03275b4eefc"
```

Revise `specs/InnerLoop.md` and `specs/MetricsAndScenarios.md` from what
this pass teaches about metric design. The specific question to answer:
CB-WP-0001 produced a fully specified metric that could not be computed,
and v1.0's "every metric names its instrument" rule was written to stop
that. Did it? If a metric can still be written without a working
instrument, the rule needs teeth — for example, requiring that the
instrument be demonstrated on real data before the metric is accepted.