clay-borg/workplans/CB-WP-0002-cost-accounting.md

---
id: CB-WP-0002
title: "Make agentic cost measurable, so D2 claims are falsifiable"
status: proposed
---

# Purpose

CB-WP-0001 specified a cost metric (AM-12, `M-D2-CST`) in full — price
sheet, formula, staleness rule — and then could not report a number,
because nothing was ever instrumented. It was recorded as
"uncomputable".

That was wrong, and the retrospective for this workplan should say so
plainly: **the data existed the whole time.** Every Claude Code session
transcript (`~/.claude/projects/<slug>/<session>.jsonl`) carries exact
per-message `usage`, including the cache breakdown. Reading it for
CB-WP-0001's session gives:

| Component | Tokens | Cost (Fable 5) |
|---|---|---|
| Output | 585,528 | $29.28 |
| Cache read | 131,863,164 | **$131.86** |
| Cache write (1h) | 4,365,668 | $87.31 |
| Input | 1,090 | $0.01 |
| **Total** | | **$248.46** |

The headline finding from that single reading is the reason this
workplan exists: **53% of the cost is cache reads**, not output. Cost in
an agentic loop is driven by context size × turn count, not by how much
the model writes. No D2 decision made on "tokens per task" would have
surfaced that.

This workplan makes cost a first-class measured dimension so that
future AM-12 equivalents support conclusions instead of decorating an
evidence file. It follows InnerLoop v1.0, including the rules that pass
earned: every metric names its instrument, and every harness carries a
positive control.

## Phase A — Instrument

## Task: Survey the available cost instruments

```task
id: CB-WP-0002-T01
status: todo
priority: high
```

Produce `research/CB-RES-0002-cost-accounting.md` per the InnerLoop
survey template. Candidates at minimum: session transcript JSONL
(`usage` per assistant message), the Custodian State Hub token-event
API (`record_token_event`, `update_task_status` token tiers,
`get_token_summary`), the Claude Code status bar, and the Anthropic
usage/billing API. Per candidate: what it reports, granularity,
accuracy, whether it survives compaction, and whether it can attribute
cost to a unit of work.

Note explicitly which are **exact** and which are **estimates** — the
prior failure was tolerating an estimate-shaped metric. Name the
benchmark-to-beat per dimension; expect the transcript to lead on
accuracy and the hub to lead on durability.

## Task: Decide the instrument and the attribution model (ADR)

```task
id: CB-WP-0002-T02
status: todo
priority: high
```

Adversarial review of T01 first (InnerLoop §Step 2), committed as
`history/YYMMDD-cost-accounting-{challenge,response}.md`.

Then `decisions/ADR-0003-cost-accounting.md`. The genuinely hard part is
**attribution**: a transcript is a flat message stream, and a "task" is
a workplan concept. Options to weigh, not assume:

- git commit timestamps as task boundaries (the loop commits per
  iteration, so boundaries already exist and are durable)
- explicit session markers emitted at task start/end
- hub task status transitions as the time index

State the expected advantage per dimension and the known failure modes
of the chosen model — in particular, what happens to attribution across
`/compact`, across resumed sessions, and for work spanning a boundary.
Gate: no collector code before this ADR is committed.

## Task: Specify the cost metrics with named instruments

```task
id: CB-WP-0002-T03
status: todo
priority: high
```

Write `specs/CostAccounting.md`: the cost model (input, output, cache
read, cache write at 5m and 1h, per the price sheet), the attribution
contract, and the acceptance metrics — each naming the **command that
produces its number**, per InnerLoop v1.0 §Step 4.

Must include a metric for the finding that motivated this workplan:
**cost composition** (what fraction is cache read vs write vs output),
not only a total. A single total would have hidden the 53%.

Revise `specs/MetricsAndScenarios.md` §1a to point at this spec, and
replace AM-12's definition with one that is computable.

## Phase B — Build and prove

## Task: Implement the cost collector

```task
id: CB-WP-0002-T04
status: todo
priority: medium
```

Implement the tool chosen in T02 (expected: `tools/cb-cost`). It reads
transcripts, applies the price sheet at
`benchmarks/baselines/model-prices.toml`, attributes cost per the T02
model, and emits both a per-task table and a composition breakdown in
the evidence-row format from T03.

**Positive control is mandatory** (InnerLoop v1.0 §Step 5): the tool
asserts that attributed tokens sum to the transcript total, and refuses
to emit numbers when they do not reconcile. An unattributed remainder is
reported as its own line, never silently dropped — the failure this
whole workplan exists to prevent was a number that looked fine.

Handle multi-model sessions: a session that switches models must price
each message at its own model's rate.

## Task: Validate against CB-WP-0001 and answer a real question

```task
id: CB-WP-0002-T05
status: todo
priority: medium
```

Run the collector over the CB-WP-0001 session and commit
`evidence/CB-EV-0002-cost-accounting.md`. Reconciliation against the
totals in this workplan's Purpose section is the acceptance test: the
tool must reproduce $248.46 (Fable 5) from the same transcript, or
explain the difference.

Then use it to answer at least one question that could not be answered
before, and record the answer. Candidates: which of T01–T09 cost most
and whether that matched its value; what a `/compact` costs; whether the
adversarial review paid for itself; how much the six T08 code iterations
cost relative to the research phase.

A tool that produces numbers nobody draws a conclusion from has not
cleared the bar that CB-WP-0001's AM-12 failed to clear.

## Task: Wire cost into the loop

```task
id: CB-WP-0002-T06
status: todo
priority: low
```

Make cost collection automatic rather than remembered: add the
collector to the evidence checklist in `specs/InnerLoop.md`, emit hub
token events so the Token Cost dashboard reflects real numbers rather
than the 1000/500 heuristic fallback, and add a `make cost` target to
the one command surface.

## Task: Retrospective

```task
id: CB-WP-0002-T07
status: todo
priority: low
```

Revise `specs/InnerLoop.md` and `specs/MetricsAndScenarios.md` from what
this pass teaches about metric design. The specific question to answer:
CB-WP-0001 produced a fully specified metric that could not be computed,
and v1.0's "every metric names its instrument" rule was written to stop
that. Did it? If a metric can still be written without a working
instrument, the rule needs teeth — for example, requiring that the
instrument be demonstrated on real data before the metric is accepted.