clay-borg/research/CB-RES-0002-cost-accounting.md

# CB-RES-0002: agentic cost accounting

capability: meta.loop.cost-accounting
status: draft
tier: L (structural L, chaos d10=2 → no override)
runnable-baseline: invoked — every candidate below was exercised against the
CB-WP-0001 session on this machine, not cited
review-trail: history/260731-cost-accounting-research.md

Survey of instruments that can attribute the USD cost of agentic work to a
unit of work, so that M-D2-CST (`specs/MetricsAndScenarios.md` §1a) becomes
computable. CB-WP-0001 specified that metric completely and recorded it as
*uncomputable*; the premise of this workplan is that the data existed the
whole time.

That premise survives. The workplan's **numbers do not** — see §Correction.

---

## Correction to this workplan's own Purpose section

CB-WP-0002's Purpose reports the CB-WP-0001 session at **$248.46**, from
131,863,164 cache-read tokens priced at Fable 5. Both halves are wrong, and
in the same direction — too high. The survey found this by re-deriving the
number rather than adopting it.

**Error 1 — per-line summation double-counts.** A single API response is
written to the transcript as *several* JSONL lines, split by content block
(`thinking`, `text`, `tool_use`), and **every one of those lines repeats the
complete `usage` object**. Measured on the CB-WP-0001 transcript: 657
assistant lines carry only 346 distinct `requestId`s. Group sizes run 1–6:

| lines per requestId | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|
| groups | 140 | 120 | 76 | 6 | 5 | 1 |

Positive control on the dedup: across all 208 multi-line groups the `usage`
object is byte-identical (208/208 identical, 0 differing), and no group
mixes models. The duplication is a transcript-format artifact, not repeated
billing. Summing per line inflates by ≈1.9×.

**Error 2 — single-model pricing on a multi-model session.** The session ran
three models, not one:

| model | API responses (deduped) |
|---|---|
| claude-opus-5 | 382 lines → majority |
| claude-fable-5 | 250 lines |
| claude-sonnet-5 | 24 lines |
| `<synthetic>` | 1 (no usage; an error placeholder) |

Pricing everything at Fable 5's $10/$50 overstates the Opus and Sonnet
majority. §1a *already required* per-model pricing; the Purpose section did
not apply its own rule.

**Corrected totals** for the same transcript, all three methods run over
the identical unpinned line set so the methods are comparable:

| method | responses | output | cache read | cost |
|---|---|---|---|---|
| per-line, all-Fable (the Purpose method) | 657 | 700,690 | 161,408,840 | $289.12 |
| per-line, per-model | 657 | 700,690 | 161,408,840 | $210.05 |
| **deduped, per-model (correct)** | **346** | **318,230** | **81,100,498** | **$93.15** |

**Third error, found while verifying the second: the transcript is a live
file.** Re-running the deduped figure minutes later returned 356
responses and $94.04 — this survey's own session appends to the same
JSONL it is measuring. An unpinned total is not a repeatable number. The
acceptance target is therefore pinned by timestamp:

| CB-WP-0001, pinned ≤ `2026-07-31T02:17:59Z` (commit `fc76445`) | value |
|---|---|
| responses | 339 (206 opus-5, 118 fable-5, 14 sonnet-5, 1 synthetic) |
| output | 313,900 tok → $10.66 |
| cache read | 80,453,702 tok → $59.59 |
| cache write 1h | 1,672,854 tok → $21.95 |
| input | 676 tok → $0.00 |
| **total** | **$92.21** — 88.4% cache, 256:1 cache-read:output |

The reported figure was **~2.7× the real cost**. This is the fourth
instance of the harness-does-nothing error class from
`history/260731-inner-loop-retrospective.md`, wearing a new coat: not a
harness that measured nothing, but an arithmetic that measured the same
thing twice. Both produce a number that looks fine.

The qualitative headline survives the correction and gets stronger: cache
reads are **81.1M tokens against 318k of output**, ~255:1. Cost in an
agentic loop is context × turns.

---

## Candidates

### C1 — Session transcript JSONL

`~/.claude/projects/<slug>/<session>.jsonl`, one JSON object per line.
Assistant lines carry `message.usage` with exact billing counters:
`input_tokens`, `output_tokens`, `cache_read_input_tokens`, and
`cache_creation.{ephemeral_1h,ephemeral_5m}_input_tokens`, plus
`message.model`, `requestId`, and an ISO-8601 `timestamp`.

- **Granularity:** per API response, once deduplicated by `requestId`.
- **Accuracy:** exact — these are the counters the invoice is computed from.
  There is no sampling or rounding.
- **Verified non-issue:** `usage.iterations[]` is a sub-breakdown, not an
  additional charge. Checked all 654 usage-bearing lines: the iteration
  outputs sum exactly to the top-level `output_tokens` in every case, and
  no message had more than one iteration. Summing `iterations` *instead of*
  the top-level fields is safe; summing *both* would double-count.
- **Survives compaction:** yes. `/compact` writes a summary message into the
  same file (`isCompactSummary`, `compactMetadata`) and the session
  continues; no usage is lost. Compaction is visible as an event, so its
  cost is itself measurable.
- **Attribution:** none built in — a transcript is a flat message stream
  with timestamps. It must be joined against an external time index.
- **Blind spot found:** subagent cost is **not** in the main transcript.
  `isSidechain` is `false` on all 657 lines; subagent work lives in
  `<session>/subagents/agent-*.jsonl` with an `agent-*.meta.json` naming
  the agentType and model. CB-WP-0001 spawned one (the adversarial review).
  A collector reading only the main file silently under-reports.

### C2 — Custodian State Hub token API

`record_token_event`, the `update_task_status` token tiers, and
`get_token_summary`. Exercised against CB-WP-0001's workplan
(`a1b434dc-…`), which returned:

```text
tokens_in 362,000   tokens_out 39,100   event_count 7   by model: claude-fable-5
```

- **Granularity:** per task — the best of any candidate, and the only one
  that is natively *about* the unit of work.
- **Accuracy:** poor, and structurally so. Three independent defects:
  1. **The schema has no cache fields.** `tokens_in`/`tokens_out` cannot
     represent the finding this workplan exists to report. Cache reads are
     ~87% of real spend here and the hub cannot express them at any
     fidelity.
  2. **The recorded numbers are estimates.** 7 events for 9 tasks, at
     round figures — the skill's Tier-3 heuristic (1000/500) and Tier-1
     eyeball estimates. Against a deduped transcript output of 318,230,
     the hub's 39,100 is off by ~8×; against total input it is off by
     ~450×.
  3. **Model attribution is wrong.** Everything is filed under
     `claude-fable-5` on a session that was majority Opus 5.
- **Survives compaction:** yes — it is server-side and independent of the
  client.
- **Verdict:** durable and task-shaped, but its numbers are unusable as a
  cost source. Its role is as a **sink** for numbers computed elsewhere,
  not a source. Even as a sink it can only carry a lossy projection until
  the schema grows cache fields.

### C3 — Claude Code status bar

- **Granularity:** whole session, live.
- **Accuracy:** unknown and unauditable — it is rendered text.
- **Machine-readable:** no. Not configured here (`statusLine` is absent
  from `~/.claude/settings.json`), and it is not reachable from inside a
  tool call regardless.
- **Verdict:** eliminated. The ralph-workplan skill's "read tokens from the
  status bar" instruction is the proximate cause of C2's bad numbers — it
  asks an agent to report a figure it cannot read, and an agent that cannot
  read it estimates instead. This should be raised against the skill.

### C4 — Anthropic usage / billing API

- **Granularity:** organization and API-key, by day.
- **Accuracy:** authoritative — it *is* the invoice.
- **Attribution:** none to a task, and none to a session. Cannot separate
  clay-borg from the other twenty-plus projects on this machine.
- **Availability:** requires an admin key; none is configured here.
- **Verdict:** not usable for M-D2-CST, but valuable as an **external
  reconciliation check** if an admin key is ever provisioned — it is the
  only candidate that can catch a systematic error in C1's price model.
  Left as a stated non-dependency.

### C5 — Git commit history (attribution index, not a cost source)

Not a cost instrument; the missing half of C1. The loop already commits per
task iteration with the task in the subject line, giving durable, timestamped
boundaries at exactly the granularity M-D2-CST wants:

```text
a09d76f 2026-07-31T02:14:34+02:00 T08 iter 1: scenario runner executes; …
b58a913 2026-07-31T02:19:34+02:00 T08 iter 2: Reveal, Resolve, End; …
```

- Boundaries are 3–15 minutes apart across CB-WP-0001 — finer than a task.
- Durable, versioned, and free; requires no change to how work is done.
- **Known hazards:** commit timestamps are local (`+02:00`), transcript
  timestamps are UTC — a naive join is off by the offset. Work before the
  first commit and after the last has no enclosing interval. Commits made
  outside a session (or by `fix-consistency`) create empty intervals.

---

## Baselines (benchmark-to-beat)

| Dimension | Baseline holder | Metric | Value | Provenance |
|---|---|---|---|---|
| D1 ease of specification | C2 hub | fields needed to record a task's cost | 4 (`task_id`, `tokens_in`, `tokens_out`, `model`) — but cannot express cache | measured (API schema) |
| D2 efficiency | C2 hub | cost of producing a number | ~0 (one API call) — number is an estimate, off by ~8× on output | measured |
| D2 efficiency | C1 transcript | cost of producing a number | one file read, 5.1 MB, ~1 s; exact | measured |
| D3 speed | C1 transcript | parse of a full session | 2,040 lines / 5.1 MB in <1 s in CPython | measured |
| D4 optionality | C4 billing API | authority | invoice-grade, zero attribution, needs admin key | measured (absent) |
| **Accuracy (the deciding axis)** | **C1 transcript** | agreement with billing counters | exact by construction | measured |

**Benchmark-to-beat for the collector:** reproduce **$92.21** for session
`8cbd5701` pinned at `2026-07-31T02:17:59Z`, from the committed price
sheet, with an unattributed remainder reported as its own line and
reconciliation asserted rather than assumed.

---

## Verdict

**C1 (transcript) leads on accuracy and is the only exact candidate.**
**C5 (git commits) supplies the attribution index C1 lacks.** C2 is the
durable sink. C3 is eliminated. C4 is an optional external check.

The expected shape is therefore: parse transcripts → dedup by `requestId` →
price per message at its own model's rate from
`benchmarks/baselines/model-prices.toml` → attribute to a task by joining
message timestamps against commit intervals → emit per-task cost and a
composition breakdown → push a lossy summary to C2.

**What none of them do well — the surpass opportunity.** Every candidate
reports *totals*. None reports **composition**, and composition is where
the actionable finding lives: 81.1M cache-read tokens against 318k of
output means cost is driven by how much context is re-read per turn, which
no total can show. A metric that had reported only dollars would have been
correct and useless.

**Risks in the baselines themselves.**

1. **The $248.46 figure was wrong and was nearly adopted as this
   workplan's acceptance target.** T05's reconciliation test must be
   against a number this survey re-derived, not against the Purpose
   section. The Purpose section needs correcting.
2. **Dedup is load-bearing.** If the transcript format ever splits one
   response across two `requestId`s, dedup silently under-reports —
   the opposite error, and the more dangerous one. The collector must
   assert its dedup assumption (identical usage within a group) at
   runtime rather than trusting this survey's one-time check.
3. **Subagent transcripts are a separate tree.** Measured: CB-WP-0001's
   one subagent (adversarial review, Fable 5, 7 responses, 158,096 cache
   reads) cost **$0.66**, invisible to any collector reading only the
   main file. Small here; not small for a pass that fans out.
4. **The price sheet has a 90-day staleness rule** (§1a) and no automated
   check. Every number this capability produces inherits that.
5. **The transcript is append-live.** It is written by the session that
   reads it, so any total is a reading at an instant. Every committed
   number from this capability states its pin (timestamp or commit), and
   the collector takes a pin argument rather than defaulting to "all".
6. **Attribution across `/compact` and resumed sessions is unproven.** The
   join is by wall-clock; a compaction inside a task boundary is fine, but
   two sessions interleaved on one repo would mis-attribute. Not exercised
   here — CB-WP-0001 ran in a single session.