T01: cost-accounting survey — and a 2.7x correction to our own numbers
Tier L (structural L, chaos d10=2). Every candidate exercised locally
against the CB-WP-0001 transcript rather than cited.
Verdict: session transcript JSONL is the only exact instrument; git
commits supply the attribution index it lacks; the State Hub is a
durable sink whose schema cannot express cache tokens at all; the
status bar and billing API are eliminated on availability.
Three errors found in this workplan's own Purpose section, all
overstating cost:
- one API response is written as up to six JSONL lines, each
repeating the same usage object (657 lines, 346 responses)
- a multi-model session was priced entirely at Fable 5 rates,
violating a per-model rule MetricsAndScenarios §1a already had
- the transcript is a live file, so an unpinned total is not
repeatable (346 -> 356 responses between two runs minutes apart)
CB-WP-0001 pinned at fc76445: $92.21, not $248.46. The qualitative
finding sharpens rather than weakens — 88.4% of spend is cache, at
256:1 cache-read to output tokens.
Neither error was of the harness-does-nothing class the positive
control was written for; both sums ran over real data. What caught
them was re-deriving a number instead of quoting it. Raised for T07
and for CB-WP-0003 T01.
Also measured: the one subagent (adversarial review) cost $0.66 and is
invisible to a collector reading only the main transcript.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
fc76445aec
commit
060eb8cf6a
4 changed files with 385 additions and 12 deletions
258
research/CB-RES-0002-cost-accounting.md
Normal file
258
research/CB-RES-0002-cost-accounting.md
Normal file
|
|
@ -0,0 +1,258 @@
|
|||
# CB-RES-0002: agentic cost accounting
|
||||
|
||||
capability: meta.loop.cost-accounting
|
||||
status: draft
|
||||
tier: L (structural L, chaos d10=2 → no override)
|
||||
runnable-baseline: invoked — every candidate below was exercised against the
|
||||
CB-WP-0001 session on this machine, not cited
|
||||
review-trail: history/260731-cost-accounting-research.md
|
||||
|
||||
Survey of instruments that can attribute the USD cost of agentic work to a
|
||||
unit of work, so that M-D2-CST (`specs/MetricsAndScenarios.md` §1a) becomes
|
||||
computable. CB-WP-0001 specified that metric completely and recorded it as
|
||||
*uncomputable*; the premise of this workplan is that the data existed the
|
||||
whole time.
|
||||
|
||||
That premise survives. The workplan's **numbers do not** — see §Correction.
|
||||
|
||||
---
|
||||
|
||||
## Correction to this workplan's own Purpose section
|
||||
|
||||
CB-WP-0002's Purpose reports the CB-WP-0001 session at **$248.46**, from
|
||||
131,863,164 cache-read tokens priced at Fable 5. Both halves are wrong, and
|
||||
in the same direction — too high. The survey found this by re-deriving the
|
||||
number rather than adopting it.
|
||||
|
||||
**Error 1 — per-line summation double-counts.** A single API response is
|
||||
written to the transcript as *several* JSONL lines, split by content block
|
||||
(`thinking`, `text`, `tool_use`), and **every one of those lines repeats the
|
||||
complete `usage` object**. Measured on the CB-WP-0001 transcript: 657
|
||||
assistant lines carry only 346 distinct `requestId`s. Group sizes run 1–6:
|
||||
|
||||
| lines per requestId | 1 | 2 | 3 | 4 | 5 | 6 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| groups | 140 | 120 | 76 | 6 | 5 | 1 |
|
||||
|
||||
Positive control on the dedup: across all 208 multi-line groups the `usage`
|
||||
object is byte-identical (208/208 identical, 0 differing), and no group
|
||||
mixes models. The duplication is a transcript-format artifact, not repeated
|
||||
billing. Summing per line inflates by ≈1.9×.
|
||||
|
||||
**Error 2 — single-model pricing on a multi-model session.** The session ran
|
||||
three models, not one:
|
||||
|
||||
| model | API responses (deduped) |
|
||||
|---|---|
|
||||
| claude-opus-5 | 382 lines → majority |
|
||||
| claude-fable-5 | 250 lines |
|
||||
| claude-sonnet-5 | 24 lines |
|
||||
| `<synthetic>` | 1 (no usage; an error placeholder) |
|
||||
|
||||
Pricing everything at Fable 5's $10/$50 overstates the Opus and Sonnet
|
||||
majority. §1a *already required* per-model pricing; the Purpose section did
|
||||
not apply its own rule.
|
||||
|
||||
**Corrected totals** for the same transcript, all three methods run over
|
||||
the identical unpinned line set so the methods are comparable:
|
||||
|
||||
| method | responses | output | cache read | cost |
|
||||
|---|---|---|---|---|
|
||||
| per-line, all-Fable (the Purpose method) | 657 | 700,690 | 161,408,840 | $289.12 |
|
||||
| per-line, per-model | 657 | 700,690 | 161,408,840 | $210.05 |
|
||||
| **deduped, per-model (correct)** | **346** | **318,230** | **81,100,498** | **$93.15** |
|
||||
|
||||
**Third error, found while verifying the second: the transcript is a live
|
||||
file.** Re-running the deduped figure minutes later returned 356
|
||||
responses and $94.04 — this survey's own session appends to the same
|
||||
JSONL it is measuring. An unpinned total is not a repeatable number. The
|
||||
acceptance target is therefore pinned by timestamp:
|
||||
|
||||
| CB-WP-0001, pinned ≤ `2026-07-31T02:17:59Z` (commit `fc76445`) | value |
|
||||
|---|---|
|
||||
| responses | 339 (206 opus-5, 118 fable-5, 14 sonnet-5, 1 synthetic) |
|
||||
| output | 313,900 tok → $10.66 |
|
||||
| cache read | 80,453,702 tok → $59.59 |
|
||||
| cache write 1h | 1,672,854 tok → $21.95 |
|
||||
| input | 676 tok → $0.00 |
|
||||
| **total** | **$92.21** — 88.4% cache, 256:1 cache-read:output |
|
||||
|
||||
The reported figure was **~2.7× the real cost**. This is the fourth
|
||||
instance of the harness-does-nothing error class from
|
||||
`history/260731-inner-loop-retrospective.md`, wearing a new coat: not a
|
||||
harness that measured nothing, but an arithmetic that measured the same
|
||||
thing twice. Both produce a number that looks fine.
|
||||
|
||||
The qualitative headline survives the correction and gets stronger: cache
|
||||
reads are **81.1M tokens against 318k of output**, ~255:1. Cost in an
|
||||
agentic loop is context × turns.
|
||||
|
||||
---
|
||||
|
||||
## Candidates
|
||||
|
||||
### C1 — Session transcript JSONL
|
||||
|
||||
`~/.claude/projects/<slug>/<session>.jsonl`, one JSON object per line.
|
||||
Assistant lines carry `message.usage` with exact billing counters:
|
||||
`input_tokens`, `output_tokens`, `cache_read_input_tokens`, and
|
||||
`cache_creation.{ephemeral_1h,ephemeral_5m}_input_tokens`, plus
|
||||
`message.model`, `requestId`, and an ISO-8601 `timestamp`.
|
||||
|
||||
- **Granularity:** per API response, once deduplicated by `requestId`.
|
||||
- **Accuracy:** exact — these are the counters the invoice is computed from.
|
||||
There is no sampling or rounding.
|
||||
- **Verified non-issue:** `usage.iterations[]` is a sub-breakdown, not an
|
||||
additional charge. Checked all 654 usage-bearing lines: the iteration
|
||||
outputs sum exactly to the top-level `output_tokens` in every case, and
|
||||
no message had more than one iteration. Summing `iterations` *instead of*
|
||||
the top-level fields is safe; summing *both* would double-count.
|
||||
- **Survives compaction:** yes. `/compact` writes a summary message into the
|
||||
same file (`isCompactSummary`, `compactMetadata`) and the session
|
||||
continues; no usage is lost. Compaction is visible as an event, so its
|
||||
cost is itself measurable.
|
||||
- **Attribution:** none built in — a transcript is a flat message stream
|
||||
with timestamps. It must be joined against an external time index.
|
||||
- **Blind spot found:** subagent cost is **not** in the main transcript.
|
||||
`isSidechain` is `false` on all 657 lines; subagent work lives in
|
||||
`<session>/subagents/agent-*.jsonl` with an `agent-*.meta.json` naming
|
||||
the agentType and model. CB-WP-0001 spawned one (the adversarial review).
|
||||
A collector reading only the main file silently under-reports.
|
||||
|
||||
### C2 — Custodian State Hub token API
|
||||
|
||||
`record_token_event`, the `update_task_status` token tiers, and
|
||||
`get_token_summary`. Exercised against CB-WP-0001's workplan
|
||||
(`a1b434dc-…`), which returned:
|
||||
|
||||
```text
|
||||
tokens_in 362,000 tokens_out 39,100 event_count 7 by model: claude-fable-5
|
||||
```
|
||||
|
||||
- **Granularity:** per task — the best of any candidate, and the only one
|
||||
that is natively *about* the unit of work.
|
||||
- **Accuracy:** poor, and structurally so. Three independent defects:
|
||||
1. **The schema has no cache fields.** `tokens_in`/`tokens_out` cannot
|
||||
represent the finding this workplan exists to report. Cache reads are
|
||||
~87% of real spend here and the hub cannot express them at any
|
||||
fidelity.
|
||||
2. **The recorded numbers are estimates.** 7 events for 9 tasks, at
|
||||
round figures — the skill's Tier-3 heuristic (1000/500) and Tier-1
|
||||
eyeball estimates. Against a deduped transcript output of 318,230,
|
||||
the hub's 39,100 is off by ~8×; against total input it is off by
|
||||
~450×.
|
||||
3. **Model attribution is wrong.** Everything is filed under
|
||||
`claude-fable-5` on a session that was majority Opus 5.
|
||||
- **Survives compaction:** yes — it is server-side and independent of the
|
||||
client.
|
||||
- **Verdict:** durable and task-shaped, but its numbers are unusable as a
|
||||
cost source. Its role is as a **sink** for numbers computed elsewhere,
|
||||
not a source. Even as a sink it can only carry a lossy projection until
|
||||
the schema grows cache fields.
|
||||
|
||||
### C3 — Claude Code status bar
|
||||
|
||||
- **Granularity:** whole session, live.
|
||||
- **Accuracy:** unknown and unauditable — it is rendered text.
|
||||
- **Machine-readable:** no. Not configured here (`statusLine` is absent
|
||||
from `~/.claude/settings.json`), and it is not reachable from inside a
|
||||
tool call regardless.
|
||||
- **Verdict:** eliminated. The ralph-workplan skill's "read tokens from the
|
||||
status bar" instruction is the proximate cause of C2's bad numbers — it
|
||||
asks an agent to report a figure it cannot read, and an agent that cannot
|
||||
read it estimates instead. This should be raised against the skill.
|
||||
|
||||
### C4 — Anthropic usage / billing API
|
||||
|
||||
- **Granularity:** organization and API-key, by day.
|
||||
- **Accuracy:** authoritative — it *is* the invoice.
|
||||
- **Attribution:** none to a task, and none to a session. Cannot separate
|
||||
clay-borg from the other twenty-plus projects on this machine.
|
||||
- **Availability:** requires an admin key; none is configured here.
|
||||
- **Verdict:** not usable for M-D2-CST, but valuable as an **external
|
||||
reconciliation check** if an admin key is ever provisioned — it is the
|
||||
only candidate that can catch a systematic error in C1's price model.
|
||||
Left as a stated non-dependency.
|
||||
|
||||
### C5 — Git commit history (attribution index, not a cost source)
|
||||
|
||||
Not a cost instrument; the missing half of C1. The loop already commits per
|
||||
task iteration with the task in the subject line, giving durable, timestamped
|
||||
boundaries at exactly the granularity M-D2-CST wants:
|
||||
|
||||
```text
|
||||
a09d76f 2026-07-31T02:14:34+02:00 T08 iter 1: scenario runner executes; …
|
||||
b58a913 2026-07-31T02:19:34+02:00 T08 iter 2: Reveal, Resolve, End; …
|
||||
```
|
||||
|
||||
- Boundaries are 3–15 minutes apart across CB-WP-0001 — finer than a task.
|
||||
- Durable, versioned, and free; requires no change to how work is done.
|
||||
- **Known hazards:** commit timestamps are local (`+02:00`), transcript
|
||||
timestamps are UTC — a naive join is off by the offset. Work before the
|
||||
first commit and after the last has no enclosing interval. Commits made
|
||||
outside a session (or by `fix-consistency`) create empty intervals.
|
||||
|
||||
---
|
||||
|
||||
## Baselines (benchmark-to-beat)
|
||||
|
||||
| Dimension | Baseline holder | Metric | Value | Provenance |
|
||||
|---|---|---|---|---|
|
||||
| D1 ease of specification | C2 hub | fields needed to record a task's cost | 4 (`task_id`, `tokens_in`, `tokens_out`, `model`) — but cannot express cache | measured (API schema) |
|
||||
| D2 efficiency | C2 hub | cost of producing a number | ~0 (one API call) — number is an estimate, off by ~8× on output | measured |
|
||||
| D2 efficiency | C1 transcript | cost of producing a number | one file read, 5.1 MB, ~1 s; exact | measured |
|
||||
| D3 speed | C1 transcript | parse of a full session | 2,040 lines / 5.1 MB in <1 s in CPython | measured |
|
||||
| D4 optionality | C4 billing API | authority | invoice-grade, zero attribution, needs admin key | measured (absent) |
|
||||
| **Accuracy (the deciding axis)** | **C1 transcript** | agreement with billing counters | exact by construction | measured |
|
||||
|
||||
**Benchmark-to-beat for the collector:** reproduce **$92.21** for session
|
||||
`8cbd5701` pinned at `2026-07-31T02:17:59Z`, from the committed price
|
||||
sheet, with an unattributed remainder reported as its own line and
|
||||
reconciliation asserted rather than assumed.
|
||||
|
||||
---
|
||||
|
||||
## Verdict
|
||||
|
||||
**C1 (transcript) leads on accuracy and is the only exact candidate.**
|
||||
**C5 (git commits) supplies the attribution index C1 lacks.** C2 is the
|
||||
durable sink. C3 is eliminated. C4 is an optional external check.
|
||||
|
||||
The expected shape is therefore: parse transcripts → dedup by `requestId` →
|
||||
price per message at its own model's rate from
|
||||
`benchmarks/baselines/model-prices.toml` → attribute to a task by joining
|
||||
message timestamps against commit intervals → emit per-task cost and a
|
||||
composition breakdown → push a lossy summary to C2.
|
||||
|
||||
**What none of them do well — the surpass opportunity.** Every candidate
|
||||
reports *totals*. None reports **composition**, and composition is where
|
||||
the actionable finding lives: 81.1M cache-read tokens against 318k of
|
||||
output means cost is driven by how much context is re-read per turn, which
|
||||
no total can show. A metric that had reported only dollars would have been
|
||||
correct and useless.
|
||||
|
||||
**Risks in the baselines themselves.**
|
||||
|
||||
1. **The $248.46 figure was wrong and was nearly adopted as this
|
||||
workplan's acceptance target.** T05's reconciliation test must be
|
||||
against a number this survey re-derived, not against the Purpose
|
||||
section. The Purpose section needs correcting.
|
||||
2. **Dedup is load-bearing.** If the transcript format ever splits one
|
||||
response across two `requestId`s, dedup silently under-reports —
|
||||
the opposite error, and the more dangerous one. The collector must
|
||||
assert its dedup assumption (identical usage within a group) at
|
||||
runtime rather than trusting this survey's one-time check.
|
||||
3. **Subagent transcripts are a separate tree.** Measured: CB-WP-0001's
|
||||
one subagent (adversarial review, Fable 5, 7 responses, 158,096 cache
|
||||
reads) cost **$0.66**, invisible to any collector reading only the
|
||||
main file. Small here; not small for a pass that fans out.
|
||||
4. **The price sheet has a 90-day staleness rule** (§1a) and no automated
|
||||
check. Every number this capability produces inherits that.
|
||||
5. **The transcript is append-live.** It is written by the session that
|
||||
reads it, so any total is a reading at an instant. Every committed
|
||||
number from this capability states its pin (timestamp or commit), and
|
||||
the collector takes a pin argument rather than defaulting to "all".
|
||||
6. **Attribution across `/compact` and resumed sessions is unproven.** The
|
||||
join is by wall-clock; a compaction inside a task boundary is fine, but
|
||||
two sessions interleaved on one repo would mis-attribute. Not exercised
|
||||
here — CB-WP-0001 ran in a single session.
|
||||
Loading…
Add table
Add a link
Reference in a new issue