119 lines
6.4 KiB
Markdown
119 lines
6.4 KiB
Markdown
|
# ADR-0003: cost accounting — instrument and attribution model
|
|||
|
|
|
|||
|
|
status: accepted
|
|||
|
|
date: 2026-07-31
|
|||
|
|
tier: L (structural L, chaos d10=2 → no override)
|
|||
|
|
research: [CB-RES-0002](../research/CB-RES-0002-cost-accounting.md) (approved)
|
|||
|
|
review: history/260731-cost-accounting-{challenge,response}.md
|
|||
|
|
supersedes: the "uncomputable" disposition of AM-12 / M-D2-CST in
|
|||
|
|
[CB-EV-0001](../evidence/CB-EV-0001-game-kernel.md)
|
|||
|
|
|
|||
|
|
## Context
|
|||
|
|
|
|||
|
|
M-D2-CST was fully specified in CB-WP-0001 and never instrumented, so D2
|
|||
|
|
claims about implementation efficiency rest on nothing measured. The survey
|
|||
|
|
established that the data existed the whole time, and that reading it
|
|||
|
|
correctly is harder than it looks: this workplan's own opening figure was
|
|||
|
|
wrong by 2.7×, and the survey's first draft set an acceptance target only a
|
|||
|
|
*broken* collector could hit.
|
|||
|
|
|
|||
|
|
## Decision
|
|||
|
|
|
|||
|
|
### D1 — Instrument: session transcript JSONL (C1)
|
|||
|
|
|
|||
|
|
Cost is computed from `~/.claude/projects/<slug>/**/*.jsonl`, deduplicated
|
|||
|
|
by `requestId`, priced per message against
|
|||
|
|
`benchmarks/baselines/model-prices.toml`.
|
|||
|
|
|
|||
|
|
Rejected: the State Hub as a *source* (its schema has no cache fields, so it
|
|||
|
|
cannot represent 88% of spend, and its recorded numbers are ~100% in error
|
|||
|
|
against the same work); the status bar (not machine-readable from a tool
|
|||
|
|
call); the billing API (no session or task attribution, no admin key). The
|
|||
|
|
hub remains the durable **sink**; the billing API remains an optional
|
|||
|
|
external reconciliation check if an admin key ever exists.
|
|||
|
|
|
|||
|
|
### D2 — Attribution: git commit intervals, scoped by session
|
|||
|
|
|
|||
|
|
A message is attributed to the task named by the **next commit at or after
|
|||
|
|
it**, within its own session:
|
|||
|
|
|
|||
|
|
```text
|
|||
|
|
interval := (prev_commit_time, this_commit_time] # ending-at-commit
|
|||
|
|
scope := sessionId # never wall-clock alone
|
|||
|
|
task := the T## tag in the commit subject, else UNATTRIBUTED
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Ending-at-commit is the only convention consistent with the loop's
|
|||
|
|
commit-at-end-of-task pattern; the alternative shifts every task's cost one
|
|||
|
|
interval. Session scoping is not optional: two sessions overlap 4 h 13 m on
|
|||
|
|
this repo carrying ~$12 that no wall-clock join can separate.
|
|||
|
|
|
|||
|
|
Rejected: **explicit session markers** at task start/end — they require the
|
|||
|
|
agent to remember, and the failure this workplan exists to fix was caused by
|
|||
|
|
exactly that kind of remembered step; nothing that depends on discipline
|
|||
|
|
gets to be the primary index. Rejected: **hub status transitions** as the
|
|||
|
|
time index — they are written after the fact, sometimes in a batch, and one
|
|||
|
|
was written for CB-WP-0001 nine minutes after the work it bounds.
|
|||
|
|
|
|||
|
|
### D3 — Reported shape: composition, not a total
|
|||
|
|
|
|||
|
|
Every report carries the cost split by component (input / output / cache
|
|||
|
|
read / cache write per TTL) alongside the total. A single total would have
|
|||
|
|
concealed the finding that motivated the workplan.
|
|||
|
|
|
|||
|
|
## Expected advantage per dimension
|
|||
|
|
|
|||
|
|
| Dim | Expectation vs the C2 baseline | Basis |
|
|||
|
|
|---|---|---|
|
|||
|
|
| **D1 ease of specification** | **worse.** The hub is one API call; this is a parser with a dedup rule, a TTL-aware price model, a session scope, and a commit join. Four moving parts against one. | measured: the survey needed three corrections to get the parse right |
|
|||
|
|
| **D2 efficiency** | **better, decisively.** C2's recorded numbers are ~100% in error on the same work ($0.03-equivalent recorded against $92.21 actual). C1 is exact by construction. Cost to produce: one file read, ~1 s. | measured |
|
|||
|
|
| **D3 speed** | **parity.** 2,040 lines / 5.1 MB parsed in <1 s; a hub call is a network round trip. Neither is a bottleneck. | measured |
|
|||
|
|
| **D4 optionality** | **better.** The transcript is a file on disk in a documented shape; the hub is a service that must be running. The collector degrades to "no data" rather than "wrong data" when a transcript is absent. | reasoned |
|
|||
|
|
|
|||
|
|
An honest summary: **we are buying accuracy with specification complexity.**
|
|||
|
|
That trade is right here only because the alternative is not "a simpler
|
|||
|
|
correct number" but "a number that is wrong by two orders of magnitude",
|
|||
|
|
which is what CB-WP-0001 actually recorded.
|
|||
|
|
|
|||
|
|
## Known failure modes of the chosen model
|
|||
|
|
|
|||
|
|
Stated rather than discovered later. Each becomes a test in T04.
|
|||
|
|
|
|||
|
|
1. **33% of spend has no task.** Only 14 of 33 commits name a task; the rest
|
|||
|
|
hold $30.32 of $92.21. Per-task tables are a view over two-thirds of the
|
|||
|
|
money and must say so wherever reported.
|
|||
|
|
2. **Work spanning a boundary is assigned whole to the later task.** A
|
|||
|
|
message before a commit belongs to that commit's task even if the thinking
|
|||
|
|
began earlier. Accepted: the loop commits per task, so the error is
|
|||
|
|
bounded by one interval (p90 17.7 min, max 36.8 min).
|
|||
|
|
3. **Uncommitted work is invisible.** Cost incurred after the last commit
|
|||
|
|
has no enclosing interval. Reported as an open remainder, never dropped.
|
|||
|
|
4. **`/compact` is safe; resumed and concurrent sessions are the risk.**
|
|||
|
|
Compaction stays within one file and one session. Two agents on one repo
|
|||
|
|
are separable only because `sessionId` exists — this is why D2 scopes by
|
|||
|
|
it.
|
|||
|
|
5. **The price sheet cannot express a time-boxed rate.** Sonnet's intro
|
|||
|
|
price is a TOML comment. $0.17 at the pin; the schema defect is the real
|
|||
|
|
issue and is deferred to T03 with a stated deadline of 2026-08-31, when
|
|||
|
|
the intro rate expires and the sheet becomes silently wrong.
|
|||
|
|
6. **Dedup is load-bearing in the dangerous direction.** If the format ever
|
|||
|
|
splits one response across two `requestId`s, the collector *under*-reports
|
|||
|
|
and nothing looks wrong. T04 asserts the dedup invariant at runtime
|
|||
|
|
(identical `usage` within a group) rather than trusting the survey's
|
|||
|
|
one-time check.
|
|||
|
|
7. **The subagent tree is a separate enumeration.** Missing it under-reports
|
|||
|
|
silently; it was $0.66 here and will not stay small on a fan-out pass.
|
|||
|
|
|
|||
|
|
## Consequences
|
|||
|
|
|
|||
|
|
- **Gate satisfied:** T04 may now write collector code. Nothing before this.
|
|||
|
|
- T03 specifies the metrics with this ADR's contracts; T05's acceptance test
|
|||
|
|
is $92.87 as two components, with a 33% unattributed line.
|
|||
|
|
- The hub's token API keeps receiving events, now as a lossy projection of a
|
|||
|
|
computed number rather than as an estimate. Its schema gap (no cache
|
|||
|
|
fields) is recorded as a limitation of the dashboard, not of the metric.
|
|||
|
|
- **Raised out of scope, for the ralph-workplan skill:** its instruction to
|
|||
|
|
"read tokens from the Claude Code status bar" asks an agent for a figure
|
|||
|
|
it cannot read, and is the proximate cause of the hub's bad numbers. The
|
|||
|
|
skill lives outside this repo; flagged for the maintainer.
|