tegwick
c3e6e72ab2
Instrument: session transcript JSONL, deduped by requestId, priced per model and per cache TTL. Hub rejected as a source (no cache fields, ~100% error on the same work), kept as sink. Status bar and billing API rejected on availability. Attribution: (prev_commit, this_commit] ending-at-commit, scoped by sessionId. Explicit session markers rejected because they depend on an agent remembering a step — the exact failure this workplan exists to fix. Hub status transitions rejected as an after-the-fact time index. Stated honestly: we buy accuracy with specification complexity, and D1 gets worse. That trade is only right because the alternative on offer was not a simpler correct number but one wrong by two orders of magnitude. Seven failure modes recorded up front, each becoming a T04 test — most importantly that dedup fails in the under-reporting direction, so the invariant is asserted at runtime rather than trusted. Gate satisfied: T04 may now write collector code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
118 lines
6.4 KiB
Markdown
118 lines
6.4 KiB
Markdown
# ADR-0003: cost accounting — instrument and attribution model
|
||
|
||
status: accepted
|
||
date: 2026-07-31
|
||
tier: L (structural L, chaos d10=2 → no override)
|
||
research: [CB-RES-0002](../research/CB-RES-0002-cost-accounting.md) (approved)
|
||
review: history/260731-cost-accounting-{challenge,response}.md
|
||
supersedes: the "uncomputable" disposition of AM-12 / M-D2-CST in
|
||
[CB-EV-0001](../evidence/CB-EV-0001-game-kernel.md)
|
||
|
||
## Context
|
||
|
||
M-D2-CST was fully specified in CB-WP-0001 and never instrumented, so D2
|
||
claims about implementation efficiency rest on nothing measured. The survey
|
||
established that the data existed the whole time, and that reading it
|
||
correctly is harder than it looks: this workplan's own opening figure was
|
||
wrong by 2.7×, and the survey's first draft set an acceptance target only a
|
||
*broken* collector could hit.
|
||
|
||
## Decision
|
||
|
||
### D1 — Instrument: session transcript JSONL (C1)
|
||
|
||
Cost is computed from `~/.claude/projects/<slug>/**/*.jsonl`, deduplicated
|
||
by `requestId`, priced per message against
|
||
`benchmarks/baselines/model-prices.toml`.
|
||
|
||
Rejected: the State Hub as a *source* (its schema has no cache fields, so it
|
||
cannot represent 88% of spend, and its recorded numbers are ~100% in error
|
||
against the same work); the status bar (not machine-readable from a tool
|
||
call); the billing API (no session or task attribution, no admin key). The
|
||
hub remains the durable **sink**; the billing API remains an optional
|
||
external reconciliation check if an admin key ever exists.
|
||
|
||
### D2 — Attribution: git commit intervals, scoped by session
|
||
|
||
A message is attributed to the task named by the **next commit at or after
|
||
it**, within its own session:
|
||
|
||
```text
|
||
interval := (prev_commit_time, this_commit_time] # ending-at-commit
|
||
scope := sessionId # never wall-clock alone
|
||
task := the T## tag in the commit subject, else UNATTRIBUTED
|
||
```
|
||
|
||
Ending-at-commit is the only convention consistent with the loop's
|
||
commit-at-end-of-task pattern; the alternative shifts every task's cost one
|
||
interval. Session scoping is not optional: two sessions overlap 4 h 13 m on
|
||
this repo carrying ~$12 that no wall-clock join can separate.
|
||
|
||
Rejected: **explicit session markers** at task start/end — they require the
|
||
agent to remember, and the failure this workplan exists to fix was caused by
|
||
exactly that kind of remembered step; nothing that depends on discipline
|
||
gets to be the primary index. Rejected: **hub status transitions** as the
|
||
time index — they are written after the fact, sometimes in a batch, and one
|
||
was written for CB-WP-0001 nine minutes after the work it bounds.
|
||
|
||
### D3 — Reported shape: composition, not a total
|
||
|
||
Every report carries the cost split by component (input / output / cache
|
||
read / cache write per TTL) alongside the total. A single total would have
|
||
concealed the finding that motivated the workplan.
|
||
|
||
## Expected advantage per dimension
|
||
|
||
| Dim | Expectation vs the C2 baseline | Basis |
|
||
|---|---|---|
|
||
| **D1 ease of specification** | **worse.** The hub is one API call; this is a parser with a dedup rule, a TTL-aware price model, a session scope, and a commit join. Four moving parts against one. | measured: the survey needed three corrections to get the parse right |
|
||
| **D2 efficiency** | **better, decisively.** C2's recorded numbers are ~100% in error on the same work ($0.03-equivalent recorded against $92.21 actual). C1 is exact by construction. Cost to produce: one file read, ~1 s. | measured |
|
||
| **D3 speed** | **parity.** 2,040 lines / 5.1 MB parsed in <1 s; a hub call is a network round trip. Neither is a bottleneck. | measured |
|
||
| **D4 optionality** | **better.** The transcript is a file on disk in a documented shape; the hub is a service that must be running. The collector degrades to "no data" rather than "wrong data" when a transcript is absent. | reasoned |
|
||
|
||
An honest summary: **we are buying accuracy with specification complexity.**
|
||
That trade is right here only because the alternative is not "a simpler
|
||
correct number" but "a number that is wrong by two orders of magnitude",
|
||
which is what CB-WP-0001 actually recorded.
|
||
|
||
## Known failure modes of the chosen model
|
||
|
||
Stated rather than discovered later. Each becomes a test in T04.
|
||
|
||
1. **33% of spend has no task.** Only 14 of 33 commits name a task; the rest
|
||
hold $30.32 of $92.21. Per-task tables are a view over two-thirds of the
|
||
money and must say so wherever reported.
|
||
2. **Work spanning a boundary is assigned whole to the later task.** A
|
||
message before a commit belongs to that commit's task even if the thinking
|
||
began earlier. Accepted: the loop commits per task, so the error is
|
||
bounded by one interval (p90 17.7 min, max 36.8 min).
|
||
3. **Uncommitted work is invisible.** Cost incurred after the last commit
|
||
has no enclosing interval. Reported as an open remainder, never dropped.
|
||
4. **`/compact` is safe; resumed and concurrent sessions are the risk.**
|
||
Compaction stays within one file and one session. Two agents on one repo
|
||
are separable only because `sessionId` exists — this is why D2 scopes by
|
||
it.
|
||
5. **The price sheet cannot express a time-boxed rate.** Sonnet's intro
|
||
price is a TOML comment. $0.17 at the pin; the schema defect is the real
|
||
issue and is deferred to T03 with a stated deadline of 2026-08-31, when
|
||
the intro rate expires and the sheet becomes silently wrong.
|
||
6. **Dedup is load-bearing in the dangerous direction.** If the format ever
|
||
splits one response across two `requestId`s, the collector *under*-reports
|
||
and nothing looks wrong. T04 asserts the dedup invariant at runtime
|
||
(identical `usage` within a group) rather than trusting the survey's
|
||
one-time check.
|
||
7. **The subagent tree is a separate enumeration.** Missing it under-reports
|
||
silently; it was $0.66 here and will not stay small on a fan-out pass.
|
||
|
||
## Consequences
|
||
|
||
- **Gate satisfied:** T04 may now write collector code. Nothing before this.
|
||
- T03 specifies the metrics with this ADR's contracts; T05's acceptance test
|
||
is $92.87 as two components, with a 33% unattributed line.
|
||
- The hub's token API keeps receiving events, now as a lossy projection of a
|
||
computed number rather than as an estimate. Its schema gap (no cache
|
||
fields) is recorded as a limitation of the dashboard, not of the metric.
|
||
- **Raised out of scope, for the ralph-workplan skill:** its instruction to
|
||
"read tokens from the Claude Code status bar" asks an agent for a figure
|
||
it cannot read, and is the proximate cause of the hub's bad numbers. The
|
||
skill lives outside this repo; flagged for the maintainer.
|