T02: ADR-0003 — transcript as instrument, commit intervals as attribution

Instrument: session transcript JSONL, deduped by requestId, priced per
model and per cache TTL. Hub rejected as a source (no cache fields, ~100%
error on the same work), kept as sink. Status bar and billing API
rejected on availability.

Attribution: (prev_commit, this_commit] ending-at-commit, scoped by
sessionId. Explicit session markers rejected because they depend on an
agent remembering a step — the exact failure this workplan exists to fix.
Hub status transitions rejected as an after-the-fact time index.

Stated honestly: we buy accuracy with specification complexity, and D1
gets worse. That trade is only right because the alternative on offer was
not a simpler correct number but one wrong by two orders of magnitude.

Seven failure modes recorded up front, each becoming a T04 test — most
importantly that dedup fails in the under-reporting direction, so the
invariant is asserted at runtime rather than trusted.

Gate satisfied: T04 may now write collector code.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 08:44:27 +02:00
parent ac3ac2a8ce
commit c3e6e72ab2
2 changed files with 119 additions and 1 deletions

View file

@ -0,0 +1,118 @@
# ADR-0003: cost accounting — instrument and attribution model
status: accepted
date: 2026-07-31
tier: L (structural L, chaos d10=2 → no override)
research: [CB-RES-0002](../research/CB-RES-0002-cost-accounting.md) (approved)
review: history/260731-cost-accounting-{challenge,response}.md
supersedes: the "uncomputable" disposition of AM-12 / M-D2-CST in
[CB-EV-0001](../evidence/CB-EV-0001-game-kernel.md)
## Context
M-D2-CST was fully specified in CB-WP-0001 and never instrumented, so D2
claims about implementation efficiency rest on nothing measured. The survey
established that the data existed the whole time, and that reading it
correctly is harder than it looks: this workplan's own opening figure was
wrong by 2.7×, and the survey's first draft set an acceptance target only a
*broken* collector could hit.
## Decision
### D1 — Instrument: session transcript JSONL (C1)
Cost is computed from `~/.claude/projects/<slug>/**/*.jsonl`, deduplicated
by `requestId`, priced per message against
`benchmarks/baselines/model-prices.toml`.
Rejected: the State Hub as a *source* (its schema has no cache fields, so it
cannot represent 88% of spend, and its recorded numbers are ~100% in error
against the same work); the status bar (not machine-readable from a tool
call); the billing API (no session or task attribution, no admin key). The
hub remains the durable **sink**; the billing API remains an optional
external reconciliation check if an admin key ever exists.
### D2 — Attribution: git commit intervals, scoped by session
A message is attributed to the task named by the **next commit at or after
it**, within its own session:
```text
interval := (prev_commit_time, this_commit_time] # ending-at-commit
scope := sessionId # never wall-clock alone
task := the T## tag in the commit subject, else UNATTRIBUTED
```
Ending-at-commit is the only convention consistent with the loop's
commit-at-end-of-task pattern; the alternative shifts every task's cost one
interval. Session scoping is not optional: two sessions overlap 4 h 13 m on
this repo carrying ~$12 that no wall-clock join can separate.
Rejected: **explicit session markers** at task start/end — they require the
agent to remember, and the failure this workplan exists to fix was caused by
exactly that kind of remembered step; nothing that depends on discipline
gets to be the primary index. Rejected: **hub status transitions** as the
time index — they are written after the fact, sometimes in a batch, and one
was written for CB-WP-0001 nine minutes after the work it bounds.
### D3 — Reported shape: composition, not a total
Every report carries the cost split by component (input / output / cache
read / cache write per TTL) alongside the total. A single total would have
concealed the finding that motivated the workplan.
## Expected advantage per dimension
| Dim | Expectation vs the C2 baseline | Basis |
|---|---|---|
| **D1 ease of specification** | **worse.** The hub is one API call; this is a parser with a dedup rule, a TTL-aware price model, a session scope, and a commit join. Four moving parts against one. | measured: the survey needed three corrections to get the parse right |
| **D2 efficiency** | **better, decisively.** C2's recorded numbers are ~100% in error on the same work ($0.03-equivalent recorded against $92.21 actual). C1 is exact by construction. Cost to produce: one file read, ~1 s. | measured |
| **D3 speed** | **parity.** 2,040 lines / 5.1 MB parsed in <1 s; a hub call is a network round trip. Neither is a bottleneck. | measured |
| **D4 optionality** | **better.** The transcript is a file on disk in a documented shape; the hub is a service that must be running. The collector degrades to "no data" rather than "wrong data" when a transcript is absent. | reasoned |
An honest summary: **we are buying accuracy with specification complexity.**
That trade is right here only because the alternative is not "a simpler
correct number" but "a number that is wrong by two orders of magnitude",
which is what CB-WP-0001 actually recorded.
## Known failure modes of the chosen model
Stated rather than discovered later. Each becomes a test in T04.
1. **33% of spend has no task.** Only 14 of 33 commits name a task; the rest
hold $30.32 of $92.21. Per-task tables are a view over two-thirds of the
money and must say so wherever reported.
2. **Work spanning a boundary is assigned whole to the later task.** A
message before a commit belongs to that commit's task even if the thinking
began earlier. Accepted: the loop commits per task, so the error is
bounded by one interval (p90 17.7 min, max 36.8 min).
3. **Uncommitted work is invisible.** Cost incurred after the last commit
has no enclosing interval. Reported as an open remainder, never dropped.
4. **`/compact` is safe; resumed and concurrent sessions are the risk.**
Compaction stays within one file and one session. Two agents on one repo
are separable only because `sessionId` exists — this is why D2 scopes by
it.
5. **The price sheet cannot express a time-boxed rate.** Sonnet's intro
price is a TOML comment. $0.17 at the pin; the schema defect is the real
issue and is deferred to T03 with a stated deadline of 2026-08-31, when
the intro rate expires and the sheet becomes silently wrong.
6. **Dedup is load-bearing in the dangerous direction.** If the format ever
splits one response across two `requestId`s, the collector *under*-reports
and nothing looks wrong. T04 asserts the dedup invariant at runtime
(identical `usage` within a group) rather than trusting the survey's
one-time check.
7. **The subagent tree is a separate enumeration.** Missing it under-reports
silently; it was $0.66 here and will not stay small on a fan-out pass.
## Consequences
- **Gate satisfied:** T04 may now write collector code. Nothing before this.
- T03 specifies the metrics with this ADR's contracts; T05's acceptance test
is $92.87 as two components, with a 33% unattributed line.
- The hub's token API keeps receiving events, now as a lossy projection of a
computed number rather than as an estimate. Its schema gap (no cache
fields) is recorded as a limitation of the dashboard, not of the metric.
- **Raised out of scope, for the ralph-workplan skill:** its instruction to
"read tokens from the Claude Code status bar" asks an agent for a figure
it cannot read, and is the proximate cause of the hub's bad numbers. The
skill lives outside this repo; flagged for the maintainer.

View file

@ -87,7 +87,7 @@ accuracy and the hub to lead on durability.
```task
id: CB-WP-0002-T02
status: todo
status: done
priority: high
state_hub_task_id: "eae248ab-f29f-4f11-9d20-e8145b0d822d"
```