T02: ADR-0003 — transcript as instrument, commit intervals as attribution
Instrument: session transcript JSONL, deduped by requestId, priced per model and per cache TTL. Hub rejected as a source (no cache fields, ~100% error on the same work), kept as sink. Status bar and billing API rejected on availability. Attribution: (prev_commit, this_commit] ending-at-commit, scoped by sessionId. Explicit session markers rejected because they depend on an agent remembering a step — the exact failure this workplan exists to fix. Hub status transitions rejected as an after-the-fact time index. Stated honestly: we buy accuracy with specification complexity, and D1 gets worse. That trade is only right because the alternative on offer was not a simpler correct number but one wrong by two orders of magnitude. Seven failure modes recorded up front, each becoming a T04 test — most importantly that dedup fails in the under-reporting direction, so the invariant is asserted at runtime rather than trusted. Gate satisfied: T04 may now write collector code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
ac3ac2a8ce
commit
c3e6e72ab2
2 changed files with 119 additions and 1 deletions
118
decisions/ADR-0003-cost-accounting.md
Normal file
118
decisions/ADR-0003-cost-accounting.md
Normal file
|
|
@ -0,0 +1,118 @@
|
|||
# ADR-0003: cost accounting — instrument and attribution model
|
||||
|
||||
status: accepted
|
||||
date: 2026-07-31
|
||||
tier: L (structural L, chaos d10=2 → no override)
|
||||
research: [CB-RES-0002](../research/CB-RES-0002-cost-accounting.md) (approved)
|
||||
review: history/260731-cost-accounting-{challenge,response}.md
|
||||
supersedes: the "uncomputable" disposition of AM-12 / M-D2-CST in
|
||||
[CB-EV-0001](../evidence/CB-EV-0001-game-kernel.md)
|
||||
|
||||
## Context
|
||||
|
||||
M-D2-CST was fully specified in CB-WP-0001 and never instrumented, so D2
|
||||
claims about implementation efficiency rest on nothing measured. The survey
|
||||
established that the data existed the whole time, and that reading it
|
||||
correctly is harder than it looks: this workplan's own opening figure was
|
||||
wrong by 2.7×, and the survey's first draft set an acceptance target only a
|
||||
*broken* collector could hit.
|
||||
|
||||
## Decision
|
||||
|
||||
### D1 — Instrument: session transcript JSONL (C1)
|
||||
|
||||
Cost is computed from `~/.claude/projects/<slug>/**/*.jsonl`, deduplicated
|
||||
by `requestId`, priced per message against
|
||||
`benchmarks/baselines/model-prices.toml`.
|
||||
|
||||
Rejected: the State Hub as a *source* (its schema has no cache fields, so it
|
||||
cannot represent 88% of spend, and its recorded numbers are ~100% in error
|
||||
against the same work); the status bar (not machine-readable from a tool
|
||||
call); the billing API (no session or task attribution, no admin key). The
|
||||
hub remains the durable **sink**; the billing API remains an optional
|
||||
external reconciliation check if an admin key ever exists.
|
||||
|
||||
### D2 — Attribution: git commit intervals, scoped by session
|
||||
|
||||
A message is attributed to the task named by the **next commit at or after
|
||||
it**, within its own session:
|
||||
|
||||
```text
|
||||
interval := (prev_commit_time, this_commit_time] # ending-at-commit
|
||||
scope := sessionId # never wall-clock alone
|
||||
task := the T## tag in the commit subject, else UNATTRIBUTED
|
||||
```
|
||||
|
||||
Ending-at-commit is the only convention consistent with the loop's
|
||||
commit-at-end-of-task pattern; the alternative shifts every task's cost one
|
||||
interval. Session scoping is not optional: two sessions overlap 4 h 13 m on
|
||||
this repo carrying ~$12 that no wall-clock join can separate.
|
||||
|
||||
Rejected: **explicit session markers** at task start/end — they require the
|
||||
agent to remember, and the failure this workplan exists to fix was caused by
|
||||
exactly that kind of remembered step; nothing that depends on discipline
|
||||
gets to be the primary index. Rejected: **hub status transitions** as the
|
||||
time index — they are written after the fact, sometimes in a batch, and one
|
||||
was written for CB-WP-0001 nine minutes after the work it bounds.
|
||||
|
||||
### D3 — Reported shape: composition, not a total
|
||||
|
||||
Every report carries the cost split by component (input / output / cache
|
||||
read / cache write per TTL) alongside the total. A single total would have
|
||||
concealed the finding that motivated the workplan.
|
||||
|
||||
## Expected advantage per dimension
|
||||
|
||||
| Dim | Expectation vs the C2 baseline | Basis |
|
||||
|---|---|---|
|
||||
| **D1 ease of specification** | **worse.** The hub is one API call; this is a parser with a dedup rule, a TTL-aware price model, a session scope, and a commit join. Four moving parts against one. | measured: the survey needed three corrections to get the parse right |
|
||||
| **D2 efficiency** | **better, decisively.** C2's recorded numbers are ~100% in error on the same work ($0.03-equivalent recorded against $92.21 actual). C1 is exact by construction. Cost to produce: one file read, ~1 s. | measured |
|
||||
| **D3 speed** | **parity.** 2,040 lines / 5.1 MB parsed in <1 s; a hub call is a network round trip. Neither is a bottleneck. | measured |
|
||||
| **D4 optionality** | **better.** The transcript is a file on disk in a documented shape; the hub is a service that must be running. The collector degrades to "no data" rather than "wrong data" when a transcript is absent. | reasoned |
|
||||
|
||||
An honest summary: **we are buying accuracy with specification complexity.**
|
||||
That trade is right here only because the alternative is not "a simpler
|
||||
correct number" but "a number that is wrong by two orders of magnitude",
|
||||
which is what CB-WP-0001 actually recorded.
|
||||
|
||||
## Known failure modes of the chosen model
|
||||
|
||||
Stated rather than discovered later. Each becomes a test in T04.
|
||||
|
||||
1. **33% of spend has no task.** Only 14 of 33 commits name a task; the rest
|
||||
hold $30.32 of $92.21. Per-task tables are a view over two-thirds of the
|
||||
money and must say so wherever reported.
|
||||
2. **Work spanning a boundary is assigned whole to the later task.** A
|
||||
message before a commit belongs to that commit's task even if the thinking
|
||||
began earlier. Accepted: the loop commits per task, so the error is
|
||||
bounded by one interval (p90 17.7 min, max 36.8 min).
|
||||
3. **Uncommitted work is invisible.** Cost incurred after the last commit
|
||||
has no enclosing interval. Reported as an open remainder, never dropped.
|
||||
4. **`/compact` is safe; resumed and concurrent sessions are the risk.**
|
||||
Compaction stays within one file and one session. Two agents on one repo
|
||||
are separable only because `sessionId` exists — this is why D2 scopes by
|
||||
it.
|
||||
5. **The price sheet cannot express a time-boxed rate.** Sonnet's intro
|
||||
price is a TOML comment. $0.17 at the pin; the schema defect is the real
|
||||
issue and is deferred to T03 with a stated deadline of 2026-08-31, when
|
||||
the intro rate expires and the sheet becomes silently wrong.
|
||||
6. **Dedup is load-bearing in the dangerous direction.** If the format ever
|
||||
splits one response across two `requestId`s, the collector *under*-reports
|
||||
and nothing looks wrong. T04 asserts the dedup invariant at runtime
|
||||
(identical `usage` within a group) rather than trusting the survey's
|
||||
one-time check.
|
||||
7. **The subagent tree is a separate enumeration.** Missing it under-reports
|
||||
silently; it was $0.66 here and will not stay small on a fan-out pass.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Gate satisfied:** T04 may now write collector code. Nothing before this.
|
||||
- T03 specifies the metrics with this ADR's contracts; T05's acceptance test
|
||||
is $92.87 as two components, with a 33% unattributed line.
|
||||
- The hub's token API keeps receiving events, now as a lossy projection of a
|
||||
computed number rather than as an estimate. Its schema gap (no cache
|
||||
fields) is recorded as a limitation of the dashboard, not of the metric.
|
||||
- **Raised out of scope, for the ralph-workplan skill:** its instruction to
|
||||
"read tokens from the Claude Code status bar" asks an agent for a figure
|
||||
it cannot read, and is the proximate cause of the hub's bad numbers. The
|
||||
skill lives outside this repo; flagged for the maintainer.
|
||||
|
|
@ -87,7 +87,7 @@ accuracy and the hub to lead on durability.
|
|||
|
||||
```task
|
||||
id: CB-WP-0002-T02
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "eae248ab-f29f-4f11-9d20-e8145b0d822d"
|
||||
```
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue