diff --git a/decisions/ADR-0003-cost-accounting.md b/decisions/ADR-0003-cost-accounting.md new file mode 100644 index 0000000..fe27a1b --- /dev/null +++ b/decisions/ADR-0003-cost-accounting.md @@ -0,0 +1,118 @@ +# ADR-0003: cost accounting — instrument and attribution model + +status: accepted +date: 2026-07-31 +tier: L (structural L, chaos d10=2 → no override) +research: [CB-RES-0002](../research/CB-RES-0002-cost-accounting.md) (approved) +review: history/260731-cost-accounting-{challenge,response}.md +supersedes: the "uncomputable" disposition of AM-12 / M-D2-CST in +[CB-EV-0001](../evidence/CB-EV-0001-game-kernel.md) + +## Context + +M-D2-CST was fully specified in CB-WP-0001 and never instrumented, so D2 +claims about implementation efficiency rest on nothing measured. The survey +established that the data existed the whole time, and that reading it +correctly is harder than it looks: this workplan's own opening figure was +wrong by 2.7×, and the survey's first draft set an acceptance target only a +*broken* collector could hit. + +## Decision + +### D1 — Instrument: session transcript JSONL (C1) + +Cost is computed from `~/.claude/projects//**/*.jsonl`, deduplicated +by `requestId`, priced per message against +`benchmarks/baselines/model-prices.toml`. + +Rejected: the State Hub as a *source* (its schema has no cache fields, so it +cannot represent 88% of spend, and its recorded numbers are ~100% in error +against the same work); the status bar (not machine-readable from a tool +call); the billing API (no session or task attribution, no admin key). The +hub remains the durable **sink**; the billing API remains an optional +external reconciliation check if an admin key ever exists. + +### D2 — Attribution: git commit intervals, scoped by session + +A message is attributed to the task named by the **next commit at or after +it**, within its own session: + +```text +interval := (prev_commit_time, this_commit_time] # ending-at-commit +scope := sessionId # never wall-clock alone +task := the T## tag in the commit subject, else UNATTRIBUTED +``` + +Ending-at-commit is the only convention consistent with the loop's +commit-at-end-of-task pattern; the alternative shifts every task's cost one +interval. Session scoping is not optional: two sessions overlap 4 h 13 m on +this repo carrying ~$12 that no wall-clock join can separate. + +Rejected: **explicit session markers** at task start/end — they require the +agent to remember, and the failure this workplan exists to fix was caused by +exactly that kind of remembered step; nothing that depends on discipline +gets to be the primary index. Rejected: **hub status transitions** as the +time index — they are written after the fact, sometimes in a batch, and one +was written for CB-WP-0001 nine minutes after the work it bounds. + +### D3 — Reported shape: composition, not a total + +Every report carries the cost split by component (input / output / cache +read / cache write per TTL) alongside the total. A single total would have +concealed the finding that motivated the workplan. + +## Expected advantage per dimension + +| Dim | Expectation vs the C2 baseline | Basis | +|---|---|---| +| **D1 ease of specification** | **worse.** The hub is one API call; this is a parser with a dedup rule, a TTL-aware price model, a session scope, and a commit join. Four moving parts against one. | measured: the survey needed three corrections to get the parse right | +| **D2 efficiency** | **better, decisively.** C2's recorded numbers are ~100% in error on the same work ($0.03-equivalent recorded against $92.21 actual). C1 is exact by construction. Cost to produce: one file read, ~1 s. | measured | +| **D3 speed** | **parity.** 2,040 lines / 5.1 MB parsed in <1 s; a hub call is a network round trip. Neither is a bottleneck. | measured | +| **D4 optionality** | **better.** The transcript is a file on disk in a documented shape; the hub is a service that must be running. The collector degrades to "no data" rather than "wrong data" when a transcript is absent. | reasoned | + +An honest summary: **we are buying accuracy with specification complexity.** +That trade is right here only because the alternative is not "a simpler +correct number" but "a number that is wrong by two orders of magnitude", +which is what CB-WP-0001 actually recorded. + +## Known failure modes of the chosen model + +Stated rather than discovered later. Each becomes a test in T04. + +1. **33% of spend has no task.** Only 14 of 33 commits name a task; the rest + hold $30.32 of $92.21. Per-task tables are a view over two-thirds of the + money and must say so wherever reported. +2. **Work spanning a boundary is assigned whole to the later task.** A + message before a commit belongs to that commit's task even if the thinking + began earlier. Accepted: the loop commits per task, so the error is + bounded by one interval (p90 17.7 min, max 36.8 min). +3. **Uncommitted work is invisible.** Cost incurred after the last commit + has no enclosing interval. Reported as an open remainder, never dropped. +4. **`/compact` is safe; resumed and concurrent sessions are the risk.** + Compaction stays within one file and one session. Two agents on one repo + are separable only because `sessionId` exists — this is why D2 scopes by + it. +5. **The price sheet cannot express a time-boxed rate.** Sonnet's intro + price is a TOML comment. $0.17 at the pin; the schema defect is the real + issue and is deferred to T03 with a stated deadline of 2026-08-31, when + the intro rate expires and the sheet becomes silently wrong. +6. **Dedup is load-bearing in the dangerous direction.** If the format ever + splits one response across two `requestId`s, the collector *under*-reports + and nothing looks wrong. T04 asserts the dedup invariant at runtime + (identical `usage` within a group) rather than trusting the survey's + one-time check. +7. **The subagent tree is a separate enumeration.** Missing it under-reports + silently; it was $0.66 here and will not stay small on a fan-out pass. + +## Consequences + +- **Gate satisfied:** T04 may now write collector code. Nothing before this. +- T03 specifies the metrics with this ADR's contracts; T05's acceptance test + is $92.87 as two components, with a 33% unattributed line. +- The hub's token API keeps receiving events, now as a lossy projection of a + computed number rather than as an estimate. Its schema gap (no cache + fields) is recorded as a limitation of the dashboard, not of the metric. +- **Raised out of scope, for the ralph-workplan skill:** its instruction to + "read tokens from the Claude Code status bar" asks an agent for a figure + it cannot read, and is the proximate cause of the hub's bad numbers. The + skill lives outside this repo; flagged for the maintainer. diff --git a/workplans/CB-WP-0002-cost-accounting.md b/workplans/CB-WP-0002-cost-accounting.md index c5a82f5..72693c3 100644 --- a/workplans/CB-WP-0002-cost-accounting.md +++ b/workplans/CB-WP-0002-cost-accounting.md @@ -87,7 +87,7 @@ accuracy and the hub to lead on durability. ```task id: CB-WP-0002-T02 -status: todo +status: done priority: high state_hub_task_id: "eae248ab-f29f-4f11-9d20-e8145b0d822d" ```