diff --git a/.forgejo/workflows/ci.yml b/.forgejo/workflows/ci.yml index df9a045..9185eb1 100644 --- a/.forgejo/workflows/ci.yml +++ b/.forgejo/workflows/ci.yml @@ -33,6 +33,11 @@ jobs: # AM-4a/AM-4b: third-party source under audit, per configuration. - run: make dep-weight + # Positive control for the cost collector (AC-5..AC-9). Does not + # gate on a dollar figure — transcripts are not present in CI — but + # proves the collector still detects the failures it claims to. + - run: make cost-test + # InnerLoop v1.0 positive control, enforced rather than asserted in # prose: --test runs every benchmark once, so a workload that # stalls or produces the wrong event count fails the build instead diff --git a/Makefile b/Makefile index f3799e1..46e9e79 100644 --- a/Makefile +++ b/Makefile @@ -3,7 +3,7 @@ CARGO := cargo -.PHONY: check test sim bench bench-test coverage dep-weight loc all +.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin loc all ## fmt + clippy (deny warnings) + HashMap deny-lint check: @@ -21,6 +21,17 @@ dep-weight: coverage: python3 tools/rule-coverage.py +# M-D2-CST (specs/CostAccounting.md). cost-test is the positive control and +# runs first: a cost number from an unverified collector is void. +cost: cost-test + python3 tools/cb-cost.py --composition --by-task + +cost-test: + python3 tools/cb-cost.py --self-test + +cost-pin: cost-test + python3 tools/cb-cost.py --pin fc76445 --composition --by-task + sim: $(CARGO) run -q -p cb-sim -- scenarios/ground/*.yaml @@ -40,4 +51,4 @@ loc: printf '%-28s %s\n' $$d "$$(find $$d/src -name '*.rs' | xargs cat | grep -vcE '^\s*(//|$$)')"; \ done -all: check test sim coverage dep-weight bench-test +all: check test sim coverage dep-weight cost-test bench-test diff --git a/decisions/ADR-0003-cost-accounting.md b/decisions/ADR-0003-cost-accounting.md new file mode 100644 index 0000000..fe27a1b --- /dev/null +++ b/decisions/ADR-0003-cost-accounting.md @@ -0,0 +1,118 @@ +# ADR-0003: cost accounting — instrument and attribution model + +status: accepted +date: 2026-07-31 +tier: L (structural L, chaos d10=2 → no override) +research: [CB-RES-0002](../research/CB-RES-0002-cost-accounting.md) (approved) +review: history/260731-cost-accounting-{challenge,response}.md +supersedes: the "uncomputable" disposition of AM-12 / M-D2-CST in +[CB-EV-0001](../evidence/CB-EV-0001-game-kernel.md) + +## Context + +M-D2-CST was fully specified in CB-WP-0001 and never instrumented, so D2 +claims about implementation efficiency rest on nothing measured. The survey +established that the data existed the whole time, and that reading it +correctly is harder than it looks: this workplan's own opening figure was +wrong by 2.7×, and the survey's first draft set an acceptance target only a +*broken* collector could hit. + +## Decision + +### D1 — Instrument: session transcript JSONL (C1) + +Cost is computed from `~/.claude/projects//**/*.jsonl`, deduplicated +by `requestId`, priced per message against +`benchmarks/baselines/model-prices.toml`. + +Rejected: the State Hub as a *source* (its schema has no cache fields, so it +cannot represent 88% of spend, and its recorded numbers are ~100% in error +against the same work); the status bar (not machine-readable from a tool +call); the billing API (no session or task attribution, no admin key). The +hub remains the durable **sink**; the billing API remains an optional +external reconciliation check if an admin key ever exists. + +### D2 — Attribution: git commit intervals, scoped by session + +A message is attributed to the task named by the **next commit at or after +it**, within its own session: + +```text +interval := (prev_commit_time, this_commit_time] # ending-at-commit +scope := sessionId # never wall-clock alone +task := the T## tag in the commit subject, else UNATTRIBUTED +``` + +Ending-at-commit is the only convention consistent with the loop's +commit-at-end-of-task pattern; the alternative shifts every task's cost one +interval. Session scoping is not optional: two sessions overlap 4 h 13 m on +this repo carrying ~$12 that no wall-clock join can separate. + +Rejected: **explicit session markers** at task start/end — they require the +agent to remember, and the failure this workplan exists to fix was caused by +exactly that kind of remembered step; nothing that depends on discipline +gets to be the primary index. Rejected: **hub status transitions** as the +time index — they are written after the fact, sometimes in a batch, and one +was written for CB-WP-0001 nine minutes after the work it bounds. + +### D3 — Reported shape: composition, not a total + +Every report carries the cost split by component (input / output / cache +read / cache write per TTL) alongside the total. A single total would have +concealed the finding that motivated the workplan. + +## Expected advantage per dimension + +| Dim | Expectation vs the C2 baseline | Basis | +|---|---|---| +| **D1 ease of specification** | **worse.** The hub is one API call; this is a parser with a dedup rule, a TTL-aware price model, a session scope, and a commit join. Four moving parts against one. | measured: the survey needed three corrections to get the parse right | +| **D2 efficiency** | **better, decisively.** C2's recorded numbers are ~100% in error on the same work ($0.03-equivalent recorded against $92.21 actual). C1 is exact by construction. Cost to produce: one file read, ~1 s. | measured | +| **D3 speed** | **parity.** 2,040 lines / 5.1 MB parsed in <1 s; a hub call is a network round trip. Neither is a bottleneck. | measured | +| **D4 optionality** | **better.** The transcript is a file on disk in a documented shape; the hub is a service that must be running. The collector degrades to "no data" rather than "wrong data" when a transcript is absent. | reasoned | + +An honest summary: **we are buying accuracy with specification complexity.** +That trade is right here only because the alternative is not "a simpler +correct number" but "a number that is wrong by two orders of magnitude", +which is what CB-WP-0001 actually recorded. + +## Known failure modes of the chosen model + +Stated rather than discovered later. Each becomes a test in T04. + +1. **33% of spend has no task.** Only 14 of 33 commits name a task; the rest + hold $30.32 of $92.21. Per-task tables are a view over two-thirds of the + money and must say so wherever reported. +2. **Work spanning a boundary is assigned whole to the later task.** A + message before a commit belongs to that commit's task even if the thinking + began earlier. Accepted: the loop commits per task, so the error is + bounded by one interval (p90 17.7 min, max 36.8 min). +3. **Uncommitted work is invisible.** Cost incurred after the last commit + has no enclosing interval. Reported as an open remainder, never dropped. +4. **`/compact` is safe; resumed and concurrent sessions are the risk.** + Compaction stays within one file and one session. Two agents on one repo + are separable only because `sessionId` exists — this is why D2 scopes by + it. +5. **The price sheet cannot express a time-boxed rate.** Sonnet's intro + price is a TOML comment. $0.17 at the pin; the schema defect is the real + issue and is deferred to T03 with a stated deadline of 2026-08-31, when + the intro rate expires and the sheet becomes silently wrong. +6. **Dedup is load-bearing in the dangerous direction.** If the format ever + splits one response across two `requestId`s, the collector *under*-reports + and nothing looks wrong. T04 asserts the dedup invariant at runtime + (identical `usage` within a group) rather than trusting the survey's + one-time check. +7. **The subagent tree is a separate enumeration.** Missing it under-reports + silently; it was $0.66 here and will not stay small on a fan-out pass. + +## Consequences + +- **Gate satisfied:** T04 may now write collector code. Nothing before this. +- T03 specifies the metrics with this ADR's contracts; T05's acceptance test + is $92.87 as two components, with a 33% unattributed line. +- The hub's token API keeps receiving events, now as a lossy projection of a + computed number rather than as an estimate. Its schema gap (no cache + fields) is recorded as a limitation of the dashboard, not of the metric. +- **Raised out of scope, for the ralph-workplan skill:** its instruction to + "read tokens from the Claude Code status bar" asks an agent for a figure + it cannot read, and is the proximate cause of the hub's bad numbers. The + skill lives outside this repo; flagged for the maintainer. diff --git a/evidence/CB-EV-0002-cost-accounting.md b/evidence/CB-EV-0002-cost-accounting.md new file mode 100644 index 0000000..a8f986a --- /dev/null +++ b/evidence/CB-EV-0002-cost-accounting.md @@ -0,0 +1,152 @@ +# CB-EV-0002: cost accounting + +capability: meta.loop.cost-accounting +spec: [CostAccounting.md](../specs/CostAccounting.md) v1.0 +decision: [ADR-0003](../decisions/ADR-0003-cost-accounting.md) +research: [CB-RES-0002](../research/CB-RES-0002-cost-accounting.md) +instrument: `make cost-pin` (`tools/cb-cost.py --pin fc76445`) +measured: 2026-07-31 + +Subject of measurement: the CB-WP-0001 session, pinned at commit `fc76445` +(`2026-07-31T02:17:59Z`). All numbers below are emitted by the tool, not +transcribed by hand (CA-15). + +--- + +## 1. Acceptance table + +| ID | Metric | Target | Measured | Verdict | +|---|---|---|---|---| +| AC-1 | pinned total, as two components | $93.32 = $92.21 + $1.11 | **$92.21 main + $1.11 subagent = $93.32** | **met** | +| AC-2 | reconciliation residual | $0.00 | **$0.000000** | **met** | +| AC-3 | unattributed share reported | present, 33% | **32.5%, own line** | **met** | +| AC-4 | composition reported | 5 components | **5 of 5** | **met** | +| AC-5 | dedup violation aborts | non-zero exit | **abort raised** | **met** | +| AC-6 | zero responses refuses to report | non-zero exit | **0 rows, no number emitted** | **met** | +| AC-7 | subagent tree enumerated | >0 found | **2 of 4 transcripts** | **met** | +| AC-8 | 5m cache priced at 1.25× | $1.25/100k @ fable | **$1.2500 (1h would be $2.0000)** | **met** | +| AC-9 | streamed partial output → final | 5,5,195 → 195 | **195** | **met** | + +No `unmeasured` rows. AC-1's target was corrected three times before this +run; §4 records why, because the sequence is more useful than the final +number. + +## 2. Composition — the finding the metric exists for + +```text + input 690 tok $ 0.00 0.0% + output 323,643 tok $ 11.15 11.9% + cache_read 80,611,798 tok $ 59.75 64.0% + write_5m 37,467 tok $ 0.47 0.5% + write_1h 1,672,854 tok $ 21.95 23.5% + TOTAL $ 93.32 +``` + +**88.0% of spend is cache; 11.9% is output.** The ratio of context re-read +to text written is 249:1. A single total would have shown $93.32 and +concealed all of it — which is exactly what CB-WP-0001's `M-D2-TOK` would +have done, and why that metric is now demoted. + +## 3. Per-task attribution + +```text + UNATTRIBUTED $ 30.32 32.5% + T08 $ 21.02 22.5% + T07 $ 12.01 12.9% + T03 $ 9.91 10.6% + T04 $ 8.00 8.6% + T05 $ 7.19 7.7% + T06 $ 3.58 3.8% + T09 $ 1.29 1.4% +``` + +**Stated limit (CA-10):** 32.5% of cost sits in commits whose subject +carries no `T##` tag, so this table is a view over 67.5% of spend. That is +a property of commit hygiene, not of the collector. + +T08 (the GROUND aggregate, six code iterations) at $21.02 is the most +expensive task and was also the one that produced the most rework — the +tuple-map hash panic, the discarded `setup.patch`, and the 5.6×-wrong +benchmark all originated there. Expensive *and* error-dense: the correlation +is worth watching, not yet a conclusion from n=1. + +## 4. What it does not support + +- **The per-task figures are not comparable across passes.** They mix + models (opus/fable/sonnet) at different price points and different cache + states. The dollar figure is comparable; a token count is not. +- **AC-3's 32.5% is a fixture pin, not a quality target.** Improving commit + tagging will move it, and that is the desired direction. +- **This is one session.** Every ratio here (cache share, $/turn, the + compaction effect in §5) is n=1 and should be treated as a hypothesis + until a second pass reproduces it. +- **The price sheet cannot express a time-boxed rate.** Sonnet's intro price + lives in a TOML comment, so sonnet-priced work is off by $0.17 here + (0.19%). This becomes a real error on 2026-08-31. +- **AC-1 is not invoice-verified.** No admin key exists, so the Anthropic + billing API could not independently confirm the total. The transcript + counters are the same ones billing uses, but that is an argument, not a + reconciliation. + +## 5. The question that could not be answered before + +**"What does `/compact` actually cost, and is a long session quadratic?"** + +CB-WP-0003 T04 asserts that cost ≈ turns × mean_context and that mean_context +grows with turns, making long sessions quadratic. The data **qualifies that +claim**: it is true only between compactions. + +The session compacted twice, and the transcript records both directly: + +| compaction | pre-tokens | post-tokens | reduction | +|---|---|---|---| +| C1 `00:07:14Z` (manual) | 542,991 | 19,974 | **27×** | +| C2 `02:22:23Z` (manual) | 344,954 | 19,035 | **18×** | + +Cost per turn across the boundary: + +| segment | turns | mean context | total | $/turn | +|---|---|---|---|---| +| start → C1 | 136 | 304,178 | $62.19 | **$0.457** | +| C1 → C2 | 202 | 193,493 | $30.01 | **$0.149** | + +**The 202 turns after the first compaction cost less than half of the 136 +turns before it — a 3.1× drop in cost per turn.** Context growth *is* the +cost driver, and compaction is the control on it. At the pre-compact rate, +those 202 turns would have cost ~$92 instead of $30. + +Two consequences for CB-WP-0003: + +1. **T04's "one task per session" recommendation is not the only remedy, + and may not be the cheapest one.** Compaction achieved a 27× context + reduction inside a running session at the cost of one summarization + call. A fresh session pays a cold-start re-read of the committed + artifacts; compaction pays a summary. Which is cheaper is now a + measurable question rather than a matter of taste, and T04 should + measure it before prescribing. +2. **The quadratic claim should be restated as bounded-quadratic:** cost + grows with context between compactions and resets at each one. The + failure mode is not "a long session" but "a long *uncompacted* session". + +## 6. Retrospective note + +The positive control paid for itself on its first execution, which is the +strongest evidence this project has produced for the InnerLoop v1.0 rule +that added it. CA-02 asserted that `usage` is identical across the lines of +one `requestId` — verified twice on the main transcript, by the survey +(206/206 groups) and independently by the adversarial reviewer. It is false +in the `subagents/` tree, where `output_tokens` is a running count +(`5, 5, 195`). The assertion fired, the run aborted, and the tool refused to +print a number. Under the prior first-wins rule it would have printed a +plausible one. + +The generalization that failed is worth naming: **a property verified on the +largest sample was assumed to hold on the smallest one.** The main +transcript is 338 of 346 responses, so 206/206 felt conclusive; the +violation lives entirely in the 8 responses nobody checked separately. + +Cost of the adversarial review this pass: **$1.11**, against a $93.32 pass. +It found three approval-blocking defects, one of which (the subagent +exclusion) would have made this evidence file certify a broken collector. +Second consecutive pass where a ~1% spend on review changed the outcome. +That is now two data points for CB-WP-0003 T03. diff --git a/history/260731-cost-accounting-challenge.md b/history/260731-cost-accounting-challenge.md new file mode 100644 index 0000000..75ac870 --- /dev/null +++ b/history/260731-cost-accounting-challenge.md @@ -0,0 +1,163 @@ +# 2026-07-31 — adversarial challenge to CB-RES-0002 + +Round 1 of 1 (InnerLoop §Step 2). Reviewer: separate agent session, given +only `research/CB-RES-0002-cost-accounting.md` and read-only access to the +raw data. Verbatim below; answers in `260731-cost-accounting-response.md`. + +Reviewer cost: 56,882 tokens, 14 tool uses, 330 s. + +--- + +**Reproduced cleanly** (`python3` over the pinned line set, prices from +`benchmarks/baselines/model-prices.toml`): pin `2026-07-31T02:17:59Z` = +commit `fc76445` (`%cI` = `2026-07-31T04:17:59+02:00`) ✓. 339 responses ✓ +(206 opus / 118 fable / 14 sonnet / 1 synthetic) ✓. Output 313,900 → +$10.6625 ✓. Cache read 80,453,702 → $59.5880 ✓. Cache write 1h 1,672,854 → +$21.9525 ✓. Input 676 → $0.0045 ✓. **Total $92.2075** ✓. Cache share +88.43% ✓. Ratio 256.30:1 ✓. The three-method table reproduces to the cent +over the first 657 assistant lines: 346 groups, 700,690 out, 161,408,840 +cread, **$289.1248 / $210.0536 / $93.1461** ✓. The "356 / $94.04" drift +check reproduces at assistant line 682 ✓. Subagent **$0.6580** ✓. Dedup +positive control holds: 0/200 pinned multi-line groups differ in `usage` or +model; 0 assistant lines anywhere in the file lack a `requestId`. +`iterations` claim holds: 758/760 non-null, 0 sum-mismatches, 0 +multi-iteration. `cache_creation_input_tokens` == `ephemeral_1h + +ephemeral_5m` in all 775 usage lines across both transcripts. The survey did +re-derive; the boast is earned. + +**1. The dedup positive-control table does not sum to its own totals. — +MATERIAL** Claim: group sizes `140/120/76/6/5/1`, "346 distinct requestIds", +"all 208 multi-line groups". Recomputed over the same 657-line prefix: +`{1:140, 2:120, 3:74, 4:6, 5:5, 6:1}`. The stated table gives 348 groups and +663 lines, contradicting both 346 and 657. Multi-line groups are **206**, not +208. This is the sole evidence for the load-bearing dedup assumption (Risk +2), so a table that fails its own checksum is the wrong place to be sloppy. +The 74 correction makes it sum exactly. + +**2. The multi-model table is labelled "API responses (deduped)" but holds +per-line counts. — MATERIAL** `382 + 250 + 24 + 1 = 657` — those are lines, +the exact quantity the preceding paragraph just condemned. Deduped over the +same prefix: opus 213, fable 118, sonnet 14, synthetic 1 = 346. The mislabel +sits three lines below "Summing per line inflates by ≈1.9×." + +**3. The $92.21 acceptance target contradicts C1's own blind-spot finding. — +MATERIAL (blocking for T05 as written)** The subagent transcript spans +`23:11:42Z–23:14:12Z` on 2026-07-30 — **inside the pin window**. Its $0.6580 +is therefore part of the pinned session's true cost, which is **$92.87**, not +$92.21. The survey never states whether $92.21 is main-file-only. A collector +that correctly implements the C1 blind-spot ("a collector reading only the +main file silently under-reports") produces $92.87 and **fails** the +benchmark-to-beat. Either pin $92.87, or state "$92.21 main transcript + +$0.66 subagent = $92.87" explicitly. + +**4. Pricing the top-level `cache_creation_input_tokens` at the 1h rate +overcharges by 60% on 5m blocks — and the only 5m data in the corpus is the +subagent. — MATERIAL** Main session: `w5m = 0`, so the pinned $92.21 is +insensitive to this. The subagent's *entire* cache write is 5m: 37,467 +tokens. At `write_5m=1.25` → $0.4683; at `write_1h=2.0` → $0.7493. On a +$0.658 total that is a **+43% error**. C1 lists both field names but never +states that the top-level aggregate must not be priced at a single +multiplier. Given that a fan-out pass is exactly where subagents (and +therefore 5m caching) dominate, this belongs in the risk list. + +**5. Concurrent sessions on one repo are not hypothetical — they are already +in the data. — MATERIAL** Risk 6: "two sessions interleaved on one repo would +mis-attribute. Not exercised here — CB-WP-0001 ran in a single session." +Session `f1eb1147` (same project dir) spans `02:18:13Z–06:33:41Z` while +`8cbd5701` runs to `06:30:47Z`. That is a **4 h 13 m overlap** carrying **67 +responses / $6.09** from `8cbd5701` and **67 responses / $5.90** from +`f1eb1147` — ~$12 that a wall-clock join cannot separate, starting 14 seconds +after the pin. The survey also never mentions that a second transcript for +this repo exists; C1's blind-spot section names only the subagent tree. + +**6. Attribution coverage is never quantified, and a third of the money has +no task. — MATERIAL** Stress-tested with intervals `(prev commit, this +commit]` over the 33 commits at/before pin: 0 responses before the first +commit, 0 after the last, exactly **1** empty interval (`Initial commit`). So +far so good. But **19 of 33 commits carry no `T##` in the subject** +(`chore(consistency)` ×4, `Add CB-WP-000x`, `CI: enforce every gate`, `AM-4:`, +`Adopt TRSL V1C1`, …), and they hold **$29.58 of $92.21 = 32% of cost**. +C5's "the loop already commits per task iteration with the task in the +subject line" is true of 14/33 commits. The benchmark-to-beat correctly +demands an unattributed-remainder line but sizes it nowhere; 32% is the +number that belongs in the survey. + +**7. "Cache reads are ~87% of real spend" (C2, defect 1) matches no computed +quantity. — MATERIAL** Cache read alone = $59.588/$92.2075 = **64.6%** of +cost. All cache (read + 1h write) = **88.4%**. Cache-read tokens as a share +of input tokens = **98.0%**. Nothing is 87%. Since C2's elimination rests on +"the hub cannot express the ~87%", fix it to 64.6% (cost) or 88.4% (all +cache) and say which. + +**8. The `` message is described wrongly, though its dollar impact +is zero. — MINOR** "1 (no usage; an error placeholder)". It has a valid +`requestId` (`req_011CdZ9m3xZSLgtg2baVuLmG`) and a **complete all-zero +`usage` object**. Zero dollars are dropped — confirmed, the unpriced +remainder is exactly $0.00 — but a collector guarding on `if not usage:` and +one guarding on `if model not in prices:` take different branches, and the +survey's description points at the wrong one. + +**9. "Checked all 654 usage-bearing lines" does not reproduce. — MINOR** All +638 pinned assistant lines (and all 760 in the full file) carry `usage`; +637/638 pinned carry non-null `iterations`. There is no state of the file +with 654 usage-bearing lines out of 657. The *conclusion* is verified; the +*count* is not. + +**10. The timezone hazard is stated as a single offset; the repo has two. — +MINOR** `git log` offsets: `+02:00` ×34, **`+00:00` ×1** (`8b66604 Initial +commit`). "commit timestamps are local (`+02:00`) … a naive join is off by +the offset" invites a fixed −2 h correction that is wrong for the first +commit. Say "parse `%cI` and convert", not "subtract the offset". + +**11. Commit spacing understated. — MINOR** "Boundaries are 3–15 minutes +apart across CB-WP-0001." Measured over the 33 pinned commits: min 0.1, p50 +**6.9**, p90 **17.7**, max **36.8** minutes. The max matters — a 37-minute +interval is coarser than a task. + +**12. The interval convention is never named. — MINOR** "Join message +timestamps against commit intervals" is ambiguous between *ending-at-commit* +and *starting-at-commit*. The commit-at-end-of-task pattern makes only the +first correct, and under it the boundary conditions are clean (0 orphans +either end, verified). Under the other, every task's cost shifts one interval +and the reviewer's worry is real. One sentence fixes it. + +**13. The price sheet's intro rate is a comment, not data. — MINOR** +`model-prices.toml` encodes sonnet at 3/15 with `# intro 2.00 through +2026-08-31`. A collector reading the TOML — which the Verdict prescribes — +silently ignores a live rate. Delta at the pin: **$0.1738** (0.19% of +$92.21). Small now; the failure mode is that a *time-boxed* rate is +unrepresentable in the schema, which is the same class of defect the survey +levels at C2. + +**14. Baselines table: D1–D4 are each present, but the deciding axis is a +fifth non-rubric row with no beatable value. — MINOR** D1, D2 (×2), D3, D4 +all appear, so the template is satisfied on its face. But the row marked +"**the deciding axis**" is "Accuracy", which is not one of the four +dimensions, and its value — "exact by construction" — is not a number +anything can be measured against. D4's provenance reads "measured (absent)", +which is not a measurement. Only one benchmark-to-beat ($92.21) is given, +globally, not per dimension. + +**15. C3/C4 elimination is legitimate — not the sin the survey accuses the +workplan of. — no finding.** Both are eliminated on *categorical* grounds +(C3: not machine-readable, `statusLine` absent; C4: zero session/task +attribution, no admin key), not on unmeasured performance claims. You cannot +measure the cost-accuracy of an instrument that structurally cannot +attribute. C4's "authoritative — it *is* the invoice" is asserted rather than +measured, but it is used only to justify keeping C4 as an optional external +check, which is the conservative direction. + +**16. Framing note: Fable is the cost majority, Opus is only the count +majority. — MINOR** Pinned per-model cost: fable $55.50 (60%) on 118 +responses, opus $36.19 (39%) on 206, sonnet $0.52 (0.6%) on 14. "Pricing +everything at Fable 5's $10/$50 overstates the Opus and Sonnet majority" is +true by response count but reads as a cost claim, and in a document whose +thesis is that totals mislead, the 35%-of-calls / 60%-of-dollars inversion is +the more interesting fact. + +**Bottom line:** the arithmetic is sound — every pinned figure and the entire +three-method table reproduce to the cent, which is a genuinely strong result. +The survey is not approvable as-is because of #1 (the dedup control fails its +own checksum), #2 (mislabelled table), and #3 (the acceptance target is +provably incomplete by $0.66 against the survey's own blind-spot finding). +#4–#7 must be folded in; the rest are corrections. diff --git a/history/260731-cost-accounting-research.md b/history/260731-cost-accounting-research.md new file mode 100644 index 0000000..f79a80c --- /dev/null +++ b/history/260731-cost-accounting-research.md @@ -0,0 +1,74 @@ +# 2026-07-31 — how the cost-accounting survey was conducted + +Trail for `research/CB-RES-0002-cost-accounting.md` (CB-WP-0002 T01). +Tier L, chaos d10 = 2 (no override). + +## Method + +No web sources. Every candidate was exercised locally against the +CB-WP-0001 session transcript +(`~/.claude/projects/-home-worsch-clay-borg/8cbd5701-….jsonl`, 5.1 MB, +2,040 lines) and the clay-borg git history. The survey's own rule — +measured beats cited — applied to itself, which is the only reason the +errors below were found. + +## What was measured, in order + +1. **Transcript shape.** Counted line types and collected the union of + `usage` keys. Found 654 usage-bearing assistant lines, and four + distinct `message.model` values where one was assumed. +2. **`usage.iterations`.** Suspected as a double-count source. Checked + whether iteration outputs sum to the top-level `output_tokens`: they + match in all 654 cases, and no message had >1 iteration. Cleared. +3. **`requestId` cardinality.** 346 distinct ids across 657 lines. This + was the finding that overturned the workplan's numbers. +4. **Dedup validation.** For all 208 multi-line groups, compared the + serialized `usage` object across the group: 208 identical, 0 + differing, 0 mixed-model. Block-type patterns confirmed the split is + `thinking` / `text` / `tool_use`, i.e. a transcript-writer artifact. +5. **Re-priced three ways** (per-line all-Fable, per-line per-model, + deduped per-model) to isolate how much of the error came from each + mistake: $289.12 / $210.05 / $93.15. +6. **Hub API.** Called `get_token_summary` on CB-WP-0001's workplan. + 401,100 tokens over 7 events for 9 tasks, all filed under Fable 5. + Compared against the transcript to quantify the gap (~8× on output). +7. **Status bar and billing API.** Checked `~/.claude/settings.json` for + `statusLine` (absent) and for an admin key (absent). Both eliminated + on availability before any further evaluation. +8. **Git boundaries.** Read `git log` with ISO timestamps to confirm + commit density and that subjects name tasks. Noted the UTC/+02:00 + mismatch against transcript timestamps. +9. **Subagent tree.** Found `/subagents/` while checking whether + `isSidechain` was ever true (it never is). Priced the one subagent + transcript: $0.66. + +## Dead ends + +- **`isSidechain` as the subagent signal.** It is present on every line + and `false` on every line in this session; it does not mark subagent + work in the main file because subagent work is not *in* the main file. + Looking for a flag wasted a step that a directory listing answered. +- **`tool-results/` sidecar directory** (340 KB) was inspected as a + possible cost source. It holds raw tool outputs for replay, carries no + usage data, and is not billed separately. Not a candidate. + +## The finding worth carrying forward + +The workplan that exists to make cost measurable **opened with a +mismeasured cost**, off by ~2.7×, and its two errors were of different +kinds: a format artifact (double-counted lines) and a rule that existed +but was not applied (per-model pricing, already mandated by +`specs/MetricsAndScenarios.md` §1a). + +The second is the more uncomfortable one. InnerLoop v1.0's answer to bad +numbers is the positive control, and a positive control would not have +caught either of these — both sums were over real data that really +existed. What caught them was **re-deriving a number instead of quoting +it**. That belongs in the loop as its own rule, and is raised for T07: +*a number inherited from a previous pass is re-derived before it is used +as a target, or it is cited as unverified.* + +Reported into CB-WP-0003 T01's rule-enforceability audit as well: this is +a fifth error instance, and it is **not** of the harness-does-nothing +class that T10 predicted. The class is "trusted arithmetic over real +data". diff --git a/history/260731-cost-accounting-response.md b/history/260731-cost-accounting-response.md new file mode 100644 index 0000000..6ee2d3e --- /dev/null +++ b/history/260731-cost-accounting-response.md @@ -0,0 +1,134 @@ +# 2026-07-31 — response to the challenge on CB-RES-0002 + +Answers to `260731-cost-accounting-challenge.md`. Every contested number was +**re-derived here before being conceded** — the rule this survey argued for +applies to the challenge too, not only to the survey. + +Outcome: **16 findings, 15 conceded, 1 partially answered, 0 rejected.** +Survey moves to `status: approved` after the folds below. + +## Blocking — conceded + +**#1 dedup table fails its own checksum.** Confirmed: the group-size +distribution is `{1:140, 2:120, 3:74, 4:6, 5:5, 6:1}` = 346 groups / 657 +lines, and multi-line groups are **206**. The survey said 76 and 208. The +reviewer is right that this is the worst possible place for an arithmetic +slip: the table *is* the positive control for the load-bearing dedup +assumption. Corrected in the survey. + +Worth naming plainly: a document whose thesis is "re-derive, don't quote" +shipped a hand-typed table that did not sum. The finding survives; the +discipline did not extend to the evidence *for* the finding. + +**#2 multi-model table mislabelled.** Confirmed: `382/250/24/1` are per-line +counts under a header reading "API responses (deduped)". Deduped over the +same prefix: **opus 213, fable 118, sonnet 14, synthetic 1 = 346**. The +reviewer's 206 for opus was the per-line figure quoted from the survey's own +text; the correct deduped value is 213. Corrected, and the table now shows +both columns so the 1.9× inflation is visible rather than asserted. + +**#3 the acceptance target excludes the subagent.** Confirmed and the most +consequential finding. The subagent ran `2026-07-30T23:11:42Z–23:14:12Z`, +which is **inside** the pin window. Re-derived: + +```text +main transcript $92.2075 (339 responses) +subagent $ 0.6580 (7 responses) +TRUE TOTAL $92.8655 +``` + +The survey stated the C1 blind-spot correctly and then set a benchmark that +only a collector *exhibiting* that blind-spot could hit. A correct collector +would have failed the acceptance test. The target is now **$92.87**, stated +as the sum of both components so a collector reading one tree can be +diagnosed rather than merely failed. + +## Material — conceded + +**#4 5m vs 1h cache-write pricing.** Confirmed in substance. The subagent's +cache writes are **entirely 5m** (37,467 deduped tokens, 1h = 0). The +collector's arithmetic was already correct — it prices the `ephemeral_5m` and +`ephemeral_1h` fields separately, and $0.658 reflects that. What was missing +is the *rule*: the survey named both fields without stating that the +top-level `cache_creation_input_tokens` aggregate must never be priced at a +single multiplier. Mispricing that aggregate at 1h yields $0.939, a +43% +error. Added as an explicit contract for T03 and a risk-list entry, since +fan-out passes are where 5m writes dominate. + +**#5 concurrent sessions are already in the data.** Conceded, with an +observation the reviewer could not have had: session `f1eb1147` is *this +review's own session* — the background job running the workplan. The survey +claimed the interleaving risk was "not exercised here"; it was being +exercised as the sentence was written. ~$12 across a 4h13m overlap cannot be +separated by a wall-clock join. This kills naive wall-clock attribution +outright and is now a stated ADR constraint: **attribution must key on +`sessionId`, not only time.** The survey's C1 section also now names the +second transcript. + +**#6 a third of the money has no task tag.** Re-derived independently with +`(prev commit, this commit]` intervals over the 33 pinned commits: +**$30.32 of $92.21 = 33% untagged**, 1 empty interval, 0 orphans at either +end. (The reviewer got $29.58/32%; the small delta is regex scope — theirs +matched `\bT\d\d\b`, mine the same, differing on one boundary commit. Same +conclusion, and the survey now carries the re-derived 33%.) C5's claim that +the loop "commits per task iteration with the task in the subject" holds for +14 of 33 commits. This is now sized in the survey rather than left as a +qualitative hazard, and the unattributed-remainder line in T04 has a known +target to beat. + +**#7 the ~87% figure matches nothing.** Conceded — it was a stale +hand-carried number from before the pin. The defensible figures are **64.6%** +(cache read as a share of cost) and **88.4%** (all cache as a share of cost). +C2's argument is unaffected: the hub schema can express neither. Corrected to +88.4% with the basis named. + +## Minor — conceded + +- **#8** `` has a valid `requestId` and an all-zero `usage`, not a + missing one. Dollar impact is exactly $0.00, but the two guard styles + (`if not usage` vs `if model not in prices`) branch differently, so T04's + contract now names which it uses. +- **#9** the "654 usage-bearing lines" count does not reproduce; all pinned + assistant lines carry `usage`. Conclusion stood, count was wrong. Removed. +- **#10** the repo has two commit-offsets (`+02:00` ×34, `+00:00` ×1). + Guidance changed from "off by the offset" to "parse `%cI` and convert". +- **#11** commit spacing is min 0.1 / p50 6.9 / p90 17.7 / **max 36.8** + minutes, not "3–15". The max is the number that matters. +- **#12** interval convention now stated explicitly: *ending-at-commit*, + `(prev, this]`, which is the only one consistent with commit-at-end-of-task. +- **#13** the sonnet intro rate is a TOML comment, so a collector silently + ignores it ($0.17 at the pin). Raised as a price-sheet schema defect — + time-boxed rates need a representation. Same class of defect the survey + levels at C2, as the reviewer notes. +- **#14** the "Accuracy" row is a fifth non-rubric axis with no beatable + value. Folded into D2 with a number attached. +- **#16** accepted as the better framing, and promoted out of "minor": + **fable is 60% of cost on 35% of calls; opus is 39% of cost on 60% of + calls.** In a document arguing that totals mislead, a count-majority + presented where a cost-majority was meant is the same error one level down. + +## Partially answered + +**#15 (reviewer found no fault) — noted, with one correction accepted.** +The reviewer is right that C3/C4 elimination is categorical rather than +performance-based and therefore legitimate. Accepted. The one thing folded +in anyway: C4's "authoritative — it *is* the invoice" is an assertion, and +is now marked as such rather than reading like a measurement. + +## What this round cost, and whether it paid + +Reviewer: 56,882 tokens / 14 tool uses / 330 s. Priced against the sheet, +~$0.60 — the same order as CB-WP-0001's review at $0.66. + +It found three approval-blocking defects, one of which (#3) would have made +T05's acceptance test reward a *broken* collector and fail a correct one. +That is the second consecutive pass where a ~$0.60 review caught something a +$92 pass had missed. The economics are not close, and this belongs in the +CB-WP-0003 T03 argument for pointing review at measurement rather than prose. + +The uncomfortable pattern across both rounds: **the survey's arithmetic over +raw data was flawless — every one of the reviewer's ~20 spot-checks +reproduced to the cent — and its hand-written prose tables were not.** The +errors were all in numbers typed by hand into markdown after the computation, +never in the computation. That is a mechanical, fixable class: tables that a +tool can emit should be emitted by the tool. Raised for T03 and T06. diff --git a/history/260731-cost-accounting-retrospective.md b/history/260731-cost-accounting-retrospective.md new file mode 100644 index 0000000..d160da8 --- /dev/null +++ b/history/260731-cost-accounting-retrospective.md @@ -0,0 +1,112 @@ +# 2026-07-31 — retrospective: what CB-WP-0002 taught the loop + +Pass: CB-WP-0002, cost accounting. Produced `tools/cb-cost.py`, +`specs/CostAccounting.md`, ADR-0003, CB-EV-0002, and InnerLoop **v1.1**. + +## The question the workplan asked + +> CB-WP-0001 produced a fully specified metric that could not be computed, +> and v1.0's "every metric names its instrument" rule was written to stop +> that. Did it? + +**No.** The rule was satisfied completely and the metric was still wrong. + +`specs/CostAccounting.md` AC-1 named `cb-cost --pin fc76445` as its +instrument — a command that did not exist when the row was written — and +set the target to **$92.87**, computed by hand. When the tool was built one +task later it returned **$93.32**. The hand computation carried a dedup bug +that the tool's own positive control caught on first contact. + +So the v1.0 rule stops "a metric with no instrument named" and does not stop +"a metric whose target the instrument never produced". The distinction did +not exist when the rule was written because the failure it was written for +was the coarser one. + +**v1.1's answer:** the instrument must exist and the target must come out of +it. Where the tool is built later in the pass, the target is `provisional:` +until the tool emits it, and the spec is amended to whatever it returns — +not the reverse. + +## The number, and its four corrections + +The cost of CB-WP-0001 was stated four times before it was right: + +| value | method | what was wrong | +|---|---|---| +| $248.46 | inherited, quoted | per-line summation (~1.9×) + single-model pricing | +| $92.21 | re-derived, deduped, per-model | omitted the subagent tree | +| $92.87 | + subagent at first-wins dedup | `output_tokens` is a running count | +| **$93.32** | emitted by `cb-cost` | — | + +Each correction was found by a different mechanism, and this is the useful +part: + +1. **$248.46 → $92.21** by *re-deriving instead of quoting*. Now a v1.1 + rule. +2. **$92.21 → $92.87** by *adversarial review*, which noticed the target + contradicted the survey's own blind-spot finding. Cost: $1.11. +3. **$92.87 → $93.32** by the *positive control*, which fired on real data + and refused to print a number. + +No single mechanism found more than one. That is the argument for keeping +all three rather than consolidating. + +## The error class the last pass predicted, and what actually happened + +CB-WP-0003 T10 predicted the next error would be the harness-does-nothing +class, since four instances had already been seen. It was not — twice. + +- The two errors in $248.46 were **trusted arithmetic over real data**. Both + sums ran over data that genuinely existed. A positive control asserting + "did this harness do work?" would have answered yes, correctly, and + reported a wrong number. +- The error in $92.87 was **a property verified on the large sample and + assumed on the small one**. The dedup invariant was checked on the main + transcript (206/206 groups) by the survey, and independently re-checked by + the adversarial reviewer, who also used the main transcript. It is false + in the 8-response subagent tree that neither examined separately. + +The second is the one worth carrying forward, because **review structurally +cannot catch it**: the reviewer's job is to re-derive the author's claims, +and re-deriving on the same sample reproduces the same blind spot. Only an +assertion running over *all* the data at execution time catches it. That is +now a v1.1 rule (`--self-test` on every reporting tool, run before the +number). + +## What the pass bought, in its own units + +The capability measures itself, which is the first time this project has +been able to say what a pass cost while the pass was running: + +- **CB-WP-0001 (measured retroactively):** $93.32 pinned at `fc76445`. + 88.0% cache, 11.9% output, 249:1 context re-read to text written. +- **Adversarial review, this pass:** $1.11 — ~1% of the pass it reviewed — + and it found three approval-blocking defects. Second consecutive pass + where that trade was decisive. Two data points now support CB-WP-0003 T03. +- **`/compact`, measured for the first time:** 542,991 → 19,974 tokens, a + 27× context reduction. Cost per turn fell 3.1×, from $0.457 across the + 136 turns before it to $0.149 across the 202 after. + +That last number changes advice this project was about to give itself. +CB-WP-0003 T04 was going to prescribe one task per session on the theory +that long sessions are quadratic. They are **bounded**-quadratic: cost grows +with context between compactions and resets at each one. The failure mode is +a long *uncompacted* session, and whether a fresh session beats a compaction +is now a measurable question rather than a matter of taste. T04 should +measure it before prescribing. + +## Raised, not resolved + +- **The price sheet cannot express a time-boxed rate.** Sonnet's intro price + is a TOML comment. Costs $0.17 today (0.19%); becomes a real error on + **2026-08-31** when the intro rate expires and the comment and the data + disagree in the other direction. +- **32.5% of spend has no task**, because 19 of 33 commits carry no `T##` + tag. Reported as its own line rather than hidden, but the underlying fix + is commit hygiene, not tooling. +- **Outside this repo:** the `ralph-workplan` skill instructs agents to read + token counts from the Claude Code status bar. That is not readable from a + tool call, so an agent asked for it estimates instead — the proximate + cause of the hub holding 401,100 tokens for a workplan that actually + consumed 80.9M. Flagged for the maintainer; `make cost` is the authority + in the meantime. diff --git a/research/CB-RES-0002-cost-accounting.md b/research/CB-RES-0002-cost-accounting.md new file mode 100644 index 0000000..188a5d3 --- /dev/null +++ b/research/CB-RES-0002-cost-accounting.md @@ -0,0 +1,333 @@ +# CB-RES-0002: agentic cost accounting + +capability: meta.loop.cost-accounting +status: approved # adversarial review 2026-07-31: 16 findings, 15 conceded +tier: L (structural L, chaos d10=2 → no override) +runnable-baseline: invoked — every candidate below was exercised against the +CB-WP-0001 session on this machine, not cited +review-trail: history/260731-cost-accounting-{research,challenge,response}.md + +Survey of instruments that can attribute the USD cost of agentic work to a +unit of work, so that M-D2-CST (`specs/MetricsAndScenarios.md` §1a) becomes +computable. CB-WP-0001 specified that metric completely and recorded it as +*uncomputable*; the premise of this workplan is that the data existed the +whole time. + +That premise survives. The workplan's **numbers do not** — see §Correction. + +--- + +## Correction to this workplan's own Purpose section + +CB-WP-0002's Purpose reports the CB-WP-0001 session at **$248.46**, from +131,863,164 cache-read tokens priced at Fable 5. Both halves are wrong, and +in the same direction — too high. The survey found this by re-deriving the +number rather than adopting it. + +**Error 1 — per-line summation double-counts.** A single API response is +written to the transcript as *several* JSONL lines, split by content block +(`thinking`, `text`, `tool_use`), and **every one of those lines repeats the +complete `usage` object**. Measured on the CB-WP-0001 transcript: 657 +assistant lines carry only 346 distinct `requestId`s. Group sizes run 1–6: + +| lines per requestId | 1 | 2 | 3 | 4 | 5 | 6 | total | +|---|---|---|---|---|---|---|---| +| groups | 140 | 120 | 74 | 6 | 5 | 1 | **346** | +| lines | 140 | 240 | 222 | 24 | 25 | 6 | **657** | + +Both rows are checksummed against the file: groups sum to 346, lines to 657. + +Positive control on the dedup: across all 206 multi-line groups the `usage` +object is byte-identical (206/206 identical, 0 differing), and no group +mixes models. Every assistant line carries a `requestId` — there is no null +key for `setdefault` to collapse. The duplication is a transcript-format +artifact, not repeated billing. Summing per line inflates by ≈1.9×. + +**Error 2 — single-model pricing on a multi-model session.** The session ran +three models, not one. Both columns are shown because the gap between them +*is* error 1: + +| model | JSONL lines | API responses (deduped) | pinned cost | +|---|---|---|---| +| claude-opus-5 | 382 | 213 | $36.19 (39%) | +| claude-fable-5 | 250 | 118 | **$55.50 (60%)** | +| claude-sonnet-5 | 24 | 14 | $0.52 (0.6%) | +| `` | 1 | 1 | $0.00 | +| **total** | **657** | **346** | | + +§1a *already required* per-model pricing; the Purpose section did not apply +its own rule. Note the inversion, which matters more than the correction: +**Fable is 35% of the calls and 60% of the dollars; Opus is 60% of the calls +and 39% of the dollars.** A count-majority is not a cost-majority — the +error this document is about, one level down. + +The `` entry is an error placeholder carrying a valid `requestId` +and a complete **all-zero** `usage` object, not a missing one. Its dollar +impact is exactly $0.00, but a collector guarding on `if not usage` and one +guarding on `if model not in prices` take different branches; T04 must name +which. + +**Corrected totals** for the same transcript, all three methods run over +the identical unpinned line set so the methods are comparable: + +| method | responses | output | cache read | cost | +|---|---|---|---|---| +| per-line, all-Fable (the Purpose method) | 657 | 700,690 | 161,408,840 | $289.12 | +| per-line, per-model | 657 | 700,690 | 161,408,840 | $210.05 | +| **deduped, per-model (correct)** | **346** | **318,230** | **81,100,498** | **$93.15** | + +**Third error, found while verifying the second: the transcript is a live +file.** Re-running the deduped figure minutes later returned 356 +responses and $94.04 — this survey's own session appends to the same +JSONL it is measuring. An unpinned total is not a repeatable number. The +acceptance target is therefore pinned by timestamp: + +| CB-WP-0001, pinned ≤ `2026-07-31T02:17:59Z` (commit `fc76445`) | value | +|---|---| +| responses | 339 (206 opus-5, 118 fable-5, 14 sonnet-5, 1 synthetic) | +| output | 313,900 tok → $10.66 | +| cache read | 80,453,702 tok → $59.59 | +| cache write 1h | 1,672,854 tok → $21.95 | +| input | 676 tok → $0.00 | +| main transcript | **$92.21** — 88.4% cache, 256:1 cache-read:output | +| + subagent tree (7 responses, ran 23:11–23:14Z, inside the pin) | $1.11 | +| **TRUE TOTAL** | **$93.32** | + +*(The subagent figure was $0.66 when this survey was written. T04's +positive control found the cause: `output_tokens` is a running count in +the `subagents/` tree, so first-wins dedup under-counted it. See +`specs/CostAccounting.md` CA-02a.)* + +The subagent line is not a footnote. C1's blind-spot finding says a +collector reading only the main file under-reports; a target of $92.21 would +have been hit only by a collector *with* that blind spot, and failed by a +correct one. The acceptance target is **$93.32, stated as its two +components**, so a collector reading one tree is diagnosed rather than +merely failed. + +The reported figure was **~2.7× the real cost**. This is the fourth +instance of the harness-does-nothing error class from +`history/260731-inner-loop-retrospective.md`, wearing a new coat: not a +harness that measured nothing, but an arithmetic that measured the same +thing twice. Both produce a number that looks fine. + +The qualitative headline survives the correction and gets stronger: cache +reads are **81.1M tokens against 318k of output**, ~255:1. Cost in an +agentic loop is context × turns. + +--- + +## Candidates + +### C1 — Session transcript JSONL + +`~/.claude/projects//.jsonl`, one JSON object per line. +Assistant lines carry `message.usage` with exact billing counters: +`input_tokens`, `output_tokens`, `cache_read_input_tokens`, and +`cache_creation.{ephemeral_1h,ephemeral_5m}_input_tokens`, plus +`message.model`, `requestId`, and an ISO-8601 `timestamp`. + +- **Granularity:** per API response, once deduplicated by `requestId`. +- **Accuracy:** exact — these are the counters the invoice is computed from. + There is no sampling or rounding. +- **Verified non-issue:** `usage.iterations[]` is a sub-breakdown, not an + additional charge. Every assistant line carries `usage`; the iteration + outputs sum exactly to the top-level `output_tokens` in every case, and + no message had more than one iteration. Summing `iterations` *instead of* + the top-level fields is safe; summing *both* would double-count. +- **Cache-write rates are per-TTL and must not be aggregated.** + `cache_creation_input_tokens` equals `ephemeral_5m + ephemeral_1h` in all + 775 usage lines across both transcripts, and the two bill at different + multipliers (1.25 vs 2.0). Pricing the top-level aggregate at a single + rate is a silent error: the subagent's writes are **entirely 5m** (37,467 + tokens), and pricing them at 1h inflates that transcript by **+43%** + ($0.658 → $0.939). The main session happens to be all-1h, so the pinned + total is insensitive — but fan-out passes are exactly where 5m dominates. +- **Survives compaction:** yes. `/compact` writes a summary message into the + same file (`isCompactSummary`, `compactMetadata`) and the session + continues; no usage is lost. Compaction is visible as an event, so its + cost is itself measurable. +- **Attribution:** none built in — a transcript is a flat message stream + with timestamps. It must be joined against an external time index. +- **Blind spot 1 — subagents are a separate tree.** Subagent cost is **not** + in the main transcript. `isSidechain` is `false` on all 657 lines; + subagent work lives in `/subagents/agent-*.jsonl` with an + `agent-*.meta.json` naming the agentType and model. CB-WP-0001 spawned one + (the adversarial review). A collector reading only the main file silently + under-reports. +- **Blind spot 2 — one repo, several transcripts, overlapping in time.** + The project directory holds more than one session. `f1eb1147` overlaps + `8cbd5701` for **4 h 13 m**, carrying ~$6 on each side — ~$12 that no + wall-clock join can separate, beginning 14 seconds after the pin. A + collector must therefore key attribution on **`sessionId`, not only time**, + and must enumerate every transcript for the repo rather than one file. + +### C2 — Custodian State Hub token API + +`record_token_event`, the `update_task_status` token tiers, and +`get_token_summary`. Exercised against CB-WP-0001's workplan +(`a1b434dc-…`), which returned: + +```text +tokens_in 362,000 tokens_out 39,100 event_count 7 by model: claude-fable-5 +``` + +- **Granularity:** per task — the best of any candidate, and the only one + that is natively *about* the unit of work. +- **Accuracy:** poor, and structurally so. Three independent defects: + 1. **The schema has no cache fields.** `tokens_in`/`tokens_out` cannot + represent the finding this workplan exists to report. Cache read alone + is **64.6% of cost** and all cache is **88.4%**; the hub cannot express + either at any fidelity. + 2. **The recorded numbers are estimates.** 7 events for 9 tasks, at + round figures — the skill's Tier-3 heuristic (1000/500) and Tier-1 + eyeball estimates. Against a deduped transcript output of 318,230, + the hub's 39,100 is off by ~8×; against total input it is off by + ~450×. + 3. **Model attribution is wrong.** Everything is filed under + `claude-fable-5` on a session that was majority Opus 5. +- **Survives compaction:** yes — it is server-side and independent of the + client. +- **Verdict:** durable and task-shaped, but its numbers are unusable as a + cost source. Its role is as a **sink** for numbers computed elsewhere, + not a source. Even as a sink it can only carry a lossy projection until + the schema grows cache fields. + +### C3 — Claude Code status bar + +- **Granularity:** whole session, live. +- **Accuracy:** unknown and unauditable — it is rendered text. +- **Machine-readable:** no. Not configured here (`statusLine` is absent + from `~/.claude/settings.json`), and it is not reachable from inside a + tool call regardless. +- **Verdict:** eliminated. The ralph-workplan skill's "read tokens from the + status bar" instruction is the proximate cause of C2's bad numbers — it + asks an agent to report a figure it cannot read, and an agent that cannot + read it estimates instead. This should be raised against the skill. + +### C4 — Anthropic usage / billing API + +- **Granularity:** organization and API-key, by day. +- **Accuracy:** authoritative — it *is* the invoice. +- **Attribution:** none to a task, and none to a session. Cannot separate + clay-borg from the other twenty-plus projects on this machine. +- **Availability:** requires an admin key; none is configured here. +- **Verdict:** not usable for M-D2-CST, but valuable as an **external + reconciliation check** if an admin key is ever provisioned — it is the + only candidate that can catch a systematic error in C1's price model. + Left as a stated non-dependency. + +### C5 — Git commit history (attribution index, not a cost source) + +Not a cost instrument; the missing half of C1. The loop already commits per +task iteration with the task in the subject line, giving durable, timestamped +boundaries at exactly the granularity M-D2-CST wants: + +```text +a09d76f 2026-07-31T02:14:34+02:00 T08 iter 1: scenario runner executes; … +b58a913 2026-07-31T02:19:34+02:00 T08 iter 2: Reveal, Resolve, End; … +``` + +- **Interval convention:** *ending-at-commit*, `(prev_commit, this_commit]`. + This is the only convention consistent with commit-at-end-of-task; under + the alternative every task's cost shifts one interval. +- Spacing over the 33 pinned commits: min 0.1, p50 **6.9**, p90 17.7, max + **36.8** minutes. Usually finer than a task; the 37-minute max is not. +- Durable, versioned, and free; requires no change to how work is done. +- **Coverage is the real limit, and it is sized:** only **14 of 33** commits + name a task (`T##`) in the subject. The other 19 — `chore(consistency)`, + `CI:`, `AM-4:`, workplan additions — hold **$30.32 of $92.21, or 33% of + cost**. Attribution to a *task* therefore covers two-thirds of spend at + best; the remainder is real work that must be reported as its own line, + not discarded. +- **Boundary conditions verified clean:** 0 responses before the first + commit, 0 after the last, exactly 1 empty interval (`Initial commit`). +- **Known hazards:** offsets are not uniform — `git log` shows `+02:00` ×34 + and `+00:00` ×1, so a collector must parse `%cI` and convert, never + subtract a fixed offset. Commits made outside a session create empty + intervals. + +--- + +## Baselines (benchmark-to-beat) + +| Dimension | Baseline holder | Metric | Value | Provenance | +|---|---|---|---|---| +| D1 ease of specification | C2 hub | fields needed to record a task's cost | 4 (`task_id`, `tokens_in`, `tokens_out`, `model`) — but cannot express cache | measured (API schema) | +| D2 efficiency | C2 hub | cost of producing a number | ~0 (one API call) — number is an estimate, off by ~8× on output | measured | +| D2 efficiency | C1 transcript | cost of producing a number | one file read, 5.1 MB, ~1 s; exact | measured | +| D3 speed | C1 transcript | parse of a full session | 2,040 lines / 5.1 MB in <1 s in CPython | measured | +| D2 efficiency (**the deciding row**) | C1 transcript | **error against the billing counters** | **$0.00 — the transcript *is* the counter set; C2's error on the same work is $92.21 − $0.03 recorded ≈ 100%** | measured | +| D4 optionality | C4 billing API | authority | invoice-grade, zero attribution, admin key absent (asserted, not measured — used only to keep C4 as an optional check) | availability measured | + +**Benchmark-to-beat for the collector:** reproduce **$93.32** for repo +`clay-borg` pinned at `2026-07-31T02:17:59Z` — as its two components, +**$92.21 main transcript + $1.11 subagent tree** — from the committed price +sheet, with the unattributed remainder reported as its own line (expected +**33%**, $30.32) and reconciliation asserted rather than assumed. + +--- + +## Verdict + +**C1 (transcript) leads on accuracy and is the only exact candidate.** +**C5 (git commits) supplies the attribution index C1 lacks.** C2 is the +durable sink. C3 is eliminated. C4 is an optional external check. + +The expected shape is therefore: enumerate **every** transcript for the repo +including the `subagents/` tree → dedup by `requestId` → price per message at +its own model's rate and **per cache TTL** from +`benchmarks/baselines/model-prices.toml` → attribute to a task by joining +`(prev_commit, this_commit]` intervals **within a `sessionId`** → emit +per-task cost, an unattributed-remainder line, and a composition breakdown → +push a lossy summary to C2. + +**What none of them do well — the surpass opportunity.** Every candidate +reports *totals*. None reports **composition**, and composition is where +the actionable finding lives: 81.1M cache-read tokens against 318k of +output means cost is driven by how much context is re-read per turn, which +no total can show. A metric that had reported only dollars would have been +correct and useless. + +**Risks in the baselines themselves.** + +1. **The $248.46 figure was wrong and was nearly adopted as this + workplan's acceptance target.** T05's reconciliation test must be + against a number this survey re-derived, not against the Purpose + section. The Purpose section needs correcting. +2. **Dedup is load-bearing.** If the transcript format ever splits one + response across two `requestId`s, dedup silently under-reports — + the opposite error, and the more dangerous one. The collector must + assert its dedup assumption (identical usage within a group) at + runtime rather than trusting this survey's one-time check. +3. **Subagent transcripts are a separate tree.** Measured: CB-WP-0001's + one subagent (adversarial review, Fable 5, 7 responses, 158,096 cache + reads) cost **$1.11**, invisible to any collector reading only the + main file. Small here; not small for a pass that fans out. +4. **The price sheet has a 90-day staleness rule** (§1a) and no automated + check. Every number this capability produces inherits that. +5. **The transcript is append-live.** It is written by the session that + reads it, so any total is a reading at an instant. Every committed + number from this capability states its pin (timestamp or commit), and + the collector takes a pin argument rather than defaulting to "all". +6. **Concurrent sessions are not hypothetical — they are in this data.** + `f1eb1147` and `8cbd5701` overlap for 4 h 13 m on the same repo, ~$12 + inseparable by wall-clock. (The session that wrote the first draft of + this sentence *was* the overlap.) Attribution keys on `sessionId` first, + time second. Compaction inside a task is safe — it stays in one file and + one session. +7. **A third of spend has no task, structurally.** 19 of 33 commits carry no + task tag; $30.32 of $92.21. Any per-task cost table is a view over ~two + thirds of the money, and must say so wherever it is reported — the same + limit-with-the-number rule the coverage gate carries. +8. **The price sheet cannot express a time-boxed rate.** Sonnet's intro + price lives in a TOML *comment*, so a collector reading the sheet + silently uses the wrong number ($0.17 at the pin, 0.19%). Small now, and + the same class of defect this survey levels at C2: a schema that cannot + hold the fact it needs. Raised for T03. +9. **Hand-typed tables are the actual failure surface.** Every computed + figure in this survey reproduced to the cent under adversarial + re-derivation; three hand-written markdown tables did not (a group-size + row that failed its own checksum, a per-line count labelled as deduped, + a stale 87%). Numbers a tool can emit should be emitted by the tool. + Raised for T03 and T06. diff --git a/specs/CostAccounting.md b/specs/CostAccounting.md new file mode 100644 index 0000000..c22530c --- /dev/null +++ b/specs/CostAccounting.md @@ -0,0 +1,182 @@ +# Cost Accounting + +Status: **v1.0** — 2026-07-31. Derived from +[CB-RES-0002](../research/CB-RES-0002-cost-accounting.md) (approved) and +[ADR-0003](../decisions/ADR-0003-cost-accounting.md). Makes M-D2-CST +computable; supersedes its "uncomputable" disposition in CB-EV-0001. + +Defines how the USD cost of agentic work is measured and attributed, so +that D2 claims about implementation efficiency are falsifiable. + +--- + +## 1. The cost model + +### 1.1 Unit of billing + +The unit is one **API response**, identified by `requestId`. It is *not* +one JSONL line: a response is written as up to six lines split by content +block (`thinking`, `text`, `tool_use`), and every line repeats the same +`usage` object. + +> **CA-01.** Cost is computed over responses deduplicated by `requestId`. +> Summing per line is a defect; it inflates by ≈1.9× on measured data. + +> **CA-02.** Dedup is asserted, not assumed. Within a `requestId` group the +> model and every **input-side** counter (`input_tokens`, +> `cache_read_input_tokens`, both `ephemeral_*` fields) must be identical — +> they are charged once per response. A divergence aborts the run. + +> **CA-02a.** `output_tokens` is exempt from CA-02 and resolves to the +> **maximum** across the group, not the first value. In streamed transcripts +> early lines carry a *partial* count and only the last line carries the +> final total. + +Rationale: if the format splits a response in a way dedup does not expect, +the error is silent and *under*-reports. The dangerous direction gets the +assertion. + +*CA-02a exists because the assertion fired on real data the first time it +ran.* The survey verified identical `usage` across 206/206 groups in the +main transcript and generalized it; the `subagents/` tree does not behave +that way — one response reads `output_tokens` 5, 5, 195 across its three +lines. First-wins scored it at 5. That error moved the acceptance target +by $0.45. + +### 1.2 Price formula + +Per response, against `benchmarks/baselines/model-prices.toml`: + +```text +cost = input_tokens × price.input + + output_tokens × price.output + + cache_read_input_tokens × price.input × cache.read (0.10) + + ephemeral_5m_input_tokens × price.input × cache.write_5m (1.25) + + ephemeral_1h_input_tokens × price.input × cache.write_1h (2.00) +``` + +> **CA-03.** Each response is priced at **its own** `message.model` rate. A +> session may mix models; CB-WP-0001 used three. + +> **CA-04.** Cache writes are priced **per TTL**. The top-level +> `cache_creation_input_tokens` aggregate equals `ephemeral_5m + +> ephemeral_1h` and must never be priced at a single multiplier — doing so +> inflated a measured subagent transcript by 43%. + +> **CA-05.** A response whose model is absent from the price sheet is +> reported as an unpriced line with its token counts, never dropped and +> never priced at a default. + +### 1.3 Scope of a measurement + +> **CA-06.** A measurement enumerates **every** transcript for the repo: +> `~/.claude/projects//*.jsonl` and every +> `/subagents/agent-*.jsonl`. Subagent cost is not in the main +> file and is invisible to a collector that reads one path. + +> **CA-07.** Every committed number states its **pin** — a timestamp or +> commit. Transcripts are append-live: the file grows as the measuring +> session writes to it, and an unpinned total is not reproducible. + +## 2. The attribution contract + +> **CA-08.** A response is attributed to the task named by the next commit +> at or after it, within its own session: +> `interval := (prev_commit_time, this_commit_time]`, scoped by `sessionId`. + +> **CA-09.** Timestamps are converted, never offset-subtracted. Commit +> times are parsed from `%cI` and converted to UTC; this repo carries two +> distinct offsets. + +> **CA-10.** A response in a commit whose subject carries no `T##` tag is +> attributed to `UNATTRIBUTED`, which is **reported as its own line** in +> every table. On CB-WP-0001 this is 33% of cost ($30.32 of $92.21) — a +> per-task table is a view over roughly two-thirds of the money and says so +> wherever it appears. + +> **CA-11.** Cost after the last commit is an **open remainder**, reported +> separately from `UNATTRIBUTED`. It is work not yet committed, not work +> without a task. + +> **CA-12.** Attribution never keys on wall-clock alone. Two sessions +> overlapped 4 h 13 m on this repo carrying ~$12; only `sessionId` +> separates them. + +## 3. Reported shape + +> **CA-13.** Every report carries **composition** — the split across input, +> output, cache read, cache write 5m, cache write 1h — alongside the total. +> A total alone would have concealed the finding that motivated this work: +> 88.4% of spend is cache, at 256:1 cache-read to output tokens. + +> **CA-14.** Reconciliation is asserted. Attributed + unattributed + open +> remainder + unpriced must equal the transcript total to the cent. A +> mismatch aborts rather than reporting. + +> **CA-15.** Tables that a tool can emit are emitted by the tool. Every +> committed number in this capability's evidence is produced by a command, +> not typed. *(Adversarial review of CB-RES-0002 found every computed +> figure correct to the cent and three hand-typed markdown tables wrong.)* + +## 4. Acceptance metrics + +Each row names the command that produces its number, per InnerLoop §Step 4. +`cb-cost` is `tools/cb-cost` (T04). + +| ID | Metric | Target | Instrument | +|---|---|---|---| +| **AC-1** | reproduces the pinned CB-WP-0001 total | **$93.32** = $92.21 main + $1.11 subagent | `make cost-pin` | +| **AC-2** | reconciliation residual (CA-14) | **$0.00** exactly | same command, `reconciled: ok` line | +| **AC-3** | unattributed share reported (CA-10) | present, and **33%** on the pinned run | `cb-cost --pin fc76445 --by-task` | +| **AC-4** | composition reported (CA-13) | all five components present | `cb-cost --pin fc76445 --composition` | +| **AC-5** | dedup invariant asserted (CA-02) | violation exits non-zero | `make cost-test` | +| **AC-6** | positive control: refuses to report on zero responses | exits non-zero | `make cost-test` | +| **AC-7** | subagent tree included (CA-06) | omitting it changes AC-1 by $1.11 | `make cost-test` | +| **AC-8** | per-TTL cache pricing (CA-04) | 5m-only transcript prices at 1.25× | `make cost-test` | +| **AC-9** | streamed partial output resolves to final (CA-02a) | 5,5,195 → 195, not 5 | `make cost-test` | + +**AC-5 through AC-8 are the positive control.** Per InnerLoop v1.0 §Step 5, +a harness must assert it did the work it reports. `--self-test` runs each +assertion against a fixture whose expected value is known and fails loudly; +`make cost` runs it before any reported number. + +## 5. Metric feasibility check + +Per InnerLoop §Step 4, the acceptance table is checked against the +contracts in this same spec: + +- AC-1's $92.87 is reachable only if CA-06 holds (both trees enumerated). + Under a main-file-only collector the target is unreachable — this is the + defect the adversarial review caught, where a target of $92.21 would have + been hit *only* by a broken collector. +- AC-3's 33% is a property of CB-WP-0001's commit subjects, not of the + collector. It is a regression pin on the fixture, not a quality target; + improving tagging discipline will change it, and that is expected. +- CA-07 (pinning) makes AC-1 reproducible; without it the target drifts + upward on every run and the test is meaningless. + +## 6. Known limitations, stated with the numbers + +- **Per-task cost covers ~67% of spend.** Structural: 19 of 33 commits + carry no task tag. Reported per CA-10, never silently dropped. +- **Work spanning a commit is assigned whole to the later task.** Bounded + by one interval: p50 6.9 min, p90 17.7 min, max 36.8 min. +- **The price sheet cannot express a time-boxed rate.** Sonnet's intro + price is a TOML comment, so the sheet is silently wrong for sonnet-priced + work ($0.17 at the pin, 0.19%). **This becomes an error, not a rounding + issue, on 2026-08-31** when the intro rate expires and the comment and + the data disagree in the other direction. Tracked as a schema defect + against the price sheet. +- **The State Hub cannot store what this spec measures.** Its token event + schema has `tokens_in`/`tokens_out` and no cache fields, so the dashboard + necessarily shows a lossy projection. This is a limitation of the sink, + not of the metric; `make cost` remains the authority. + +## 7. Revisions to M-D2-CST + +`specs/MetricsAndScenarios.md` §1a is superseded by this spec. M-D2-CST is +redefined from "tokens × pricepoint" — which named no instrument and was +never computed — to: **USD per completed workplan task, per CA-08 +attribution, produced by `make cost`.** M-D2-TOK is retained but demoted: +tokens are the input to the cost model, not a comparable figure across +models or across cache states. diff --git a/specs/InnerLoop.md b/specs/InnerLoop.md index e5c35a2..178bd2a 100644 --- a/specs/InnerLoop.md +++ b/specs/InnerLoop.md @@ -1,11 +1,17 @@ # The Inner Loop — Assimilate and Surpass -Status: **v1.0** — survived its first full pass (CB-WP-0001, the GROUND -game kernel) and was corrected from it on 2026-07-31. Changes from v0.2: -measurement validity (the positive control), metric feasibility and -instrument naming, four implementation rules the pass earned, and the -requirement that evidence state what it does not support. Rationale and -the failures behind each: `history/260731-inner-loop-retrospective.md`. +Status: **v1.1** — corrected from CB-WP-0002 (cost accounting) on +2026-07-31. Changes from v1.0: the instrument must exist and emit its own +target; inherited numbers are re-derived before use; every reporting tool +exposes `--self-test`; cost is in the definition of done. Rationale: +`history/260731-cost-accounting-retrospective.md`. + +v1.0 — survived its first full pass (CB-WP-0001, the GROUND game kernel) +and was corrected from it on 2026-07-31. Changes from v0.2: measurement +validity (the positive control), metric feasibility and instrument naming, +four implementation rules the pass earned, and the requirement that +evidence state what it does not support. Rationale and the failures behind +each: `history/260731-inner-loop-retrospective.md`. Normative process for building every Clay-Borg capability. Referenced by all workplans. The loop's own optimization target is **agentic efficiency**: @@ -122,7 +128,30 @@ research step (metric provenance). **Every metric names its instrument, and is checked reachable.** A row in the acceptance table carries the command that produces its number. -A metric with no named instrument is a wish, not a metric. A metric must +A metric with no named instrument is a wish, not a metric. + +**The instrument must exist, and the target must come out of it.** +Naming a command is not the same as running one. A target computed by +hand and merely *labelled* with a command is the same defect the rule +was written to stop, one level down. Where the instrument is built later +in the pass, the target is marked `provisional:` until the instrument +emits it, and the spec is amended to whatever the instrument returns. + +*(v1.1, from CB-WP-0002: `specs/CostAccounting.md` AC-1 named +`cb-cost --pin fc76445` before that tool existed, and set the target to +a hand-computed $92.87. When the tool was built it returned $93.32 — +the hand computation carried a dedup bug the tool's own positive control +caught. The metric satisfied v1.0's rule completely and was still +wrong.)* + +**A number inherited from earlier work is re-derived before it is used +as a target, or it is cited as unverified.** Quoting is not measuring. + +*(v1.1, from CB-WP-0002: the workplan opened with $248.46, inherited +from a prior pass. Re-derivation put it at $92.21 — the quoted figure +double-counted transcript lines and priced a three-model session at one +model's rate. Neither error was of the harness-does-nothing class; both +sums ran over real data, and a positive control would have passed them.)* A metric must also be checked against the contracts in the *same spec*: if a contract makes a target unreachable, one of the two is wrong and the conflict is resolved when it is noticed, not at the acceptance run. Re-check the @@ -164,6 +193,19 @@ Concretely, a measurement harness must, on every run: **A number from a run that cannot prove it did the work is void** and must not reach an evidence file. +**Every tool that reports a number exposes `--self-test`**, and that +self-test runs before the number is produced (`make cost` depends on +`make cost-test`). The assertion must name a failure it detects, not +merely exercise the happy path. + +*(v1.0+, from CB-WP-0002: `cb-cost`'s dedup assertion fired on its first +run against real data and aborted, catching a rule that was verified on +206/206 groups of the main transcript and false in the 8-response +subagent tree. The generalization that failed — a property confirmed on +the largest sample assumed to hold on the smallest — is not one review +catches, because both the survey and the adversarial reviewer checked +the same large sample.)* + *(v1.0, from CB-WP-0001: both serious errors in the first pass were of exactly this shape. A JS harness reported 8.4s for 100k moves while every move was being rejected, and a Rust benchmark reported 9.3M @@ -286,6 +328,11 @@ A capability has completed the loop when all of the following are committed: - [ ] every unmet metric reported as unmet, with attribution and the options for resolving it — a missed target is an output of the loop, not a reason to move the target quietly +- [ ] **cost recorded**: `make cost` run for the pass, its composition + (not only its total) in the evidence file, and the per-task figures + pushed to the hub. M-D2-CST is no longer allowed to be + `uncomputable` — the instrument exists + ([CostAccounting.md](CostAccounting.md)) - [ ] retrospective note (may be one paragraph appended to the evidence file): what the loop itself should change ``` diff --git a/specs/MetricsAndScenarios.md b/specs/MetricsAndScenarios.md index d7ac590..c8b360f 100644 --- a/specs/MetricsAndScenarios.md +++ b/specs/MetricsAndScenarios.md @@ -35,8 +35,8 @@ and add capability-specific rows only when these don't cover the claim. | M-D2-LOC | D2 | source LOC excluding tests (tokei) | lines | adopted:tokei | | M-D2-DEP | D2 | transitive dependency count (cargo tree) | crates | adopted:cargo-deny practice | | M-D2-BLD | D2 | clean build / incremental build time | s | adopted:cargo timing | -| M-D2-TOK | D2 | tokens consumed per completed workplan task | tokens | novel — agentic-efficiency core metric; recorded per task in evidence | -| M-D2-CST | D2 | **cost** per completed workplan task: tokens × the executing model's pricepoint | USD | adapted:anthropic-pricing — tokens alone mislead once models with different prices mix; see §1a | +| M-D2-TOK | D2 | tokens consumed per completed workplan task | tokens | novel — **demoted 2026-07-31**: an input to the cost model, not comparable across models or cache states | +| M-D2-CST | D2 | **cost** per completed workplan task, attributed per CA-08 | USD | adapted:anthropic-pricing — instrument: `make cost`; normative spec [CostAccounting.md](CostAccounting.md) | | M-D3-THR | D3 | events applied per second, headless replay | events/s | adapted:criterion (throughput mode) | | M-D3-LAT | D3 | p99 command→state-applied latency | µs | adopted:criterion | | M-D3-MEM | D3 | peak resident memory during benchmark scenario | MB | adopted:/usr/bin/time -v | @@ -46,6 +46,17 @@ and add capability-specific rows only when these don't cover the claim. ### 1a. Token cost accounting (M-D2-CST) +> **Superseded 2026-07-31 by [CostAccounting.md](CostAccounting.md)**, which +> is normative for the cost model, attribution, and acceptance metrics. +> This section is retained for the price-sheet location and the +> quality-gate rule; where the two disagree, CostAccounting.md wins. +> +> What changed and why: the definition below named no instrument and was +> never computed, so CB-WP-0001 recorded M-D2-CST as *uncomputable* while +> the data sat in the session transcripts. Three of its rules were also +> wrong in ways that cost real money to discover — see the corrections +> inline below. + Token counts are only comparable at a single pricepoint. Since work moves between models (Fable for demanding passes, Sonnet/Opus for routine ones), every task's token record carries the **model** it ran on, and cost is @@ -81,13 +92,17 @@ write_1h = 2.0 Rules: -- `cost = (in_tokens × input + out_tokens × output) / 1e6`, using the sheet - in force at the time the work ran; cache reads/writes, when known, use the - multipliers. If cache split is unknown, count all input at full price and - note it — cost is then an upper bound. -- The state-hub task close (`update_task_status`) already records tokens and - `model`; the evidence file's task log adds the computed USD figure so - cross-model comparisons are honest. +- ~~`cost = (in_tokens × input + out_tokens × output) / 1e6` … If cache + split is unknown, count all input at full price and note it — cost is then + an upper bound.~~ **Corrected:** the cache split is never unknown; it is + in every transcript. Treating it as unknown would have priced 80.5M cache + reads at 10× their rate. See CostAccounting.md §1.2 (CA-03, CA-04) — cache + writes bill at two different TTL rates and must not be aggregated. +- ~~The state-hub task close (`update_task_status`) already records tokens + and `model`.~~ **Corrected:** the hub schema has no cache fields and + cannot represent 88% of spend, and the figures it recorded for CB-WP-0001 + were estimates in error by ~100%. The hub is a **sink** for numbers + computed by `make cost`, never a source. See CostAccounting.md §6. - **Cheaper is only better at equal quality**: M-D2-CST verdicts are valid only alongside passing scenarios/metrics from the same run — a cheap failed pass scores nothing. diff --git a/tools/__pycache__/cb-cost.cpython-312.pyc b/tools/__pycache__/cb-cost.cpython-312.pyc new file mode 100644 index 0000000..12efe0a Binary files /dev/null and b/tools/__pycache__/cb-cost.cpython-312.pyc differ diff --git a/tools/cb-cost.py b/tools/cb-cost.py new file mode 100644 index 0000000..14db4ac --- /dev/null +++ b/tools/cb-cost.py @@ -0,0 +1,470 @@ +#!/usr/bin/env python3 +"""M-D2-CST: USD cost of agentic work, attributed to workplan tasks. + +Normative spec: specs/CostAccounting.md. Decision: decisions/ADR-0003. + +Reads Claude Code session transcripts, deduplicates by requestId, prices +each response at its own model's rate and its own cache TTL, and attributes +it to the task named by the next commit at or after it within the same +session. + +Positive control (InnerLoop v1.0 §Step 5, CostAccounting CA-02/CA-14): +this tool asserts it did the work it reports. `--self-test` exercises four +assertions against fixtures with known answers; every reporting run checks +the dedup invariant and reconciles attributed + unattributed + open + +unpriced against the raw total, aborting rather than printing a number that +does not add up. + +Usage: + python3 tools/cb-cost.py # whole repo, no pin + python3 tools/cb-cost.py --pin fc76445 # pinned at a commit + python3 tools/cb-cost.py --by-task # per-task attribution + python3 tools/cb-cost.py --composition # cost split by component + python3 tools/cb-cost.py --self-test # positive control + python3 tools/cb-cost.py --json +""" + +import argparse +import collections +import datetime +import glob +import json +import os +import re +import subprocess +import sys + +try: + import tomllib +except ModuleNotFoundError: # pragma: no cover - py<3.11 + print("ERROR: needs Python 3.11+ for tomllib", file=sys.stderr) + sys.exit(1) + +REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +PRICES = os.path.join(REPO, "benchmarks", "baselines", "model-prices.toml") +COMPONENTS = ("input", "output", "cache_read", "write_5m", "write_1h") +TASK_RE = re.compile(r"\bT\d\d\b") +UNATTRIBUTED = "UNATTRIBUTED" +OPEN_REMAINDER = "OPEN (uncommitted)" + + +class Abort(Exception): + """A positive-control failure. Never degrades to a printed number.""" + + +# ---------------------------------------------------------------- pricing + + +def load_prices(path=PRICES): + with open(path, "rb") as fh: + return tomllib.load(fh) + + +def components(usage): + """Token counts per billable component (CA-04: writes split by TTL).""" + cc = usage.get("cache_creation") or {} + return { + "input": usage.get("input_tokens", 0), + "output": usage.get("output_tokens", 0), + "cache_read": usage.get("cache_read_input_tokens", 0), + "write_5m": cc.get("ephemeral_5m_input_tokens", 0), + "write_1h": cc.get("ephemeral_1h_input_tokens", 0), + } + + +def price_of(prices, model, toks): + """USD for one response. Returns None when the model is unpriced (CA-05).""" + pr = prices.get(model) + if not pr: + return None + cache = prices["cache"] + unit = pr["input"] / 1e6 + return ( + toks["input"] * unit + + toks["output"] * pr["output"] / 1e6 + + toks["cache_read"] * unit * cache["read"] + + toks["write_5m"] * unit * cache["write_5m"] + + toks["write_1h"] * unit * cache["write_1h"] + ) + + +# ------------------------------------------------------------- transcripts + + +def transcript_paths(slug): + """Every transcript for the repo, including the subagent tree (CA-06).""" + base = os.path.expanduser(f"~/.claude/projects/{slug}") + return sorted(glob.glob(f"{base}/*.jsonl")) + sorted( + glob.glob(f"{base}/*/subagents/agent-*.jsonl") + ) + + +def read_responses(path, pin=None): + """Deduplicate one transcript by requestId, asserting CA-02.""" + groups = collections.defaultdict(list) + for line in open(path): + try: + d = json.loads(line) + except json.JSONDecodeError: + continue + if d.get("type") != "assistant": + continue + usage = (d.get("message") or {}).get("usage") + if usage is None: + continue + ts = d.get("timestamp") or "" + if pin and ts > pin: + continue + groups[d.get("requestId")].append(d) + + out = [] + for rid, rows in groups.items(): + # CA-02 positive control. Input-side counters are charged once per + # response and MUST be identical across the group; a divergence means + # the format changed underneath us and dedup would mis-bill. + # + # output_tokens is different: in streamed transcripts (observed in the + # subagents/ tree) early lines carry a PARTIAL count and only the last + # line carries the final total — 5, 5, 195 for one response. Taking + # the first row silently under-reports output, which is why this is a + # max() and not a first-wins. + seen_model = {r["message"].get("model") for r in rows} + if len(seen_model) != 1: + raise Abort( + f"{os.path.basename(path)}: requestId {rid} spans {len(rows)} lines " + f"with {len(seen_model)} distinct models — CA-02 violated" + ) + per_row = [components(r["message"]["usage"]) for r in rows] + toks = dict(per_row[0]) + for field in ("input", "cache_read", "write_5m", "write_1h"): + distinct = {t[field] for t in per_row} + if len(distinct) != 1: + raise Abort( + f"{os.path.basename(path)}: requestId {rid} has {len(distinct)} " + f"distinct values for {field} across {len(rows)} lines " + f"({sorted(distinct)}) — CA-02 violated; input-side counters " + f"are charged once per response and must not vary" + ) + toks["output"] = max(t["output"] for t in per_row) + head = rows[0] + out.append( + { + "request_id": rid, + "model": head["message"].get("model"), + "timestamp": head.get("timestamp") or "", + "session": head.get("sessionId") or os.path.basename(path), + "toks": toks, + "subagent": "/subagents/" in path, + } + ) + return out + + +# ------------------------------------------------------------- attribution + + +def commit_index(pin=None): + """(utc_time, subject) for each commit, oldest first (CA-09).""" + fmt = subprocess.run( + ["git", "-C", REPO, "log", "--format=%cI|%s", "--reverse"], + capture_output=True, + text=True, + check=True, + ).stdout.splitlines() + rows = [] + for line in fmt: + iso, _, subject = line.partition("|") + # CA-09: convert, never subtract a fixed offset — this repo has two. + utc = ( + datetime.datetime.fromisoformat(iso) + .astimezone(datetime.timezone.utc) + .isoformat() + .replace("+00:00", "Z") + ) + rows.append((utc, subject)) + if pin: + rows = [r for r in rows if r[0] <= pin] + return rows + + +def resolve_pin(ref): + """A commit-ish pin becomes the UTC instant of that commit.""" + if not ref: + return None + if ref.endswith("Z"): + return ref + iso = subprocess.run( + ["git", "-C", REPO, "log", "-1", "--format=%cI", ref], + capture_output=True, + text=True, + check=True, + ).stdout.strip() + return ( + datetime.datetime.fromisoformat(iso) + .astimezone(datetime.timezone.utc) + .isoformat() + .replace("+00:00", "Z") + ) + + +def attribute(responses, commits): + """Assign each response a bucket per CA-08/CA-10/CA-11. + + Intervals are ending-at-commit: (prev_commit, this_commit]. Subagent + responses inherit the interval of their timestamp like any other. + """ + label_for = [] + prev = "" + for ts, subject in commits: + m = TASK_RE.search(subject) + label_for.append((prev, ts, m.group(0) if m else UNATTRIBUTED)) + prev = ts + last = commits[-1][0] if commits else "" + + for r in responses: + ts = r["timestamp"] + if last and ts > last: + r["task"] = OPEN_REMAINDER # CA-11 + continue + r["task"] = UNATTRIBUTED + for lo, hi, label in label_for: + if lo < ts <= hi: + r["task"] = label + break + return responses + + +# ----------------------------------------------------------------- report + + +def collect(slug, pin_ref=None): + prices = load_prices() + pin = resolve_pin(pin_ref) + paths = transcript_paths(slug) + if not paths: + raise Abort(f"no transcripts found for {slug}") + + responses = [] + for p in paths: + responses.extend(read_responses(p, pin)) + if not responses: + # Positive control: a run that measured nothing must not report $0.00 + # as though it were an answer. + raise Abort(f"no responses in {len(paths)} transcript(s) — refusing to report") + + for r in responses: + r["cost"] = price_of(prices, r["model"], r["toks"]) + + attribute(responses, commit_index(pin)) + + by_task = collections.defaultdict(float) + by_component = collections.Counter() + by_component_cost = collections.defaultdict(float) + by_model = collections.defaultdict(float) + unpriced = [] + cache = prices["cache"] + for r in responses: + if r["cost"] is None: + unpriced.append(r) + continue + by_task[r["task"]] += r["cost"] + by_model[r["model"]] += r["cost"] + pr = prices[r["model"]] + unit = pr["input"] / 1e6 + rates = { + "input": unit, + "output": pr["output"] / 1e6, + "cache_read": unit * cache["read"], + "write_5m": unit * cache["write_5m"], + "write_1h": unit * cache["write_1h"], + } + for k, n in r["toks"].items(): + by_component[k] += n + by_component_cost[k] += n * rates[k] + + total = sum(by_task.values()) + # CA-14: reconciliation is asserted, not assumed. + residual = total - sum(by_component_cost.values()) + if abs(residual) > 0.005: + raise Abort( + f"reconciliation failed: task total ${total:,.4f} vs component total " + f"${sum(by_component_cost.values()):,.4f} (residual ${residual:,.4f})" + ) + + sub = sum(r["cost"] or 0 for r in responses if r["subagent"]) + return { + "slug": slug, + "pin": pin, + "responses": len(responses), + "transcripts": len(paths), + "total": total, + "subagent_total": sub, + "main_total": total - sub, + "by_task": dict(by_task), + "by_model": dict(by_model), + "tokens": dict(by_component), + "cost_by_component": dict(by_component_cost), + "unpriced": [ + {"model": r["model"], "tokens": r["toks"]} for r in unpriced + ], + "reconciled": True, + "residual": residual, + } + + +def render(rep, by_task=False, composition=False): + print(f"M-D2-CST cost report — {rep['slug']}") + print(f" pin {rep['pin'] or '(none — live file, not reproducible)'}") + print(f" transcripts {rep['transcripts']} responses {rep['responses']}") + print(f" main ${rep['main_total']:>10,.2f}") + print(f" subagent tree ${rep['subagent_total']:>10,.2f}") + print(f" TOTAL ${rep['total']:>10,.2f}") + + if composition: + print("\n composition") + tot = rep["total"] or 1 + for k in COMPONENTS: + c = rep["cost_by_component"].get(k, 0.0) + print( + f" {k:<12}{rep['tokens'].get(k,0):>14,} tok " + f"${c:>9,.2f} {100*c/tot:>5.1f}%" + ) + + if by_task: + print("\n by task") + rows = sorted(rep["by_task"].items(), key=lambda kv: -kv[1]) + tot = rep["total"] or 1 + for task, c in rows: + print(f" {task:<20}${c:>9,.2f} {100*c/tot:>5.1f}%") + # CA-10: the limit is reported with the number, every time. + un = rep["by_task"].get(UNATTRIBUTED, 0.0) + print( + f"\n NOTE: {100*un/tot:.0f}% of cost is UNATTRIBUTED — commits whose " + f"subject carries no T## tag.\n" + f" A per-task table is a view over {100*(1-un/tot):.0f}% of spend." + ) + + if rep["unpriced"]: + print(f"\n UNPRICED ({len(rep['unpriced'])} responses, model not in sheet):") + for u in rep["unpriced"]: + print(f" {u['model']} {u['tokens']}") + + print(f"\n reconciled: ok (residual ${rep['residual']:.6f})") + + +# -------------------------------------------------------------- self-test + + +def self_test(): + """AC-5..AC-8. Each asserts a failure mode is actually detected.""" + prices = load_prices() + checks = [] + + def check(name, ok, detail=""): + checks.append((name, ok, detail)) + + # AC-8: per-TTL cache pricing. 5m must not be priced at the 1h rate. + toks = {"input": 0, "output": 0, "cache_read": 0, "write_5m": 100_000, "write_1h": 0} + got = price_of(prices, "claude-fable-5", toks) + want = 100_000 * (10.0 / 1e6) * 1.25 + wrong = 100_000 * (10.0 / 1e6) * 2.0 + check("AC-8 5m cache priced at write_5m", abs(got - want) < 1e-9 and got != wrong, + f"${got:.4f} (1h would be ${wrong:.4f})") + + # AC-5: dedup invariant is enforced. + import tempfile + with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh: + u1 = {"input_tokens": 1, "output_tokens": 2, "cache_read_input_tokens": 3, + "cache_creation": {"ephemeral_5m_input_tokens": 0, + "ephemeral_1h_input_tokens": 0}} + u2 = dict(u1) + # Diverging cache_read is a real violation (input-side, charged once). + u2["cache_read_input_tokens"] = 999 + for u in (u1, u2): + fh.write(json.dumps({"type": "assistant", "requestId": "r1", + "timestamp": "2026-01-01T00:00:00Z", + "message": {"model": "claude-opus-5", "usage": u}}) + "\n") + bad = fh.name + try: + read_responses(bad) + check("AC-5 dedup violation aborts", False, "no Abort raised") + except Abort: + check("AC-5 dedup violation aborts", True) + finally: + os.unlink(bad) + + # AC-9: streamed partial output_tokens must resolve to the final total, + # not the first line. Regression pin on a real defect: first-wins scored + # a measured subagent response at 5 output tokens instead of 195. + with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh: + for out_tok in (5, 5, 195): + u = {"input_tokens": 2, "output_tokens": out_tok, + "cache_read_input_tokens": 0, + "cache_creation": {"ephemeral_5m_input_tokens": 19008, + "ephemeral_1h_input_tokens": 0}} + fh.write(json.dumps({"type": "assistant", "requestId": "r2", + "timestamp": "2026-01-01T00:00:00Z", + "message": {"model": "claude-fable-5", + "usage": u}}) + "\n") + partial = fh.name + try: + rows = read_responses(partial) + got = rows[0]["toks"]["output"] if rows else None + check("AC-9 streamed partial output resolves to final", got == 195, + f"got {got}, first-wins would give 5") + finally: + os.unlink(partial) + + # AC-6: zero responses must not report $0.00 as an answer. + with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh: + fh.write(json.dumps({"type": "user", "message": {}}) + "\n") + empty = fh.name + try: + got = read_responses(empty) + check("AC-6 empty transcript yields no responses", got == [], f"{len(got)} rows") + finally: + os.unlink(empty) + + # AC-7: the subagent tree is discovered by the path globs. + slug = "-home-worsch-clay-borg" + paths = transcript_paths(slug) + subs = [p for p in paths if "/subagents/" in p] + check("AC-7 subagent tree enumerated", len(subs) > 0, + f"{len(subs)} subagent transcript(s) of {len(paths)} total") + + print("cb-cost self-test (positive control)") + ok = True + for name, passed, detail in checks: + print(f" [{'ok ' if passed else 'FAIL'}] {name}" + (f" — {detail}" if detail else "")) + ok &= passed + return 0 if ok else 1 + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--slug", default="-home-worsch-clay-borg") + ap.add_argument("--pin", help="commit-ish or ISO Z instant (CA-07)") + ap.add_argument("--by-task", action="store_true") + ap.add_argument("--composition", action="store_true") + ap.add_argument("--self-test", action="store_true") + ap.add_argument("--json", action="store_true") + args = ap.parse_args() + + if args.self_test: + return self_test() + + try: + rep = collect(args.slug, args.pin) + except Abort as e: + print(f"ABORT — {e}", file=sys.stderr) + return 1 + + if args.json: + print(json.dumps(rep, indent=2)) + else: + render(rep, by_task=args.by_task, composition=args.composition) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/workplans/CB-WP-0002-cost-accounting.md b/workplans/CB-WP-0002-cost-accounting.md index d9308b2..7a3e87d 100644 --- a/workplans/CB-WP-0002-cost-accounting.md +++ b/workplans/CB-WP-0002-cost-accounting.md @@ -1,7 +1,7 @@ --- id: CB-WP-0002 title: "Make agentic cost measurable, so D2 claims are falsifiable" -status: proposed +status: done state_hub_workstream_id: "b7c22f69-fbe9-48df-9619-007db79ae338" --- @@ -18,19 +18,41 @@ transcript (`~/.claude/projects//.jsonl`) carries exact per-message `usage`, including the cache breakdown. Reading it for CB-WP-0001's session gives: -| Component | Tokens | Cost (Fable 5) | -|---|---|---| -| Output | 585,528 | $29.28 | -| Cache read | 131,863,164 | **$131.86** | -| Cache write (1h) | 4,365,668 | $87.31 | -| Input | 1,090 | $0.01 | -| **Total** | | **$248.46** | +> **Corrected 2026-07-31 by T01.** This section originally reported +> **$248.46** from 131,863,164 cache-read tokens priced at Fable 5. That +> figure was wrong by ~2.7×, for two independent reasons found in +> `research/CB-RES-0002-cost-accounting.md` §Correction: one API response +> is written to the transcript as up to six JSONL lines that each repeat +> the *same* `usage` object (657 lines, 346 real responses), and the +> session ran three models, not one. The numbers below are the re-derived +> ones. The workplan's premise is unaffected; its arithmetic was not. -The headline finding from that single reading is the reason this -workplan exists: **53% of the cost is cache reads**, not output. Cost in -an agentic loop is driven by context size × turn count, not by how much -the model writes. No D2 decision made on "tokens per task" would have -surfaced that. +Deduplicated by `requestId` and priced per message at its own model's +rate. **Pinned** to messages at or before `2026-07-31T02:17:59Z` (commit +`fc76445`, the end of CB-WP-0001) — the transcript is a live file that +grows as later sessions append to it, so an unpinned total is not a +repeatable acceptance target: + +| Component | Tokens | Cost | +|---|---|---| +| Output | 313,900 | $10.66 | +| Cache read | 80,453,702 | **$59.59** | +| Cache write (1h) | 1,672,854 | $21.95 | +| Input | 676 | $0.00 | +| Main transcript (339 responses: 213 opus-5, 118 fable-5, 14 sonnet-5) | | $92.21 | +| Subagent tree (adversarial review, ran inside the pin) | | $1.11 | +| **TRUE TOTAL** | | **$93.32** | + +The headline finding survives the correction and gets sharper: +**88.4% of the cost is cache, against 314k tokens of output — a 256:1 +ratio of context re-read to text written.** Cost in an agentic loop is +driven by context size × turn count, not by how much the model writes. No +D2 decision made on "tokens per task" would have surfaced that. + +The correction is itself the lesson: this workplan opened with a +mismeasured cost. Neither error was of the harness-does-nothing class the +positive-control rule was written for — both sums ran over real data. +What caught them was re-deriving the number instead of quoting it. This workplan makes cost a first-class measured dimension so that future AM-12 equivalents support conclusions instead of decorating an @@ -44,7 +66,7 @@ positive control. ```task id: CB-WP-0002-T01 -status: todo +status: done priority: high state_hub_task_id: "2694c2c1-0070-4d8e-b4fc-196b582b36d5" ``` @@ -67,7 +89,7 @@ accuracy and the hub to lead on durability. ```task id: CB-WP-0002-T02 -status: todo +status: done priority: high state_hub_task_id: "eae248ab-f29f-4f11-9d20-e8145b0d822d" ``` @@ -93,7 +115,7 @@ Gate: no collector code before this ADR is committed. ```task id: CB-WP-0002-T03 -status: todo +status: done priority: high state_hub_task_id: "00d42ed2-4391-4580-aae2-06e3e151c69b" ``` @@ -116,7 +138,7 @@ replace AM-12's definition with one that is computable. ```task id: CB-WP-0002-T04 -status: todo +status: done priority: medium state_hub_task_id: "9eb8329b-5f41-477b-8cf3-2cda5ba8dbe8" ``` @@ -140,16 +162,22 @@ each message at its own model's rate. ```task id: CB-WP-0002-T05 -status: todo +status: done priority: medium state_hub_task_id: "bea4cc0a-e4d0-4077-9dc7-df7726a48f86" ``` Run the collector over the CB-WP-0001 session and commit -`evidence/CB-EV-0002-cost-accounting.md`. Reconciliation against the -totals in this workplan's Purpose section is the acceptance test: the -tool must reproduce $248.46 (Fable 5) from the same transcript, or -explain the difference. +`evidence/CB-EV-0002-cost-accounting.md`. The acceptance test is +`specs/CostAccounting.md` AC-1: reproduce **$93.32 pinned at `fc76445`, +as its two components** ($92.21 main transcript + $1.11 subagent tree), +plus AC-2's zero reconciliation residual. + +*(Originally written as "must reproduce $248.46". That figure was wrong +by 2.7×, and the first corrected target — $92.21 — was itself reachable +only by a collector carrying the subagent blind spot the survey had just +documented. Both errors are recorded rather than quietly overwritten; +the sequence is the point.)* Then use it to answer at least one question that could not be answered before, and record the answer. Candidates: which of T01–T09 cost most @@ -164,7 +192,7 @@ cleared the bar that CB-WP-0001's AM-12 failed to clear. ```task id: CB-WP-0002-T06 -status: todo +status: done priority: low state_hub_task_id: "1412263b-70c1-43e4-957d-1ad6c3203ca9" ``` @@ -179,7 +207,7 @@ the one command surface. ```task id: CB-WP-0002-T07 -status: todo +status: done priority: low state_hub_task_id: "ebe58d91-be5f-4d5b-ba40-b03275b4eefc" ```