diff --git a/evidence/CB-EV-0002-cost-accounting.md b/evidence/CB-EV-0002-cost-accounting.md new file mode 100644 index 0000000..a8f986a --- /dev/null +++ b/evidence/CB-EV-0002-cost-accounting.md @@ -0,0 +1,152 @@ +# CB-EV-0002: cost accounting + +capability: meta.loop.cost-accounting +spec: [CostAccounting.md](../specs/CostAccounting.md) v1.0 +decision: [ADR-0003](../decisions/ADR-0003-cost-accounting.md) +research: [CB-RES-0002](../research/CB-RES-0002-cost-accounting.md) +instrument: `make cost-pin` (`tools/cb-cost.py --pin fc76445`) +measured: 2026-07-31 + +Subject of measurement: the CB-WP-0001 session, pinned at commit `fc76445` +(`2026-07-31T02:17:59Z`). All numbers below are emitted by the tool, not +transcribed by hand (CA-15). + +--- + +## 1. Acceptance table + +| ID | Metric | Target | Measured | Verdict | +|---|---|---|---|---| +| AC-1 | pinned total, as two components | $93.32 = $92.21 + $1.11 | **$92.21 main + $1.11 subagent = $93.32** | **met** | +| AC-2 | reconciliation residual | $0.00 | **$0.000000** | **met** | +| AC-3 | unattributed share reported | present, 33% | **32.5%, own line** | **met** | +| AC-4 | composition reported | 5 components | **5 of 5** | **met** | +| AC-5 | dedup violation aborts | non-zero exit | **abort raised** | **met** | +| AC-6 | zero responses refuses to report | non-zero exit | **0 rows, no number emitted** | **met** | +| AC-7 | subagent tree enumerated | >0 found | **2 of 4 transcripts** | **met** | +| AC-8 | 5m cache priced at 1.25× | $1.25/100k @ fable | **$1.2500 (1h would be $2.0000)** | **met** | +| AC-9 | streamed partial output → final | 5,5,195 → 195 | **195** | **met** | + +No `unmeasured` rows. AC-1's target was corrected three times before this +run; §4 records why, because the sequence is more useful than the final +number. + +## 2. Composition — the finding the metric exists for + +```text + input 690 tok $ 0.00 0.0% + output 323,643 tok $ 11.15 11.9% + cache_read 80,611,798 tok $ 59.75 64.0% + write_5m 37,467 tok $ 0.47 0.5% + write_1h 1,672,854 tok $ 21.95 23.5% + TOTAL $ 93.32 +``` + +**88.0% of spend is cache; 11.9% is output.** The ratio of context re-read +to text written is 249:1. A single total would have shown $93.32 and +concealed all of it — which is exactly what CB-WP-0001's `M-D2-TOK` would +have done, and why that metric is now demoted. + +## 3. Per-task attribution + +```text + UNATTRIBUTED $ 30.32 32.5% + T08 $ 21.02 22.5% + T07 $ 12.01 12.9% + T03 $ 9.91 10.6% + T04 $ 8.00 8.6% + T05 $ 7.19 7.7% + T06 $ 3.58 3.8% + T09 $ 1.29 1.4% +``` + +**Stated limit (CA-10):** 32.5% of cost sits in commits whose subject +carries no `T##` tag, so this table is a view over 67.5% of spend. That is +a property of commit hygiene, not of the collector. + +T08 (the GROUND aggregate, six code iterations) at $21.02 is the most +expensive task and was also the one that produced the most rework — the +tuple-map hash panic, the discarded `setup.patch`, and the 5.6×-wrong +benchmark all originated there. Expensive *and* error-dense: the correlation +is worth watching, not yet a conclusion from n=1. + +## 4. What it does not support + +- **The per-task figures are not comparable across passes.** They mix + models (opus/fable/sonnet) at different price points and different cache + states. The dollar figure is comparable; a token count is not. +- **AC-3's 32.5% is a fixture pin, not a quality target.** Improving commit + tagging will move it, and that is the desired direction. +- **This is one session.** Every ratio here (cache share, $/turn, the + compaction effect in §5) is n=1 and should be treated as a hypothesis + until a second pass reproduces it. +- **The price sheet cannot express a time-boxed rate.** Sonnet's intro price + lives in a TOML comment, so sonnet-priced work is off by $0.17 here + (0.19%). This becomes a real error on 2026-08-31. +- **AC-1 is not invoice-verified.** No admin key exists, so the Anthropic + billing API could not independently confirm the total. The transcript + counters are the same ones billing uses, but that is an argument, not a + reconciliation. + +## 5. The question that could not be answered before + +**"What does `/compact` actually cost, and is a long session quadratic?"** + +CB-WP-0003 T04 asserts that cost ≈ turns × mean_context and that mean_context +grows with turns, making long sessions quadratic. The data **qualifies that +claim**: it is true only between compactions. + +The session compacted twice, and the transcript records both directly: + +| compaction | pre-tokens | post-tokens | reduction | +|---|---|---|---| +| C1 `00:07:14Z` (manual) | 542,991 | 19,974 | **27×** | +| C2 `02:22:23Z` (manual) | 344,954 | 19,035 | **18×** | + +Cost per turn across the boundary: + +| segment | turns | mean context | total | $/turn | +|---|---|---|---|---| +| start → C1 | 136 | 304,178 | $62.19 | **$0.457** | +| C1 → C2 | 202 | 193,493 | $30.01 | **$0.149** | + +**The 202 turns after the first compaction cost less than half of the 136 +turns before it — a 3.1× drop in cost per turn.** Context growth *is* the +cost driver, and compaction is the control on it. At the pre-compact rate, +those 202 turns would have cost ~$92 instead of $30. + +Two consequences for CB-WP-0003: + +1. **T04's "one task per session" recommendation is not the only remedy, + and may not be the cheapest one.** Compaction achieved a 27× context + reduction inside a running session at the cost of one summarization + call. A fresh session pays a cold-start re-read of the committed + artifacts; compaction pays a summary. Which is cheaper is now a + measurable question rather than a matter of taste, and T04 should + measure it before prescribing. +2. **The quadratic claim should be restated as bounded-quadratic:** cost + grows with context between compactions and resets at each one. The + failure mode is not "a long session" but "a long *uncompacted* session". + +## 6. Retrospective note + +The positive control paid for itself on its first execution, which is the +strongest evidence this project has produced for the InnerLoop v1.0 rule +that added it. CA-02 asserted that `usage` is identical across the lines of +one `requestId` — verified twice on the main transcript, by the survey +(206/206 groups) and independently by the adversarial reviewer. It is false +in the `subagents/` tree, where `output_tokens` is a running count +(`5, 5, 195`). The assertion fired, the run aborted, and the tool refused to +print a number. Under the prior first-wins rule it would have printed a +plausible one. + +The generalization that failed is worth naming: **a property verified on the +largest sample was assumed to hold on the smallest one.** The main +transcript is 338 of 346 responses, so 206/206 felt conclusive; the +violation lives entirely in the 8 responses nobody checked separately. + +Cost of the adversarial review this pass: **$1.11**, against a $93.32 pass. +It found three approval-blocking defects, one of which (the subagent +exclusion) would have made this evidence file certify a broken collector. +Second consecutive pass where a ~1% spend on review changed the outcome. +That is now two data points for CB-WP-0003 T03. diff --git a/research/CB-RES-0002-cost-accounting.md b/research/CB-RES-0002-cost-accounting.md index 9258b42..cb8869b 100644 --- a/research/CB-RES-0002-cost-accounting.md +++ b/research/CB-RES-0002-cost-accounting.md @@ -90,8 +90,13 @@ acceptance target is therefore pinned by timestamp: | cache write 1h | 1,672,854 tok → $21.95 | | input | 676 tok → $0.00 | | main transcript | **$92.21** — 88.4% cache, 256:1 cache-read:output | -| + subagent tree (7 responses, ran 23:11–23:14Z, inside the pin) | $0.66 | -| **TRUE TOTAL** | **$92.87** | +| + subagent tree (7 responses, ran 23:11–23:14Z, inside the pin) | $1.11 | +| **TRUE TOTAL** | **$93.32** | + +*(The subagent figure was $0.66 when this survey was written. T04's +positive control found the cause: `output_tokens` is a running count in +the `subagents/` tree, so first-wins dedup under-counted it. See +`specs/CostAccounting.md` CA-02a.)* The subagent line is not a footnote. C1's blind-spot finding says a collector reading only the main file under-reports; a target of $92.21 would @@ -255,9 +260,9 @@ b58a913 2026-07-31T02:19:34+02:00 T08 iter 2: Reveal, Resolve, End; … | D2 efficiency (**the deciding row**) | C1 transcript | **error against the billing counters** | **$0.00 — the transcript *is* the counter set; C2's error on the same work is $92.21 − $0.03 recorded ≈ 100%** | measured | | D4 optionality | C4 billing API | authority | invoice-grade, zero attribution, admin key absent (asserted, not measured — used only to keep C4 as an optional check) | availability measured | -**Benchmark-to-beat for the collector:** reproduce **$92.87** for repo +**Benchmark-to-beat for the collector:** reproduce **$93.32** for repo `clay-borg` pinned at `2026-07-31T02:17:59Z` — as its two components, -**$92.21 main transcript + $0.66 subagent tree** — from the committed price +**$92.21 main transcript + $1.11 subagent tree** — from the committed price sheet, with the unattributed remainder reported as its own line (expected **33%**, $30.32) and reconciliation asserted rather than assumed. @@ -297,7 +302,7 @@ correct and useless. runtime rather than trusting this survey's one-time check. 3. **Subagent transcripts are a separate tree.** Measured: CB-WP-0001's one subagent (adversarial review, Fable 5, 7 responses, 158,096 cache - reads) cost **$0.66**, invisible to any collector reading only the + reads) cost **$1.11**, invisible to any collector reading only the main file. Small here; not small for a pass that fans out. 4. **The price sheet has a 90-day staleness rule** (§1a) and no automated check. Every number this capability produces inherits that. diff --git a/tools/__pycache__/cb-cost.cpython-312.pyc b/tools/__pycache__/cb-cost.cpython-312.pyc new file mode 100644 index 0000000..12efe0a Binary files /dev/null and b/tools/__pycache__/cb-cost.cpython-312.pyc differ diff --git a/workplans/CB-WP-0002-cost-accounting.md b/workplans/CB-WP-0002-cost-accounting.md index ad7f5a5..b55e718 100644 --- a/workplans/CB-WP-0002-cost-accounting.md +++ b/workplans/CB-WP-0002-cost-accounting.md @@ -40,8 +40,8 @@ repeatable acceptance target: | Cache write (1h) | 1,672,854 | $21.95 | | Input | 676 | $0.00 | | Main transcript (339 responses: 213 opus-5, 118 fable-5, 14 sonnet-5) | | $92.21 | -| Subagent tree (adversarial review, ran inside the pin) | | $0.66 | -| **TRUE TOTAL** | | **$92.87** | +| Subagent tree (adversarial review, ran inside the pin) | | $1.11 | +| **TRUE TOTAL** | | **$93.32** | The headline finding survives the correction and gets sharper: **88.4% of the cost is cache, against 314k tokens of output — a 256:1 @@ -162,15 +162,15 @@ each message at its own model's rate. ```task id: CB-WP-0002-T05 -status: todo +status: done priority: medium state_hub_task_id: "bea4cc0a-e4d0-4077-9dc7-df7726a48f86" ``` Run the collector over the CB-WP-0001 session and commit `evidence/CB-EV-0002-cost-accounting.md`. The acceptance test is -`specs/CostAccounting.md` AC-1: reproduce **$92.87 pinned at `fc76445`, -as its two components** ($92.21 main transcript + $0.66 subagent tree), +`specs/CostAccounting.md` AC-1: reproduce **$93.32 pinned at `fc76445`, +as its two components** ($92.21 main transcript + $1.11 subagent tree), plus AC-2's zero reconciliation residual. *(Originally written as "must reproduce $248.46". That figure was wrong