clay-borg/evidence/CB-EV-0002-cost-accounting.md

CB-EV-0002: cost accounting

capability: meta.loop.cost-accounting spec: CostAccounting.md v1.0 decision: ADR-0003 research: CB-RES-0002 instrument: make cost-pin (tools/cb-cost.py --pin fc76445) measured: 2026-07-31

Subject of measurement: the CB-WP-0001 session, pinned at commit fc76445 (2026-07-31T02:17:59Z). All numbers below are emitted by the tool, not transcribed by hand (CA-15).

---

1. Acceptance table

ID	Metric	Target	Measured	Verdict
AC-1	pinned total, as two components	$93.32 = $92.21 + $1.11	$92.21 main + $1.11 subagent = $93.32	met
AC-2	reconciliation residual	$0.00	$0.000000	met
AC-3	unattributed share reported	present, 33%	32.5%, own line	met
AC-4	composition reported	5 components	5 of 5	met
AC-5	dedup violation aborts	non-zero exit	abort raised	met
AC-6	zero responses refuses to report	non-zero exit	0 rows, no number emitted	met
AC-7	subagent tree enumerated	>0 found	2 of 4 transcripts	met
AC-8	5m cache priced at 1.25×	$1.25/100k @ fable	$1.2500 (1h would be $2.0000)	met
AC-9	streamed partial output → final	5,5,195 → 195	195	met

No unmeasured rows. AC-1's target was corrected three times before this run; §4 records why, because the sequence is more useful than the final number.

2. Composition — the finding the metric exists for

    input                  690 tok  $     0.00    0.0%
    output             323,643 tok  $    11.15   11.9%
    cache_read      80,611,798 tok  $    59.75   64.0%
    write_5m            37,467 tok  $     0.47    0.5%
    write_1h         1,672,854 tok  $    21.95   23.5%
    TOTAL                          $    93.32

88.0% of spend is cache; 11.9% is output. The ratio of context re-read to text written is 249:1. A single total would have shown $93.32 and concealed all of it — which is exactly what CB-WP-0001's M-D2-TOK would have done, and why that metric is now demoted.

3. Per-task attribution

    UNATTRIBUTED        $    30.32   32.5%
    T08                 $    21.02   22.5%
    T07                 $    12.01   12.9%
    T03                 $     9.91   10.6%
    T04                 $     8.00    8.6%
    T05                 $     7.19    7.7%
    T06                 $     3.58    3.8%
    T09                 $     1.29    1.4%

Stated limit (CA-10): 32.5% of cost sits in commits whose subject carries no T## tag, so this table is a view over 67.5% of spend. That is a property of commit hygiene, not of the collector.

T08 (the GROUND aggregate, six code iterations) at $21.02 is the most expensive task and was also the one that produced the most rework — the tuple-map hash panic, the discarded setup.patch, and the 5.6×-wrong benchmark all originated there. Expensive and error-dense: the correlation is worth watching, not yet a conclusion from n=1.

4. What it does not support

• The per-task figures are not comparable across passes. They mix models (opus/fable/sonnet) at different price points and different cache states. The dollar figure is comparable; a token count is not.
• AC-3's 32.5% is a fixture pin, not a quality target. Improving commit tagging will move it, and that is the desired direction.
• This is one session. Every ratio here (cache share, $/turn, the compaction effect in §5) is n=1 and should be treated as a hypothesis until a second pass reproduces it.
• The price sheet cannot express a time-boxed rate. Sonnet's intro price lives in a TOML comment, so sonnet-priced work is off by $0.17 here (0.19%). This becomes a real error on 2026-08-31.
• AC-1 is not invoice-verified. No admin key exists, so the Anthropic billing API could not independently confirm the total. The transcript counters are the same ones billing uses, but that is an argument, not a reconciliation.

5. The question that could not be answered before

"What does /compact actually cost, and is a long session quadratic?"

CB-WP-0003 T04 asserts that cost ≈ turns × mean_context and that mean_context grows with turns, making long sessions quadratic. The data qualifies that claim: it is true only between compactions.

The session compacted twice, and the transcript records both directly:

compaction	pre-tokens	post-tokens	reduction
C1 00:07:14Z (manual)	542,991	19,974	27×
C2 02:22:23Z (manual)	344,954	19,035	18×

Cost per turn across the boundary:

segment	turns	mean context	total	$/turn
start → C1	136	304,178	$62.19	$0.457
C1 → C2	202	193,493	$30.01	$0.149

The 202 turns after the first compaction cost less than half of the 136 turns before it — a 3.1× drop in cost per turn. Context growth is the cost driver, and compaction is the control on it. At the pre-compact rate, those 202 turns would have cost ~$92 instead of $30.

Two consequences for CB-WP-0003:

1. T04's "one task per session" recommendation is not the only remedy, and may not be the cheapest one. Compaction achieved a 27× context reduction inside a running session at the cost of one summarization call. A fresh session pays a cold-start re-read of the committed artifacts; compaction pays a summary. Which is cheaper is now a measurable question rather than a matter of taste, and T04 should measure it before prescribing.
2. The quadratic claim should be restated as bounded-quadratic: cost grows with context between compactions and resets at each one. The failure mode is not "a long session" but "a long uncompacted session".

6. Retrospective note

The positive control paid for itself on its first execution, which is the strongest evidence this project has produced for the InnerLoop v1.0 rule that added it. CA-02 asserted that usage is identical across the lines of one requestId — verified twice on the main transcript, by the survey (206/206 groups) and independently by the adversarial reviewer. It is false in the subagents/ tree, where output_tokens is a running count (5, 5, 195). The assertion fired, the run aborted, and the tool refused to print a number. Under the prior first-wins rule it would have printed a plausible one.

The generalization that failed is worth naming: a property verified on the largest sample was assumed to hold on the smallest one. The main transcript is 338 of 346 responses, so 206/206 felt conclusive; the violation lives entirely in the 8 responses nobody checked separately.

Cost of the adversarial review this pass: $1.11, against a $93.32 pass. It found three approval-blocking defects, one of which (the subagent exclusion) would have made this evidence file certify a broken collector. Second consecutive pass where a ~1% spend on review changed the outcome. That is now two data points for CB-WP-0003 T03.