coulomb/clay-borg
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

T04: tools/cb-cost.py — and its positive control fires on first contact

Browse source 
Collector per ADR-0003: enumerates every transcript including the
subagents/ tree, dedups by requestId, prices per model and per cache
TTL, attributes on (prev_commit, this_commit] intervals, and reconciles
to the cent or aborts.

The positive control caught a real defect on its very first run against
real data, which is the entire argument for writing it:

  CA-02 assumed usage is identical across the lines of one requestId.
  True in the main transcript (206/206 groups, verified twice — by the
  survey and by the adversarial reviewer). FALSE in the subagents/
  tree, where output_tokens is a running count: one response reads
  5, 5, 195 across its three lines. First-wins scored it at 5.

So CA-02 now splits: input-side counters are charged once and must be
identical (assertion retained); output_tokens resolves to the max
(CA-02a). AC-9 pins the exact 5,5,195 case as a regression test.

The acceptance target moved again as a result, for the third time:
$248.46 -> $92.21 -> $92.87 -> $93.32. AC-1 was computed by the same
first-wins method the tool just disproved, so the tool failing its
target was the tool being correct. Target updated, not the tool.

Measured at the pin: $92.21 main + $1.11 subagent = $93.32, residual
$0.000000. 32.5% UNATTRIBUTED, reported as its own line per CA-10.

make cost / cost-test / cost-pin wired; cost-test added to `make all`
and to CI, where it gates the collector's assertions without needing
transcripts present.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 08:50:04 +02:00
parent 900467017f
commit e008b1e787
5 changed files with 514 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

36
specs/CostAccounting.md
View file

@ -22,13 +22,26 @@ block (`thinking`, `text`, `tool_use`), and every line repeats the same
> **CA-01.** Cost is computed over responses deduplicated by `requestId`.
> Summing per line is a defect; it inflates by ≈1.9× on measured data.
> **CA-02.** Dedup is asserted, not assumed. Within a `requestId` group,
> `usage` must be identical across lines and the model must not vary. A
> violation aborts the run rather than producing a number.
> **CA-02.** Dedup is asserted, not assumed. Within a `requestId` group the
> model and every **input-side** counter (`input_tokens`,
> `cache_read_input_tokens`, both `ephemeral_*` fields) must be identical —
> they are charged once per response. A divergence aborts the run.
Rationale for CA-02: if the format ever splits one response across two
`requestId`s, dedup *under*-reports and nothing looks wrong. The dangerous
direction gets the assertion.
> **CA-02a.** `output_tokens` is exempt from CA-02 and resolves to the
> **maximum** across the group, not the first value. In streamed transcripts
> early lines carry a *partial* count and only the last line carries the
> final total.
Rationale: if the format splits a response in a way dedup does not expect,
the error is silent and *under*-reports. The dangerous direction gets the
assertion.
*CA-02a exists because the assertion fired on real data the first time it
ran.* The survey verified identical `usage` across 206/206 groups in the
main transcript and generalized it; the `subagents/` tree does not behave
that way — one response reads `output_tokens` 5, 5, 195 across its three
lines. First-wins scored it at 5. That error moved the acceptance target
by $0.45.
### 1.2 Price formula
@ -112,14 +125,15 @@ Each row names the command that produces its number, per InnerLoop §Step 4.
| ID | Metric | Target | Instrument |
|---|---|---|---|
| **AC-1** | reproduces the pinned CB-WP-0001 total | **$92.87** = $92.21 main + $0.66 subagent | `cb-cost --repo clay-borg --pin fc76445` |
| **AC-1** | reproduces the pinned CB-WP-0001 total | **$93.32** = $92.21 main + $1.11 subagent | `make cost-pin` |
| **AC-2** | reconciliation residual (CA-14) | **$0.00** exactly | same command, `reconciled: ok` line |
| **AC-3** | unattributed share reported (CA-10) | present, and **33%** on the pinned run | `cb-cost --pin fc76445 --by-task` |
| **AC-4** | composition reported (CA-13) | all five components present | `cb-cost --pin fc76445 --composition` |
| **AC-5** | dedup invariant asserted (CA-02) | violation exits non-zero | `cb-cost --self-test` |
| **AC-6** | positive control: refuses to report on zero responses | exits non-zero | `cb-cost --self-test` |
| **AC-7** | subagent tree included (CA-06) | omitting it changes AC-1 by $0.66 | `cb-cost --self-test` |
| **AC-8** | per-TTL cache pricing (CA-04) | 5m-only transcript prices at 1.25× | `cb-cost --self-test` |
| **AC-5** | dedup invariant asserted (CA-02) | violation exits non-zero | `make cost-test` |
| **AC-6** | positive control: refuses to report on zero responses | exits non-zero | `make cost-test` |
| **AC-7** | subagent tree included (CA-06) | omitting it changes AC-1 by $1.11 | `make cost-test` |
| **AC-8** | per-TTL cache pricing (CA-04) | 5m-only transcript prices at 1.25× | `make cost-test` |
| **AC-9** | streamed partial output resolves to final (CA-02a) | 5,5,195 → 195, not 5 | `make cost-test` |
**AC-5 through AC-8 are the positive control.** Per InnerLoop v1.0 §Step 5,
a harness must assert it did the work it reports. `--self-test` runs each