coulomb/clay-borg
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

T04: tools/cb-cost.py — and its positive control fires on first contact

Browse source 
Collector per ADR-0003: enumerates every transcript including the
subagents/ tree, dedups by requestId, prices per model and per cache
TTL, attributes on (prev_commit, this_commit] intervals, and reconciles
to the cent or aborts.

The positive control caught a real defect on its very first run against
real data, which is the entire argument for writing it:

  CA-02 assumed usage is identical across the lines of one requestId.
  True in the main transcript (206/206 groups, verified twice — by the
  survey and by the adversarial reviewer). FALSE in the subagents/
  tree, where output_tokens is a running count: one response reads
  5, 5, 195 across its three lines. First-wins scored it at 5.

So CA-02 now splits: input-side counters are charged once and must be
identical (assertion retained); output_tokens resolves to the max
(CA-02a). AC-9 pins the exact 5,5,195 case as a regression test.

The acceptance target moved again as a result, for the third time:
$248.46 -> $92.21 -> $92.87 -> $93.32. AC-1 was computed by the same
first-wins method the tool just disproved, so the tool failing its
target was the tool being correct. Target updated, not the tool.

Measured at the pin: $92.21 main + $1.11 subagent = $93.32, residual
$0.000000. 32.5% UNATTRIBUTED, reported as its own line per CA-10.

make cost / cost-test / cost-pin wired; cost-test added to `make all`
and to CI, where it gates the collector's assertions without needing
transcripts present.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 08:50:04 +02:00
parent 900467017f
commit e008b1e787
5 changed files with 514 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

15
Makefile
View file

@ -3,7 +3,7 @@
CARGO := cargo
.PHONY: check test sim bench bench-test coverage dep-weight loc all
.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin loc all
## fmt + clippy (deny warnings) + HashMap deny-lint
check:
@ -21,6 +21,17 @@ dep-weight:
coverage:
python3 tools/rule-coverage.py
# M-D2-CST (specs/CostAccounting.md). cost-test is the positive control and
# runs first: a cost number from an unverified collector is void.
cost: cost-test
python3 tools/cb-cost.py --composition --by-task
cost-test:
python3 tools/cb-cost.py --self-test
cost-pin: cost-test
python3 tools/cb-cost.py --pin fc76445 --composition --by-task
sim:
$(CARGO) run -q -p cb-sim -- scenarios/ground/*.yaml
@ -40,4 +51,4 @@ loc:
printf '%-28s %s\n' $$d "$$(find $$d/src -name '*.rs' | xargs cat | grep -vcE '^\s*(//|$$)')"; \
done
all: check test sim coverage dep-weight bench-test
all: check test sim coverage dep-weight cost-test bench-test