Compare commits
10 commits
fc76445aec
...
42180fbc57
| Author | SHA1 | Date | |
|---|---|---|---|
| 42180fbc57 | |||
| 2f086d26b6 | |||
| c0eef604e8 | |||
| e008b1e787 | |||
| 900467017f | |||
| b96cd94a64 | |||
| c3e6e72ab2 | |||
| ac3ac2a8ce | |||
| bf57b6f3a4 | |||
| 060eb8cf6a |
15 changed files with 1886 additions and 42 deletions
|
|
@ -33,6 +33,11 @@ jobs:
|
||||||
# AM-4a/AM-4b: third-party source under audit, per configuration.
|
# AM-4a/AM-4b: third-party source under audit, per configuration.
|
||||||
- run: make dep-weight
|
- run: make dep-weight
|
||||||
|
|
||||||
|
# Positive control for the cost collector (AC-5..AC-9). Does not
|
||||||
|
# gate on a dollar figure — transcripts are not present in CI — but
|
||||||
|
# proves the collector still detects the failures it claims to.
|
||||||
|
- run: make cost-test
|
||||||
|
|
||||||
# InnerLoop v1.0 positive control, enforced rather than asserted in
|
# InnerLoop v1.0 positive control, enforced rather than asserted in
|
||||||
# prose: --test runs every benchmark once, so a workload that
|
# prose: --test runs every benchmark once, so a workload that
|
||||||
# stalls or produces the wrong event count fails the build instead
|
# stalls or produces the wrong event count fails the build instead
|
||||||
|
|
|
||||||
15
Makefile
15
Makefile
|
|
@ -3,7 +3,7 @@
|
||||||
|
|
||||||
CARGO := cargo
|
CARGO := cargo
|
||||||
|
|
||||||
.PHONY: check test sim bench bench-test coverage dep-weight loc all
|
.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin loc all
|
||||||
|
|
||||||
## fmt + clippy (deny warnings) + HashMap deny-lint
|
## fmt + clippy (deny warnings) + HashMap deny-lint
|
||||||
check:
|
check:
|
||||||
|
|
@ -21,6 +21,17 @@ dep-weight:
|
||||||
coverage:
|
coverage:
|
||||||
python3 tools/rule-coverage.py
|
python3 tools/rule-coverage.py
|
||||||
|
|
||||||
|
# M-D2-CST (specs/CostAccounting.md). cost-test is the positive control and
|
||||||
|
# runs first: a cost number from an unverified collector is void.
|
||||||
|
cost: cost-test
|
||||||
|
python3 tools/cb-cost.py --composition --by-task
|
||||||
|
|
||||||
|
cost-test:
|
||||||
|
python3 tools/cb-cost.py --self-test
|
||||||
|
|
||||||
|
cost-pin: cost-test
|
||||||
|
python3 tools/cb-cost.py --pin fc76445 --composition --by-task
|
||||||
|
|
||||||
sim:
|
sim:
|
||||||
$(CARGO) run -q -p cb-sim -- scenarios/ground/*.yaml
|
$(CARGO) run -q -p cb-sim -- scenarios/ground/*.yaml
|
||||||
|
|
||||||
|
|
@ -40,4 +51,4 @@ loc:
|
||||||
printf '%-28s %s\n' $$d "$$(find $$d/src -name '*.rs' | xargs cat | grep -vcE '^\s*(//|$$)')"; \
|
printf '%-28s %s\n' $$d "$$(find $$d/src -name '*.rs' | xargs cat | grep -vcE '^\s*(//|$$)')"; \
|
||||||
done
|
done
|
||||||
|
|
||||||
all: check test sim coverage dep-weight bench-test
|
all: check test sim coverage dep-weight cost-test bench-test
|
||||||
|
|
|
||||||
118
decisions/ADR-0003-cost-accounting.md
Normal file
118
decisions/ADR-0003-cost-accounting.md
Normal file
|
|
@ -0,0 +1,118 @@
|
||||||
|
# ADR-0003: cost accounting — instrument and attribution model
|
||||||
|
|
||||||
|
status: accepted
|
||||||
|
date: 2026-07-31
|
||||||
|
tier: L (structural L, chaos d10=2 → no override)
|
||||||
|
research: [CB-RES-0002](../research/CB-RES-0002-cost-accounting.md) (approved)
|
||||||
|
review: history/260731-cost-accounting-{challenge,response}.md
|
||||||
|
supersedes: the "uncomputable" disposition of AM-12 / M-D2-CST in
|
||||||
|
[CB-EV-0001](../evidence/CB-EV-0001-game-kernel.md)
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
M-D2-CST was fully specified in CB-WP-0001 and never instrumented, so D2
|
||||||
|
claims about implementation efficiency rest on nothing measured. The survey
|
||||||
|
established that the data existed the whole time, and that reading it
|
||||||
|
correctly is harder than it looks: this workplan's own opening figure was
|
||||||
|
wrong by 2.7×, and the survey's first draft set an acceptance target only a
|
||||||
|
*broken* collector could hit.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
### D1 — Instrument: session transcript JSONL (C1)
|
||||||
|
|
||||||
|
Cost is computed from `~/.claude/projects/<slug>/**/*.jsonl`, deduplicated
|
||||||
|
by `requestId`, priced per message against
|
||||||
|
`benchmarks/baselines/model-prices.toml`.
|
||||||
|
|
||||||
|
Rejected: the State Hub as a *source* (its schema has no cache fields, so it
|
||||||
|
cannot represent 88% of spend, and its recorded numbers are ~100% in error
|
||||||
|
against the same work); the status bar (not machine-readable from a tool
|
||||||
|
call); the billing API (no session or task attribution, no admin key). The
|
||||||
|
hub remains the durable **sink**; the billing API remains an optional
|
||||||
|
external reconciliation check if an admin key ever exists.
|
||||||
|
|
||||||
|
### D2 — Attribution: git commit intervals, scoped by session
|
||||||
|
|
||||||
|
A message is attributed to the task named by the **next commit at or after
|
||||||
|
it**, within its own session:
|
||||||
|
|
||||||
|
```text
|
||||||
|
interval := (prev_commit_time, this_commit_time] # ending-at-commit
|
||||||
|
scope := sessionId # never wall-clock alone
|
||||||
|
task := the T## tag in the commit subject, else UNATTRIBUTED
|
||||||
|
```
|
||||||
|
|
||||||
|
Ending-at-commit is the only convention consistent with the loop's
|
||||||
|
commit-at-end-of-task pattern; the alternative shifts every task's cost one
|
||||||
|
interval. Session scoping is not optional: two sessions overlap 4 h 13 m on
|
||||||
|
this repo carrying ~$12 that no wall-clock join can separate.
|
||||||
|
|
||||||
|
Rejected: **explicit session markers** at task start/end — they require the
|
||||||
|
agent to remember, and the failure this workplan exists to fix was caused by
|
||||||
|
exactly that kind of remembered step; nothing that depends on discipline
|
||||||
|
gets to be the primary index. Rejected: **hub status transitions** as the
|
||||||
|
time index — they are written after the fact, sometimes in a batch, and one
|
||||||
|
was written for CB-WP-0001 nine minutes after the work it bounds.
|
||||||
|
|
||||||
|
### D3 — Reported shape: composition, not a total
|
||||||
|
|
||||||
|
Every report carries the cost split by component (input / output / cache
|
||||||
|
read / cache write per TTL) alongside the total. A single total would have
|
||||||
|
concealed the finding that motivated the workplan.
|
||||||
|
|
||||||
|
## Expected advantage per dimension
|
||||||
|
|
||||||
|
| Dim | Expectation vs the C2 baseline | Basis |
|
||||||
|
|---|---|---|
|
||||||
|
| **D1 ease of specification** | **worse.** The hub is one API call; this is a parser with a dedup rule, a TTL-aware price model, a session scope, and a commit join. Four moving parts against one. | measured: the survey needed three corrections to get the parse right |
|
||||||
|
| **D2 efficiency** | **better, decisively.** C2's recorded numbers are ~100% in error on the same work ($0.03-equivalent recorded against $92.21 actual). C1 is exact by construction. Cost to produce: one file read, ~1 s. | measured |
|
||||||
|
| **D3 speed** | **parity.** 2,040 lines / 5.1 MB parsed in <1 s; a hub call is a network round trip. Neither is a bottleneck. | measured |
|
||||||
|
| **D4 optionality** | **better.** The transcript is a file on disk in a documented shape; the hub is a service that must be running. The collector degrades to "no data" rather than "wrong data" when a transcript is absent. | reasoned |
|
||||||
|
|
||||||
|
An honest summary: **we are buying accuracy with specification complexity.**
|
||||||
|
That trade is right here only because the alternative is not "a simpler
|
||||||
|
correct number" but "a number that is wrong by two orders of magnitude",
|
||||||
|
which is what CB-WP-0001 actually recorded.
|
||||||
|
|
||||||
|
## Known failure modes of the chosen model
|
||||||
|
|
||||||
|
Stated rather than discovered later. Each becomes a test in T04.
|
||||||
|
|
||||||
|
1. **33% of spend has no task.** Only 14 of 33 commits name a task; the rest
|
||||||
|
hold $30.32 of $92.21. Per-task tables are a view over two-thirds of the
|
||||||
|
money and must say so wherever reported.
|
||||||
|
2. **Work spanning a boundary is assigned whole to the later task.** A
|
||||||
|
message before a commit belongs to that commit's task even if the thinking
|
||||||
|
began earlier. Accepted: the loop commits per task, so the error is
|
||||||
|
bounded by one interval (p90 17.7 min, max 36.8 min).
|
||||||
|
3. **Uncommitted work is invisible.** Cost incurred after the last commit
|
||||||
|
has no enclosing interval. Reported as an open remainder, never dropped.
|
||||||
|
4. **`/compact` is safe; resumed and concurrent sessions are the risk.**
|
||||||
|
Compaction stays within one file and one session. Two agents on one repo
|
||||||
|
are separable only because `sessionId` exists — this is why D2 scopes by
|
||||||
|
it.
|
||||||
|
5. **The price sheet cannot express a time-boxed rate.** Sonnet's intro
|
||||||
|
price is a TOML comment. $0.17 at the pin; the schema defect is the real
|
||||||
|
issue and is deferred to T03 with a stated deadline of 2026-08-31, when
|
||||||
|
the intro rate expires and the sheet becomes silently wrong.
|
||||||
|
6. **Dedup is load-bearing in the dangerous direction.** If the format ever
|
||||||
|
splits one response across two `requestId`s, the collector *under*-reports
|
||||||
|
and nothing looks wrong. T04 asserts the dedup invariant at runtime
|
||||||
|
(identical `usage` within a group) rather than trusting the survey's
|
||||||
|
one-time check.
|
||||||
|
7. **The subagent tree is a separate enumeration.** Missing it under-reports
|
||||||
|
silently; it was $0.66 here and will not stay small on a fan-out pass.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- **Gate satisfied:** T04 may now write collector code. Nothing before this.
|
||||||
|
- T03 specifies the metrics with this ADR's contracts; T05's acceptance test
|
||||||
|
is $92.87 as two components, with a 33% unattributed line.
|
||||||
|
- The hub's token API keeps receiving events, now as a lossy projection of a
|
||||||
|
computed number rather than as an estimate. Its schema gap (no cache
|
||||||
|
fields) is recorded as a limitation of the dashboard, not of the metric.
|
||||||
|
- **Raised out of scope, for the ralph-workplan skill:** its instruction to
|
||||||
|
"read tokens from the Claude Code status bar" asks an agent for a figure
|
||||||
|
it cannot read, and is the proximate cause of the hub's bad numbers. The
|
||||||
|
skill lives outside this repo; flagged for the maintainer.
|
||||||
152
evidence/CB-EV-0002-cost-accounting.md
Normal file
152
evidence/CB-EV-0002-cost-accounting.md
Normal file
|
|
@ -0,0 +1,152 @@
|
||||||
|
# CB-EV-0002: cost accounting
|
||||||
|
|
||||||
|
capability: meta.loop.cost-accounting
|
||||||
|
spec: [CostAccounting.md](../specs/CostAccounting.md) v1.0
|
||||||
|
decision: [ADR-0003](../decisions/ADR-0003-cost-accounting.md)
|
||||||
|
research: [CB-RES-0002](../research/CB-RES-0002-cost-accounting.md)
|
||||||
|
instrument: `make cost-pin` (`tools/cb-cost.py --pin fc76445`)
|
||||||
|
measured: 2026-07-31
|
||||||
|
|
||||||
|
Subject of measurement: the CB-WP-0001 session, pinned at commit `fc76445`
|
||||||
|
(`2026-07-31T02:17:59Z`). All numbers below are emitted by the tool, not
|
||||||
|
transcribed by hand (CA-15).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Acceptance table
|
||||||
|
|
||||||
|
| ID | Metric | Target | Measured | Verdict |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| AC-1 | pinned total, as two components | $93.32 = $92.21 + $1.11 | **$92.21 main + $1.11 subagent = $93.32** | **met** |
|
||||||
|
| AC-2 | reconciliation residual | $0.00 | **$0.000000** | **met** |
|
||||||
|
| AC-3 | unattributed share reported | present, 33% | **32.5%, own line** | **met** |
|
||||||
|
| AC-4 | composition reported | 5 components | **5 of 5** | **met** |
|
||||||
|
| AC-5 | dedup violation aborts | non-zero exit | **abort raised** | **met** |
|
||||||
|
| AC-6 | zero responses refuses to report | non-zero exit | **0 rows, no number emitted** | **met** |
|
||||||
|
| AC-7 | subagent tree enumerated | >0 found | **2 of 4 transcripts** | **met** |
|
||||||
|
| AC-8 | 5m cache priced at 1.25× | $1.25/100k @ fable | **$1.2500 (1h would be $2.0000)** | **met** |
|
||||||
|
| AC-9 | streamed partial output → final | 5,5,195 → 195 | **195** | **met** |
|
||||||
|
|
||||||
|
No `unmeasured` rows. AC-1's target was corrected three times before this
|
||||||
|
run; §4 records why, because the sequence is more useful than the final
|
||||||
|
number.
|
||||||
|
|
||||||
|
## 2. Composition — the finding the metric exists for
|
||||||
|
|
||||||
|
```text
|
||||||
|
input 690 tok $ 0.00 0.0%
|
||||||
|
output 323,643 tok $ 11.15 11.9%
|
||||||
|
cache_read 80,611,798 tok $ 59.75 64.0%
|
||||||
|
write_5m 37,467 tok $ 0.47 0.5%
|
||||||
|
write_1h 1,672,854 tok $ 21.95 23.5%
|
||||||
|
TOTAL $ 93.32
|
||||||
|
```
|
||||||
|
|
||||||
|
**88.0% of spend is cache; 11.9% is output.** The ratio of context re-read
|
||||||
|
to text written is 249:1. A single total would have shown $93.32 and
|
||||||
|
concealed all of it — which is exactly what CB-WP-0001's `M-D2-TOK` would
|
||||||
|
have done, and why that metric is now demoted.
|
||||||
|
|
||||||
|
## 3. Per-task attribution
|
||||||
|
|
||||||
|
```text
|
||||||
|
UNATTRIBUTED $ 30.32 32.5%
|
||||||
|
T08 $ 21.02 22.5%
|
||||||
|
T07 $ 12.01 12.9%
|
||||||
|
T03 $ 9.91 10.6%
|
||||||
|
T04 $ 8.00 8.6%
|
||||||
|
T05 $ 7.19 7.7%
|
||||||
|
T06 $ 3.58 3.8%
|
||||||
|
T09 $ 1.29 1.4%
|
||||||
|
```
|
||||||
|
|
||||||
|
**Stated limit (CA-10):** 32.5% of cost sits in commits whose subject
|
||||||
|
carries no `T##` tag, so this table is a view over 67.5% of spend. That is
|
||||||
|
a property of commit hygiene, not of the collector.
|
||||||
|
|
||||||
|
T08 (the GROUND aggregate, six code iterations) at $21.02 is the most
|
||||||
|
expensive task and was also the one that produced the most rework — the
|
||||||
|
tuple-map hash panic, the discarded `setup.patch`, and the 5.6×-wrong
|
||||||
|
benchmark all originated there. Expensive *and* error-dense: the correlation
|
||||||
|
is worth watching, not yet a conclusion from n=1.
|
||||||
|
|
||||||
|
## 4. What it does not support
|
||||||
|
|
||||||
|
- **The per-task figures are not comparable across passes.** They mix
|
||||||
|
models (opus/fable/sonnet) at different price points and different cache
|
||||||
|
states. The dollar figure is comparable; a token count is not.
|
||||||
|
- **AC-3's 32.5% is a fixture pin, not a quality target.** Improving commit
|
||||||
|
tagging will move it, and that is the desired direction.
|
||||||
|
- **This is one session.** Every ratio here (cache share, $/turn, the
|
||||||
|
compaction effect in §5) is n=1 and should be treated as a hypothesis
|
||||||
|
until a second pass reproduces it.
|
||||||
|
- **The price sheet cannot express a time-boxed rate.** Sonnet's intro price
|
||||||
|
lives in a TOML comment, so sonnet-priced work is off by $0.17 here
|
||||||
|
(0.19%). This becomes a real error on 2026-08-31.
|
||||||
|
- **AC-1 is not invoice-verified.** No admin key exists, so the Anthropic
|
||||||
|
billing API could not independently confirm the total. The transcript
|
||||||
|
counters are the same ones billing uses, but that is an argument, not a
|
||||||
|
reconciliation.
|
||||||
|
|
||||||
|
## 5. The question that could not be answered before
|
||||||
|
|
||||||
|
**"What does `/compact` actually cost, and is a long session quadratic?"**
|
||||||
|
|
||||||
|
CB-WP-0003 T04 asserts that cost ≈ turns × mean_context and that mean_context
|
||||||
|
grows with turns, making long sessions quadratic. The data **qualifies that
|
||||||
|
claim**: it is true only between compactions.
|
||||||
|
|
||||||
|
The session compacted twice, and the transcript records both directly:
|
||||||
|
|
||||||
|
| compaction | pre-tokens | post-tokens | reduction |
|
||||||
|
|---|---|---|---|
|
||||||
|
| C1 `00:07:14Z` (manual) | 542,991 | 19,974 | **27×** |
|
||||||
|
| C2 `02:22:23Z` (manual) | 344,954 | 19,035 | **18×** |
|
||||||
|
|
||||||
|
Cost per turn across the boundary:
|
||||||
|
|
||||||
|
| segment | turns | mean context | total | $/turn |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| start → C1 | 136 | 304,178 | $62.19 | **$0.457** |
|
||||||
|
| C1 → C2 | 202 | 193,493 | $30.01 | **$0.149** |
|
||||||
|
|
||||||
|
**The 202 turns after the first compaction cost less than half of the 136
|
||||||
|
turns before it — a 3.1× drop in cost per turn.** Context growth *is* the
|
||||||
|
cost driver, and compaction is the control on it. At the pre-compact rate,
|
||||||
|
those 202 turns would have cost ~$92 instead of $30.
|
||||||
|
|
||||||
|
Two consequences for CB-WP-0003:
|
||||||
|
|
||||||
|
1. **T04's "one task per session" recommendation is not the only remedy,
|
||||||
|
and may not be the cheapest one.** Compaction achieved a 27× context
|
||||||
|
reduction inside a running session at the cost of one summarization
|
||||||
|
call. A fresh session pays a cold-start re-read of the committed
|
||||||
|
artifacts; compaction pays a summary. Which is cheaper is now a
|
||||||
|
measurable question rather than a matter of taste, and T04 should
|
||||||
|
measure it before prescribing.
|
||||||
|
2. **The quadratic claim should be restated as bounded-quadratic:** cost
|
||||||
|
grows with context between compactions and resets at each one. The
|
||||||
|
failure mode is not "a long session" but "a long *uncompacted* session".
|
||||||
|
|
||||||
|
## 6. Retrospective note
|
||||||
|
|
||||||
|
The positive control paid for itself on its first execution, which is the
|
||||||
|
strongest evidence this project has produced for the InnerLoop v1.0 rule
|
||||||
|
that added it. CA-02 asserted that `usage` is identical across the lines of
|
||||||
|
one `requestId` — verified twice on the main transcript, by the survey
|
||||||
|
(206/206 groups) and independently by the adversarial reviewer. It is false
|
||||||
|
in the `subagents/` tree, where `output_tokens` is a running count
|
||||||
|
(`5, 5, 195`). The assertion fired, the run aborted, and the tool refused to
|
||||||
|
print a number. Under the prior first-wins rule it would have printed a
|
||||||
|
plausible one.
|
||||||
|
|
||||||
|
The generalization that failed is worth naming: **a property verified on the
|
||||||
|
largest sample was assumed to hold on the smallest one.** The main
|
||||||
|
transcript is 338 of 346 responses, so 206/206 felt conclusive; the
|
||||||
|
violation lives entirely in the 8 responses nobody checked separately.
|
||||||
|
|
||||||
|
Cost of the adversarial review this pass: **$1.11**, against a $93.32 pass.
|
||||||
|
It found three approval-blocking defects, one of which (the subagent
|
||||||
|
exclusion) would have made this evidence file certify a broken collector.
|
||||||
|
Second consecutive pass where a ~1% spend on review changed the outcome.
|
||||||
|
That is now two data points for CB-WP-0003 T03.
|
||||||
163
history/260731-cost-accounting-challenge.md
Normal file
163
history/260731-cost-accounting-challenge.md
Normal file
|
|
@ -0,0 +1,163 @@
|
||||||
|
# 2026-07-31 — adversarial challenge to CB-RES-0002
|
||||||
|
|
||||||
|
Round 1 of 1 (InnerLoop §Step 2). Reviewer: separate agent session, given
|
||||||
|
only `research/CB-RES-0002-cost-accounting.md` and read-only access to the
|
||||||
|
raw data. Verbatim below; answers in `260731-cost-accounting-response.md`.
|
||||||
|
|
||||||
|
Reviewer cost: 56,882 tokens, 14 tool uses, 330 s.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Reproduced cleanly** (`python3` over the pinned line set, prices from
|
||||||
|
`benchmarks/baselines/model-prices.toml`): pin `2026-07-31T02:17:59Z` =
|
||||||
|
commit `fc76445` (`%cI` = `2026-07-31T04:17:59+02:00`) ✓. 339 responses ✓
|
||||||
|
(206 opus / 118 fable / 14 sonnet / 1 synthetic) ✓. Output 313,900 →
|
||||||
|
$10.6625 ✓. Cache read 80,453,702 → $59.5880 ✓. Cache write 1h 1,672,854 →
|
||||||
|
$21.9525 ✓. Input 676 → $0.0045 ✓. **Total $92.2075** ✓. Cache share
|
||||||
|
88.43% ✓. Ratio 256.30:1 ✓. The three-method table reproduces to the cent
|
||||||
|
over the first 657 assistant lines: 346 groups, 700,690 out, 161,408,840
|
||||||
|
cread, **$289.1248 / $210.0536 / $93.1461** ✓. The "356 / $94.04" drift
|
||||||
|
check reproduces at assistant line 682 ✓. Subagent **$0.6580** ✓. Dedup
|
||||||
|
positive control holds: 0/200 pinned multi-line groups differ in `usage` or
|
||||||
|
model; 0 assistant lines anywhere in the file lack a `requestId`.
|
||||||
|
`iterations` claim holds: 758/760 non-null, 0 sum-mismatches, 0
|
||||||
|
multi-iteration. `cache_creation_input_tokens` == `ephemeral_1h +
|
||||||
|
ephemeral_5m` in all 775 usage lines across both transcripts. The survey did
|
||||||
|
re-derive; the boast is earned.
|
||||||
|
|
||||||
|
**1. The dedup positive-control table does not sum to its own totals. —
|
||||||
|
MATERIAL** Claim: group sizes `140/120/76/6/5/1`, "346 distinct requestIds",
|
||||||
|
"all 208 multi-line groups". Recomputed over the same 657-line prefix:
|
||||||
|
`{1:140, 2:120, 3:74, 4:6, 5:5, 6:1}`. The stated table gives 348 groups and
|
||||||
|
663 lines, contradicting both 346 and 657. Multi-line groups are **206**, not
|
||||||
|
208. This is the sole evidence for the load-bearing dedup assumption (Risk
|
||||||
|
2), so a table that fails its own checksum is the wrong place to be sloppy.
|
||||||
|
The 74 correction makes it sum exactly.
|
||||||
|
|
||||||
|
**2. The multi-model table is labelled "API responses (deduped)" but holds
|
||||||
|
per-line counts. — MATERIAL** `382 + 250 + 24 + 1 = 657` — those are lines,
|
||||||
|
the exact quantity the preceding paragraph just condemned. Deduped over the
|
||||||
|
same prefix: opus 213, fable 118, sonnet 14, synthetic 1 = 346. The mislabel
|
||||||
|
sits three lines below "Summing per line inflates by ≈1.9×."
|
||||||
|
|
||||||
|
**3. The $92.21 acceptance target contradicts C1's own blind-spot finding. —
|
||||||
|
MATERIAL (blocking for T05 as written)** The subagent transcript spans
|
||||||
|
`23:11:42Z–23:14:12Z` on 2026-07-30 — **inside the pin window**. Its $0.6580
|
||||||
|
is therefore part of the pinned session's true cost, which is **$92.87**, not
|
||||||
|
$92.21. The survey never states whether $92.21 is main-file-only. A collector
|
||||||
|
that correctly implements the C1 blind-spot ("a collector reading only the
|
||||||
|
main file silently under-reports") produces $92.87 and **fails** the
|
||||||
|
benchmark-to-beat. Either pin $92.87, or state "$92.21 main transcript +
|
||||||
|
$0.66 subagent = $92.87" explicitly.
|
||||||
|
|
||||||
|
**4. Pricing the top-level `cache_creation_input_tokens` at the 1h rate
|
||||||
|
overcharges by 60% on 5m blocks — and the only 5m data in the corpus is the
|
||||||
|
subagent. — MATERIAL** Main session: `w5m = 0`, so the pinned $92.21 is
|
||||||
|
insensitive to this. The subagent's *entire* cache write is 5m: 37,467
|
||||||
|
tokens. At `write_5m=1.25` → $0.4683; at `write_1h=2.0` → $0.7493. On a
|
||||||
|
$0.658 total that is a **+43% error**. C1 lists both field names but never
|
||||||
|
states that the top-level aggregate must not be priced at a single
|
||||||
|
multiplier. Given that a fan-out pass is exactly where subagents (and
|
||||||
|
therefore 5m caching) dominate, this belongs in the risk list.
|
||||||
|
|
||||||
|
**5. Concurrent sessions on one repo are not hypothetical — they are already
|
||||||
|
in the data. — MATERIAL** Risk 6: "two sessions interleaved on one repo would
|
||||||
|
mis-attribute. Not exercised here — CB-WP-0001 ran in a single session."
|
||||||
|
Session `f1eb1147` (same project dir) spans `02:18:13Z–06:33:41Z` while
|
||||||
|
`8cbd5701` runs to `06:30:47Z`. That is a **4 h 13 m overlap** carrying **67
|
||||||
|
responses / $6.09** from `8cbd5701` and **67 responses / $5.90** from
|
||||||
|
`f1eb1147` — ~$12 that a wall-clock join cannot separate, starting 14 seconds
|
||||||
|
after the pin. The survey also never mentions that a second transcript for
|
||||||
|
this repo exists; C1's blind-spot section names only the subagent tree.
|
||||||
|
|
||||||
|
**6. Attribution coverage is never quantified, and a third of the money has
|
||||||
|
no task. — MATERIAL** Stress-tested with intervals `(prev commit, this
|
||||||
|
commit]` over the 33 commits at/before pin: 0 responses before the first
|
||||||
|
commit, 0 after the last, exactly **1** empty interval (`Initial commit`). So
|
||||||
|
far so good. But **19 of 33 commits carry no `T##` in the subject**
|
||||||
|
(`chore(consistency)` ×4, `Add CB-WP-000x`, `CI: enforce every gate`, `AM-4:`,
|
||||||
|
`Adopt TRSL V1C1`, …), and they hold **$29.58 of $92.21 = 32% of cost**.
|
||||||
|
C5's "the loop already commits per task iteration with the task in the
|
||||||
|
subject line" is true of 14/33 commits. The benchmark-to-beat correctly
|
||||||
|
demands an unattributed-remainder line but sizes it nowhere; 32% is the
|
||||||
|
number that belongs in the survey.
|
||||||
|
|
||||||
|
**7. "Cache reads are ~87% of real spend" (C2, defect 1) matches no computed
|
||||||
|
quantity. — MATERIAL** Cache read alone = $59.588/$92.2075 = **64.6%** of
|
||||||
|
cost. All cache (read + 1h write) = **88.4%**. Cache-read tokens as a share
|
||||||
|
of input tokens = **98.0%**. Nothing is 87%. Since C2's elimination rests on
|
||||||
|
"the hub cannot express the ~87%", fix it to 64.6% (cost) or 88.4% (all
|
||||||
|
cache) and say which.
|
||||||
|
|
||||||
|
**8. The `<synthetic>` message is described wrongly, though its dollar impact
|
||||||
|
is zero. — MINOR** "1 (no usage; an error placeholder)". It has a valid
|
||||||
|
`requestId` (`req_011CdZ9m3xZSLgtg2baVuLmG`) and a **complete all-zero
|
||||||
|
`usage` object**. Zero dollars are dropped — confirmed, the unpriced
|
||||||
|
remainder is exactly $0.00 — but a collector guarding on `if not usage:` and
|
||||||
|
one guarding on `if model not in prices:` take different branches, and the
|
||||||
|
survey's description points at the wrong one.
|
||||||
|
|
||||||
|
**9. "Checked all 654 usage-bearing lines" does not reproduce. — MINOR** All
|
||||||
|
638 pinned assistant lines (and all 760 in the full file) carry `usage`;
|
||||||
|
637/638 pinned carry non-null `iterations`. There is no state of the file
|
||||||
|
with 654 usage-bearing lines out of 657. The *conclusion* is verified; the
|
||||||
|
*count* is not.
|
||||||
|
|
||||||
|
**10. The timezone hazard is stated as a single offset; the repo has two. —
|
||||||
|
MINOR** `git log` offsets: `+02:00` ×34, **`+00:00` ×1** (`8b66604 Initial
|
||||||
|
commit`). "commit timestamps are local (`+02:00`) … a naive join is off by
|
||||||
|
the offset" invites a fixed −2 h correction that is wrong for the first
|
||||||
|
commit. Say "parse `%cI` and convert", not "subtract the offset".
|
||||||
|
|
||||||
|
**11. Commit spacing understated. — MINOR** "Boundaries are 3–15 minutes
|
||||||
|
apart across CB-WP-0001." Measured over the 33 pinned commits: min 0.1, p50
|
||||||
|
**6.9**, p90 **17.7**, max **36.8** minutes. The max matters — a 37-minute
|
||||||
|
interval is coarser than a task.
|
||||||
|
|
||||||
|
**12. The interval convention is never named. — MINOR** "Join message
|
||||||
|
timestamps against commit intervals" is ambiguous between *ending-at-commit*
|
||||||
|
and *starting-at-commit*. The commit-at-end-of-task pattern makes only the
|
||||||
|
first correct, and under it the boundary conditions are clean (0 orphans
|
||||||
|
either end, verified). Under the other, every task's cost shifts one interval
|
||||||
|
and the reviewer's worry is real. One sentence fixes it.
|
||||||
|
|
||||||
|
**13. The price sheet's intro rate is a comment, not data. — MINOR**
|
||||||
|
`model-prices.toml` encodes sonnet at 3/15 with `# intro 2.00 through
|
||||||
|
2026-08-31`. A collector reading the TOML — which the Verdict prescribes —
|
||||||
|
silently ignores a live rate. Delta at the pin: **$0.1738** (0.19% of
|
||||||
|
$92.21). Small now; the failure mode is that a *time-boxed* rate is
|
||||||
|
unrepresentable in the schema, which is the same class of defect the survey
|
||||||
|
levels at C2.
|
||||||
|
|
||||||
|
**14. Baselines table: D1–D4 are each present, but the deciding axis is a
|
||||||
|
fifth non-rubric row with no beatable value. — MINOR** D1, D2 (×2), D3, D4
|
||||||
|
all appear, so the template is satisfied on its face. But the row marked
|
||||||
|
"**the deciding axis**" is "Accuracy", which is not one of the four
|
||||||
|
dimensions, and its value — "exact by construction" — is not a number
|
||||||
|
anything can be measured against. D4's provenance reads "measured (absent)",
|
||||||
|
which is not a measurement. Only one benchmark-to-beat ($92.21) is given,
|
||||||
|
globally, not per dimension.
|
||||||
|
|
||||||
|
**15. C3/C4 elimination is legitimate — not the sin the survey accuses the
|
||||||
|
workplan of. — no finding.** Both are eliminated on *categorical* grounds
|
||||||
|
(C3: not machine-readable, `statusLine` absent; C4: zero session/task
|
||||||
|
attribution, no admin key), not on unmeasured performance claims. You cannot
|
||||||
|
measure the cost-accuracy of an instrument that structurally cannot
|
||||||
|
attribute. C4's "authoritative — it *is* the invoice" is asserted rather than
|
||||||
|
measured, but it is used only to justify keeping C4 as an optional external
|
||||||
|
check, which is the conservative direction.
|
||||||
|
|
||||||
|
**16. Framing note: Fable is the cost majority, Opus is only the count
|
||||||
|
majority. — MINOR** Pinned per-model cost: fable $55.50 (60%) on 118
|
||||||
|
responses, opus $36.19 (39%) on 206, sonnet $0.52 (0.6%) on 14. "Pricing
|
||||||
|
everything at Fable 5's $10/$50 overstates the Opus and Sonnet majority" is
|
||||||
|
true by response count but reads as a cost claim, and in a document whose
|
||||||
|
thesis is that totals mislead, the 35%-of-calls / 60%-of-dollars inversion is
|
||||||
|
the more interesting fact.
|
||||||
|
|
||||||
|
**Bottom line:** the arithmetic is sound — every pinned figure and the entire
|
||||||
|
three-method table reproduce to the cent, which is a genuinely strong result.
|
||||||
|
The survey is not approvable as-is because of #1 (the dedup control fails its
|
||||||
|
own checksum), #2 (mislabelled table), and #3 (the acceptance target is
|
||||||
|
provably incomplete by $0.66 against the survey's own blind-spot finding).
|
||||||
|
#4–#7 must be folded in; the rest are corrections.
|
||||||
74
history/260731-cost-accounting-research.md
Normal file
74
history/260731-cost-accounting-research.md
Normal file
|
|
@ -0,0 +1,74 @@
|
||||||
|
# 2026-07-31 — how the cost-accounting survey was conducted
|
||||||
|
|
||||||
|
Trail for `research/CB-RES-0002-cost-accounting.md` (CB-WP-0002 T01).
|
||||||
|
Tier L, chaos d10 = 2 (no override).
|
||||||
|
|
||||||
|
## Method
|
||||||
|
|
||||||
|
No web sources. Every candidate was exercised locally against the
|
||||||
|
CB-WP-0001 session transcript
|
||||||
|
(`~/.claude/projects/-home-worsch-clay-borg/8cbd5701-….jsonl`, 5.1 MB,
|
||||||
|
2,040 lines) and the clay-borg git history. The survey's own rule —
|
||||||
|
measured beats cited — applied to itself, which is the only reason the
|
||||||
|
errors below were found.
|
||||||
|
|
||||||
|
## What was measured, in order
|
||||||
|
|
||||||
|
1. **Transcript shape.** Counted line types and collected the union of
|
||||||
|
`usage` keys. Found 654 usage-bearing assistant lines, and four
|
||||||
|
distinct `message.model` values where one was assumed.
|
||||||
|
2. **`usage.iterations`.** Suspected as a double-count source. Checked
|
||||||
|
whether iteration outputs sum to the top-level `output_tokens`: they
|
||||||
|
match in all 654 cases, and no message had >1 iteration. Cleared.
|
||||||
|
3. **`requestId` cardinality.** 346 distinct ids across 657 lines. This
|
||||||
|
was the finding that overturned the workplan's numbers.
|
||||||
|
4. **Dedup validation.** For all 208 multi-line groups, compared the
|
||||||
|
serialized `usage` object across the group: 208 identical, 0
|
||||||
|
differing, 0 mixed-model. Block-type patterns confirmed the split is
|
||||||
|
`thinking` / `text` / `tool_use`, i.e. a transcript-writer artifact.
|
||||||
|
5. **Re-priced three ways** (per-line all-Fable, per-line per-model,
|
||||||
|
deduped per-model) to isolate how much of the error came from each
|
||||||
|
mistake: $289.12 / $210.05 / $93.15.
|
||||||
|
6. **Hub API.** Called `get_token_summary` on CB-WP-0001's workplan.
|
||||||
|
401,100 tokens over 7 events for 9 tasks, all filed under Fable 5.
|
||||||
|
Compared against the transcript to quantify the gap (~8× on output).
|
||||||
|
7. **Status bar and billing API.** Checked `~/.claude/settings.json` for
|
||||||
|
`statusLine` (absent) and for an admin key (absent). Both eliminated
|
||||||
|
on availability before any further evaluation.
|
||||||
|
8. **Git boundaries.** Read `git log` with ISO timestamps to confirm
|
||||||
|
commit density and that subjects name tasks. Noted the UTC/+02:00
|
||||||
|
mismatch against transcript timestamps.
|
||||||
|
9. **Subagent tree.** Found `<session>/subagents/` while checking whether
|
||||||
|
`isSidechain` was ever true (it never is). Priced the one subagent
|
||||||
|
transcript: $0.66.
|
||||||
|
|
||||||
|
## Dead ends
|
||||||
|
|
||||||
|
- **`isSidechain` as the subagent signal.** It is present on every line
|
||||||
|
and `false` on every line in this session; it does not mark subagent
|
||||||
|
work in the main file because subagent work is not *in* the main file.
|
||||||
|
Looking for a flag wasted a step that a directory listing answered.
|
||||||
|
- **`tool-results/` sidecar directory** (340 KB) was inspected as a
|
||||||
|
possible cost source. It holds raw tool outputs for replay, carries no
|
||||||
|
usage data, and is not billed separately. Not a candidate.
|
||||||
|
|
||||||
|
## The finding worth carrying forward
|
||||||
|
|
||||||
|
The workplan that exists to make cost measurable **opened with a
|
||||||
|
mismeasured cost**, off by ~2.7×, and its two errors were of different
|
||||||
|
kinds: a format artifact (double-counted lines) and a rule that existed
|
||||||
|
but was not applied (per-model pricing, already mandated by
|
||||||
|
`specs/MetricsAndScenarios.md` §1a).
|
||||||
|
|
||||||
|
The second is the more uncomfortable one. InnerLoop v1.0's answer to bad
|
||||||
|
numbers is the positive control, and a positive control would not have
|
||||||
|
caught either of these — both sums were over real data that really
|
||||||
|
existed. What caught them was **re-deriving a number instead of quoting
|
||||||
|
it**. That belongs in the loop as its own rule, and is raised for T07:
|
||||||
|
*a number inherited from a previous pass is re-derived before it is used
|
||||||
|
as a target, or it is cited as unverified.*
|
||||||
|
|
||||||
|
Reported into CB-WP-0003 T01's rule-enforceability audit as well: this is
|
||||||
|
a fifth error instance, and it is **not** of the harness-does-nothing
|
||||||
|
class that T10 predicted. The class is "trusted arithmetic over real
|
||||||
|
data".
|
||||||
134
history/260731-cost-accounting-response.md
Normal file
134
history/260731-cost-accounting-response.md
Normal file
|
|
@ -0,0 +1,134 @@
|
||||||
|
# 2026-07-31 — response to the challenge on CB-RES-0002
|
||||||
|
|
||||||
|
Answers to `260731-cost-accounting-challenge.md`. Every contested number was
|
||||||
|
**re-derived here before being conceded** — the rule this survey argued for
|
||||||
|
applies to the challenge too, not only to the survey.
|
||||||
|
|
||||||
|
Outcome: **16 findings, 15 conceded, 1 partially answered, 0 rejected.**
|
||||||
|
Survey moves to `status: approved` after the folds below.
|
||||||
|
|
||||||
|
## Blocking — conceded
|
||||||
|
|
||||||
|
**#1 dedup table fails its own checksum.** Confirmed: the group-size
|
||||||
|
distribution is `{1:140, 2:120, 3:74, 4:6, 5:5, 6:1}` = 346 groups / 657
|
||||||
|
lines, and multi-line groups are **206**. The survey said 76 and 208. The
|
||||||
|
reviewer is right that this is the worst possible place for an arithmetic
|
||||||
|
slip: the table *is* the positive control for the load-bearing dedup
|
||||||
|
assumption. Corrected in the survey.
|
||||||
|
|
||||||
|
Worth naming plainly: a document whose thesis is "re-derive, don't quote"
|
||||||
|
shipped a hand-typed table that did not sum. The finding survives; the
|
||||||
|
discipline did not extend to the evidence *for* the finding.
|
||||||
|
|
||||||
|
**#2 multi-model table mislabelled.** Confirmed: `382/250/24/1` are per-line
|
||||||
|
counts under a header reading "API responses (deduped)". Deduped over the
|
||||||
|
same prefix: **opus 213, fable 118, sonnet 14, synthetic 1 = 346**. The
|
||||||
|
reviewer's 206 for opus was the per-line figure quoted from the survey's own
|
||||||
|
text; the correct deduped value is 213. Corrected, and the table now shows
|
||||||
|
both columns so the 1.9× inflation is visible rather than asserted.
|
||||||
|
|
||||||
|
**#3 the acceptance target excludes the subagent.** Confirmed and the most
|
||||||
|
consequential finding. The subagent ran `2026-07-30T23:11:42Z–23:14:12Z`,
|
||||||
|
which is **inside** the pin window. Re-derived:
|
||||||
|
|
||||||
|
```text
|
||||||
|
main transcript $92.2075 (339 responses)
|
||||||
|
subagent $ 0.6580 (7 responses)
|
||||||
|
TRUE TOTAL $92.8655
|
||||||
|
```
|
||||||
|
|
||||||
|
The survey stated the C1 blind-spot correctly and then set a benchmark that
|
||||||
|
only a collector *exhibiting* that blind-spot could hit. A correct collector
|
||||||
|
would have failed the acceptance test. The target is now **$92.87**, stated
|
||||||
|
as the sum of both components so a collector reading one tree can be
|
||||||
|
diagnosed rather than merely failed.
|
||||||
|
|
||||||
|
## Material — conceded
|
||||||
|
|
||||||
|
**#4 5m vs 1h cache-write pricing.** Confirmed in substance. The subagent's
|
||||||
|
cache writes are **entirely 5m** (37,467 deduped tokens, 1h = 0). The
|
||||||
|
collector's arithmetic was already correct — it prices the `ephemeral_5m` and
|
||||||
|
`ephemeral_1h` fields separately, and $0.658 reflects that. What was missing
|
||||||
|
is the *rule*: the survey named both fields without stating that the
|
||||||
|
top-level `cache_creation_input_tokens` aggregate must never be priced at a
|
||||||
|
single multiplier. Mispricing that aggregate at 1h yields $0.939, a +43%
|
||||||
|
error. Added as an explicit contract for T03 and a risk-list entry, since
|
||||||
|
fan-out passes are where 5m writes dominate.
|
||||||
|
|
||||||
|
**#5 concurrent sessions are already in the data.** Conceded, with an
|
||||||
|
observation the reviewer could not have had: session `f1eb1147` is *this
|
||||||
|
review's own session* — the background job running the workplan. The survey
|
||||||
|
claimed the interleaving risk was "not exercised here"; it was being
|
||||||
|
exercised as the sentence was written. ~$12 across a 4h13m overlap cannot be
|
||||||
|
separated by a wall-clock join. This kills naive wall-clock attribution
|
||||||
|
outright and is now a stated ADR constraint: **attribution must key on
|
||||||
|
`sessionId`, not only time.** The survey's C1 section also now names the
|
||||||
|
second transcript.
|
||||||
|
|
||||||
|
**#6 a third of the money has no task tag.** Re-derived independently with
|
||||||
|
`(prev commit, this commit]` intervals over the 33 pinned commits:
|
||||||
|
**$30.32 of $92.21 = 33% untagged**, 1 empty interval, 0 orphans at either
|
||||||
|
end. (The reviewer got $29.58/32%; the small delta is regex scope — theirs
|
||||||
|
matched `\bT\d\d\b`, mine the same, differing on one boundary commit. Same
|
||||||
|
conclusion, and the survey now carries the re-derived 33%.) C5's claim that
|
||||||
|
the loop "commits per task iteration with the task in the subject" holds for
|
||||||
|
14 of 33 commits. This is now sized in the survey rather than left as a
|
||||||
|
qualitative hazard, and the unattributed-remainder line in T04 has a known
|
||||||
|
target to beat.
|
||||||
|
|
||||||
|
**#7 the ~87% figure matches nothing.** Conceded — it was a stale
|
||||||
|
hand-carried number from before the pin. The defensible figures are **64.6%**
|
||||||
|
(cache read as a share of cost) and **88.4%** (all cache as a share of cost).
|
||||||
|
C2's argument is unaffected: the hub schema can express neither. Corrected to
|
||||||
|
88.4% with the basis named.
|
||||||
|
|
||||||
|
## Minor — conceded
|
||||||
|
|
||||||
|
- **#8** `<synthetic>` has a valid `requestId` and an all-zero `usage`, not a
|
||||||
|
missing one. Dollar impact is exactly $0.00, but the two guard styles
|
||||||
|
(`if not usage` vs `if model not in prices`) branch differently, so T04's
|
||||||
|
contract now names which it uses.
|
||||||
|
- **#9** the "654 usage-bearing lines" count does not reproduce; all pinned
|
||||||
|
assistant lines carry `usage`. Conclusion stood, count was wrong. Removed.
|
||||||
|
- **#10** the repo has two commit-offsets (`+02:00` ×34, `+00:00` ×1).
|
||||||
|
Guidance changed from "off by the offset" to "parse `%cI` and convert".
|
||||||
|
- **#11** commit spacing is min 0.1 / p50 6.9 / p90 17.7 / **max 36.8**
|
||||||
|
minutes, not "3–15". The max is the number that matters.
|
||||||
|
- **#12** interval convention now stated explicitly: *ending-at-commit*,
|
||||||
|
`(prev, this]`, which is the only one consistent with commit-at-end-of-task.
|
||||||
|
- **#13** the sonnet intro rate is a TOML comment, so a collector silently
|
||||||
|
ignores it ($0.17 at the pin). Raised as a price-sheet schema defect —
|
||||||
|
time-boxed rates need a representation. Same class of defect the survey
|
||||||
|
levels at C2, as the reviewer notes.
|
||||||
|
- **#14** the "Accuracy" row is a fifth non-rubric axis with no beatable
|
||||||
|
value. Folded into D2 with a number attached.
|
||||||
|
- **#16** accepted as the better framing, and promoted out of "minor":
|
||||||
|
**fable is 60% of cost on 35% of calls; opus is 39% of cost on 60% of
|
||||||
|
calls.** In a document arguing that totals mislead, a count-majority
|
||||||
|
presented where a cost-majority was meant is the same error one level down.
|
||||||
|
|
||||||
|
## Partially answered
|
||||||
|
|
||||||
|
**#15 (reviewer found no fault) — noted, with one correction accepted.**
|
||||||
|
The reviewer is right that C3/C4 elimination is categorical rather than
|
||||||
|
performance-based and therefore legitimate. Accepted. The one thing folded
|
||||||
|
in anyway: C4's "authoritative — it *is* the invoice" is an assertion, and
|
||||||
|
is now marked as such rather than reading like a measurement.
|
||||||
|
|
||||||
|
## What this round cost, and whether it paid
|
||||||
|
|
||||||
|
Reviewer: 56,882 tokens / 14 tool uses / 330 s. Priced against the sheet,
|
||||||
|
~$0.60 — the same order as CB-WP-0001's review at $0.66.
|
||||||
|
|
||||||
|
It found three approval-blocking defects, one of which (#3) would have made
|
||||||
|
T05's acceptance test reward a *broken* collector and fail a correct one.
|
||||||
|
That is the second consecutive pass where a ~$0.60 review caught something a
|
||||||
|
$92 pass had missed. The economics are not close, and this belongs in the
|
||||||
|
CB-WP-0003 T03 argument for pointing review at measurement rather than prose.
|
||||||
|
|
||||||
|
The uncomfortable pattern across both rounds: **the survey's arithmetic over
|
||||||
|
raw data was flawless — every one of the reviewer's ~20 spot-checks
|
||||||
|
reproduced to the cent — and its hand-written prose tables were not.** The
|
||||||
|
errors were all in numbers typed by hand into markdown after the computation,
|
||||||
|
never in the computation. That is a mechanical, fixable class: tables that a
|
||||||
|
tool can emit should be emitted by the tool. Raised for T03 and T06.
|
||||||
112
history/260731-cost-accounting-retrospective.md
Normal file
112
history/260731-cost-accounting-retrospective.md
Normal file
|
|
@ -0,0 +1,112 @@
|
||||||
|
# 2026-07-31 — retrospective: what CB-WP-0002 taught the loop
|
||||||
|
|
||||||
|
Pass: CB-WP-0002, cost accounting. Produced `tools/cb-cost.py`,
|
||||||
|
`specs/CostAccounting.md`, ADR-0003, CB-EV-0002, and InnerLoop **v1.1**.
|
||||||
|
|
||||||
|
## The question the workplan asked
|
||||||
|
|
||||||
|
> CB-WP-0001 produced a fully specified metric that could not be computed,
|
||||||
|
> and v1.0's "every metric names its instrument" rule was written to stop
|
||||||
|
> that. Did it?
|
||||||
|
|
||||||
|
**No.** The rule was satisfied completely and the metric was still wrong.
|
||||||
|
|
||||||
|
`specs/CostAccounting.md` AC-1 named `cb-cost --pin fc76445` as its
|
||||||
|
instrument — a command that did not exist when the row was written — and
|
||||||
|
set the target to **$92.87**, computed by hand. When the tool was built one
|
||||||
|
task later it returned **$93.32**. The hand computation carried a dedup bug
|
||||||
|
that the tool's own positive control caught on first contact.
|
||||||
|
|
||||||
|
So the v1.0 rule stops "a metric with no instrument named" and does not stop
|
||||||
|
"a metric whose target the instrument never produced". The distinction did
|
||||||
|
not exist when the rule was written because the failure it was written for
|
||||||
|
was the coarser one.
|
||||||
|
|
||||||
|
**v1.1's answer:** the instrument must exist and the target must come out of
|
||||||
|
it. Where the tool is built later in the pass, the target is `provisional:`
|
||||||
|
until the tool emits it, and the spec is amended to whatever it returns —
|
||||||
|
not the reverse.
|
||||||
|
|
||||||
|
## The number, and its four corrections
|
||||||
|
|
||||||
|
The cost of CB-WP-0001 was stated four times before it was right:
|
||||||
|
|
||||||
|
| value | method | what was wrong |
|
||||||
|
|---|---|---|
|
||||||
|
| $248.46 | inherited, quoted | per-line summation (~1.9×) + single-model pricing |
|
||||||
|
| $92.21 | re-derived, deduped, per-model | omitted the subagent tree |
|
||||||
|
| $92.87 | + subagent at first-wins dedup | `output_tokens` is a running count |
|
||||||
|
| **$93.32** | emitted by `cb-cost` | — |
|
||||||
|
|
||||||
|
Each correction was found by a different mechanism, and this is the useful
|
||||||
|
part:
|
||||||
|
|
||||||
|
1. **$248.46 → $92.21** by *re-deriving instead of quoting*. Now a v1.1
|
||||||
|
rule.
|
||||||
|
2. **$92.21 → $92.87** by *adversarial review*, which noticed the target
|
||||||
|
contradicted the survey's own blind-spot finding. Cost: $1.11.
|
||||||
|
3. **$92.87 → $93.32** by the *positive control*, which fired on real data
|
||||||
|
and refused to print a number.
|
||||||
|
|
||||||
|
No single mechanism found more than one. That is the argument for keeping
|
||||||
|
all three rather than consolidating.
|
||||||
|
|
||||||
|
## The error class the last pass predicted, and what actually happened
|
||||||
|
|
||||||
|
CB-WP-0003 T10 predicted the next error would be the harness-does-nothing
|
||||||
|
class, since four instances had already been seen. It was not — twice.
|
||||||
|
|
||||||
|
- The two errors in $248.46 were **trusted arithmetic over real data**. Both
|
||||||
|
sums ran over data that genuinely existed. A positive control asserting
|
||||||
|
"did this harness do work?" would have answered yes, correctly, and
|
||||||
|
reported a wrong number.
|
||||||
|
- The error in $92.87 was **a property verified on the large sample and
|
||||||
|
assumed on the small one**. The dedup invariant was checked on the main
|
||||||
|
transcript (206/206 groups) by the survey, and independently re-checked by
|
||||||
|
the adversarial reviewer, who also used the main transcript. It is false
|
||||||
|
in the 8-response subagent tree that neither examined separately.
|
||||||
|
|
||||||
|
The second is the one worth carrying forward, because **review structurally
|
||||||
|
cannot catch it**: the reviewer's job is to re-derive the author's claims,
|
||||||
|
and re-deriving on the same sample reproduces the same blind spot. Only an
|
||||||
|
assertion running over *all* the data at execution time catches it. That is
|
||||||
|
now a v1.1 rule (`--self-test` on every reporting tool, run before the
|
||||||
|
number).
|
||||||
|
|
||||||
|
## What the pass bought, in its own units
|
||||||
|
|
||||||
|
The capability measures itself, which is the first time this project has
|
||||||
|
been able to say what a pass cost while the pass was running:
|
||||||
|
|
||||||
|
- **CB-WP-0001 (measured retroactively):** $93.32 pinned at `fc76445`.
|
||||||
|
88.0% cache, 11.9% output, 249:1 context re-read to text written.
|
||||||
|
- **Adversarial review, this pass:** $1.11 — ~1% of the pass it reviewed —
|
||||||
|
and it found three approval-blocking defects. Second consecutive pass
|
||||||
|
where that trade was decisive. Two data points now support CB-WP-0003 T03.
|
||||||
|
- **`/compact`, measured for the first time:** 542,991 → 19,974 tokens, a
|
||||||
|
27× context reduction. Cost per turn fell 3.1×, from $0.457 across the
|
||||||
|
136 turns before it to $0.149 across the 202 after.
|
||||||
|
|
||||||
|
That last number changes advice this project was about to give itself.
|
||||||
|
CB-WP-0003 T04 was going to prescribe one task per session on the theory
|
||||||
|
that long sessions are quadratic. They are **bounded**-quadratic: cost grows
|
||||||
|
with context between compactions and resets at each one. The failure mode is
|
||||||
|
a long *uncompacted* session, and whether a fresh session beats a compaction
|
||||||
|
is now a measurable question rather than a matter of taste. T04 should
|
||||||
|
measure it before prescribing.
|
||||||
|
|
||||||
|
## Raised, not resolved
|
||||||
|
|
||||||
|
- **The price sheet cannot express a time-boxed rate.** Sonnet's intro price
|
||||||
|
is a TOML comment. Costs $0.17 today (0.19%); becomes a real error on
|
||||||
|
**2026-08-31** when the intro rate expires and the comment and the data
|
||||||
|
disagree in the other direction.
|
||||||
|
- **32.5% of spend has no task**, because 19 of 33 commits carry no `T##`
|
||||||
|
tag. Reported as its own line rather than hidden, but the underlying fix
|
||||||
|
is commit hygiene, not tooling.
|
||||||
|
- **Outside this repo:** the `ralph-workplan` skill instructs agents to read
|
||||||
|
token counts from the Claude Code status bar. That is not readable from a
|
||||||
|
tool call, so an agent asked for it estimates instead — the proximate
|
||||||
|
cause of the hub holding 401,100 tokens for a workplan that actually
|
||||||
|
consumed 80.9M. Flagged for the maintainer; `make cost` is the authority
|
||||||
|
in the meantime.
|
||||||
333
research/CB-RES-0002-cost-accounting.md
Normal file
333
research/CB-RES-0002-cost-accounting.md
Normal file
|
|
@ -0,0 +1,333 @@
|
||||||
|
# CB-RES-0002: agentic cost accounting
|
||||||
|
|
||||||
|
capability: meta.loop.cost-accounting
|
||||||
|
status: approved # adversarial review 2026-07-31: 16 findings, 15 conceded
|
||||||
|
tier: L (structural L, chaos d10=2 → no override)
|
||||||
|
runnable-baseline: invoked — every candidate below was exercised against the
|
||||||
|
CB-WP-0001 session on this machine, not cited
|
||||||
|
review-trail: history/260731-cost-accounting-{research,challenge,response}.md
|
||||||
|
|
||||||
|
Survey of instruments that can attribute the USD cost of agentic work to a
|
||||||
|
unit of work, so that M-D2-CST (`specs/MetricsAndScenarios.md` §1a) becomes
|
||||||
|
computable. CB-WP-0001 specified that metric completely and recorded it as
|
||||||
|
*uncomputable*; the premise of this workplan is that the data existed the
|
||||||
|
whole time.
|
||||||
|
|
||||||
|
That premise survives. The workplan's **numbers do not** — see §Correction.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Correction to this workplan's own Purpose section
|
||||||
|
|
||||||
|
CB-WP-0002's Purpose reports the CB-WP-0001 session at **$248.46**, from
|
||||||
|
131,863,164 cache-read tokens priced at Fable 5. Both halves are wrong, and
|
||||||
|
in the same direction — too high. The survey found this by re-deriving the
|
||||||
|
number rather than adopting it.
|
||||||
|
|
||||||
|
**Error 1 — per-line summation double-counts.** A single API response is
|
||||||
|
written to the transcript as *several* JSONL lines, split by content block
|
||||||
|
(`thinking`, `text`, `tool_use`), and **every one of those lines repeats the
|
||||||
|
complete `usage` object**. Measured on the CB-WP-0001 transcript: 657
|
||||||
|
assistant lines carry only 346 distinct `requestId`s. Group sizes run 1–6:
|
||||||
|
|
||||||
|
| lines per requestId | 1 | 2 | 3 | 4 | 5 | 6 | total |
|
||||||
|
|---|---|---|---|---|---|---|---|
|
||||||
|
| groups | 140 | 120 | 74 | 6 | 5 | 1 | **346** |
|
||||||
|
| lines | 140 | 240 | 222 | 24 | 25 | 6 | **657** |
|
||||||
|
|
||||||
|
Both rows are checksummed against the file: groups sum to 346, lines to 657.
|
||||||
|
|
||||||
|
Positive control on the dedup: across all 206 multi-line groups the `usage`
|
||||||
|
object is byte-identical (206/206 identical, 0 differing), and no group
|
||||||
|
mixes models. Every assistant line carries a `requestId` — there is no null
|
||||||
|
key for `setdefault` to collapse. The duplication is a transcript-format
|
||||||
|
artifact, not repeated billing. Summing per line inflates by ≈1.9×.
|
||||||
|
|
||||||
|
**Error 2 — single-model pricing on a multi-model session.** The session ran
|
||||||
|
three models, not one. Both columns are shown because the gap between them
|
||||||
|
*is* error 1:
|
||||||
|
|
||||||
|
| model | JSONL lines | API responses (deduped) | pinned cost |
|
||||||
|
|---|---|---|---|
|
||||||
|
| claude-opus-5 | 382 | 213 | $36.19 (39%) |
|
||||||
|
| claude-fable-5 | 250 | 118 | **$55.50 (60%)** |
|
||||||
|
| claude-sonnet-5 | 24 | 14 | $0.52 (0.6%) |
|
||||||
|
| `<synthetic>` | 1 | 1 | $0.00 |
|
||||||
|
| **total** | **657** | **346** | |
|
||||||
|
|
||||||
|
§1a *already required* per-model pricing; the Purpose section did not apply
|
||||||
|
its own rule. Note the inversion, which matters more than the correction:
|
||||||
|
**Fable is 35% of the calls and 60% of the dollars; Opus is 60% of the calls
|
||||||
|
and 39% of the dollars.** A count-majority is not a cost-majority — the
|
||||||
|
error this document is about, one level down.
|
||||||
|
|
||||||
|
The `<synthetic>` entry is an error placeholder carrying a valid `requestId`
|
||||||
|
and a complete **all-zero** `usage` object, not a missing one. Its dollar
|
||||||
|
impact is exactly $0.00, but a collector guarding on `if not usage` and one
|
||||||
|
guarding on `if model not in prices` take different branches; T04 must name
|
||||||
|
which.
|
||||||
|
|
||||||
|
**Corrected totals** for the same transcript, all three methods run over
|
||||||
|
the identical unpinned line set so the methods are comparable:
|
||||||
|
|
||||||
|
| method | responses | output | cache read | cost |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| per-line, all-Fable (the Purpose method) | 657 | 700,690 | 161,408,840 | $289.12 |
|
||||||
|
| per-line, per-model | 657 | 700,690 | 161,408,840 | $210.05 |
|
||||||
|
| **deduped, per-model (correct)** | **346** | **318,230** | **81,100,498** | **$93.15** |
|
||||||
|
|
||||||
|
**Third error, found while verifying the second: the transcript is a live
|
||||||
|
file.** Re-running the deduped figure minutes later returned 356
|
||||||
|
responses and $94.04 — this survey's own session appends to the same
|
||||||
|
JSONL it is measuring. An unpinned total is not a repeatable number. The
|
||||||
|
acceptance target is therefore pinned by timestamp:
|
||||||
|
|
||||||
|
| CB-WP-0001, pinned ≤ `2026-07-31T02:17:59Z` (commit `fc76445`) | value |
|
||||||
|
|---|---|
|
||||||
|
| responses | 339 (206 opus-5, 118 fable-5, 14 sonnet-5, 1 synthetic) |
|
||||||
|
| output | 313,900 tok → $10.66 |
|
||||||
|
| cache read | 80,453,702 tok → $59.59 |
|
||||||
|
| cache write 1h | 1,672,854 tok → $21.95 |
|
||||||
|
| input | 676 tok → $0.00 |
|
||||||
|
| main transcript | **$92.21** — 88.4% cache, 256:1 cache-read:output |
|
||||||
|
| + subagent tree (7 responses, ran 23:11–23:14Z, inside the pin) | $1.11 |
|
||||||
|
| **TRUE TOTAL** | **$93.32** |
|
||||||
|
|
||||||
|
*(The subagent figure was $0.66 when this survey was written. T04's
|
||||||
|
positive control found the cause: `output_tokens` is a running count in
|
||||||
|
the `subagents/` tree, so first-wins dedup under-counted it. See
|
||||||
|
`specs/CostAccounting.md` CA-02a.)*
|
||||||
|
|
||||||
|
The subagent line is not a footnote. C1's blind-spot finding says a
|
||||||
|
collector reading only the main file under-reports; a target of $92.21 would
|
||||||
|
have been hit only by a collector *with* that blind spot, and failed by a
|
||||||
|
correct one. The acceptance target is **$93.32, stated as its two
|
||||||
|
components**, so a collector reading one tree is diagnosed rather than
|
||||||
|
merely failed.
|
||||||
|
|
||||||
|
The reported figure was **~2.7× the real cost**. This is the fourth
|
||||||
|
instance of the harness-does-nothing error class from
|
||||||
|
`history/260731-inner-loop-retrospective.md`, wearing a new coat: not a
|
||||||
|
harness that measured nothing, but an arithmetic that measured the same
|
||||||
|
thing twice. Both produce a number that looks fine.
|
||||||
|
|
||||||
|
The qualitative headline survives the correction and gets stronger: cache
|
||||||
|
reads are **81.1M tokens against 318k of output**, ~255:1. Cost in an
|
||||||
|
agentic loop is context × turns.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Candidates
|
||||||
|
|
||||||
|
### C1 — Session transcript JSONL
|
||||||
|
|
||||||
|
`~/.claude/projects/<slug>/<session>.jsonl`, one JSON object per line.
|
||||||
|
Assistant lines carry `message.usage` with exact billing counters:
|
||||||
|
`input_tokens`, `output_tokens`, `cache_read_input_tokens`, and
|
||||||
|
`cache_creation.{ephemeral_1h,ephemeral_5m}_input_tokens`, plus
|
||||||
|
`message.model`, `requestId`, and an ISO-8601 `timestamp`.
|
||||||
|
|
||||||
|
- **Granularity:** per API response, once deduplicated by `requestId`.
|
||||||
|
- **Accuracy:** exact — these are the counters the invoice is computed from.
|
||||||
|
There is no sampling or rounding.
|
||||||
|
- **Verified non-issue:** `usage.iterations[]` is a sub-breakdown, not an
|
||||||
|
additional charge. Every assistant line carries `usage`; the iteration
|
||||||
|
outputs sum exactly to the top-level `output_tokens` in every case, and
|
||||||
|
no message had more than one iteration. Summing `iterations` *instead of*
|
||||||
|
the top-level fields is safe; summing *both* would double-count.
|
||||||
|
- **Cache-write rates are per-TTL and must not be aggregated.**
|
||||||
|
`cache_creation_input_tokens` equals `ephemeral_5m + ephemeral_1h` in all
|
||||||
|
775 usage lines across both transcripts, and the two bill at different
|
||||||
|
multipliers (1.25 vs 2.0). Pricing the top-level aggregate at a single
|
||||||
|
rate is a silent error: the subagent's writes are **entirely 5m** (37,467
|
||||||
|
tokens), and pricing them at 1h inflates that transcript by **+43%**
|
||||||
|
($0.658 → $0.939). The main session happens to be all-1h, so the pinned
|
||||||
|
total is insensitive — but fan-out passes are exactly where 5m dominates.
|
||||||
|
- **Survives compaction:** yes. `/compact` writes a summary message into the
|
||||||
|
same file (`isCompactSummary`, `compactMetadata`) and the session
|
||||||
|
continues; no usage is lost. Compaction is visible as an event, so its
|
||||||
|
cost is itself measurable.
|
||||||
|
- **Attribution:** none built in — a transcript is a flat message stream
|
||||||
|
with timestamps. It must be joined against an external time index.
|
||||||
|
- **Blind spot 1 — subagents are a separate tree.** Subagent cost is **not**
|
||||||
|
in the main transcript. `isSidechain` is `false` on all 657 lines;
|
||||||
|
subagent work lives in `<session>/subagents/agent-*.jsonl` with an
|
||||||
|
`agent-*.meta.json` naming the agentType and model. CB-WP-0001 spawned one
|
||||||
|
(the adversarial review). A collector reading only the main file silently
|
||||||
|
under-reports.
|
||||||
|
- **Blind spot 2 — one repo, several transcripts, overlapping in time.**
|
||||||
|
The project directory holds more than one session. `f1eb1147` overlaps
|
||||||
|
`8cbd5701` for **4 h 13 m**, carrying ~$6 on each side — ~$12 that no
|
||||||
|
wall-clock join can separate, beginning 14 seconds after the pin. A
|
||||||
|
collector must therefore key attribution on **`sessionId`, not only time**,
|
||||||
|
and must enumerate every transcript for the repo rather than one file.
|
||||||
|
|
||||||
|
### C2 — Custodian State Hub token API
|
||||||
|
|
||||||
|
`record_token_event`, the `update_task_status` token tiers, and
|
||||||
|
`get_token_summary`. Exercised against CB-WP-0001's workplan
|
||||||
|
(`a1b434dc-…`), which returned:
|
||||||
|
|
||||||
|
```text
|
||||||
|
tokens_in 362,000 tokens_out 39,100 event_count 7 by model: claude-fable-5
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Granularity:** per task — the best of any candidate, and the only one
|
||||||
|
that is natively *about* the unit of work.
|
||||||
|
- **Accuracy:** poor, and structurally so. Three independent defects:
|
||||||
|
1. **The schema has no cache fields.** `tokens_in`/`tokens_out` cannot
|
||||||
|
represent the finding this workplan exists to report. Cache read alone
|
||||||
|
is **64.6% of cost** and all cache is **88.4%**; the hub cannot express
|
||||||
|
either at any fidelity.
|
||||||
|
2. **The recorded numbers are estimates.** 7 events for 9 tasks, at
|
||||||
|
round figures — the skill's Tier-3 heuristic (1000/500) and Tier-1
|
||||||
|
eyeball estimates. Against a deduped transcript output of 318,230,
|
||||||
|
the hub's 39,100 is off by ~8×; against total input it is off by
|
||||||
|
~450×.
|
||||||
|
3. **Model attribution is wrong.** Everything is filed under
|
||||||
|
`claude-fable-5` on a session that was majority Opus 5.
|
||||||
|
- **Survives compaction:** yes — it is server-side and independent of the
|
||||||
|
client.
|
||||||
|
- **Verdict:** durable and task-shaped, but its numbers are unusable as a
|
||||||
|
cost source. Its role is as a **sink** for numbers computed elsewhere,
|
||||||
|
not a source. Even as a sink it can only carry a lossy projection until
|
||||||
|
the schema grows cache fields.
|
||||||
|
|
||||||
|
### C3 — Claude Code status bar
|
||||||
|
|
||||||
|
- **Granularity:** whole session, live.
|
||||||
|
- **Accuracy:** unknown and unauditable — it is rendered text.
|
||||||
|
- **Machine-readable:** no. Not configured here (`statusLine` is absent
|
||||||
|
from `~/.claude/settings.json`), and it is not reachable from inside a
|
||||||
|
tool call regardless.
|
||||||
|
- **Verdict:** eliminated. The ralph-workplan skill's "read tokens from the
|
||||||
|
status bar" instruction is the proximate cause of C2's bad numbers — it
|
||||||
|
asks an agent to report a figure it cannot read, and an agent that cannot
|
||||||
|
read it estimates instead. This should be raised against the skill.
|
||||||
|
|
||||||
|
### C4 — Anthropic usage / billing API
|
||||||
|
|
||||||
|
- **Granularity:** organization and API-key, by day.
|
||||||
|
- **Accuracy:** authoritative — it *is* the invoice.
|
||||||
|
- **Attribution:** none to a task, and none to a session. Cannot separate
|
||||||
|
clay-borg from the other twenty-plus projects on this machine.
|
||||||
|
- **Availability:** requires an admin key; none is configured here.
|
||||||
|
- **Verdict:** not usable for M-D2-CST, but valuable as an **external
|
||||||
|
reconciliation check** if an admin key is ever provisioned — it is the
|
||||||
|
only candidate that can catch a systematic error in C1's price model.
|
||||||
|
Left as a stated non-dependency.
|
||||||
|
|
||||||
|
### C5 — Git commit history (attribution index, not a cost source)
|
||||||
|
|
||||||
|
Not a cost instrument; the missing half of C1. The loop already commits per
|
||||||
|
task iteration with the task in the subject line, giving durable, timestamped
|
||||||
|
boundaries at exactly the granularity M-D2-CST wants:
|
||||||
|
|
||||||
|
```text
|
||||||
|
a09d76f 2026-07-31T02:14:34+02:00 T08 iter 1: scenario runner executes; …
|
||||||
|
b58a913 2026-07-31T02:19:34+02:00 T08 iter 2: Reveal, Resolve, End; …
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Interval convention:** *ending-at-commit*, `(prev_commit, this_commit]`.
|
||||||
|
This is the only convention consistent with commit-at-end-of-task; under
|
||||||
|
the alternative every task's cost shifts one interval.
|
||||||
|
- Spacing over the 33 pinned commits: min 0.1, p50 **6.9**, p90 17.7, max
|
||||||
|
**36.8** minutes. Usually finer than a task; the 37-minute max is not.
|
||||||
|
- Durable, versioned, and free; requires no change to how work is done.
|
||||||
|
- **Coverage is the real limit, and it is sized:** only **14 of 33** commits
|
||||||
|
name a task (`T##`) in the subject. The other 19 — `chore(consistency)`,
|
||||||
|
`CI:`, `AM-4:`, workplan additions — hold **$30.32 of $92.21, or 33% of
|
||||||
|
cost**. Attribution to a *task* therefore covers two-thirds of spend at
|
||||||
|
best; the remainder is real work that must be reported as its own line,
|
||||||
|
not discarded.
|
||||||
|
- **Boundary conditions verified clean:** 0 responses before the first
|
||||||
|
commit, 0 after the last, exactly 1 empty interval (`Initial commit`).
|
||||||
|
- **Known hazards:** offsets are not uniform — `git log` shows `+02:00` ×34
|
||||||
|
and `+00:00` ×1, so a collector must parse `%cI` and convert, never
|
||||||
|
subtract a fixed offset. Commits made outside a session create empty
|
||||||
|
intervals.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Baselines (benchmark-to-beat)
|
||||||
|
|
||||||
|
| Dimension | Baseline holder | Metric | Value | Provenance |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| D1 ease of specification | C2 hub | fields needed to record a task's cost | 4 (`task_id`, `tokens_in`, `tokens_out`, `model`) — but cannot express cache | measured (API schema) |
|
||||||
|
| D2 efficiency | C2 hub | cost of producing a number | ~0 (one API call) — number is an estimate, off by ~8× on output | measured |
|
||||||
|
| D2 efficiency | C1 transcript | cost of producing a number | one file read, 5.1 MB, ~1 s; exact | measured |
|
||||||
|
| D3 speed | C1 transcript | parse of a full session | 2,040 lines / 5.1 MB in <1 s in CPython | measured |
|
||||||
|
| D2 efficiency (**the deciding row**) | C1 transcript | **error against the billing counters** | **$0.00 — the transcript *is* the counter set; C2's error on the same work is $92.21 − $0.03 recorded ≈ 100%** | measured |
|
||||||
|
| D4 optionality | C4 billing API | authority | invoice-grade, zero attribution, admin key absent (asserted, not measured — used only to keep C4 as an optional check) | availability measured |
|
||||||
|
|
||||||
|
**Benchmark-to-beat for the collector:** reproduce **$93.32** for repo
|
||||||
|
`clay-borg` pinned at `2026-07-31T02:17:59Z` — as its two components,
|
||||||
|
**$92.21 main transcript + $1.11 subagent tree** — from the committed price
|
||||||
|
sheet, with the unattributed remainder reported as its own line (expected
|
||||||
|
**33%**, $30.32) and reconciliation asserted rather than assumed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Verdict
|
||||||
|
|
||||||
|
**C1 (transcript) leads on accuracy and is the only exact candidate.**
|
||||||
|
**C5 (git commits) supplies the attribution index C1 lacks.** C2 is the
|
||||||
|
durable sink. C3 is eliminated. C4 is an optional external check.
|
||||||
|
|
||||||
|
The expected shape is therefore: enumerate **every** transcript for the repo
|
||||||
|
including the `subagents/` tree → dedup by `requestId` → price per message at
|
||||||
|
its own model's rate and **per cache TTL** from
|
||||||
|
`benchmarks/baselines/model-prices.toml` → attribute to a task by joining
|
||||||
|
`(prev_commit, this_commit]` intervals **within a `sessionId`** → emit
|
||||||
|
per-task cost, an unattributed-remainder line, and a composition breakdown →
|
||||||
|
push a lossy summary to C2.
|
||||||
|
|
||||||
|
**What none of them do well — the surpass opportunity.** Every candidate
|
||||||
|
reports *totals*. None reports **composition**, and composition is where
|
||||||
|
the actionable finding lives: 81.1M cache-read tokens against 318k of
|
||||||
|
output means cost is driven by how much context is re-read per turn, which
|
||||||
|
no total can show. A metric that had reported only dollars would have been
|
||||||
|
correct and useless.
|
||||||
|
|
||||||
|
**Risks in the baselines themselves.**
|
||||||
|
|
||||||
|
1. **The $248.46 figure was wrong and was nearly adopted as this
|
||||||
|
workplan's acceptance target.** T05's reconciliation test must be
|
||||||
|
against a number this survey re-derived, not against the Purpose
|
||||||
|
section. The Purpose section needs correcting.
|
||||||
|
2. **Dedup is load-bearing.** If the transcript format ever splits one
|
||||||
|
response across two `requestId`s, dedup silently under-reports —
|
||||||
|
the opposite error, and the more dangerous one. The collector must
|
||||||
|
assert its dedup assumption (identical usage within a group) at
|
||||||
|
runtime rather than trusting this survey's one-time check.
|
||||||
|
3. **Subagent transcripts are a separate tree.** Measured: CB-WP-0001's
|
||||||
|
one subagent (adversarial review, Fable 5, 7 responses, 158,096 cache
|
||||||
|
reads) cost **$1.11**, invisible to any collector reading only the
|
||||||
|
main file. Small here; not small for a pass that fans out.
|
||||||
|
4. **The price sheet has a 90-day staleness rule** (§1a) and no automated
|
||||||
|
check. Every number this capability produces inherits that.
|
||||||
|
5. **The transcript is append-live.** It is written by the session that
|
||||||
|
reads it, so any total is a reading at an instant. Every committed
|
||||||
|
number from this capability states its pin (timestamp or commit), and
|
||||||
|
the collector takes a pin argument rather than defaulting to "all".
|
||||||
|
6. **Concurrent sessions are not hypothetical — they are in this data.**
|
||||||
|
`f1eb1147` and `8cbd5701` overlap for 4 h 13 m on the same repo, ~$12
|
||||||
|
inseparable by wall-clock. (The session that wrote the first draft of
|
||||||
|
this sentence *was* the overlap.) Attribution keys on `sessionId` first,
|
||||||
|
time second. Compaction inside a task is safe — it stays in one file and
|
||||||
|
one session.
|
||||||
|
7. **A third of spend has no task, structurally.** 19 of 33 commits carry no
|
||||||
|
task tag; $30.32 of $92.21. Any per-task cost table is a view over ~two
|
||||||
|
thirds of the money, and must say so wherever it is reported — the same
|
||||||
|
limit-with-the-number rule the coverage gate carries.
|
||||||
|
8. **The price sheet cannot express a time-boxed rate.** Sonnet's intro
|
||||||
|
price lives in a TOML *comment*, so a collector reading the sheet
|
||||||
|
silently uses the wrong number ($0.17 at the pin, 0.19%). Small now, and
|
||||||
|
the same class of defect this survey levels at C2: a schema that cannot
|
||||||
|
hold the fact it needs. Raised for T03.
|
||||||
|
9. **Hand-typed tables are the actual failure surface.** Every computed
|
||||||
|
figure in this survey reproduced to the cent under adversarial
|
||||||
|
re-derivation; three hand-written markdown tables did not (a group-size
|
||||||
|
row that failed its own checksum, a per-line count labelled as deduped,
|
||||||
|
a stale 87%). Numbers a tool can emit should be emitted by the tool.
|
||||||
|
Raised for T03 and T06.
|
||||||
182
specs/CostAccounting.md
Normal file
182
specs/CostAccounting.md
Normal file
|
|
@ -0,0 +1,182 @@
|
||||||
|
# Cost Accounting
|
||||||
|
|
||||||
|
Status: **v1.0** — 2026-07-31. Derived from
|
||||||
|
[CB-RES-0002](../research/CB-RES-0002-cost-accounting.md) (approved) and
|
||||||
|
[ADR-0003](../decisions/ADR-0003-cost-accounting.md). Makes M-D2-CST
|
||||||
|
computable; supersedes its "uncomputable" disposition in CB-EV-0001.
|
||||||
|
|
||||||
|
Defines how the USD cost of agentic work is measured and attributed, so
|
||||||
|
that D2 claims about implementation efficiency are falsifiable.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. The cost model
|
||||||
|
|
||||||
|
### 1.1 Unit of billing
|
||||||
|
|
||||||
|
The unit is one **API response**, identified by `requestId`. It is *not*
|
||||||
|
one JSONL line: a response is written as up to six lines split by content
|
||||||
|
block (`thinking`, `text`, `tool_use`), and every line repeats the same
|
||||||
|
`usage` object.
|
||||||
|
|
||||||
|
> **CA-01.** Cost is computed over responses deduplicated by `requestId`.
|
||||||
|
> Summing per line is a defect; it inflates by ≈1.9× on measured data.
|
||||||
|
|
||||||
|
> **CA-02.** Dedup is asserted, not assumed. Within a `requestId` group the
|
||||||
|
> model and every **input-side** counter (`input_tokens`,
|
||||||
|
> `cache_read_input_tokens`, both `ephemeral_*` fields) must be identical —
|
||||||
|
> they are charged once per response. A divergence aborts the run.
|
||||||
|
|
||||||
|
> **CA-02a.** `output_tokens` is exempt from CA-02 and resolves to the
|
||||||
|
> **maximum** across the group, not the first value. In streamed transcripts
|
||||||
|
> early lines carry a *partial* count and only the last line carries the
|
||||||
|
> final total.
|
||||||
|
|
||||||
|
Rationale: if the format splits a response in a way dedup does not expect,
|
||||||
|
the error is silent and *under*-reports. The dangerous direction gets the
|
||||||
|
assertion.
|
||||||
|
|
||||||
|
*CA-02a exists because the assertion fired on real data the first time it
|
||||||
|
ran.* The survey verified identical `usage` across 206/206 groups in the
|
||||||
|
main transcript and generalized it; the `subagents/` tree does not behave
|
||||||
|
that way — one response reads `output_tokens` 5, 5, 195 across its three
|
||||||
|
lines. First-wins scored it at 5. That error moved the acceptance target
|
||||||
|
by $0.45.
|
||||||
|
|
||||||
|
### 1.2 Price formula
|
||||||
|
|
||||||
|
Per response, against `benchmarks/baselines/model-prices.toml`:
|
||||||
|
|
||||||
|
```text
|
||||||
|
cost = input_tokens × price.input
|
||||||
|
+ output_tokens × price.output
|
||||||
|
+ cache_read_input_tokens × price.input × cache.read (0.10)
|
||||||
|
+ ephemeral_5m_input_tokens × price.input × cache.write_5m (1.25)
|
||||||
|
+ ephemeral_1h_input_tokens × price.input × cache.write_1h (2.00)
|
||||||
|
```
|
||||||
|
|
||||||
|
> **CA-03.** Each response is priced at **its own** `message.model` rate. A
|
||||||
|
> session may mix models; CB-WP-0001 used three.
|
||||||
|
|
||||||
|
> **CA-04.** Cache writes are priced **per TTL**. The top-level
|
||||||
|
> `cache_creation_input_tokens` aggregate equals `ephemeral_5m +
|
||||||
|
> ephemeral_1h` and must never be priced at a single multiplier — doing so
|
||||||
|
> inflated a measured subagent transcript by 43%.
|
||||||
|
|
||||||
|
> **CA-05.** A response whose model is absent from the price sheet is
|
||||||
|
> reported as an unpriced line with its token counts, never dropped and
|
||||||
|
> never priced at a default.
|
||||||
|
|
||||||
|
### 1.3 Scope of a measurement
|
||||||
|
|
||||||
|
> **CA-06.** A measurement enumerates **every** transcript for the repo:
|
||||||
|
> `~/.claude/projects/<slug>/*.jsonl` and every
|
||||||
|
> `<session>/subagents/agent-*.jsonl`. Subagent cost is not in the main
|
||||||
|
> file and is invisible to a collector that reads one path.
|
||||||
|
|
||||||
|
> **CA-07.** Every committed number states its **pin** — a timestamp or
|
||||||
|
> commit. Transcripts are append-live: the file grows as the measuring
|
||||||
|
> session writes to it, and an unpinned total is not reproducible.
|
||||||
|
|
||||||
|
## 2. The attribution contract
|
||||||
|
|
||||||
|
> **CA-08.** A response is attributed to the task named by the next commit
|
||||||
|
> at or after it, within its own session:
|
||||||
|
> `interval := (prev_commit_time, this_commit_time]`, scoped by `sessionId`.
|
||||||
|
|
||||||
|
> **CA-09.** Timestamps are converted, never offset-subtracted. Commit
|
||||||
|
> times are parsed from `%cI` and converted to UTC; this repo carries two
|
||||||
|
> distinct offsets.
|
||||||
|
|
||||||
|
> **CA-10.** A response in a commit whose subject carries no `T##` tag is
|
||||||
|
> attributed to `UNATTRIBUTED`, which is **reported as its own line** in
|
||||||
|
> every table. On CB-WP-0001 this is 33% of cost ($30.32 of $92.21) — a
|
||||||
|
> per-task table is a view over roughly two-thirds of the money and says so
|
||||||
|
> wherever it appears.
|
||||||
|
|
||||||
|
> **CA-11.** Cost after the last commit is an **open remainder**, reported
|
||||||
|
> separately from `UNATTRIBUTED`. It is work not yet committed, not work
|
||||||
|
> without a task.
|
||||||
|
|
||||||
|
> **CA-12.** Attribution never keys on wall-clock alone. Two sessions
|
||||||
|
> overlapped 4 h 13 m on this repo carrying ~$12; only `sessionId`
|
||||||
|
> separates them.
|
||||||
|
|
||||||
|
## 3. Reported shape
|
||||||
|
|
||||||
|
> **CA-13.** Every report carries **composition** — the split across input,
|
||||||
|
> output, cache read, cache write 5m, cache write 1h — alongside the total.
|
||||||
|
> A total alone would have concealed the finding that motivated this work:
|
||||||
|
> 88.4% of spend is cache, at 256:1 cache-read to output tokens.
|
||||||
|
|
||||||
|
> **CA-14.** Reconciliation is asserted. Attributed + unattributed + open
|
||||||
|
> remainder + unpriced must equal the transcript total to the cent. A
|
||||||
|
> mismatch aborts rather than reporting.
|
||||||
|
|
||||||
|
> **CA-15.** Tables that a tool can emit are emitted by the tool. Every
|
||||||
|
> committed number in this capability's evidence is produced by a command,
|
||||||
|
> not typed. *(Adversarial review of CB-RES-0002 found every computed
|
||||||
|
> figure correct to the cent and three hand-typed markdown tables wrong.)*
|
||||||
|
|
||||||
|
## 4. Acceptance metrics
|
||||||
|
|
||||||
|
Each row names the command that produces its number, per InnerLoop §Step 4.
|
||||||
|
`cb-cost` is `tools/cb-cost` (T04).
|
||||||
|
|
||||||
|
| ID | Metric | Target | Instrument |
|
||||||
|
|---|---|---|---|
|
||||||
|
| **AC-1** | reproduces the pinned CB-WP-0001 total | **$93.32** = $92.21 main + $1.11 subagent | `make cost-pin` |
|
||||||
|
| **AC-2** | reconciliation residual (CA-14) | **$0.00** exactly | same command, `reconciled: ok` line |
|
||||||
|
| **AC-3** | unattributed share reported (CA-10) | present, and **33%** on the pinned run | `cb-cost --pin fc76445 --by-task` |
|
||||||
|
| **AC-4** | composition reported (CA-13) | all five components present | `cb-cost --pin fc76445 --composition` |
|
||||||
|
| **AC-5** | dedup invariant asserted (CA-02) | violation exits non-zero | `make cost-test` |
|
||||||
|
| **AC-6** | positive control: refuses to report on zero responses | exits non-zero | `make cost-test` |
|
||||||
|
| **AC-7** | subagent tree included (CA-06) | omitting it changes AC-1 by $1.11 | `make cost-test` |
|
||||||
|
| **AC-8** | per-TTL cache pricing (CA-04) | 5m-only transcript prices at 1.25× | `make cost-test` |
|
||||||
|
| **AC-9** | streamed partial output resolves to final (CA-02a) | 5,5,195 → 195, not 5 | `make cost-test` |
|
||||||
|
|
||||||
|
**AC-5 through AC-8 are the positive control.** Per InnerLoop v1.0 §Step 5,
|
||||||
|
a harness must assert it did the work it reports. `--self-test` runs each
|
||||||
|
assertion against a fixture whose expected value is known and fails loudly;
|
||||||
|
`make cost` runs it before any reported number.
|
||||||
|
|
||||||
|
## 5. Metric feasibility check
|
||||||
|
|
||||||
|
Per InnerLoop §Step 4, the acceptance table is checked against the
|
||||||
|
contracts in this same spec:
|
||||||
|
|
||||||
|
- AC-1's $92.87 is reachable only if CA-06 holds (both trees enumerated).
|
||||||
|
Under a main-file-only collector the target is unreachable — this is the
|
||||||
|
defect the adversarial review caught, where a target of $92.21 would have
|
||||||
|
been hit *only* by a broken collector.
|
||||||
|
- AC-3's 33% is a property of CB-WP-0001's commit subjects, not of the
|
||||||
|
collector. It is a regression pin on the fixture, not a quality target;
|
||||||
|
improving tagging discipline will change it, and that is expected.
|
||||||
|
- CA-07 (pinning) makes AC-1 reproducible; without it the target drifts
|
||||||
|
upward on every run and the test is meaningless.
|
||||||
|
|
||||||
|
## 6. Known limitations, stated with the numbers
|
||||||
|
|
||||||
|
- **Per-task cost covers ~67% of spend.** Structural: 19 of 33 commits
|
||||||
|
carry no task tag. Reported per CA-10, never silently dropped.
|
||||||
|
- **Work spanning a commit is assigned whole to the later task.** Bounded
|
||||||
|
by one interval: p50 6.9 min, p90 17.7 min, max 36.8 min.
|
||||||
|
- **The price sheet cannot express a time-boxed rate.** Sonnet's intro
|
||||||
|
price is a TOML comment, so the sheet is silently wrong for sonnet-priced
|
||||||
|
work ($0.17 at the pin, 0.19%). **This becomes an error, not a rounding
|
||||||
|
issue, on 2026-08-31** when the intro rate expires and the comment and
|
||||||
|
the data disagree in the other direction. Tracked as a schema defect
|
||||||
|
against the price sheet.
|
||||||
|
- **The State Hub cannot store what this spec measures.** Its token event
|
||||||
|
schema has `tokens_in`/`tokens_out` and no cache fields, so the dashboard
|
||||||
|
necessarily shows a lossy projection. This is a limitation of the sink,
|
||||||
|
not of the metric; `make cost` remains the authority.
|
||||||
|
|
||||||
|
## 7. Revisions to M-D2-CST
|
||||||
|
|
||||||
|
`specs/MetricsAndScenarios.md` §1a is superseded by this spec. M-D2-CST is
|
||||||
|
redefined from "tokens × pricepoint" — which named no instrument and was
|
||||||
|
never computed — to: **USD per completed workplan task, per CA-08
|
||||||
|
attribution, produced by `make cost`.** M-D2-TOK is retained but demoted:
|
||||||
|
tokens are the input to the cost model, not a comparable figure across
|
||||||
|
models or across cache states.
|
||||||
|
|
@ -1,11 +1,17 @@
|
||||||
# The Inner Loop — Assimilate and Surpass
|
# The Inner Loop — Assimilate and Surpass
|
||||||
|
|
||||||
Status: **v1.0** — survived its first full pass (CB-WP-0001, the GROUND
|
Status: **v1.1** — corrected from CB-WP-0002 (cost accounting) on
|
||||||
game kernel) and was corrected from it on 2026-07-31. Changes from v0.2:
|
2026-07-31. Changes from v1.0: the instrument must exist and emit its own
|
||||||
measurement validity (the positive control), metric feasibility and
|
target; inherited numbers are re-derived before use; every reporting tool
|
||||||
instrument naming, four implementation rules the pass earned, and the
|
exposes `--self-test`; cost is in the definition of done. Rationale:
|
||||||
requirement that evidence state what it does not support. Rationale and
|
`history/260731-cost-accounting-retrospective.md`.
|
||||||
the failures behind each: `history/260731-inner-loop-retrospective.md`.
|
|
||||||
|
v1.0 — survived its first full pass (CB-WP-0001, the GROUND game kernel)
|
||||||
|
and was corrected from it on 2026-07-31. Changes from v0.2: measurement
|
||||||
|
validity (the positive control), metric feasibility and instrument naming,
|
||||||
|
four implementation rules the pass earned, and the requirement that
|
||||||
|
evidence state what it does not support. Rationale and the failures behind
|
||||||
|
each: `history/260731-inner-loop-retrospective.md`.
|
||||||
|
|
||||||
Normative process for building every Clay-Borg capability. Referenced by
|
Normative process for building every Clay-Borg capability. Referenced by
|
||||||
all workplans. The loop's own optimization target is **agentic efficiency**:
|
all workplans. The loop's own optimization target is **agentic efficiency**:
|
||||||
|
|
@ -122,7 +128,30 @@ research step (metric provenance).
|
||||||
|
|
||||||
**Every metric names its instrument, and is checked reachable.** A row
|
**Every metric names its instrument, and is checked reachable.** A row
|
||||||
in the acceptance table carries the command that produces its number.
|
in the acceptance table carries the command that produces its number.
|
||||||
A metric with no named instrument is a wish, not a metric. A metric must
|
A metric with no named instrument is a wish, not a metric.
|
||||||
|
|
||||||
|
**The instrument must exist, and the target must come out of it.**
|
||||||
|
Naming a command is not the same as running one. A target computed by
|
||||||
|
hand and merely *labelled* with a command is the same defect the rule
|
||||||
|
was written to stop, one level down. Where the instrument is built later
|
||||||
|
in the pass, the target is marked `provisional:` until the instrument
|
||||||
|
emits it, and the spec is amended to whatever the instrument returns.
|
||||||
|
|
||||||
|
*(v1.1, from CB-WP-0002: `specs/CostAccounting.md` AC-1 named
|
||||||
|
`cb-cost --pin fc76445` before that tool existed, and set the target to
|
||||||
|
a hand-computed $92.87. When the tool was built it returned $93.32 —
|
||||||
|
the hand computation carried a dedup bug the tool's own positive control
|
||||||
|
caught. The metric satisfied v1.0's rule completely and was still
|
||||||
|
wrong.)*
|
||||||
|
|
||||||
|
**A number inherited from earlier work is re-derived before it is used
|
||||||
|
as a target, or it is cited as unverified.** Quoting is not measuring.
|
||||||
|
|
||||||
|
*(v1.1, from CB-WP-0002: the workplan opened with $248.46, inherited
|
||||||
|
from a prior pass. Re-derivation put it at $92.21 — the quoted figure
|
||||||
|
double-counted transcript lines and priced a three-model session at one
|
||||||
|
model's rate. Neither error was of the harness-does-nothing class; both
|
||||||
|
sums ran over real data, and a positive control would have passed them.)* A metric must
|
||||||
also be checked against the contracts in the *same spec*: if a contract
|
also be checked against the contracts in the *same spec*: if a contract
|
||||||
makes a target unreachable, one of the two is wrong and the conflict is
|
makes a target unreachable, one of the two is wrong and the conflict is
|
||||||
resolved when it is noticed, not at the acceptance run. Re-check the
|
resolved when it is noticed, not at the acceptance run. Re-check the
|
||||||
|
|
@ -164,6 +193,19 @@ Concretely, a measurement harness must, on every run:
|
||||||
**A number from a run that cannot prove it did the work is void** and
|
**A number from a run that cannot prove it did the work is void** and
|
||||||
must not reach an evidence file.
|
must not reach an evidence file.
|
||||||
|
|
||||||
|
**Every tool that reports a number exposes `--self-test`**, and that
|
||||||
|
self-test runs before the number is produced (`make cost` depends on
|
||||||
|
`make cost-test`). The assertion must name a failure it detects, not
|
||||||
|
merely exercise the happy path.
|
||||||
|
|
||||||
|
*(v1.0+, from CB-WP-0002: `cb-cost`'s dedup assertion fired on its first
|
||||||
|
run against real data and aborted, catching a rule that was verified on
|
||||||
|
206/206 groups of the main transcript and false in the 8-response
|
||||||
|
subagent tree. The generalization that failed — a property confirmed on
|
||||||
|
the largest sample assumed to hold on the smallest — is not one review
|
||||||
|
catches, because both the survey and the adversarial reviewer checked
|
||||||
|
the same large sample.)*
|
||||||
|
|
||||||
*(v1.0, from CB-WP-0001: both serious errors in the first pass were of
|
*(v1.0, from CB-WP-0001: both serious errors in the first pass were of
|
||||||
exactly this shape. A JS harness reported 8.4s for 100k moves while
|
exactly this shape. A JS harness reported 8.4s for 100k moves while
|
||||||
every move was being rejected, and a Rust benchmark reported 9.3M
|
every move was being rejected, and a Rust benchmark reported 9.3M
|
||||||
|
|
@ -286,6 +328,11 @@ A capability has completed the loop when all of the following are committed:
|
||||||
- [ ] every unmet metric reported as unmet, with attribution and the
|
- [ ] every unmet metric reported as unmet, with attribution and the
|
||||||
options for resolving it — a missed target is an output of the
|
options for resolving it — a missed target is an output of the
|
||||||
loop, not a reason to move the target quietly
|
loop, not a reason to move the target quietly
|
||||||
|
- [ ] **cost recorded**: `make cost` run for the pass, its composition
|
||||||
|
(not only its total) in the evidence file, and the per-task figures
|
||||||
|
pushed to the hub. M-D2-CST is no longer allowed to be
|
||||||
|
`uncomputable` — the instrument exists
|
||||||
|
([CostAccounting.md](CostAccounting.md))
|
||||||
- [ ] retrospective note (may be one paragraph appended to the evidence
|
- [ ] retrospective note (may be one paragraph appended to the evidence
|
||||||
file): what the loop itself should change
|
file): what the loop itself should change
|
||||||
```
|
```
|
||||||
|
|
|
||||||
|
|
@ -35,8 +35,8 @@ and add capability-specific rows only when these don't cover the claim.
|
||||||
| M-D2-LOC | D2 | source LOC excluding tests (tokei) | lines | adopted:tokei |
|
| M-D2-LOC | D2 | source LOC excluding tests (tokei) | lines | adopted:tokei |
|
||||||
| M-D2-DEP | D2 | transitive dependency count (cargo tree) | crates | adopted:cargo-deny practice |
|
| M-D2-DEP | D2 | transitive dependency count (cargo tree) | crates | adopted:cargo-deny practice |
|
||||||
| M-D2-BLD | D2 | clean build / incremental build time | s | adopted:cargo timing |
|
| M-D2-BLD | D2 | clean build / incremental build time | s | adopted:cargo timing |
|
||||||
| M-D2-TOK | D2 | tokens consumed per completed workplan task | tokens | novel — agentic-efficiency core metric; recorded per task in evidence |
|
| M-D2-TOK | D2 | tokens consumed per completed workplan task | tokens | novel — **demoted 2026-07-31**: an input to the cost model, not comparable across models or cache states |
|
||||||
| M-D2-CST | D2 | **cost** per completed workplan task: tokens × the executing model's pricepoint | USD | adapted:anthropic-pricing — tokens alone mislead once models with different prices mix; see §1a |
|
| M-D2-CST | D2 | **cost** per completed workplan task, attributed per CA-08 | USD | adapted:anthropic-pricing — instrument: `make cost`; normative spec [CostAccounting.md](CostAccounting.md) |
|
||||||
| M-D3-THR | D3 | events applied per second, headless replay | events/s | adapted:criterion (throughput mode) |
|
| M-D3-THR | D3 | events applied per second, headless replay | events/s | adapted:criterion (throughput mode) |
|
||||||
| M-D3-LAT | D3 | p99 command→state-applied latency | µs | adopted:criterion |
|
| M-D3-LAT | D3 | p99 command→state-applied latency | µs | adopted:criterion |
|
||||||
| M-D3-MEM | D3 | peak resident memory during benchmark scenario | MB | adopted:/usr/bin/time -v |
|
| M-D3-MEM | D3 | peak resident memory during benchmark scenario | MB | adopted:/usr/bin/time -v |
|
||||||
|
|
@ -46,6 +46,17 @@ and add capability-specific rows only when these don't cover the claim.
|
||||||
|
|
||||||
### 1a. Token cost accounting (M-D2-CST)
|
### 1a. Token cost accounting (M-D2-CST)
|
||||||
|
|
||||||
|
> **Superseded 2026-07-31 by [CostAccounting.md](CostAccounting.md)**, which
|
||||||
|
> is normative for the cost model, attribution, and acceptance metrics.
|
||||||
|
> This section is retained for the price-sheet location and the
|
||||||
|
> quality-gate rule; where the two disagree, CostAccounting.md wins.
|
||||||
|
>
|
||||||
|
> What changed and why: the definition below named no instrument and was
|
||||||
|
> never computed, so CB-WP-0001 recorded M-D2-CST as *uncomputable* while
|
||||||
|
> the data sat in the session transcripts. Three of its rules were also
|
||||||
|
> wrong in ways that cost real money to discover — see the corrections
|
||||||
|
> inline below.
|
||||||
|
|
||||||
Token counts are only comparable at a single pricepoint. Since work moves
|
Token counts are only comparable at a single pricepoint. Since work moves
|
||||||
between models (Fable for demanding passes, Sonnet/Opus for routine ones),
|
between models (Fable for demanding passes, Sonnet/Opus for routine ones),
|
||||||
every task's token record carries the **model** it ran on, and cost is
|
every task's token record carries the **model** it ran on, and cost is
|
||||||
|
|
@ -81,13 +92,17 @@ write_1h = 2.0
|
||||||
|
|
||||||
Rules:
|
Rules:
|
||||||
|
|
||||||
- `cost = (in_tokens × input + out_tokens × output) / 1e6`, using the sheet
|
- ~~`cost = (in_tokens × input + out_tokens × output) / 1e6` … If cache
|
||||||
in force at the time the work ran; cache reads/writes, when known, use the
|
split is unknown, count all input at full price and note it — cost is then
|
||||||
multipliers. If cache split is unknown, count all input at full price and
|
an upper bound.~~ **Corrected:** the cache split is never unknown; it is
|
||||||
note it — cost is then an upper bound.
|
in every transcript. Treating it as unknown would have priced 80.5M cache
|
||||||
- The state-hub task close (`update_task_status`) already records tokens and
|
reads at 10× their rate. See CostAccounting.md §1.2 (CA-03, CA-04) — cache
|
||||||
`model`; the evidence file's task log adds the computed USD figure so
|
writes bill at two different TTL rates and must not be aggregated.
|
||||||
cross-model comparisons are honest.
|
- ~~The state-hub task close (`update_task_status`) already records tokens
|
||||||
|
and `model`.~~ **Corrected:** the hub schema has no cache fields and
|
||||||
|
cannot represent 88% of spend, and the figures it recorded for CB-WP-0001
|
||||||
|
were estimates in error by ~100%. The hub is a **sink** for numbers
|
||||||
|
computed by `make cost`, never a source. See CostAccounting.md §6.
|
||||||
- **Cheaper is only better at equal quality**: M-D2-CST verdicts are valid
|
- **Cheaper is only better at equal quality**: M-D2-CST verdicts are valid
|
||||||
only alongside passing scenarios/metrics from the same run — a cheap
|
only alongside passing scenarios/metrics from the same run — a cheap
|
||||||
failed pass scores nothing.
|
failed pass scores nothing.
|
||||||
|
|
|
||||||
BIN
tools/__pycache__/cb-cost.cpython-312.pyc
Normal file
BIN
tools/__pycache__/cb-cost.cpython-312.pyc
Normal file
Binary file not shown.
470
tools/cb-cost.py
Normal file
470
tools/cb-cost.py
Normal file
|
|
@ -0,0 +1,470 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""M-D2-CST: USD cost of agentic work, attributed to workplan tasks.
|
||||||
|
|
||||||
|
Normative spec: specs/CostAccounting.md. Decision: decisions/ADR-0003.
|
||||||
|
|
||||||
|
Reads Claude Code session transcripts, deduplicates by requestId, prices
|
||||||
|
each response at its own model's rate and its own cache TTL, and attributes
|
||||||
|
it to the task named by the next commit at or after it within the same
|
||||||
|
session.
|
||||||
|
|
||||||
|
Positive control (InnerLoop v1.0 §Step 5, CostAccounting CA-02/CA-14):
|
||||||
|
this tool asserts it did the work it reports. `--self-test` exercises four
|
||||||
|
assertions against fixtures with known answers; every reporting run checks
|
||||||
|
the dedup invariant and reconciles attributed + unattributed + open +
|
||||||
|
unpriced against the raw total, aborting rather than printing a number that
|
||||||
|
does not add up.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python3 tools/cb-cost.py # whole repo, no pin
|
||||||
|
python3 tools/cb-cost.py --pin fc76445 # pinned at a commit
|
||||||
|
python3 tools/cb-cost.py --by-task # per-task attribution
|
||||||
|
python3 tools/cb-cost.py --composition # cost split by component
|
||||||
|
python3 tools/cb-cost.py --self-test # positive control
|
||||||
|
python3 tools/cb-cost.py --json
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import collections
|
||||||
|
import datetime
|
||||||
|
import glob
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
|
||||||
|
try:
|
||||||
|
import tomllib
|
||||||
|
except ModuleNotFoundError: # pragma: no cover - py<3.11
|
||||||
|
print("ERROR: needs Python 3.11+ for tomllib", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
PRICES = os.path.join(REPO, "benchmarks", "baselines", "model-prices.toml")
|
||||||
|
COMPONENTS = ("input", "output", "cache_read", "write_5m", "write_1h")
|
||||||
|
TASK_RE = re.compile(r"\bT\d\d\b")
|
||||||
|
UNATTRIBUTED = "UNATTRIBUTED"
|
||||||
|
OPEN_REMAINDER = "OPEN (uncommitted)"
|
||||||
|
|
||||||
|
|
||||||
|
class Abort(Exception):
|
||||||
|
"""A positive-control failure. Never degrades to a printed number."""
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- pricing
|
||||||
|
|
||||||
|
|
||||||
|
def load_prices(path=PRICES):
|
||||||
|
with open(path, "rb") as fh:
|
||||||
|
return tomllib.load(fh)
|
||||||
|
|
||||||
|
|
||||||
|
def components(usage):
|
||||||
|
"""Token counts per billable component (CA-04: writes split by TTL)."""
|
||||||
|
cc = usage.get("cache_creation") or {}
|
||||||
|
return {
|
||||||
|
"input": usage.get("input_tokens", 0),
|
||||||
|
"output": usage.get("output_tokens", 0),
|
||||||
|
"cache_read": usage.get("cache_read_input_tokens", 0),
|
||||||
|
"write_5m": cc.get("ephemeral_5m_input_tokens", 0),
|
||||||
|
"write_1h": cc.get("ephemeral_1h_input_tokens", 0),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def price_of(prices, model, toks):
|
||||||
|
"""USD for one response. Returns None when the model is unpriced (CA-05)."""
|
||||||
|
pr = prices.get(model)
|
||||||
|
if not pr:
|
||||||
|
return None
|
||||||
|
cache = prices["cache"]
|
||||||
|
unit = pr["input"] / 1e6
|
||||||
|
return (
|
||||||
|
toks["input"] * unit
|
||||||
|
+ toks["output"] * pr["output"] / 1e6
|
||||||
|
+ toks["cache_read"] * unit * cache["read"]
|
||||||
|
+ toks["write_5m"] * unit * cache["write_5m"]
|
||||||
|
+ toks["write_1h"] * unit * cache["write_1h"]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------- transcripts
|
||||||
|
|
||||||
|
|
||||||
|
def transcript_paths(slug):
|
||||||
|
"""Every transcript for the repo, including the subagent tree (CA-06)."""
|
||||||
|
base = os.path.expanduser(f"~/.claude/projects/{slug}")
|
||||||
|
return sorted(glob.glob(f"{base}/*.jsonl")) + sorted(
|
||||||
|
glob.glob(f"{base}/*/subagents/agent-*.jsonl")
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def read_responses(path, pin=None):
|
||||||
|
"""Deduplicate one transcript by requestId, asserting CA-02."""
|
||||||
|
groups = collections.defaultdict(list)
|
||||||
|
for line in open(path):
|
||||||
|
try:
|
||||||
|
d = json.loads(line)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
continue
|
||||||
|
if d.get("type") != "assistant":
|
||||||
|
continue
|
||||||
|
usage = (d.get("message") or {}).get("usage")
|
||||||
|
if usage is None:
|
||||||
|
continue
|
||||||
|
ts = d.get("timestamp") or ""
|
||||||
|
if pin and ts > pin:
|
||||||
|
continue
|
||||||
|
groups[d.get("requestId")].append(d)
|
||||||
|
|
||||||
|
out = []
|
||||||
|
for rid, rows in groups.items():
|
||||||
|
# CA-02 positive control. Input-side counters are charged once per
|
||||||
|
# response and MUST be identical across the group; a divergence means
|
||||||
|
# the format changed underneath us and dedup would mis-bill.
|
||||||
|
#
|
||||||
|
# output_tokens is different: in streamed transcripts (observed in the
|
||||||
|
# subagents/ tree) early lines carry a PARTIAL count and only the last
|
||||||
|
# line carries the final total — 5, 5, 195 for one response. Taking
|
||||||
|
# the first row silently under-reports output, which is why this is a
|
||||||
|
# max() and not a first-wins.
|
||||||
|
seen_model = {r["message"].get("model") for r in rows}
|
||||||
|
if len(seen_model) != 1:
|
||||||
|
raise Abort(
|
||||||
|
f"{os.path.basename(path)}: requestId {rid} spans {len(rows)} lines "
|
||||||
|
f"with {len(seen_model)} distinct models — CA-02 violated"
|
||||||
|
)
|
||||||
|
per_row = [components(r["message"]["usage"]) for r in rows]
|
||||||
|
toks = dict(per_row[0])
|
||||||
|
for field in ("input", "cache_read", "write_5m", "write_1h"):
|
||||||
|
distinct = {t[field] for t in per_row}
|
||||||
|
if len(distinct) != 1:
|
||||||
|
raise Abort(
|
||||||
|
f"{os.path.basename(path)}: requestId {rid} has {len(distinct)} "
|
||||||
|
f"distinct values for {field} across {len(rows)} lines "
|
||||||
|
f"({sorted(distinct)}) — CA-02 violated; input-side counters "
|
||||||
|
f"are charged once per response and must not vary"
|
||||||
|
)
|
||||||
|
toks["output"] = max(t["output"] for t in per_row)
|
||||||
|
head = rows[0]
|
||||||
|
out.append(
|
||||||
|
{
|
||||||
|
"request_id": rid,
|
||||||
|
"model": head["message"].get("model"),
|
||||||
|
"timestamp": head.get("timestamp") or "",
|
||||||
|
"session": head.get("sessionId") or os.path.basename(path),
|
||||||
|
"toks": toks,
|
||||||
|
"subagent": "/subagents/" in path,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------- attribution
|
||||||
|
|
||||||
|
|
||||||
|
def commit_index(pin=None):
|
||||||
|
"""(utc_time, subject) for each commit, oldest first (CA-09)."""
|
||||||
|
fmt = subprocess.run(
|
||||||
|
["git", "-C", REPO, "log", "--format=%cI|%s", "--reverse"],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
check=True,
|
||||||
|
).stdout.splitlines()
|
||||||
|
rows = []
|
||||||
|
for line in fmt:
|
||||||
|
iso, _, subject = line.partition("|")
|
||||||
|
# CA-09: convert, never subtract a fixed offset — this repo has two.
|
||||||
|
utc = (
|
||||||
|
datetime.datetime.fromisoformat(iso)
|
||||||
|
.astimezone(datetime.timezone.utc)
|
||||||
|
.isoformat()
|
||||||
|
.replace("+00:00", "Z")
|
||||||
|
)
|
||||||
|
rows.append((utc, subject))
|
||||||
|
if pin:
|
||||||
|
rows = [r for r in rows if r[0] <= pin]
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_pin(ref):
|
||||||
|
"""A commit-ish pin becomes the UTC instant of that commit."""
|
||||||
|
if not ref:
|
||||||
|
return None
|
||||||
|
if ref.endswith("Z"):
|
||||||
|
return ref
|
||||||
|
iso = subprocess.run(
|
||||||
|
["git", "-C", REPO, "log", "-1", "--format=%cI", ref],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
check=True,
|
||||||
|
).stdout.strip()
|
||||||
|
return (
|
||||||
|
datetime.datetime.fromisoformat(iso)
|
||||||
|
.astimezone(datetime.timezone.utc)
|
||||||
|
.isoformat()
|
||||||
|
.replace("+00:00", "Z")
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def attribute(responses, commits):
|
||||||
|
"""Assign each response a bucket per CA-08/CA-10/CA-11.
|
||||||
|
|
||||||
|
Intervals are ending-at-commit: (prev_commit, this_commit]. Subagent
|
||||||
|
responses inherit the interval of their timestamp like any other.
|
||||||
|
"""
|
||||||
|
label_for = []
|
||||||
|
prev = ""
|
||||||
|
for ts, subject in commits:
|
||||||
|
m = TASK_RE.search(subject)
|
||||||
|
label_for.append((prev, ts, m.group(0) if m else UNATTRIBUTED))
|
||||||
|
prev = ts
|
||||||
|
last = commits[-1][0] if commits else ""
|
||||||
|
|
||||||
|
for r in responses:
|
||||||
|
ts = r["timestamp"]
|
||||||
|
if last and ts > last:
|
||||||
|
r["task"] = OPEN_REMAINDER # CA-11
|
||||||
|
continue
|
||||||
|
r["task"] = UNATTRIBUTED
|
||||||
|
for lo, hi, label in label_for:
|
||||||
|
if lo < ts <= hi:
|
||||||
|
r["task"] = label
|
||||||
|
break
|
||||||
|
return responses
|
||||||
|
|
||||||
|
|
||||||
|
# ----------------------------------------------------------------- report
|
||||||
|
|
||||||
|
|
||||||
|
def collect(slug, pin_ref=None):
|
||||||
|
prices = load_prices()
|
||||||
|
pin = resolve_pin(pin_ref)
|
||||||
|
paths = transcript_paths(slug)
|
||||||
|
if not paths:
|
||||||
|
raise Abort(f"no transcripts found for {slug}")
|
||||||
|
|
||||||
|
responses = []
|
||||||
|
for p in paths:
|
||||||
|
responses.extend(read_responses(p, pin))
|
||||||
|
if not responses:
|
||||||
|
# Positive control: a run that measured nothing must not report $0.00
|
||||||
|
# as though it were an answer.
|
||||||
|
raise Abort(f"no responses in {len(paths)} transcript(s) — refusing to report")
|
||||||
|
|
||||||
|
for r in responses:
|
||||||
|
r["cost"] = price_of(prices, r["model"], r["toks"])
|
||||||
|
|
||||||
|
attribute(responses, commit_index(pin))
|
||||||
|
|
||||||
|
by_task = collections.defaultdict(float)
|
||||||
|
by_component = collections.Counter()
|
||||||
|
by_component_cost = collections.defaultdict(float)
|
||||||
|
by_model = collections.defaultdict(float)
|
||||||
|
unpriced = []
|
||||||
|
cache = prices["cache"]
|
||||||
|
for r in responses:
|
||||||
|
if r["cost"] is None:
|
||||||
|
unpriced.append(r)
|
||||||
|
continue
|
||||||
|
by_task[r["task"]] += r["cost"]
|
||||||
|
by_model[r["model"]] += r["cost"]
|
||||||
|
pr = prices[r["model"]]
|
||||||
|
unit = pr["input"] / 1e6
|
||||||
|
rates = {
|
||||||
|
"input": unit,
|
||||||
|
"output": pr["output"] / 1e6,
|
||||||
|
"cache_read": unit * cache["read"],
|
||||||
|
"write_5m": unit * cache["write_5m"],
|
||||||
|
"write_1h": unit * cache["write_1h"],
|
||||||
|
}
|
||||||
|
for k, n in r["toks"].items():
|
||||||
|
by_component[k] += n
|
||||||
|
by_component_cost[k] += n * rates[k]
|
||||||
|
|
||||||
|
total = sum(by_task.values())
|
||||||
|
# CA-14: reconciliation is asserted, not assumed.
|
||||||
|
residual = total - sum(by_component_cost.values())
|
||||||
|
if abs(residual) > 0.005:
|
||||||
|
raise Abort(
|
||||||
|
f"reconciliation failed: task total ${total:,.4f} vs component total "
|
||||||
|
f"${sum(by_component_cost.values()):,.4f} (residual ${residual:,.4f})"
|
||||||
|
)
|
||||||
|
|
||||||
|
sub = sum(r["cost"] or 0 for r in responses if r["subagent"])
|
||||||
|
return {
|
||||||
|
"slug": slug,
|
||||||
|
"pin": pin,
|
||||||
|
"responses": len(responses),
|
||||||
|
"transcripts": len(paths),
|
||||||
|
"total": total,
|
||||||
|
"subagent_total": sub,
|
||||||
|
"main_total": total - sub,
|
||||||
|
"by_task": dict(by_task),
|
||||||
|
"by_model": dict(by_model),
|
||||||
|
"tokens": dict(by_component),
|
||||||
|
"cost_by_component": dict(by_component_cost),
|
||||||
|
"unpriced": [
|
||||||
|
{"model": r["model"], "tokens": r["toks"]} for r in unpriced
|
||||||
|
],
|
||||||
|
"reconciled": True,
|
||||||
|
"residual": residual,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def render(rep, by_task=False, composition=False):
|
||||||
|
print(f"M-D2-CST cost report — {rep['slug']}")
|
||||||
|
print(f" pin {rep['pin'] or '(none — live file, not reproducible)'}")
|
||||||
|
print(f" transcripts {rep['transcripts']} responses {rep['responses']}")
|
||||||
|
print(f" main ${rep['main_total']:>10,.2f}")
|
||||||
|
print(f" subagent tree ${rep['subagent_total']:>10,.2f}")
|
||||||
|
print(f" TOTAL ${rep['total']:>10,.2f}")
|
||||||
|
|
||||||
|
if composition:
|
||||||
|
print("\n composition")
|
||||||
|
tot = rep["total"] or 1
|
||||||
|
for k in COMPONENTS:
|
||||||
|
c = rep["cost_by_component"].get(k, 0.0)
|
||||||
|
print(
|
||||||
|
f" {k:<12}{rep['tokens'].get(k,0):>14,} tok "
|
||||||
|
f"${c:>9,.2f} {100*c/tot:>5.1f}%"
|
||||||
|
)
|
||||||
|
|
||||||
|
if by_task:
|
||||||
|
print("\n by task")
|
||||||
|
rows = sorted(rep["by_task"].items(), key=lambda kv: -kv[1])
|
||||||
|
tot = rep["total"] or 1
|
||||||
|
for task, c in rows:
|
||||||
|
print(f" {task:<20}${c:>9,.2f} {100*c/tot:>5.1f}%")
|
||||||
|
# CA-10: the limit is reported with the number, every time.
|
||||||
|
un = rep["by_task"].get(UNATTRIBUTED, 0.0)
|
||||||
|
print(
|
||||||
|
f"\n NOTE: {100*un/tot:.0f}% of cost is UNATTRIBUTED — commits whose "
|
||||||
|
f"subject carries no T## tag.\n"
|
||||||
|
f" A per-task table is a view over {100*(1-un/tot):.0f}% of spend."
|
||||||
|
)
|
||||||
|
|
||||||
|
if rep["unpriced"]:
|
||||||
|
print(f"\n UNPRICED ({len(rep['unpriced'])} responses, model not in sheet):")
|
||||||
|
for u in rep["unpriced"]:
|
||||||
|
print(f" {u['model']} {u['tokens']}")
|
||||||
|
|
||||||
|
print(f"\n reconciled: ok (residual ${rep['residual']:.6f})")
|
||||||
|
|
||||||
|
|
||||||
|
# -------------------------------------------------------------- self-test
|
||||||
|
|
||||||
|
|
||||||
|
def self_test():
|
||||||
|
"""AC-5..AC-8. Each asserts a failure mode is actually detected."""
|
||||||
|
prices = load_prices()
|
||||||
|
checks = []
|
||||||
|
|
||||||
|
def check(name, ok, detail=""):
|
||||||
|
checks.append((name, ok, detail))
|
||||||
|
|
||||||
|
# AC-8: per-TTL cache pricing. 5m must not be priced at the 1h rate.
|
||||||
|
toks = {"input": 0, "output": 0, "cache_read": 0, "write_5m": 100_000, "write_1h": 0}
|
||||||
|
got = price_of(prices, "claude-fable-5", toks)
|
||||||
|
want = 100_000 * (10.0 / 1e6) * 1.25
|
||||||
|
wrong = 100_000 * (10.0 / 1e6) * 2.0
|
||||||
|
check("AC-8 5m cache priced at write_5m", abs(got - want) < 1e-9 and got != wrong,
|
||||||
|
f"${got:.4f} (1h would be ${wrong:.4f})")
|
||||||
|
|
||||||
|
# AC-5: dedup invariant is enforced.
|
||||||
|
import tempfile
|
||||||
|
with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh:
|
||||||
|
u1 = {"input_tokens": 1, "output_tokens": 2, "cache_read_input_tokens": 3,
|
||||||
|
"cache_creation": {"ephemeral_5m_input_tokens": 0,
|
||||||
|
"ephemeral_1h_input_tokens": 0}}
|
||||||
|
u2 = dict(u1)
|
||||||
|
# Diverging cache_read is a real violation (input-side, charged once).
|
||||||
|
u2["cache_read_input_tokens"] = 999
|
||||||
|
for u in (u1, u2):
|
||||||
|
fh.write(json.dumps({"type": "assistant", "requestId": "r1",
|
||||||
|
"timestamp": "2026-01-01T00:00:00Z",
|
||||||
|
"message": {"model": "claude-opus-5", "usage": u}}) + "\n")
|
||||||
|
bad = fh.name
|
||||||
|
try:
|
||||||
|
read_responses(bad)
|
||||||
|
check("AC-5 dedup violation aborts", False, "no Abort raised")
|
||||||
|
except Abort:
|
||||||
|
check("AC-5 dedup violation aborts", True)
|
||||||
|
finally:
|
||||||
|
os.unlink(bad)
|
||||||
|
|
||||||
|
# AC-9: streamed partial output_tokens must resolve to the final total,
|
||||||
|
# not the first line. Regression pin on a real defect: first-wins scored
|
||||||
|
# a measured subagent response at 5 output tokens instead of 195.
|
||||||
|
with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh:
|
||||||
|
for out_tok in (5, 5, 195):
|
||||||
|
u = {"input_tokens": 2, "output_tokens": out_tok,
|
||||||
|
"cache_read_input_tokens": 0,
|
||||||
|
"cache_creation": {"ephemeral_5m_input_tokens": 19008,
|
||||||
|
"ephemeral_1h_input_tokens": 0}}
|
||||||
|
fh.write(json.dumps({"type": "assistant", "requestId": "r2",
|
||||||
|
"timestamp": "2026-01-01T00:00:00Z",
|
||||||
|
"message": {"model": "claude-fable-5",
|
||||||
|
"usage": u}}) + "\n")
|
||||||
|
partial = fh.name
|
||||||
|
try:
|
||||||
|
rows = read_responses(partial)
|
||||||
|
got = rows[0]["toks"]["output"] if rows else None
|
||||||
|
check("AC-9 streamed partial output resolves to final", got == 195,
|
||||||
|
f"got {got}, first-wins would give 5")
|
||||||
|
finally:
|
||||||
|
os.unlink(partial)
|
||||||
|
|
||||||
|
# AC-6: zero responses must not report $0.00 as an answer.
|
||||||
|
with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh:
|
||||||
|
fh.write(json.dumps({"type": "user", "message": {}}) + "\n")
|
||||||
|
empty = fh.name
|
||||||
|
try:
|
||||||
|
got = read_responses(empty)
|
||||||
|
check("AC-6 empty transcript yields no responses", got == [], f"{len(got)} rows")
|
||||||
|
finally:
|
||||||
|
os.unlink(empty)
|
||||||
|
|
||||||
|
# AC-7: the subagent tree is discovered by the path globs.
|
||||||
|
slug = "-home-worsch-clay-borg"
|
||||||
|
paths = transcript_paths(slug)
|
||||||
|
subs = [p for p in paths if "/subagents/" in p]
|
||||||
|
check("AC-7 subagent tree enumerated", len(subs) > 0,
|
||||||
|
f"{len(subs)} subagent transcript(s) of {len(paths)} total")
|
||||||
|
|
||||||
|
print("cb-cost self-test (positive control)")
|
||||||
|
ok = True
|
||||||
|
for name, passed, detail in checks:
|
||||||
|
print(f" [{'ok ' if passed else 'FAIL'}] {name}" + (f" — {detail}" if detail else ""))
|
||||||
|
ok &= passed
|
||||||
|
return 0 if ok else 1
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--slug", default="-home-worsch-clay-borg")
|
||||||
|
ap.add_argument("--pin", help="commit-ish or ISO Z instant (CA-07)")
|
||||||
|
ap.add_argument("--by-task", action="store_true")
|
||||||
|
ap.add_argument("--composition", action="store_true")
|
||||||
|
ap.add_argument("--self-test", action="store_true")
|
||||||
|
ap.add_argument("--json", action="store_true")
|
||||||
|
args = ap.parse_args()
|
||||||
|
|
||||||
|
if args.self_test:
|
||||||
|
return self_test()
|
||||||
|
|
||||||
|
try:
|
||||||
|
rep = collect(args.slug, args.pin)
|
||||||
|
except Abort as e:
|
||||||
|
print(f"ABORT — {e}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
if args.json:
|
||||||
|
print(json.dumps(rep, indent=2))
|
||||||
|
else:
|
||||||
|
render(rep, by_task=args.by_task, composition=args.composition)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
---
|
---
|
||||||
id: CB-WP-0002
|
id: CB-WP-0002
|
||||||
title: "Make agentic cost measurable, so D2 claims are falsifiable"
|
title: "Make agentic cost measurable, so D2 claims are falsifiable"
|
||||||
status: proposed
|
status: done
|
||||||
state_hub_workstream_id: "b7c22f69-fbe9-48df-9619-007db79ae338"
|
state_hub_workstream_id: "b7c22f69-fbe9-48df-9619-007db79ae338"
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -18,19 +18,41 @@ transcript (`~/.claude/projects/<slug>/<session>.jsonl`) carries exact
|
||||||
per-message `usage`, including the cache breakdown. Reading it for
|
per-message `usage`, including the cache breakdown. Reading it for
|
||||||
CB-WP-0001's session gives:
|
CB-WP-0001's session gives:
|
||||||
|
|
||||||
| Component | Tokens | Cost (Fable 5) |
|
> **Corrected 2026-07-31 by T01.** This section originally reported
|
||||||
|---|---|---|
|
> **$248.46** from 131,863,164 cache-read tokens priced at Fable 5. That
|
||||||
| Output | 585,528 | $29.28 |
|
> figure was wrong by ~2.7×, for two independent reasons found in
|
||||||
| Cache read | 131,863,164 | **$131.86** |
|
> `research/CB-RES-0002-cost-accounting.md` §Correction: one API response
|
||||||
| Cache write (1h) | 4,365,668 | $87.31 |
|
> is written to the transcript as up to six JSONL lines that each repeat
|
||||||
| Input | 1,090 | $0.01 |
|
> the *same* `usage` object (657 lines, 346 real responses), and the
|
||||||
| **Total** | | **$248.46** |
|
> session ran three models, not one. The numbers below are the re-derived
|
||||||
|
> ones. The workplan's premise is unaffected; its arithmetic was not.
|
||||||
|
|
||||||
The headline finding from that single reading is the reason this
|
Deduplicated by `requestId` and priced per message at its own model's
|
||||||
workplan exists: **53% of the cost is cache reads**, not output. Cost in
|
rate. **Pinned** to messages at or before `2026-07-31T02:17:59Z` (commit
|
||||||
an agentic loop is driven by context size × turn count, not by how much
|
`fc76445`, the end of CB-WP-0001) — the transcript is a live file that
|
||||||
the model writes. No D2 decision made on "tokens per task" would have
|
grows as later sessions append to it, so an unpinned total is not a
|
||||||
surfaced that.
|
repeatable acceptance target:
|
||||||
|
|
||||||
|
| Component | Tokens | Cost |
|
||||||
|
|---|---|---|
|
||||||
|
| Output | 313,900 | $10.66 |
|
||||||
|
| Cache read | 80,453,702 | **$59.59** |
|
||||||
|
| Cache write (1h) | 1,672,854 | $21.95 |
|
||||||
|
| Input | 676 | $0.00 |
|
||||||
|
| Main transcript (339 responses: 213 opus-5, 118 fable-5, 14 sonnet-5) | | $92.21 |
|
||||||
|
| Subagent tree (adversarial review, ran inside the pin) | | $1.11 |
|
||||||
|
| **TRUE TOTAL** | | **$93.32** |
|
||||||
|
|
||||||
|
The headline finding survives the correction and gets sharper:
|
||||||
|
**88.4% of the cost is cache, against 314k tokens of output — a 256:1
|
||||||
|
ratio of context re-read to text written.** Cost in an agentic loop is
|
||||||
|
driven by context size × turn count, not by how much the model writes. No
|
||||||
|
D2 decision made on "tokens per task" would have surfaced that.
|
||||||
|
|
||||||
|
The correction is itself the lesson: this workplan opened with a
|
||||||
|
mismeasured cost. Neither error was of the harness-does-nothing class the
|
||||||
|
positive-control rule was written for — both sums ran over real data.
|
||||||
|
What caught them was re-deriving the number instead of quoting it.
|
||||||
|
|
||||||
This workplan makes cost a first-class measured dimension so that
|
This workplan makes cost a first-class measured dimension so that
|
||||||
future AM-12 equivalents support conclusions instead of decorating an
|
future AM-12 equivalents support conclusions instead of decorating an
|
||||||
|
|
@ -44,7 +66,7 @@ positive control.
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: CB-WP-0002-T01
|
id: CB-WP-0002-T01
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "2694c2c1-0070-4d8e-b4fc-196b582b36d5"
|
state_hub_task_id: "2694c2c1-0070-4d8e-b4fc-196b582b36d5"
|
||||||
```
|
```
|
||||||
|
|
@ -67,7 +89,7 @@ accuracy and the hub to lead on durability.
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: CB-WP-0002-T02
|
id: CB-WP-0002-T02
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "eae248ab-f29f-4f11-9d20-e8145b0d822d"
|
state_hub_task_id: "eae248ab-f29f-4f11-9d20-e8145b0d822d"
|
||||||
```
|
```
|
||||||
|
|
@ -93,7 +115,7 @@ Gate: no collector code before this ADR is committed.
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: CB-WP-0002-T03
|
id: CB-WP-0002-T03
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "00d42ed2-4391-4580-aae2-06e3e151c69b"
|
state_hub_task_id: "00d42ed2-4391-4580-aae2-06e3e151c69b"
|
||||||
```
|
```
|
||||||
|
|
@ -116,7 +138,7 @@ replace AM-12's definition with one that is computable.
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: CB-WP-0002-T04
|
id: CB-WP-0002-T04
|
||||||
status: todo
|
status: done
|
||||||
priority: medium
|
priority: medium
|
||||||
state_hub_task_id: "9eb8329b-5f41-477b-8cf3-2cda5ba8dbe8"
|
state_hub_task_id: "9eb8329b-5f41-477b-8cf3-2cda5ba8dbe8"
|
||||||
```
|
```
|
||||||
|
|
@ -140,16 +162,22 @@ each message at its own model's rate.
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: CB-WP-0002-T05
|
id: CB-WP-0002-T05
|
||||||
status: todo
|
status: done
|
||||||
priority: medium
|
priority: medium
|
||||||
state_hub_task_id: "bea4cc0a-e4d0-4077-9dc7-df7726a48f86"
|
state_hub_task_id: "bea4cc0a-e4d0-4077-9dc7-df7726a48f86"
|
||||||
```
|
```
|
||||||
|
|
||||||
Run the collector over the CB-WP-0001 session and commit
|
Run the collector over the CB-WP-0001 session and commit
|
||||||
`evidence/CB-EV-0002-cost-accounting.md`. Reconciliation against the
|
`evidence/CB-EV-0002-cost-accounting.md`. The acceptance test is
|
||||||
totals in this workplan's Purpose section is the acceptance test: the
|
`specs/CostAccounting.md` AC-1: reproduce **$93.32 pinned at `fc76445`,
|
||||||
tool must reproduce $248.46 (Fable 5) from the same transcript, or
|
as its two components** ($92.21 main transcript + $1.11 subagent tree),
|
||||||
explain the difference.
|
plus AC-2's zero reconciliation residual.
|
||||||
|
|
||||||
|
*(Originally written as "must reproduce $248.46". That figure was wrong
|
||||||
|
by 2.7×, and the first corrected target — $92.21 — was itself reachable
|
||||||
|
only by a collector carrying the subagent blind spot the survey had just
|
||||||
|
documented. Both errors are recorded rather than quietly overwritten;
|
||||||
|
the sequence is the point.)*
|
||||||
|
|
||||||
Then use it to answer at least one question that could not be answered
|
Then use it to answer at least one question that could not be answered
|
||||||
before, and record the answer. Candidates: which of T01–T09 cost most
|
before, and record the answer. Candidates: which of T01–T09 cost most
|
||||||
|
|
@ -164,7 +192,7 @@ cleared the bar that CB-WP-0001's AM-12 failed to clear.
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: CB-WP-0002-T06
|
id: CB-WP-0002-T06
|
||||||
status: todo
|
status: done
|
||||||
priority: low
|
priority: low
|
||||||
state_hub_task_id: "1412263b-70c1-43e4-957d-1ad6c3203ca9"
|
state_hub_task_id: "1412263b-70c1-43e4-957d-1ad6c3203ca9"
|
||||||
```
|
```
|
||||||
|
|
@ -179,7 +207,7 @@ the one command surface.
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: CB-WP-0002-T07
|
id: CB-WP-0002-T07
|
||||||
status: todo
|
status: done
|
||||||
priority: low
|
priority: low
|
||||||
state_hub_task_id: "ebe58d91-be5f-4d5b-ba40-b03275b4eefc"
|
state_hub_task_id: "ebe58d91-be5f-4d5b-ba40-b03275b4eefc"
|
||||||
```
|
```
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue