Compare commits
10 commits
fc76445aec
...
42180fbc57
| Author | SHA1 | Date | |
|---|---|---|---|
| 42180fbc57 | |||
| 2f086d26b6 | |||
| c0eef604e8 | |||
| e008b1e787 | |||
| 900467017f | |||
| b96cd94a64 | |||
| c3e6e72ab2 | |||
| ac3ac2a8ce | |||
| bf57b6f3a4 | |||
| 060eb8cf6a |
15 changed files with 1886 additions and 42 deletions
|
|
@ -33,6 +33,11 @@ jobs:
|
|||
# AM-4a/AM-4b: third-party source under audit, per configuration.
|
||||
- run: make dep-weight
|
||||
|
||||
# Positive control for the cost collector (AC-5..AC-9). Does not
|
||||
# gate on a dollar figure — transcripts are not present in CI — but
|
||||
# proves the collector still detects the failures it claims to.
|
||||
- run: make cost-test
|
||||
|
||||
# InnerLoop v1.0 positive control, enforced rather than asserted in
|
||||
# prose: --test runs every benchmark once, so a workload that
|
||||
# stalls or produces the wrong event count fails the build instead
|
||||
|
|
|
|||
15
Makefile
15
Makefile
|
|
@ -3,7 +3,7 @@
|
|||
|
||||
CARGO := cargo
|
||||
|
||||
.PHONY: check test sim bench bench-test coverage dep-weight loc all
|
||||
.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin loc all
|
||||
|
||||
## fmt + clippy (deny warnings) + HashMap deny-lint
|
||||
check:
|
||||
|
|
@ -21,6 +21,17 @@ dep-weight:
|
|||
coverage:
|
||||
python3 tools/rule-coverage.py
|
||||
|
||||
# M-D2-CST (specs/CostAccounting.md). cost-test is the positive control and
|
||||
# runs first: a cost number from an unverified collector is void.
|
||||
cost: cost-test
|
||||
python3 tools/cb-cost.py --composition --by-task
|
||||
|
||||
cost-test:
|
||||
python3 tools/cb-cost.py --self-test
|
||||
|
||||
cost-pin: cost-test
|
||||
python3 tools/cb-cost.py --pin fc76445 --composition --by-task
|
||||
|
||||
sim:
|
||||
$(CARGO) run -q -p cb-sim -- scenarios/ground/*.yaml
|
||||
|
||||
|
|
@ -40,4 +51,4 @@ loc:
|
|||
printf '%-28s %s\n' $$d "$$(find $$d/src -name '*.rs' | xargs cat | grep -vcE '^\s*(//|$$)')"; \
|
||||
done
|
||||
|
||||
all: check test sim coverage dep-weight bench-test
|
||||
all: check test sim coverage dep-weight cost-test bench-test
|
||||
|
|
|
|||
118
decisions/ADR-0003-cost-accounting.md
Normal file
118
decisions/ADR-0003-cost-accounting.md
Normal file
|
|
@ -0,0 +1,118 @@
|
|||
# ADR-0003: cost accounting — instrument and attribution model
|
||||
|
||||
status: accepted
|
||||
date: 2026-07-31
|
||||
tier: L (structural L, chaos d10=2 → no override)
|
||||
research: [CB-RES-0002](../research/CB-RES-0002-cost-accounting.md) (approved)
|
||||
review: history/260731-cost-accounting-{challenge,response}.md
|
||||
supersedes: the "uncomputable" disposition of AM-12 / M-D2-CST in
|
||||
[CB-EV-0001](../evidence/CB-EV-0001-game-kernel.md)
|
||||
|
||||
## Context
|
||||
|
||||
M-D2-CST was fully specified in CB-WP-0001 and never instrumented, so D2
|
||||
claims about implementation efficiency rest on nothing measured. The survey
|
||||
established that the data existed the whole time, and that reading it
|
||||
correctly is harder than it looks: this workplan's own opening figure was
|
||||
wrong by 2.7×, and the survey's first draft set an acceptance target only a
|
||||
*broken* collector could hit.
|
||||
|
||||
## Decision
|
||||
|
||||
### D1 — Instrument: session transcript JSONL (C1)
|
||||
|
||||
Cost is computed from `~/.claude/projects/<slug>/**/*.jsonl`, deduplicated
|
||||
by `requestId`, priced per message against
|
||||
`benchmarks/baselines/model-prices.toml`.
|
||||
|
||||
Rejected: the State Hub as a *source* (its schema has no cache fields, so it
|
||||
cannot represent 88% of spend, and its recorded numbers are ~100% in error
|
||||
against the same work); the status bar (not machine-readable from a tool
|
||||
call); the billing API (no session or task attribution, no admin key). The
|
||||
hub remains the durable **sink**; the billing API remains an optional
|
||||
external reconciliation check if an admin key ever exists.
|
||||
|
||||
### D2 — Attribution: git commit intervals, scoped by session
|
||||
|
||||
A message is attributed to the task named by the **next commit at or after
|
||||
it**, within its own session:
|
||||
|
||||
```text
|
||||
interval := (prev_commit_time, this_commit_time] # ending-at-commit
|
||||
scope := sessionId # never wall-clock alone
|
||||
task := the T## tag in the commit subject, else UNATTRIBUTED
|
||||
```
|
||||
|
||||
Ending-at-commit is the only convention consistent with the loop's
|
||||
commit-at-end-of-task pattern; the alternative shifts every task's cost one
|
||||
interval. Session scoping is not optional: two sessions overlap 4 h 13 m on
|
||||
this repo carrying ~$12 that no wall-clock join can separate.
|
||||
|
||||
Rejected: **explicit session markers** at task start/end — they require the
|
||||
agent to remember, and the failure this workplan exists to fix was caused by
|
||||
exactly that kind of remembered step; nothing that depends on discipline
|
||||
gets to be the primary index. Rejected: **hub status transitions** as the
|
||||
time index — they are written after the fact, sometimes in a batch, and one
|
||||
was written for CB-WP-0001 nine minutes after the work it bounds.
|
||||
|
||||
### D3 — Reported shape: composition, not a total
|
||||
|
||||
Every report carries the cost split by component (input / output / cache
|
||||
read / cache write per TTL) alongside the total. A single total would have
|
||||
concealed the finding that motivated the workplan.
|
||||
|
||||
## Expected advantage per dimension
|
||||
|
||||
| Dim | Expectation vs the C2 baseline | Basis |
|
||||
|---|---|---|
|
||||
| **D1 ease of specification** | **worse.** The hub is one API call; this is a parser with a dedup rule, a TTL-aware price model, a session scope, and a commit join. Four moving parts against one. | measured: the survey needed three corrections to get the parse right |
|
||||
| **D2 efficiency** | **better, decisively.** C2's recorded numbers are ~100% in error on the same work ($0.03-equivalent recorded against $92.21 actual). C1 is exact by construction. Cost to produce: one file read, ~1 s. | measured |
|
||||
| **D3 speed** | **parity.** 2,040 lines / 5.1 MB parsed in <1 s; a hub call is a network round trip. Neither is a bottleneck. | measured |
|
||||
| **D4 optionality** | **better.** The transcript is a file on disk in a documented shape; the hub is a service that must be running. The collector degrades to "no data" rather than "wrong data" when a transcript is absent. | reasoned |
|
||||
|
||||
An honest summary: **we are buying accuracy with specification complexity.**
|
||||
That trade is right here only because the alternative is not "a simpler
|
||||
correct number" but "a number that is wrong by two orders of magnitude",
|
||||
which is what CB-WP-0001 actually recorded.
|
||||
|
||||
## Known failure modes of the chosen model
|
||||
|
||||
Stated rather than discovered later. Each becomes a test in T04.
|
||||
|
||||
1. **33% of spend has no task.** Only 14 of 33 commits name a task; the rest
|
||||
hold $30.32 of $92.21. Per-task tables are a view over two-thirds of the
|
||||
money and must say so wherever reported.
|
||||
2. **Work spanning a boundary is assigned whole to the later task.** A
|
||||
message before a commit belongs to that commit's task even if the thinking
|
||||
began earlier. Accepted: the loop commits per task, so the error is
|
||||
bounded by one interval (p90 17.7 min, max 36.8 min).
|
||||
3. **Uncommitted work is invisible.** Cost incurred after the last commit
|
||||
has no enclosing interval. Reported as an open remainder, never dropped.
|
||||
4. **`/compact` is safe; resumed and concurrent sessions are the risk.**
|
||||
Compaction stays within one file and one session. Two agents on one repo
|
||||
are separable only because `sessionId` exists — this is why D2 scopes by
|
||||
it.
|
||||
5. **The price sheet cannot express a time-boxed rate.** Sonnet's intro
|
||||
price is a TOML comment. $0.17 at the pin; the schema defect is the real
|
||||
issue and is deferred to T03 with a stated deadline of 2026-08-31, when
|
||||
the intro rate expires and the sheet becomes silently wrong.
|
||||
6. **Dedup is load-bearing in the dangerous direction.** If the format ever
|
||||
splits one response across two `requestId`s, the collector *under*-reports
|
||||
and nothing looks wrong. T04 asserts the dedup invariant at runtime
|
||||
(identical `usage` within a group) rather than trusting the survey's
|
||||
one-time check.
|
||||
7. **The subagent tree is a separate enumeration.** Missing it under-reports
|
||||
silently; it was $0.66 here and will not stay small on a fan-out pass.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Gate satisfied:** T04 may now write collector code. Nothing before this.
|
||||
- T03 specifies the metrics with this ADR's contracts; T05's acceptance test
|
||||
is $92.87 as two components, with a 33% unattributed line.
|
||||
- The hub's token API keeps receiving events, now as a lossy projection of a
|
||||
computed number rather than as an estimate. Its schema gap (no cache
|
||||
fields) is recorded as a limitation of the dashboard, not of the metric.
|
||||
- **Raised out of scope, for the ralph-workplan skill:** its instruction to
|
||||
"read tokens from the Claude Code status bar" asks an agent for a figure
|
||||
it cannot read, and is the proximate cause of the hub's bad numbers. The
|
||||
skill lives outside this repo; flagged for the maintainer.
|
||||
152
evidence/CB-EV-0002-cost-accounting.md
Normal file
152
evidence/CB-EV-0002-cost-accounting.md
Normal file
|
|
@ -0,0 +1,152 @@
|
|||
# CB-EV-0002: cost accounting
|
||||
|
||||
capability: meta.loop.cost-accounting
|
||||
spec: [CostAccounting.md](../specs/CostAccounting.md) v1.0
|
||||
decision: [ADR-0003](../decisions/ADR-0003-cost-accounting.md)
|
||||
research: [CB-RES-0002](../research/CB-RES-0002-cost-accounting.md)
|
||||
instrument: `make cost-pin` (`tools/cb-cost.py --pin fc76445`)
|
||||
measured: 2026-07-31
|
||||
|
||||
Subject of measurement: the CB-WP-0001 session, pinned at commit `fc76445`
|
||||
(`2026-07-31T02:17:59Z`). All numbers below are emitted by the tool, not
|
||||
transcribed by hand (CA-15).
|
||||
|
||||
---
|
||||
|
||||
## 1. Acceptance table
|
||||
|
||||
| ID | Metric | Target | Measured | Verdict |
|
||||
|---|---|---|---|---|
|
||||
| AC-1 | pinned total, as two components | $93.32 = $92.21 + $1.11 | **$92.21 main + $1.11 subagent = $93.32** | **met** |
|
||||
| AC-2 | reconciliation residual | $0.00 | **$0.000000** | **met** |
|
||||
| AC-3 | unattributed share reported | present, 33% | **32.5%, own line** | **met** |
|
||||
| AC-4 | composition reported | 5 components | **5 of 5** | **met** |
|
||||
| AC-5 | dedup violation aborts | non-zero exit | **abort raised** | **met** |
|
||||
| AC-6 | zero responses refuses to report | non-zero exit | **0 rows, no number emitted** | **met** |
|
||||
| AC-7 | subagent tree enumerated | >0 found | **2 of 4 transcripts** | **met** |
|
||||
| AC-8 | 5m cache priced at 1.25× | $1.25/100k @ fable | **$1.2500 (1h would be $2.0000)** | **met** |
|
||||
| AC-9 | streamed partial output → final | 5,5,195 → 195 | **195** | **met** |
|
||||
|
||||
No `unmeasured` rows. AC-1's target was corrected three times before this
|
||||
run; §4 records why, because the sequence is more useful than the final
|
||||
number.
|
||||
|
||||
## 2. Composition — the finding the metric exists for
|
||||
|
||||
```text
|
||||
input 690 tok $ 0.00 0.0%
|
||||
output 323,643 tok $ 11.15 11.9%
|
||||
cache_read 80,611,798 tok $ 59.75 64.0%
|
||||
write_5m 37,467 tok $ 0.47 0.5%
|
||||
write_1h 1,672,854 tok $ 21.95 23.5%
|
||||
TOTAL $ 93.32
|
||||
```
|
||||
|
||||
**88.0% of spend is cache; 11.9% is output.** The ratio of context re-read
|
||||
to text written is 249:1. A single total would have shown $93.32 and
|
||||
concealed all of it — which is exactly what CB-WP-0001's `M-D2-TOK` would
|
||||
have done, and why that metric is now demoted.
|
||||
|
||||
## 3. Per-task attribution
|
||||
|
||||
```text
|
||||
UNATTRIBUTED $ 30.32 32.5%
|
||||
T08 $ 21.02 22.5%
|
||||
T07 $ 12.01 12.9%
|
||||
T03 $ 9.91 10.6%
|
||||
T04 $ 8.00 8.6%
|
||||
T05 $ 7.19 7.7%
|
||||
T06 $ 3.58 3.8%
|
||||
T09 $ 1.29 1.4%
|
||||
```
|
||||
|
||||
**Stated limit (CA-10):** 32.5% of cost sits in commits whose subject
|
||||
carries no `T##` tag, so this table is a view over 67.5% of spend. That is
|
||||
a property of commit hygiene, not of the collector.
|
||||
|
||||
T08 (the GROUND aggregate, six code iterations) at $21.02 is the most
|
||||
expensive task and was also the one that produced the most rework — the
|
||||
tuple-map hash panic, the discarded `setup.patch`, and the 5.6×-wrong
|
||||
benchmark all originated there. Expensive *and* error-dense: the correlation
|
||||
is worth watching, not yet a conclusion from n=1.
|
||||
|
||||
## 4. What it does not support
|
||||
|
||||
- **The per-task figures are not comparable across passes.** They mix
|
||||
models (opus/fable/sonnet) at different price points and different cache
|
||||
states. The dollar figure is comparable; a token count is not.
|
||||
- **AC-3's 32.5% is a fixture pin, not a quality target.** Improving commit
|
||||
tagging will move it, and that is the desired direction.
|
||||
- **This is one session.** Every ratio here (cache share, $/turn, the
|
||||
compaction effect in §5) is n=1 and should be treated as a hypothesis
|
||||
until a second pass reproduces it.
|
||||
- **The price sheet cannot express a time-boxed rate.** Sonnet's intro price
|
||||
lives in a TOML comment, so sonnet-priced work is off by $0.17 here
|
||||
(0.19%). This becomes a real error on 2026-08-31.
|
||||
- **AC-1 is not invoice-verified.** No admin key exists, so the Anthropic
|
||||
billing API could not independently confirm the total. The transcript
|
||||
counters are the same ones billing uses, but that is an argument, not a
|
||||
reconciliation.
|
||||
|
||||
## 5. The question that could not be answered before
|
||||
|
||||
**"What does `/compact` actually cost, and is a long session quadratic?"**
|
||||
|
||||
CB-WP-0003 T04 asserts that cost ≈ turns × mean_context and that mean_context
|
||||
grows with turns, making long sessions quadratic. The data **qualifies that
|
||||
claim**: it is true only between compactions.
|
||||
|
||||
The session compacted twice, and the transcript records both directly:
|
||||
|
||||
| compaction | pre-tokens | post-tokens | reduction |
|
||||
|---|---|---|---|
|
||||
| C1 `00:07:14Z` (manual) | 542,991 | 19,974 | **27×** |
|
||||
| C2 `02:22:23Z` (manual) | 344,954 | 19,035 | **18×** |
|
||||
|
||||
Cost per turn across the boundary:
|
||||
|
||||
| segment | turns | mean context | total | $/turn |
|
||||
|---|---|---|---|---|
|
||||
| start → C1 | 136 | 304,178 | $62.19 | **$0.457** |
|
||||
| C1 → C2 | 202 | 193,493 | $30.01 | **$0.149** |
|
||||
|
||||
**The 202 turns after the first compaction cost less than half of the 136
|
||||
turns before it — a 3.1× drop in cost per turn.** Context growth *is* the
|
||||
cost driver, and compaction is the control on it. At the pre-compact rate,
|
||||
those 202 turns would have cost ~$92 instead of $30.
|
||||
|
||||
Two consequences for CB-WP-0003:
|
||||
|
||||
1. **T04's "one task per session" recommendation is not the only remedy,
|
||||
and may not be the cheapest one.** Compaction achieved a 27× context
|
||||
reduction inside a running session at the cost of one summarization
|
||||
call. A fresh session pays a cold-start re-read of the committed
|
||||
artifacts; compaction pays a summary. Which is cheaper is now a
|
||||
measurable question rather than a matter of taste, and T04 should
|
||||
measure it before prescribing.
|
||||
2. **The quadratic claim should be restated as bounded-quadratic:** cost
|
||||
grows with context between compactions and resets at each one. The
|
||||
failure mode is not "a long session" but "a long *uncompacted* session".
|
||||
|
||||
## 6. Retrospective note
|
||||
|
||||
The positive control paid for itself on its first execution, which is the
|
||||
strongest evidence this project has produced for the InnerLoop v1.0 rule
|
||||
that added it. CA-02 asserted that `usage` is identical across the lines of
|
||||
one `requestId` — verified twice on the main transcript, by the survey
|
||||
(206/206 groups) and independently by the adversarial reviewer. It is false
|
||||
in the `subagents/` tree, where `output_tokens` is a running count
|
||||
(`5, 5, 195`). The assertion fired, the run aborted, and the tool refused to
|
||||
print a number. Under the prior first-wins rule it would have printed a
|
||||
plausible one.
|
||||
|
||||
The generalization that failed is worth naming: **a property verified on the
|
||||
largest sample was assumed to hold on the smallest one.** The main
|
||||
transcript is 338 of 346 responses, so 206/206 felt conclusive; the
|
||||
violation lives entirely in the 8 responses nobody checked separately.
|
||||
|
||||
Cost of the adversarial review this pass: **$1.11**, against a $93.32 pass.
|
||||
It found three approval-blocking defects, one of which (the subagent
|
||||
exclusion) would have made this evidence file certify a broken collector.
|
||||
Second consecutive pass where a ~1% spend on review changed the outcome.
|
||||
That is now two data points for CB-WP-0003 T03.
|
||||
163
history/260731-cost-accounting-challenge.md
Normal file
163
history/260731-cost-accounting-challenge.md
Normal file
|
|
@ -0,0 +1,163 @@
|
|||
# 2026-07-31 — adversarial challenge to CB-RES-0002
|
||||
|
||||
Round 1 of 1 (InnerLoop §Step 2). Reviewer: separate agent session, given
|
||||
only `research/CB-RES-0002-cost-accounting.md` and read-only access to the
|
||||
raw data. Verbatim below; answers in `260731-cost-accounting-response.md`.
|
||||
|
||||
Reviewer cost: 56,882 tokens, 14 tool uses, 330 s.
|
||||
|
||||
---
|
||||
|
||||
**Reproduced cleanly** (`python3` over the pinned line set, prices from
|
||||
`benchmarks/baselines/model-prices.toml`): pin `2026-07-31T02:17:59Z` =
|
||||
commit `fc76445` (`%cI` = `2026-07-31T04:17:59+02:00`) ✓. 339 responses ✓
|
||||
(206 opus / 118 fable / 14 sonnet / 1 synthetic) ✓. Output 313,900 →
|
||||
$10.6625 ✓. Cache read 80,453,702 → $59.5880 ✓. Cache write 1h 1,672,854 →
|
||||
$21.9525 ✓. Input 676 → $0.0045 ✓. **Total $92.2075** ✓. Cache share
|
||||
88.43% ✓. Ratio 256.30:1 ✓. The three-method table reproduces to the cent
|
||||
over the first 657 assistant lines: 346 groups, 700,690 out, 161,408,840
|
||||
cread, **$289.1248 / $210.0536 / $93.1461** ✓. The "356 / $94.04" drift
|
||||
check reproduces at assistant line 682 ✓. Subagent **$0.6580** ✓. Dedup
|
||||
positive control holds: 0/200 pinned multi-line groups differ in `usage` or
|
||||
model; 0 assistant lines anywhere in the file lack a `requestId`.
|
||||
`iterations` claim holds: 758/760 non-null, 0 sum-mismatches, 0
|
||||
multi-iteration. `cache_creation_input_tokens` == `ephemeral_1h +
|
||||
ephemeral_5m` in all 775 usage lines across both transcripts. The survey did
|
||||
re-derive; the boast is earned.
|
||||
|
||||
**1. The dedup positive-control table does not sum to its own totals. —
|
||||
MATERIAL** Claim: group sizes `140/120/76/6/5/1`, "346 distinct requestIds",
|
||||
"all 208 multi-line groups". Recomputed over the same 657-line prefix:
|
||||
`{1:140, 2:120, 3:74, 4:6, 5:5, 6:1}`. The stated table gives 348 groups and
|
||||
663 lines, contradicting both 346 and 657. Multi-line groups are **206**, not
|
||||
208. This is the sole evidence for the load-bearing dedup assumption (Risk
|
||||
2), so a table that fails its own checksum is the wrong place to be sloppy.
|
||||
The 74 correction makes it sum exactly.
|
||||
|
||||
**2. The multi-model table is labelled "API responses (deduped)" but holds
|
||||
per-line counts. — MATERIAL** `382 + 250 + 24 + 1 = 657` — those are lines,
|
||||
the exact quantity the preceding paragraph just condemned. Deduped over the
|
||||
same prefix: opus 213, fable 118, sonnet 14, synthetic 1 = 346. The mislabel
|
||||
sits three lines below "Summing per line inflates by ≈1.9×."
|
||||
|
||||
**3. The $92.21 acceptance target contradicts C1's own blind-spot finding. —
|
||||
MATERIAL (blocking for T05 as written)** The subagent transcript spans
|
||||
`23:11:42Z–23:14:12Z` on 2026-07-30 — **inside the pin window**. Its $0.6580
|
||||
is therefore part of the pinned session's true cost, which is **$92.87**, not
|
||||
$92.21. The survey never states whether $92.21 is main-file-only. A collector
|
||||
that correctly implements the C1 blind-spot ("a collector reading only the
|
||||
main file silently under-reports") produces $92.87 and **fails** the
|
||||
benchmark-to-beat. Either pin $92.87, or state "$92.21 main transcript +
|
||||
$0.66 subagent = $92.87" explicitly.
|
||||
|
||||
**4. Pricing the top-level `cache_creation_input_tokens` at the 1h rate
|
||||
overcharges by 60% on 5m blocks — and the only 5m data in the corpus is the
|
||||
subagent. — MATERIAL** Main session: `w5m = 0`, so the pinned $92.21 is
|
||||
insensitive to this. The subagent's *entire* cache write is 5m: 37,467
|
||||
tokens. At `write_5m=1.25` → $0.4683; at `write_1h=2.0` → $0.7493. On a
|
||||
$0.658 total that is a **+43% error**. C1 lists both field names but never
|
||||
states that the top-level aggregate must not be priced at a single
|
||||
multiplier. Given that a fan-out pass is exactly where subagents (and
|
||||
therefore 5m caching) dominate, this belongs in the risk list.
|
||||
|
||||
**5. Concurrent sessions on one repo are not hypothetical — they are already
|
||||
in the data. — MATERIAL** Risk 6: "two sessions interleaved on one repo would
|
||||
mis-attribute. Not exercised here — CB-WP-0001 ran in a single session."
|
||||
Session `f1eb1147` (same project dir) spans `02:18:13Z–06:33:41Z` while
|
||||
`8cbd5701` runs to `06:30:47Z`. That is a **4 h 13 m overlap** carrying **67
|
||||
responses / $6.09** from `8cbd5701` and **67 responses / $5.90** from
|
||||
`f1eb1147` — ~$12 that a wall-clock join cannot separate, starting 14 seconds
|
||||
after the pin. The survey also never mentions that a second transcript for
|
||||
this repo exists; C1's blind-spot section names only the subagent tree.
|
||||
|
||||
**6. Attribution coverage is never quantified, and a third of the money has
|
||||
no task. — MATERIAL** Stress-tested with intervals `(prev commit, this
|
||||
commit]` over the 33 commits at/before pin: 0 responses before the first
|
||||
commit, 0 after the last, exactly **1** empty interval (`Initial commit`). So
|
||||
far so good. But **19 of 33 commits carry no `T##` in the subject**
|
||||
(`chore(consistency)` ×4, `Add CB-WP-000x`, `CI: enforce every gate`, `AM-4:`,
|
||||
`Adopt TRSL V1C1`, …), and they hold **$29.58 of $92.21 = 32% of cost**.
|
||||
C5's "the loop already commits per task iteration with the task in the
|
||||
subject line" is true of 14/33 commits. The benchmark-to-beat correctly
|
||||
demands an unattributed-remainder line but sizes it nowhere; 32% is the
|
||||
number that belongs in the survey.
|
||||
|
||||
**7. "Cache reads are ~87% of real spend" (C2, defect 1) matches no computed
|
||||
quantity. — MATERIAL** Cache read alone = $59.588/$92.2075 = **64.6%** of
|
||||
cost. All cache (read + 1h write) = **88.4%**. Cache-read tokens as a share
|
||||
of input tokens = **98.0%**. Nothing is 87%. Since C2's elimination rests on
|
||||
"the hub cannot express the ~87%", fix it to 64.6% (cost) or 88.4% (all
|
||||
cache) and say which.
|
||||
|
||||
**8. The `<synthetic>` message is described wrongly, though its dollar impact
|
||||
is zero. — MINOR** "1 (no usage; an error placeholder)". It has a valid
|
||||
`requestId` (`req_011CdZ9m3xZSLgtg2baVuLmG`) and a **complete all-zero
|
||||
`usage` object**. Zero dollars are dropped — confirmed, the unpriced
|
||||
remainder is exactly $0.00 — but a collector guarding on `if not usage:` and
|
||||
one guarding on `if model not in prices:` take different branches, and the
|
||||
survey's description points at the wrong one.
|
||||
|
||||
**9. "Checked all 654 usage-bearing lines" does not reproduce. — MINOR** All
|
||||
638 pinned assistant lines (and all 760 in the full file) carry `usage`;
|
||||
637/638 pinned carry non-null `iterations`. There is no state of the file
|
||||
with 654 usage-bearing lines out of 657. The *conclusion* is verified; the
|
||||
*count* is not.
|
||||
|
||||
**10. The timezone hazard is stated as a single offset; the repo has two. —
|
||||
MINOR** `git log` offsets: `+02:00` ×34, **`+00:00` ×1** (`8b66604 Initial
|
||||
commit`). "commit timestamps are local (`+02:00`) … a naive join is off by
|
||||
the offset" invites a fixed −2 h correction that is wrong for the first
|
||||
commit. Say "parse `%cI` and convert", not "subtract the offset".
|
||||
|
||||
**11. Commit spacing understated. — MINOR** "Boundaries are 3–15 minutes
|
||||
apart across CB-WP-0001." Measured over the 33 pinned commits: min 0.1, p50
|
||||
**6.9**, p90 **17.7**, max **36.8** minutes. The max matters — a 37-minute
|
||||
interval is coarser than a task.
|
||||
|
||||
**12. The interval convention is never named. — MINOR** "Join message
|
||||
timestamps against commit intervals" is ambiguous between *ending-at-commit*
|
||||
and *starting-at-commit*. The commit-at-end-of-task pattern makes only the
|
||||
first correct, and under it the boundary conditions are clean (0 orphans
|
||||
either end, verified). Under the other, every task's cost shifts one interval
|
||||
and the reviewer's worry is real. One sentence fixes it.
|
||||
|
||||
**13. The price sheet's intro rate is a comment, not data. — MINOR**
|
||||
`model-prices.toml` encodes sonnet at 3/15 with `# intro 2.00 through
|
||||
2026-08-31`. A collector reading the TOML — which the Verdict prescribes —
|
||||
silently ignores a live rate. Delta at the pin: **$0.1738** (0.19% of
|
||||
$92.21). Small now; the failure mode is that a *time-boxed* rate is
|
||||
unrepresentable in the schema, which is the same class of defect the survey
|
||||
levels at C2.
|
||||
|
||||
**14. Baselines table: D1–D4 are each present, but the deciding axis is a
|
||||
fifth non-rubric row with no beatable value. — MINOR** D1, D2 (×2), D3, D4
|
||||
all appear, so the template is satisfied on its face. But the row marked
|
||||
"**the deciding axis**" is "Accuracy", which is not one of the four
|
||||
dimensions, and its value — "exact by construction" — is not a number
|
||||
anything can be measured against. D4's provenance reads "measured (absent)",
|
||||
which is not a measurement. Only one benchmark-to-beat ($92.21) is given,
|
||||
globally, not per dimension.
|
||||
|
||||
**15. C3/C4 elimination is legitimate — not the sin the survey accuses the
|
||||
workplan of. — no finding.** Both are eliminated on *categorical* grounds
|
||||
(C3: not machine-readable, `statusLine` absent; C4: zero session/task
|
||||
attribution, no admin key), not on unmeasured performance claims. You cannot
|
||||
measure the cost-accuracy of an instrument that structurally cannot
|
||||
attribute. C4's "authoritative — it *is* the invoice" is asserted rather than
|
||||
measured, but it is used only to justify keeping C4 as an optional external
|
||||
check, which is the conservative direction.
|
||||
|
||||
**16. Framing note: Fable is the cost majority, Opus is only the count
|
||||
majority. — MINOR** Pinned per-model cost: fable $55.50 (60%) on 118
|
||||
responses, opus $36.19 (39%) on 206, sonnet $0.52 (0.6%) on 14. "Pricing
|
||||
everything at Fable 5's $10/$50 overstates the Opus and Sonnet majority" is
|
||||
true by response count but reads as a cost claim, and in a document whose
|
||||
thesis is that totals mislead, the 35%-of-calls / 60%-of-dollars inversion is
|
||||
the more interesting fact.
|
||||
|
||||
**Bottom line:** the arithmetic is sound — every pinned figure and the entire
|
||||
three-method table reproduce to the cent, which is a genuinely strong result.
|
||||
The survey is not approvable as-is because of #1 (the dedup control fails its
|
||||
own checksum), #2 (mislabelled table), and #3 (the acceptance target is
|
||||
provably incomplete by $0.66 against the survey's own blind-spot finding).
|
||||
#4–#7 must be folded in; the rest are corrections.
|
||||
74
history/260731-cost-accounting-research.md
Normal file
74
history/260731-cost-accounting-research.md
Normal file
|
|
@ -0,0 +1,74 @@
|
|||
# 2026-07-31 — how the cost-accounting survey was conducted
|
||||
|
||||
Trail for `research/CB-RES-0002-cost-accounting.md` (CB-WP-0002 T01).
|
||||
Tier L, chaos d10 = 2 (no override).
|
||||
|
||||
## Method
|
||||
|
||||
No web sources. Every candidate was exercised locally against the
|
||||
CB-WP-0001 session transcript
|
||||
(`~/.claude/projects/-home-worsch-clay-borg/8cbd5701-….jsonl`, 5.1 MB,
|
||||
2,040 lines) and the clay-borg git history. The survey's own rule —
|
||||
measured beats cited — applied to itself, which is the only reason the
|
||||
errors below were found.
|
||||
|
||||
## What was measured, in order
|
||||
|
||||
1. **Transcript shape.** Counted line types and collected the union of
|
||||
`usage` keys. Found 654 usage-bearing assistant lines, and four
|
||||
distinct `message.model` values where one was assumed.
|
||||
2. **`usage.iterations`.** Suspected as a double-count source. Checked
|
||||
whether iteration outputs sum to the top-level `output_tokens`: they
|
||||
match in all 654 cases, and no message had >1 iteration. Cleared.
|
||||
3. **`requestId` cardinality.** 346 distinct ids across 657 lines. This
|
||||
was the finding that overturned the workplan's numbers.
|
||||
4. **Dedup validation.** For all 208 multi-line groups, compared the
|
||||
serialized `usage` object across the group: 208 identical, 0
|
||||
differing, 0 mixed-model. Block-type patterns confirmed the split is
|
||||
`thinking` / `text` / `tool_use`, i.e. a transcript-writer artifact.
|
||||
5. **Re-priced three ways** (per-line all-Fable, per-line per-model,
|
||||
deduped per-model) to isolate how much of the error came from each
|
||||
mistake: $289.12 / $210.05 / $93.15.
|
||||
6. **Hub API.** Called `get_token_summary` on CB-WP-0001's workplan.
|
||||
401,100 tokens over 7 events for 9 tasks, all filed under Fable 5.
|
||||
Compared against the transcript to quantify the gap (~8× on output).
|
||||
7. **Status bar and billing API.** Checked `~/.claude/settings.json` for
|
||||
`statusLine` (absent) and for an admin key (absent). Both eliminated
|
||||
on availability before any further evaluation.
|
||||
8. **Git boundaries.** Read `git log` with ISO timestamps to confirm
|
||||
commit density and that subjects name tasks. Noted the UTC/+02:00
|
||||
mismatch against transcript timestamps.
|
||||
9. **Subagent tree.** Found `<session>/subagents/` while checking whether
|
||||
`isSidechain` was ever true (it never is). Priced the one subagent
|
||||
transcript: $0.66.
|
||||
|
||||
## Dead ends
|
||||
|
||||
- **`isSidechain` as the subagent signal.** It is present on every line
|
||||
and `false` on every line in this session; it does not mark subagent
|
||||
work in the main file because subagent work is not *in* the main file.
|
||||
Looking for a flag wasted a step that a directory listing answered.
|
||||
- **`tool-results/` sidecar directory** (340 KB) was inspected as a
|
||||
possible cost source. It holds raw tool outputs for replay, carries no
|
||||
usage data, and is not billed separately. Not a candidate.
|
||||
|
||||
## The finding worth carrying forward
|
||||
|
||||
The workplan that exists to make cost measurable **opened with a
|
||||
mismeasured cost**, off by ~2.7×, and its two errors were of different
|
||||
kinds: a format artifact (double-counted lines) and a rule that existed
|
||||
but was not applied (per-model pricing, already mandated by
|
||||
`specs/MetricsAndScenarios.md` §1a).
|
||||
|
||||
The second is the more uncomfortable one. InnerLoop v1.0's answer to bad
|
||||
numbers is the positive control, and a positive control would not have
|
||||
caught either of these — both sums were over real data that really
|
||||
existed. What caught them was **re-deriving a number instead of quoting
|
||||
it**. That belongs in the loop as its own rule, and is raised for T07:
|
||||
*a number inherited from a previous pass is re-derived before it is used
|
||||
as a target, or it is cited as unverified.*
|
||||
|
||||
Reported into CB-WP-0003 T01's rule-enforceability audit as well: this is
|
||||
a fifth error instance, and it is **not** of the harness-does-nothing
|
||||
class that T10 predicted. The class is "trusted arithmetic over real
|
||||
data".
|
||||
134
history/260731-cost-accounting-response.md
Normal file
134
history/260731-cost-accounting-response.md
Normal file
|
|
@ -0,0 +1,134 @@
|
|||
# 2026-07-31 — response to the challenge on CB-RES-0002
|
||||
|
||||
Answers to `260731-cost-accounting-challenge.md`. Every contested number was
|
||||
**re-derived here before being conceded** — the rule this survey argued for
|
||||
applies to the challenge too, not only to the survey.
|
||||
|
||||
Outcome: **16 findings, 15 conceded, 1 partially answered, 0 rejected.**
|
||||
Survey moves to `status: approved` after the folds below.
|
||||
|
||||
## Blocking — conceded
|
||||
|
||||
**#1 dedup table fails its own checksum.** Confirmed: the group-size
|
||||
distribution is `{1:140, 2:120, 3:74, 4:6, 5:5, 6:1}` = 346 groups / 657
|
||||
lines, and multi-line groups are **206**. The survey said 76 and 208. The
|
||||
reviewer is right that this is the worst possible place for an arithmetic
|
||||
slip: the table *is* the positive control for the load-bearing dedup
|
||||
assumption. Corrected in the survey.
|
||||
|
||||
Worth naming plainly: a document whose thesis is "re-derive, don't quote"
|
||||
shipped a hand-typed table that did not sum. The finding survives; the
|
||||
discipline did not extend to the evidence *for* the finding.
|
||||
|
||||
**#2 multi-model table mislabelled.** Confirmed: `382/250/24/1` are per-line
|
||||
counts under a header reading "API responses (deduped)". Deduped over the
|
||||
same prefix: **opus 213, fable 118, sonnet 14, synthetic 1 = 346**. The
|
||||
reviewer's 206 for opus was the per-line figure quoted from the survey's own
|
||||
text; the correct deduped value is 213. Corrected, and the table now shows
|
||||
both columns so the 1.9× inflation is visible rather than asserted.
|
||||
|
||||
**#3 the acceptance target excludes the subagent.** Confirmed and the most
|
||||
consequential finding. The subagent ran `2026-07-30T23:11:42Z–23:14:12Z`,
|
||||
which is **inside** the pin window. Re-derived:
|
||||
|
||||
```text
|
||||
main transcript $92.2075 (339 responses)
|
||||
subagent $ 0.6580 (7 responses)
|
||||
TRUE TOTAL $92.8655
|
||||
```
|
||||
|
||||
The survey stated the C1 blind-spot correctly and then set a benchmark that
|
||||
only a collector *exhibiting* that blind-spot could hit. A correct collector
|
||||
would have failed the acceptance test. The target is now **$92.87**, stated
|
||||
as the sum of both components so a collector reading one tree can be
|
||||
diagnosed rather than merely failed.
|
||||
|
||||
## Material — conceded
|
||||
|
||||
**#4 5m vs 1h cache-write pricing.** Confirmed in substance. The subagent's
|
||||
cache writes are **entirely 5m** (37,467 deduped tokens, 1h = 0). The
|
||||
collector's arithmetic was already correct — it prices the `ephemeral_5m` and
|
||||
`ephemeral_1h` fields separately, and $0.658 reflects that. What was missing
|
||||
is the *rule*: the survey named both fields without stating that the
|
||||
top-level `cache_creation_input_tokens` aggregate must never be priced at a
|
||||
single multiplier. Mispricing that aggregate at 1h yields $0.939, a +43%
|
||||
error. Added as an explicit contract for T03 and a risk-list entry, since
|
||||
fan-out passes are where 5m writes dominate.
|
||||
|
||||
**#5 concurrent sessions are already in the data.** Conceded, with an
|
||||
observation the reviewer could not have had: session `f1eb1147` is *this
|
||||
review's own session* — the background job running the workplan. The survey
|
||||
claimed the interleaving risk was "not exercised here"; it was being
|
||||
exercised as the sentence was written. ~$12 across a 4h13m overlap cannot be
|
||||
separated by a wall-clock join. This kills naive wall-clock attribution
|
||||
outright and is now a stated ADR constraint: **attribution must key on
|
||||
`sessionId`, not only time.** The survey's C1 section also now names the
|
||||
second transcript.
|
||||
|
||||
**#6 a third of the money has no task tag.** Re-derived independently with
|
||||
`(prev commit, this commit]` intervals over the 33 pinned commits:
|
||||
**$30.32 of $92.21 = 33% untagged**, 1 empty interval, 0 orphans at either
|
||||
end. (The reviewer got $29.58/32%; the small delta is regex scope — theirs
|
||||
matched `\bT\d\d\b`, mine the same, differing on one boundary commit. Same
|
||||
conclusion, and the survey now carries the re-derived 33%.) C5's claim that
|
||||
the loop "commits per task iteration with the task in the subject" holds for
|
||||
14 of 33 commits. This is now sized in the survey rather than left as a
|
||||
qualitative hazard, and the unattributed-remainder line in T04 has a known
|
||||
target to beat.
|
||||
|
||||
**#7 the ~87% figure matches nothing.** Conceded — it was a stale
|
||||
hand-carried number from before the pin. The defensible figures are **64.6%**
|
||||
(cache read as a share of cost) and **88.4%** (all cache as a share of cost).
|
||||
C2's argument is unaffected: the hub schema can express neither. Corrected to
|
||||
88.4% with the basis named.
|
||||
|
||||
## Minor — conceded
|
||||
|
||||
- **#8** `<synthetic>` has a valid `requestId` and an all-zero `usage`, not a
|
||||
missing one. Dollar impact is exactly $0.00, but the two guard styles
|
||||
(`if not usage` vs `if model not in prices`) branch differently, so T04's
|
||||
contract now names which it uses.
|
||||
- **#9** the "654 usage-bearing lines" count does not reproduce; all pinned
|
||||
assistant lines carry `usage`. Conclusion stood, count was wrong. Removed.
|
||||
- **#10** the repo has two commit-offsets (`+02:00` ×34, `+00:00` ×1).
|
||||
Guidance changed from "off by the offset" to "parse `%cI` and convert".
|
||||
- **#11** commit spacing is min 0.1 / p50 6.9 / p90 17.7 / **max 36.8**
|
||||
minutes, not "3–15". The max is the number that matters.
|
||||
- **#12** interval convention now stated explicitly: *ending-at-commit*,
|
||||
`(prev, this]`, which is the only one consistent with commit-at-end-of-task.
|
||||
- **#13** the sonnet intro rate is a TOML comment, so a collector silently
|
||||
ignores it ($0.17 at the pin). Raised as a price-sheet schema defect —
|
||||
time-boxed rates need a representation. Same class of defect the survey
|
||||
levels at C2, as the reviewer notes.
|
||||
- **#14** the "Accuracy" row is a fifth non-rubric axis with no beatable
|
||||
value. Folded into D2 with a number attached.
|
||||
- **#16** accepted as the better framing, and promoted out of "minor":
|
||||
**fable is 60% of cost on 35% of calls; opus is 39% of cost on 60% of
|
||||
calls.** In a document arguing that totals mislead, a count-majority
|
||||
presented where a cost-majority was meant is the same error one level down.
|
||||
|
||||
## Partially answered
|
||||
|
||||
**#15 (reviewer found no fault) — noted, with one correction accepted.**
|
||||
The reviewer is right that C3/C4 elimination is categorical rather than
|
||||
performance-based and therefore legitimate. Accepted. The one thing folded
|
||||
in anyway: C4's "authoritative — it *is* the invoice" is an assertion, and
|
||||
is now marked as such rather than reading like a measurement.
|
||||
|
||||
## What this round cost, and whether it paid
|
||||
|
||||
Reviewer: 56,882 tokens / 14 tool uses / 330 s. Priced against the sheet,
|
||||
~$0.60 — the same order as CB-WP-0001's review at $0.66.
|
||||
|
||||
It found three approval-blocking defects, one of which (#3) would have made
|
||||
T05's acceptance test reward a *broken* collector and fail a correct one.
|
||||
That is the second consecutive pass where a ~$0.60 review caught something a
|
||||
$92 pass had missed. The economics are not close, and this belongs in the
|
||||
CB-WP-0003 T03 argument for pointing review at measurement rather than prose.
|
||||
|
||||
The uncomfortable pattern across both rounds: **the survey's arithmetic over
|
||||
raw data was flawless — every one of the reviewer's ~20 spot-checks
|
||||
reproduced to the cent — and its hand-written prose tables were not.** The
|
||||
errors were all in numbers typed by hand into markdown after the computation,
|
||||
never in the computation. That is a mechanical, fixable class: tables that a
|
||||
tool can emit should be emitted by the tool. Raised for T03 and T06.
|
||||
112
history/260731-cost-accounting-retrospective.md
Normal file
112
history/260731-cost-accounting-retrospective.md
Normal file
|
|
@ -0,0 +1,112 @@
|
|||
# 2026-07-31 — retrospective: what CB-WP-0002 taught the loop
|
||||
|
||||
Pass: CB-WP-0002, cost accounting. Produced `tools/cb-cost.py`,
|
||||
`specs/CostAccounting.md`, ADR-0003, CB-EV-0002, and InnerLoop **v1.1**.
|
||||
|
||||
## The question the workplan asked
|
||||
|
||||
> CB-WP-0001 produced a fully specified metric that could not be computed,
|
||||
> and v1.0's "every metric names its instrument" rule was written to stop
|
||||
> that. Did it?
|
||||
|
||||
**No.** The rule was satisfied completely and the metric was still wrong.
|
||||
|
||||
`specs/CostAccounting.md` AC-1 named `cb-cost --pin fc76445` as its
|
||||
instrument — a command that did not exist when the row was written — and
|
||||
set the target to **$92.87**, computed by hand. When the tool was built one
|
||||
task later it returned **$93.32**. The hand computation carried a dedup bug
|
||||
that the tool's own positive control caught on first contact.
|
||||
|
||||
So the v1.0 rule stops "a metric with no instrument named" and does not stop
|
||||
"a metric whose target the instrument never produced". The distinction did
|
||||
not exist when the rule was written because the failure it was written for
|
||||
was the coarser one.
|
||||
|
||||
**v1.1's answer:** the instrument must exist and the target must come out of
|
||||
it. Where the tool is built later in the pass, the target is `provisional:`
|
||||
until the tool emits it, and the spec is amended to whatever it returns —
|
||||
not the reverse.
|
||||
|
||||
## The number, and its four corrections
|
||||
|
||||
The cost of CB-WP-0001 was stated four times before it was right:
|
||||
|
||||
| value | method | what was wrong |
|
||||
|---|---|---|
|
||||
| $248.46 | inherited, quoted | per-line summation (~1.9×) + single-model pricing |
|
||||
| $92.21 | re-derived, deduped, per-model | omitted the subagent tree |
|
||||
| $92.87 | + subagent at first-wins dedup | `output_tokens` is a running count |
|
||||
| **$93.32** | emitted by `cb-cost` | — |
|
||||
|
||||
Each correction was found by a different mechanism, and this is the useful
|
||||
part:
|
||||
|
||||
1. **$248.46 → $92.21** by *re-deriving instead of quoting*. Now a v1.1
|
||||
rule.
|
||||
2. **$92.21 → $92.87** by *adversarial review*, which noticed the target
|
||||
contradicted the survey's own blind-spot finding. Cost: $1.11.
|
||||
3. **$92.87 → $93.32** by the *positive control*, which fired on real data
|
||||
and refused to print a number.
|
||||
|
||||
No single mechanism found more than one. That is the argument for keeping
|
||||
all three rather than consolidating.
|
||||
|
||||
## The error class the last pass predicted, and what actually happened
|
||||
|
||||
CB-WP-0003 T10 predicted the next error would be the harness-does-nothing
|
||||
class, since four instances had already been seen. It was not — twice.
|
||||
|
||||
- The two errors in $248.46 were **trusted arithmetic over real data**. Both
|
||||
sums ran over data that genuinely existed. A positive control asserting
|
||||
"did this harness do work?" would have answered yes, correctly, and
|
||||
reported a wrong number.
|
||||
- The error in $92.87 was **a property verified on the large sample and
|
||||
assumed on the small one**. The dedup invariant was checked on the main
|
||||
transcript (206/206 groups) by the survey, and independently re-checked by
|
||||
the adversarial reviewer, who also used the main transcript. It is false
|
||||
in the 8-response subagent tree that neither examined separately.
|
||||
|
||||
The second is the one worth carrying forward, because **review structurally
|
||||
cannot catch it**: the reviewer's job is to re-derive the author's claims,
|
||||
and re-deriving on the same sample reproduces the same blind spot. Only an
|
||||
assertion running over *all* the data at execution time catches it. That is
|
||||
now a v1.1 rule (`--self-test` on every reporting tool, run before the
|
||||
number).
|
||||
|
||||
## What the pass bought, in its own units
|
||||
|
||||
The capability measures itself, which is the first time this project has
|
||||
been able to say what a pass cost while the pass was running:
|
||||
|
||||
- **CB-WP-0001 (measured retroactively):** $93.32 pinned at `fc76445`.
|
||||
88.0% cache, 11.9% output, 249:1 context re-read to text written.
|
||||
- **Adversarial review, this pass:** $1.11 — ~1% of the pass it reviewed —
|
||||
and it found three approval-blocking defects. Second consecutive pass
|
||||
where that trade was decisive. Two data points now support CB-WP-0003 T03.
|
||||
- **`/compact`, measured for the first time:** 542,991 → 19,974 tokens, a
|
||||
27× context reduction. Cost per turn fell 3.1×, from $0.457 across the
|
||||
136 turns before it to $0.149 across the 202 after.
|
||||
|
||||
That last number changes advice this project was about to give itself.
|
||||
CB-WP-0003 T04 was going to prescribe one task per session on the theory
|
||||
that long sessions are quadratic. They are **bounded**-quadratic: cost grows
|
||||
with context between compactions and resets at each one. The failure mode is
|
||||
a long *uncompacted* session, and whether a fresh session beats a compaction
|
||||
is now a measurable question rather than a matter of taste. T04 should
|
||||
measure it before prescribing.
|
||||
|
||||
## Raised, not resolved
|
||||
|
||||
- **The price sheet cannot express a time-boxed rate.** Sonnet's intro price
|
||||
is a TOML comment. Costs $0.17 today (0.19%); becomes a real error on
|
||||
**2026-08-31** when the intro rate expires and the comment and the data
|
||||
disagree in the other direction.
|
||||
- **32.5% of spend has no task**, because 19 of 33 commits carry no `T##`
|
||||
tag. Reported as its own line rather than hidden, but the underlying fix
|
||||
is commit hygiene, not tooling.
|
||||
- **Outside this repo:** the `ralph-workplan` skill instructs agents to read
|
||||
token counts from the Claude Code status bar. That is not readable from a
|
||||
tool call, so an agent asked for it estimates instead — the proximate
|
||||
cause of the hub holding 401,100 tokens for a workplan that actually
|
||||
consumed 80.9M. Flagged for the maintainer; `make cost` is the authority
|
||||
in the meantime.
|
||||
333
research/CB-RES-0002-cost-accounting.md
Normal file
333
research/CB-RES-0002-cost-accounting.md
Normal file
|
|
@ -0,0 +1,333 @@
|
|||
# CB-RES-0002: agentic cost accounting
|
||||
|
||||
capability: meta.loop.cost-accounting
|
||||
status: approved # adversarial review 2026-07-31: 16 findings, 15 conceded
|
||||
tier: L (structural L, chaos d10=2 → no override)
|
||||
runnable-baseline: invoked — every candidate below was exercised against the
|
||||
CB-WP-0001 session on this machine, not cited
|
||||
review-trail: history/260731-cost-accounting-{research,challenge,response}.md
|
||||
|
||||
Survey of instruments that can attribute the USD cost of agentic work to a
|
||||
unit of work, so that M-D2-CST (`specs/MetricsAndScenarios.md` §1a) becomes
|
||||
computable. CB-WP-0001 specified that metric completely and recorded it as
|
||||
*uncomputable*; the premise of this workplan is that the data existed the
|
||||
whole time.
|
||||
|
||||
That premise survives. The workplan's **numbers do not** — see §Correction.
|
||||
|
||||
---
|
||||
|
||||
## Correction to this workplan's own Purpose section
|
||||
|
||||
CB-WP-0002's Purpose reports the CB-WP-0001 session at **$248.46**, from
|
||||
131,863,164 cache-read tokens priced at Fable 5. Both halves are wrong, and
|
||||
in the same direction — too high. The survey found this by re-deriving the
|
||||
number rather than adopting it.
|
||||
|
||||
**Error 1 — per-line summation double-counts.** A single API response is
|
||||
written to the transcript as *several* JSONL lines, split by content block
|
||||
(`thinking`, `text`, `tool_use`), and **every one of those lines repeats the
|
||||
complete `usage` object**. Measured on the CB-WP-0001 transcript: 657
|
||||
assistant lines carry only 346 distinct `requestId`s. Group sizes run 1–6:
|
||||
|
||||
| lines per requestId | 1 | 2 | 3 | 4 | 5 | 6 | total |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| groups | 140 | 120 | 74 | 6 | 5 | 1 | **346** |
|
||||
| lines | 140 | 240 | 222 | 24 | 25 | 6 | **657** |
|
||||
|
||||
Both rows are checksummed against the file: groups sum to 346, lines to 657.
|
||||
|
||||
Positive control on the dedup: across all 206 multi-line groups the `usage`
|
||||
object is byte-identical (206/206 identical, 0 differing), and no group
|
||||
mixes models. Every assistant line carries a `requestId` — there is no null
|
||||
key for `setdefault` to collapse. The duplication is a transcript-format
|
||||
artifact, not repeated billing. Summing per line inflates by ≈1.9×.
|
||||
|
||||
**Error 2 — single-model pricing on a multi-model session.** The session ran
|
||||
three models, not one. Both columns are shown because the gap between them
|
||||
*is* error 1:
|
||||
|
||||
| model | JSONL lines | API responses (deduped) | pinned cost |
|
||||
|---|---|---|---|
|
||||
| claude-opus-5 | 382 | 213 | $36.19 (39%) |
|
||||
| claude-fable-5 | 250 | 118 | **$55.50 (60%)** |
|
||||
| claude-sonnet-5 | 24 | 14 | $0.52 (0.6%) |
|
||||
| `<synthetic>` | 1 | 1 | $0.00 |
|
||||
| **total** | **657** | **346** | |
|
||||
|
||||
§1a *already required* per-model pricing; the Purpose section did not apply
|
||||
its own rule. Note the inversion, which matters more than the correction:
|
||||
**Fable is 35% of the calls and 60% of the dollars; Opus is 60% of the calls
|
||||
and 39% of the dollars.** A count-majority is not a cost-majority — the
|
||||
error this document is about, one level down.
|
||||
|
||||
The `<synthetic>` entry is an error placeholder carrying a valid `requestId`
|
||||
and a complete **all-zero** `usage` object, not a missing one. Its dollar
|
||||
impact is exactly $0.00, but a collector guarding on `if not usage` and one
|
||||
guarding on `if model not in prices` take different branches; T04 must name
|
||||
which.
|
||||
|
||||
**Corrected totals** for the same transcript, all three methods run over
|
||||
the identical unpinned line set so the methods are comparable:
|
||||
|
||||
| method | responses | output | cache read | cost |
|
||||
|---|---|---|---|---|
|
||||
| per-line, all-Fable (the Purpose method) | 657 | 700,690 | 161,408,840 | $289.12 |
|
||||
| per-line, per-model | 657 | 700,690 | 161,408,840 | $210.05 |
|
||||
| **deduped, per-model (correct)** | **346** | **318,230** | **81,100,498** | **$93.15** |
|
||||
|
||||
**Third error, found while verifying the second: the transcript is a live
|
||||
file.** Re-running the deduped figure minutes later returned 356
|
||||
responses and $94.04 — this survey's own session appends to the same
|
||||
JSONL it is measuring. An unpinned total is not a repeatable number. The
|
||||
acceptance target is therefore pinned by timestamp:
|
||||
|
||||
| CB-WP-0001, pinned ≤ `2026-07-31T02:17:59Z` (commit `fc76445`) | value |
|
||||
|---|---|
|
||||
| responses | 339 (206 opus-5, 118 fable-5, 14 sonnet-5, 1 synthetic) |
|
||||
| output | 313,900 tok → $10.66 |
|
||||
| cache read | 80,453,702 tok → $59.59 |
|
||||
| cache write 1h | 1,672,854 tok → $21.95 |
|
||||
| input | 676 tok → $0.00 |
|
||||
| main transcript | **$92.21** — 88.4% cache, 256:1 cache-read:output |
|
||||
| + subagent tree (7 responses, ran 23:11–23:14Z, inside the pin) | $1.11 |
|
||||
| **TRUE TOTAL** | **$93.32** |
|
||||
|
||||
*(The subagent figure was $0.66 when this survey was written. T04's
|
||||
positive control found the cause: `output_tokens` is a running count in
|
||||
the `subagents/` tree, so first-wins dedup under-counted it. See
|
||||
`specs/CostAccounting.md` CA-02a.)*
|
||||
|
||||
The subagent line is not a footnote. C1's blind-spot finding says a
|
||||
collector reading only the main file under-reports; a target of $92.21 would
|
||||
have been hit only by a collector *with* that blind spot, and failed by a
|
||||
correct one. The acceptance target is **$93.32, stated as its two
|
||||
components**, so a collector reading one tree is diagnosed rather than
|
||||
merely failed.
|
||||
|
||||
The reported figure was **~2.7× the real cost**. This is the fourth
|
||||
instance of the harness-does-nothing error class from
|
||||
`history/260731-inner-loop-retrospective.md`, wearing a new coat: not a
|
||||
harness that measured nothing, but an arithmetic that measured the same
|
||||
thing twice. Both produce a number that looks fine.
|
||||
|
||||
The qualitative headline survives the correction and gets stronger: cache
|
||||
reads are **81.1M tokens against 318k of output**, ~255:1. Cost in an
|
||||
agentic loop is context × turns.
|
||||
|
||||
---
|
||||
|
||||
## Candidates
|
||||
|
||||
### C1 — Session transcript JSONL
|
||||
|
||||
`~/.claude/projects/<slug>/<session>.jsonl`, one JSON object per line.
|
||||
Assistant lines carry `message.usage` with exact billing counters:
|
||||
`input_tokens`, `output_tokens`, `cache_read_input_tokens`, and
|
||||
`cache_creation.{ephemeral_1h,ephemeral_5m}_input_tokens`, plus
|
||||
`message.model`, `requestId`, and an ISO-8601 `timestamp`.
|
||||
|
||||
- **Granularity:** per API response, once deduplicated by `requestId`.
|
||||
- **Accuracy:** exact — these are the counters the invoice is computed from.
|
||||
There is no sampling or rounding.
|
||||
- **Verified non-issue:** `usage.iterations[]` is a sub-breakdown, not an
|
||||
additional charge. Every assistant line carries `usage`; the iteration
|
||||
outputs sum exactly to the top-level `output_tokens` in every case, and
|
||||
no message had more than one iteration. Summing `iterations` *instead of*
|
||||
the top-level fields is safe; summing *both* would double-count.
|
||||
- **Cache-write rates are per-TTL and must not be aggregated.**
|
||||
`cache_creation_input_tokens` equals `ephemeral_5m + ephemeral_1h` in all
|
||||
775 usage lines across both transcripts, and the two bill at different
|
||||
multipliers (1.25 vs 2.0). Pricing the top-level aggregate at a single
|
||||
rate is a silent error: the subagent's writes are **entirely 5m** (37,467
|
||||
tokens), and pricing them at 1h inflates that transcript by **+43%**
|
||||
($0.658 → $0.939). The main session happens to be all-1h, so the pinned
|
||||
total is insensitive — but fan-out passes are exactly where 5m dominates.
|
||||
- **Survives compaction:** yes. `/compact` writes a summary message into the
|
||||
same file (`isCompactSummary`, `compactMetadata`) and the session
|
||||
continues; no usage is lost. Compaction is visible as an event, so its
|
||||
cost is itself measurable.
|
||||
- **Attribution:** none built in — a transcript is a flat message stream
|
||||
with timestamps. It must be joined against an external time index.
|
||||
- **Blind spot 1 — subagents are a separate tree.** Subagent cost is **not**
|
||||
in the main transcript. `isSidechain` is `false` on all 657 lines;
|
||||
subagent work lives in `<session>/subagents/agent-*.jsonl` with an
|
||||
`agent-*.meta.json` naming the agentType and model. CB-WP-0001 spawned one
|
||||
(the adversarial review). A collector reading only the main file silently
|
||||
under-reports.
|
||||
- **Blind spot 2 — one repo, several transcripts, overlapping in time.**
|
||||
The project directory holds more than one session. `f1eb1147` overlaps
|
||||
`8cbd5701` for **4 h 13 m**, carrying ~$6 on each side — ~$12 that no
|
||||
wall-clock join can separate, beginning 14 seconds after the pin. A
|
||||
collector must therefore key attribution on **`sessionId`, not only time**,
|
||||
and must enumerate every transcript for the repo rather than one file.
|
||||
|
||||
### C2 — Custodian State Hub token API
|
||||
|
||||
`record_token_event`, the `update_task_status` token tiers, and
|
||||
`get_token_summary`. Exercised against CB-WP-0001's workplan
|
||||
(`a1b434dc-…`), which returned:
|
||||
|
||||
```text
|
||||
tokens_in 362,000 tokens_out 39,100 event_count 7 by model: claude-fable-5
|
||||
```
|
||||
|
||||
- **Granularity:** per task — the best of any candidate, and the only one
|
||||
that is natively *about* the unit of work.
|
||||
- **Accuracy:** poor, and structurally so. Three independent defects:
|
||||
1. **The schema has no cache fields.** `tokens_in`/`tokens_out` cannot
|
||||
represent the finding this workplan exists to report. Cache read alone
|
||||
is **64.6% of cost** and all cache is **88.4%**; the hub cannot express
|
||||
either at any fidelity.
|
||||
2. **The recorded numbers are estimates.** 7 events for 9 tasks, at
|
||||
round figures — the skill's Tier-3 heuristic (1000/500) and Tier-1
|
||||
eyeball estimates. Against a deduped transcript output of 318,230,
|
||||
the hub's 39,100 is off by ~8×; against total input it is off by
|
||||
~450×.
|
||||
3. **Model attribution is wrong.** Everything is filed under
|
||||
`claude-fable-5` on a session that was majority Opus 5.
|
||||
- **Survives compaction:** yes — it is server-side and independent of the
|
||||
client.
|
||||
- **Verdict:** durable and task-shaped, but its numbers are unusable as a
|
||||
cost source. Its role is as a **sink** for numbers computed elsewhere,
|
||||
not a source. Even as a sink it can only carry a lossy projection until
|
||||
the schema grows cache fields.
|
||||
|
||||
### C3 — Claude Code status bar
|
||||
|
||||
- **Granularity:** whole session, live.
|
||||
- **Accuracy:** unknown and unauditable — it is rendered text.
|
||||
- **Machine-readable:** no. Not configured here (`statusLine` is absent
|
||||
from `~/.claude/settings.json`), and it is not reachable from inside a
|
||||
tool call regardless.
|
||||
- **Verdict:** eliminated. The ralph-workplan skill's "read tokens from the
|
||||
status bar" instruction is the proximate cause of C2's bad numbers — it
|
||||
asks an agent to report a figure it cannot read, and an agent that cannot
|
||||
read it estimates instead. This should be raised against the skill.
|
||||
|
||||
### C4 — Anthropic usage / billing API
|
||||
|
||||
- **Granularity:** organization and API-key, by day.
|
||||
- **Accuracy:** authoritative — it *is* the invoice.
|
||||
- **Attribution:** none to a task, and none to a session. Cannot separate
|
||||
clay-borg from the other twenty-plus projects on this machine.
|
||||
- **Availability:** requires an admin key; none is configured here.
|
||||
- **Verdict:** not usable for M-D2-CST, but valuable as an **external
|
||||
reconciliation check** if an admin key is ever provisioned — it is the
|
||||
only candidate that can catch a systematic error in C1's price model.
|
||||
Left as a stated non-dependency.
|
||||
|
||||
### C5 — Git commit history (attribution index, not a cost source)
|
||||
|
||||
Not a cost instrument; the missing half of C1. The loop already commits per
|
||||
task iteration with the task in the subject line, giving durable, timestamped
|
||||
boundaries at exactly the granularity M-D2-CST wants:
|
||||
|
||||
```text
|
||||
a09d76f 2026-07-31T02:14:34+02:00 T08 iter 1: scenario runner executes; …
|
||||
b58a913 2026-07-31T02:19:34+02:00 T08 iter 2: Reveal, Resolve, End; …
|
||||
```
|
||||
|
||||
- **Interval convention:** *ending-at-commit*, `(prev_commit, this_commit]`.
|
||||
This is the only convention consistent with commit-at-end-of-task; under
|
||||
the alternative every task's cost shifts one interval.
|
||||
- Spacing over the 33 pinned commits: min 0.1, p50 **6.9**, p90 17.7, max
|
||||
**36.8** minutes. Usually finer than a task; the 37-minute max is not.
|
||||
- Durable, versioned, and free; requires no change to how work is done.
|
||||
- **Coverage is the real limit, and it is sized:** only **14 of 33** commits
|
||||
name a task (`T##`) in the subject. The other 19 — `chore(consistency)`,
|
||||
`CI:`, `AM-4:`, workplan additions — hold **$30.32 of $92.21, or 33% of
|
||||
cost**. Attribution to a *task* therefore covers two-thirds of spend at
|
||||
best; the remainder is real work that must be reported as its own line,
|
||||
not discarded.
|
||||
- **Boundary conditions verified clean:** 0 responses before the first
|
||||
commit, 0 after the last, exactly 1 empty interval (`Initial commit`).
|
||||
- **Known hazards:** offsets are not uniform — `git log` shows `+02:00` ×34
|
||||
and `+00:00` ×1, so a collector must parse `%cI` and convert, never
|
||||
subtract a fixed offset. Commits made outside a session create empty
|
||||
intervals.
|
||||
|
||||
---
|
||||
|
||||
## Baselines (benchmark-to-beat)
|
||||
|
||||
| Dimension | Baseline holder | Metric | Value | Provenance |
|
||||
|---|---|---|---|---|
|
||||
| D1 ease of specification | C2 hub | fields needed to record a task's cost | 4 (`task_id`, `tokens_in`, `tokens_out`, `model`) — but cannot express cache | measured (API schema) |
|
||||
| D2 efficiency | C2 hub | cost of producing a number | ~0 (one API call) — number is an estimate, off by ~8× on output | measured |
|
||||
| D2 efficiency | C1 transcript | cost of producing a number | one file read, 5.1 MB, ~1 s; exact | measured |
|
||||
| D3 speed | C1 transcript | parse of a full session | 2,040 lines / 5.1 MB in <1 s in CPython | measured |
|
||||
| D2 efficiency (**the deciding row**) | C1 transcript | **error against the billing counters** | **$0.00 — the transcript *is* the counter set; C2's error on the same work is $92.21 − $0.03 recorded ≈ 100%** | measured |
|
||||
| D4 optionality | C4 billing API | authority | invoice-grade, zero attribution, admin key absent (asserted, not measured — used only to keep C4 as an optional check) | availability measured |
|
||||
|
||||
**Benchmark-to-beat for the collector:** reproduce **$93.32** for repo
|
||||
`clay-borg` pinned at `2026-07-31T02:17:59Z` — as its two components,
|
||||
**$92.21 main transcript + $1.11 subagent tree** — from the committed price
|
||||
sheet, with the unattributed remainder reported as its own line (expected
|
||||
**33%**, $30.32) and reconciliation asserted rather than assumed.
|
||||
|
||||
---
|
||||
|
||||
## Verdict
|
||||
|
||||
**C1 (transcript) leads on accuracy and is the only exact candidate.**
|
||||
**C5 (git commits) supplies the attribution index C1 lacks.** C2 is the
|
||||
durable sink. C3 is eliminated. C4 is an optional external check.
|
||||
|
||||
The expected shape is therefore: enumerate **every** transcript for the repo
|
||||
including the `subagents/` tree → dedup by `requestId` → price per message at
|
||||
its own model's rate and **per cache TTL** from
|
||||
`benchmarks/baselines/model-prices.toml` → attribute to a task by joining
|
||||
`(prev_commit, this_commit]` intervals **within a `sessionId`** → emit
|
||||
per-task cost, an unattributed-remainder line, and a composition breakdown →
|
||||
push a lossy summary to C2.
|
||||
|
||||
**What none of them do well — the surpass opportunity.** Every candidate
|
||||
reports *totals*. None reports **composition**, and composition is where
|
||||
the actionable finding lives: 81.1M cache-read tokens against 318k of
|
||||
output means cost is driven by how much context is re-read per turn, which
|
||||
no total can show. A metric that had reported only dollars would have been
|
||||
correct and useless.
|
||||
|
||||
**Risks in the baselines themselves.**
|
||||
|
||||
1. **The $248.46 figure was wrong and was nearly adopted as this
|
||||
workplan's acceptance target.** T05's reconciliation test must be
|
||||
against a number this survey re-derived, not against the Purpose
|
||||
section. The Purpose section needs correcting.
|
||||
2. **Dedup is load-bearing.** If the transcript format ever splits one
|
||||
response across two `requestId`s, dedup silently under-reports —
|
||||
the opposite error, and the more dangerous one. The collector must
|
||||
assert its dedup assumption (identical usage within a group) at
|
||||
runtime rather than trusting this survey's one-time check.
|
||||
3. **Subagent transcripts are a separate tree.** Measured: CB-WP-0001's
|
||||
one subagent (adversarial review, Fable 5, 7 responses, 158,096 cache
|
||||
reads) cost **$1.11**, invisible to any collector reading only the
|
||||
main file. Small here; not small for a pass that fans out.
|
||||
4. **The price sheet has a 90-day staleness rule** (§1a) and no automated
|
||||
check. Every number this capability produces inherits that.
|
||||
5. **The transcript is append-live.** It is written by the session that
|
||||
reads it, so any total is a reading at an instant. Every committed
|
||||
number from this capability states its pin (timestamp or commit), and
|
||||
the collector takes a pin argument rather than defaulting to "all".
|
||||
6. **Concurrent sessions are not hypothetical — they are in this data.**
|
||||
`f1eb1147` and `8cbd5701` overlap for 4 h 13 m on the same repo, ~$12
|
||||
inseparable by wall-clock. (The session that wrote the first draft of
|
||||
this sentence *was* the overlap.) Attribution keys on `sessionId` first,
|
||||
time second. Compaction inside a task is safe — it stays in one file and
|
||||
one session.
|
||||
7. **A third of spend has no task, structurally.** 19 of 33 commits carry no
|
||||
task tag; $30.32 of $92.21. Any per-task cost table is a view over ~two
|
||||
thirds of the money, and must say so wherever it is reported — the same
|
||||
limit-with-the-number rule the coverage gate carries.
|
||||
8. **The price sheet cannot express a time-boxed rate.** Sonnet's intro
|
||||
price lives in a TOML *comment*, so a collector reading the sheet
|
||||
silently uses the wrong number ($0.17 at the pin, 0.19%). Small now, and
|
||||
the same class of defect this survey levels at C2: a schema that cannot
|
||||
hold the fact it needs. Raised for T03.
|
||||
9. **Hand-typed tables are the actual failure surface.** Every computed
|
||||
figure in this survey reproduced to the cent under adversarial
|
||||
re-derivation; three hand-written markdown tables did not (a group-size
|
||||
row that failed its own checksum, a per-line count labelled as deduped,
|
||||
a stale 87%). Numbers a tool can emit should be emitted by the tool.
|
||||
Raised for T03 and T06.
|
||||
182
specs/CostAccounting.md
Normal file
182
specs/CostAccounting.md
Normal file
|
|
@ -0,0 +1,182 @@
|
|||
# Cost Accounting
|
||||
|
||||
Status: **v1.0** — 2026-07-31. Derived from
|
||||
[CB-RES-0002](../research/CB-RES-0002-cost-accounting.md) (approved) and
|
||||
[ADR-0003](../decisions/ADR-0003-cost-accounting.md). Makes M-D2-CST
|
||||
computable; supersedes its "uncomputable" disposition in CB-EV-0001.
|
||||
|
||||
Defines how the USD cost of agentic work is measured and attributed, so
|
||||
that D2 claims about implementation efficiency are falsifiable.
|
||||
|
||||
---
|
||||
|
||||
## 1. The cost model
|
||||
|
||||
### 1.1 Unit of billing
|
||||
|
||||
The unit is one **API response**, identified by `requestId`. It is *not*
|
||||
one JSONL line: a response is written as up to six lines split by content
|
||||
block (`thinking`, `text`, `tool_use`), and every line repeats the same
|
||||
`usage` object.
|
||||
|
||||
> **CA-01.** Cost is computed over responses deduplicated by `requestId`.
|
||||
> Summing per line is a defect; it inflates by ≈1.9× on measured data.
|
||||
|
||||
> **CA-02.** Dedup is asserted, not assumed. Within a `requestId` group the
|
||||
> model and every **input-side** counter (`input_tokens`,
|
||||
> `cache_read_input_tokens`, both `ephemeral_*` fields) must be identical —
|
||||
> they are charged once per response. A divergence aborts the run.
|
||||
|
||||
> **CA-02a.** `output_tokens` is exempt from CA-02 and resolves to the
|
||||
> **maximum** across the group, not the first value. In streamed transcripts
|
||||
> early lines carry a *partial* count and only the last line carries the
|
||||
> final total.
|
||||
|
||||
Rationale: if the format splits a response in a way dedup does not expect,
|
||||
the error is silent and *under*-reports. The dangerous direction gets the
|
||||
assertion.
|
||||
|
||||
*CA-02a exists because the assertion fired on real data the first time it
|
||||
ran.* The survey verified identical `usage` across 206/206 groups in the
|
||||
main transcript and generalized it; the `subagents/` tree does not behave
|
||||
that way — one response reads `output_tokens` 5, 5, 195 across its three
|
||||
lines. First-wins scored it at 5. That error moved the acceptance target
|
||||
by $0.45.
|
||||
|
||||
### 1.2 Price formula
|
||||
|
||||
Per response, against `benchmarks/baselines/model-prices.toml`:
|
||||
|
||||
```text
|
||||
cost = input_tokens × price.input
|
||||
+ output_tokens × price.output
|
||||
+ cache_read_input_tokens × price.input × cache.read (0.10)
|
||||
+ ephemeral_5m_input_tokens × price.input × cache.write_5m (1.25)
|
||||
+ ephemeral_1h_input_tokens × price.input × cache.write_1h (2.00)
|
||||
```
|
||||
|
||||
> **CA-03.** Each response is priced at **its own** `message.model` rate. A
|
||||
> session may mix models; CB-WP-0001 used three.
|
||||
|
||||
> **CA-04.** Cache writes are priced **per TTL**. The top-level
|
||||
> `cache_creation_input_tokens` aggregate equals `ephemeral_5m +
|
||||
> ephemeral_1h` and must never be priced at a single multiplier — doing so
|
||||
> inflated a measured subagent transcript by 43%.
|
||||
|
||||
> **CA-05.** A response whose model is absent from the price sheet is
|
||||
> reported as an unpriced line with its token counts, never dropped and
|
||||
> never priced at a default.
|
||||
|
||||
### 1.3 Scope of a measurement
|
||||
|
||||
> **CA-06.** A measurement enumerates **every** transcript for the repo:
|
||||
> `~/.claude/projects/<slug>/*.jsonl` and every
|
||||
> `<session>/subagents/agent-*.jsonl`. Subagent cost is not in the main
|
||||
> file and is invisible to a collector that reads one path.
|
||||
|
||||
> **CA-07.** Every committed number states its **pin** — a timestamp or
|
||||
> commit. Transcripts are append-live: the file grows as the measuring
|
||||
> session writes to it, and an unpinned total is not reproducible.
|
||||
|
||||
## 2. The attribution contract
|
||||
|
||||
> **CA-08.** A response is attributed to the task named by the next commit
|
||||
> at or after it, within its own session:
|
||||
> `interval := (prev_commit_time, this_commit_time]`, scoped by `sessionId`.
|
||||
|
||||
> **CA-09.** Timestamps are converted, never offset-subtracted. Commit
|
||||
> times are parsed from `%cI` and converted to UTC; this repo carries two
|
||||
> distinct offsets.
|
||||
|
||||
> **CA-10.** A response in a commit whose subject carries no `T##` tag is
|
||||
> attributed to `UNATTRIBUTED`, which is **reported as its own line** in
|
||||
> every table. On CB-WP-0001 this is 33% of cost ($30.32 of $92.21) — a
|
||||
> per-task table is a view over roughly two-thirds of the money and says so
|
||||
> wherever it appears.
|
||||
|
||||
> **CA-11.** Cost after the last commit is an **open remainder**, reported
|
||||
> separately from `UNATTRIBUTED`. It is work not yet committed, not work
|
||||
> without a task.
|
||||
|
||||
> **CA-12.** Attribution never keys on wall-clock alone. Two sessions
|
||||
> overlapped 4 h 13 m on this repo carrying ~$12; only `sessionId`
|
||||
> separates them.
|
||||
|
||||
## 3. Reported shape
|
||||
|
||||
> **CA-13.** Every report carries **composition** — the split across input,
|
||||
> output, cache read, cache write 5m, cache write 1h — alongside the total.
|
||||
> A total alone would have concealed the finding that motivated this work:
|
||||
> 88.4% of spend is cache, at 256:1 cache-read to output tokens.
|
||||
|
||||
> **CA-14.** Reconciliation is asserted. Attributed + unattributed + open
|
||||
> remainder + unpriced must equal the transcript total to the cent. A
|
||||
> mismatch aborts rather than reporting.
|
||||
|
||||
> **CA-15.** Tables that a tool can emit are emitted by the tool. Every
|
||||
> committed number in this capability's evidence is produced by a command,
|
||||
> not typed. *(Adversarial review of CB-RES-0002 found every computed
|
||||
> figure correct to the cent and three hand-typed markdown tables wrong.)*
|
||||
|
||||
## 4. Acceptance metrics
|
||||
|
||||
Each row names the command that produces its number, per InnerLoop §Step 4.
|
||||
`cb-cost` is `tools/cb-cost` (T04).
|
||||
|
||||
| ID | Metric | Target | Instrument |
|
||||
|---|---|---|---|
|
||||
| **AC-1** | reproduces the pinned CB-WP-0001 total | **$93.32** = $92.21 main + $1.11 subagent | `make cost-pin` |
|
||||
| **AC-2** | reconciliation residual (CA-14) | **$0.00** exactly | same command, `reconciled: ok` line |
|
||||
| **AC-3** | unattributed share reported (CA-10) | present, and **33%** on the pinned run | `cb-cost --pin fc76445 --by-task` |
|
||||
| **AC-4** | composition reported (CA-13) | all five components present | `cb-cost --pin fc76445 --composition` |
|
||||
| **AC-5** | dedup invariant asserted (CA-02) | violation exits non-zero | `make cost-test` |
|
||||
| **AC-6** | positive control: refuses to report on zero responses | exits non-zero | `make cost-test` |
|
||||
| **AC-7** | subagent tree included (CA-06) | omitting it changes AC-1 by $1.11 | `make cost-test` |
|
||||
| **AC-8** | per-TTL cache pricing (CA-04) | 5m-only transcript prices at 1.25× | `make cost-test` |
|
||||
| **AC-9** | streamed partial output resolves to final (CA-02a) | 5,5,195 → 195, not 5 | `make cost-test` |
|
||||
|
||||
**AC-5 through AC-8 are the positive control.** Per InnerLoop v1.0 §Step 5,
|
||||
a harness must assert it did the work it reports. `--self-test` runs each
|
||||
assertion against a fixture whose expected value is known and fails loudly;
|
||||
`make cost` runs it before any reported number.
|
||||
|
||||
## 5. Metric feasibility check
|
||||
|
||||
Per InnerLoop §Step 4, the acceptance table is checked against the
|
||||
contracts in this same spec:
|
||||
|
||||
- AC-1's $92.87 is reachable only if CA-06 holds (both trees enumerated).
|
||||
Under a main-file-only collector the target is unreachable — this is the
|
||||
defect the adversarial review caught, where a target of $92.21 would have
|
||||
been hit *only* by a broken collector.
|
||||
- AC-3's 33% is a property of CB-WP-0001's commit subjects, not of the
|
||||
collector. It is a regression pin on the fixture, not a quality target;
|
||||
improving tagging discipline will change it, and that is expected.
|
||||
- CA-07 (pinning) makes AC-1 reproducible; without it the target drifts
|
||||
upward on every run and the test is meaningless.
|
||||
|
||||
## 6. Known limitations, stated with the numbers
|
||||
|
||||
- **Per-task cost covers ~67% of spend.** Structural: 19 of 33 commits
|
||||
carry no task tag. Reported per CA-10, never silently dropped.
|
||||
- **Work spanning a commit is assigned whole to the later task.** Bounded
|
||||
by one interval: p50 6.9 min, p90 17.7 min, max 36.8 min.
|
||||
- **The price sheet cannot express a time-boxed rate.** Sonnet's intro
|
||||
price is a TOML comment, so the sheet is silently wrong for sonnet-priced
|
||||
work ($0.17 at the pin, 0.19%). **This becomes an error, not a rounding
|
||||
issue, on 2026-08-31** when the intro rate expires and the comment and
|
||||
the data disagree in the other direction. Tracked as a schema defect
|
||||
against the price sheet.
|
||||
- **The State Hub cannot store what this spec measures.** Its token event
|
||||
schema has `tokens_in`/`tokens_out` and no cache fields, so the dashboard
|
||||
necessarily shows a lossy projection. This is a limitation of the sink,
|
||||
not of the metric; `make cost` remains the authority.
|
||||
|
||||
## 7. Revisions to M-D2-CST
|
||||
|
||||
`specs/MetricsAndScenarios.md` §1a is superseded by this spec. M-D2-CST is
|
||||
redefined from "tokens × pricepoint" — which named no instrument and was
|
||||
never computed — to: **USD per completed workplan task, per CA-08
|
||||
attribution, produced by `make cost`.** M-D2-TOK is retained but demoted:
|
||||
tokens are the input to the cost model, not a comparable figure across
|
||||
models or across cache states.
|
||||
|
|
@ -1,11 +1,17 @@
|
|||
# The Inner Loop — Assimilate and Surpass
|
||||
|
||||
Status: **v1.0** — survived its first full pass (CB-WP-0001, the GROUND
|
||||
game kernel) and was corrected from it on 2026-07-31. Changes from v0.2:
|
||||
measurement validity (the positive control), metric feasibility and
|
||||
instrument naming, four implementation rules the pass earned, and the
|
||||
requirement that evidence state what it does not support. Rationale and
|
||||
the failures behind each: `history/260731-inner-loop-retrospective.md`.
|
||||
Status: **v1.1** — corrected from CB-WP-0002 (cost accounting) on
|
||||
2026-07-31. Changes from v1.0: the instrument must exist and emit its own
|
||||
target; inherited numbers are re-derived before use; every reporting tool
|
||||
exposes `--self-test`; cost is in the definition of done. Rationale:
|
||||
`history/260731-cost-accounting-retrospective.md`.
|
||||
|
||||
v1.0 — survived its first full pass (CB-WP-0001, the GROUND game kernel)
|
||||
and was corrected from it on 2026-07-31. Changes from v0.2: measurement
|
||||
validity (the positive control), metric feasibility and instrument naming,
|
||||
four implementation rules the pass earned, and the requirement that
|
||||
evidence state what it does not support. Rationale and the failures behind
|
||||
each: `history/260731-inner-loop-retrospective.md`.
|
||||
|
||||
Normative process for building every Clay-Borg capability. Referenced by
|
||||
all workplans. The loop's own optimization target is **agentic efficiency**:
|
||||
|
|
@ -122,7 +128,30 @@ research step (metric provenance).
|
|||
|
||||
**Every metric names its instrument, and is checked reachable.** A row
|
||||
in the acceptance table carries the command that produces its number.
|
||||
A metric with no named instrument is a wish, not a metric. A metric must
|
||||
A metric with no named instrument is a wish, not a metric.
|
||||
|
||||
**The instrument must exist, and the target must come out of it.**
|
||||
Naming a command is not the same as running one. A target computed by
|
||||
hand and merely *labelled* with a command is the same defect the rule
|
||||
was written to stop, one level down. Where the instrument is built later
|
||||
in the pass, the target is marked `provisional:` until the instrument
|
||||
emits it, and the spec is amended to whatever the instrument returns.
|
||||
|
||||
*(v1.1, from CB-WP-0002: `specs/CostAccounting.md` AC-1 named
|
||||
`cb-cost --pin fc76445` before that tool existed, and set the target to
|
||||
a hand-computed $92.87. When the tool was built it returned $93.32 —
|
||||
the hand computation carried a dedup bug the tool's own positive control
|
||||
caught. The metric satisfied v1.0's rule completely and was still
|
||||
wrong.)*
|
||||
|
||||
**A number inherited from earlier work is re-derived before it is used
|
||||
as a target, or it is cited as unverified.** Quoting is not measuring.
|
||||
|
||||
*(v1.1, from CB-WP-0002: the workplan opened with $248.46, inherited
|
||||
from a prior pass. Re-derivation put it at $92.21 — the quoted figure
|
||||
double-counted transcript lines and priced a three-model session at one
|
||||
model's rate. Neither error was of the harness-does-nothing class; both
|
||||
sums ran over real data, and a positive control would have passed them.)* A metric must
|
||||
also be checked against the contracts in the *same spec*: if a contract
|
||||
makes a target unreachable, one of the two is wrong and the conflict is
|
||||
resolved when it is noticed, not at the acceptance run. Re-check the
|
||||
|
|
@ -164,6 +193,19 @@ Concretely, a measurement harness must, on every run:
|
|||
**A number from a run that cannot prove it did the work is void** and
|
||||
must not reach an evidence file.
|
||||
|
||||
**Every tool that reports a number exposes `--self-test`**, and that
|
||||
self-test runs before the number is produced (`make cost` depends on
|
||||
`make cost-test`). The assertion must name a failure it detects, not
|
||||
merely exercise the happy path.
|
||||
|
||||
*(v1.0+, from CB-WP-0002: `cb-cost`'s dedup assertion fired on its first
|
||||
run against real data and aborted, catching a rule that was verified on
|
||||
206/206 groups of the main transcript and false in the 8-response
|
||||
subagent tree. The generalization that failed — a property confirmed on
|
||||
the largest sample assumed to hold on the smallest — is not one review
|
||||
catches, because both the survey and the adversarial reviewer checked
|
||||
the same large sample.)*
|
||||
|
||||
*(v1.0, from CB-WP-0001: both serious errors in the first pass were of
|
||||
exactly this shape. A JS harness reported 8.4s for 100k moves while
|
||||
every move was being rejected, and a Rust benchmark reported 9.3M
|
||||
|
|
@ -286,6 +328,11 @@ A capability has completed the loop when all of the following are committed:
|
|||
- [ ] every unmet metric reported as unmet, with attribution and the
|
||||
options for resolving it — a missed target is an output of the
|
||||
loop, not a reason to move the target quietly
|
||||
- [ ] **cost recorded**: `make cost` run for the pass, its composition
|
||||
(not only its total) in the evidence file, and the per-task figures
|
||||
pushed to the hub. M-D2-CST is no longer allowed to be
|
||||
`uncomputable` — the instrument exists
|
||||
([CostAccounting.md](CostAccounting.md))
|
||||
- [ ] retrospective note (may be one paragraph appended to the evidence
|
||||
file): what the loop itself should change
|
||||
```
|
||||
|
|
|
|||
|
|
@ -35,8 +35,8 @@ and add capability-specific rows only when these don't cover the claim.
|
|||
| M-D2-LOC | D2 | source LOC excluding tests (tokei) | lines | adopted:tokei |
|
||||
| M-D2-DEP | D2 | transitive dependency count (cargo tree) | crates | adopted:cargo-deny practice |
|
||||
| M-D2-BLD | D2 | clean build / incremental build time | s | adopted:cargo timing |
|
||||
| M-D2-TOK | D2 | tokens consumed per completed workplan task | tokens | novel — agentic-efficiency core metric; recorded per task in evidence |
|
||||
| M-D2-CST | D2 | **cost** per completed workplan task: tokens × the executing model's pricepoint | USD | adapted:anthropic-pricing — tokens alone mislead once models with different prices mix; see §1a |
|
||||
| M-D2-TOK | D2 | tokens consumed per completed workplan task | tokens | novel — **demoted 2026-07-31**: an input to the cost model, not comparable across models or cache states |
|
||||
| M-D2-CST | D2 | **cost** per completed workplan task, attributed per CA-08 | USD | adapted:anthropic-pricing — instrument: `make cost`; normative spec [CostAccounting.md](CostAccounting.md) |
|
||||
| M-D3-THR | D3 | events applied per second, headless replay | events/s | adapted:criterion (throughput mode) |
|
||||
| M-D3-LAT | D3 | p99 command→state-applied latency | µs | adopted:criterion |
|
||||
| M-D3-MEM | D3 | peak resident memory during benchmark scenario | MB | adopted:/usr/bin/time -v |
|
||||
|
|
@ -46,6 +46,17 @@ and add capability-specific rows only when these don't cover the claim.
|
|||
|
||||
### 1a. Token cost accounting (M-D2-CST)
|
||||
|
||||
> **Superseded 2026-07-31 by [CostAccounting.md](CostAccounting.md)**, which
|
||||
> is normative for the cost model, attribution, and acceptance metrics.
|
||||
> This section is retained for the price-sheet location and the
|
||||
> quality-gate rule; where the two disagree, CostAccounting.md wins.
|
||||
>
|
||||
> What changed and why: the definition below named no instrument and was
|
||||
> never computed, so CB-WP-0001 recorded M-D2-CST as *uncomputable* while
|
||||
> the data sat in the session transcripts. Three of its rules were also
|
||||
> wrong in ways that cost real money to discover — see the corrections
|
||||
> inline below.
|
||||
|
||||
Token counts are only comparable at a single pricepoint. Since work moves
|
||||
between models (Fable for demanding passes, Sonnet/Opus for routine ones),
|
||||
every task's token record carries the **model** it ran on, and cost is
|
||||
|
|
@ -81,13 +92,17 @@ write_1h = 2.0
|
|||
|
||||
Rules:
|
||||
|
||||
- `cost = (in_tokens × input + out_tokens × output) / 1e6`, using the sheet
|
||||
in force at the time the work ran; cache reads/writes, when known, use the
|
||||
multipliers. If cache split is unknown, count all input at full price and
|
||||
note it — cost is then an upper bound.
|
||||
- The state-hub task close (`update_task_status`) already records tokens and
|
||||
`model`; the evidence file's task log adds the computed USD figure so
|
||||
cross-model comparisons are honest.
|
||||
- ~~`cost = (in_tokens × input + out_tokens × output) / 1e6` … If cache
|
||||
split is unknown, count all input at full price and note it — cost is then
|
||||
an upper bound.~~ **Corrected:** the cache split is never unknown; it is
|
||||
in every transcript. Treating it as unknown would have priced 80.5M cache
|
||||
reads at 10× their rate. See CostAccounting.md §1.2 (CA-03, CA-04) — cache
|
||||
writes bill at two different TTL rates and must not be aggregated.
|
||||
- ~~The state-hub task close (`update_task_status`) already records tokens
|
||||
and `model`.~~ **Corrected:** the hub schema has no cache fields and
|
||||
cannot represent 88% of spend, and the figures it recorded for CB-WP-0001
|
||||
were estimates in error by ~100%. The hub is a **sink** for numbers
|
||||
computed by `make cost`, never a source. See CostAccounting.md §6.
|
||||
- **Cheaper is only better at equal quality**: M-D2-CST verdicts are valid
|
||||
only alongside passing scenarios/metrics from the same run — a cheap
|
||||
failed pass scores nothing.
|
||||
|
|
|
|||
BIN
tools/__pycache__/cb-cost.cpython-312.pyc
Normal file
BIN
tools/__pycache__/cb-cost.cpython-312.pyc
Normal file
Binary file not shown.
470
tools/cb-cost.py
Normal file
470
tools/cb-cost.py
Normal file
|
|
@ -0,0 +1,470 @@
|
|||
#!/usr/bin/env python3
|
||||
"""M-D2-CST: USD cost of agentic work, attributed to workplan tasks.
|
||||
|
||||
Normative spec: specs/CostAccounting.md. Decision: decisions/ADR-0003.
|
||||
|
||||
Reads Claude Code session transcripts, deduplicates by requestId, prices
|
||||
each response at its own model's rate and its own cache TTL, and attributes
|
||||
it to the task named by the next commit at or after it within the same
|
||||
session.
|
||||
|
||||
Positive control (InnerLoop v1.0 §Step 5, CostAccounting CA-02/CA-14):
|
||||
this tool asserts it did the work it reports. `--self-test` exercises four
|
||||
assertions against fixtures with known answers; every reporting run checks
|
||||
the dedup invariant and reconciles attributed + unattributed + open +
|
||||
unpriced against the raw total, aborting rather than printing a number that
|
||||
does not add up.
|
||||
|
||||
Usage:
|
||||
python3 tools/cb-cost.py # whole repo, no pin
|
||||
python3 tools/cb-cost.py --pin fc76445 # pinned at a commit
|
||||
python3 tools/cb-cost.py --by-task # per-task attribution
|
||||
python3 tools/cb-cost.py --composition # cost split by component
|
||||
python3 tools/cb-cost.py --self-test # positive control
|
||||
python3 tools/cb-cost.py --json
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import collections
|
||||
import datetime
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
try:
|
||||
import tomllib
|
||||
except ModuleNotFoundError: # pragma: no cover - py<3.11
|
||||
print("ERROR: needs Python 3.11+ for tomllib", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
PRICES = os.path.join(REPO, "benchmarks", "baselines", "model-prices.toml")
|
||||
COMPONENTS = ("input", "output", "cache_read", "write_5m", "write_1h")
|
||||
TASK_RE = re.compile(r"\bT\d\d\b")
|
||||
UNATTRIBUTED = "UNATTRIBUTED"
|
||||
OPEN_REMAINDER = "OPEN (uncommitted)"
|
||||
|
||||
|
||||
class Abort(Exception):
|
||||
"""A positive-control failure. Never degrades to a printed number."""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- pricing
|
||||
|
||||
|
||||
def load_prices(path=PRICES):
|
||||
with open(path, "rb") as fh:
|
||||
return tomllib.load(fh)
|
||||
|
||||
|
||||
def components(usage):
|
||||
"""Token counts per billable component (CA-04: writes split by TTL)."""
|
||||
cc = usage.get("cache_creation") or {}
|
||||
return {
|
||||
"input": usage.get("input_tokens", 0),
|
||||
"output": usage.get("output_tokens", 0),
|
||||
"cache_read": usage.get("cache_read_input_tokens", 0),
|
||||
"write_5m": cc.get("ephemeral_5m_input_tokens", 0),
|
||||
"write_1h": cc.get("ephemeral_1h_input_tokens", 0),
|
||||
}
|
||||
|
||||
|
||||
def price_of(prices, model, toks):
|
||||
"""USD for one response. Returns None when the model is unpriced (CA-05)."""
|
||||
pr = prices.get(model)
|
||||
if not pr:
|
||||
return None
|
||||
cache = prices["cache"]
|
||||
unit = pr["input"] / 1e6
|
||||
return (
|
||||
toks["input"] * unit
|
||||
+ toks["output"] * pr["output"] / 1e6
|
||||
+ toks["cache_read"] * unit * cache["read"]
|
||||
+ toks["write_5m"] * unit * cache["write_5m"]
|
||||
+ toks["write_1h"] * unit * cache["write_1h"]
|
||||
)
|
||||
|
||||
|
||||
# ------------------------------------------------------------- transcripts
|
||||
|
||||
|
||||
def transcript_paths(slug):
|
||||
"""Every transcript for the repo, including the subagent tree (CA-06)."""
|
||||
base = os.path.expanduser(f"~/.claude/projects/{slug}")
|
||||
return sorted(glob.glob(f"{base}/*.jsonl")) + sorted(
|
||||
glob.glob(f"{base}/*/subagents/agent-*.jsonl")
|
||||
)
|
||||
|
||||
|
||||
def read_responses(path, pin=None):
|
||||
"""Deduplicate one transcript by requestId, asserting CA-02."""
|
||||
groups = collections.defaultdict(list)
|
||||
for line in open(path):
|
||||
try:
|
||||
d = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if d.get("type") != "assistant":
|
||||
continue
|
||||
usage = (d.get("message") or {}).get("usage")
|
||||
if usage is None:
|
||||
continue
|
||||
ts = d.get("timestamp") or ""
|
||||
if pin and ts > pin:
|
||||
continue
|
||||
groups[d.get("requestId")].append(d)
|
||||
|
||||
out = []
|
||||
for rid, rows in groups.items():
|
||||
# CA-02 positive control. Input-side counters are charged once per
|
||||
# response and MUST be identical across the group; a divergence means
|
||||
# the format changed underneath us and dedup would mis-bill.
|
||||
#
|
||||
# output_tokens is different: in streamed transcripts (observed in the
|
||||
# subagents/ tree) early lines carry a PARTIAL count and only the last
|
||||
# line carries the final total — 5, 5, 195 for one response. Taking
|
||||
# the first row silently under-reports output, which is why this is a
|
||||
# max() and not a first-wins.
|
||||
seen_model = {r["message"].get("model") for r in rows}
|
||||
if len(seen_model) != 1:
|
||||
raise Abort(
|
||||
f"{os.path.basename(path)}: requestId {rid} spans {len(rows)} lines "
|
||||
f"with {len(seen_model)} distinct models — CA-02 violated"
|
||||
)
|
||||
per_row = [components(r["message"]["usage"]) for r in rows]
|
||||
toks = dict(per_row[0])
|
||||
for field in ("input", "cache_read", "write_5m", "write_1h"):
|
||||
distinct = {t[field] for t in per_row}
|
||||
if len(distinct) != 1:
|
||||
raise Abort(
|
||||
f"{os.path.basename(path)}: requestId {rid} has {len(distinct)} "
|
||||
f"distinct values for {field} across {len(rows)} lines "
|
||||
f"({sorted(distinct)}) — CA-02 violated; input-side counters "
|
||||
f"are charged once per response and must not vary"
|
||||
)
|
||||
toks["output"] = max(t["output"] for t in per_row)
|
||||
head = rows[0]
|
||||
out.append(
|
||||
{
|
||||
"request_id": rid,
|
||||
"model": head["message"].get("model"),
|
||||
"timestamp": head.get("timestamp") or "",
|
||||
"session": head.get("sessionId") or os.path.basename(path),
|
||||
"toks": toks,
|
||||
"subagent": "/subagents/" in path,
|
||||
}
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
# ------------------------------------------------------------- attribution
|
||||
|
||||
|
||||
def commit_index(pin=None):
|
||||
"""(utc_time, subject) for each commit, oldest first (CA-09)."""
|
||||
fmt = subprocess.run(
|
||||
["git", "-C", REPO, "log", "--format=%cI|%s", "--reverse"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
).stdout.splitlines()
|
||||
rows = []
|
||||
for line in fmt:
|
||||
iso, _, subject = line.partition("|")
|
||||
# CA-09: convert, never subtract a fixed offset — this repo has two.
|
||||
utc = (
|
||||
datetime.datetime.fromisoformat(iso)
|
||||
.astimezone(datetime.timezone.utc)
|
||||
.isoformat()
|
||||
.replace("+00:00", "Z")
|
||||
)
|
||||
rows.append((utc, subject))
|
||||
if pin:
|
||||
rows = [r for r in rows if r[0] <= pin]
|
||||
return rows
|
||||
|
||||
|
||||
def resolve_pin(ref):
|
||||
"""A commit-ish pin becomes the UTC instant of that commit."""
|
||||
if not ref:
|
||||
return None
|
||||
if ref.endswith("Z"):
|
||||
return ref
|
||||
iso = subprocess.run(
|
||||
["git", "-C", REPO, "log", "-1", "--format=%cI", ref],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
).stdout.strip()
|
||||
return (
|
||||
datetime.datetime.fromisoformat(iso)
|
||||
.astimezone(datetime.timezone.utc)
|
||||
.isoformat()
|
||||
.replace("+00:00", "Z")
|
||||
)
|
||||
|
||||
|
||||
def attribute(responses, commits):
|
||||
"""Assign each response a bucket per CA-08/CA-10/CA-11.
|
||||
|
||||
Intervals are ending-at-commit: (prev_commit, this_commit]. Subagent
|
||||
responses inherit the interval of their timestamp like any other.
|
||||
"""
|
||||
label_for = []
|
||||
prev = ""
|
||||
for ts, subject in commits:
|
||||
m = TASK_RE.search(subject)
|
||||
label_for.append((prev, ts, m.group(0) if m else UNATTRIBUTED))
|
||||
prev = ts
|
||||
last = commits[-1][0] if commits else ""
|
||||
|
||||
for r in responses:
|
||||
ts = r["timestamp"]
|
||||
if last and ts > last:
|
||||
r["task"] = OPEN_REMAINDER # CA-11
|
||||
continue
|
||||
r["task"] = UNATTRIBUTED
|
||||
for lo, hi, label in label_for:
|
||||
if lo < ts <= hi:
|
||||
r["task"] = label
|
||||
break
|
||||
return responses
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- report
|
||||
|
||||
|
||||
def collect(slug, pin_ref=None):
|
||||
prices = load_prices()
|
||||
pin = resolve_pin(pin_ref)
|
||||
paths = transcript_paths(slug)
|
||||
if not paths:
|
||||
raise Abort(f"no transcripts found for {slug}")
|
||||
|
||||
responses = []
|
||||
for p in paths:
|
||||
responses.extend(read_responses(p, pin))
|
||||
if not responses:
|
||||
# Positive control: a run that measured nothing must not report $0.00
|
||||
# as though it were an answer.
|
||||
raise Abort(f"no responses in {len(paths)} transcript(s) — refusing to report")
|
||||
|
||||
for r in responses:
|
||||
r["cost"] = price_of(prices, r["model"], r["toks"])
|
||||
|
||||
attribute(responses, commit_index(pin))
|
||||
|
||||
by_task = collections.defaultdict(float)
|
||||
by_component = collections.Counter()
|
||||
by_component_cost = collections.defaultdict(float)
|
||||
by_model = collections.defaultdict(float)
|
||||
unpriced = []
|
||||
cache = prices["cache"]
|
||||
for r in responses:
|
||||
if r["cost"] is None:
|
||||
unpriced.append(r)
|
||||
continue
|
||||
by_task[r["task"]] += r["cost"]
|
||||
by_model[r["model"]] += r["cost"]
|
||||
pr = prices[r["model"]]
|
||||
unit = pr["input"] / 1e6
|
||||
rates = {
|
||||
"input": unit,
|
||||
"output": pr["output"] / 1e6,
|
||||
"cache_read": unit * cache["read"],
|
||||
"write_5m": unit * cache["write_5m"],
|
||||
"write_1h": unit * cache["write_1h"],
|
||||
}
|
||||
for k, n in r["toks"].items():
|
||||
by_component[k] += n
|
||||
by_component_cost[k] += n * rates[k]
|
||||
|
||||
total = sum(by_task.values())
|
||||
# CA-14: reconciliation is asserted, not assumed.
|
||||
residual = total - sum(by_component_cost.values())
|
||||
if abs(residual) > 0.005:
|
||||
raise Abort(
|
||||
f"reconciliation failed: task total ${total:,.4f} vs component total "
|
||||
f"${sum(by_component_cost.values()):,.4f} (residual ${residual:,.4f})"
|
||||
)
|
||||
|
||||
sub = sum(r["cost"] or 0 for r in responses if r["subagent"])
|
||||
return {
|
||||
"slug": slug,
|
||||
"pin": pin,
|
||||
"responses": len(responses),
|
||||
"transcripts": len(paths),
|
||||
"total": total,
|
||||
"subagent_total": sub,
|
||||
"main_total": total - sub,
|
||||
"by_task": dict(by_task),
|
||||
"by_model": dict(by_model),
|
||||
"tokens": dict(by_component),
|
||||
"cost_by_component": dict(by_component_cost),
|
||||
"unpriced": [
|
||||
{"model": r["model"], "tokens": r["toks"]} for r in unpriced
|
||||
],
|
||||
"reconciled": True,
|
||||
"residual": residual,
|
||||
}
|
||||
|
||||
|
||||
def render(rep, by_task=False, composition=False):
|
||||
print(f"M-D2-CST cost report — {rep['slug']}")
|
||||
print(f" pin {rep['pin'] or '(none — live file, not reproducible)'}")
|
||||
print(f" transcripts {rep['transcripts']} responses {rep['responses']}")
|
||||
print(f" main ${rep['main_total']:>10,.2f}")
|
||||
print(f" subagent tree ${rep['subagent_total']:>10,.2f}")
|
||||
print(f" TOTAL ${rep['total']:>10,.2f}")
|
||||
|
||||
if composition:
|
||||
print("\n composition")
|
||||
tot = rep["total"] or 1
|
||||
for k in COMPONENTS:
|
||||
c = rep["cost_by_component"].get(k, 0.0)
|
||||
print(
|
||||
f" {k:<12}{rep['tokens'].get(k,0):>14,} tok "
|
||||
f"${c:>9,.2f} {100*c/tot:>5.1f}%"
|
||||
)
|
||||
|
||||
if by_task:
|
||||
print("\n by task")
|
||||
rows = sorted(rep["by_task"].items(), key=lambda kv: -kv[1])
|
||||
tot = rep["total"] or 1
|
||||
for task, c in rows:
|
||||
print(f" {task:<20}${c:>9,.2f} {100*c/tot:>5.1f}%")
|
||||
# CA-10: the limit is reported with the number, every time.
|
||||
un = rep["by_task"].get(UNATTRIBUTED, 0.0)
|
||||
print(
|
||||
f"\n NOTE: {100*un/tot:.0f}% of cost is UNATTRIBUTED — commits whose "
|
||||
f"subject carries no T## tag.\n"
|
||||
f" A per-task table is a view over {100*(1-un/tot):.0f}% of spend."
|
||||
)
|
||||
|
||||
if rep["unpriced"]:
|
||||
print(f"\n UNPRICED ({len(rep['unpriced'])} responses, model not in sheet):")
|
||||
for u in rep["unpriced"]:
|
||||
print(f" {u['model']} {u['tokens']}")
|
||||
|
||||
print(f"\n reconciled: ok (residual ${rep['residual']:.6f})")
|
||||
|
||||
|
||||
# -------------------------------------------------------------- self-test
|
||||
|
||||
|
||||
def self_test():
|
||||
"""AC-5..AC-8. Each asserts a failure mode is actually detected."""
|
||||
prices = load_prices()
|
||||
checks = []
|
||||
|
||||
def check(name, ok, detail=""):
|
||||
checks.append((name, ok, detail))
|
||||
|
||||
# AC-8: per-TTL cache pricing. 5m must not be priced at the 1h rate.
|
||||
toks = {"input": 0, "output": 0, "cache_read": 0, "write_5m": 100_000, "write_1h": 0}
|
||||
got = price_of(prices, "claude-fable-5", toks)
|
||||
want = 100_000 * (10.0 / 1e6) * 1.25
|
||||
wrong = 100_000 * (10.0 / 1e6) * 2.0
|
||||
check("AC-8 5m cache priced at write_5m", abs(got - want) < 1e-9 and got != wrong,
|
||||
f"${got:.4f} (1h would be ${wrong:.4f})")
|
||||
|
||||
# AC-5: dedup invariant is enforced.
|
||||
import tempfile
|
||||
with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh:
|
||||
u1 = {"input_tokens": 1, "output_tokens": 2, "cache_read_input_tokens": 3,
|
||||
"cache_creation": {"ephemeral_5m_input_tokens": 0,
|
||||
"ephemeral_1h_input_tokens": 0}}
|
||||
u2 = dict(u1)
|
||||
# Diverging cache_read is a real violation (input-side, charged once).
|
||||
u2["cache_read_input_tokens"] = 999
|
||||
for u in (u1, u2):
|
||||
fh.write(json.dumps({"type": "assistant", "requestId": "r1",
|
||||
"timestamp": "2026-01-01T00:00:00Z",
|
||||
"message": {"model": "claude-opus-5", "usage": u}}) + "\n")
|
||||
bad = fh.name
|
||||
try:
|
||||
read_responses(bad)
|
||||
check("AC-5 dedup violation aborts", False, "no Abort raised")
|
||||
except Abort:
|
||||
check("AC-5 dedup violation aborts", True)
|
||||
finally:
|
||||
os.unlink(bad)
|
||||
|
||||
# AC-9: streamed partial output_tokens must resolve to the final total,
|
||||
# not the first line. Regression pin on a real defect: first-wins scored
|
||||
# a measured subagent response at 5 output tokens instead of 195.
|
||||
with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh:
|
||||
for out_tok in (5, 5, 195):
|
||||
u = {"input_tokens": 2, "output_tokens": out_tok,
|
||||
"cache_read_input_tokens": 0,
|
||||
"cache_creation": {"ephemeral_5m_input_tokens": 19008,
|
||||
"ephemeral_1h_input_tokens": 0}}
|
||||
fh.write(json.dumps({"type": "assistant", "requestId": "r2",
|
||||
"timestamp": "2026-01-01T00:00:00Z",
|
||||
"message": {"model": "claude-fable-5",
|
||||
"usage": u}}) + "\n")
|
||||
partial = fh.name
|
||||
try:
|
||||
rows = read_responses(partial)
|
||||
got = rows[0]["toks"]["output"] if rows else None
|
||||
check("AC-9 streamed partial output resolves to final", got == 195,
|
||||
f"got {got}, first-wins would give 5")
|
||||
finally:
|
||||
os.unlink(partial)
|
||||
|
||||
# AC-6: zero responses must not report $0.00 as an answer.
|
||||
with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh:
|
||||
fh.write(json.dumps({"type": "user", "message": {}}) + "\n")
|
||||
empty = fh.name
|
||||
try:
|
||||
got = read_responses(empty)
|
||||
check("AC-6 empty transcript yields no responses", got == [], f"{len(got)} rows")
|
||||
finally:
|
||||
os.unlink(empty)
|
||||
|
||||
# AC-7: the subagent tree is discovered by the path globs.
|
||||
slug = "-home-worsch-clay-borg"
|
||||
paths = transcript_paths(slug)
|
||||
subs = [p for p in paths if "/subagents/" in p]
|
||||
check("AC-7 subagent tree enumerated", len(subs) > 0,
|
||||
f"{len(subs)} subagent transcript(s) of {len(paths)} total")
|
||||
|
||||
print("cb-cost self-test (positive control)")
|
||||
ok = True
|
||||
for name, passed, detail in checks:
|
||||
print(f" [{'ok ' if passed else 'FAIL'}] {name}" + (f" — {detail}" if detail else ""))
|
||||
ok &= passed
|
||||
return 0 if ok else 1
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--slug", default="-home-worsch-clay-borg")
|
||||
ap.add_argument("--pin", help="commit-ish or ISO Z instant (CA-07)")
|
||||
ap.add_argument("--by-task", action="store_true")
|
||||
ap.add_argument("--composition", action="store_true")
|
||||
ap.add_argument("--self-test", action="store_true")
|
||||
ap.add_argument("--json", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
if args.self_test:
|
||||
return self_test()
|
||||
|
||||
try:
|
||||
rep = collect(args.slug, args.pin)
|
||||
except Abort as e:
|
||||
print(f"ABORT — {e}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if args.json:
|
||||
print(json.dumps(rep, indent=2))
|
||||
else:
|
||||
render(rep, by_task=args.by_task, composition=args.composition)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
id: CB-WP-0002
|
||||
title: "Make agentic cost measurable, so D2 claims are falsifiable"
|
||||
status: proposed
|
||||
status: done
|
||||
state_hub_workstream_id: "b7c22f69-fbe9-48df-9619-007db79ae338"
|
||||
---
|
||||
|
||||
|
|
@ -18,19 +18,41 @@ transcript (`~/.claude/projects/<slug>/<session>.jsonl`) carries exact
|
|||
per-message `usage`, including the cache breakdown. Reading it for
|
||||
CB-WP-0001's session gives:
|
||||
|
||||
| Component | Tokens | Cost (Fable 5) |
|
||||
|---|---|---|
|
||||
| Output | 585,528 | $29.28 |
|
||||
| Cache read | 131,863,164 | **$131.86** |
|
||||
| Cache write (1h) | 4,365,668 | $87.31 |
|
||||
| Input | 1,090 | $0.01 |
|
||||
| **Total** | | **$248.46** |
|
||||
> **Corrected 2026-07-31 by T01.** This section originally reported
|
||||
> **$248.46** from 131,863,164 cache-read tokens priced at Fable 5. That
|
||||
> figure was wrong by ~2.7×, for two independent reasons found in
|
||||
> `research/CB-RES-0002-cost-accounting.md` §Correction: one API response
|
||||
> is written to the transcript as up to six JSONL lines that each repeat
|
||||
> the *same* `usage` object (657 lines, 346 real responses), and the
|
||||
> session ran three models, not one. The numbers below are the re-derived
|
||||
> ones. The workplan's premise is unaffected; its arithmetic was not.
|
||||
|
||||
The headline finding from that single reading is the reason this
|
||||
workplan exists: **53% of the cost is cache reads**, not output. Cost in
|
||||
an agentic loop is driven by context size × turn count, not by how much
|
||||
the model writes. No D2 decision made on "tokens per task" would have
|
||||
surfaced that.
|
||||
Deduplicated by `requestId` and priced per message at its own model's
|
||||
rate. **Pinned** to messages at or before `2026-07-31T02:17:59Z` (commit
|
||||
`fc76445`, the end of CB-WP-0001) — the transcript is a live file that
|
||||
grows as later sessions append to it, so an unpinned total is not a
|
||||
repeatable acceptance target:
|
||||
|
||||
| Component | Tokens | Cost |
|
||||
|---|---|---|
|
||||
| Output | 313,900 | $10.66 |
|
||||
| Cache read | 80,453,702 | **$59.59** |
|
||||
| Cache write (1h) | 1,672,854 | $21.95 |
|
||||
| Input | 676 | $0.00 |
|
||||
| Main transcript (339 responses: 213 opus-5, 118 fable-5, 14 sonnet-5) | | $92.21 |
|
||||
| Subagent tree (adversarial review, ran inside the pin) | | $1.11 |
|
||||
| **TRUE TOTAL** | | **$93.32** |
|
||||
|
||||
The headline finding survives the correction and gets sharper:
|
||||
**88.4% of the cost is cache, against 314k tokens of output — a 256:1
|
||||
ratio of context re-read to text written.** Cost in an agentic loop is
|
||||
driven by context size × turn count, not by how much the model writes. No
|
||||
D2 decision made on "tokens per task" would have surfaced that.
|
||||
|
||||
The correction is itself the lesson: this workplan opened with a
|
||||
mismeasured cost. Neither error was of the harness-does-nothing class the
|
||||
positive-control rule was written for — both sums ran over real data.
|
||||
What caught them was re-deriving the number instead of quoting it.
|
||||
|
||||
This workplan makes cost a first-class measured dimension so that
|
||||
future AM-12 equivalents support conclusions instead of decorating an
|
||||
|
|
@ -44,7 +66,7 @@ positive control.
|
|||
|
||||
```task
|
||||
id: CB-WP-0002-T01
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "2694c2c1-0070-4d8e-b4fc-196b582b36d5"
|
||||
```
|
||||
|
|
@ -67,7 +89,7 @@ accuracy and the hub to lead on durability.
|
|||
|
||||
```task
|
||||
id: CB-WP-0002-T02
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "eae248ab-f29f-4f11-9d20-e8145b0d822d"
|
||||
```
|
||||
|
|
@ -93,7 +115,7 @@ Gate: no collector code before this ADR is committed.
|
|||
|
||||
```task
|
||||
id: CB-WP-0002-T03
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "00d42ed2-4391-4580-aae2-06e3e151c69b"
|
||||
```
|
||||
|
|
@ -116,7 +138,7 @@ replace AM-12's definition with one that is computable.
|
|||
|
||||
```task
|
||||
id: CB-WP-0002-T04
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "9eb8329b-5f41-477b-8cf3-2cda5ba8dbe8"
|
||||
```
|
||||
|
|
@ -140,16 +162,22 @@ each message at its own model's rate.
|
|||
|
||||
```task
|
||||
id: CB-WP-0002-T05
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "bea4cc0a-e4d0-4077-9dc7-df7726a48f86"
|
||||
```
|
||||
|
||||
Run the collector over the CB-WP-0001 session and commit
|
||||
`evidence/CB-EV-0002-cost-accounting.md`. Reconciliation against the
|
||||
totals in this workplan's Purpose section is the acceptance test: the
|
||||
tool must reproduce $248.46 (Fable 5) from the same transcript, or
|
||||
explain the difference.
|
||||
`evidence/CB-EV-0002-cost-accounting.md`. The acceptance test is
|
||||
`specs/CostAccounting.md` AC-1: reproduce **$93.32 pinned at `fc76445`,
|
||||
as its two components** ($92.21 main transcript + $1.11 subagent tree),
|
||||
plus AC-2's zero reconciliation residual.
|
||||
|
||||
*(Originally written as "must reproduce $248.46". That figure was wrong
|
||||
by 2.7×, and the first corrected target — $92.21 — was itself reachable
|
||||
only by a collector carrying the subagent blind spot the survey had just
|
||||
documented. Both errors are recorded rather than quietly overwritten;
|
||||
the sequence is the point.)*
|
||||
|
||||
Then use it to answer at least one question that could not be answered
|
||||
before, and record the answer. Candidates: which of T01–T09 cost most
|
||||
|
|
@ -164,7 +192,7 @@ cleared the bar that CB-WP-0001's AM-12 failed to clear.
|
|||
|
||||
```task
|
||||
id: CB-WP-0002-T06
|
||||
status: todo
|
||||
status: done
|
||||
priority: low
|
||||
state_hub_task_id: "1412263b-70c1-43e4-957d-1ad6c3203ca9"
|
||||
```
|
||||
|
|
@ -179,7 +207,7 @@ the one command surface.
|
|||
|
||||
```task
|
||||
id: CB-WP-0002-T07
|
||||
status: todo
|
||||
status: done
|
||||
priority: low
|
||||
state_hub_task_id: "ebe58d91-be5f-4d5b-ba40-b03275b4eefc"
|
||||
```
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue