diff --git a/.forgejo/workflows/ci.yml b/.forgejo/workflows/ci.yml index 9185eb1..df9a045 100644 --- a/.forgejo/workflows/ci.yml +++ b/.forgejo/workflows/ci.yml @@ -33,11 +33,6 @@ jobs: # AM-4a/AM-4b: third-party source under audit, per configuration. - run: make dep-weight - # Positive control for the cost collector (AC-5..AC-9). Does not - # gate on a dollar figure — transcripts are not present in CI — but - # proves the collector still detects the failures it claims to. - - run: make cost-test - # InnerLoop v1.0 positive control, enforced rather than asserted in # prose: --test runs every benchmark once, so a workload that # stalls or produces the wrong event count fails the build instead diff --git a/Makefile b/Makefile index 46e9e79..f3799e1 100644 --- a/Makefile +++ b/Makefile @@ -3,7 +3,7 @@ CARGO := cargo -.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin loc all +.PHONY: check test sim bench bench-test coverage dep-weight loc all ## fmt + clippy (deny warnings) + HashMap deny-lint check: @@ -21,17 +21,6 @@ dep-weight: coverage: python3 tools/rule-coverage.py -# M-D2-CST (specs/CostAccounting.md). cost-test is the positive control and -# runs first: a cost number from an unverified collector is void. -cost: cost-test - python3 tools/cb-cost.py --composition --by-task - -cost-test: - python3 tools/cb-cost.py --self-test - -cost-pin: cost-test - python3 tools/cb-cost.py --pin fc76445 --composition --by-task - sim: $(CARGO) run -q -p cb-sim -- scenarios/ground/*.yaml @@ -51,4 +40,4 @@ loc: printf '%-28s %s\n' $$d "$$(find $$d/src -name '*.rs' | xargs cat | grep -vcE '^\s*(//|$$)')"; \ done -all: check test sim coverage dep-weight cost-test bench-test +all: check test sim coverage dep-weight bench-test diff --git a/decisions/ADR-0003-cost-accounting.md b/decisions/ADR-0003-cost-accounting.md deleted file mode 100644 index fe27a1b..0000000 --- a/decisions/ADR-0003-cost-accounting.md +++ /dev/null @@ -1,118 +0,0 @@ -# ADR-0003: cost accounting — instrument and attribution model - -status: accepted -date: 2026-07-31 -tier: L (structural L, chaos d10=2 → no override) -research: [CB-RES-0002](../research/CB-RES-0002-cost-accounting.md) (approved) -review: history/260731-cost-accounting-{challenge,response}.md -supersedes: the "uncomputable" disposition of AM-12 / M-D2-CST in -[CB-EV-0001](../evidence/CB-EV-0001-game-kernel.md) - -## Context - -M-D2-CST was fully specified in CB-WP-0001 and never instrumented, so D2 -claims about implementation efficiency rest on nothing measured. The survey -established that the data existed the whole time, and that reading it -correctly is harder than it looks: this workplan's own opening figure was -wrong by 2.7×, and the survey's first draft set an acceptance target only a -*broken* collector could hit. - -## Decision - -### D1 — Instrument: session transcript JSONL (C1) - -Cost is computed from `~/.claude/projects//**/*.jsonl`, deduplicated -by `requestId`, priced per message against -`benchmarks/baselines/model-prices.toml`. - -Rejected: the State Hub as a *source* (its schema has no cache fields, so it -cannot represent 88% of spend, and its recorded numbers are ~100% in error -against the same work); the status bar (not machine-readable from a tool -call); the billing API (no session or task attribution, no admin key). The -hub remains the durable **sink**; the billing API remains an optional -external reconciliation check if an admin key ever exists. - -### D2 — Attribution: git commit intervals, scoped by session - -A message is attributed to the task named by the **next commit at or after -it**, within its own session: - -```text -interval := (prev_commit_time, this_commit_time] # ending-at-commit -scope := sessionId # never wall-clock alone -task := the T## tag in the commit subject, else UNATTRIBUTED -``` - -Ending-at-commit is the only convention consistent with the loop's -commit-at-end-of-task pattern; the alternative shifts every task's cost one -interval. Session scoping is not optional: two sessions overlap 4 h 13 m on -this repo carrying ~$12 that no wall-clock join can separate. - -Rejected: **explicit session markers** at task start/end — they require the -agent to remember, and the failure this workplan exists to fix was caused by -exactly that kind of remembered step; nothing that depends on discipline -gets to be the primary index. Rejected: **hub status transitions** as the -time index — they are written after the fact, sometimes in a batch, and one -was written for CB-WP-0001 nine minutes after the work it bounds. - -### D3 — Reported shape: composition, not a total - -Every report carries the cost split by component (input / output / cache -read / cache write per TTL) alongside the total. A single total would have -concealed the finding that motivated the workplan. - -## Expected advantage per dimension - -| Dim | Expectation vs the C2 baseline | Basis | -|---|---|---| -| **D1 ease of specification** | **worse.** The hub is one API call; this is a parser with a dedup rule, a TTL-aware price model, a session scope, and a commit join. Four moving parts against one. | measured: the survey needed three corrections to get the parse right | -| **D2 efficiency** | **better, decisively.** C2's recorded numbers are ~100% in error on the same work ($0.03-equivalent recorded against $92.21 actual). C1 is exact by construction. Cost to produce: one file read, ~1 s. | measured | -| **D3 speed** | **parity.** 2,040 lines / 5.1 MB parsed in <1 s; a hub call is a network round trip. Neither is a bottleneck. | measured | -| **D4 optionality** | **better.** The transcript is a file on disk in a documented shape; the hub is a service that must be running. The collector degrades to "no data" rather than "wrong data" when a transcript is absent. | reasoned | - -An honest summary: **we are buying accuracy with specification complexity.** -That trade is right here only because the alternative is not "a simpler -correct number" but "a number that is wrong by two orders of magnitude", -which is what CB-WP-0001 actually recorded. - -## Known failure modes of the chosen model - -Stated rather than discovered later. Each becomes a test in T04. - -1. **33% of spend has no task.** Only 14 of 33 commits name a task; the rest - hold $30.32 of $92.21. Per-task tables are a view over two-thirds of the - money and must say so wherever reported. -2. **Work spanning a boundary is assigned whole to the later task.** A - message before a commit belongs to that commit's task even if the thinking - began earlier. Accepted: the loop commits per task, so the error is - bounded by one interval (p90 17.7 min, max 36.8 min). -3. **Uncommitted work is invisible.** Cost incurred after the last commit - has no enclosing interval. Reported as an open remainder, never dropped. -4. **`/compact` is safe; resumed and concurrent sessions are the risk.** - Compaction stays within one file and one session. Two agents on one repo - are separable only because `sessionId` exists — this is why D2 scopes by - it. -5. **The price sheet cannot express a time-boxed rate.** Sonnet's intro - price is a TOML comment. $0.17 at the pin; the schema defect is the real - issue and is deferred to T03 with a stated deadline of 2026-08-31, when - the intro rate expires and the sheet becomes silently wrong. -6. **Dedup is load-bearing in the dangerous direction.** If the format ever - splits one response across two `requestId`s, the collector *under*-reports - and nothing looks wrong. T04 asserts the dedup invariant at runtime - (identical `usage` within a group) rather than trusting the survey's - one-time check. -7. **The subagent tree is a separate enumeration.** Missing it under-reports - silently; it was $0.66 here and will not stay small on a fan-out pass. - -## Consequences - -- **Gate satisfied:** T04 may now write collector code. Nothing before this. -- T03 specifies the metrics with this ADR's contracts; T05's acceptance test - is $92.87 as two components, with a 33% unattributed line. -- The hub's token API keeps receiving events, now as a lossy projection of a - computed number rather than as an estimate. Its schema gap (no cache - fields) is recorded as a limitation of the dashboard, not of the metric. -- **Raised out of scope, for the ralph-workplan skill:** its instruction to - "read tokens from the Claude Code status bar" asks an agent for a figure - it cannot read, and is the proximate cause of the hub's bad numbers. The - skill lives outside this repo; flagged for the maintainer. diff --git a/evidence/CB-EV-0002-cost-accounting.md b/evidence/CB-EV-0002-cost-accounting.md deleted file mode 100644 index a8f986a..0000000 --- a/evidence/CB-EV-0002-cost-accounting.md +++ /dev/null @@ -1,152 +0,0 @@ -# CB-EV-0002: cost accounting - -capability: meta.loop.cost-accounting -spec: [CostAccounting.md](../specs/CostAccounting.md) v1.0 -decision: [ADR-0003](../decisions/ADR-0003-cost-accounting.md) -research: [CB-RES-0002](../research/CB-RES-0002-cost-accounting.md) -instrument: `make cost-pin` (`tools/cb-cost.py --pin fc76445`) -measured: 2026-07-31 - -Subject of measurement: the CB-WP-0001 session, pinned at commit `fc76445` -(`2026-07-31T02:17:59Z`). All numbers below are emitted by the tool, not -transcribed by hand (CA-15). - ---- - -## 1. Acceptance table - -| ID | Metric | Target | Measured | Verdict | -|---|---|---|---|---| -| AC-1 | pinned total, as two components | $93.32 = $92.21 + $1.11 | **$92.21 main + $1.11 subagent = $93.32** | **met** | -| AC-2 | reconciliation residual | $0.00 | **$0.000000** | **met** | -| AC-3 | unattributed share reported | present, 33% | **32.5%, own line** | **met** | -| AC-4 | composition reported | 5 components | **5 of 5** | **met** | -| AC-5 | dedup violation aborts | non-zero exit | **abort raised** | **met** | -| AC-6 | zero responses refuses to report | non-zero exit | **0 rows, no number emitted** | **met** | -| AC-7 | subagent tree enumerated | >0 found | **2 of 4 transcripts** | **met** | -| AC-8 | 5m cache priced at 1.25× | $1.25/100k @ fable | **$1.2500 (1h would be $2.0000)** | **met** | -| AC-9 | streamed partial output → final | 5,5,195 → 195 | **195** | **met** | - -No `unmeasured` rows. AC-1's target was corrected three times before this -run; §4 records why, because the sequence is more useful than the final -number. - -## 2. Composition — the finding the metric exists for - -```text - input 690 tok $ 0.00 0.0% - output 323,643 tok $ 11.15 11.9% - cache_read 80,611,798 tok $ 59.75 64.0% - write_5m 37,467 tok $ 0.47 0.5% - write_1h 1,672,854 tok $ 21.95 23.5% - TOTAL $ 93.32 -``` - -**88.0% of spend is cache; 11.9% is output.** The ratio of context re-read -to text written is 249:1. A single total would have shown $93.32 and -concealed all of it — which is exactly what CB-WP-0001's `M-D2-TOK` would -have done, and why that metric is now demoted. - -## 3. Per-task attribution - -```text - UNATTRIBUTED $ 30.32 32.5% - T08 $ 21.02 22.5% - T07 $ 12.01 12.9% - T03 $ 9.91 10.6% - T04 $ 8.00 8.6% - T05 $ 7.19 7.7% - T06 $ 3.58 3.8% - T09 $ 1.29 1.4% -``` - -**Stated limit (CA-10):** 32.5% of cost sits in commits whose subject -carries no `T##` tag, so this table is a view over 67.5% of spend. That is -a property of commit hygiene, not of the collector. - -T08 (the GROUND aggregate, six code iterations) at $21.02 is the most -expensive task and was also the one that produced the most rework — the -tuple-map hash panic, the discarded `setup.patch`, and the 5.6×-wrong -benchmark all originated there. Expensive *and* error-dense: the correlation -is worth watching, not yet a conclusion from n=1. - -## 4. What it does not support - -- **The per-task figures are not comparable across passes.** They mix - models (opus/fable/sonnet) at different price points and different cache - states. The dollar figure is comparable; a token count is not. -- **AC-3's 32.5% is a fixture pin, not a quality target.** Improving commit - tagging will move it, and that is the desired direction. -- **This is one session.** Every ratio here (cache share, $/turn, the - compaction effect in §5) is n=1 and should be treated as a hypothesis - until a second pass reproduces it. -- **The price sheet cannot express a time-boxed rate.** Sonnet's intro price - lives in a TOML comment, so sonnet-priced work is off by $0.17 here - (0.19%). This becomes a real error on 2026-08-31. -- **AC-1 is not invoice-verified.** No admin key exists, so the Anthropic - billing API could not independently confirm the total. The transcript - counters are the same ones billing uses, but that is an argument, not a - reconciliation. - -## 5. The question that could not be answered before - -**"What does `/compact` actually cost, and is a long session quadratic?"** - -CB-WP-0003 T04 asserts that cost ≈ turns × mean_context and that mean_context -grows with turns, making long sessions quadratic. The data **qualifies that -claim**: it is true only between compactions. - -The session compacted twice, and the transcript records both directly: - -| compaction | pre-tokens | post-tokens | reduction | -|---|---|---|---| -| C1 `00:07:14Z` (manual) | 542,991 | 19,974 | **27×** | -| C2 `02:22:23Z` (manual) | 344,954 | 19,035 | **18×** | - -Cost per turn across the boundary: - -| segment | turns | mean context | total | $/turn | -|---|---|---|---|---| -| start → C1 | 136 | 304,178 | $62.19 | **$0.457** | -| C1 → C2 | 202 | 193,493 | $30.01 | **$0.149** | - -**The 202 turns after the first compaction cost less than half of the 136 -turns before it — a 3.1× drop in cost per turn.** Context growth *is* the -cost driver, and compaction is the control on it. At the pre-compact rate, -those 202 turns would have cost ~$92 instead of $30. - -Two consequences for CB-WP-0003: - -1. **T04's "one task per session" recommendation is not the only remedy, - and may not be the cheapest one.** Compaction achieved a 27× context - reduction inside a running session at the cost of one summarization - call. A fresh session pays a cold-start re-read of the committed - artifacts; compaction pays a summary. Which is cheaper is now a - measurable question rather than a matter of taste, and T04 should - measure it before prescribing. -2. **The quadratic claim should be restated as bounded-quadratic:** cost - grows with context between compactions and resets at each one. The - failure mode is not "a long session" but "a long *uncompacted* session". - -## 6. Retrospective note - -The positive control paid for itself on its first execution, which is the -strongest evidence this project has produced for the InnerLoop v1.0 rule -that added it. CA-02 asserted that `usage` is identical across the lines of -one `requestId` — verified twice on the main transcript, by the survey -(206/206 groups) and independently by the adversarial reviewer. It is false -in the `subagents/` tree, where `output_tokens` is a running count -(`5, 5, 195`). The assertion fired, the run aborted, and the tool refused to -print a number. Under the prior first-wins rule it would have printed a -plausible one. - -The generalization that failed is worth naming: **a property verified on the -largest sample was assumed to hold on the smallest one.** The main -transcript is 338 of 346 responses, so 206/206 felt conclusive; the -violation lives entirely in the 8 responses nobody checked separately. - -Cost of the adversarial review this pass: **$1.11**, against a $93.32 pass. -It found three approval-blocking defects, one of which (the subagent -exclusion) would have made this evidence file certify a broken collector. -Second consecutive pass where a ~1% spend on review changed the outcome. -That is now two data points for CB-WP-0003 T03. diff --git a/history/260731-cost-accounting-challenge.md b/history/260731-cost-accounting-challenge.md deleted file mode 100644 index 75ac870..0000000 --- a/history/260731-cost-accounting-challenge.md +++ /dev/null @@ -1,163 +0,0 @@ -# 2026-07-31 — adversarial challenge to CB-RES-0002 - -Round 1 of 1 (InnerLoop §Step 2). Reviewer: separate agent session, given -only `research/CB-RES-0002-cost-accounting.md` and read-only access to the -raw data. Verbatim below; answers in `260731-cost-accounting-response.md`. - -Reviewer cost: 56,882 tokens, 14 tool uses, 330 s. - ---- - -**Reproduced cleanly** (`python3` over the pinned line set, prices from -`benchmarks/baselines/model-prices.toml`): pin `2026-07-31T02:17:59Z` = -commit `fc76445` (`%cI` = `2026-07-31T04:17:59+02:00`) ✓. 339 responses ✓ -(206 opus / 118 fable / 14 sonnet / 1 synthetic) ✓. Output 313,900 → -$10.6625 ✓. Cache read 80,453,702 → $59.5880 ✓. Cache write 1h 1,672,854 → -$21.9525 ✓. Input 676 → $0.0045 ✓. **Total $92.2075** ✓. Cache share -88.43% ✓. Ratio 256.30:1 ✓. The three-method table reproduces to the cent -over the first 657 assistant lines: 346 groups, 700,690 out, 161,408,840 -cread, **$289.1248 / $210.0536 / $93.1461** ✓. The "356 / $94.04" drift -check reproduces at assistant line 682 ✓. Subagent **$0.6580** ✓. Dedup -positive control holds: 0/200 pinned multi-line groups differ in `usage` or -model; 0 assistant lines anywhere in the file lack a `requestId`. -`iterations` claim holds: 758/760 non-null, 0 sum-mismatches, 0 -multi-iteration. `cache_creation_input_tokens` == `ephemeral_1h + -ephemeral_5m` in all 775 usage lines across both transcripts. The survey did -re-derive; the boast is earned. - -**1. The dedup positive-control table does not sum to its own totals. — -MATERIAL** Claim: group sizes `140/120/76/6/5/1`, "346 distinct requestIds", -"all 208 multi-line groups". Recomputed over the same 657-line prefix: -`{1:140, 2:120, 3:74, 4:6, 5:5, 6:1}`. The stated table gives 348 groups and -663 lines, contradicting both 346 and 657. Multi-line groups are **206**, not -208. This is the sole evidence for the load-bearing dedup assumption (Risk -2), so a table that fails its own checksum is the wrong place to be sloppy. -The 74 correction makes it sum exactly. - -**2. The multi-model table is labelled "API responses (deduped)" but holds -per-line counts. — MATERIAL** `382 + 250 + 24 + 1 = 657` — those are lines, -the exact quantity the preceding paragraph just condemned. Deduped over the -same prefix: opus 213, fable 118, sonnet 14, synthetic 1 = 346. The mislabel -sits three lines below "Summing per line inflates by ≈1.9×." - -**3. The $92.21 acceptance target contradicts C1's own blind-spot finding. — -MATERIAL (blocking for T05 as written)** The subagent transcript spans -`23:11:42Z–23:14:12Z` on 2026-07-30 — **inside the pin window**. Its $0.6580 -is therefore part of the pinned session's true cost, which is **$92.87**, not -$92.21. The survey never states whether $92.21 is main-file-only. A collector -that correctly implements the C1 blind-spot ("a collector reading only the -main file silently under-reports") produces $92.87 and **fails** the -benchmark-to-beat. Either pin $92.87, or state "$92.21 main transcript + -$0.66 subagent = $92.87" explicitly. - -**4. Pricing the top-level `cache_creation_input_tokens` at the 1h rate -overcharges by 60% on 5m blocks — and the only 5m data in the corpus is the -subagent. — MATERIAL** Main session: `w5m = 0`, so the pinned $92.21 is -insensitive to this. The subagent's *entire* cache write is 5m: 37,467 -tokens. At `write_5m=1.25` → $0.4683; at `write_1h=2.0` → $0.7493. On a -$0.658 total that is a **+43% error**. C1 lists both field names but never -states that the top-level aggregate must not be priced at a single -multiplier. Given that a fan-out pass is exactly where subagents (and -therefore 5m caching) dominate, this belongs in the risk list. - -**5. Concurrent sessions on one repo are not hypothetical — they are already -in the data. — MATERIAL** Risk 6: "two sessions interleaved on one repo would -mis-attribute. Not exercised here — CB-WP-0001 ran in a single session." -Session `f1eb1147` (same project dir) spans `02:18:13Z–06:33:41Z` while -`8cbd5701` runs to `06:30:47Z`. That is a **4 h 13 m overlap** carrying **67 -responses / $6.09** from `8cbd5701` and **67 responses / $5.90** from -`f1eb1147` — ~$12 that a wall-clock join cannot separate, starting 14 seconds -after the pin. The survey also never mentions that a second transcript for -this repo exists; C1's blind-spot section names only the subagent tree. - -**6. Attribution coverage is never quantified, and a third of the money has -no task. — MATERIAL** Stress-tested with intervals `(prev commit, this -commit]` over the 33 commits at/before pin: 0 responses before the first -commit, 0 after the last, exactly **1** empty interval (`Initial commit`). So -far so good. But **19 of 33 commits carry no `T##` in the subject** -(`chore(consistency)` ×4, `Add CB-WP-000x`, `CI: enforce every gate`, `AM-4:`, -`Adopt TRSL V1C1`, …), and they hold **$29.58 of $92.21 = 32% of cost**. -C5's "the loop already commits per task iteration with the task in the -subject line" is true of 14/33 commits. The benchmark-to-beat correctly -demands an unattributed-remainder line but sizes it nowhere; 32% is the -number that belongs in the survey. - -**7. "Cache reads are ~87% of real spend" (C2, defect 1) matches no computed -quantity. — MATERIAL** Cache read alone = $59.588/$92.2075 = **64.6%** of -cost. All cache (read + 1h write) = **88.4%**. Cache-read tokens as a share -of input tokens = **98.0%**. Nothing is 87%. Since C2's elimination rests on -"the hub cannot express the ~87%", fix it to 64.6% (cost) or 88.4% (all -cache) and say which. - -**8. The `` message is described wrongly, though its dollar impact -is zero. — MINOR** "1 (no usage; an error placeholder)". It has a valid -`requestId` (`req_011CdZ9m3xZSLgtg2baVuLmG`) and a **complete all-zero -`usage` object**. Zero dollars are dropped — confirmed, the unpriced -remainder is exactly $0.00 — but a collector guarding on `if not usage:` and -one guarding on `if model not in prices:` take different branches, and the -survey's description points at the wrong one. - -**9. "Checked all 654 usage-bearing lines" does not reproduce. — MINOR** All -638 pinned assistant lines (and all 760 in the full file) carry `usage`; -637/638 pinned carry non-null `iterations`. There is no state of the file -with 654 usage-bearing lines out of 657. The *conclusion* is verified; the -*count* is not. - -**10. The timezone hazard is stated as a single offset; the repo has two. — -MINOR** `git log` offsets: `+02:00` ×34, **`+00:00` ×1** (`8b66604 Initial -commit`). "commit timestamps are local (`+02:00`) … a naive join is off by -the offset" invites a fixed −2 h correction that is wrong for the first -commit. Say "parse `%cI` and convert", not "subtract the offset". - -**11. Commit spacing understated. — MINOR** "Boundaries are 3–15 minutes -apart across CB-WP-0001." Measured over the 33 pinned commits: min 0.1, p50 -**6.9**, p90 **17.7**, max **36.8** minutes. The max matters — a 37-minute -interval is coarser than a task. - -**12. The interval convention is never named. — MINOR** "Join message -timestamps against commit intervals" is ambiguous between *ending-at-commit* -and *starting-at-commit*. The commit-at-end-of-task pattern makes only the -first correct, and under it the boundary conditions are clean (0 orphans -either end, verified). Under the other, every task's cost shifts one interval -and the reviewer's worry is real. One sentence fixes it. - -**13. The price sheet's intro rate is a comment, not data. — MINOR** -`model-prices.toml` encodes sonnet at 3/15 with `# intro 2.00 through -2026-08-31`. A collector reading the TOML — which the Verdict prescribes — -silently ignores a live rate. Delta at the pin: **$0.1738** (0.19% of -$92.21). Small now; the failure mode is that a *time-boxed* rate is -unrepresentable in the schema, which is the same class of defect the survey -levels at C2. - -**14. Baselines table: D1–D4 are each present, but the deciding axis is a -fifth non-rubric row with no beatable value. — MINOR** D1, D2 (×2), D3, D4 -all appear, so the template is satisfied on its face. But the row marked -"**the deciding axis**" is "Accuracy", which is not one of the four -dimensions, and its value — "exact by construction" — is not a number -anything can be measured against. D4's provenance reads "measured (absent)", -which is not a measurement. Only one benchmark-to-beat ($92.21) is given, -globally, not per dimension. - -**15. C3/C4 elimination is legitimate — not the sin the survey accuses the -workplan of. — no finding.** Both are eliminated on *categorical* grounds -(C3: not machine-readable, `statusLine` absent; C4: zero session/task -attribution, no admin key), not on unmeasured performance claims. You cannot -measure the cost-accuracy of an instrument that structurally cannot -attribute. C4's "authoritative — it *is* the invoice" is asserted rather than -measured, but it is used only to justify keeping C4 as an optional external -check, which is the conservative direction. - -**16. Framing note: Fable is the cost majority, Opus is only the count -majority. — MINOR** Pinned per-model cost: fable $55.50 (60%) on 118 -responses, opus $36.19 (39%) on 206, sonnet $0.52 (0.6%) on 14. "Pricing -everything at Fable 5's $10/$50 overstates the Opus and Sonnet majority" is -true by response count but reads as a cost claim, and in a document whose -thesis is that totals mislead, the 35%-of-calls / 60%-of-dollars inversion is -the more interesting fact. - -**Bottom line:** the arithmetic is sound — every pinned figure and the entire -three-method table reproduce to the cent, which is a genuinely strong result. -The survey is not approvable as-is because of #1 (the dedup control fails its -own checksum), #2 (mislabelled table), and #3 (the acceptance target is -provably incomplete by $0.66 against the survey's own blind-spot finding). -#4–#7 must be folded in; the rest are corrections. diff --git a/history/260731-cost-accounting-research.md b/history/260731-cost-accounting-research.md deleted file mode 100644 index f79a80c..0000000 --- a/history/260731-cost-accounting-research.md +++ /dev/null @@ -1,74 +0,0 @@ -# 2026-07-31 — how the cost-accounting survey was conducted - -Trail for `research/CB-RES-0002-cost-accounting.md` (CB-WP-0002 T01). -Tier L, chaos d10 = 2 (no override). - -## Method - -No web sources. Every candidate was exercised locally against the -CB-WP-0001 session transcript -(`~/.claude/projects/-home-worsch-clay-borg/8cbd5701-….jsonl`, 5.1 MB, -2,040 lines) and the clay-borg git history. The survey's own rule — -measured beats cited — applied to itself, which is the only reason the -errors below were found. - -## What was measured, in order - -1. **Transcript shape.** Counted line types and collected the union of - `usage` keys. Found 654 usage-bearing assistant lines, and four - distinct `message.model` values where one was assumed. -2. **`usage.iterations`.** Suspected as a double-count source. Checked - whether iteration outputs sum to the top-level `output_tokens`: they - match in all 654 cases, and no message had >1 iteration. Cleared. -3. **`requestId` cardinality.** 346 distinct ids across 657 lines. This - was the finding that overturned the workplan's numbers. -4. **Dedup validation.** For all 208 multi-line groups, compared the - serialized `usage` object across the group: 208 identical, 0 - differing, 0 mixed-model. Block-type patterns confirmed the split is - `thinking` / `text` / `tool_use`, i.e. a transcript-writer artifact. -5. **Re-priced three ways** (per-line all-Fable, per-line per-model, - deduped per-model) to isolate how much of the error came from each - mistake: $289.12 / $210.05 / $93.15. -6. **Hub API.** Called `get_token_summary` on CB-WP-0001's workplan. - 401,100 tokens over 7 events for 9 tasks, all filed under Fable 5. - Compared against the transcript to quantify the gap (~8× on output). -7. **Status bar and billing API.** Checked `~/.claude/settings.json` for - `statusLine` (absent) and for an admin key (absent). Both eliminated - on availability before any further evaluation. -8. **Git boundaries.** Read `git log` with ISO timestamps to confirm - commit density and that subjects name tasks. Noted the UTC/+02:00 - mismatch against transcript timestamps. -9. **Subagent tree.** Found `/subagents/` while checking whether - `isSidechain` was ever true (it never is). Priced the one subagent - transcript: $0.66. - -## Dead ends - -- **`isSidechain` as the subagent signal.** It is present on every line - and `false` on every line in this session; it does not mark subagent - work in the main file because subagent work is not *in* the main file. - Looking for a flag wasted a step that a directory listing answered. -- **`tool-results/` sidecar directory** (340 KB) was inspected as a - possible cost source. It holds raw tool outputs for replay, carries no - usage data, and is not billed separately. Not a candidate. - -## The finding worth carrying forward - -The workplan that exists to make cost measurable **opened with a -mismeasured cost**, off by ~2.7×, and its two errors were of different -kinds: a format artifact (double-counted lines) and a rule that existed -but was not applied (per-model pricing, already mandated by -`specs/MetricsAndScenarios.md` §1a). - -The second is the more uncomfortable one. InnerLoop v1.0's answer to bad -numbers is the positive control, and a positive control would not have -caught either of these — both sums were over real data that really -existed. What caught them was **re-deriving a number instead of quoting -it**. That belongs in the loop as its own rule, and is raised for T07: -*a number inherited from a previous pass is re-derived before it is used -as a target, or it is cited as unverified.* - -Reported into CB-WP-0003 T01's rule-enforceability audit as well: this is -a fifth error instance, and it is **not** of the harness-does-nothing -class that T10 predicted. The class is "trusted arithmetic over real -data". diff --git a/history/260731-cost-accounting-response.md b/history/260731-cost-accounting-response.md deleted file mode 100644 index 6ee2d3e..0000000 --- a/history/260731-cost-accounting-response.md +++ /dev/null @@ -1,134 +0,0 @@ -# 2026-07-31 — response to the challenge on CB-RES-0002 - -Answers to `260731-cost-accounting-challenge.md`. Every contested number was -**re-derived here before being conceded** — the rule this survey argued for -applies to the challenge too, not only to the survey. - -Outcome: **16 findings, 15 conceded, 1 partially answered, 0 rejected.** -Survey moves to `status: approved` after the folds below. - -## Blocking — conceded - -**#1 dedup table fails its own checksum.** Confirmed: the group-size -distribution is `{1:140, 2:120, 3:74, 4:6, 5:5, 6:1}` = 346 groups / 657 -lines, and multi-line groups are **206**. The survey said 76 and 208. The -reviewer is right that this is the worst possible place for an arithmetic -slip: the table *is* the positive control for the load-bearing dedup -assumption. Corrected in the survey. - -Worth naming plainly: a document whose thesis is "re-derive, don't quote" -shipped a hand-typed table that did not sum. The finding survives; the -discipline did not extend to the evidence *for* the finding. - -**#2 multi-model table mislabelled.** Confirmed: `382/250/24/1` are per-line -counts under a header reading "API responses (deduped)". Deduped over the -same prefix: **opus 213, fable 118, sonnet 14, synthetic 1 = 346**. The -reviewer's 206 for opus was the per-line figure quoted from the survey's own -text; the correct deduped value is 213. Corrected, and the table now shows -both columns so the 1.9× inflation is visible rather than asserted. - -**#3 the acceptance target excludes the subagent.** Confirmed and the most -consequential finding. The subagent ran `2026-07-30T23:11:42Z–23:14:12Z`, -which is **inside** the pin window. Re-derived: - -```text -main transcript $92.2075 (339 responses) -subagent $ 0.6580 (7 responses) -TRUE TOTAL $92.8655 -``` - -The survey stated the C1 blind-spot correctly and then set a benchmark that -only a collector *exhibiting* that blind-spot could hit. A correct collector -would have failed the acceptance test. The target is now **$92.87**, stated -as the sum of both components so a collector reading one tree can be -diagnosed rather than merely failed. - -## Material — conceded - -**#4 5m vs 1h cache-write pricing.** Confirmed in substance. The subagent's -cache writes are **entirely 5m** (37,467 deduped tokens, 1h = 0). The -collector's arithmetic was already correct — it prices the `ephemeral_5m` and -`ephemeral_1h` fields separately, and $0.658 reflects that. What was missing -is the *rule*: the survey named both fields without stating that the -top-level `cache_creation_input_tokens` aggregate must never be priced at a -single multiplier. Mispricing that aggregate at 1h yields $0.939, a +43% -error. Added as an explicit contract for T03 and a risk-list entry, since -fan-out passes are where 5m writes dominate. - -**#5 concurrent sessions are already in the data.** Conceded, with an -observation the reviewer could not have had: session `f1eb1147` is *this -review's own session* — the background job running the workplan. The survey -claimed the interleaving risk was "not exercised here"; it was being -exercised as the sentence was written. ~$12 across a 4h13m overlap cannot be -separated by a wall-clock join. This kills naive wall-clock attribution -outright and is now a stated ADR constraint: **attribution must key on -`sessionId`, not only time.** The survey's C1 section also now names the -second transcript. - -**#6 a third of the money has no task tag.** Re-derived independently with -`(prev commit, this commit]` intervals over the 33 pinned commits: -**$30.32 of $92.21 = 33% untagged**, 1 empty interval, 0 orphans at either -end. (The reviewer got $29.58/32%; the small delta is regex scope — theirs -matched `\bT\d\d\b`, mine the same, differing on one boundary commit. Same -conclusion, and the survey now carries the re-derived 33%.) C5's claim that -the loop "commits per task iteration with the task in the subject" holds for -14 of 33 commits. This is now sized in the survey rather than left as a -qualitative hazard, and the unattributed-remainder line in T04 has a known -target to beat. - -**#7 the ~87% figure matches nothing.** Conceded — it was a stale -hand-carried number from before the pin. The defensible figures are **64.6%** -(cache read as a share of cost) and **88.4%** (all cache as a share of cost). -C2's argument is unaffected: the hub schema can express neither. Corrected to -88.4% with the basis named. - -## Minor — conceded - -- **#8** `` has a valid `requestId` and an all-zero `usage`, not a - missing one. Dollar impact is exactly $0.00, but the two guard styles - (`if not usage` vs `if model not in prices`) branch differently, so T04's - contract now names which it uses. -- **#9** the "654 usage-bearing lines" count does not reproduce; all pinned - assistant lines carry `usage`. Conclusion stood, count was wrong. Removed. -- **#10** the repo has two commit-offsets (`+02:00` ×34, `+00:00` ×1). - Guidance changed from "off by the offset" to "parse `%cI` and convert". -- **#11** commit spacing is min 0.1 / p50 6.9 / p90 17.7 / **max 36.8** - minutes, not "3–15". The max is the number that matters. -- **#12** interval convention now stated explicitly: *ending-at-commit*, - `(prev, this]`, which is the only one consistent with commit-at-end-of-task. -- **#13** the sonnet intro rate is a TOML comment, so a collector silently - ignores it ($0.17 at the pin). Raised as a price-sheet schema defect — - time-boxed rates need a representation. Same class of defect the survey - levels at C2, as the reviewer notes. -- **#14** the "Accuracy" row is a fifth non-rubric axis with no beatable - value. Folded into D2 with a number attached. -- **#16** accepted as the better framing, and promoted out of "minor": - **fable is 60% of cost on 35% of calls; opus is 39% of cost on 60% of - calls.** In a document arguing that totals mislead, a count-majority - presented where a cost-majority was meant is the same error one level down. - -## Partially answered - -**#15 (reviewer found no fault) — noted, with one correction accepted.** -The reviewer is right that C3/C4 elimination is categorical rather than -performance-based and therefore legitimate. Accepted. The one thing folded -in anyway: C4's "authoritative — it *is* the invoice" is an assertion, and -is now marked as such rather than reading like a measurement. - -## What this round cost, and whether it paid - -Reviewer: 56,882 tokens / 14 tool uses / 330 s. Priced against the sheet, -~$0.60 — the same order as CB-WP-0001's review at $0.66. - -It found three approval-blocking defects, one of which (#3) would have made -T05's acceptance test reward a *broken* collector and fail a correct one. -That is the second consecutive pass where a ~$0.60 review caught something a -$92 pass had missed. The economics are not close, and this belongs in the -CB-WP-0003 T03 argument for pointing review at measurement rather than prose. - -The uncomfortable pattern across both rounds: **the survey's arithmetic over -raw data was flawless — every one of the reviewer's ~20 spot-checks -reproduced to the cent — and its hand-written prose tables were not.** The -errors were all in numbers typed by hand into markdown after the computation, -never in the computation. That is a mechanical, fixable class: tables that a -tool can emit should be emitted by the tool. Raised for T03 and T06. diff --git a/history/260731-cost-accounting-retrospective.md b/history/260731-cost-accounting-retrospective.md deleted file mode 100644 index d160da8..0000000 --- a/history/260731-cost-accounting-retrospective.md +++ /dev/null @@ -1,112 +0,0 @@ -# 2026-07-31 — retrospective: what CB-WP-0002 taught the loop - -Pass: CB-WP-0002, cost accounting. Produced `tools/cb-cost.py`, -`specs/CostAccounting.md`, ADR-0003, CB-EV-0002, and InnerLoop **v1.1**. - -## The question the workplan asked - -> CB-WP-0001 produced a fully specified metric that could not be computed, -> and v1.0's "every metric names its instrument" rule was written to stop -> that. Did it? - -**No.** The rule was satisfied completely and the metric was still wrong. - -`specs/CostAccounting.md` AC-1 named `cb-cost --pin fc76445` as its -instrument — a command that did not exist when the row was written — and -set the target to **$92.87**, computed by hand. When the tool was built one -task later it returned **$93.32**. The hand computation carried a dedup bug -that the tool's own positive control caught on first contact. - -So the v1.0 rule stops "a metric with no instrument named" and does not stop -"a metric whose target the instrument never produced". The distinction did -not exist when the rule was written because the failure it was written for -was the coarser one. - -**v1.1's answer:** the instrument must exist and the target must come out of -it. Where the tool is built later in the pass, the target is `provisional:` -until the tool emits it, and the spec is amended to whatever it returns — -not the reverse. - -## The number, and its four corrections - -The cost of CB-WP-0001 was stated four times before it was right: - -| value | method | what was wrong | -|---|---|---| -| $248.46 | inherited, quoted | per-line summation (~1.9×) + single-model pricing | -| $92.21 | re-derived, deduped, per-model | omitted the subagent tree | -| $92.87 | + subagent at first-wins dedup | `output_tokens` is a running count | -| **$93.32** | emitted by `cb-cost` | — | - -Each correction was found by a different mechanism, and this is the useful -part: - -1. **$248.46 → $92.21** by *re-deriving instead of quoting*. Now a v1.1 - rule. -2. **$92.21 → $92.87** by *adversarial review*, which noticed the target - contradicted the survey's own blind-spot finding. Cost: $1.11. -3. **$92.87 → $93.32** by the *positive control*, which fired on real data - and refused to print a number. - -No single mechanism found more than one. That is the argument for keeping -all three rather than consolidating. - -## The error class the last pass predicted, and what actually happened - -CB-WP-0003 T10 predicted the next error would be the harness-does-nothing -class, since four instances had already been seen. It was not — twice. - -- The two errors in $248.46 were **trusted arithmetic over real data**. Both - sums ran over data that genuinely existed. A positive control asserting - "did this harness do work?" would have answered yes, correctly, and - reported a wrong number. -- The error in $92.87 was **a property verified on the large sample and - assumed on the small one**. The dedup invariant was checked on the main - transcript (206/206 groups) by the survey, and independently re-checked by - the adversarial reviewer, who also used the main transcript. It is false - in the 8-response subagent tree that neither examined separately. - -The second is the one worth carrying forward, because **review structurally -cannot catch it**: the reviewer's job is to re-derive the author's claims, -and re-deriving on the same sample reproduces the same blind spot. Only an -assertion running over *all* the data at execution time catches it. That is -now a v1.1 rule (`--self-test` on every reporting tool, run before the -number). - -## What the pass bought, in its own units - -The capability measures itself, which is the first time this project has -been able to say what a pass cost while the pass was running: - -- **CB-WP-0001 (measured retroactively):** $93.32 pinned at `fc76445`. - 88.0% cache, 11.9% output, 249:1 context re-read to text written. -- **Adversarial review, this pass:** $1.11 — ~1% of the pass it reviewed — - and it found three approval-blocking defects. Second consecutive pass - where that trade was decisive. Two data points now support CB-WP-0003 T03. -- **`/compact`, measured for the first time:** 542,991 → 19,974 tokens, a - 27× context reduction. Cost per turn fell 3.1×, from $0.457 across the - 136 turns before it to $0.149 across the 202 after. - -That last number changes advice this project was about to give itself. -CB-WP-0003 T04 was going to prescribe one task per session on the theory -that long sessions are quadratic. They are **bounded**-quadratic: cost grows -with context between compactions and resets at each one. The failure mode is -a long *uncompacted* session, and whether a fresh session beats a compaction -is now a measurable question rather than a matter of taste. T04 should -measure it before prescribing. - -## Raised, not resolved - -- **The price sheet cannot express a time-boxed rate.** Sonnet's intro price - is a TOML comment. Costs $0.17 today (0.19%); becomes a real error on - **2026-08-31** when the intro rate expires and the comment and the data - disagree in the other direction. -- **32.5% of spend has no task**, because 19 of 33 commits carry no `T##` - tag. Reported as its own line rather than hidden, but the underlying fix - is commit hygiene, not tooling. -- **Outside this repo:** the `ralph-workplan` skill instructs agents to read - token counts from the Claude Code status bar. That is not readable from a - tool call, so an agent asked for it estimates instead — the proximate - cause of the hub holding 401,100 tokens for a workplan that actually - consumed 80.9M. Flagged for the maintainer; `make cost` is the authority - in the meantime. diff --git a/research/CB-RES-0002-cost-accounting.md b/research/CB-RES-0002-cost-accounting.md deleted file mode 100644 index 188a5d3..0000000 --- a/research/CB-RES-0002-cost-accounting.md +++ /dev/null @@ -1,333 +0,0 @@ -# CB-RES-0002: agentic cost accounting - -capability: meta.loop.cost-accounting -status: approved # adversarial review 2026-07-31: 16 findings, 15 conceded -tier: L (structural L, chaos d10=2 → no override) -runnable-baseline: invoked — every candidate below was exercised against the -CB-WP-0001 session on this machine, not cited -review-trail: history/260731-cost-accounting-{research,challenge,response}.md - -Survey of instruments that can attribute the USD cost of agentic work to a -unit of work, so that M-D2-CST (`specs/MetricsAndScenarios.md` §1a) becomes -computable. CB-WP-0001 specified that metric completely and recorded it as -*uncomputable*; the premise of this workplan is that the data existed the -whole time. - -That premise survives. The workplan's **numbers do not** — see §Correction. - ---- - -## Correction to this workplan's own Purpose section - -CB-WP-0002's Purpose reports the CB-WP-0001 session at **$248.46**, from -131,863,164 cache-read tokens priced at Fable 5. Both halves are wrong, and -in the same direction — too high. The survey found this by re-deriving the -number rather than adopting it. - -**Error 1 — per-line summation double-counts.** A single API response is -written to the transcript as *several* JSONL lines, split by content block -(`thinking`, `text`, `tool_use`), and **every one of those lines repeats the -complete `usage` object**. Measured on the CB-WP-0001 transcript: 657 -assistant lines carry only 346 distinct `requestId`s. Group sizes run 1–6: - -| lines per requestId | 1 | 2 | 3 | 4 | 5 | 6 | total | -|---|---|---|---|---|---|---|---| -| groups | 140 | 120 | 74 | 6 | 5 | 1 | **346** | -| lines | 140 | 240 | 222 | 24 | 25 | 6 | **657** | - -Both rows are checksummed against the file: groups sum to 346, lines to 657. - -Positive control on the dedup: across all 206 multi-line groups the `usage` -object is byte-identical (206/206 identical, 0 differing), and no group -mixes models. Every assistant line carries a `requestId` — there is no null -key for `setdefault` to collapse. The duplication is a transcript-format -artifact, not repeated billing. Summing per line inflates by ≈1.9×. - -**Error 2 — single-model pricing on a multi-model session.** The session ran -three models, not one. Both columns are shown because the gap between them -*is* error 1: - -| model | JSONL lines | API responses (deduped) | pinned cost | -|---|---|---|---| -| claude-opus-5 | 382 | 213 | $36.19 (39%) | -| claude-fable-5 | 250 | 118 | **$55.50 (60%)** | -| claude-sonnet-5 | 24 | 14 | $0.52 (0.6%) | -| `` | 1 | 1 | $0.00 | -| **total** | **657** | **346** | | - -§1a *already required* per-model pricing; the Purpose section did not apply -its own rule. Note the inversion, which matters more than the correction: -**Fable is 35% of the calls and 60% of the dollars; Opus is 60% of the calls -and 39% of the dollars.** A count-majority is not a cost-majority — the -error this document is about, one level down. - -The `` entry is an error placeholder carrying a valid `requestId` -and a complete **all-zero** `usage` object, not a missing one. Its dollar -impact is exactly $0.00, but a collector guarding on `if not usage` and one -guarding on `if model not in prices` take different branches; T04 must name -which. - -**Corrected totals** for the same transcript, all three methods run over -the identical unpinned line set so the methods are comparable: - -| method | responses | output | cache read | cost | -|---|---|---|---|---| -| per-line, all-Fable (the Purpose method) | 657 | 700,690 | 161,408,840 | $289.12 | -| per-line, per-model | 657 | 700,690 | 161,408,840 | $210.05 | -| **deduped, per-model (correct)** | **346** | **318,230** | **81,100,498** | **$93.15** | - -**Third error, found while verifying the second: the transcript is a live -file.** Re-running the deduped figure minutes later returned 356 -responses and $94.04 — this survey's own session appends to the same -JSONL it is measuring. An unpinned total is not a repeatable number. The -acceptance target is therefore pinned by timestamp: - -| CB-WP-0001, pinned ≤ `2026-07-31T02:17:59Z` (commit `fc76445`) | value | -|---|---| -| responses | 339 (206 opus-5, 118 fable-5, 14 sonnet-5, 1 synthetic) | -| output | 313,900 tok → $10.66 | -| cache read | 80,453,702 tok → $59.59 | -| cache write 1h | 1,672,854 tok → $21.95 | -| input | 676 tok → $0.00 | -| main transcript | **$92.21** — 88.4% cache, 256:1 cache-read:output | -| + subagent tree (7 responses, ran 23:11–23:14Z, inside the pin) | $1.11 | -| **TRUE TOTAL** | **$93.32** | - -*(The subagent figure was $0.66 when this survey was written. T04's -positive control found the cause: `output_tokens` is a running count in -the `subagents/` tree, so first-wins dedup under-counted it. See -`specs/CostAccounting.md` CA-02a.)* - -The subagent line is not a footnote. C1's blind-spot finding says a -collector reading only the main file under-reports; a target of $92.21 would -have been hit only by a collector *with* that blind spot, and failed by a -correct one. The acceptance target is **$93.32, stated as its two -components**, so a collector reading one tree is diagnosed rather than -merely failed. - -The reported figure was **~2.7× the real cost**. This is the fourth -instance of the harness-does-nothing error class from -`history/260731-inner-loop-retrospective.md`, wearing a new coat: not a -harness that measured nothing, but an arithmetic that measured the same -thing twice. Both produce a number that looks fine. - -The qualitative headline survives the correction and gets stronger: cache -reads are **81.1M tokens against 318k of output**, ~255:1. Cost in an -agentic loop is context × turns. - ---- - -## Candidates - -### C1 — Session transcript JSONL - -`~/.claude/projects//.jsonl`, one JSON object per line. -Assistant lines carry `message.usage` with exact billing counters: -`input_tokens`, `output_tokens`, `cache_read_input_tokens`, and -`cache_creation.{ephemeral_1h,ephemeral_5m}_input_tokens`, plus -`message.model`, `requestId`, and an ISO-8601 `timestamp`. - -- **Granularity:** per API response, once deduplicated by `requestId`. -- **Accuracy:** exact — these are the counters the invoice is computed from. - There is no sampling or rounding. -- **Verified non-issue:** `usage.iterations[]` is a sub-breakdown, not an - additional charge. Every assistant line carries `usage`; the iteration - outputs sum exactly to the top-level `output_tokens` in every case, and - no message had more than one iteration. Summing `iterations` *instead of* - the top-level fields is safe; summing *both* would double-count. -- **Cache-write rates are per-TTL and must not be aggregated.** - `cache_creation_input_tokens` equals `ephemeral_5m + ephemeral_1h` in all - 775 usage lines across both transcripts, and the two bill at different - multipliers (1.25 vs 2.0). Pricing the top-level aggregate at a single - rate is a silent error: the subagent's writes are **entirely 5m** (37,467 - tokens), and pricing them at 1h inflates that transcript by **+43%** - ($0.658 → $0.939). The main session happens to be all-1h, so the pinned - total is insensitive — but fan-out passes are exactly where 5m dominates. -- **Survives compaction:** yes. `/compact` writes a summary message into the - same file (`isCompactSummary`, `compactMetadata`) and the session - continues; no usage is lost. Compaction is visible as an event, so its - cost is itself measurable. -- **Attribution:** none built in — a transcript is a flat message stream - with timestamps. It must be joined against an external time index. -- **Blind spot 1 — subagents are a separate tree.** Subagent cost is **not** - in the main transcript. `isSidechain` is `false` on all 657 lines; - subagent work lives in `/subagents/agent-*.jsonl` with an - `agent-*.meta.json` naming the agentType and model. CB-WP-0001 spawned one - (the adversarial review). A collector reading only the main file silently - under-reports. -- **Blind spot 2 — one repo, several transcripts, overlapping in time.** - The project directory holds more than one session. `f1eb1147` overlaps - `8cbd5701` for **4 h 13 m**, carrying ~$6 on each side — ~$12 that no - wall-clock join can separate, beginning 14 seconds after the pin. A - collector must therefore key attribution on **`sessionId`, not only time**, - and must enumerate every transcript for the repo rather than one file. - -### C2 — Custodian State Hub token API - -`record_token_event`, the `update_task_status` token tiers, and -`get_token_summary`. Exercised against CB-WP-0001's workplan -(`a1b434dc-…`), which returned: - -```text -tokens_in 362,000 tokens_out 39,100 event_count 7 by model: claude-fable-5 -``` - -- **Granularity:** per task — the best of any candidate, and the only one - that is natively *about* the unit of work. -- **Accuracy:** poor, and structurally so. Three independent defects: - 1. **The schema has no cache fields.** `tokens_in`/`tokens_out` cannot - represent the finding this workplan exists to report. Cache read alone - is **64.6% of cost** and all cache is **88.4%**; the hub cannot express - either at any fidelity. - 2. **The recorded numbers are estimates.** 7 events for 9 tasks, at - round figures — the skill's Tier-3 heuristic (1000/500) and Tier-1 - eyeball estimates. Against a deduped transcript output of 318,230, - the hub's 39,100 is off by ~8×; against total input it is off by - ~450×. - 3. **Model attribution is wrong.** Everything is filed under - `claude-fable-5` on a session that was majority Opus 5. -- **Survives compaction:** yes — it is server-side and independent of the - client. -- **Verdict:** durable and task-shaped, but its numbers are unusable as a - cost source. Its role is as a **sink** for numbers computed elsewhere, - not a source. Even as a sink it can only carry a lossy projection until - the schema grows cache fields. - -### C3 — Claude Code status bar - -- **Granularity:** whole session, live. -- **Accuracy:** unknown and unauditable — it is rendered text. -- **Machine-readable:** no. Not configured here (`statusLine` is absent - from `~/.claude/settings.json`), and it is not reachable from inside a - tool call regardless. -- **Verdict:** eliminated. The ralph-workplan skill's "read tokens from the - status bar" instruction is the proximate cause of C2's bad numbers — it - asks an agent to report a figure it cannot read, and an agent that cannot - read it estimates instead. This should be raised against the skill. - -### C4 — Anthropic usage / billing API - -- **Granularity:** organization and API-key, by day. -- **Accuracy:** authoritative — it *is* the invoice. -- **Attribution:** none to a task, and none to a session. Cannot separate - clay-borg from the other twenty-plus projects on this machine. -- **Availability:** requires an admin key; none is configured here. -- **Verdict:** not usable for M-D2-CST, but valuable as an **external - reconciliation check** if an admin key is ever provisioned — it is the - only candidate that can catch a systematic error in C1's price model. - Left as a stated non-dependency. - -### C5 — Git commit history (attribution index, not a cost source) - -Not a cost instrument; the missing half of C1. The loop already commits per -task iteration with the task in the subject line, giving durable, timestamped -boundaries at exactly the granularity M-D2-CST wants: - -```text -a09d76f 2026-07-31T02:14:34+02:00 T08 iter 1: scenario runner executes; … -b58a913 2026-07-31T02:19:34+02:00 T08 iter 2: Reveal, Resolve, End; … -``` - -- **Interval convention:** *ending-at-commit*, `(prev_commit, this_commit]`. - This is the only convention consistent with commit-at-end-of-task; under - the alternative every task's cost shifts one interval. -- Spacing over the 33 pinned commits: min 0.1, p50 **6.9**, p90 17.7, max - **36.8** minutes. Usually finer than a task; the 37-minute max is not. -- Durable, versioned, and free; requires no change to how work is done. -- **Coverage is the real limit, and it is sized:** only **14 of 33** commits - name a task (`T##`) in the subject. The other 19 — `chore(consistency)`, - `CI:`, `AM-4:`, workplan additions — hold **$30.32 of $92.21, or 33% of - cost**. Attribution to a *task* therefore covers two-thirds of spend at - best; the remainder is real work that must be reported as its own line, - not discarded. -- **Boundary conditions verified clean:** 0 responses before the first - commit, 0 after the last, exactly 1 empty interval (`Initial commit`). -- **Known hazards:** offsets are not uniform — `git log` shows `+02:00` ×34 - and `+00:00` ×1, so a collector must parse `%cI` and convert, never - subtract a fixed offset. Commits made outside a session create empty - intervals. - ---- - -## Baselines (benchmark-to-beat) - -| Dimension | Baseline holder | Metric | Value | Provenance | -|---|---|---|---|---| -| D1 ease of specification | C2 hub | fields needed to record a task's cost | 4 (`task_id`, `tokens_in`, `tokens_out`, `model`) — but cannot express cache | measured (API schema) | -| D2 efficiency | C2 hub | cost of producing a number | ~0 (one API call) — number is an estimate, off by ~8× on output | measured | -| D2 efficiency | C1 transcript | cost of producing a number | one file read, 5.1 MB, ~1 s; exact | measured | -| D3 speed | C1 transcript | parse of a full session | 2,040 lines / 5.1 MB in <1 s in CPython | measured | -| D2 efficiency (**the deciding row**) | C1 transcript | **error against the billing counters** | **$0.00 — the transcript *is* the counter set; C2's error on the same work is $92.21 − $0.03 recorded ≈ 100%** | measured | -| D4 optionality | C4 billing API | authority | invoice-grade, zero attribution, admin key absent (asserted, not measured — used only to keep C4 as an optional check) | availability measured | - -**Benchmark-to-beat for the collector:** reproduce **$93.32** for repo -`clay-borg` pinned at `2026-07-31T02:17:59Z` — as its two components, -**$92.21 main transcript + $1.11 subagent tree** — from the committed price -sheet, with the unattributed remainder reported as its own line (expected -**33%**, $30.32) and reconciliation asserted rather than assumed. - ---- - -## Verdict - -**C1 (transcript) leads on accuracy and is the only exact candidate.** -**C5 (git commits) supplies the attribution index C1 lacks.** C2 is the -durable sink. C3 is eliminated. C4 is an optional external check. - -The expected shape is therefore: enumerate **every** transcript for the repo -including the `subagents/` tree → dedup by `requestId` → price per message at -its own model's rate and **per cache TTL** from -`benchmarks/baselines/model-prices.toml` → attribute to a task by joining -`(prev_commit, this_commit]` intervals **within a `sessionId`** → emit -per-task cost, an unattributed-remainder line, and a composition breakdown → -push a lossy summary to C2. - -**What none of them do well — the surpass opportunity.** Every candidate -reports *totals*. None reports **composition**, and composition is where -the actionable finding lives: 81.1M cache-read tokens against 318k of -output means cost is driven by how much context is re-read per turn, which -no total can show. A metric that had reported only dollars would have been -correct and useless. - -**Risks in the baselines themselves.** - -1. **The $248.46 figure was wrong and was nearly adopted as this - workplan's acceptance target.** T05's reconciliation test must be - against a number this survey re-derived, not against the Purpose - section. The Purpose section needs correcting. -2. **Dedup is load-bearing.** If the transcript format ever splits one - response across two `requestId`s, dedup silently under-reports — - the opposite error, and the more dangerous one. The collector must - assert its dedup assumption (identical usage within a group) at - runtime rather than trusting this survey's one-time check. -3. **Subagent transcripts are a separate tree.** Measured: CB-WP-0001's - one subagent (adversarial review, Fable 5, 7 responses, 158,096 cache - reads) cost **$1.11**, invisible to any collector reading only the - main file. Small here; not small for a pass that fans out. -4. **The price sheet has a 90-day staleness rule** (§1a) and no automated - check. Every number this capability produces inherits that. -5. **The transcript is append-live.** It is written by the session that - reads it, so any total is a reading at an instant. Every committed - number from this capability states its pin (timestamp or commit), and - the collector takes a pin argument rather than defaulting to "all". -6. **Concurrent sessions are not hypothetical — they are in this data.** - `f1eb1147` and `8cbd5701` overlap for 4 h 13 m on the same repo, ~$12 - inseparable by wall-clock. (The session that wrote the first draft of - this sentence *was* the overlap.) Attribution keys on `sessionId` first, - time second. Compaction inside a task is safe — it stays in one file and - one session. -7. **A third of spend has no task, structurally.** 19 of 33 commits carry no - task tag; $30.32 of $92.21. Any per-task cost table is a view over ~two - thirds of the money, and must say so wherever it is reported — the same - limit-with-the-number rule the coverage gate carries. -8. **The price sheet cannot express a time-boxed rate.** Sonnet's intro - price lives in a TOML *comment*, so a collector reading the sheet - silently uses the wrong number ($0.17 at the pin, 0.19%). Small now, and - the same class of defect this survey levels at C2: a schema that cannot - hold the fact it needs. Raised for T03. -9. **Hand-typed tables are the actual failure surface.** Every computed - figure in this survey reproduced to the cent under adversarial - re-derivation; three hand-written markdown tables did not (a group-size - row that failed its own checksum, a per-line count labelled as deduped, - a stale 87%). Numbers a tool can emit should be emitted by the tool. - Raised for T03 and T06. diff --git a/specs/CostAccounting.md b/specs/CostAccounting.md deleted file mode 100644 index c22530c..0000000 --- a/specs/CostAccounting.md +++ /dev/null @@ -1,182 +0,0 @@ -# Cost Accounting - -Status: **v1.0** — 2026-07-31. Derived from -[CB-RES-0002](../research/CB-RES-0002-cost-accounting.md) (approved) and -[ADR-0003](../decisions/ADR-0003-cost-accounting.md). Makes M-D2-CST -computable; supersedes its "uncomputable" disposition in CB-EV-0001. - -Defines how the USD cost of agentic work is measured and attributed, so -that D2 claims about implementation efficiency are falsifiable. - ---- - -## 1. The cost model - -### 1.1 Unit of billing - -The unit is one **API response**, identified by `requestId`. It is *not* -one JSONL line: a response is written as up to six lines split by content -block (`thinking`, `text`, `tool_use`), and every line repeats the same -`usage` object. - -> **CA-01.** Cost is computed over responses deduplicated by `requestId`. -> Summing per line is a defect; it inflates by ≈1.9× on measured data. - -> **CA-02.** Dedup is asserted, not assumed. Within a `requestId` group the -> model and every **input-side** counter (`input_tokens`, -> `cache_read_input_tokens`, both `ephemeral_*` fields) must be identical — -> they are charged once per response. A divergence aborts the run. - -> **CA-02a.** `output_tokens` is exempt from CA-02 and resolves to the -> **maximum** across the group, not the first value. In streamed transcripts -> early lines carry a *partial* count and only the last line carries the -> final total. - -Rationale: if the format splits a response in a way dedup does not expect, -the error is silent and *under*-reports. The dangerous direction gets the -assertion. - -*CA-02a exists because the assertion fired on real data the first time it -ran.* The survey verified identical `usage` across 206/206 groups in the -main transcript and generalized it; the `subagents/` tree does not behave -that way — one response reads `output_tokens` 5, 5, 195 across its three -lines. First-wins scored it at 5. That error moved the acceptance target -by $0.45. - -### 1.2 Price formula - -Per response, against `benchmarks/baselines/model-prices.toml`: - -```text -cost = input_tokens × price.input - + output_tokens × price.output - + cache_read_input_tokens × price.input × cache.read (0.10) - + ephemeral_5m_input_tokens × price.input × cache.write_5m (1.25) - + ephemeral_1h_input_tokens × price.input × cache.write_1h (2.00) -``` - -> **CA-03.** Each response is priced at **its own** `message.model` rate. A -> session may mix models; CB-WP-0001 used three. - -> **CA-04.** Cache writes are priced **per TTL**. The top-level -> `cache_creation_input_tokens` aggregate equals `ephemeral_5m + -> ephemeral_1h` and must never be priced at a single multiplier — doing so -> inflated a measured subagent transcript by 43%. - -> **CA-05.** A response whose model is absent from the price sheet is -> reported as an unpriced line with its token counts, never dropped and -> never priced at a default. - -### 1.3 Scope of a measurement - -> **CA-06.** A measurement enumerates **every** transcript for the repo: -> `~/.claude/projects//*.jsonl` and every -> `/subagents/agent-*.jsonl`. Subagent cost is not in the main -> file and is invisible to a collector that reads one path. - -> **CA-07.** Every committed number states its **pin** — a timestamp or -> commit. Transcripts are append-live: the file grows as the measuring -> session writes to it, and an unpinned total is not reproducible. - -## 2. The attribution contract - -> **CA-08.** A response is attributed to the task named by the next commit -> at or after it, within its own session: -> `interval := (prev_commit_time, this_commit_time]`, scoped by `sessionId`. - -> **CA-09.** Timestamps are converted, never offset-subtracted. Commit -> times are parsed from `%cI` and converted to UTC; this repo carries two -> distinct offsets. - -> **CA-10.** A response in a commit whose subject carries no `T##` tag is -> attributed to `UNATTRIBUTED`, which is **reported as its own line** in -> every table. On CB-WP-0001 this is 33% of cost ($30.32 of $92.21) — a -> per-task table is a view over roughly two-thirds of the money and says so -> wherever it appears. - -> **CA-11.** Cost after the last commit is an **open remainder**, reported -> separately from `UNATTRIBUTED`. It is work not yet committed, not work -> without a task. - -> **CA-12.** Attribution never keys on wall-clock alone. Two sessions -> overlapped 4 h 13 m on this repo carrying ~$12; only `sessionId` -> separates them. - -## 3. Reported shape - -> **CA-13.** Every report carries **composition** — the split across input, -> output, cache read, cache write 5m, cache write 1h — alongside the total. -> A total alone would have concealed the finding that motivated this work: -> 88.4% of spend is cache, at 256:1 cache-read to output tokens. - -> **CA-14.** Reconciliation is asserted. Attributed + unattributed + open -> remainder + unpriced must equal the transcript total to the cent. A -> mismatch aborts rather than reporting. - -> **CA-15.** Tables that a tool can emit are emitted by the tool. Every -> committed number in this capability's evidence is produced by a command, -> not typed. *(Adversarial review of CB-RES-0002 found every computed -> figure correct to the cent and three hand-typed markdown tables wrong.)* - -## 4. Acceptance metrics - -Each row names the command that produces its number, per InnerLoop §Step 4. -`cb-cost` is `tools/cb-cost` (T04). - -| ID | Metric | Target | Instrument | -|---|---|---|---| -| **AC-1** | reproduces the pinned CB-WP-0001 total | **$93.32** = $92.21 main + $1.11 subagent | `make cost-pin` | -| **AC-2** | reconciliation residual (CA-14) | **$0.00** exactly | same command, `reconciled: ok` line | -| **AC-3** | unattributed share reported (CA-10) | present, and **33%** on the pinned run | `cb-cost --pin fc76445 --by-task` | -| **AC-4** | composition reported (CA-13) | all five components present | `cb-cost --pin fc76445 --composition` | -| **AC-5** | dedup invariant asserted (CA-02) | violation exits non-zero | `make cost-test` | -| **AC-6** | positive control: refuses to report on zero responses | exits non-zero | `make cost-test` | -| **AC-7** | subagent tree included (CA-06) | omitting it changes AC-1 by $1.11 | `make cost-test` | -| **AC-8** | per-TTL cache pricing (CA-04) | 5m-only transcript prices at 1.25× | `make cost-test` | -| **AC-9** | streamed partial output resolves to final (CA-02a) | 5,5,195 → 195, not 5 | `make cost-test` | - -**AC-5 through AC-8 are the positive control.** Per InnerLoop v1.0 §Step 5, -a harness must assert it did the work it reports. `--self-test` runs each -assertion against a fixture whose expected value is known and fails loudly; -`make cost` runs it before any reported number. - -## 5. Metric feasibility check - -Per InnerLoop §Step 4, the acceptance table is checked against the -contracts in this same spec: - -- AC-1's $92.87 is reachable only if CA-06 holds (both trees enumerated). - Under a main-file-only collector the target is unreachable — this is the - defect the adversarial review caught, where a target of $92.21 would have - been hit *only* by a broken collector. -- AC-3's 33% is a property of CB-WP-0001's commit subjects, not of the - collector. It is a regression pin on the fixture, not a quality target; - improving tagging discipline will change it, and that is expected. -- CA-07 (pinning) makes AC-1 reproducible; without it the target drifts - upward on every run and the test is meaningless. - -## 6. Known limitations, stated with the numbers - -- **Per-task cost covers ~67% of spend.** Structural: 19 of 33 commits - carry no task tag. Reported per CA-10, never silently dropped. -- **Work spanning a commit is assigned whole to the later task.** Bounded - by one interval: p50 6.9 min, p90 17.7 min, max 36.8 min. -- **The price sheet cannot express a time-boxed rate.** Sonnet's intro - price is a TOML comment, so the sheet is silently wrong for sonnet-priced - work ($0.17 at the pin, 0.19%). **This becomes an error, not a rounding - issue, on 2026-08-31** when the intro rate expires and the comment and - the data disagree in the other direction. Tracked as a schema defect - against the price sheet. -- **The State Hub cannot store what this spec measures.** Its token event - schema has `tokens_in`/`tokens_out` and no cache fields, so the dashboard - necessarily shows a lossy projection. This is a limitation of the sink, - not of the metric; `make cost` remains the authority. - -## 7. Revisions to M-D2-CST - -`specs/MetricsAndScenarios.md` §1a is superseded by this spec. M-D2-CST is -redefined from "tokens × pricepoint" — which named no instrument and was -never computed — to: **USD per completed workplan task, per CA-08 -attribution, produced by `make cost`.** M-D2-TOK is retained but demoted: -tokens are the input to the cost model, not a comparable figure across -models or across cache states. diff --git a/specs/InnerLoop.md b/specs/InnerLoop.md index 178bd2a..e5c35a2 100644 --- a/specs/InnerLoop.md +++ b/specs/InnerLoop.md @@ -1,17 +1,11 @@ # The Inner Loop — Assimilate and Surpass -Status: **v1.1** — corrected from CB-WP-0002 (cost accounting) on -2026-07-31. Changes from v1.0: the instrument must exist and emit its own -target; inherited numbers are re-derived before use; every reporting tool -exposes `--self-test`; cost is in the definition of done. Rationale: -`history/260731-cost-accounting-retrospective.md`. - -v1.0 — survived its first full pass (CB-WP-0001, the GROUND game kernel) -and was corrected from it on 2026-07-31. Changes from v0.2: measurement -validity (the positive control), metric feasibility and instrument naming, -four implementation rules the pass earned, and the requirement that -evidence state what it does not support. Rationale and the failures behind -each: `history/260731-inner-loop-retrospective.md`. +Status: **v1.0** — survived its first full pass (CB-WP-0001, the GROUND +game kernel) and was corrected from it on 2026-07-31. Changes from v0.2: +measurement validity (the positive control), metric feasibility and +instrument naming, four implementation rules the pass earned, and the +requirement that evidence state what it does not support. Rationale and +the failures behind each: `history/260731-inner-loop-retrospective.md`. Normative process for building every Clay-Borg capability. Referenced by all workplans. The loop's own optimization target is **agentic efficiency**: @@ -128,30 +122,7 @@ research step (metric provenance). **Every metric names its instrument, and is checked reachable.** A row in the acceptance table carries the command that produces its number. -A metric with no named instrument is a wish, not a metric. - -**The instrument must exist, and the target must come out of it.** -Naming a command is not the same as running one. A target computed by -hand and merely *labelled* with a command is the same defect the rule -was written to stop, one level down. Where the instrument is built later -in the pass, the target is marked `provisional:` until the instrument -emits it, and the spec is amended to whatever the instrument returns. - -*(v1.1, from CB-WP-0002: `specs/CostAccounting.md` AC-1 named -`cb-cost --pin fc76445` before that tool existed, and set the target to -a hand-computed $92.87. When the tool was built it returned $93.32 — -the hand computation carried a dedup bug the tool's own positive control -caught. The metric satisfied v1.0's rule completely and was still -wrong.)* - -**A number inherited from earlier work is re-derived before it is used -as a target, or it is cited as unverified.** Quoting is not measuring. - -*(v1.1, from CB-WP-0002: the workplan opened with $248.46, inherited -from a prior pass. Re-derivation put it at $92.21 — the quoted figure -double-counted transcript lines and priced a three-model session at one -model's rate. Neither error was of the harness-does-nothing class; both -sums ran over real data, and a positive control would have passed them.)* A metric must +A metric with no named instrument is a wish, not a metric. A metric must also be checked against the contracts in the *same spec*: if a contract makes a target unreachable, one of the two is wrong and the conflict is resolved when it is noticed, not at the acceptance run. Re-check the @@ -193,19 +164,6 @@ Concretely, a measurement harness must, on every run: **A number from a run that cannot prove it did the work is void** and must not reach an evidence file. -**Every tool that reports a number exposes `--self-test`**, and that -self-test runs before the number is produced (`make cost` depends on -`make cost-test`). The assertion must name a failure it detects, not -merely exercise the happy path. - -*(v1.0+, from CB-WP-0002: `cb-cost`'s dedup assertion fired on its first -run against real data and aborted, catching a rule that was verified on -206/206 groups of the main transcript and false in the 8-response -subagent tree. The generalization that failed — a property confirmed on -the largest sample assumed to hold on the smallest — is not one review -catches, because both the survey and the adversarial reviewer checked -the same large sample.)* - *(v1.0, from CB-WP-0001: both serious errors in the first pass were of exactly this shape. A JS harness reported 8.4s for 100k moves while every move was being rejected, and a Rust benchmark reported 9.3M @@ -328,11 +286,6 @@ A capability has completed the loop when all of the following are committed: - [ ] every unmet metric reported as unmet, with attribution and the options for resolving it — a missed target is an output of the loop, not a reason to move the target quietly -- [ ] **cost recorded**: `make cost` run for the pass, its composition - (not only its total) in the evidence file, and the per-task figures - pushed to the hub. M-D2-CST is no longer allowed to be - `uncomputable` — the instrument exists - ([CostAccounting.md](CostAccounting.md)) - [ ] retrospective note (may be one paragraph appended to the evidence file): what the loop itself should change ``` diff --git a/specs/MetricsAndScenarios.md b/specs/MetricsAndScenarios.md index c8b360f..d7ac590 100644 --- a/specs/MetricsAndScenarios.md +++ b/specs/MetricsAndScenarios.md @@ -35,8 +35,8 @@ and add capability-specific rows only when these don't cover the claim. | M-D2-LOC | D2 | source LOC excluding tests (tokei) | lines | adopted:tokei | | M-D2-DEP | D2 | transitive dependency count (cargo tree) | crates | adopted:cargo-deny practice | | M-D2-BLD | D2 | clean build / incremental build time | s | adopted:cargo timing | -| M-D2-TOK | D2 | tokens consumed per completed workplan task | tokens | novel — **demoted 2026-07-31**: an input to the cost model, not comparable across models or cache states | -| M-D2-CST | D2 | **cost** per completed workplan task, attributed per CA-08 | USD | adapted:anthropic-pricing — instrument: `make cost`; normative spec [CostAccounting.md](CostAccounting.md) | +| M-D2-TOK | D2 | tokens consumed per completed workplan task | tokens | novel — agentic-efficiency core metric; recorded per task in evidence | +| M-D2-CST | D2 | **cost** per completed workplan task: tokens × the executing model's pricepoint | USD | adapted:anthropic-pricing — tokens alone mislead once models with different prices mix; see §1a | | M-D3-THR | D3 | events applied per second, headless replay | events/s | adapted:criterion (throughput mode) | | M-D3-LAT | D3 | p99 command→state-applied latency | µs | adopted:criterion | | M-D3-MEM | D3 | peak resident memory during benchmark scenario | MB | adopted:/usr/bin/time -v | @@ -46,17 +46,6 @@ and add capability-specific rows only when these don't cover the claim. ### 1a. Token cost accounting (M-D2-CST) -> **Superseded 2026-07-31 by [CostAccounting.md](CostAccounting.md)**, which -> is normative for the cost model, attribution, and acceptance metrics. -> This section is retained for the price-sheet location and the -> quality-gate rule; where the two disagree, CostAccounting.md wins. -> -> What changed and why: the definition below named no instrument and was -> never computed, so CB-WP-0001 recorded M-D2-CST as *uncomputable* while -> the data sat in the session transcripts. Three of its rules were also -> wrong in ways that cost real money to discover — see the corrections -> inline below. - Token counts are only comparable at a single pricepoint. Since work moves between models (Fable for demanding passes, Sonnet/Opus for routine ones), every task's token record carries the **model** it ran on, and cost is @@ -92,17 +81,13 @@ write_1h = 2.0 Rules: -- ~~`cost = (in_tokens × input + out_tokens × output) / 1e6` … If cache - split is unknown, count all input at full price and note it — cost is then - an upper bound.~~ **Corrected:** the cache split is never unknown; it is - in every transcript. Treating it as unknown would have priced 80.5M cache - reads at 10× their rate. See CostAccounting.md §1.2 (CA-03, CA-04) — cache - writes bill at two different TTL rates and must not be aggregated. -- ~~The state-hub task close (`update_task_status`) already records tokens - and `model`.~~ **Corrected:** the hub schema has no cache fields and - cannot represent 88% of spend, and the figures it recorded for CB-WP-0001 - were estimates in error by ~100%. The hub is a **sink** for numbers - computed by `make cost`, never a source. See CostAccounting.md §6. +- `cost = (in_tokens × input + out_tokens × output) / 1e6`, using the sheet + in force at the time the work ran; cache reads/writes, when known, use the + multipliers. If cache split is unknown, count all input at full price and + note it — cost is then an upper bound. +- The state-hub task close (`update_task_status`) already records tokens and + `model`; the evidence file's task log adds the computed USD figure so + cross-model comparisons are honest. - **Cheaper is only better at equal quality**: M-D2-CST verdicts are valid only alongside passing scenarios/metrics from the same run — a cheap failed pass scores nothing. diff --git a/tools/__pycache__/cb-cost.cpython-312.pyc b/tools/__pycache__/cb-cost.cpython-312.pyc deleted file mode 100644 index 12efe0a..0000000 Binary files a/tools/__pycache__/cb-cost.cpython-312.pyc and /dev/null differ diff --git a/tools/cb-cost.py b/tools/cb-cost.py deleted file mode 100644 index 14db4ac..0000000 --- a/tools/cb-cost.py +++ /dev/null @@ -1,470 +0,0 @@ -#!/usr/bin/env python3 -"""M-D2-CST: USD cost of agentic work, attributed to workplan tasks. - -Normative spec: specs/CostAccounting.md. Decision: decisions/ADR-0003. - -Reads Claude Code session transcripts, deduplicates by requestId, prices -each response at its own model's rate and its own cache TTL, and attributes -it to the task named by the next commit at or after it within the same -session. - -Positive control (InnerLoop v1.0 §Step 5, CostAccounting CA-02/CA-14): -this tool asserts it did the work it reports. `--self-test` exercises four -assertions against fixtures with known answers; every reporting run checks -the dedup invariant and reconciles attributed + unattributed + open + -unpriced against the raw total, aborting rather than printing a number that -does not add up. - -Usage: - python3 tools/cb-cost.py # whole repo, no pin - python3 tools/cb-cost.py --pin fc76445 # pinned at a commit - python3 tools/cb-cost.py --by-task # per-task attribution - python3 tools/cb-cost.py --composition # cost split by component - python3 tools/cb-cost.py --self-test # positive control - python3 tools/cb-cost.py --json -""" - -import argparse -import collections -import datetime -import glob -import json -import os -import re -import subprocess -import sys - -try: - import tomllib -except ModuleNotFoundError: # pragma: no cover - py<3.11 - print("ERROR: needs Python 3.11+ for tomllib", file=sys.stderr) - sys.exit(1) - -REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) -PRICES = os.path.join(REPO, "benchmarks", "baselines", "model-prices.toml") -COMPONENTS = ("input", "output", "cache_read", "write_5m", "write_1h") -TASK_RE = re.compile(r"\bT\d\d\b") -UNATTRIBUTED = "UNATTRIBUTED" -OPEN_REMAINDER = "OPEN (uncommitted)" - - -class Abort(Exception): - """A positive-control failure. Never degrades to a printed number.""" - - -# ---------------------------------------------------------------- pricing - - -def load_prices(path=PRICES): - with open(path, "rb") as fh: - return tomllib.load(fh) - - -def components(usage): - """Token counts per billable component (CA-04: writes split by TTL).""" - cc = usage.get("cache_creation") or {} - return { - "input": usage.get("input_tokens", 0), - "output": usage.get("output_tokens", 0), - "cache_read": usage.get("cache_read_input_tokens", 0), - "write_5m": cc.get("ephemeral_5m_input_tokens", 0), - "write_1h": cc.get("ephemeral_1h_input_tokens", 0), - } - - -def price_of(prices, model, toks): - """USD for one response. Returns None when the model is unpriced (CA-05).""" - pr = prices.get(model) - if not pr: - return None - cache = prices["cache"] - unit = pr["input"] / 1e6 - return ( - toks["input"] * unit - + toks["output"] * pr["output"] / 1e6 - + toks["cache_read"] * unit * cache["read"] - + toks["write_5m"] * unit * cache["write_5m"] - + toks["write_1h"] * unit * cache["write_1h"] - ) - - -# ------------------------------------------------------------- transcripts - - -def transcript_paths(slug): - """Every transcript for the repo, including the subagent tree (CA-06).""" - base = os.path.expanduser(f"~/.claude/projects/{slug}") - return sorted(glob.glob(f"{base}/*.jsonl")) + sorted( - glob.glob(f"{base}/*/subagents/agent-*.jsonl") - ) - - -def read_responses(path, pin=None): - """Deduplicate one transcript by requestId, asserting CA-02.""" - groups = collections.defaultdict(list) - for line in open(path): - try: - d = json.loads(line) - except json.JSONDecodeError: - continue - if d.get("type") != "assistant": - continue - usage = (d.get("message") or {}).get("usage") - if usage is None: - continue - ts = d.get("timestamp") or "" - if pin and ts > pin: - continue - groups[d.get("requestId")].append(d) - - out = [] - for rid, rows in groups.items(): - # CA-02 positive control. Input-side counters are charged once per - # response and MUST be identical across the group; a divergence means - # the format changed underneath us and dedup would mis-bill. - # - # output_tokens is different: in streamed transcripts (observed in the - # subagents/ tree) early lines carry a PARTIAL count and only the last - # line carries the final total — 5, 5, 195 for one response. Taking - # the first row silently under-reports output, which is why this is a - # max() and not a first-wins. - seen_model = {r["message"].get("model") for r in rows} - if len(seen_model) != 1: - raise Abort( - f"{os.path.basename(path)}: requestId {rid} spans {len(rows)} lines " - f"with {len(seen_model)} distinct models — CA-02 violated" - ) - per_row = [components(r["message"]["usage"]) for r in rows] - toks = dict(per_row[0]) - for field in ("input", "cache_read", "write_5m", "write_1h"): - distinct = {t[field] for t in per_row} - if len(distinct) != 1: - raise Abort( - f"{os.path.basename(path)}: requestId {rid} has {len(distinct)} " - f"distinct values for {field} across {len(rows)} lines " - f"({sorted(distinct)}) — CA-02 violated; input-side counters " - f"are charged once per response and must not vary" - ) - toks["output"] = max(t["output"] for t in per_row) - head = rows[0] - out.append( - { - "request_id": rid, - "model": head["message"].get("model"), - "timestamp": head.get("timestamp") or "", - "session": head.get("sessionId") or os.path.basename(path), - "toks": toks, - "subagent": "/subagents/" in path, - } - ) - return out - - -# ------------------------------------------------------------- attribution - - -def commit_index(pin=None): - """(utc_time, subject) for each commit, oldest first (CA-09).""" - fmt = subprocess.run( - ["git", "-C", REPO, "log", "--format=%cI|%s", "--reverse"], - capture_output=True, - text=True, - check=True, - ).stdout.splitlines() - rows = [] - for line in fmt: - iso, _, subject = line.partition("|") - # CA-09: convert, never subtract a fixed offset — this repo has two. - utc = ( - datetime.datetime.fromisoformat(iso) - .astimezone(datetime.timezone.utc) - .isoformat() - .replace("+00:00", "Z") - ) - rows.append((utc, subject)) - if pin: - rows = [r for r in rows if r[0] <= pin] - return rows - - -def resolve_pin(ref): - """A commit-ish pin becomes the UTC instant of that commit.""" - if not ref: - return None - if ref.endswith("Z"): - return ref - iso = subprocess.run( - ["git", "-C", REPO, "log", "-1", "--format=%cI", ref], - capture_output=True, - text=True, - check=True, - ).stdout.strip() - return ( - datetime.datetime.fromisoformat(iso) - .astimezone(datetime.timezone.utc) - .isoformat() - .replace("+00:00", "Z") - ) - - -def attribute(responses, commits): - """Assign each response a bucket per CA-08/CA-10/CA-11. - - Intervals are ending-at-commit: (prev_commit, this_commit]. Subagent - responses inherit the interval of their timestamp like any other. - """ - label_for = [] - prev = "" - for ts, subject in commits: - m = TASK_RE.search(subject) - label_for.append((prev, ts, m.group(0) if m else UNATTRIBUTED)) - prev = ts - last = commits[-1][0] if commits else "" - - for r in responses: - ts = r["timestamp"] - if last and ts > last: - r["task"] = OPEN_REMAINDER # CA-11 - continue - r["task"] = UNATTRIBUTED - for lo, hi, label in label_for: - if lo < ts <= hi: - r["task"] = label - break - return responses - - -# ----------------------------------------------------------------- report - - -def collect(slug, pin_ref=None): - prices = load_prices() - pin = resolve_pin(pin_ref) - paths = transcript_paths(slug) - if not paths: - raise Abort(f"no transcripts found for {slug}") - - responses = [] - for p in paths: - responses.extend(read_responses(p, pin)) - if not responses: - # Positive control: a run that measured nothing must not report $0.00 - # as though it were an answer. - raise Abort(f"no responses in {len(paths)} transcript(s) — refusing to report") - - for r in responses: - r["cost"] = price_of(prices, r["model"], r["toks"]) - - attribute(responses, commit_index(pin)) - - by_task = collections.defaultdict(float) - by_component = collections.Counter() - by_component_cost = collections.defaultdict(float) - by_model = collections.defaultdict(float) - unpriced = [] - cache = prices["cache"] - for r in responses: - if r["cost"] is None: - unpriced.append(r) - continue - by_task[r["task"]] += r["cost"] - by_model[r["model"]] += r["cost"] - pr = prices[r["model"]] - unit = pr["input"] / 1e6 - rates = { - "input": unit, - "output": pr["output"] / 1e6, - "cache_read": unit * cache["read"], - "write_5m": unit * cache["write_5m"], - "write_1h": unit * cache["write_1h"], - } - for k, n in r["toks"].items(): - by_component[k] += n - by_component_cost[k] += n * rates[k] - - total = sum(by_task.values()) - # CA-14: reconciliation is asserted, not assumed. - residual = total - sum(by_component_cost.values()) - if abs(residual) > 0.005: - raise Abort( - f"reconciliation failed: task total ${total:,.4f} vs component total " - f"${sum(by_component_cost.values()):,.4f} (residual ${residual:,.4f})" - ) - - sub = sum(r["cost"] or 0 for r in responses if r["subagent"]) - return { - "slug": slug, - "pin": pin, - "responses": len(responses), - "transcripts": len(paths), - "total": total, - "subagent_total": sub, - "main_total": total - sub, - "by_task": dict(by_task), - "by_model": dict(by_model), - "tokens": dict(by_component), - "cost_by_component": dict(by_component_cost), - "unpriced": [ - {"model": r["model"], "tokens": r["toks"]} for r in unpriced - ], - "reconciled": True, - "residual": residual, - } - - -def render(rep, by_task=False, composition=False): - print(f"M-D2-CST cost report — {rep['slug']}") - print(f" pin {rep['pin'] or '(none — live file, not reproducible)'}") - print(f" transcripts {rep['transcripts']} responses {rep['responses']}") - print(f" main ${rep['main_total']:>10,.2f}") - print(f" subagent tree ${rep['subagent_total']:>10,.2f}") - print(f" TOTAL ${rep['total']:>10,.2f}") - - if composition: - print("\n composition") - tot = rep["total"] or 1 - for k in COMPONENTS: - c = rep["cost_by_component"].get(k, 0.0) - print( - f" {k:<12}{rep['tokens'].get(k,0):>14,} tok " - f"${c:>9,.2f} {100*c/tot:>5.1f}%" - ) - - if by_task: - print("\n by task") - rows = sorted(rep["by_task"].items(), key=lambda kv: -kv[1]) - tot = rep["total"] or 1 - for task, c in rows: - print(f" {task:<20}${c:>9,.2f} {100*c/tot:>5.1f}%") - # CA-10: the limit is reported with the number, every time. - un = rep["by_task"].get(UNATTRIBUTED, 0.0) - print( - f"\n NOTE: {100*un/tot:.0f}% of cost is UNATTRIBUTED — commits whose " - f"subject carries no T## tag.\n" - f" A per-task table is a view over {100*(1-un/tot):.0f}% of spend." - ) - - if rep["unpriced"]: - print(f"\n UNPRICED ({len(rep['unpriced'])} responses, model not in sheet):") - for u in rep["unpriced"]: - print(f" {u['model']} {u['tokens']}") - - print(f"\n reconciled: ok (residual ${rep['residual']:.6f})") - - -# -------------------------------------------------------------- self-test - - -def self_test(): - """AC-5..AC-8. Each asserts a failure mode is actually detected.""" - prices = load_prices() - checks = [] - - def check(name, ok, detail=""): - checks.append((name, ok, detail)) - - # AC-8: per-TTL cache pricing. 5m must not be priced at the 1h rate. - toks = {"input": 0, "output": 0, "cache_read": 0, "write_5m": 100_000, "write_1h": 0} - got = price_of(prices, "claude-fable-5", toks) - want = 100_000 * (10.0 / 1e6) * 1.25 - wrong = 100_000 * (10.0 / 1e6) * 2.0 - check("AC-8 5m cache priced at write_5m", abs(got - want) < 1e-9 and got != wrong, - f"${got:.4f} (1h would be ${wrong:.4f})") - - # AC-5: dedup invariant is enforced. - import tempfile - with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh: - u1 = {"input_tokens": 1, "output_tokens": 2, "cache_read_input_tokens": 3, - "cache_creation": {"ephemeral_5m_input_tokens": 0, - "ephemeral_1h_input_tokens": 0}} - u2 = dict(u1) - # Diverging cache_read is a real violation (input-side, charged once). - u2["cache_read_input_tokens"] = 999 - for u in (u1, u2): - fh.write(json.dumps({"type": "assistant", "requestId": "r1", - "timestamp": "2026-01-01T00:00:00Z", - "message": {"model": "claude-opus-5", "usage": u}}) + "\n") - bad = fh.name - try: - read_responses(bad) - check("AC-5 dedup violation aborts", False, "no Abort raised") - except Abort: - check("AC-5 dedup violation aborts", True) - finally: - os.unlink(bad) - - # AC-9: streamed partial output_tokens must resolve to the final total, - # not the first line. Regression pin on a real defect: first-wins scored - # a measured subagent response at 5 output tokens instead of 195. - with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh: - for out_tok in (5, 5, 195): - u = {"input_tokens": 2, "output_tokens": out_tok, - "cache_read_input_tokens": 0, - "cache_creation": {"ephemeral_5m_input_tokens": 19008, - "ephemeral_1h_input_tokens": 0}} - fh.write(json.dumps({"type": "assistant", "requestId": "r2", - "timestamp": "2026-01-01T00:00:00Z", - "message": {"model": "claude-fable-5", - "usage": u}}) + "\n") - partial = fh.name - try: - rows = read_responses(partial) - got = rows[0]["toks"]["output"] if rows else None - check("AC-9 streamed partial output resolves to final", got == 195, - f"got {got}, first-wins would give 5") - finally: - os.unlink(partial) - - # AC-6: zero responses must not report $0.00 as an answer. - with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False) as fh: - fh.write(json.dumps({"type": "user", "message": {}}) + "\n") - empty = fh.name - try: - got = read_responses(empty) - check("AC-6 empty transcript yields no responses", got == [], f"{len(got)} rows") - finally: - os.unlink(empty) - - # AC-7: the subagent tree is discovered by the path globs. - slug = "-home-worsch-clay-borg" - paths = transcript_paths(slug) - subs = [p for p in paths if "/subagents/" in p] - check("AC-7 subagent tree enumerated", len(subs) > 0, - f"{len(subs)} subagent transcript(s) of {len(paths)} total") - - print("cb-cost self-test (positive control)") - ok = True - for name, passed, detail in checks: - print(f" [{'ok ' if passed else 'FAIL'}] {name}" + (f" — {detail}" if detail else "")) - ok &= passed - return 0 if ok else 1 - - -def main(): - ap = argparse.ArgumentParser() - ap.add_argument("--slug", default="-home-worsch-clay-borg") - ap.add_argument("--pin", help="commit-ish or ISO Z instant (CA-07)") - ap.add_argument("--by-task", action="store_true") - ap.add_argument("--composition", action="store_true") - ap.add_argument("--self-test", action="store_true") - ap.add_argument("--json", action="store_true") - args = ap.parse_args() - - if args.self_test: - return self_test() - - try: - rep = collect(args.slug, args.pin) - except Abort as e: - print(f"ABORT — {e}", file=sys.stderr) - return 1 - - if args.json: - print(json.dumps(rep, indent=2)) - else: - render(rep, by_task=args.by_task, composition=args.composition) - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/workplans/CB-WP-0002-cost-accounting.md b/workplans/CB-WP-0002-cost-accounting.md index 7a3e87d..d9308b2 100644 --- a/workplans/CB-WP-0002-cost-accounting.md +++ b/workplans/CB-WP-0002-cost-accounting.md @@ -1,7 +1,7 @@ --- id: CB-WP-0002 title: "Make agentic cost measurable, so D2 claims are falsifiable" -status: done +status: proposed state_hub_workstream_id: "b7c22f69-fbe9-48df-9619-007db79ae338" --- @@ -18,41 +18,19 @@ transcript (`~/.claude/projects//.jsonl`) carries exact per-message `usage`, including the cache breakdown. Reading it for CB-WP-0001's session gives: -> **Corrected 2026-07-31 by T01.** This section originally reported -> **$248.46** from 131,863,164 cache-read tokens priced at Fable 5. That -> figure was wrong by ~2.7×, for two independent reasons found in -> `research/CB-RES-0002-cost-accounting.md` §Correction: one API response -> is written to the transcript as up to six JSONL lines that each repeat -> the *same* `usage` object (657 lines, 346 real responses), and the -> session ran three models, not one. The numbers below are the re-derived -> ones. The workplan's premise is unaffected; its arithmetic was not. - -Deduplicated by `requestId` and priced per message at its own model's -rate. **Pinned** to messages at or before `2026-07-31T02:17:59Z` (commit -`fc76445`, the end of CB-WP-0001) — the transcript is a live file that -grows as later sessions append to it, so an unpinned total is not a -repeatable acceptance target: - -| Component | Tokens | Cost | +| Component | Tokens | Cost (Fable 5) | |---|---|---| -| Output | 313,900 | $10.66 | -| Cache read | 80,453,702 | **$59.59** | -| Cache write (1h) | 1,672,854 | $21.95 | -| Input | 676 | $0.00 | -| Main transcript (339 responses: 213 opus-5, 118 fable-5, 14 sonnet-5) | | $92.21 | -| Subagent tree (adversarial review, ran inside the pin) | | $1.11 | -| **TRUE TOTAL** | | **$93.32** | +| Output | 585,528 | $29.28 | +| Cache read | 131,863,164 | **$131.86** | +| Cache write (1h) | 4,365,668 | $87.31 | +| Input | 1,090 | $0.01 | +| **Total** | | **$248.46** | -The headline finding survives the correction and gets sharper: -**88.4% of the cost is cache, against 314k tokens of output — a 256:1 -ratio of context re-read to text written.** Cost in an agentic loop is -driven by context size × turn count, not by how much the model writes. No -D2 decision made on "tokens per task" would have surfaced that. - -The correction is itself the lesson: this workplan opened with a -mismeasured cost. Neither error was of the harness-does-nothing class the -positive-control rule was written for — both sums ran over real data. -What caught them was re-deriving the number instead of quoting it. +The headline finding from that single reading is the reason this +workplan exists: **53% of the cost is cache reads**, not output. Cost in +an agentic loop is driven by context size × turn count, not by how much +the model writes. No D2 decision made on "tokens per task" would have +surfaced that. This workplan makes cost a first-class measured dimension so that future AM-12 equivalents support conclusions instead of decorating an @@ -66,7 +44,7 @@ positive control. ```task id: CB-WP-0002-T01 -status: done +status: todo priority: high state_hub_task_id: "2694c2c1-0070-4d8e-b4fc-196b582b36d5" ``` @@ -89,7 +67,7 @@ accuracy and the hub to lead on durability. ```task id: CB-WP-0002-T02 -status: done +status: todo priority: high state_hub_task_id: "eae248ab-f29f-4f11-9d20-e8145b0d822d" ``` @@ -115,7 +93,7 @@ Gate: no collector code before this ADR is committed. ```task id: CB-WP-0002-T03 -status: done +status: todo priority: high state_hub_task_id: "00d42ed2-4391-4580-aae2-06e3e151c69b" ``` @@ -138,7 +116,7 @@ replace AM-12's definition with one that is computable. ```task id: CB-WP-0002-T04 -status: done +status: todo priority: medium state_hub_task_id: "9eb8329b-5f41-477b-8cf3-2cda5ba8dbe8" ``` @@ -162,22 +140,16 @@ each message at its own model's rate. ```task id: CB-WP-0002-T05 -status: done +status: todo priority: medium state_hub_task_id: "bea4cc0a-e4d0-4077-9dc7-df7726a48f86" ``` Run the collector over the CB-WP-0001 session and commit -`evidence/CB-EV-0002-cost-accounting.md`. The acceptance test is -`specs/CostAccounting.md` AC-1: reproduce **$93.32 pinned at `fc76445`, -as its two components** ($92.21 main transcript + $1.11 subagent tree), -plus AC-2's zero reconciliation residual. - -*(Originally written as "must reproduce $248.46". That figure was wrong -by 2.7×, and the first corrected target — $92.21 — was itself reachable -only by a collector carrying the subagent blind spot the survey had just -documented. Both errors are recorded rather than quietly overwritten; -the sequence is the point.)* +`evidence/CB-EV-0002-cost-accounting.md`. Reconciliation against the +totals in this workplan's Purpose section is the acceptance test: the +tool must reproduce $248.46 (Fable 5) from the same transcript, or +explain the difference. Then use it to answer at least one question that could not be answered before, and record the answer. Candidates: which of T01–T09 cost most @@ -192,7 +164,7 @@ cleared the bar that CB-WP-0001's AM-12 failed to clear. ```task id: CB-WP-0002-T06 -status: done +status: todo priority: low state_hub_task_id: "1412263b-70c1-43e4-957d-1ad6c3203ca9" ``` @@ -207,7 +179,7 @@ the one command surface. ```task id: CB-WP-0002-T07 -status: done +status: todo priority: low state_hub_task_id: "ebe58d91-be5f-4d5b-ba40-b03275b4eefc" ```