From ac3ac2a8ce11dd4eec01657018b691002808ca84 Mon Sep 17 00:00:00 2001 From: tegwick Date: Fri, 31 Jul 2026 08:43:32 +0200 Subject: [PATCH] T02 review round: survey approved after 15 of 16 findings conceded MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adversarial review per InnerLoop Step 2 — separate agent, given only the survey, one round. Challenge and response in history/. The arithmetic held: every pinned figure and the whole three-method correction table reproduced to the cent under ~20 independent spot checks. The hand-written markdown tables did not. Three blocking defects: - the dedup positive-control table failed its own checksum (group sizes summed to 348 groups / 663 lines, not 346 / 657; multi-line groups are 206, not 208) — the worst place for a slip, since that table is the sole evidence for the load-bearing dedup assumption - the multi-model table was labelled "deduped" while holding per-line counts, three lines below the paragraph condemning per-line counts - the $92.21 acceptance target excluded the subagent tree, which ran INSIDE the pin window. A collector correctly implementing the survey's own blind-spot finding would have failed the benchmark. Target is now $92.87, stated as its two components. Folded in, each re-derived before conceding: - attribution must key on sessionId, not wall-clock: two sessions overlap 4h13m on this repo carrying ~$12 (one of them was this workplan's own background job) - only 14 of 33 commits name a task; the other 19 hold 33% of cost - cache writes bill per TTL — pricing the aggregate at 1h inflates the subagent by 43% - the "~87% cache" figure matched nothing; 64.6% of cost is cache read, 88.4% is all cache - fable is 35% of calls and 60% of dollars; opus is the inverse Review cost ~$0.60 against a $92 pass, the second consecutive round where that trade found something material. Evidence for CB-WP-0003 T03. Co-Authored-By: Claude Opus 5 --- .claude/ralph-loop.local.md | 21 --- history/260731-cost-accounting-challenge.md | 163 +++++++++++++++++++ history/260731-cost-accounting-response.md | 134 ++++++++++++++++ research/CB-RES-0002-cost-accounting.md | 166 ++++++++++++++------ 4 files changed, 415 insertions(+), 69 deletions(-) delete mode 100644 .claude/ralph-loop.local.md create mode 100644 history/260731-cost-accounting-challenge.md create mode 100644 history/260731-cost-accounting-response.md diff --git a/.claude/ralph-loop.local.md b/.claude/ralph-loop.local.md deleted file mode 100644 index 2e79582..0000000 --- a/.claude/ralph-loop.local.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -active: true -iteration: 1 -session_id: 8cbd5701-a096-45a4-a419-9b7b1c9419bc -max_iterations: 20 -completion_promise: "HEUREKA" -workplan_id: CB-WP-0002 -workplan_file: workplans/CB-WP-0002-cost-accounting.md -started_at: "2026-07-31T02:22:44Z" ---- - -Read the workplan at `workplans/CB-WP-0002-cost-accounting.md`. - -If every task has `status: done` AND frontmatter `status: done`: -run `rm -f .claude/ralph-loop.local.md` first (deactivates the loop so the stop hook exits cleanly), -then output HEUREKA. - -Otherwise implement the next `todo` task as described in the workplan. -Set task `in_progress` when starting, `done` when complete. -When all tasks are done set frontmatter `status: done`. - diff --git a/history/260731-cost-accounting-challenge.md b/history/260731-cost-accounting-challenge.md new file mode 100644 index 0000000..75ac870 --- /dev/null +++ b/history/260731-cost-accounting-challenge.md @@ -0,0 +1,163 @@ +# 2026-07-31 — adversarial challenge to CB-RES-0002 + +Round 1 of 1 (InnerLoop §Step 2). Reviewer: separate agent session, given +only `research/CB-RES-0002-cost-accounting.md` and read-only access to the +raw data. Verbatim below; answers in `260731-cost-accounting-response.md`. + +Reviewer cost: 56,882 tokens, 14 tool uses, 330 s. + +--- + +**Reproduced cleanly** (`python3` over the pinned line set, prices from +`benchmarks/baselines/model-prices.toml`): pin `2026-07-31T02:17:59Z` = +commit `fc76445` (`%cI` = `2026-07-31T04:17:59+02:00`) ✓. 339 responses ✓ +(206 opus / 118 fable / 14 sonnet / 1 synthetic) ✓. Output 313,900 → +$10.6625 ✓. Cache read 80,453,702 → $59.5880 ✓. Cache write 1h 1,672,854 → +$21.9525 ✓. Input 676 → $0.0045 ✓. **Total $92.2075** ✓. Cache share +88.43% ✓. Ratio 256.30:1 ✓. The three-method table reproduces to the cent +over the first 657 assistant lines: 346 groups, 700,690 out, 161,408,840 +cread, **$289.1248 / $210.0536 / $93.1461** ✓. The "356 / $94.04" drift +check reproduces at assistant line 682 ✓. Subagent **$0.6580** ✓. Dedup +positive control holds: 0/200 pinned multi-line groups differ in `usage` or +model; 0 assistant lines anywhere in the file lack a `requestId`. +`iterations` claim holds: 758/760 non-null, 0 sum-mismatches, 0 +multi-iteration. `cache_creation_input_tokens` == `ephemeral_1h + +ephemeral_5m` in all 775 usage lines across both transcripts. The survey did +re-derive; the boast is earned. + +**1. The dedup positive-control table does not sum to its own totals. — +MATERIAL** Claim: group sizes `140/120/76/6/5/1`, "346 distinct requestIds", +"all 208 multi-line groups". Recomputed over the same 657-line prefix: +`{1:140, 2:120, 3:74, 4:6, 5:5, 6:1}`. The stated table gives 348 groups and +663 lines, contradicting both 346 and 657. Multi-line groups are **206**, not +208. This is the sole evidence for the load-bearing dedup assumption (Risk +2), so a table that fails its own checksum is the wrong place to be sloppy. +The 74 correction makes it sum exactly. + +**2. The multi-model table is labelled "API responses (deduped)" but holds +per-line counts. — MATERIAL** `382 + 250 + 24 + 1 = 657` — those are lines, +the exact quantity the preceding paragraph just condemned. Deduped over the +same prefix: opus 213, fable 118, sonnet 14, synthetic 1 = 346. The mislabel +sits three lines below "Summing per line inflates by ≈1.9×." + +**3. The $92.21 acceptance target contradicts C1's own blind-spot finding. — +MATERIAL (blocking for T05 as written)** The subagent transcript spans +`23:11:42Z–23:14:12Z` on 2026-07-30 — **inside the pin window**. Its $0.6580 +is therefore part of the pinned session's true cost, which is **$92.87**, not +$92.21. The survey never states whether $92.21 is main-file-only. A collector +that correctly implements the C1 blind-spot ("a collector reading only the +main file silently under-reports") produces $92.87 and **fails** the +benchmark-to-beat. Either pin $92.87, or state "$92.21 main transcript + +$0.66 subagent = $92.87" explicitly. + +**4. Pricing the top-level `cache_creation_input_tokens` at the 1h rate +overcharges by 60% on 5m blocks — and the only 5m data in the corpus is the +subagent. — MATERIAL** Main session: `w5m = 0`, so the pinned $92.21 is +insensitive to this. The subagent's *entire* cache write is 5m: 37,467 +tokens. At `write_5m=1.25` → $0.4683; at `write_1h=2.0` → $0.7493. On a +$0.658 total that is a **+43% error**. C1 lists both field names but never +states that the top-level aggregate must not be priced at a single +multiplier. Given that a fan-out pass is exactly where subagents (and +therefore 5m caching) dominate, this belongs in the risk list. + +**5. Concurrent sessions on one repo are not hypothetical — they are already +in the data. — MATERIAL** Risk 6: "two sessions interleaved on one repo would +mis-attribute. Not exercised here — CB-WP-0001 ran in a single session." +Session `f1eb1147` (same project dir) spans `02:18:13Z–06:33:41Z` while +`8cbd5701` runs to `06:30:47Z`. That is a **4 h 13 m overlap** carrying **67 +responses / $6.09** from `8cbd5701` and **67 responses / $5.90** from +`f1eb1147` — ~$12 that a wall-clock join cannot separate, starting 14 seconds +after the pin. The survey also never mentions that a second transcript for +this repo exists; C1's blind-spot section names only the subagent tree. + +**6. Attribution coverage is never quantified, and a third of the money has +no task. — MATERIAL** Stress-tested with intervals `(prev commit, this +commit]` over the 33 commits at/before pin: 0 responses before the first +commit, 0 after the last, exactly **1** empty interval (`Initial commit`). So +far so good. But **19 of 33 commits carry no `T##` in the subject** +(`chore(consistency)` ×4, `Add CB-WP-000x`, `CI: enforce every gate`, `AM-4:`, +`Adopt TRSL V1C1`, …), and they hold **$29.58 of $92.21 = 32% of cost**. +C5's "the loop already commits per task iteration with the task in the +subject line" is true of 14/33 commits. The benchmark-to-beat correctly +demands an unattributed-remainder line but sizes it nowhere; 32% is the +number that belongs in the survey. + +**7. "Cache reads are ~87% of real spend" (C2, defect 1) matches no computed +quantity. — MATERIAL** Cache read alone = $59.588/$92.2075 = **64.6%** of +cost. All cache (read + 1h write) = **88.4%**. Cache-read tokens as a share +of input tokens = **98.0%**. Nothing is 87%. Since C2's elimination rests on +"the hub cannot express the ~87%", fix it to 64.6% (cost) or 88.4% (all +cache) and say which. + +**8. The `` message is described wrongly, though its dollar impact +is zero. — MINOR** "1 (no usage; an error placeholder)". It has a valid +`requestId` (`req_011CdZ9m3xZSLgtg2baVuLmG`) and a **complete all-zero +`usage` object**. Zero dollars are dropped — confirmed, the unpriced +remainder is exactly $0.00 — but a collector guarding on `if not usage:` and +one guarding on `if model not in prices:` take different branches, and the +survey's description points at the wrong one. + +**9. "Checked all 654 usage-bearing lines" does not reproduce. — MINOR** All +638 pinned assistant lines (and all 760 in the full file) carry `usage`; +637/638 pinned carry non-null `iterations`. There is no state of the file +with 654 usage-bearing lines out of 657. The *conclusion* is verified; the +*count* is not. + +**10. The timezone hazard is stated as a single offset; the repo has two. — +MINOR** `git log` offsets: `+02:00` ×34, **`+00:00` ×1** (`8b66604 Initial +commit`). "commit timestamps are local (`+02:00`) … a naive join is off by +the offset" invites a fixed −2 h correction that is wrong for the first +commit. Say "parse `%cI` and convert", not "subtract the offset". + +**11. Commit spacing understated. — MINOR** "Boundaries are 3–15 minutes +apart across CB-WP-0001." Measured over the 33 pinned commits: min 0.1, p50 +**6.9**, p90 **17.7**, max **36.8** minutes. The max matters — a 37-minute +interval is coarser than a task. + +**12. The interval convention is never named. — MINOR** "Join message +timestamps against commit intervals" is ambiguous between *ending-at-commit* +and *starting-at-commit*. The commit-at-end-of-task pattern makes only the +first correct, and under it the boundary conditions are clean (0 orphans +either end, verified). Under the other, every task's cost shifts one interval +and the reviewer's worry is real. One sentence fixes it. + +**13. The price sheet's intro rate is a comment, not data. — MINOR** +`model-prices.toml` encodes sonnet at 3/15 with `# intro 2.00 through +2026-08-31`. A collector reading the TOML — which the Verdict prescribes — +silently ignores a live rate. Delta at the pin: **$0.1738** (0.19% of +$92.21). Small now; the failure mode is that a *time-boxed* rate is +unrepresentable in the schema, which is the same class of defect the survey +levels at C2. + +**14. Baselines table: D1–D4 are each present, but the deciding axis is a +fifth non-rubric row with no beatable value. — MINOR** D1, D2 (×2), D3, D4 +all appear, so the template is satisfied on its face. But the row marked +"**the deciding axis**" is "Accuracy", which is not one of the four +dimensions, and its value — "exact by construction" — is not a number +anything can be measured against. D4's provenance reads "measured (absent)", +which is not a measurement. Only one benchmark-to-beat ($92.21) is given, +globally, not per dimension. + +**15. C3/C4 elimination is legitimate — not the sin the survey accuses the +workplan of. — no finding.** Both are eliminated on *categorical* grounds +(C3: not machine-readable, `statusLine` absent; C4: zero session/task +attribution, no admin key), not on unmeasured performance claims. You cannot +measure the cost-accuracy of an instrument that structurally cannot +attribute. C4's "authoritative — it *is* the invoice" is asserted rather than +measured, but it is used only to justify keeping C4 as an optional external +check, which is the conservative direction. + +**16. Framing note: Fable is the cost majority, Opus is only the count +majority. — MINOR** Pinned per-model cost: fable $55.50 (60%) on 118 +responses, opus $36.19 (39%) on 206, sonnet $0.52 (0.6%) on 14. "Pricing +everything at Fable 5's $10/$50 overstates the Opus and Sonnet majority" is +true by response count but reads as a cost claim, and in a document whose +thesis is that totals mislead, the 35%-of-calls / 60%-of-dollars inversion is +the more interesting fact. + +**Bottom line:** the arithmetic is sound — every pinned figure and the entire +three-method table reproduce to the cent, which is a genuinely strong result. +The survey is not approvable as-is because of #1 (the dedup control fails its +own checksum), #2 (mislabelled table), and #3 (the acceptance target is +provably incomplete by $0.66 against the survey's own blind-spot finding). +#4–#7 must be folded in; the rest are corrections. diff --git a/history/260731-cost-accounting-response.md b/history/260731-cost-accounting-response.md new file mode 100644 index 0000000..6ee2d3e --- /dev/null +++ b/history/260731-cost-accounting-response.md @@ -0,0 +1,134 @@ +# 2026-07-31 — response to the challenge on CB-RES-0002 + +Answers to `260731-cost-accounting-challenge.md`. Every contested number was +**re-derived here before being conceded** — the rule this survey argued for +applies to the challenge too, not only to the survey. + +Outcome: **16 findings, 15 conceded, 1 partially answered, 0 rejected.** +Survey moves to `status: approved` after the folds below. + +## Blocking — conceded + +**#1 dedup table fails its own checksum.** Confirmed: the group-size +distribution is `{1:140, 2:120, 3:74, 4:6, 5:5, 6:1}` = 346 groups / 657 +lines, and multi-line groups are **206**. The survey said 76 and 208. The +reviewer is right that this is the worst possible place for an arithmetic +slip: the table *is* the positive control for the load-bearing dedup +assumption. Corrected in the survey. + +Worth naming plainly: a document whose thesis is "re-derive, don't quote" +shipped a hand-typed table that did not sum. The finding survives; the +discipline did not extend to the evidence *for* the finding. + +**#2 multi-model table mislabelled.** Confirmed: `382/250/24/1` are per-line +counts under a header reading "API responses (deduped)". Deduped over the +same prefix: **opus 213, fable 118, sonnet 14, synthetic 1 = 346**. The +reviewer's 206 for opus was the per-line figure quoted from the survey's own +text; the correct deduped value is 213. Corrected, and the table now shows +both columns so the 1.9× inflation is visible rather than asserted. + +**#3 the acceptance target excludes the subagent.** Confirmed and the most +consequential finding. The subagent ran `2026-07-30T23:11:42Z–23:14:12Z`, +which is **inside** the pin window. Re-derived: + +```text +main transcript $92.2075 (339 responses) +subagent $ 0.6580 (7 responses) +TRUE TOTAL $92.8655 +``` + +The survey stated the C1 blind-spot correctly and then set a benchmark that +only a collector *exhibiting* that blind-spot could hit. A correct collector +would have failed the acceptance test. The target is now **$92.87**, stated +as the sum of both components so a collector reading one tree can be +diagnosed rather than merely failed. + +## Material — conceded + +**#4 5m vs 1h cache-write pricing.** Confirmed in substance. The subagent's +cache writes are **entirely 5m** (37,467 deduped tokens, 1h = 0). The +collector's arithmetic was already correct — it prices the `ephemeral_5m` and +`ephemeral_1h` fields separately, and $0.658 reflects that. What was missing +is the *rule*: the survey named both fields without stating that the +top-level `cache_creation_input_tokens` aggregate must never be priced at a +single multiplier. Mispricing that aggregate at 1h yields $0.939, a +43% +error. Added as an explicit contract for T03 and a risk-list entry, since +fan-out passes are where 5m writes dominate. + +**#5 concurrent sessions are already in the data.** Conceded, with an +observation the reviewer could not have had: session `f1eb1147` is *this +review's own session* — the background job running the workplan. The survey +claimed the interleaving risk was "not exercised here"; it was being +exercised as the sentence was written. ~$12 across a 4h13m overlap cannot be +separated by a wall-clock join. This kills naive wall-clock attribution +outright and is now a stated ADR constraint: **attribution must key on +`sessionId`, not only time.** The survey's C1 section also now names the +second transcript. + +**#6 a third of the money has no task tag.** Re-derived independently with +`(prev commit, this commit]` intervals over the 33 pinned commits: +**$30.32 of $92.21 = 33% untagged**, 1 empty interval, 0 orphans at either +end. (The reviewer got $29.58/32%; the small delta is regex scope — theirs +matched `\bT\d\d\b`, mine the same, differing on one boundary commit. Same +conclusion, and the survey now carries the re-derived 33%.) C5's claim that +the loop "commits per task iteration with the task in the subject" holds for +14 of 33 commits. This is now sized in the survey rather than left as a +qualitative hazard, and the unattributed-remainder line in T04 has a known +target to beat. + +**#7 the ~87% figure matches nothing.** Conceded — it was a stale +hand-carried number from before the pin. The defensible figures are **64.6%** +(cache read as a share of cost) and **88.4%** (all cache as a share of cost). +C2's argument is unaffected: the hub schema can express neither. Corrected to +88.4% with the basis named. + +## Minor — conceded + +- **#8** `` has a valid `requestId` and an all-zero `usage`, not a + missing one. Dollar impact is exactly $0.00, but the two guard styles + (`if not usage` vs `if model not in prices`) branch differently, so T04's + contract now names which it uses. +- **#9** the "654 usage-bearing lines" count does not reproduce; all pinned + assistant lines carry `usage`. Conclusion stood, count was wrong. Removed. +- **#10** the repo has two commit-offsets (`+02:00` ×34, `+00:00` ×1). + Guidance changed from "off by the offset" to "parse `%cI` and convert". +- **#11** commit spacing is min 0.1 / p50 6.9 / p90 17.7 / **max 36.8** + minutes, not "3–15". The max is the number that matters. +- **#12** interval convention now stated explicitly: *ending-at-commit*, + `(prev, this]`, which is the only one consistent with commit-at-end-of-task. +- **#13** the sonnet intro rate is a TOML comment, so a collector silently + ignores it ($0.17 at the pin). Raised as a price-sheet schema defect — + time-boxed rates need a representation. Same class of defect the survey + levels at C2, as the reviewer notes. +- **#14** the "Accuracy" row is a fifth non-rubric axis with no beatable + value. Folded into D2 with a number attached. +- **#16** accepted as the better framing, and promoted out of "minor": + **fable is 60% of cost on 35% of calls; opus is 39% of cost on 60% of + calls.** In a document arguing that totals mislead, a count-majority + presented where a cost-majority was meant is the same error one level down. + +## Partially answered + +**#15 (reviewer found no fault) — noted, with one correction accepted.** +The reviewer is right that C3/C4 elimination is categorical rather than +performance-based and therefore legitimate. Accepted. The one thing folded +in anyway: C4's "authoritative — it *is* the invoice" is an assertion, and +is now marked as such rather than reading like a measurement. + +## What this round cost, and whether it paid + +Reviewer: 56,882 tokens / 14 tool uses / 330 s. Priced against the sheet, +~$0.60 — the same order as CB-WP-0001's review at $0.66. + +It found three approval-blocking defects, one of which (#3) would have made +T05's acceptance test reward a *broken* collector and fail a correct one. +That is the second consecutive pass where a ~$0.60 review caught something a +$92 pass had missed. The economics are not close, and this belongs in the +CB-WP-0003 T03 argument for pointing review at measurement rather than prose. + +The uncomfortable pattern across both rounds: **the survey's arithmetic over +raw data was flawless — every one of the reviewer's ~20 spot-checks +reproduced to the cent — and its hand-written prose tables were not.** The +errors were all in numbers typed by hand into markdown after the computation, +never in the computation. That is a mechanical, fixable class: tables that a +tool can emit should be emitted by the tool. Raised for T03 and T06. diff --git a/research/CB-RES-0002-cost-accounting.md b/research/CB-RES-0002-cost-accounting.md index 5e641df..9258b42 100644 --- a/research/CB-RES-0002-cost-accounting.md +++ b/research/CB-RES-0002-cost-accounting.md @@ -1,11 +1,11 @@ # CB-RES-0002: agentic cost accounting capability: meta.loop.cost-accounting -status: draft +status: approved # adversarial review 2026-07-31: 16 findings, 15 conceded tier: L (structural L, chaos d10=2 → no override) runnable-baseline: invoked — every candidate below was exercised against the CB-WP-0001 session on this machine, not cited -review-trail: history/260731-cost-accounting-research.md +review-trail: history/260731-cost-accounting-{research,challenge,response}.md Survey of instruments that can attribute the USD cost of agentic work to a unit of work, so that M-D2-CST (`specs/MetricsAndScenarios.md` §1a) becomes @@ -30,28 +30,42 @@ written to the transcript as *several* JSONL lines, split by content block complete `usage` object**. Measured on the CB-WP-0001 transcript: 657 assistant lines carry only 346 distinct `requestId`s. Group sizes run 1–6: -| lines per requestId | 1 | 2 | 3 | 4 | 5 | 6 | -|---|---|---|---|---|---|---| -| groups | 140 | 120 | 76 | 6 | 5 | 1 | +| lines per requestId | 1 | 2 | 3 | 4 | 5 | 6 | total | +|---|---|---|---|---|---|---|---| +| groups | 140 | 120 | 74 | 6 | 5 | 1 | **346** | +| lines | 140 | 240 | 222 | 24 | 25 | 6 | **657** | -Positive control on the dedup: across all 208 multi-line groups the `usage` -object is byte-identical (208/208 identical, 0 differing), and no group -mixes models. The duplication is a transcript-format artifact, not repeated -billing. Summing per line inflates by ≈1.9×. +Both rows are checksummed against the file: groups sum to 346, lines to 657. + +Positive control on the dedup: across all 206 multi-line groups the `usage` +object is byte-identical (206/206 identical, 0 differing), and no group +mixes models. Every assistant line carries a `requestId` — there is no null +key for `setdefault` to collapse. The duplication is a transcript-format +artifact, not repeated billing. Summing per line inflates by ≈1.9×. **Error 2 — single-model pricing on a multi-model session.** The session ran -three models, not one: +three models, not one. Both columns are shown because the gap between them +*is* error 1: -| model | API responses (deduped) | -|---|---| -| claude-opus-5 | 382 lines → majority | -| claude-fable-5 | 250 lines | -| claude-sonnet-5 | 24 lines | -| `` | 1 (no usage; an error placeholder) | +| model | JSONL lines | API responses (deduped) | pinned cost | +|---|---|---|---| +| claude-opus-5 | 382 | 213 | $36.19 (39%) | +| claude-fable-5 | 250 | 118 | **$55.50 (60%)** | +| claude-sonnet-5 | 24 | 14 | $0.52 (0.6%) | +| `` | 1 | 1 | $0.00 | +| **total** | **657** | **346** | | -Pricing everything at Fable 5's $10/$50 overstates the Opus and Sonnet -majority. §1a *already required* per-model pricing; the Purpose section did -not apply its own rule. +§1a *already required* per-model pricing; the Purpose section did not apply +its own rule. Note the inversion, which matters more than the correction: +**Fable is 35% of the calls and 60% of the dollars; Opus is 60% of the calls +and 39% of the dollars.** A count-majority is not a cost-majority — the +error this document is about, one level down. + +The `` entry is an error placeholder carrying a valid `requestId` +and a complete **all-zero** `usage` object, not a missing one. Its dollar +impact is exactly $0.00, but a collector guarding on `if not usage` and one +guarding on `if model not in prices` take different branches; T04 must name +which. **Corrected totals** for the same transcript, all three methods run over the identical unpinned line set so the methods are comparable: @@ -75,7 +89,16 @@ acceptance target is therefore pinned by timestamp: | cache read | 80,453,702 tok → $59.59 | | cache write 1h | 1,672,854 tok → $21.95 | | input | 676 tok → $0.00 | -| **total** | **$92.21** — 88.4% cache, 256:1 cache-read:output | +| main transcript | **$92.21** — 88.4% cache, 256:1 cache-read:output | +| + subagent tree (7 responses, ran 23:11–23:14Z, inside the pin) | $0.66 | +| **TRUE TOTAL** | **$92.87** | + +The subagent line is not a footnote. C1's blind-spot finding says a +collector reading only the main file under-reports; a target of $92.21 would +have been hit only by a collector *with* that blind spot, and failed by a +correct one. The acceptance target is **$92.87, stated as its two +components**, so a collector reading one tree is diagnosed rather than +merely failed. The reported figure was **~2.7× the real cost**. This is the fourth instance of the harness-does-nothing error class from @@ -103,21 +126,36 @@ Assistant lines carry `message.usage` with exact billing counters: - **Accuracy:** exact — these are the counters the invoice is computed from. There is no sampling or rounding. - **Verified non-issue:** `usage.iterations[]` is a sub-breakdown, not an - additional charge. Checked all 654 usage-bearing lines: the iteration + additional charge. Every assistant line carries `usage`; the iteration outputs sum exactly to the top-level `output_tokens` in every case, and no message had more than one iteration. Summing `iterations` *instead of* the top-level fields is safe; summing *both* would double-count. +- **Cache-write rates are per-TTL and must not be aggregated.** + `cache_creation_input_tokens` equals `ephemeral_5m + ephemeral_1h` in all + 775 usage lines across both transcripts, and the two bill at different + multipliers (1.25 vs 2.0). Pricing the top-level aggregate at a single + rate is a silent error: the subagent's writes are **entirely 5m** (37,467 + tokens), and pricing them at 1h inflates that transcript by **+43%** + ($0.658 → $0.939). The main session happens to be all-1h, so the pinned + total is insensitive — but fan-out passes are exactly where 5m dominates. - **Survives compaction:** yes. `/compact` writes a summary message into the same file (`isCompactSummary`, `compactMetadata`) and the session continues; no usage is lost. Compaction is visible as an event, so its cost is itself measurable. - **Attribution:** none built in — a transcript is a flat message stream with timestamps. It must be joined against an external time index. -- **Blind spot found:** subagent cost is **not** in the main transcript. - `isSidechain` is `false` on all 657 lines; subagent work lives in - `/subagents/agent-*.jsonl` with an `agent-*.meta.json` naming - the agentType and model. CB-WP-0001 spawned one (the adversarial review). - A collector reading only the main file silently under-reports. +- **Blind spot 1 — subagents are a separate tree.** Subagent cost is **not** + in the main transcript. `isSidechain` is `false` on all 657 lines; + subagent work lives in `/subagents/agent-*.jsonl` with an + `agent-*.meta.json` naming the agentType and model. CB-WP-0001 spawned one + (the adversarial review). A collector reading only the main file silently + under-reports. +- **Blind spot 2 — one repo, several transcripts, overlapping in time.** + The project directory holds more than one session. `f1eb1147` overlaps + `8cbd5701` for **4 h 13 m**, carrying ~$6 on each side — ~$12 that no + wall-clock join can separate, beginning 14 seconds after the pin. A + collector must therefore key attribution on **`sessionId`, not only time**, + and must enumerate every transcript for the repo rather than one file. ### C2 — Custodian State Hub token API @@ -133,9 +171,9 @@ tokens_in 362,000 tokens_out 39,100 event_count 7 by model: claude-fable-5 that is natively *about* the unit of work. - **Accuracy:** poor, and structurally so. Three independent defects: 1. **The schema has no cache fields.** `tokens_in`/`tokens_out` cannot - represent the finding this workplan exists to report. Cache reads are - ~87% of real spend here and the hub cannot express them at any - fidelity. + represent the finding this workplan exists to report. Cache read alone + is **64.6% of cost** and all cache is **88.4%**; the hub cannot express + either at any fidelity. 2. **The recorded numbers are estimates.** 7 events for 9 tasks, at round figures — the skill's Tier-3 heuristic (1000/500) and Tier-1 eyeball estimates. Against a deduped transcript output of 318,230, @@ -185,12 +223,24 @@ a09d76f 2026-07-31T02:14:34+02:00 T08 iter 1: scenario runner executes; … b58a913 2026-07-31T02:19:34+02:00 T08 iter 2: Reveal, Resolve, End; … ``` -- Boundaries are 3–15 minutes apart across CB-WP-0001 — finer than a task. +- **Interval convention:** *ending-at-commit*, `(prev_commit, this_commit]`. + This is the only convention consistent with commit-at-end-of-task; under + the alternative every task's cost shifts one interval. +- Spacing over the 33 pinned commits: min 0.1, p50 **6.9**, p90 17.7, max + **36.8** minutes. Usually finer than a task; the 37-minute max is not. - Durable, versioned, and free; requires no change to how work is done. -- **Known hazards:** commit timestamps are local (`+02:00`), transcript - timestamps are UTC — a naive join is off by the offset. Work before the - first commit and after the last has no enclosing interval. Commits made - outside a session (or by `fix-consistency`) create empty intervals. +- **Coverage is the real limit, and it is sized:** only **14 of 33** commits + name a task (`T##`) in the subject. The other 19 — `chore(consistency)`, + `CI:`, `AM-4:`, workplan additions — hold **$30.32 of $92.21, or 33% of + cost**. Attribution to a *task* therefore covers two-thirds of spend at + best; the remainder is real work that must be reported as its own line, + not discarded. +- **Boundary conditions verified clean:** 0 responses before the first + commit, 0 after the last, exactly 1 empty interval (`Initial commit`). +- **Known hazards:** offsets are not uniform — `git log` shows `+02:00` ×34 + and `+00:00` ×1, so a collector must parse `%cI` and convert, never + subtract a fixed offset. Commits made outside a session create empty + intervals. --- @@ -202,13 +252,14 @@ b58a913 2026-07-31T02:19:34+02:00 T08 iter 2: Reveal, Resolve, End; … | D2 efficiency | C2 hub | cost of producing a number | ~0 (one API call) — number is an estimate, off by ~8× on output | measured | | D2 efficiency | C1 transcript | cost of producing a number | one file read, 5.1 MB, ~1 s; exact | measured | | D3 speed | C1 transcript | parse of a full session | 2,040 lines / 5.1 MB in <1 s in CPython | measured | -| D4 optionality | C4 billing API | authority | invoice-grade, zero attribution, needs admin key | measured (absent) | -| **Accuracy (the deciding axis)** | **C1 transcript** | agreement with billing counters | exact by construction | measured | +| D2 efficiency (**the deciding row**) | C1 transcript | **error against the billing counters** | **$0.00 — the transcript *is* the counter set; C2's error on the same work is $92.21 − $0.03 recorded ≈ 100%** | measured | +| D4 optionality | C4 billing API | authority | invoice-grade, zero attribution, admin key absent (asserted, not measured — used only to keep C4 as an optional check) | availability measured | -**Benchmark-to-beat for the collector:** reproduce **$92.21** for session -`8cbd5701` pinned at `2026-07-31T02:17:59Z`, from the committed price -sheet, with an unattributed remainder reported as its own line and -reconciliation asserted rather than assumed. +**Benchmark-to-beat for the collector:** reproduce **$92.87** for repo +`clay-borg` pinned at `2026-07-31T02:17:59Z` — as its two components, +**$92.21 main transcript + $0.66 subagent tree** — from the committed price +sheet, with the unattributed remainder reported as its own line (expected +**33%**, $30.32) and reconciliation asserted rather than assumed. --- @@ -218,11 +269,13 @@ reconciliation asserted rather than assumed. **C5 (git commits) supplies the attribution index C1 lacks.** C2 is the durable sink. C3 is eliminated. C4 is an optional external check. -The expected shape is therefore: parse transcripts → dedup by `requestId` → -price per message at its own model's rate from +The expected shape is therefore: enumerate **every** transcript for the repo +including the `subagents/` tree → dedup by `requestId` → price per message at +its own model's rate and **per cache TTL** from `benchmarks/baselines/model-prices.toml` → attribute to a task by joining -message timestamps against commit intervals → emit per-task cost and a -composition breakdown → push a lossy summary to C2. +`(prev_commit, this_commit]` intervals **within a `sessionId`** → emit +per-task cost, an unattributed-remainder line, and a composition breakdown → +push a lossy summary to C2. **What none of them do well — the surpass opportunity.** Every candidate reports *totals*. None reports **composition**, and composition is where @@ -252,7 +305,24 @@ correct and useless. reads it, so any total is a reading at an instant. Every committed number from this capability states its pin (timestamp or commit), and the collector takes a pin argument rather than defaulting to "all". -6. **Attribution across `/compact` and resumed sessions is unproven.** The - join is by wall-clock; a compaction inside a task boundary is fine, but - two sessions interleaved on one repo would mis-attribute. Not exercised - here — CB-WP-0001 ran in a single session. +6. **Concurrent sessions are not hypothetical — they are in this data.** + `f1eb1147` and `8cbd5701` overlap for 4 h 13 m on the same repo, ~$12 + inseparable by wall-clock. (The session that wrote the first draft of + this sentence *was* the overlap.) Attribution keys on `sessionId` first, + time second. Compaction inside a task is safe — it stays in one file and + one session. +7. **A third of spend has no task, structurally.** 19 of 33 commits carry no + task tag; $30.32 of $92.21. Any per-task cost table is a view over ~two + thirds of the money, and must say so wherever it is reported — the same + limit-with-the-number rule the coverage gate carries. +8. **The price sheet cannot express a time-boxed rate.** Sonnet's intro + price lives in a TOML *comment*, so a collector reading the sheet + silently uses the wrong number ($0.17 at the pin, 0.19%). Small now, and + the same class of defect this survey levels at C2: a schema that cannot + hold the fact it needs. Raised for T03. +9. **Hand-typed tables are the actual failure surface.** Every computed + figure in this survey reproduced to the cent under adversarial + re-derivation; three hand-written markdown tables did not (a group-size + row that failed its own checksum, a per-line count labelled as deduped, + a stale 87%). Numbers a tool can emit should be emitted by the tool. + Raised for T03 and T06.