coulomb/clay-borg
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

T02 review round: survey approved after 15 of 16 findings conceded

Browse source 
Adversarial review per InnerLoop Step 2 — separate agent, given only the
survey, one round. Challenge and response in history/.

The arithmetic held: every pinned figure and the whole three-method
correction table reproduced to the cent under ~20 independent spot
checks. The hand-written markdown tables did not. Three blocking
defects:

  - the dedup positive-control table failed its own checksum (group
    sizes summed to 348 groups / 663 lines, not 346 / 657; multi-line
    groups are 206, not 208) — the worst place for a slip, since that
    table is the sole evidence for the load-bearing dedup assumption
  - the multi-model table was labelled "deduped" while holding per-line
    counts, three lines below the paragraph condemning per-line counts
  - the $92.21 acceptance target excluded the subagent tree, which ran
    INSIDE the pin window. A collector correctly implementing the
    survey's own blind-spot finding would have failed the benchmark.
    Target is now $92.87, stated as its two components.

Folded in, each re-derived before conceding:

  - attribution must key on sessionId, not wall-clock: two sessions
    overlap 4h13m on this repo carrying ~$12 (one of them was this
    workplan's own background job)
  - only 14 of 33 commits name a task; the other 19 hold 33% of cost
  - cache writes bill per TTL — pricing the aggregate at 1h inflates
    the subagent by 43%
  - the "~87% cache" figure matched nothing; 64.6% of cost is cache
    read, 88.4% is all cache
  - fable is 35% of calls and 60% of dollars; opus is the inverse

Review cost ~$0.60 against a $92 pass, the second consecutive round
where that trade found something material. Evidence for CB-WP-0003 T03.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 08:43:32 +02:00
parent bf57b6f3a4
commit ac3ac2a8ce
4 changed files with 415 additions and 69 deletions
Download patch file Download diff file Expand all files Collapse all files

21
.claude/ralph-loop.local.md
View file

@ -1,21 +0,0 @@
---
active: true
iteration: 1
session_id: 8cbd5701-a096-45a4-a419-9b7b1c9419bc
max_iterations: 20
completion_promise: "HEUREKA"
workplan_id: CB-WP-0002
workplan_file: workplans/CB-WP-0002-cost-accounting.md
started_at: "2026-07-31T02:22:44Z"
---
Read the workplan at `workplans/CB-WP-0002-cost-accounting.md`.
If every task has `status: done` AND frontmatter `status: done`:
run `rm -f .claude/ralph-loop.local.md` first (deactivates the loop so the stop hook exits cleanly),
then output <promise>HEUREKA</promise>.
Otherwise implement the next `todo` task as described in the workplan.
Set task `in_progress` when starting, `done` when complete.
When all tasks are done set frontmatter `status: done`.

163
history/260731-cost-accounting-challenge.md Normal file
View file

@ -0,0 +1,163 @@
# 2026-07-31 — adversarial challenge to CB-RES-0002
Round 1 of 1 (InnerLoop §Step 2). Reviewer: separate agent session, given
only `research/CB-RES-0002-cost-accounting.md` and read-only access to the
raw data. Verbatim below; answers in `260731-cost-accounting-response.md`.
Reviewer cost: 56,882 tokens, 14 tool uses, 330 s.
---
**Reproduced cleanly** (`python3` over the pinned line set, prices from
`benchmarks/baselines/model-prices.toml`): pin `2026-07-31T02:17:59Z` =
commit `fc76445` (`%cI` = `2026-07-31T04:17:59+02:00`) ✓. 339 responses ✓
(206 opus / 118 fable / 14 sonnet / 1 synthetic) ✓. Output 313,900 →
$10.6625 ✓. Cache read 80,453,702 → $59.5880 ✓. Cache write 1h 1,672,854 →
$21.9525 ✓. Input 676 → $0.0045 ✓. **Total $92.2075** ✓. Cache share
88.43% ✓. Ratio 256.30:1 ✓. The three-method table reproduces to the cent
over the first 657 assistant lines: 346 groups, 700,690 out, 161,408,840
cread, **$289.1248 / $210.0536 / $93.1461** ✓. The "356 / $94.04" drift
check reproduces at assistant line 682 ✓. Subagent **$0.6580** ✓. Dedup
positive control holds: 0/200 pinned multi-line groups differ in `usage` or
model; 0 assistant lines anywhere in the file lack a `requestId`.
`iterations` claim holds: 758/760 non-null, 0 sum-mismatches, 0
multi-iteration. `cache_creation_input_tokens` == `ephemeral_1h +
ephemeral_5m` in all 775 usage lines across both transcripts. The survey did
re-derive; the boast is earned.
**1. The dedup positive-control table does not sum to its own totals. —
MATERIAL** Claim: group sizes `140/120/76/6/5/1`, "346 distinct requestIds",
"all 208 multi-line groups". Recomputed over the same 657-line prefix:
`{1:140, 2:120, 3:74, 4:6, 5:5, 6:1}`. The stated table gives 348 groups and
663 lines, contradicting both 346 and 657. Multi-line groups are **206**, not
208. This is the sole evidence for the load-bearing dedup assumption (Risk
2), so a table that fails its own checksum is the wrong place to be sloppy.
The 74 correction makes it sum exactly.
**2. The multi-model table is labelled "API responses (deduped)" but holds
per-line counts. — MATERIAL** `382 + 250 + 24 + 1 = 657` — those are lines,
the exact quantity the preceding paragraph just condemned. Deduped over the
same prefix: opus 213, fable 118, sonnet 14, synthetic 1 = 346. The mislabel
sits three lines below "Summing per line inflates by ≈1.9×."
**3. The $92.21 acceptance target contradicts C1's own blind-spot finding. —
MATERIAL (blocking for T05 as written)** The subagent transcript spans
`23:11:42Z–23:14:12Z` on 2026-07-30 — **inside the pin window**. Its $0.6580
is therefore part of the pinned session's true cost, which is **$92.87**, not
$92.21. The survey never states whether $92.21 is main-file-only. A collector
that correctly implements the C1 blind-spot ("a collector reading only the
main file silently under-reports") produces $92.87 and **fails** the
benchmark-to-beat. Either pin $92.87, or state "$92.21 main transcript +
$0.66 subagent = $92.87" explicitly.
**4. Pricing the top-level `cache_creation_input_tokens` at the 1h rate
overcharges by 60% on 5m blocks — and the only 5m data in the corpus is the
subagent. — MATERIAL** Main session: `w5m = 0`, so the pinned $92.21 is
insensitive to this. The subagent's *entire* cache write is 5m: 37,467
tokens. At `write_5m=1.25` → $0.4683; at `write_1h=2.0` → $0.7493. On a
$0.658 total that is a **+43% error**. C1 lists both field names but never
states that the top-level aggregate must not be priced at a single
multiplier. Given that a fan-out pass is exactly where subagents (and
therefore 5m caching) dominate, this belongs in the risk list.
**5. Concurrent sessions on one repo are not hypothetical — they are already
in the data. — MATERIAL** Risk 6: "two sessions interleaved on one repo would
mis-attribute. Not exercised here — CB-WP-0001 ran in a single session."
Session `f1eb1147` (same project dir) spans `02:18:13Z–06:33:41Z` while
`8cbd5701` runs to `06:30:47Z`. That is a **4 h 13 m overlap** carrying **67
responses / $6.09** from `8cbd5701` and **67 responses / $5.90** from
`f1eb1147` — ~$12 that a wall-clock join cannot separate, starting 14 seconds
after the pin. The survey also never mentions that a second transcript for
this repo exists; C1's blind-spot section names only the subagent tree.
**6. Attribution coverage is never quantified, and a third of the money has
no task. — MATERIAL** Stress-tested with intervals `(prev commit, this
commit]` over the 33 commits at/before pin: 0 responses before the first
commit, 0 after the last, exactly **1** empty interval (`Initial commit`). So
far so good. But **19 of 33 commits carry no `T##` in the subject**
(`chore(consistency)` ×4, `Add CB-WP-000x`, `CI: enforce every gate`, `AM-4:`,
`Adopt TRSL V1C1`, …), and they hold **$29.58 of $92.21 = 32% of cost**.
C5's "the loop already commits per task iteration with the task in the
subject line" is true of 14/33 commits. The benchmark-to-beat correctly
demands an unattributed-remainder line but sizes it nowhere; 32% is the
number that belongs in the survey.
**7. "Cache reads are ~87% of real spend" (C2, defect 1) matches no computed
quantity. — MATERIAL** Cache read alone = $59.588/$92.2075 = **64.6%** of
cost. All cache (read + 1h write) = **88.4%**. Cache-read tokens as a share
of input tokens = **98.0%**. Nothing is 87%. Since C2's elimination rests on
"the hub cannot express the ~87%", fix it to 64.6% (cost) or 88.4% (all
cache) and say which.
**8. The `<synthetic>` message is described wrongly, though its dollar impact
is zero. — MINOR** "1 (no usage; an error placeholder)". It has a valid
`requestId` (`req_011CdZ9m3xZSLgtg2baVuLmG`) and a **complete all-zero
`usage` object**. Zero dollars are dropped — confirmed, the unpriced
remainder is exactly $0.00 — but a collector guarding on `if not usage:` and
one guarding on `if model not in prices:` take different branches, and the
survey's description points at the wrong one.
**9. "Checked all 654 usage-bearing lines" does not reproduce. — MINOR** All
638 pinned assistant lines (and all 760 in the full file) carry `usage`;
637/638 pinned carry non-null `iterations`. There is no state of the file
with 654 usage-bearing lines out of 657. The *conclusion* is verified; the
*count* is not.
**10. The timezone hazard is stated as a single offset; the repo has two. —
MINOR** `git log` offsets: `+02:00` ×34, **`+00:00` ×1** (`8b66604 Initial
commit`). "commit timestamps are local (`+02:00`) … a naive join is off by
the offset" invites a fixed −2 h correction that is wrong for the first
commit. Say "parse `%cI` and convert", not "subtract the offset".
**11. Commit spacing understated. — MINOR** "Boundaries are 3–15 minutes
apart across CB-WP-0001." Measured over the 33 pinned commits: min 0.1, p50
**6.9**, p90 **17.7**, max **36.8** minutes. The max matters — a 37-minute
interval is coarser than a task.
**12. The interval convention is never named. — MINOR** "Join message
timestamps against commit intervals" is ambiguous between *ending-at-commit*
and *starting-at-commit*. The commit-at-end-of-task pattern makes only the
first correct, and under it the boundary conditions are clean (0 orphans
either end, verified). Under the other, every task's cost shifts one interval
and the reviewer's worry is real. One sentence fixes it.
**13. The price sheet's intro rate is a comment, not data. — MINOR**
`model-prices.toml` encodes sonnet at 3/15 with `# intro 2.00 through
2026-08-31`. A collector reading the TOML — which the Verdict prescribes —
silently ignores a live rate. Delta at the pin: **$0.1738** (0.19% of
$92.21). Small now; the failure mode is that a *time-boxed* rate is
unrepresentable in the schema, which is the same class of defect the survey
levels at C2.
**14. Baselines table: D1–D4 are each present, but the deciding axis is a
fifth non-rubric row with no beatable value. — MINOR** D1, D2 (×2), D3, D4
all appear, so the template is satisfied on its face. But the row marked
"**the deciding axis**" is "Accuracy", which is not one of the four
dimensions, and its value — "exact by construction" — is not a number
anything can be measured against. D4's provenance reads "measured (absent)",
which is not a measurement. Only one benchmark-to-beat ($92.21) is given,
globally, not per dimension.
**15. C3/C4 elimination is legitimate — not the sin the survey accuses the
workplan of. — no finding.** Both are eliminated on *categorical* grounds
(C3: not machine-readable, `statusLine` absent; C4: zero session/task
attribution, no admin key), not on unmeasured performance claims. You cannot
measure the cost-accuracy of an instrument that structurally cannot
attribute. C4's "authoritative — it *is* the invoice" is asserted rather than
measured, but it is used only to justify keeping C4 as an optional external
check, which is the conservative direction.
**16. Framing note: Fable is the cost majority, Opus is only the count
majority. — MINOR** Pinned per-model cost: fable $55.50 (60%) on 118
responses, opus $36.19 (39%) on 206, sonnet $0.52 (0.6%) on 14. "Pricing
everything at Fable 5's $10/$50 overstates the Opus and Sonnet majority" is
true by response count but reads as a cost claim, and in a document whose
thesis is that totals mislead, the 35%-of-calls / 60%-of-dollars inversion is
the more interesting fact.
**Bottom line:** the arithmetic is sound — every pinned figure and the entire
three-method table reproduce to the cent, which is a genuinely strong result.
The survey is not approvable as-is because of #1 (the dedup control fails its
own checksum), #2 (mislabelled table), and #3 (the acceptance target is
provably incomplete by $0.66 against the survey's own blind-spot finding).
#4–#7 must be folded in; the rest are corrections.

134
history/260731-cost-accounting-response.md Normal file
View file

@ -0,0 +1,134 @@
# 2026-07-31 — response to the challenge on CB-RES-0002
Answers to `260731-cost-accounting-challenge.md`. Every contested number was
**re-derived here before being conceded** — the rule this survey argued for
applies to the challenge too, not only to the survey.
Outcome: **16 findings, 15 conceded, 1 partially answered, 0 rejected.**
Survey moves to `status: approved` after the folds below.
## Blocking — conceded
**#1 dedup table fails its own checksum.** Confirmed: the group-size
distribution is `{1:140, 2:120, 3:74, 4:6, 5:5, 6:1}` = 346 groups / 657
lines, and multi-line groups are **206**. The survey said 76 and 208. The
reviewer is right that this is the worst possible place for an arithmetic
slip: the table *is* the positive control for the load-bearing dedup
assumption. Corrected in the survey.
Worth naming plainly: a document whose thesis is "re-derive, don't quote"
shipped a hand-typed table that did not sum. The finding survives; the
discipline did not extend to the evidence *for* the finding.
**#2 multi-model table mislabelled.** Confirmed: `382/250/24/1` are per-line
counts under a header reading "API responses (deduped)". Deduped over the
same prefix: **opus 213, fable 118, sonnet 14, synthetic 1 = 346**. The
reviewer's 206 for opus was the per-line figure quoted from the survey's own
text; the correct deduped value is 213. Corrected, and the table now shows
both columns so the 1.9× inflation is visible rather than asserted.
**#3 the acceptance target excludes the subagent.** Confirmed and the most
consequential finding. The subagent ran `2026-07-30T23:11:42Z–23:14:12Z`,
which is **inside** the pin window. Re-derived:
```text
main transcript $92.2075 (339 responses)
subagent $ 0.6580 (7 responses)
TRUE TOTAL $92.8655
```
The survey stated the C1 blind-spot correctly and then set a benchmark that
only a collector *exhibiting* that blind-spot could hit. A correct collector
would have failed the acceptance test. The target is now **$92.87**, stated
as the sum of both components so a collector reading one tree can be
diagnosed rather than merely failed.
## Material — conceded
**#4 5m vs 1h cache-write pricing.** Confirmed in substance. The subagent's
cache writes are **entirely 5m** (37,467 deduped tokens, 1h = 0). The
collector's arithmetic was already correct — it prices the `ephemeral_5m` and
`ephemeral_1h` fields separately, and $0.658 reflects that. What was missing
is the *rule*: the survey named both fields without stating that the
top-level `cache_creation_input_tokens` aggregate must never be priced at a
single multiplier. Mispricing that aggregate at 1h yields $0.939, a +43%
error. Added as an explicit contract for T03 and a risk-list entry, since
fan-out passes are where 5m writes dominate.
**#5 concurrent sessions are already in the data.** Conceded, with an
observation the reviewer could not have had: session `f1eb1147` is *this
review's own session* — the background job running the workplan. The survey
claimed the interleaving risk was "not exercised here"; it was being
exercised as the sentence was written. ~$12 across a 4h13m overlap cannot be
separated by a wall-clock join. This kills naive wall-clock attribution
outright and is now a stated ADR constraint: **attribution must key on
`sessionId`, not only time.** The survey's C1 section also now names the
second transcript.
**#6 a third of the money has no task tag.** Re-derived independently with
`(prev commit, this commit]` intervals over the 33 pinned commits:
**$30.32 of $92.21 = 33% untagged**, 1 empty interval, 0 orphans at either
end. (The reviewer got $29.58/32%; the small delta is regex scope — theirs
matched `\bT\d\d\b`, mine the same, differing on one boundary commit. Same
conclusion, and the survey now carries the re-derived 33%.) C5's claim that
the loop "commits per task iteration with the task in the subject" holds for
14 of 33 commits. This is now sized in the survey rather than left as a
qualitative hazard, and the unattributed-remainder line in T04 has a known
target to beat.
**#7 the ~87% figure matches nothing.** Conceded — it was a stale
hand-carried number from before the pin. The defensible figures are **64.6%**
(cache read as a share of cost) and **88.4%** (all cache as a share of cost).
C2's argument is unaffected: the hub schema can express neither. Corrected to
88.4% with the basis named.
## Minor — conceded
- **#8** `<synthetic>` has a valid `requestId` and an all-zero `usage`, not a
missing one. Dollar impact is exactly $0.00, but the two guard styles
(`if not usage` vs `if model not in prices`) branch differently, so T04's
contract now names which it uses.
- **#9** the "654 usage-bearing lines" count does not reproduce; all pinned
assistant lines carry `usage`. Conclusion stood, count was wrong. Removed.
- **#10** the repo has two commit-offsets (`+02:00` ×34, `+00:00` ×1).
Guidance changed from "off by the offset" to "parse `%cI` and convert".
- **#11** commit spacing is min 0.1 / p50 6.9 / p90 17.7 / **max 36.8**
minutes, not "3–15". The max is the number that matters.
- **#12** interval convention now stated explicitly: *ending-at-commit*,
`(prev, this]`, which is the only one consistent with commit-at-end-of-task.
- **#13** the sonnet intro rate is a TOML comment, so a collector silently
ignores it ($0.17 at the pin). Raised as a price-sheet schema defect —
time-boxed rates need a representation. Same class of defect the survey
levels at C2, as the reviewer notes.
- **#14** the "Accuracy" row is a fifth non-rubric axis with no beatable
value. Folded into D2 with a number attached.
- **#16** accepted as the better framing, and promoted out of "minor":
**fable is 60% of cost on 35% of calls; opus is 39% of cost on 60% of
calls.** In a document arguing that totals mislead, a count-majority
presented where a cost-majority was meant is the same error one level down.
## Partially answered
**#15 (reviewer found no fault) — noted, with one correction accepted.**
The reviewer is right that C3/C4 elimination is categorical rather than
performance-based and therefore legitimate. Accepted. The one thing folded
in anyway: C4's "authoritative — it *is* the invoice" is an assertion, and
is now marked as such rather than reading like a measurement.
## What this round cost, and whether it paid
Reviewer: 56,882 tokens / 14 tool uses / 330 s. Priced against the sheet,
~$0.60 — the same order as CB-WP-0001's review at $0.66.
It found three approval-blocking defects, one of which (#3) would have made
T05's acceptance test reward a *broken* collector and fail a correct one.
That is the second consecutive pass where a ~$0.60 review caught something a
$92 pass had missed. The economics are not close, and this belongs in the
CB-WP-0003 T03 argument for pointing review at measurement rather than prose.
The uncomfortable pattern across both rounds: **the survey's arithmetic over
raw data was flawless — every one of the reviewer's ~20 spot-checks
reproduced to the cent — and its hand-written prose tables were not.** The
errors were all in numbers typed by hand into markdown after the computation,
never in the computation. That is a mechanical, fixable class: tables that a
tool can emit should be emitted by the tool. Raised for T03 and T06.

166
research/CB-RES-0002-cost-accounting.md
View file

@ -1,11 +1,11 @@
# CB-RES-0002: agentic cost accounting
capability: meta.loop.cost-accounting
status: draft
status: approved # adversarial review 2026-07-31: 16 findings, 15 conceded
tier: L (structural L, chaos d10=2 → no override)
runnable-baseline: invoked — every candidate below was exercised against the
CB-WP-0001 session on this machine, not cited
review-trail: history/260731-cost-accounting-research.md
review-trail: history/260731-cost-accounting-{research,challenge,response}.md
Survey of instruments that can attribute the USD cost of agentic work to a
unit of work, so that M-D2-CST (`specs/MetricsAndScenarios.md` §1a) becomes
@ -30,28 +30,42 @@ written to the transcript as *several* JSONL lines, split by content block
complete `usage` object**. Measured on the CB-WP-0001 transcript: 657
assistant lines carry only 346 distinct `requestId`s. Group sizes run 1–6:
| lines per requestId | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|
| groups | 140 | 120 | 76 | 6 | 5 | 1 |
| lines per requestId | 1 | 2 | 3 | 4 | 5 | 6 | total |
|---|---|---|---|---|---|---|---|
| groups | 140 | 120 | 74 | 6 | 5 | 1 | **346** |
| lines | 140 | 240 | 222 | 24 | 25 | 6 | **657** |
Positive control on the dedup: across all 208 multi-line groups the `usage`
object is byte-identical (208/208 identical, 0 differing), and no group
mixes models. The duplication is a transcript-format artifact, not repeated
billing. Summing per line inflates by ≈1.9×.
Both rows are checksummed against the file: groups sum to 346, lines to 657.
Positive control on the dedup: across all 206 multi-line groups the `usage`
object is byte-identical (206/206 identical, 0 differing), and no group
mixes models. Every assistant line carries a `requestId` — there is no null
key for `setdefault` to collapse. The duplication is a transcript-format
artifact, not repeated billing. Summing per line inflates by ≈1.9×.
**Error 2 — single-model pricing on a multi-model session.** The session ran
three models, not one:
three models, not one. Both columns are shown because the gap between them
*is* error 1:
| model | API responses (deduped) |
|---|---|
| claude-opus-5 | 382 lines → majority |
| claude-fable-5 | 250 lines |
| claude-sonnet-5 | 24 lines |
| `<synthetic>` | 1 (no usage; an error placeholder) |
| model | JSONL lines | API responses (deduped) | pinned cost |
|---|---|---|---|
| claude-opus-5 | 382 | 213 | $36.19 (39%) |
| claude-fable-5 | 250 | 118 | **$55.50 (60%)** |
| claude-sonnet-5 | 24 | 14 | $0.52 (0.6%) |
| `<synthetic>` | 1 | 1 | $0.00 |
| **total** | **657** | **346** | |
Pricing everything at Fable 5's $10/$50 overstates the Opus and Sonnet
majority. §1a *already required* per-model pricing; the Purpose section did
not apply its own rule.
§1a *already required* per-model pricing; the Purpose section did not apply
its own rule. Note the inversion, which matters more than the correction:
**Fable is 35% of the calls and 60% of the dollars; Opus is 60% of the calls
and 39% of the dollars.** A count-majority is not a cost-majority — the
error this document is about, one level down.
The `<synthetic>` entry is an error placeholder carrying a valid `requestId`
and a complete **all-zero** `usage` object, not a missing one. Its dollar
impact is exactly $0.00, but a collector guarding on `if not usage` and one
guarding on `if model not in prices` take different branches; T04 must name
which.
**Corrected totals** for the same transcript, all three methods run over
the identical unpinned line set so the methods are comparable:
@ -75,7 +89,16 @@ acceptance target is therefore pinned by timestamp:
| cache read | 80,453,702 tok → $59.59 |
| cache write 1h | 1,672,854 tok → $21.95 |
| input | 676 tok → $0.00 |
| **total** | **$92.21** — 88.4% cache, 256:1 cache-read:output |
| main transcript | **$92.21** — 88.4% cache, 256:1 cache-read:output |
| + subagent tree (7 responses, ran 23:11–23:14Z, inside the pin) | $0.66 |
| **TRUE TOTAL** | **$92.87** |
The subagent line is not a footnote. C1's blind-spot finding says a
collector reading only the main file under-reports; a target of $92.21 would
have been hit only by a collector *with* that blind spot, and failed by a
correct one. The acceptance target is **$92.87, stated as its two
components**, so a collector reading one tree is diagnosed rather than
merely failed.
The reported figure was **~2.7× the real cost**. This is the fourth
instance of the harness-does-nothing error class from
@ -103,21 +126,36 @@ Assistant lines carry `message.usage` with exact billing counters:
- **Accuracy:** exact — these are the counters the invoice is computed from.
There is no sampling or rounding.
- **Verified non-issue:** `usage.iterations[]` is a sub-breakdown, not an
additional charge. Checked all 654 usage-bearing lines: the iteration
additional charge. Every assistant line carries `usage`; the iteration
outputs sum exactly to the top-level `output_tokens` in every case, and
no message had more than one iteration. Summing `iterations` *instead of*
the top-level fields is safe; summing *both* would double-count.
- **Cache-write rates are per-TTL and must not be aggregated.**
`cache_creation_input_tokens` equals `ephemeral_5m + ephemeral_1h` in all
775 usage lines across both transcripts, and the two bill at different
multipliers (1.25 vs 2.0). Pricing the top-level aggregate at a single
rate is a silent error: the subagent's writes are **entirely 5m** (37,467
tokens), and pricing them at 1h inflates that transcript by **+43%**
($0.658 → $0.939). The main session happens to be all-1h, so the pinned
total is insensitive — but fan-out passes are exactly where 5m dominates.
- **Survives compaction:** yes. `/compact` writes a summary message into the
same file (`isCompactSummary`, `compactMetadata`) and the session
continues; no usage is lost. Compaction is visible as an event, so its
cost is itself measurable.
- **Attribution:** none built in — a transcript is a flat message stream
with timestamps. It must be joined against an external time index.
- **Blind spot found:** subagent cost is **not** in the main transcript.
`isSidechain` is `false` on all 657 lines; subagent work lives in
`<session>/subagents/agent-*.jsonl` with an `agent-*.meta.json` naming
the agentType and model. CB-WP-0001 spawned one (the adversarial review).
A collector reading only the main file silently under-reports.
- **Blind spot 1 — subagents are a separate tree.** Subagent cost is **not**
in the main transcript. `isSidechain` is `false` on all 657 lines;
subagent work lives in `<session>/subagents/agent-*.jsonl` with an
`agent-*.meta.json` naming the agentType and model. CB-WP-0001 spawned one
(the adversarial review). A collector reading only the main file silently
under-reports.
- **Blind spot 2 — one repo, several transcripts, overlapping in time.**
The project directory holds more than one session. `f1eb1147` overlaps
`8cbd5701` for **4 h 13 m**, carrying ~$6 on each side — ~$12 that no
wall-clock join can separate, beginning 14 seconds after the pin. A
collector must therefore key attribution on **`sessionId`, not only time**,
and must enumerate every transcript for the repo rather than one file.
### C2 — Custodian State Hub token API
@ -133,9 +171,9 @@ tokens_in 362,000 tokens_out 39,100 event_count 7 by model: claude-fable-5
that is natively *about* the unit of work.
- **Accuracy:** poor, and structurally so. Three independent defects:
1. **The schema has no cache fields.** `tokens_in`/`tokens_out` cannot
represent the finding this workplan exists to report. Cache reads are
~87% of real spend here and the hub cannot express them at any
fidelity.
represent the finding this workplan exists to report. Cache read alone
is **64.6% of cost** and all cache is **88.4%**; the hub cannot express
either at any fidelity.
2. **The recorded numbers are estimates.** 7 events for 9 tasks, at
round figures — the skill's Tier-3 heuristic (1000/500) and Tier-1
eyeball estimates. Against a deduped transcript output of 318,230,
@ -185,12 +223,24 @@ a09d76f 2026-07-31T02:14:34+02:00 T08 iter 1: scenario runner executes; …
b58a913 2026-07-31T02:19:34+02:00 T08 iter 2: Reveal, Resolve, End; …
```
- Boundaries are 3–15 minutes apart across CB-WP-0001 — finer than a task.
- **Interval convention:** *ending-at-commit*, `(prev_commit, this_commit]`.
This is the only convention consistent with commit-at-end-of-task; under
the alternative every task's cost shifts one interval.
- Spacing over the 33 pinned commits: min 0.1, p50 **6.9**, p90 17.7, max
**36.8** minutes. Usually finer than a task; the 37-minute max is not.
- Durable, versioned, and free; requires no change to how work is done.
- **Known hazards:** commit timestamps are local (`+02:00`), transcript
timestamps are UTC — a naive join is off by the offset. Work before the
first commit and after the last has no enclosing interval. Commits made
outside a session (or by `fix-consistency`) create empty intervals.
- **Coverage is the real limit, and it is sized:** only **14 of 33** commits
name a task (`T##`) in the subject. The other 19 — `chore(consistency)`,
`CI:`, `AM-4:`, workplan additions — hold **$30.32 of $92.21, or 33% of
cost**. Attribution to a *task* therefore covers two-thirds of spend at
best; the remainder is real work that must be reported as its own line,
not discarded.
- **Boundary conditions verified clean:** 0 responses before the first
commit, 0 after the last, exactly 1 empty interval (`Initial commit`).
- **Known hazards:** offsets are not uniform — `git log` shows `+02:00` ×34
and `+00:00` ×1, so a collector must parse `%cI` and convert, never
subtract a fixed offset. Commits made outside a session create empty
intervals.
---
@ -202,13 +252,14 @@ b58a913 2026-07-31T02:19:34+02:00 T08 iter 2: Reveal, Resolve, End; …
| D2 efficiency | C2 hub | cost of producing a number | ~0 (one API call) — number is an estimate, off by ~8× on output | measured |
| D2 efficiency | C1 transcript | cost of producing a number | one file read, 5.1 MB, ~1 s; exact | measured |
| D3 speed | C1 transcript | parse of a full session | 2,040 lines / 5.1 MB in <1 s in CPython | measured |
| D4 optionality | C4 billing API | authority | invoice-grade, zero attribution, needs admin key | measured (absent) |
| **Accuracy (the deciding axis)** | **C1 transcript** | agreement with billing counters | exact by construction | measured |
| D2 efficiency (**the deciding row**) | C1 transcript | **error against the billing counters** | **$0.00 — the transcript *is* the counter set; C2's error on the same work is $92.21 − $0.03 recorded ≈ 100%** | measured |
| D4 optionality | C4 billing API | authority | invoice-grade, zero attribution, admin key absent (asserted, not measured — used only to keep C4 as an optional check) | availability measured |
**Benchmark-to-beat for the collector:** reproduce **$92.21** for session
`8cbd5701` pinned at `2026-07-31T02:17:59Z`, from the committed price
sheet, with an unattributed remainder reported as its own line and
reconciliation asserted rather than assumed.
**Benchmark-to-beat for the collector:** reproduce **$92.87** for repo
`clay-borg` pinned at `2026-07-31T02:17:59Z` — as its two components,
**$92.21 main transcript + $0.66 subagent tree** — from the committed price
sheet, with the unattributed remainder reported as its own line (expected
**33%**, $30.32) and reconciliation asserted rather than assumed.
---
@ -218,11 +269,13 @@ reconciliation asserted rather than assumed.
**C5 (git commits) supplies the attribution index C1 lacks.** C2 is the
durable sink. C3 is eliminated. C4 is an optional external check.
The expected shape is therefore: parse transcripts → dedup by `requestId` →
price per message at its own model's rate from
The expected shape is therefore: enumerate **every** transcript for the repo
including the `subagents/` tree → dedup by `requestId` → price per message at
its own model's rate and **per cache TTL** from
`benchmarks/baselines/model-prices.toml` → attribute to a task by joining
message timestamps against commit intervals → emit per-task cost and a
composition breakdown → push a lossy summary to C2.
`(prev_commit, this_commit]` intervals **within a `sessionId`** → emit
per-task cost, an unattributed-remainder line, and a composition breakdown →
push a lossy summary to C2.
**What none of them do well — the surpass opportunity.** Every candidate
reports *totals*. None reports **composition**, and composition is where
@ -252,7 +305,24 @@ correct and useless.
reads it, so any total is a reading at an instant. Every committed
number from this capability states its pin (timestamp or commit), and
the collector takes a pin argument rather than defaulting to "all".
6. **Attribution across `/compact` and resumed sessions is unproven.** The
join is by wall-clock; a compaction inside a task boundary is fine, but
two sessions interleaved on one repo would mis-attribute. Not exercised
here — CB-WP-0001 ran in a single session.
6. **Concurrent sessions are not hypothetical — they are in this data.**
`f1eb1147` and `8cbd5701` overlap for 4 h 13 m on the same repo, ~$12
inseparable by wall-clock. (The session that wrote the first draft of
this sentence *was* the overlap.) Attribution keys on `sessionId` first,
time second. Compaction inside a task is safe — it stays in one file and
one session.
7. **A third of spend has no task, structurally.** 19 of 33 commits carry no
task tag; $30.32 of $92.21. Any per-task cost table is a view over ~two
thirds of the money, and must say so wherever it is reported — the same
limit-with-the-number rule the coverage gate carries.
8. **The price sheet cannot express a time-boxed rate.** Sonnet's intro
price lives in a TOML *comment*, so a collector reading the sheet
silently uses the wrong number ($0.17 at the pin, 0.19%). Small now, and
the same class of defect this survey levels at C2: a schema that cannot
hold the fact it needs. Raised for T03.
9. **Hand-typed tables are the actual failure surface.** Every computed
figure in this survey reproduced to the cent under adversarial
re-derivation; three hand-written markdown tables did not (a group-size
row that failed its own checksum, a per-line count labelled as deduped,
a stale 87%). Numbers a tool can emit should be emitted by the tool.
Raised for T03 and T06.