coulomb/clay-borg
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

T02 review round: survey approved after 15 of 16 findings conceded

Browse source 
Adversarial review per InnerLoop Step 2 — separate agent, given only the
survey, one round. Challenge and response in history/.

The arithmetic held: every pinned figure and the whole three-method
correction table reproduced to the cent under ~20 independent spot
checks. The hand-written markdown tables did not. Three blocking
defects:

  - the dedup positive-control table failed its own checksum (group
    sizes summed to 348 groups / 663 lines, not 346 / 657; multi-line
    groups are 206, not 208) — the worst place for a slip, since that
    table is the sole evidence for the load-bearing dedup assumption
  - the multi-model table was labelled "deduped" while holding per-line
    counts, three lines below the paragraph condemning per-line counts
  - the $92.21 acceptance target excluded the subagent tree, which ran
    INSIDE the pin window. A collector correctly implementing the
    survey's own blind-spot finding would have failed the benchmark.
    Target is now $92.87, stated as its two components.

Folded in, each re-derived before conceding:

  - attribution must key on sessionId, not wall-clock: two sessions
    overlap 4h13m on this repo carrying ~$12 (one of them was this
    workplan's own background job)
  - only 14 of 33 commits name a task; the other 19 hold 33% of cost
  - cache writes bill per TTL — pricing the aggregate at 1h inflates
    the subagent by 43%
  - the "~87% cache" figure matched nothing; 64.6% of cost is cache
    read, 88.4% is all cache
  - fable is 35% of calls and 60% of dollars; opus is the inverse

Review cost ~$0.60 against a $92 pass, the second consecutive round
where that trade found something material. Evidence for CB-WP-0003 T03.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 08:43:32 +02:00
parent bf57b6f3a4
commit ac3ac2a8ce
4 changed files with 415 additions and 69 deletions
Download patch file Download diff file Expand all files Collapse all files

166
research/CB-RES-0002-cost-accounting.md
View file

@ -1,11 +1,11 @@
# CB-RES-0002: agentic cost accounting
capability: meta.loop.cost-accounting
status: draft
status: approved # adversarial review 2026-07-31: 16 findings, 15 conceded
tier: L (structural L, chaos d10=2 → no override)
runnable-baseline: invoked — every candidate below was exercised against the
CB-WP-0001 session on this machine, not cited
review-trail: history/260731-cost-accounting-research.md
review-trail: history/260731-cost-accounting-{research,challenge,response}.md
Survey of instruments that can attribute the USD cost of agentic work to a
unit of work, so that M-D2-CST (`specs/MetricsAndScenarios.md` §1a) becomes
@ -30,28 +30,42 @@ written to the transcript as *several* JSONL lines, split by content block
complete `usage` object**. Measured on the CB-WP-0001 transcript: 657
assistant lines carry only 346 distinct `requestId`s. Group sizes run 1–6:
| lines per requestId | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|
| groups | 140 | 120 | 76 | 6 | 5 | 1 |
| lines per requestId | 1 | 2 | 3 | 4 | 5 | 6 | total |
|---|---|---|---|---|---|---|---|
| groups | 140 | 120 | 74 | 6 | 5 | 1 | **346** |
| lines | 140 | 240 | 222 | 24 | 25 | 6 | **657** |
Positive control on the dedup: across all 208 multi-line groups the `usage`
object is byte-identical (208/208 identical, 0 differing), and no group
mixes models. The duplication is a transcript-format artifact, not repeated
billing. Summing per line inflates by ≈1.9×.
Both rows are checksummed against the file: groups sum to 346, lines to 657.
Positive control on the dedup: across all 206 multi-line groups the `usage`
object is byte-identical (206/206 identical, 0 differing), and no group
mixes models. Every assistant line carries a `requestId` — there is no null
key for `setdefault` to collapse. The duplication is a transcript-format
artifact, not repeated billing. Summing per line inflates by ≈1.9×.
**Error 2 — single-model pricing on a multi-model session.** The session ran
three models, not one:
three models, not one. Both columns are shown because the gap between them
*is* error 1:
| model | API responses (deduped) |
|---|---|
| claude-opus-5 | 382 lines → majority |
| claude-fable-5 | 250 lines |
| claude-sonnet-5 | 24 lines |
| `<synthetic>` | 1 (no usage; an error placeholder) |
| model | JSONL lines | API responses (deduped) | pinned cost |
|---|---|---|---|
| claude-opus-5 | 382 | 213 | $36.19 (39%) |
| claude-fable-5 | 250 | 118 | **$55.50 (60%)** |
| claude-sonnet-5 | 24 | 14 | $0.52 (0.6%) |
| `<synthetic>` | 1 | 1 | $0.00 |
| **total** | **657** | **346** | |
Pricing everything at Fable 5's $10/$50 overstates the Opus and Sonnet
majority. §1a *already required* per-model pricing; the Purpose section did
not apply its own rule.
§1a *already required* per-model pricing; the Purpose section did not apply
its own rule. Note the inversion, which matters more than the correction:
**Fable is 35% of the calls and 60% of the dollars; Opus is 60% of the calls
and 39% of the dollars.** A count-majority is not a cost-majority — the
error this document is about, one level down.
The `<synthetic>` entry is an error placeholder carrying a valid `requestId`
and a complete **all-zero** `usage` object, not a missing one. Its dollar
impact is exactly $0.00, but a collector guarding on `if not usage` and one
guarding on `if model not in prices` take different branches; T04 must name
which.
**Corrected totals** for the same transcript, all three methods run over
the identical unpinned line set so the methods are comparable:
@ -75,7 +89,16 @@ acceptance target is therefore pinned by timestamp:
| cache read | 80,453,702 tok → $59.59 |
| cache write 1h | 1,672,854 tok → $21.95 |
| input | 676 tok → $0.00 |
| **total** | **$92.21** — 88.4% cache, 256:1 cache-read:output |
| main transcript | **$92.21** — 88.4% cache, 256:1 cache-read:output |
| + subagent tree (7 responses, ran 23:11–23:14Z, inside the pin) | $0.66 |
| **TRUE TOTAL** | **$92.87** |
The subagent line is not a footnote. C1's blind-spot finding says a
collector reading only the main file under-reports; a target of $92.21 would
have been hit only by a collector *with* that blind spot, and failed by a
correct one. The acceptance target is **$92.87, stated as its two
components**, so a collector reading one tree is diagnosed rather than
merely failed.
The reported figure was **~2.7× the real cost**. This is the fourth
instance of the harness-does-nothing error class from
@ -103,21 +126,36 @@ Assistant lines carry `message.usage` with exact billing counters:
- **Accuracy:** exact — these are the counters the invoice is computed from.
There is no sampling or rounding.
- **Verified non-issue:** `usage.iterations[]` is a sub-breakdown, not an
additional charge. Checked all 654 usage-bearing lines: the iteration
additional charge. Every assistant line carries `usage`; the iteration
outputs sum exactly to the top-level `output_tokens` in every case, and
no message had more than one iteration. Summing `iterations` *instead of*
the top-level fields is safe; summing *both* would double-count.
- **Cache-write rates are per-TTL and must not be aggregated.**
`cache_creation_input_tokens` equals `ephemeral_5m + ephemeral_1h` in all
775 usage lines across both transcripts, and the two bill at different
multipliers (1.25 vs 2.0). Pricing the top-level aggregate at a single
rate is a silent error: the subagent's writes are **entirely 5m** (37,467
tokens), and pricing them at 1h inflates that transcript by **+43%**
($0.658 → $0.939). The main session happens to be all-1h, so the pinned
total is insensitive — but fan-out passes are exactly where 5m dominates.
- **Survives compaction:** yes. `/compact` writes a summary message into the
same file (`isCompactSummary`, `compactMetadata`) and the session
continues; no usage is lost. Compaction is visible as an event, so its
cost is itself measurable.
- **Attribution:** none built in — a transcript is a flat message stream
with timestamps. It must be joined against an external time index.
- **Blind spot found:** subagent cost is **not** in the main transcript.
`isSidechain` is `false` on all 657 lines; subagent work lives in
`<session>/subagents/agent-*.jsonl` with an `agent-*.meta.json` naming
the agentType and model. CB-WP-0001 spawned one (the adversarial review).
A collector reading only the main file silently under-reports.
- **Blind spot 1 — subagents are a separate tree.** Subagent cost is **not**
in the main transcript. `isSidechain` is `false` on all 657 lines;
subagent work lives in `<session>/subagents/agent-*.jsonl` with an
`agent-*.meta.json` naming the agentType and model. CB-WP-0001 spawned one
(the adversarial review). A collector reading only the main file silently
under-reports.
- **Blind spot 2 — one repo, several transcripts, overlapping in time.**
The project directory holds more than one session. `f1eb1147` overlaps
`8cbd5701` for **4 h 13 m**, carrying ~$6 on each side — ~$12 that no
wall-clock join can separate, beginning 14 seconds after the pin. A
collector must therefore key attribution on **`sessionId`, not only time**,
and must enumerate every transcript for the repo rather than one file.
### C2 — Custodian State Hub token API
@ -133,9 +171,9 @@ tokens_in 362,000 tokens_out 39,100 event_count 7 by model: claude-fable-5
that is natively *about* the unit of work.
- **Accuracy:** poor, and structurally so. Three independent defects:
1. **The schema has no cache fields.** `tokens_in`/`tokens_out` cannot
represent the finding this workplan exists to report. Cache reads are
~87% of real spend here and the hub cannot express them at any
fidelity.
represent the finding this workplan exists to report. Cache read alone
is **64.6% of cost** and all cache is **88.4%**; the hub cannot express
either at any fidelity.
2. **The recorded numbers are estimates.** 7 events for 9 tasks, at
round figures — the skill's Tier-3 heuristic (1000/500) and Tier-1
eyeball estimates. Against a deduped transcript output of 318,230,
@ -185,12 +223,24 @@ a09d76f 2026-07-31T02:14:34+02:00 T08 iter 1: scenario runner executes; …
b58a913 2026-07-31T02:19:34+02:00 T08 iter 2: Reveal, Resolve, End; …
```
- Boundaries are 3–15 minutes apart across CB-WP-0001 — finer than a task.
- **Interval convention:** *ending-at-commit*, `(prev_commit, this_commit]`.
This is the only convention consistent with commit-at-end-of-task; under
the alternative every task's cost shifts one interval.
- Spacing over the 33 pinned commits: min 0.1, p50 **6.9**, p90 17.7, max
**36.8** minutes. Usually finer than a task; the 37-minute max is not.
- Durable, versioned, and free; requires no change to how work is done.
- **Known hazards:** commit timestamps are local (`+02:00`), transcript
timestamps are UTC — a naive join is off by the offset. Work before the
first commit and after the last has no enclosing interval. Commits made
outside a session (or by `fix-consistency`) create empty intervals.
- **Coverage is the real limit, and it is sized:** only **14 of 33** commits
name a task (`T##`) in the subject. The other 19 — `chore(consistency)`,
`CI:`, `AM-4:`, workplan additions — hold **$30.32 of $92.21, or 33% of
cost**. Attribution to a *task* therefore covers two-thirds of spend at
best; the remainder is real work that must be reported as its own line,
not discarded.
- **Boundary conditions verified clean:** 0 responses before the first
commit, 0 after the last, exactly 1 empty interval (`Initial commit`).
- **Known hazards:** offsets are not uniform — `git log` shows `+02:00` ×34
and `+00:00` ×1, so a collector must parse `%cI` and convert, never
subtract a fixed offset. Commits made outside a session create empty
intervals.
---
@ -202,13 +252,14 @@ b58a913 2026-07-31T02:19:34+02:00 T08 iter 2: Reveal, Resolve, End; …
| D2 efficiency | C2 hub | cost of producing a number | ~0 (one API call) — number is an estimate, off by ~8× on output | measured |
| D2 efficiency | C1 transcript | cost of producing a number | one file read, 5.1 MB, ~1 s; exact | measured |
| D3 speed | C1 transcript | parse of a full session | 2,040 lines / 5.1 MB in <1 s in CPython | measured |
| D4 optionality | C4 billing API | authority | invoice-grade, zero attribution, needs admin key | measured (absent) |
| **Accuracy (the deciding axis)** | **C1 transcript** | agreement with billing counters | exact by construction | measured |
| D2 efficiency (**the deciding row**) | C1 transcript | **error against the billing counters** | **$0.00 — the transcript *is* the counter set; C2's error on the same work is $92.21 − $0.03 recorded ≈ 100%** | measured |
| D4 optionality | C4 billing API | authority | invoice-grade, zero attribution, admin key absent (asserted, not measured — used only to keep C4 as an optional check) | availability measured |
**Benchmark-to-beat for the collector:** reproduce **$92.21** for session
`8cbd5701` pinned at `2026-07-31T02:17:59Z`, from the committed price
sheet, with an unattributed remainder reported as its own line and
reconciliation asserted rather than assumed.
**Benchmark-to-beat for the collector:** reproduce **$92.87** for repo
`clay-borg` pinned at `2026-07-31T02:17:59Z` — as its two components,
**$92.21 main transcript + $0.66 subagent tree** — from the committed price
sheet, with the unattributed remainder reported as its own line (expected
**33%**, $30.32) and reconciliation asserted rather than assumed.
---
@ -218,11 +269,13 @@ reconciliation asserted rather than assumed.
**C5 (git commits) supplies the attribution index C1 lacks.** C2 is the
durable sink. C3 is eliminated. C4 is an optional external check.
The expected shape is therefore: parse transcripts → dedup by `requestId` →
price per message at its own model's rate from
The expected shape is therefore: enumerate **every** transcript for the repo
including the `subagents/` tree → dedup by `requestId` → price per message at
its own model's rate and **per cache TTL** from
`benchmarks/baselines/model-prices.toml` → attribute to a task by joining
message timestamps against commit intervals → emit per-task cost and a
composition breakdown → push a lossy summary to C2.
`(prev_commit, this_commit]` intervals **within a `sessionId`** → emit
per-task cost, an unattributed-remainder line, and a composition breakdown →
push a lossy summary to C2.
**What none of them do well — the surpass opportunity.** Every candidate
reports *totals*. None reports **composition**, and composition is where
@ -252,7 +305,24 @@ correct and useless.
reads it, so any total is a reading at an instant. Every committed
number from this capability states its pin (timestamp or commit), and
the collector takes a pin argument rather than defaulting to "all".
6. **Attribution across `/compact` and resumed sessions is unproven.** The
join is by wall-clock; a compaction inside a task boundary is fine, but
two sessions interleaved on one repo would mis-attribute. Not exercised
here — CB-WP-0001 ran in a single session.
6. **Concurrent sessions are not hypothetical — they are in this data.**
`f1eb1147` and `8cbd5701` overlap for 4 h 13 m on the same repo, ~$12
inseparable by wall-clock. (The session that wrote the first draft of
this sentence *was* the overlap.) Attribution keys on `sessionId` first,
time second. Compaction inside a task is safe — it stays in one file and
one session.
7. **A third of spend has no task, structurally.** 19 of 33 commits carry no
task tag; $30.32 of $92.21. Any per-task cost table is a view over ~two
thirds of the money, and must say so wherever it is reported — the same
limit-with-the-number rule the coverage gate carries.
8. **The price sheet cannot express a time-boxed rate.** Sonnet's intro
price lives in a TOML *comment*, so a collector reading the sheet
silently uses the wrong number ($0.17 at the pin, 0.19%). Small now, and
the same class of defect this survey levels at C2: a schema that cannot
hold the fact it needs. Raised for T03.
9. **Hand-typed tables are the actual failure surface.** Every computed
figure in this survey reproduced to the cent under adversarial
re-derivation; three hand-written markdown tables did not (a group-size
row that failed its own checksum, a per-line count labelled as deduped,
a stale 87%). Numbers a tool can emit should be emitted by the tool.
Raised for T03 and T06.