coulomb/clay-borg
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

T01: cost-accounting survey — and a 2.7x correction to our own numbers

Browse source 
Tier L (structural L, chaos d10=2). Every candidate exercised locally
against the CB-WP-0001 transcript rather than cited.

Verdict: session transcript JSONL is the only exact instrument; git
commits supply the attribution index it lacks; the State Hub is a
durable sink whose schema cannot express cache tokens at all; the
status bar and billing API are eliminated on availability.

Three errors found in this workplan's own Purpose section, all
overstating cost:

  - one API response is written as up to six JSONL lines, each
    repeating the same usage object (657 lines, 346 responses)
  - a multi-model session was priced entirely at Fable 5 rates,
    violating a per-model rule MetricsAndScenarios §1a already had
  - the transcript is a live file, so an unpinned total is not
    repeatable (346 -> 356 responses between two runs minutes apart)

CB-WP-0001 pinned at fc76445: $92.21, not $248.46. The qualitative
finding sharpens rather than weakens — 88.4% of spend is cache, at
256:1 cache-read to output tokens.

Neither error was of the harness-does-nothing class the positive
control was written for; both sums ran over real data. What caught
them was re-deriving a number instead of quoting it. Raised for T07
and for CB-WP-0003 T01.

Also measured: the one subagent (adversarial review) cost $0.66 and is
invisible to a collector reading only the main transcript.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 04:27:52 +02:00
parent fc76445aec
commit 060eb8cf6a
4 changed files with 385 additions and 12 deletions
Download patch file Download diff file Expand all files Collapse all files

21
.claude/ralph-loop.local.md Normal file
View file

@ -0,0 +1,21 @@
---
active: true
iteration: 1
session_id: 8cbd5701-a096-45a4-a419-9b7b1c9419bc
max_iterations: 20
completion_promise: "HEUREKA"
workplan_id: CB-WP-0002
workplan_file: workplans/CB-WP-0002-cost-accounting.md
started_at: "2026-07-31T02:22:44Z"
---
Read the workplan at `workplans/CB-WP-0002-cost-accounting.md`.
If every task has `status: done` AND frontmatter `status: done`:
run `rm -f .claude/ralph-loop.local.md` first (deactivates the loop so the stop hook exits cleanly),
then output <promise>HEUREKA</promise>.
Otherwise implement the next `todo` task as described in the workplan.
Set task `in_progress` when starting, `done` when complete.
When all tasks are done set frontmatter `status: done`.

74
history/260731-cost-accounting-research.md Normal file
View file

@ -0,0 +1,74 @@
# 2026-07-31 — how the cost-accounting survey was conducted
Trail for `research/CB-RES-0002-cost-accounting.md` (CB-WP-0002 T01).
Tier L, chaos d10 = 2 (no override).
## Method
No web sources. Every candidate was exercised locally against the
CB-WP-0001 session transcript
(`~/.claude/projects/-home-worsch-clay-borg/8cbd5701-….jsonl`, 5.1 MB,
2,040 lines) and the clay-borg git history. The survey's own rule —
measured beats cited — applied to itself, which is the only reason the
errors below were found.
## What was measured, in order
1. **Transcript shape.** Counted line types and collected the union of
`usage` keys. Found 654 usage-bearing assistant lines, and four
distinct `message.model` values where one was assumed.
2. **`usage.iterations`.** Suspected as a double-count source. Checked
whether iteration outputs sum to the top-level `output_tokens`: they
match in all 654 cases, and no message had >1 iteration. Cleared.
3. **`requestId` cardinality.** 346 distinct ids across 657 lines. This
was the finding that overturned the workplan's numbers.
4. **Dedup validation.** For all 208 multi-line groups, compared the
serialized `usage` object across the group: 208 identical, 0
differing, 0 mixed-model. Block-type patterns confirmed the split is
`thinking` / `text` / `tool_use`, i.e. a transcript-writer artifact.
5. **Re-priced three ways** (per-line all-Fable, per-line per-model,
deduped per-model) to isolate how much of the error came from each
mistake: $289.12 / $210.05 / $93.15.
6. **Hub API.** Called `get_token_summary` on CB-WP-0001's workplan.
401,100 tokens over 7 events for 9 tasks, all filed under Fable 5.
Compared against the transcript to quantify the gap (~8× on output).
7. **Status bar and billing API.** Checked `~/.claude/settings.json` for
`statusLine` (absent) and for an admin key (absent). Both eliminated
on availability before any further evaluation.
8. **Git boundaries.** Read `git log` with ISO timestamps to confirm
commit density and that subjects name tasks. Noted the UTC/+02:00
mismatch against transcript timestamps.
9. **Subagent tree.** Found `<session>/subagents/` while checking whether
`isSidechain` was ever true (it never is). Priced the one subagent
transcript: $0.66.
## Dead ends
- **`isSidechain` as the subagent signal.** It is present on every line
and `false` on every line in this session; it does not mark subagent
work in the main file because subagent work is not *in* the main file.
Looking for a flag wasted a step that a directory listing answered.
- **`tool-results/` sidecar directory** (340 KB) was inspected as a
possible cost source. It holds raw tool outputs for replay, carries no
usage data, and is not billed separately. Not a candidate.
## The finding worth carrying forward
The workplan that exists to make cost measurable **opened with a
mismeasured cost**, off by ~2.7×, and its two errors were of different
kinds: a format artifact (double-counted lines) and a rule that existed
but was not applied (per-model pricing, already mandated by
`specs/MetricsAndScenarios.md` §1a).
The second is the more uncomfortable one. InnerLoop v1.0's answer to bad
numbers is the positive control, and a positive control would not have
caught either of these — both sums were over real data that really
existed. What caught them was **re-deriving a number instead of quoting
it**. That belongs in the loop as its own rule, and is raised for T07:
*a number inherited from a previous pass is re-derived before it is used
as a target, or it is cited as unverified.*
Reported into CB-WP-0003 T01's rule-enforceability audit as well: this is
a fifth error instance, and it is **not** of the harness-does-nothing
class that T10 predicted. The class is "trusted arithmetic over real
data".

258
research/CB-RES-0002-cost-accounting.md Normal file
View file

@ -0,0 +1,258 @@
# CB-RES-0002: agentic cost accounting
capability: meta.loop.cost-accounting
status: draft
tier: L (structural L, chaos d10=2 → no override)
runnable-baseline: invoked — every candidate below was exercised against the
CB-WP-0001 session on this machine, not cited
review-trail: history/260731-cost-accounting-research.md
Survey of instruments that can attribute the USD cost of agentic work to a
unit of work, so that M-D2-CST (`specs/MetricsAndScenarios.md` §1a) becomes
computable. CB-WP-0001 specified that metric completely and recorded it as
*uncomputable*; the premise of this workplan is that the data existed the
whole time.
That premise survives. The workplan's **numbers do not** — see §Correction.
---
## Correction to this workplan's own Purpose section
CB-WP-0002's Purpose reports the CB-WP-0001 session at **$248.46**, from
131,863,164 cache-read tokens priced at Fable 5. Both halves are wrong, and
in the same direction — too high. The survey found this by re-deriving the
number rather than adopting it.
**Error 1 — per-line summation double-counts.** A single API response is
written to the transcript as *several* JSONL lines, split by content block
(`thinking`, `text`, `tool_use`), and **every one of those lines repeats the
complete `usage` object**. Measured on the CB-WP-0001 transcript: 657
assistant lines carry only 346 distinct `requestId`s. Group sizes run 1–6:
| lines per requestId | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|
| groups | 140 | 120 | 76 | 6 | 5 | 1 |
Positive control on the dedup: across all 208 multi-line groups the `usage`
object is byte-identical (208/208 identical, 0 differing), and no group
mixes models. The duplication is a transcript-format artifact, not repeated
billing. Summing per line inflates by ≈1.9×.
**Error 2 — single-model pricing on a multi-model session.** The session ran
three models, not one:
| model | API responses (deduped) |
|---|---|
| claude-opus-5 | 382 lines → majority |
| claude-fable-5 | 250 lines |
| claude-sonnet-5 | 24 lines |
| `<synthetic>` | 1 (no usage; an error placeholder) |
Pricing everything at Fable 5's $10/$50 overstates the Opus and Sonnet
majority. §1a *already required* per-model pricing; the Purpose section did
not apply its own rule.
**Corrected totals** for the same transcript, all three methods run over
the identical unpinned line set so the methods are comparable:
| method | responses | output | cache read | cost |
|---|---|---|---|---|
| per-line, all-Fable (the Purpose method) | 657 | 700,690 | 161,408,840 | $289.12 |
| per-line, per-model | 657 | 700,690 | 161,408,840 | $210.05 |
| **deduped, per-model (correct)** | **346** | **318,230** | **81,100,498** | **$93.15** |
**Third error, found while verifying the second: the transcript is a live
file.** Re-running the deduped figure minutes later returned 356
responses and $94.04 — this survey's own session appends to the same
JSONL it is measuring. An unpinned total is not a repeatable number. The
acceptance target is therefore pinned by timestamp:
| CB-WP-0001, pinned ≤ `2026-07-31T02:17:59Z` (commit `fc76445`) | value |
|---|---|
| responses | 339 (206 opus-5, 118 fable-5, 14 sonnet-5, 1 synthetic) |
| output | 313,900 tok → $10.66 |
| cache read | 80,453,702 tok → $59.59 |
| cache write 1h | 1,672,854 tok → $21.95 |
| input | 676 tok → $0.00 |
| **total** | **$92.21** — 88.4% cache, 256:1 cache-read:output |
The reported figure was **~2.7× the real cost**. This is the fourth
instance of the harness-does-nothing error class from
`history/260731-inner-loop-retrospective.md`, wearing a new coat: not a
harness that measured nothing, but an arithmetic that measured the same
thing twice. Both produce a number that looks fine.
The qualitative headline survives the correction and gets stronger: cache
reads are **81.1M tokens against 318k of output**, ~255:1. Cost in an
agentic loop is context × turns.
---
## Candidates
### C1 — Session transcript JSONL
`~/.claude/projects/<slug>/<session>.jsonl`, one JSON object per line.
Assistant lines carry `message.usage` with exact billing counters:
`input_tokens`, `output_tokens`, `cache_read_input_tokens`, and
`cache_creation.{ephemeral_1h,ephemeral_5m}_input_tokens`, plus
`message.model`, `requestId`, and an ISO-8601 `timestamp`.
- **Granularity:** per API response, once deduplicated by `requestId`.
- **Accuracy:** exact — these are the counters the invoice is computed from.
There is no sampling or rounding.
- **Verified non-issue:** `usage.iterations[]` is a sub-breakdown, not an
additional charge. Checked all 654 usage-bearing lines: the iteration
outputs sum exactly to the top-level `output_tokens` in every case, and
no message had more than one iteration. Summing `iterations` *instead of*
the top-level fields is safe; summing *both* would double-count.
- **Survives compaction:** yes. `/compact` writes a summary message into the
same file (`isCompactSummary`, `compactMetadata`) and the session
continues; no usage is lost. Compaction is visible as an event, so its
cost is itself measurable.
- **Attribution:** none built in — a transcript is a flat message stream
with timestamps. It must be joined against an external time index.
- **Blind spot found:** subagent cost is **not** in the main transcript.
`isSidechain` is `false` on all 657 lines; subagent work lives in
`<session>/subagents/agent-*.jsonl` with an `agent-*.meta.json` naming
the agentType and model. CB-WP-0001 spawned one (the adversarial review).
A collector reading only the main file silently under-reports.
### C2 — Custodian State Hub token API
`record_token_event`, the `update_task_status` token tiers, and
`get_token_summary`. Exercised against CB-WP-0001's workplan
(`a1b434dc-…`), which returned:
```text
tokens_in 362,000 tokens_out 39,100 event_count 7 by model: claude-fable-5
```
- **Granularity:** per task — the best of any candidate, and the only one
that is natively *about* the unit of work.
- **Accuracy:** poor, and structurally so. Three independent defects:
1. **The schema has no cache fields.** `tokens_in`/`tokens_out` cannot
represent the finding this workplan exists to report. Cache reads are
~87% of real spend here and the hub cannot express them at any
fidelity.
2. **The recorded numbers are estimates.** 7 events for 9 tasks, at
round figures — the skill's Tier-3 heuristic (1000/500) and Tier-1
eyeball estimates. Against a deduped transcript output of 318,230,
the hub's 39,100 is off by ~8×; against total input it is off by
~450×.
3. **Model attribution is wrong.** Everything is filed under
`claude-fable-5` on a session that was majority Opus 5.
- **Survives compaction:** yes — it is server-side and independent of the
client.
- **Verdict:** durable and task-shaped, but its numbers are unusable as a
cost source. Its role is as a **sink** for numbers computed elsewhere,
not a source. Even as a sink it can only carry a lossy projection until
the schema grows cache fields.
### C3 — Claude Code status bar
- **Granularity:** whole session, live.
- **Accuracy:** unknown and unauditable — it is rendered text.
- **Machine-readable:** no. Not configured here (`statusLine` is absent
from `~/.claude/settings.json`), and it is not reachable from inside a
tool call regardless.
- **Verdict:** eliminated. The ralph-workplan skill's "read tokens from the
status bar" instruction is the proximate cause of C2's bad numbers — it
asks an agent to report a figure it cannot read, and an agent that cannot
read it estimates instead. This should be raised against the skill.
### C4 — Anthropic usage / billing API
- **Granularity:** organization and API-key, by day.
- **Accuracy:** authoritative — it *is* the invoice.
- **Attribution:** none to a task, and none to a session. Cannot separate
clay-borg from the other twenty-plus projects on this machine.
- **Availability:** requires an admin key; none is configured here.
- **Verdict:** not usable for M-D2-CST, but valuable as an **external
reconciliation check** if an admin key is ever provisioned — it is the
only candidate that can catch a systematic error in C1's price model.
Left as a stated non-dependency.
### C5 — Git commit history (attribution index, not a cost source)
Not a cost instrument; the missing half of C1. The loop already commits per
task iteration with the task in the subject line, giving durable, timestamped
boundaries at exactly the granularity M-D2-CST wants:
```text
a09d76f 2026-07-31T02:14:34+02:00 T08 iter 1: scenario runner executes; …
b58a913 2026-07-31T02:19:34+02:00 T08 iter 2: Reveal, Resolve, End; …
```
- Boundaries are 3–15 minutes apart across CB-WP-0001 — finer than a task.
- Durable, versioned, and free; requires no change to how work is done.
- **Known hazards:** commit timestamps are local (`+02:00`), transcript
timestamps are UTC — a naive join is off by the offset. Work before the
first commit and after the last has no enclosing interval. Commits made
outside a session (or by `fix-consistency`) create empty intervals.
---
## Baselines (benchmark-to-beat)
| Dimension | Baseline holder | Metric | Value | Provenance |
|---|---|---|---|---|
| D1 ease of specification | C2 hub | fields needed to record a task's cost | 4 (`task_id`, `tokens_in`, `tokens_out`, `model`) — but cannot express cache | measured (API schema) |
| D2 efficiency | C2 hub | cost of producing a number | ~0 (one API call) — number is an estimate, off by ~8× on output | measured |
| D2 efficiency | C1 transcript | cost of producing a number | one file read, 5.1 MB, ~1 s; exact | measured |
| D3 speed | C1 transcript | parse of a full session | 2,040 lines / 5.1 MB in <1 s in CPython | measured |
| D4 optionality | C4 billing API | authority | invoice-grade, zero attribution, needs admin key | measured (absent) |
| **Accuracy (the deciding axis)** | **C1 transcript** | agreement with billing counters | exact by construction | measured |
**Benchmark-to-beat for the collector:** reproduce **$92.21** for session
`8cbd5701` pinned at `2026-07-31T02:17:59Z`, from the committed price
sheet, with an unattributed remainder reported as its own line and
reconciliation asserted rather than assumed.
---
## Verdict
**C1 (transcript) leads on accuracy and is the only exact candidate.**
**C5 (git commits) supplies the attribution index C1 lacks.** C2 is the
durable sink. C3 is eliminated. C4 is an optional external check.
The expected shape is therefore: parse transcripts → dedup by `requestId` →
price per message at its own model's rate from
`benchmarks/baselines/model-prices.toml` → attribute to a task by joining
message timestamps against commit intervals → emit per-task cost and a
composition breakdown → push a lossy summary to C2.
**What none of them do well — the surpass opportunity.** Every candidate
reports *totals*. None reports **composition**, and composition is where
the actionable finding lives: 81.1M cache-read tokens against 318k of
output means cost is driven by how much context is re-read per turn, which
no total can show. A metric that had reported only dollars would have been
correct and useless.
**Risks in the baselines themselves.**
1. **The $248.46 figure was wrong and was nearly adopted as this
workplan's acceptance target.** T05's reconciliation test must be
against a number this survey re-derived, not against the Purpose
section. The Purpose section needs correcting.
2. **Dedup is load-bearing.** If the transcript format ever splits one
response across two `requestId`s, dedup silently under-reports —
the opposite error, and the more dangerous one. The collector must
assert its dedup assumption (identical usage within a group) at
runtime rather than trusting this survey's one-time check.
3. **Subagent transcripts are a separate tree.** Measured: CB-WP-0001's
one subagent (adversarial review, Fable 5, 7 responses, 158,096 cache
reads) cost **$0.66**, invisible to any collector reading only the
main file. Small here; not small for a pass that fans out.
4. **The price sheet has a 90-day staleness rule** (§1a) and no automated
check. Every number this capability produces inherits that.
5. **The transcript is append-live.** It is written by the session that
reads it, so any total is a reading at an instant. Every committed
number from this capability states its pin (timestamp or commit), and
the collector takes a pin argument rather than defaulting to "all".
6. **Attribution across `/compact` and resumed sessions is unproven.** The
join is by wall-clock; a compaction inside a task boundary is fine, but
two sessions interleaved on one repo would mis-attribute. Not exercised
here — CB-WP-0001 ran in a single session.

44
workplans/CB-WP-0002-cost-accounting.md
View file

@ -18,19 +18,39 @@ transcript (`~/.claude/projects/<slug>/<session>.jsonl`) carries exact
per-message `usage`, including the cache breakdown. Reading it for
CB-WP-0001's session gives:
| Component | Tokens | Cost (Fable 5) |
|---|---|---|
| Output | 585,528 | $29.28 |
| Cache read | 131,863,164 | **$131.86** |
| Cache write (1h) | 4,365,668 | $87.31 |
| Input | 1,090 | $0.01 |
| **Total** | | **$248.46** |
> **Corrected 2026-07-31 by T01.** This section originally reported
> **$248.46** from 131,863,164 cache-read tokens priced at Fable 5. That
> figure was wrong by ~2.7×, for two independent reasons found in
> `research/CB-RES-0002-cost-accounting.md` §Correction: one API response
> is written to the transcript as up to six JSONL lines that each repeat
> the *same* `usage` object (657 lines, 346 real responses), and the
> session ran three models, not one. The numbers below are the re-derived
> ones. The workplan's premise is unaffected; its arithmetic was not.
The headline finding from that single reading is the reason this
workplan exists: **53% of the cost is cache reads**, not output. Cost in
an agentic loop is driven by context size × turn count, not by how much
the model writes. No D2 decision made on "tokens per task" would have
surfaced that.
Deduplicated by `requestId` and priced per message at its own model's
rate. **Pinned** to messages at or before `2026-07-31T02:17:59Z` (commit
`fc76445`, the end of CB-WP-0001) — the transcript is a live file that
grows as later sessions append to it, so an unpinned total is not a
repeatable acceptance target:
| Component | Tokens | Cost |
|---|---|---|
| Output | 313,900 | $10.66 |
| Cache read | 80,453,702 | **$59.59** |
| Cache write (1h) | 1,672,854 | $21.95 |
| Input | 676 | $0.00 |
| **Total** (339 responses: 206 opus-5, 118 fable-5, 14 sonnet-5) | | **$92.21** |
The headline finding survives the correction and gets sharper:
**88.4% of the cost is cache, against 314k tokens of output — a 256:1
ratio of context re-read to text written.** Cost in an agentic loop is
driven by context size × turn count, not by how much the model writes. No
D2 decision made on "tokens per task" would have surfaced that.
The correction is itself the lesson: this workplan opened with a
mismeasured cost. Neither error was of the harness-does-nothing class the
positive-control rule was written for — both sums ran over real data.
What caught them was re-deriving the number instead of quoting it.
This workplan makes cost a first-class measured dimension so that
future AM-12 equivalents support conclusions instead of decorating an