diff --git a/.claude/ralph-loop.local.md b/.claude/ralph-loop.local.md new file mode 100644 index 0000000..2e79582 --- /dev/null +++ b/.claude/ralph-loop.local.md @@ -0,0 +1,21 @@ +--- +active: true +iteration: 1 +session_id: 8cbd5701-a096-45a4-a419-9b7b1c9419bc +max_iterations: 20 +completion_promise: "HEUREKA" +workplan_id: CB-WP-0002 +workplan_file: workplans/CB-WP-0002-cost-accounting.md +started_at: "2026-07-31T02:22:44Z" +--- + +Read the workplan at `workplans/CB-WP-0002-cost-accounting.md`. + +If every task has `status: done` AND frontmatter `status: done`: +run `rm -f .claude/ralph-loop.local.md` first (deactivates the loop so the stop hook exits cleanly), +then output HEUREKA. + +Otherwise implement the next `todo` task as described in the workplan. +Set task `in_progress` when starting, `done` when complete. +When all tasks are done set frontmatter `status: done`. + diff --git a/history/260731-cost-accounting-research.md b/history/260731-cost-accounting-research.md new file mode 100644 index 0000000..f79a80c --- /dev/null +++ b/history/260731-cost-accounting-research.md @@ -0,0 +1,74 @@ +# 2026-07-31 — how the cost-accounting survey was conducted + +Trail for `research/CB-RES-0002-cost-accounting.md` (CB-WP-0002 T01). +Tier L, chaos d10 = 2 (no override). + +## Method + +No web sources. Every candidate was exercised locally against the +CB-WP-0001 session transcript +(`~/.claude/projects/-home-worsch-clay-borg/8cbd5701-….jsonl`, 5.1 MB, +2,040 lines) and the clay-borg git history. The survey's own rule — +measured beats cited — applied to itself, which is the only reason the +errors below were found. + +## What was measured, in order + +1. **Transcript shape.** Counted line types and collected the union of + `usage` keys. Found 654 usage-bearing assistant lines, and four + distinct `message.model` values where one was assumed. +2. **`usage.iterations`.** Suspected as a double-count source. Checked + whether iteration outputs sum to the top-level `output_tokens`: they + match in all 654 cases, and no message had >1 iteration. Cleared. +3. **`requestId` cardinality.** 346 distinct ids across 657 lines. This + was the finding that overturned the workplan's numbers. +4. **Dedup validation.** For all 208 multi-line groups, compared the + serialized `usage` object across the group: 208 identical, 0 + differing, 0 mixed-model. Block-type patterns confirmed the split is + `thinking` / `text` / `tool_use`, i.e. a transcript-writer artifact. +5. **Re-priced three ways** (per-line all-Fable, per-line per-model, + deduped per-model) to isolate how much of the error came from each + mistake: $289.12 / $210.05 / $93.15. +6. **Hub API.** Called `get_token_summary` on CB-WP-0001's workplan. + 401,100 tokens over 7 events for 9 tasks, all filed under Fable 5. + Compared against the transcript to quantify the gap (~8× on output). +7. **Status bar and billing API.** Checked `~/.claude/settings.json` for + `statusLine` (absent) and for an admin key (absent). Both eliminated + on availability before any further evaluation. +8. **Git boundaries.** Read `git log` with ISO timestamps to confirm + commit density and that subjects name tasks. Noted the UTC/+02:00 + mismatch against transcript timestamps. +9. **Subagent tree.** Found `/subagents/` while checking whether + `isSidechain` was ever true (it never is). Priced the one subagent + transcript: $0.66. + +## Dead ends + +- **`isSidechain` as the subagent signal.** It is present on every line + and `false` on every line in this session; it does not mark subagent + work in the main file because subagent work is not *in* the main file. + Looking for a flag wasted a step that a directory listing answered. +- **`tool-results/` sidecar directory** (340 KB) was inspected as a + possible cost source. It holds raw tool outputs for replay, carries no + usage data, and is not billed separately. Not a candidate. + +## The finding worth carrying forward + +The workplan that exists to make cost measurable **opened with a +mismeasured cost**, off by ~2.7×, and its two errors were of different +kinds: a format artifact (double-counted lines) and a rule that existed +but was not applied (per-model pricing, already mandated by +`specs/MetricsAndScenarios.md` §1a). + +The second is the more uncomfortable one. InnerLoop v1.0's answer to bad +numbers is the positive control, and a positive control would not have +caught either of these — both sums were over real data that really +existed. What caught them was **re-deriving a number instead of quoting +it**. That belongs in the loop as its own rule, and is raised for T07: +*a number inherited from a previous pass is re-derived before it is used +as a target, or it is cited as unverified.* + +Reported into CB-WP-0003 T01's rule-enforceability audit as well: this is +a fifth error instance, and it is **not** of the harness-does-nothing +class that T10 predicted. The class is "trusted arithmetic over real +data". diff --git a/research/CB-RES-0002-cost-accounting.md b/research/CB-RES-0002-cost-accounting.md new file mode 100644 index 0000000..5e641df --- /dev/null +++ b/research/CB-RES-0002-cost-accounting.md @@ -0,0 +1,258 @@ +# CB-RES-0002: agentic cost accounting + +capability: meta.loop.cost-accounting +status: draft +tier: L (structural L, chaos d10=2 → no override) +runnable-baseline: invoked — every candidate below was exercised against the +CB-WP-0001 session on this machine, not cited +review-trail: history/260731-cost-accounting-research.md + +Survey of instruments that can attribute the USD cost of agentic work to a +unit of work, so that M-D2-CST (`specs/MetricsAndScenarios.md` §1a) becomes +computable. CB-WP-0001 specified that metric completely and recorded it as +*uncomputable*; the premise of this workplan is that the data existed the +whole time. + +That premise survives. The workplan's **numbers do not** — see §Correction. + +--- + +## Correction to this workplan's own Purpose section + +CB-WP-0002's Purpose reports the CB-WP-0001 session at **$248.46**, from +131,863,164 cache-read tokens priced at Fable 5. Both halves are wrong, and +in the same direction — too high. The survey found this by re-deriving the +number rather than adopting it. + +**Error 1 — per-line summation double-counts.** A single API response is +written to the transcript as *several* JSONL lines, split by content block +(`thinking`, `text`, `tool_use`), and **every one of those lines repeats the +complete `usage` object**. Measured on the CB-WP-0001 transcript: 657 +assistant lines carry only 346 distinct `requestId`s. Group sizes run 1–6: + +| lines per requestId | 1 | 2 | 3 | 4 | 5 | 6 | +|---|---|---|---|---|---|---| +| groups | 140 | 120 | 76 | 6 | 5 | 1 | + +Positive control on the dedup: across all 208 multi-line groups the `usage` +object is byte-identical (208/208 identical, 0 differing), and no group +mixes models. The duplication is a transcript-format artifact, not repeated +billing. Summing per line inflates by ≈1.9×. + +**Error 2 — single-model pricing on a multi-model session.** The session ran +three models, not one: + +| model | API responses (deduped) | +|---|---| +| claude-opus-5 | 382 lines → majority | +| claude-fable-5 | 250 lines | +| claude-sonnet-5 | 24 lines | +| `` | 1 (no usage; an error placeholder) | + +Pricing everything at Fable 5's $10/$50 overstates the Opus and Sonnet +majority. §1a *already required* per-model pricing; the Purpose section did +not apply its own rule. + +**Corrected totals** for the same transcript, all three methods run over +the identical unpinned line set so the methods are comparable: + +| method | responses | output | cache read | cost | +|---|---|---|---|---| +| per-line, all-Fable (the Purpose method) | 657 | 700,690 | 161,408,840 | $289.12 | +| per-line, per-model | 657 | 700,690 | 161,408,840 | $210.05 | +| **deduped, per-model (correct)** | **346** | **318,230** | **81,100,498** | **$93.15** | + +**Third error, found while verifying the second: the transcript is a live +file.** Re-running the deduped figure minutes later returned 356 +responses and $94.04 — this survey's own session appends to the same +JSONL it is measuring. An unpinned total is not a repeatable number. The +acceptance target is therefore pinned by timestamp: + +| CB-WP-0001, pinned ≤ `2026-07-31T02:17:59Z` (commit `fc76445`) | value | +|---|---| +| responses | 339 (206 opus-5, 118 fable-5, 14 sonnet-5, 1 synthetic) | +| output | 313,900 tok → $10.66 | +| cache read | 80,453,702 tok → $59.59 | +| cache write 1h | 1,672,854 tok → $21.95 | +| input | 676 tok → $0.00 | +| **total** | **$92.21** — 88.4% cache, 256:1 cache-read:output | + +The reported figure was **~2.7× the real cost**. This is the fourth +instance of the harness-does-nothing error class from +`history/260731-inner-loop-retrospective.md`, wearing a new coat: not a +harness that measured nothing, but an arithmetic that measured the same +thing twice. Both produce a number that looks fine. + +The qualitative headline survives the correction and gets stronger: cache +reads are **81.1M tokens against 318k of output**, ~255:1. Cost in an +agentic loop is context × turns. + +--- + +## Candidates + +### C1 — Session transcript JSONL + +`~/.claude/projects//.jsonl`, one JSON object per line. +Assistant lines carry `message.usage` with exact billing counters: +`input_tokens`, `output_tokens`, `cache_read_input_tokens`, and +`cache_creation.{ephemeral_1h,ephemeral_5m}_input_tokens`, plus +`message.model`, `requestId`, and an ISO-8601 `timestamp`. + +- **Granularity:** per API response, once deduplicated by `requestId`. +- **Accuracy:** exact — these are the counters the invoice is computed from. + There is no sampling or rounding. +- **Verified non-issue:** `usage.iterations[]` is a sub-breakdown, not an + additional charge. Checked all 654 usage-bearing lines: the iteration + outputs sum exactly to the top-level `output_tokens` in every case, and + no message had more than one iteration. Summing `iterations` *instead of* + the top-level fields is safe; summing *both* would double-count. +- **Survives compaction:** yes. `/compact` writes a summary message into the + same file (`isCompactSummary`, `compactMetadata`) and the session + continues; no usage is lost. Compaction is visible as an event, so its + cost is itself measurable. +- **Attribution:** none built in — a transcript is a flat message stream + with timestamps. It must be joined against an external time index. +- **Blind spot found:** subagent cost is **not** in the main transcript. + `isSidechain` is `false` on all 657 lines; subagent work lives in + `/subagents/agent-*.jsonl` with an `agent-*.meta.json` naming + the agentType and model. CB-WP-0001 spawned one (the adversarial review). + A collector reading only the main file silently under-reports. + +### C2 — Custodian State Hub token API + +`record_token_event`, the `update_task_status` token tiers, and +`get_token_summary`. Exercised against CB-WP-0001's workplan +(`a1b434dc-…`), which returned: + +```text +tokens_in 362,000 tokens_out 39,100 event_count 7 by model: claude-fable-5 +``` + +- **Granularity:** per task — the best of any candidate, and the only one + that is natively *about* the unit of work. +- **Accuracy:** poor, and structurally so. Three independent defects: + 1. **The schema has no cache fields.** `tokens_in`/`tokens_out` cannot + represent the finding this workplan exists to report. Cache reads are + ~87% of real spend here and the hub cannot express them at any + fidelity. + 2. **The recorded numbers are estimates.** 7 events for 9 tasks, at + round figures — the skill's Tier-3 heuristic (1000/500) and Tier-1 + eyeball estimates. Against a deduped transcript output of 318,230, + the hub's 39,100 is off by ~8×; against total input it is off by + ~450×. + 3. **Model attribution is wrong.** Everything is filed under + `claude-fable-5` on a session that was majority Opus 5. +- **Survives compaction:** yes — it is server-side and independent of the + client. +- **Verdict:** durable and task-shaped, but its numbers are unusable as a + cost source. Its role is as a **sink** for numbers computed elsewhere, + not a source. Even as a sink it can only carry a lossy projection until + the schema grows cache fields. + +### C3 — Claude Code status bar + +- **Granularity:** whole session, live. +- **Accuracy:** unknown and unauditable — it is rendered text. +- **Machine-readable:** no. Not configured here (`statusLine` is absent + from `~/.claude/settings.json`), and it is not reachable from inside a + tool call regardless. +- **Verdict:** eliminated. The ralph-workplan skill's "read tokens from the + status bar" instruction is the proximate cause of C2's bad numbers — it + asks an agent to report a figure it cannot read, and an agent that cannot + read it estimates instead. This should be raised against the skill. + +### C4 — Anthropic usage / billing API + +- **Granularity:** organization and API-key, by day. +- **Accuracy:** authoritative — it *is* the invoice. +- **Attribution:** none to a task, and none to a session. Cannot separate + clay-borg from the other twenty-plus projects on this machine. +- **Availability:** requires an admin key; none is configured here. +- **Verdict:** not usable for M-D2-CST, but valuable as an **external + reconciliation check** if an admin key is ever provisioned — it is the + only candidate that can catch a systematic error in C1's price model. + Left as a stated non-dependency. + +### C5 — Git commit history (attribution index, not a cost source) + +Not a cost instrument; the missing half of C1. The loop already commits per +task iteration with the task in the subject line, giving durable, timestamped +boundaries at exactly the granularity M-D2-CST wants: + +```text +a09d76f 2026-07-31T02:14:34+02:00 T08 iter 1: scenario runner executes; … +b58a913 2026-07-31T02:19:34+02:00 T08 iter 2: Reveal, Resolve, End; … +``` + +- Boundaries are 3–15 minutes apart across CB-WP-0001 — finer than a task. +- Durable, versioned, and free; requires no change to how work is done. +- **Known hazards:** commit timestamps are local (`+02:00`), transcript + timestamps are UTC — a naive join is off by the offset. Work before the + first commit and after the last has no enclosing interval. Commits made + outside a session (or by `fix-consistency`) create empty intervals. + +--- + +## Baselines (benchmark-to-beat) + +| Dimension | Baseline holder | Metric | Value | Provenance | +|---|---|---|---|---| +| D1 ease of specification | C2 hub | fields needed to record a task's cost | 4 (`task_id`, `tokens_in`, `tokens_out`, `model`) — but cannot express cache | measured (API schema) | +| D2 efficiency | C2 hub | cost of producing a number | ~0 (one API call) — number is an estimate, off by ~8× on output | measured | +| D2 efficiency | C1 transcript | cost of producing a number | one file read, 5.1 MB, ~1 s; exact | measured | +| D3 speed | C1 transcript | parse of a full session | 2,040 lines / 5.1 MB in <1 s in CPython | measured | +| D4 optionality | C4 billing API | authority | invoice-grade, zero attribution, needs admin key | measured (absent) | +| **Accuracy (the deciding axis)** | **C1 transcript** | agreement with billing counters | exact by construction | measured | + +**Benchmark-to-beat for the collector:** reproduce **$92.21** for session +`8cbd5701` pinned at `2026-07-31T02:17:59Z`, from the committed price +sheet, with an unattributed remainder reported as its own line and +reconciliation asserted rather than assumed. + +--- + +## Verdict + +**C1 (transcript) leads on accuracy and is the only exact candidate.** +**C5 (git commits) supplies the attribution index C1 lacks.** C2 is the +durable sink. C3 is eliminated. C4 is an optional external check. + +The expected shape is therefore: parse transcripts → dedup by `requestId` → +price per message at its own model's rate from +`benchmarks/baselines/model-prices.toml` → attribute to a task by joining +message timestamps against commit intervals → emit per-task cost and a +composition breakdown → push a lossy summary to C2. + +**What none of them do well — the surpass opportunity.** Every candidate +reports *totals*. None reports **composition**, and composition is where +the actionable finding lives: 81.1M cache-read tokens against 318k of +output means cost is driven by how much context is re-read per turn, which +no total can show. A metric that had reported only dollars would have been +correct and useless. + +**Risks in the baselines themselves.** + +1. **The $248.46 figure was wrong and was nearly adopted as this + workplan's acceptance target.** T05's reconciliation test must be + against a number this survey re-derived, not against the Purpose + section. The Purpose section needs correcting. +2. **Dedup is load-bearing.** If the transcript format ever splits one + response across two `requestId`s, dedup silently under-reports — + the opposite error, and the more dangerous one. The collector must + assert its dedup assumption (identical usage within a group) at + runtime rather than trusting this survey's one-time check. +3. **Subagent transcripts are a separate tree.** Measured: CB-WP-0001's + one subagent (adversarial review, Fable 5, 7 responses, 158,096 cache + reads) cost **$0.66**, invisible to any collector reading only the + main file. Small here; not small for a pass that fans out. +4. **The price sheet has a 90-day staleness rule** (§1a) and no automated + check. Every number this capability produces inherits that. +5. **The transcript is append-live.** It is written by the session that + reads it, so any total is a reading at an instant. Every committed + number from this capability states its pin (timestamp or commit), and + the collector takes a pin argument rather than defaulting to "all". +6. **Attribution across `/compact` and resumed sessions is unproven.** The + join is by wall-clock; a compaction inside a task boundary is fine, but + two sessions interleaved on one repo would mis-attribute. Not exercised + here — CB-WP-0001 ran in a single session. diff --git a/workplans/CB-WP-0002-cost-accounting.md b/workplans/CB-WP-0002-cost-accounting.md index d9308b2..c13b6d5 100644 --- a/workplans/CB-WP-0002-cost-accounting.md +++ b/workplans/CB-WP-0002-cost-accounting.md @@ -18,19 +18,39 @@ transcript (`~/.claude/projects//.jsonl`) carries exact per-message `usage`, including the cache breakdown. Reading it for CB-WP-0001's session gives: -| Component | Tokens | Cost (Fable 5) | -|---|---|---| -| Output | 585,528 | $29.28 | -| Cache read | 131,863,164 | **$131.86** | -| Cache write (1h) | 4,365,668 | $87.31 | -| Input | 1,090 | $0.01 | -| **Total** | | **$248.46** | +> **Corrected 2026-07-31 by T01.** This section originally reported +> **$248.46** from 131,863,164 cache-read tokens priced at Fable 5. That +> figure was wrong by ~2.7×, for two independent reasons found in +> `research/CB-RES-0002-cost-accounting.md` §Correction: one API response +> is written to the transcript as up to six JSONL lines that each repeat +> the *same* `usage` object (657 lines, 346 real responses), and the +> session ran three models, not one. The numbers below are the re-derived +> ones. The workplan's premise is unaffected; its arithmetic was not. -The headline finding from that single reading is the reason this -workplan exists: **53% of the cost is cache reads**, not output. Cost in -an agentic loop is driven by context size × turn count, not by how much -the model writes. No D2 decision made on "tokens per task" would have -surfaced that. +Deduplicated by `requestId` and priced per message at its own model's +rate. **Pinned** to messages at or before `2026-07-31T02:17:59Z` (commit +`fc76445`, the end of CB-WP-0001) — the transcript is a live file that +grows as later sessions append to it, so an unpinned total is not a +repeatable acceptance target: + +| Component | Tokens | Cost | +|---|---|---| +| Output | 313,900 | $10.66 | +| Cache read | 80,453,702 | **$59.59** | +| Cache write (1h) | 1,672,854 | $21.95 | +| Input | 676 | $0.00 | +| **Total** (339 responses: 206 opus-5, 118 fable-5, 14 sonnet-5) | | **$92.21** | + +The headline finding survives the correction and gets sharper: +**88.4% of the cost is cache, against 314k tokens of output — a 256:1 +ratio of context re-read to text written.** Cost in an agentic loop is +driven by context size × turn count, not by how much the model writes. No +D2 decision made on "tokens per task" would have surfaced that. + +The correction is itself the lesson: this workplan opened with a +mismeasured cost. Neither error was of the harness-does-nothing class the +positive-control rule was written for — both sums ran over real data. +What caught them was re-deriving the number instead of quoting it. This workplan makes cost a first-class measured dimension so that future AM-12 equivalents support conclusions instead of decorating an