From 8e11fc412ee44aea0ae40f4d1aa9e4a44543546e Mon Sep 17 00:00:00 2001 From: tegwick Date: Fri, 31 Jul 2026 03:26:03 +0200 Subject: [PATCH] Amend CB-EV-0001; add CB-WP-0002 for cost accounting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AM-4: measured what each remediation option actually buys, rather than leaving one recommendation unquantified. serde_yaml optional removes 6 crates, not 5 — ryu belongs to that group, since serde_json now uses zmij for floats. Full ladder: -6 to 27, serde_json -4 more to 23, inlining SHA-256 -8 to 19, inlining ChaCha12 -4 to 15. Only reimplementing a primitive gets under 20, so the target is unreachable without undoing K5/K7. Also records that crate count compares badly across ecosystems, and offers the alternative the count is a proxy for: 307,317 lines of third-party source under audit against 3,398 of our own. AM-12: corrected from "uncomputable" to measured. The refusal to estimate was right; the claim that no instrument existed was wrong. Session transcripts carry exact per-message usage including the cache breakdown. This session cost $248.46 at Fable 5 rates, of which 53% is cache reads — cost is driven by context size times turn count, not by output volume. What is still missing is per-task attribution, since nothing marks task boundaries in a transcript. CB-WP-0002 makes cost measurable and attributable: survey the instruments, decide the attribution model by ADR, spec metrics that include cost composition rather than a bare total, build a collector whose positive control refuses to emit unreconciled numbers, and prove it by answering a question that could not be answered before. Co-Authored-By: Claude Opus 5 --- .claude/ralph-loop.local.md | 21 --- WORK-RECORDS.md | 16 +- evidence/CB-EV-0001-game-kernel.md | 65 ++++++--- workplans/CB-WP-0002-cost-accounting.md | 185 ++++++++++++++++++++++++ 4 files changed, 240 insertions(+), 47 deletions(-) delete mode 100644 .claude/ralph-loop.local.md create mode 100644 workplans/CB-WP-0002-cost-accounting.md diff --git a/.claude/ralph-loop.local.md b/.claude/ralph-loop.local.md deleted file mode 100644 index bf5f62a..0000000 --- a/.claude/ralph-loop.local.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -active: true -iteration: 3 -session_id: 8cbd5701-a096-45a4-a419-9b7b1c9419bc -max_iterations: 20 -completion_promise: "HEUREKA" -workplan_id: CB-WP-0001 -workplan_file: workplans/CB-WP-0001-inner-loop.md -started_at: "2026-07-31T00:08:23Z" ---- - -Read the workplan at `workplans/CB-WP-0001-inner-loop.md`. - -If every task has `status: done` AND frontmatter `status: done`: -run `rm -f .claude/ralph-loop.local.md` first (deactivates the loop so the stop hook exits cleanly), -then output HEUREKA. - -Otherwise implement the next `todo` task as described in the workplan. -Set task `in_progress` when starting, `done` when complete. -When all tasks are done set frontmatter `status: done`. - diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 29c0c1f..a65d958 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -8,13 +8,13 @@ | Kind | ID | Status | Lane | Source | | --- | --- | --- | --- | --- | -| workplan | CB-WP-0001 | active | — | workplans/CB-WP-0001-inner-loop.md | +| workplan | CB-WP-0001 | done | — | workplans/CB-WP-0001-inner-loop.md | | task | CB-WP-0001-T01 | done | — | workplans/CB-WP-0001-inner-loop.md | | task | CB-WP-0001-T02 | done | — | workplans/CB-WP-0001-inner-loop.md | -| task | CB-WP-0001-T03 | todo | — | workplans/CB-WP-0001-inner-loop.md | -| task | CB-WP-0001-T04 | todo | — | workplans/CB-WP-0001-inner-loop.md | -| task | CB-WP-0001-T05 | todo | — | workplans/CB-WP-0001-inner-loop.md | -| task | CB-WP-0001-T06 | todo | — | workplans/CB-WP-0001-inner-loop.md | -| task | CB-WP-0001-T07 | todo | — | workplans/CB-WP-0001-inner-loop.md | -| task | CB-WP-0001-T08 | todo | — | workplans/CB-WP-0001-inner-loop.md | -| task | CB-WP-0001-T09 | todo | — | workplans/CB-WP-0001-inner-loop.md | +| task | CB-WP-0001-T03 | done | — | workplans/CB-WP-0001-inner-loop.md | +| task | CB-WP-0001-T04 | done | — | workplans/CB-WP-0001-inner-loop.md | +| task | CB-WP-0001-T05 | done | — | workplans/CB-WP-0001-inner-loop.md | +| task | CB-WP-0001-T06 | done | — | workplans/CB-WP-0001-inner-loop.md | +| task | CB-WP-0001-T07 | done | — | workplans/CB-WP-0001-inner-loop.md | +| task | CB-WP-0001-T08 | done | — | workplans/CB-WP-0001-inner-loop.md | +| task | CB-WP-0001-T09 | done | — | workplans/CB-WP-0001-inner-loop.md | diff --git a/evidence/CB-EV-0001-game-kernel.md b/evidence/CB-EV-0001-game-kernel.md index f69af30..1706965 100644 --- a/evidence/CB-EV-0001-game-kernel.md +++ b/evidence/CB-EV-0001-game-kernel.md @@ -1,7 +1,9 @@ # CB-EV-0001 — GROUND game kernel: acceptance evidence Status: **T08 complete, with one acceptance metric not met (AM-4).** -Recorded: 2026-07-31 +Recorded: 2026-07-31. Amended 2026-07-31 — §4 gains measured savings per +remediation option, and §5 corrects AM-12 from "uncomputable" to +measured-at-session-level; see CB-WP-0002. Workplan: CB-WP-0001, task T08 Spec: `specs/GameKernel.md` §4 (AM-1..AM-12) Baseline: `research/CB-RES-0001-game-kernel.md`, measurements in @@ -23,7 +25,7 @@ Machine: WSL2, Linux 6.18.33.2-microsoft-standard-WSL2, rustc 1.97.1, | AM-8 lint | fmt + clippy clean | clean, `-D warnings` | **met** | | AM-10 foreign types | zero `HashMap`/`HashSet` | 0 | **met** | | AM-11 impl pairs | null + reference per port | 1 of 1 (`KernelRng`) | **met, narrow** | -| AM-12 cost log | present | §5 | **met** | +| AM-12 cost | per-task USD | $248.46 session; per-task pending | **partial** | AM-2, AM-3, AM-5 and AM-9 are not reported: see §6. @@ -121,8 +123,8 @@ Attribution: |---|---|---| | `sha2` (K7 state hashing) | sha2, digest, block-buffer, crypto-common, generic-array, typenum, cpufeatures, cfg-if | 8 | | serde derive chain | serde_derive, proc-macro2, quote, syn, unicode-ident | 5 | -| serde runtime + json | serde, serde_core, serde_json, itoa, ryu, memchr, zmij | 7 | -| `serde_yaml` (scenario files only) | serde_yaml, unsafe-libyaml, indexmap, hashbrown, equivalent | 5 | +| serde runtime + json | serde, serde_core, serde_json, itoa, memchr, zmij | 6 | +| `serde_yaml` (scenario files only) | serde_yaml, unsafe-libyaml, indexmap, hashbrown, equivalent, ryu | 6 | | `rand_chacha` (K5 seeded RNG) | rand_chacha, rand_core, ppv-lite86, zerocopy | 4 | | Clay-Borg crates | cb-kernel, cb-events, cb-game-runtime, games-ground | 4 | @@ -130,20 +132,31 @@ The honest options, in order of preference: 1. **Make `serde_yaml` optional** behind a `scenarios` feature. YAML is a test-and-tooling concern; a shipped game runtime does not need it. - Removes 5 crates from the default build for no loss of capability. + Removes 6 crates from the default build for no loss of capability + (`ryu` belongs to this group, not to serde_json, which uses `zmij` + for floats — corrected after measuring the reverse-dependency graph). This is the one to do first, and it improves D4 optionality as well as D2. -2. **Revisit the target.** ≤20 was set before the K5/K7 contracts named - ChaCha and SHA-256. Those two contracts cost 12 crates between them - and are load-bearing for determinism. A target that a spec's own - contracts make unreachable is a bad target. +2. **Revisit the target, and what it measures.** Measured savings per + option: serde_yaml optional −6 (→27); replacing serde_json −4 more + (→23); inlining SHA-256 −8 (→19); inlining ChaCha12 −4 (→15). **Only + reimplementing SHA-256 or ChaCha gets under 20**, so the target is + unreachable without undoing K5/K7. + + Crate count also compares badly across ecosystems: Rust splits + crates far more finely than npm, so "33 vs 120 npm packages" flatters + us. The measurable thing crate count proxies for is third-party + source under audit: **307,317 lines** across all five groups, against + 3,398 of our own. Retargeting AM-4 on audited third-party LOC, split + into shipped-runtime and dev-toolchain, measures the real concern and + cannot be gamed by crate granularity. What we are **not** doing: hand-rolling SHA-256 or ChaCha to win a dependency count. That trades an auditable, well-tested primitive for a number on a scoreboard. -This is a T09 input: either the metric moves for a stated reason, or -option 1 lands and the remainder is justified. +Carried forward as an open decision (see the note at the head of this +file): AM-4 is re-measured once the option is chosen. ## 5. Cost log (AM-12) @@ -152,14 +165,30 @@ Per `specs/MetricsAndScenarios.md` §1a. Model: Claude Fable 5, at | Task | Model | Iterations | Notes | |---|---|---|---| -| T08 | claude-fable-5 | 6 code iterations + benchmarks | Token counts not captured per iteration; see limitation below | +| CB-WP-0001 (whole session, T01–T09) | claude-fable-5 | 6 T08 code iterations + benchmarks | $248.46 measured; per-task split pending CB-WP-0002 | -**Limitation, stated rather than fabricated:** exact per-task token -counts were not instrumented during T08, so the USD figure the metric -asks for cannot be computed honestly from this run. Recording an -estimate here would defeat the purpose of the metric. T09 should either -wire real token accounting into the loop or drop M-D2-CST as -unmeasurable in this setup. +**Correction (2026-07-31).** This section originally recorded AM-12 as +*uncomputable*. That was wrong. The declining to estimate was right; the +conclusion that no instrument existed was not. Every session transcript +(`~/.claude/projects//.jsonl`) carries exact per-message +`usage` including the cache breakdown. Read for this session: + +| Component | Tokens | Cost (Fable 5) | +|---|---|---| +| Output | 585,528 | $29.28 | +| Cache read | 131,863,164 | $131.86 | +| Cache write (1h) | 4,365,668 | $87.31 | +| Input | 1,090 | $0.01 | +| **Session total** | | **$248.46** (~$124 on Opus 5) | + +**53% of the cost is cache reads**, not output. Cost in an agentic loop +is driven by context size × turn count, which no "tokens per task" +metric would have surfaced. + +Still missing is *attribution*: this is a whole-session figure, not a +per-task one, because nothing marks task boundaries in the transcript. +That is what CB-WP-0002 is for. The AM-12 row above should be read as +"session-level cost measured; per-task attribution pending CB-WP-0002". ## 6. Metrics not reported diff --git a/workplans/CB-WP-0002-cost-accounting.md b/workplans/CB-WP-0002-cost-accounting.md new file mode 100644 index 0000000..ca4b3dd --- /dev/null +++ b/workplans/CB-WP-0002-cost-accounting.md @@ -0,0 +1,185 @@ +--- +id: CB-WP-0002 +title: "Make agentic cost measurable, so D2 claims are falsifiable" +status: proposed +--- + +# Purpose + +CB-WP-0001 specified a cost metric (AM-12, `M-D2-CST`) in full — price +sheet, formula, staleness rule — and then could not report a number, +because nothing was ever instrumented. It was recorded as +"uncomputable". + +That was wrong, and the retrospective for this workplan should say so +plainly: **the data existed the whole time.** Every Claude Code session +transcript (`~/.claude/projects//.jsonl`) carries exact +per-message `usage`, including the cache breakdown. Reading it for +CB-WP-0001's session gives: + +| Component | Tokens | Cost (Fable 5) | +|---|---|---| +| Output | 585,528 | $29.28 | +| Cache read | 131,863,164 | **$131.86** | +| Cache write (1h) | 4,365,668 | $87.31 | +| Input | 1,090 | $0.01 | +| **Total** | | **$248.46** | + +The headline finding from that single reading is the reason this +workplan exists: **53% of the cost is cache reads**, not output. Cost in +an agentic loop is driven by context size × turn count, not by how much +the model writes. No D2 decision made on "tokens per task" would have +surfaced that. + +This workplan makes cost a first-class measured dimension so that +future AM-12 equivalents support conclusions instead of decorating an +evidence file. It follows InnerLoop v1.0, including the rules that pass +earned: every metric names its instrument, and every harness carries a +positive control. + +## Phase A — Instrument + +## Task: Survey the available cost instruments + +```task +id: CB-WP-0002-T01 +status: todo +priority: high +``` + +Produce `research/CB-RES-0002-cost-accounting.md` per the InnerLoop +survey template. Candidates at minimum: session transcript JSONL +(`usage` per assistant message), the Custodian State Hub token-event +API (`record_token_event`, `update_task_status` token tiers, +`get_token_summary`), the Claude Code status bar, and the Anthropic +usage/billing API. Per candidate: what it reports, granularity, +accuracy, whether it survives compaction, and whether it can attribute +cost to a unit of work. + +Note explicitly which are **exact** and which are **estimates** — the +prior failure was tolerating an estimate-shaped metric. Name the +benchmark-to-beat per dimension; expect the transcript to lead on +accuracy and the hub to lead on durability. + +## Task: Decide the instrument and the attribution model (ADR) + +```task +id: CB-WP-0002-T02 +status: todo +priority: high +``` + +Adversarial review of T01 first (InnerLoop §Step 2), committed as +`history/YYMMDD-cost-accounting-{challenge,response}.md`. + +Then `decisions/ADR-0003-cost-accounting.md`. The genuinely hard part is +**attribution**: a transcript is a flat message stream, and a "task" is +a workplan concept. Options to weigh, not assume: + +- git commit timestamps as task boundaries (the loop commits per + iteration, so boundaries already exist and are durable) +- explicit session markers emitted at task start/end +- hub task status transitions as the time index + +State the expected advantage per dimension and the known failure modes +of the chosen model — in particular, what happens to attribution across +`/compact`, across resumed sessions, and for work spanning a boundary. +Gate: no collector code before this ADR is committed. + +## Task: Specify the cost metrics with named instruments + +```task +id: CB-WP-0002-T03 +status: todo +priority: high +``` + +Write `specs/CostAccounting.md`: the cost model (input, output, cache +read, cache write at 5m and 1h, per the price sheet), the attribution +contract, and the acceptance metrics — each naming the **command that +produces its number**, per InnerLoop v1.0 §Step 4. + +Must include a metric for the finding that motivated this workplan: +**cost composition** (what fraction is cache read vs write vs output), +not only a total. A single total would have hidden the 53%. + +Revise `specs/MetricsAndScenarios.md` §1a to point at this spec, and +replace AM-12's definition with one that is computable. + +## Phase B — Build and prove + +## Task: Implement the cost collector + +```task +id: CB-WP-0002-T04 +status: todo +priority: medium +``` + +Implement the tool chosen in T02 (expected: `tools/cb-cost`). It reads +transcripts, applies the price sheet at +`benchmarks/baselines/model-prices.toml`, attributes cost per the T02 +model, and emits both a per-task table and a composition breakdown in +the evidence-row format from T03. + +**Positive control is mandatory** (InnerLoop v1.0 §Step 5): the tool +asserts that attributed tokens sum to the transcript total, and refuses +to emit numbers when they do not reconcile. An unattributed remainder is +reported as its own line, never silently dropped — the failure this +whole workplan exists to prevent was a number that looked fine. + +Handle multi-model sessions: a session that switches models must price +each message at its own model's rate. + +## Task: Validate against CB-WP-0001 and answer a real question + +```task +id: CB-WP-0002-T05 +status: todo +priority: medium +``` + +Run the collector over the CB-WP-0001 session and commit +`evidence/CB-EV-0002-cost-accounting.md`. Reconciliation against the +totals in this workplan's Purpose section is the acceptance test: the +tool must reproduce $248.46 (Fable 5) from the same transcript, or +explain the difference. + +Then use it to answer at least one question that could not be answered +before, and record the answer. Candidates: which of T01–T09 cost most +and whether that matched its value; what a `/compact` costs; whether the +adversarial review paid for itself; how much the six T08 code iterations +cost relative to the research phase. + +A tool that produces numbers nobody draws a conclusion from has not +cleared the bar that CB-WP-0001's AM-12 failed to clear. + +## Task: Wire cost into the loop + +```task +id: CB-WP-0002-T06 +status: todo +priority: low +``` + +Make cost collection automatic rather than remembered: add the +collector to the evidence checklist in `specs/InnerLoop.md`, emit hub +token events so the Token Cost dashboard reflects real numbers rather +than the 1000/500 heuristic fallback, and add a `make cost` target to +the one command surface. + +## Task: Retrospective + +```task +id: CB-WP-0002-T07 +status: todo +priority: low +``` + +Revise `specs/InnerLoop.md` and `specs/MetricsAndScenarios.md` from what +this pass teaches about metric design. The specific question to answer: +CB-WP-0001 produced a fully specified metric that could not be computed, +and v1.0's "every metric names its instrument" rule was written to stop +that. Did it? If a metric can still be written without a working +instrument, the rule needs teeth — for example, requiring that the +instrument be demonstrated on real data before the metric is accepted.