From b96cd94a64c8d2c316d258df6b460e70944f0b43 Mon Sep 17 00:00:00 2001 From: tegwick Date: Fri, 31 Jul 2026 08:46:12 +0200 Subject: [PATCH] =?UTF-8?q?T03:=20specs/CostAccounting.md=20=E2=80=94=2015?= =?UTF-8?q?=20contracts,=208=20metrics,=20each=20with=20a=20command?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Normative spec for the cost model (dedup by requestId, per-model and per-TTL pricing), the attribution contract (ending-at-commit intervals scoped by sessionId), and the reported shape (composition, not a total). Every acceptance row names the command that produces its number, per InnerLoop Step 4. AC-5..AC-8 are the positive control: cb-cost --self-test must fail on a dedup violation, on zero responses, on a missing subagent tree, and on 5m cache priced at the 1h rate. The feasibility check earns its place here: AC-1's $92.87 is reachable only if CA-06 holds. The earlier $92.21 target was reachable only by a collector with the exact blind spot the survey documented. MetricsAndScenarios 1a superseded. Two of its rules struck through rather than deleted, because both were wrong in instructive ways: "if the cache split is unknown, count all input at full price" would have priced 80.5M cache reads at 10x, and "the hub already records tokens" named as a source what is only ever a lossy sink. M-D2-TOK demoted -- tokens are not comparable across models or cache states. Co-Authored-By: Claude Opus 5 --- specs/CostAccounting.md | 168 ++++++++++++++++++++++++ specs/MetricsAndScenarios.md | 33 +++-- workplans/CB-WP-0002-cost-accounting.md | 2 +- 3 files changed, 193 insertions(+), 10 deletions(-) create mode 100644 specs/CostAccounting.md diff --git a/specs/CostAccounting.md b/specs/CostAccounting.md new file mode 100644 index 0000000..6eb9ff3 --- /dev/null +++ b/specs/CostAccounting.md @@ -0,0 +1,168 @@ +# Cost Accounting + +Status: **v1.0** — 2026-07-31. Derived from +[CB-RES-0002](../research/CB-RES-0002-cost-accounting.md) (approved) and +[ADR-0003](../decisions/ADR-0003-cost-accounting.md). Makes M-D2-CST +computable; supersedes its "uncomputable" disposition in CB-EV-0001. + +Defines how the USD cost of agentic work is measured and attributed, so +that D2 claims about implementation efficiency are falsifiable. + +--- + +## 1. The cost model + +### 1.1 Unit of billing + +The unit is one **API response**, identified by `requestId`. It is *not* +one JSONL line: a response is written as up to six lines split by content +block (`thinking`, `text`, `tool_use`), and every line repeats the same +`usage` object. + +> **CA-01.** Cost is computed over responses deduplicated by `requestId`. +> Summing per line is a defect; it inflates by ≈1.9× on measured data. + +> **CA-02.** Dedup is asserted, not assumed. Within a `requestId` group, +> `usage` must be identical across lines and the model must not vary. A +> violation aborts the run rather than producing a number. + +Rationale for CA-02: if the format ever splits one response across two +`requestId`s, dedup *under*-reports and nothing looks wrong. The dangerous +direction gets the assertion. + +### 1.2 Price formula + +Per response, against `benchmarks/baselines/model-prices.toml`: + +```text +cost = input_tokens × price.input + + output_tokens × price.output + + cache_read_input_tokens × price.input × cache.read (0.10) + + ephemeral_5m_input_tokens × price.input × cache.write_5m (1.25) + + ephemeral_1h_input_tokens × price.input × cache.write_1h (2.00) +``` + +> **CA-03.** Each response is priced at **its own** `message.model` rate. A +> session may mix models; CB-WP-0001 used three. + +> **CA-04.** Cache writes are priced **per TTL**. The top-level +> `cache_creation_input_tokens` aggregate equals `ephemeral_5m + +> ephemeral_1h` and must never be priced at a single multiplier — doing so +> inflated a measured subagent transcript by 43%. + +> **CA-05.** A response whose model is absent from the price sheet is +> reported as an unpriced line with its token counts, never dropped and +> never priced at a default. + +### 1.3 Scope of a measurement + +> **CA-06.** A measurement enumerates **every** transcript for the repo: +> `~/.claude/projects//*.jsonl` and every +> `/subagents/agent-*.jsonl`. Subagent cost is not in the main +> file and is invisible to a collector that reads one path. + +> **CA-07.** Every committed number states its **pin** — a timestamp or +> commit. Transcripts are append-live: the file grows as the measuring +> session writes to it, and an unpinned total is not reproducible. + +## 2. The attribution contract + +> **CA-08.** A response is attributed to the task named by the next commit +> at or after it, within its own session: +> `interval := (prev_commit_time, this_commit_time]`, scoped by `sessionId`. + +> **CA-09.** Timestamps are converted, never offset-subtracted. Commit +> times are parsed from `%cI` and converted to UTC; this repo carries two +> distinct offsets. + +> **CA-10.** A response in a commit whose subject carries no `T##` tag is +> attributed to `UNATTRIBUTED`, which is **reported as its own line** in +> every table. On CB-WP-0001 this is 33% of cost ($30.32 of $92.21) — a +> per-task table is a view over roughly two-thirds of the money and says so +> wherever it appears. + +> **CA-11.** Cost after the last commit is an **open remainder**, reported +> separately from `UNATTRIBUTED`. It is work not yet committed, not work +> without a task. + +> **CA-12.** Attribution never keys on wall-clock alone. Two sessions +> overlapped 4 h 13 m on this repo carrying ~$12; only `sessionId` +> separates them. + +## 3. Reported shape + +> **CA-13.** Every report carries **composition** — the split across input, +> output, cache read, cache write 5m, cache write 1h — alongside the total. +> A total alone would have concealed the finding that motivated this work: +> 88.4% of spend is cache, at 256:1 cache-read to output tokens. + +> **CA-14.** Reconciliation is asserted. Attributed + unattributed + open +> remainder + unpriced must equal the transcript total to the cent. A +> mismatch aborts rather than reporting. + +> **CA-15.** Tables that a tool can emit are emitted by the tool. Every +> committed number in this capability's evidence is produced by a command, +> not typed. *(Adversarial review of CB-RES-0002 found every computed +> figure correct to the cent and three hand-typed markdown tables wrong.)* + +## 4. Acceptance metrics + +Each row names the command that produces its number, per InnerLoop §Step 4. +`cb-cost` is `tools/cb-cost` (T04). + +| ID | Metric | Target | Instrument | +|---|---|---|---| +| **AC-1** | reproduces the pinned CB-WP-0001 total | **$92.87** = $92.21 main + $0.66 subagent | `cb-cost --repo clay-borg --pin fc76445` | +| **AC-2** | reconciliation residual (CA-14) | **$0.00** exactly | same command, `reconciled: ok` line | +| **AC-3** | unattributed share reported (CA-10) | present, and **33%** on the pinned run | `cb-cost --pin fc76445 --by-task` | +| **AC-4** | composition reported (CA-13) | all five components present | `cb-cost --pin fc76445 --composition` | +| **AC-5** | dedup invariant asserted (CA-02) | violation exits non-zero | `cb-cost --self-test` | +| **AC-6** | positive control: refuses to report on zero responses | exits non-zero | `cb-cost --self-test` | +| **AC-7** | subagent tree included (CA-06) | omitting it changes AC-1 by $0.66 | `cb-cost --self-test` | +| **AC-8** | per-TTL cache pricing (CA-04) | 5m-only transcript prices at 1.25× | `cb-cost --self-test` | + +**AC-5 through AC-8 are the positive control.** Per InnerLoop v1.0 §Step 5, +a harness must assert it did the work it reports. `--self-test` runs each +assertion against a fixture whose expected value is known and fails loudly; +`make cost` runs it before any reported number. + +## 5. Metric feasibility check + +Per InnerLoop §Step 4, the acceptance table is checked against the +contracts in this same spec: + +- AC-1's $92.87 is reachable only if CA-06 holds (both trees enumerated). + Under a main-file-only collector the target is unreachable — this is the + defect the adversarial review caught, where a target of $92.21 would have + been hit *only* by a broken collector. +- AC-3's 33% is a property of CB-WP-0001's commit subjects, not of the + collector. It is a regression pin on the fixture, not a quality target; + improving tagging discipline will change it, and that is expected. +- CA-07 (pinning) makes AC-1 reproducible; without it the target drifts + upward on every run and the test is meaningless. + +## 6. Known limitations, stated with the numbers + +- **Per-task cost covers ~67% of spend.** Structural: 19 of 33 commits + carry no task tag. Reported per CA-10, never silently dropped. +- **Work spanning a commit is assigned whole to the later task.** Bounded + by one interval: p50 6.9 min, p90 17.7 min, max 36.8 min. +- **The price sheet cannot express a time-boxed rate.** Sonnet's intro + price is a TOML comment, so the sheet is silently wrong for sonnet-priced + work ($0.17 at the pin, 0.19%). **This becomes an error, not a rounding + issue, on 2026-08-31** when the intro rate expires and the comment and + the data disagree in the other direction. Tracked as a schema defect + against the price sheet. +- **The State Hub cannot store what this spec measures.** Its token event + schema has `tokens_in`/`tokens_out` and no cache fields, so the dashboard + necessarily shows a lossy projection. This is a limitation of the sink, + not of the metric; `make cost` remains the authority. + +## 7. Revisions to M-D2-CST + +`specs/MetricsAndScenarios.md` §1a is superseded by this spec. M-D2-CST is +redefined from "tokens × pricepoint" — which named no instrument and was +never computed — to: **USD per completed workplan task, per CA-08 +attribution, produced by `make cost`.** M-D2-TOK is retained but demoted: +tokens are the input to the cost model, not a comparable figure across +models or across cache states. diff --git a/specs/MetricsAndScenarios.md b/specs/MetricsAndScenarios.md index d7ac590..c8b360f 100644 --- a/specs/MetricsAndScenarios.md +++ b/specs/MetricsAndScenarios.md @@ -35,8 +35,8 @@ and add capability-specific rows only when these don't cover the claim. | M-D2-LOC | D2 | source LOC excluding tests (tokei) | lines | adopted:tokei | | M-D2-DEP | D2 | transitive dependency count (cargo tree) | crates | adopted:cargo-deny practice | | M-D2-BLD | D2 | clean build / incremental build time | s | adopted:cargo timing | -| M-D2-TOK | D2 | tokens consumed per completed workplan task | tokens | novel — agentic-efficiency core metric; recorded per task in evidence | -| M-D2-CST | D2 | **cost** per completed workplan task: tokens × the executing model's pricepoint | USD | adapted:anthropic-pricing — tokens alone mislead once models with different prices mix; see §1a | +| M-D2-TOK | D2 | tokens consumed per completed workplan task | tokens | novel — **demoted 2026-07-31**: an input to the cost model, not comparable across models or cache states | +| M-D2-CST | D2 | **cost** per completed workplan task, attributed per CA-08 | USD | adapted:anthropic-pricing — instrument: `make cost`; normative spec [CostAccounting.md](CostAccounting.md) | | M-D3-THR | D3 | events applied per second, headless replay | events/s | adapted:criterion (throughput mode) | | M-D3-LAT | D3 | p99 command→state-applied latency | µs | adopted:criterion | | M-D3-MEM | D3 | peak resident memory during benchmark scenario | MB | adopted:/usr/bin/time -v | @@ -46,6 +46,17 @@ and add capability-specific rows only when these don't cover the claim. ### 1a. Token cost accounting (M-D2-CST) +> **Superseded 2026-07-31 by [CostAccounting.md](CostAccounting.md)**, which +> is normative for the cost model, attribution, and acceptance metrics. +> This section is retained for the price-sheet location and the +> quality-gate rule; where the two disagree, CostAccounting.md wins. +> +> What changed and why: the definition below named no instrument and was +> never computed, so CB-WP-0001 recorded M-D2-CST as *uncomputable* while +> the data sat in the session transcripts. Three of its rules were also +> wrong in ways that cost real money to discover — see the corrections +> inline below. + Token counts are only comparable at a single pricepoint. Since work moves between models (Fable for demanding passes, Sonnet/Opus for routine ones), every task's token record carries the **model** it ran on, and cost is @@ -81,13 +92,17 @@ write_1h = 2.0 Rules: -- `cost = (in_tokens × input + out_tokens × output) / 1e6`, using the sheet - in force at the time the work ran; cache reads/writes, when known, use the - multipliers. If cache split is unknown, count all input at full price and - note it — cost is then an upper bound. -- The state-hub task close (`update_task_status`) already records tokens and - `model`; the evidence file's task log adds the computed USD figure so - cross-model comparisons are honest. +- ~~`cost = (in_tokens × input + out_tokens × output) / 1e6` … If cache + split is unknown, count all input at full price and note it — cost is then + an upper bound.~~ **Corrected:** the cache split is never unknown; it is + in every transcript. Treating it as unknown would have priced 80.5M cache + reads at 10× their rate. See CostAccounting.md §1.2 (CA-03, CA-04) — cache + writes bill at two different TTL rates and must not be aggregated. +- ~~The state-hub task close (`update_task_status`) already records tokens + and `model`.~~ **Corrected:** the hub schema has no cache fields and + cannot represent 88% of spend, and the figures it recorded for CB-WP-0001 + were estimates in error by ~100%. The hub is a **sink** for numbers + computed by `make cost`, never a source. See CostAccounting.md §6. - **Cheaper is only better at equal quality**: M-D2-CST verdicts are valid only alongside passing scenarios/metrics from the same run — a cheap failed pass scores nothing. diff --git a/workplans/CB-WP-0002-cost-accounting.md b/workplans/CB-WP-0002-cost-accounting.md index 72693c3..061ee95 100644 --- a/workplans/CB-WP-0002-cost-accounting.md +++ b/workplans/CB-WP-0002-cost-accounting.md @@ -113,7 +113,7 @@ Gate: no collector code before this ADR is committed. ```task id: CB-WP-0002-T03 -status: todo +status: done priority: high state_hub_task_id: "00d42ed2-4391-4580-aae2-06e3e151c69b" ```