coulomb/clay-borg
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

T03: specs/CostAccounting.md — 15 contracts, 8 metrics, each with a command

Browse source 
Normative spec for the cost model (dedup by requestId, per-model and
per-TTL pricing), the attribution contract (ending-at-commit intervals
scoped by sessionId), and the reported shape (composition, not a total).

Every acceptance row names the command that produces its number, per
InnerLoop Step 4. AC-5..AC-8 are the positive control: cb-cost
--self-test must fail on a dedup violation, on zero responses, on a
missing subagent tree, and on 5m cache priced at the 1h rate.

The feasibility check earns its place here: AC-1's $92.87 is reachable
only if CA-06 holds. The earlier $92.21 target was reachable only by a
collector with the exact blind spot the survey documented.

MetricsAndScenarios 1a superseded. Two of its rules struck through
rather than deleted, because both were wrong in instructive ways: "if
the cache split is unknown, count all input at full price" would have
priced 80.5M cache reads at 10x, and "the hub already records tokens"
named as a source what is only ever a lossy sink. M-D2-TOK demoted --
tokens are not comparable across models or cache states.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 08:46:12 +02:00
parent c3e6e72ab2
commit b96cd94a64
3 changed files with 193 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

33
specs/MetricsAndScenarios.md
View file

@ -35,8 +35,8 @@ and add capability-specific rows only when these don't cover the claim.
| M-D2-LOC | D2 | source LOC excluding tests (tokei) | lines | adopted:tokei |
| M-D2-DEP | D2 | transitive dependency count (cargo tree) | crates | adopted:cargo-deny practice |
| M-D2-BLD | D2 | clean build / incremental build time | s | adopted:cargo timing |
| M-D2-TOK | D2 | tokens consumed per completed workplan task | tokens | novel — agentic-efficiency core metric; recorded per task in evidence |
| M-D2-CST | D2 | **cost** per completed workplan task: tokens × the executing model's pricepoint | USD | adapted:anthropic-pricing — tokens alone mislead once models with different prices mix; see §1a |
| M-D2-TOK | D2 | tokens consumed per completed workplan task | tokens | novel — **demoted 2026-07-31**: an input to the cost model, not comparable across models or cache states |
| M-D2-CST | D2 | **cost** per completed workplan task, attributed per CA-08 | USD | adapted:anthropic-pricing — instrument: `make cost`; normative spec [CostAccounting.md](CostAccounting.md) |
| M-D3-THR | D3 | events applied per second, headless replay | events/s | adapted:criterion (throughput mode) |
| M-D3-LAT | D3 | p99 command→state-applied latency | µs | adopted:criterion |
| M-D3-MEM | D3 | peak resident memory during benchmark scenario | MB | adopted:/usr/bin/time -v |
@ -46,6 +46,17 @@ and add capability-specific rows only when these don't cover the claim.
### 1a. Token cost accounting (M-D2-CST)
> **Superseded 2026-07-31 by [CostAccounting.md](CostAccounting.md)**, which
> is normative for the cost model, attribution, and acceptance metrics.
> This section is retained for the price-sheet location and the
> quality-gate rule; where the two disagree, CostAccounting.md wins.
>
> What changed and why: the definition below named no instrument and was
> never computed, so CB-WP-0001 recorded M-D2-CST as *uncomputable* while
> the data sat in the session transcripts. Three of its rules were also
> wrong in ways that cost real money to discover — see the corrections
> inline below.
Token counts are only comparable at a single pricepoint. Since work moves
between models (Fable for demanding passes, Sonnet/Opus for routine ones),
every task's token record carries the **model** it ran on, and cost is
@ -81,13 +92,17 @@ write_1h = 2.0
Rules:
- `cost = (in_tokens × input + out_tokens × output) / 1e6`, using the sheet
in force at the time the work ran; cache reads/writes, when known, use the
multipliers. If cache split is unknown, count all input at full price and
note it — cost is then an upper bound.
- The state-hub task close (`update_task_status`) already records tokens and
`model`; the evidence file's task log adds the computed USD figure so
cross-model comparisons are honest.
- ~~`cost = (in_tokens × input + out_tokens × output) / 1e6` … If cache
split is unknown, count all input at full price and note it — cost is then
an upper bound.~~ **Corrected:** the cache split is never unknown; it is
in every transcript. Treating it as unknown would have priced 80.5M cache
reads at 10× their rate. See CostAccounting.md §1.2 (CA-03, CA-04) — cache
writes bill at two different TTL rates and must not be aggregated.
- ~~The state-hub task close (`update_task_status`) already records tokens
and `model`.~~ **Corrected:** the hub schema has no cache fields and
cannot represent 88% of spend, and the figures it recorded for CB-WP-0001
were estimates in error by ~100%. The hub is a **sink** for numbers
computed by `make cost`, never a source. See CostAccounting.md §6.
- **Cheaper is only better at equal quality**: M-D2-CST verdicts are valid
only alongside passing scenarios/metrics from the same run — a cheap
failed pass scores nothing.