diff --git a/benchmarks/baselines/model-prices.toml b/benchmarks/baselines/model-prices.toml new file mode 100644 index 0000000..3e12c08 --- /dev/null +++ b/benchmarks/baselines/model-prices.toml @@ -0,0 +1,24 @@ +# USD per million tokens; source: Anthropic pricing, recorded 2026-07-31 +# Governed by specs/MetricsAndScenarios.md §1a — refresh on price change or +# after 90 days, or new M-D2-CST 'better' verdicts are invalid. + +[claude-fable-5] +input = 10.00 +output = 50.00 + +[claude-opus-5] +input = 5.00 +output = 25.00 + +[claude-sonnet-5] +input = 3.00 # intro 2.00 through 2026-08-31 +output = 15.00 # intro 10.00 through 2026-08-31 + +[claude-haiku-4-5] +input = 1.00 +output = 5.00 + +[cache] # multipliers on the input price +read = 0.1 +write_5m = 1.25 +write_1h = 2.0 diff --git a/specs/MetricsAndScenarios.md b/specs/MetricsAndScenarios.md index d5f8c2b..d7ac590 100644 --- a/specs/MetricsAndScenarios.md +++ b/specs/MetricsAndScenarios.md @@ -36,6 +36,7 @@ and add capability-specific rows only when these don't cover the claim. | M-D2-DEP | D2 | transitive dependency count (cargo tree) | crates | adopted:cargo-deny practice | | M-D2-BLD | D2 | clean build / incremental build time | s | adopted:cargo timing | | M-D2-TOK | D2 | tokens consumed per completed workplan task | tokens | novel — agentic-efficiency core metric; recorded per task in evidence | +| M-D2-CST | D2 | **cost** per completed workplan task: tokens × the executing model's pricepoint | USD | adapted:anthropic-pricing — tokens alone mislead once models with different prices mix; see §1a | | M-D3-THR | D3 | events applied per second, headless replay | events/s | adapted:criterion (throughput mode) | | M-D3-LAT | D3 | p99 command→state-applied latency | µs | adopted:criterion | | M-D3-MEM | D3 | peak resident memory during benchmark scenario | MB | adopted:/usr/bin/time -v | @@ -43,6 +44,57 @@ and add capability-specific rows only when these don't cover the claim. | M-D4-LEAK | D4 | foreign types in canonical interfaces | count | novel — must be 0; enforced by grep/deny rule, the Clay-Borg hard rule | | M-D4-SWAP | D4 | capability has null + reference impls passing the same conformance suite | bool | adapted:hexagonal-architecture port testing | +### 1a. Token cost accounting (M-D2-CST) + +Token counts are only comparable at a single pricepoint. Since work moves +between models (Fable for demanding passes, Sonnet/Opus for routine ones), +every task's token record carries the **model** it ran on, and cost is +computed against a committed price sheet: + +```text +benchmarks/baselines/model-prices.toml # the price sheet, updated when prices change +``` + +```toml +# USD per million tokens; source: Anthropic pricing, recorded 2026-07-31 +[claude-fable-5] +input = 10.00 +output = 50.00 + +[claude-opus-5] +input = 5.00 +output = 25.00 + +[claude-sonnet-5] +input = 3.00 # intro 2.00 through 2026-08-31 +output = 15.00 # intro 10.00 through 2026-08-31 + +[claude-haiku-4-5] +input = 1.00 +output = 5.00 + +[cache] # multipliers on the input price +read = 0.1 +write_5m = 1.25 +write_1h = 2.0 +``` + +Rules: + +- `cost = (in_tokens × input + out_tokens × output) / 1e6`, using the sheet + in force at the time the work ran; cache reads/writes, when known, use the + multipliers. If cache split is unknown, count all input at full price and + note it — cost is then an upper bound. +- The state-hub task close (`update_task_status`) already records tokens and + `model`; the evidence file's task log adds the computed USD figure so + cross-model comparisons are honest. +- **Cheaper is only better at equal quality**: M-D2-CST verdicts are valid + only alongside passing scenarios/metrics from the same run — a cheap + failed pass scores nothing. +- Prices are `adopted:anthropic-pricing` with a recorded date; a stale sheet + (> 90 days or known price change) invalidates new `better` verdicts on + M-D2-CST until refreshed. + Determinism is not a metric but an **invariant**: N replays of the same seed and command log must produce bit-identical state hashes. Invariant violations fail the run regardless of metric values. @@ -161,8 +213,8 @@ commit: | M-D3-THR | 120000 events/s (boardgame.io) | 410000 events/s | better | | ...every acceptance row, no `unmeasured`... | -## Task token log -| Task | Tokens (approx) | Iterations | +## Task cost log +| Task | Model | Tokens in/out | Cost (USD, per price sheet) | Iterations | ## Retrospective