Add M-D2-CST token-cost metric with committed model price sheet
This commit is contained in:
parent
1218ee024e
commit
38ffd8b7fd
2 changed files with 78 additions and 2 deletions
24
benchmarks/baselines/model-prices.toml
Normal file
24
benchmarks/baselines/model-prices.toml
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
# USD per million tokens; source: Anthropic pricing, recorded 2026-07-31
|
||||
# Governed by specs/MetricsAndScenarios.md §1a — refresh on price change or
|
||||
# after 90 days, or new M-D2-CST 'better' verdicts are invalid.
|
||||
|
||||
[claude-fable-5]
|
||||
input = 10.00
|
||||
output = 50.00
|
||||
|
||||
[claude-opus-5]
|
||||
input = 5.00
|
||||
output = 25.00
|
||||
|
||||
[claude-sonnet-5]
|
||||
input = 3.00 # intro 2.00 through 2026-08-31
|
||||
output = 15.00 # intro 10.00 through 2026-08-31
|
||||
|
||||
[claude-haiku-4-5]
|
||||
input = 1.00
|
||||
output = 5.00
|
||||
|
||||
[cache] # multipliers on the input price
|
||||
read = 0.1
|
||||
write_5m = 1.25
|
||||
write_1h = 2.0
|
||||
|
|
@ -36,6 +36,7 @@ and add capability-specific rows only when these don't cover the claim.
|
|||
| M-D2-DEP | D2 | transitive dependency count (cargo tree) | crates | adopted:cargo-deny practice |
|
||||
| M-D2-BLD | D2 | clean build / incremental build time | s | adopted:cargo timing |
|
||||
| M-D2-TOK | D2 | tokens consumed per completed workplan task | tokens | novel — agentic-efficiency core metric; recorded per task in evidence |
|
||||
| M-D2-CST | D2 | **cost** per completed workplan task: tokens × the executing model's pricepoint | USD | adapted:anthropic-pricing — tokens alone mislead once models with different prices mix; see §1a |
|
||||
| M-D3-THR | D3 | events applied per second, headless replay | events/s | adapted:criterion (throughput mode) |
|
||||
| M-D3-LAT | D3 | p99 command→state-applied latency | µs | adopted:criterion |
|
||||
| M-D3-MEM | D3 | peak resident memory during benchmark scenario | MB | adopted:/usr/bin/time -v |
|
||||
|
|
@ -43,6 +44,57 @@ and add capability-specific rows only when these don't cover the claim.
|
|||
| M-D4-LEAK | D4 | foreign types in canonical interfaces | count | novel — must be 0; enforced by grep/deny rule, the Clay-Borg hard rule |
|
||||
| M-D4-SWAP | D4 | capability has null + reference impls passing the same conformance suite | bool | adapted:hexagonal-architecture port testing |
|
||||
|
||||
### 1a. Token cost accounting (M-D2-CST)
|
||||
|
||||
Token counts are only comparable at a single pricepoint. Since work moves
|
||||
between models (Fable for demanding passes, Sonnet/Opus for routine ones),
|
||||
every task's token record carries the **model** it ran on, and cost is
|
||||
computed against a committed price sheet:
|
||||
|
||||
```text
|
||||
benchmarks/baselines/model-prices.toml # the price sheet, updated when prices change
|
||||
```
|
||||
|
||||
```toml
|
||||
# USD per million tokens; source: Anthropic pricing, recorded 2026-07-31
|
||||
[claude-fable-5]
|
||||
input = 10.00
|
||||
output = 50.00
|
||||
|
||||
[claude-opus-5]
|
||||
input = 5.00
|
||||
output = 25.00
|
||||
|
||||
[claude-sonnet-5]
|
||||
input = 3.00 # intro 2.00 through 2026-08-31
|
||||
output = 15.00 # intro 10.00 through 2026-08-31
|
||||
|
||||
[claude-haiku-4-5]
|
||||
input = 1.00
|
||||
output = 5.00
|
||||
|
||||
[cache] # multipliers on the input price
|
||||
read = 0.1
|
||||
write_5m = 1.25
|
||||
write_1h = 2.0
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- `cost = (in_tokens × input + out_tokens × output) / 1e6`, using the sheet
|
||||
in force at the time the work ran; cache reads/writes, when known, use the
|
||||
multipliers. If cache split is unknown, count all input at full price and
|
||||
note it — cost is then an upper bound.
|
||||
- The state-hub task close (`update_task_status`) already records tokens and
|
||||
`model`; the evidence file's task log adds the computed USD figure so
|
||||
cross-model comparisons are honest.
|
||||
- **Cheaper is only better at equal quality**: M-D2-CST verdicts are valid
|
||||
only alongside passing scenarios/metrics from the same run — a cheap
|
||||
failed pass scores nothing.
|
||||
- Prices are `adopted:anthropic-pricing` with a recorded date; a stale sheet
|
||||
(> 90 days or known price change) invalidates new `better` verdicts on
|
||||
M-D2-CST until refreshed.
|
||||
|
||||
Determinism is not a metric but an **invariant**: N replays of the same
|
||||
seed and command log must produce bit-identical state hashes. Invariant
|
||||
violations fail the run regardless of metric values.
|
||||
|
|
@ -161,8 +213,8 @@ commit: <sha of measured tree>
|
|||
| M-D3-THR | 120000 events/s (boardgame.io) | 410000 events/s | better |
|
||||
| ...every acceptance row, no `unmeasured`... |
|
||||
|
||||
## Task token log
|
||||
| Task | Tokens (approx) | Iterations |
|
||||
## Task cost log
|
||||
| Task | Model | Tokens in/out | Cost (USD, per price sheet) | Iterations |
|
||||
|
||||
## Retrospective
|
||||
<what the loop itself should change — one paragraph minimum>
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue