diff --git a/specs/MetricsAndScenarios.md b/specs/MetricsAndScenarios.md index 5ba53bd..9ae2846 100644 --- a/specs/MetricsAndScenarios.md +++ b/specs/MetricsAndScenarios.md @@ -94,27 +94,11 @@ benchmarks/baselines/model-prices.toml # the price sheet, updated when prices ``` ```toml -# USD per million tokens; source: Anthropic pricing, recorded 2026-07-31 -[claude-fable-5] -input = 10.00 -output = 50.00 - -[claude-opus-5] -input = 5.00 -output = 25.00 - -[claude-sonnet-5] -input = 3.00 # intro 2.00 through 2026-08-31 -output = 15.00 # intro 10.00 through 2026-08-31 - -[claude-haiku-4-5] -input = 1.00 -output = 5.00 - -[cache] # multipliers on the input price -read = 0.1 -write_5m = 1.25 -write_1h = 2.0 +# The live sheet is benchmarks/baselines/model-prices.toml — read it, +# do not copy it. An inlined copy here went stale within hours: it still +# showed the sonnet intro rate as a `#` comment after CB-WP-0003 T11 made +# dated rates into data (promo_input / promo_output / promo_until), and +# added `recorded` + `max_age_days` for the staleness check (CA-16/CA-17). ``` Rules: @@ -135,7 +119,8 @@ Rules: failed pass scores nothing. - Prices are `adopted:anthropic-pricing` with a recorded date; a stale sheet (> 90 days or known price change) invalidates new `better` verdicts on - M-D2-CST until refreshed. + M-D2-CST until refreshed. **Now enforced** — CA-17 aborts the run rather + than leaving this as prose every verdict silently inherited. Determinism is not a metric but an **invariant**: N replays of the same seed and command log must produce bit-identical state hashes. Invariant