From 4580438f1c1a484c5406a34a57dcf7c06d01a150 Mon Sep 17 00:00:00 2001 From: tegwick Date: Fri, 31 Jul 2026 09:27:31 +0200 Subject: [PATCH] Replace the inlined price sheet with a pointer to the live file The copy in MetricsAndScenarios 1a went stale within hours of T11: it still showed the sonnet intro rate as a comment after the real sheet moved dated rates into data. A spec that duplicates a data file will always drift from it; the copy is now a pointer that says so. Co-Authored-By: Claude Opus 5 --- specs/MetricsAndScenarios.md | 29 +++++++---------------------- 1 file changed, 7 insertions(+), 22 deletions(-) diff --git a/specs/MetricsAndScenarios.md b/specs/MetricsAndScenarios.md index 5ba53bd..9ae2846 100644 --- a/specs/MetricsAndScenarios.md +++ b/specs/MetricsAndScenarios.md @@ -94,27 +94,11 @@ benchmarks/baselines/model-prices.toml # the price sheet, updated when prices ``` ```toml -# USD per million tokens; source: Anthropic pricing, recorded 2026-07-31 -[claude-fable-5] -input = 10.00 -output = 50.00 - -[claude-opus-5] -input = 5.00 -output = 25.00 - -[claude-sonnet-5] -input = 3.00 # intro 2.00 through 2026-08-31 -output = 15.00 # intro 10.00 through 2026-08-31 - -[claude-haiku-4-5] -input = 1.00 -output = 5.00 - -[cache] # multipliers on the input price -read = 0.1 -write_5m = 1.25 -write_1h = 2.0 +# The live sheet is benchmarks/baselines/model-prices.toml — read it, +# do not copy it. An inlined copy here went stale within hours: it still +# showed the sonnet intro rate as a `#` comment after CB-WP-0003 T11 made +# dated rates into data (promo_input / promo_output / promo_until), and +# added `recorded` + `max_age_days` for the staleness check (CA-16/CA-17). ``` Rules: @@ -135,7 +119,8 @@ Rules: failed pass scores nothing. - Prices are `adopted:anthropic-pricing` with a recorded date; a stale sheet (> 90 days or known price change) invalidates new `better` verdicts on - M-D2-CST until refreshed. + M-D2-CST until refreshed. **Now enforced** — CA-17 aborts the run rather + than leaving this as prose every verdict silently inherited. Determinism is not a metric but an **invariant**: N replays of the same seed and command log must produce bit-identical state hashes. Invariant