T11: dated rates and staleness become data, ahead of the 2026-08-31 flip

The price sheet had two defects of one shape -- a schema that could not
hold the fact it needed, the same criticism the cost survey levelled at
the State Hub.

  CA-16  time-boxed rates are DATA. Sonnet's intro price lived in a
         `# intro ...` comment and was invisible to the collector that
         reads the file. Now promo_input/promo_output/promo_until,
         applied per response at its own timestamp.
  CA-17  the 90-day staleness rule was prose in MetricsAndScenarios 1a
         that every M-D2-CST verdict silently inherited. Now `recorded`
         + `max_age_days` in the sheet, and a stale sheet ABORTS.

Both are exercised by make cost-test: the promo rate must apply before
2026-08-31 and lapse after, and a 102-day-old sheet must trip.

Applying CA-16 moved AC-1 from $93.32 to $93.15 -- the $0.17 CB-EV-0002
predicted, now collected rather than noted. That is a legitimate
retarget under T07's distinction: the instrument disproved the target,
and its output is in this commit. The number has now been stated five
times ($248.46, $92.21, $92.87, $93.32, $93.15), each correction from a
different mechanism.

Evidence tables regenerated from the tool rather than hand-patched,
per CA-15 -- which is the rule that exists because hand-typed tables
were the only thing the adversarial review found wrong.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 09:23:29 +02:00
parent 06628e83e1
commit db731aa0bc
6 changed files with 138 additions and 32 deletions

View file

@ -1,6 +1,15 @@
# USD per million tokens; source: Anthropic pricing, recorded 2026-07-31 # USD per million tokens. Source: Anthropic pricing.
# Governed by specs/MetricsAndScenarios.md §1a — refresh on price change or # Governed by specs/CostAccounting.md §9 — a stale sheet invalidates new
# after 90 days, or new M-D2-CST 'better' verdicts are invalid. # M-D2-CST `better` verdicts. Staleness is enforced by `make cost-test`,
# not by this comment.
#
# Dated rates are DATA, not comments. A promotional rate written as a
# comment is invisible to the collector that reads this file: the sonnet
# intro price sat in a `#` remark and was silently ignored, costing $0.17
# on a $93.32 pass and heading for a sign flip on 2026-08-31.
recorded = 2026-07-31
max_age_days = 90
[claude-fable-5] [claude-fable-5]
input = 10.00 input = 10.00
@ -11,8 +20,13 @@ input = 5.00
output = 25.00 output = 25.00
[claude-sonnet-5] [claude-sonnet-5]
input = 3.00 # intro 2.00 through 2026-08-31 input = 3.00
output = 15.00 # intro 10.00 through 2026-08-31 output = 15.00
# Promotional rate in force through `until` (inclusive); the collector
# applies it to any response whose timestamp falls on or before that date.
promo_input = 2.00
promo_output = 10.00
promo_until = 2026-08-31
[claude-haiku-4-5] [claude-haiku-4-5]
input = 1.00 input = 1.00

View file

@ -17,9 +17,9 @@ transcribed by hand (CA-15).
| ID | Metric | Target | Measured | Verdict | | ID | Metric | Target | Measured | Verdict |
|---|---|---|---|---| |---|---|---|---|---|
| AC-1 | pinned total, as two components | $93.32 = $92.21 + $1.11 | **$92.21 main + $1.11 subagent = $93.32** | **met** | | AC-1 | pinned total, as two components | $93.15 = $92.03 + $1.11 | **$92.03 main + $1.11 subagent = $93.15** | **met** |
| AC-2 | reconciliation residual | $0.00 | **$0.000000** | **met** | | AC-2 | reconciliation residual | $0.00 | **$0.000000** | **met** |
| AC-3 | unattributed share reported | present, 33% | **32.5%, own line** | **met** | | AC-3 | unattributed share reported | present, 33% | **32.4%, own line** | **met** |
| AC-4 | composition reported | 5 components | **5 of 5** | **met** | | AC-4 | composition reported | 5 components | **5 of 5** | **met** |
| AC-5 | dedup violation aborts | non-zero exit | **abort raised** | **met** | | AC-5 | dedup violation aborts | non-zero exit | **abort raised** | **met** |
| AC-6 | zero responses refuses to report | non-zero exit | **0 rows, no number emitted** | **met** | | AC-6 | zero responses refuses to report | non-zero exit | **0 rows, no number emitted** | **met** |
@ -27,7 +27,7 @@ transcribed by hand (CA-15).
| AC-8 | 5m cache priced at 1.25× | $1.25/100k @ fable | **$1.2500 (1h would be $2.0000)** | **met** | | AC-8 | 5m cache priced at 1.25× | $1.25/100k @ fable | **$1.2500 (1h would be $2.0000)** | **met** |
| AC-9 | streamed partial output → final | 5,5,195 → 195 | **195** | **met** | | AC-9 | streamed partial output → final | 5,5,195 → 195 | **195** | **met** |
No `unmeasured` rows. AC-1's target was corrected three times before this No `unmeasured` rows. AC-1's target was corrected four times before and after this
run; §4 records why, because the sequence is more useful than the final run; §4 records why, because the sequence is more useful than the final
number. number.
@ -35,23 +35,23 @@ number.
```text ```text
input 690 tok $ 0.00 0.0% input 690 tok $ 0.00 0.0%
output 323,643 tok $ 11.15 11.9% output 323,643 tok $ 11.13 11.9%
cache_read 80,611,798 tok $ 59.75 64.0% cache_read 80,611,798 tok $ 59.67 64.1%
write_5m 37,467 tok $ 0.47 0.5% write_5m 37,467 tok $ 0.47 0.5%
write_1h 1,672,854 tok $ 21.95 23.5% write_1h 1,672,854 tok $ 21.87 23.5%
TOTAL $ 93.32 TOTAL $ 93.15
``` ```
**88.0% of spend is cache; 11.9% is output.** The ratio of context re-read **88.0% of spend is cache; 11.9% is output.** The ratio of context re-read
to text written is 249:1. A single total would have shown $93.32 and to text written is 249:1. A single total would have shown $93.15 and
concealed all of it — which is exactly what CB-WP-0001's `M-D2-TOK` would concealed all of it — which is exactly what CB-WP-0001's `M-D2-TOK` would
have done, and why that metric is now demoted. have done, and why that metric is now demoted.
## 3. Per-task attribution ## 3. Per-task attribution
```text ```text
UNATTRIBUTED $ 30.32 32.5% UNATTRIBUTED $ 30.14 32.4%
T08 $ 21.02 22.5% T08 $ 21.02 22.6%
T07 $ 12.01 12.9% T07 $ 12.01 12.9%
T03 $ 9.91 10.6% T03 $ 9.91 10.6%
T04 $ 8.00 8.6% T04 $ 8.00 8.6%
@ -60,7 +60,7 @@ have done, and why that metric is now demoted.
T09 $ 1.29 1.4% T09 $ 1.29 1.4%
``` ```
**Stated limit (CA-10):** 32.5% of cost sits in commits whose subject **Stated limit (CA-10):** 32.4% of cost sits in commits whose subject
carries no `T##` tag, so this table is a view over 67.5% of spend. That is carries no `T##` tag, so this table is a view over 67.5% of spend. That is
a property of commit hygiene, not of the collector. a property of commit hygiene, not of the collector.
@ -75,14 +75,15 @@ is worth watching, not yet a conclusion from n=1.
- **The per-task figures are not comparable across passes.** They mix - **The per-task figures are not comparable across passes.** They mix
models (opus/fable/sonnet) at different price points and different cache models (opus/fable/sonnet) at different price points and different cache
states. The dollar figure is comparable; a token count is not. states. The dollar figure is comparable; a token count is not.
- **AC-3's 32.5% is a fixture pin, not a quality target.** Improving commit - **AC-3's 32.4% is a fixture pin, not a quality target.** Improving commit
tagging will move it, and that is the desired direction. tagging will move it, and that is the desired direction.
- **This is one session.** Every ratio here (cache share, $/turn, the - **This is one session.** Every ratio here (cache share, $/turn, the
compaction effect in §5) is n=1 and should be treated as a hypothesis compaction effect in §5) is n=1 and should be treated as a hypothesis
until a second pass reproduces it. until a second pass reproduces it.
- **The price sheet cannot express a time-boxed rate.** Sonnet's intro price - ~~**The price sheet cannot express a time-boxed rate.**~~ **Fixed
lives in a TOML comment, so sonnet-priced work is off by $0.17 here 2026-07-31 by CB-WP-0003 T11** (CA-16/CA-17). Applying the sonnet
(0.19%). This becomes a real error on 2026-08-31. promotional rate moved AC-1 from $93.32 to **$93.15** — the $0.17 this
section predicted, now collected rather than merely noted.
- **AC-1 is not invoice-verified.** No admin key exists, so the Anthropic - **AC-1 is not invoice-verified.** No admin key exists, so the Anthropic
billing API could not independently confirm the total. The transcript billing API could not independently confirm the total. The transcript
counters are the same ones billing uses, but that is an argument, not a counters are the same ones billing uses, but that is an argument, not a
@ -145,7 +146,7 @@ largest sample was assumed to hold on the smallest one.** The main
transcript is 338 of 346 responses, so 206/206 felt conclusive; the transcript is 338 of 346 responses, so 206/206 felt conclusive; the
violation lives entirely in the 8 responses nobody checked separately. violation lives entirely in the 8 responses nobody checked separately.
Cost of the adversarial review this pass: **$1.11**, against a $93.32 pass. Cost of the adversarial review this pass: **$1.11**, against a $93.15 pass.
It found three approval-blocking defects, one of which (the subagent It found three approval-blocking defects, one of which (the subagent
exclusion) would have made this evidence file certify a broken collector. exclusion) would have made this evidence file certify a broken collector.
Second consecutive pass where a ~1% spend on review changed the outcome. Second consecutive pass where a ~1% spend on review changed the outcome.

View file

@ -67,6 +67,16 @@ cost = input_tokens × price.input
> reported as an unpriced line with its token counts, never dropped and > reported as an unpriced line with its token counts, never dropped and
> never priced at a default. > never priced at a default.
> **CA-16.** Promotional and other time-boxed rates are **data**, not
> comments. A response is priced at the rate in force *at its own
> timestamp*: `promo_input`/`promo_output` apply on or before
> `promo_until`, and the standard rate applies after.
> **CA-17.** The price sheet carries `recorded` and `max_age_days`, and a
> sheet older than `max_age_days` **aborts the run**. A stale sheet
> invalidates `better` verdicts, so the staleness rule is a command rather
> than a sentence.
### 1.3 Scope of a measurement ### 1.3 Scope of a measurement
> **CA-06.** A measurement enumerates **every** transcript for the repo: > **CA-06.** A measurement enumerates **every** transcript for the repo:
@ -125,9 +135,9 @@ Each row names the command that produces its number, per InnerLoop §Step 4.
| ID | Metric | Target | Instrument | | ID | Metric | Target | Instrument |
|---|---|---|---| |---|---|---|---|
| **AC-1** | reproduces the pinned CB-WP-0001 total | **$93.32** = $92.21 main + $1.11 subagent | `make cost-pin` | | **AC-1** | reproduces the pinned CB-WP-0001 total | **$93.15** = $92.03 main + $1.11 subagent | `make cost-pin` |
| **AC-2** | reconciliation residual (CA-14) | **$0.00** exactly | same command, `reconciled: ok` line | | **AC-2** | reconciliation residual (CA-14) | **$0.00** exactly | same command, `reconciled: ok` line |
| **AC-3** | unattributed share reported (CA-10) | present, and **33%** on the pinned run | `cb-cost --pin fc76445 --by-task` | | **AC-3** | unattributed share reported (CA-10) | present, and **32.4%** on the pinned run | `cb-cost --pin fc76445 --by-task` |
| **AC-4** | composition reported (CA-13) | all five components present | `cb-cost --pin fc76445 --composition` | | **AC-4** | composition reported (CA-13) | all five components present | `cb-cost --pin fc76445 --composition` |
| **AC-5** | dedup invariant asserted (CA-02) | violation exits non-zero | `make cost-test` | | **AC-5** | dedup invariant asserted (CA-02) | violation exits non-zero | `make cost-test` |
| **AC-6** | positive control: refuses to report on zero responses | exits non-zero | `make cost-test` | | **AC-6** | positive control: refuses to report on zero responses | exits non-zero | `make cost-test` |
@ -145,7 +155,7 @@ assertion against a fixture whose expected value is known and fails loudly;
Per InnerLoop §Step 4, the acceptance table is checked against the Per InnerLoop §Step 4, the acceptance table is checked against the
contracts in this same spec: contracts in this same spec:
- AC-1's $93.32 is reachable only if CA-06 holds (both trees enumerated). - AC-1's $93.15 is reachable only if CA-06 holds (both trees enumerated).
Under a main-file-only collector the target is unreachable — this is the Under a main-file-only collector the target is unreachable — this is the
defect the adversarial review caught, where a target of $92.21 would have defect the adversarial review caught, where a target of $92.21 would have
been hit *only* by a broken collector. been hit *only* by a broken collector.
@ -228,6 +238,31 @@ signal — which is exactly the dead-policy failure this replaces, one level
up. The mitigation is that it costs nothing to run and is on the one up. The mitigation is that it costs nothing to run and is on the one
command surface. command surface.
## 9. The price sheet
`benchmarks/baselines/model-prices.toml` is the single source of rates.
Two defects were fixed here, both of the same shape — **a schema that
could not hold the fact it needed**, which is the criticism this
capability's survey levelled at the State Hub:
1. **A time-boxed rate lived in a comment.** Sonnet's intro price
(2.00/10.00 through 2026-08-31) was written as `# intro …` and was
therefore invisible to the collector, which used the standard rate. Cost
at the pin: **$0.17**, 0.19% of the pass. The real hazard was the
date — on **2026-08-31** the promotion expires and a comment-only
representation would have been wrong in the opposite direction, with
nothing to notice. Now expressed as `promo_*` fields and applied per
CA-16.
2. **The 90-day staleness rule had no check.** It was prose in
`MetricsAndScenarios.md` §1a and every M-D2-CST verdict silently
inherited it. Now `recorded` + `max_age_days` in the sheet, enforced by
CA-17 and exercised by `make cost-test`.
Applying CA-16 moved AC-1 from $93.32 to **$93.15**. This is a *legitimate*
retarget under the distinction CB-WP-0003 T07 draws: the instrument
disproved the target, rather than the implementation missing it. The
instrument's output is in the same commit.
## 8. Revisions to M-D2-CST ## 8. Revisions to M-D2-CST
`specs/MetricsAndScenarios.md` §1a is superseded by this spec. M-D2-CST is `specs/MetricsAndScenarios.md` §1a is superseded by this spec. M-D2-CST is

View file

@ -72,16 +72,53 @@ def components(usage):
} }
def price_of(prices, model, toks): def rates_at(prices, model, when=None):
"""Input/output rate in force for `model` at ISO instant `when` (CA-16).
Promotional rates are data, not comments. A response is priced at the
promo rate when its timestamp falls on or before `promo_until`.
"""
pr = prices.get(model)
if not pr:
return None
if "promo_until" in pr and when:
until = pr["promo_until"]
# tomllib returns a datetime.date for a bare TOML date.
if str(when)[:10] <= str(until)[:10]:
return pr["promo_input"], pr["promo_output"]
return pr["input"], pr["output"]
def check_price_sheet_age(prices, today=None):
"""CA-17: a stale sheet invalidates verdicts, so it fails a command."""
import datetime as _dt
recorded = prices.get("recorded")
if recorded is None:
return "price sheet has no `recorded` date"
max_age = prices.get("max_age_days", 90)
today = today or _dt.date.today()
if isinstance(recorded, _dt.datetime):
recorded = recorded.date()
age = (today - recorded).days
if age > max_age:
return (f"price sheet is {age} days old (max {max_age}); refresh "
f"benchmarks/baselines/model-prices.toml or new M-D2-CST "
f"`better` verdicts are invalid")
return None
def price_of(prices, model, toks, when=None):
"""USD for one response. Returns None when the model is unpriced (CA-05).""" """USD for one response. Returns None when the model is unpriced (CA-05)."""
pr = prices.get(model) pr = prices.get(model)
if not pr: if not pr:
return None return None
rin, rout = rates_at(prices, model, when)
cache = prices["cache"] cache = prices["cache"]
unit = pr["input"] / 1e6 unit = rin / 1e6
return ( return (
toks["input"] * unit toks["input"] * unit
+ toks["output"] * pr["output"] / 1e6 + toks["output"] * rout / 1e6
+ toks["cache_read"] * unit * cache["read"] + toks["cache_read"] * unit * cache["read"]
+ toks["write_5m"] * unit * cache["write_5m"] + toks["write_5m"] * unit * cache["write_5m"]
+ toks["write_1h"] * unit * cache["write_1h"] + toks["write_1h"] * unit * cache["write_1h"]
@ -281,8 +318,12 @@ def collect(slug, pin_ref=None):
# as though it were an answer. # as though it were an answer.
raise Abort(f"no responses in {len(paths)} transcript(s) — refusing to report") raise Abort(f"no responses in {len(paths)} transcript(s) — refusing to report")
stale = check_price_sheet_age(prices)
if stale:
raise Abort(stale)
for r in responses: for r in responses:
r["cost"] = price_of(prices, r["model"], r["toks"]) r["cost"] = price_of(prices, r["model"], r["toks"], r["timestamp"])
attribute(responses, commit_index(pin)) attribute(responses, commit_index(pin))
@ -298,11 +339,11 @@ def collect(slug, pin_ref=None):
continue continue
by_task[r["task"]] += r["cost"] by_task[r["task"]] += r["cost"]
by_model[r["model"]] += r["cost"] by_model[r["model"]] += r["cost"]
pr = prices[r["model"]] rin, rout = rates_at(prices, r["model"], r["timestamp"])
unit = pr["input"] / 1e6 unit = rin / 1e6
rates = { rates = {
"input": unit, "input": unit,
"output": pr["output"] / 1e6, "output": rout / 1e6,
"cache_read": unit * cache["read"], "cache_read": unit * cache["read"],
"write_5m": unit * cache["write_5m"], "write_5m": unit * cache["write_5m"],
"write_1h": unit * cache["write_1h"], "write_1h": unit * cache["write_1h"],
@ -456,6 +497,21 @@ def self_test():
finally: finally:
os.unlink(partial) os.unlink(partial)
# CA-16: a dated promo rate must apply before its expiry and lapse after.
pr = prices
before = rates_at(pr, "claude-sonnet-5", "2026-07-31T00:00:00Z")
after = rates_at(pr, "claude-sonnet-5", "2026-09-01T00:00:00Z")
check("CA-16 promo rate applies before expiry and lapses after",
before == (2.0, 10.0) and after == (3.0, 15.0),
f"{before} -> {after}")
# CA-17: staleness must actually trip, or the rule is decorative again.
import datetime as _dt
fresh = check_price_sheet_age(pr, _dt.date(2026, 8, 1))
stale = check_price_sheet_age(pr, _dt.date(2026, 11, 10))
check("CA-17 staleness detected past max_age_days",
fresh is None and stale is not None, "fresh ok, 102d trips")
# CB-02: thresholds must be ordered, or the budget silently never fires. # CB-02: thresholds must be ordered, or the budget silently never fires.
ap_defaults = {"soft": 10.00, "hard": 22.00} ap_defaults = {"soft": 10.00, "hard": 22.00}
check("CB-02 budget thresholds ordered and positive", check("CB-02 budget thresholds ordered and positive",

View file

@ -325,7 +325,7 @@ now price the evaluation itself.
```task ```task
id: CB-WP-0003-T11 id: CB-WP-0003-T11
status: todo status: done
priority: medium priority: medium
state_hub_task_id: "809215d8-8a5c-4b10-9445-9bc6db9bda42" state_hub_task_id: "809215d8-8a5c-4b10-9445-9bc6db9bda42"
``` ```