diff --git a/benchmarks/baselines/model-prices.toml b/benchmarks/baselines/model-prices.toml index 3e12c08..68cade5 100644 --- a/benchmarks/baselines/model-prices.toml +++ b/benchmarks/baselines/model-prices.toml @@ -1,6 +1,15 @@ -# USD per million tokens; source: Anthropic pricing, recorded 2026-07-31 -# Governed by specs/MetricsAndScenarios.md §1a — refresh on price change or -# after 90 days, or new M-D2-CST 'better' verdicts are invalid. +# USD per million tokens. Source: Anthropic pricing. +# Governed by specs/CostAccounting.md §9 — a stale sheet invalidates new +# M-D2-CST `better` verdicts. Staleness is enforced by `make cost-test`, +# not by this comment. +# +# Dated rates are DATA, not comments. A promotional rate written as a +# comment is invisible to the collector that reads this file: the sonnet +# intro price sat in a `#` remark and was silently ignored, costing $0.17 +# on a $93.32 pass and heading for a sign flip on 2026-08-31. + +recorded = 2026-07-31 +max_age_days = 90 [claude-fable-5] input = 10.00 @@ -11,8 +20,13 @@ input = 5.00 output = 25.00 [claude-sonnet-5] -input = 3.00 # intro 2.00 through 2026-08-31 -output = 15.00 # intro 10.00 through 2026-08-31 +input = 3.00 +output = 15.00 +# Promotional rate in force through `until` (inclusive); the collector +# applies it to any response whose timestamp falls on or before that date. +promo_input = 2.00 +promo_output = 10.00 +promo_until = 2026-08-31 [claude-haiku-4-5] input = 1.00 diff --git a/evidence/CB-EV-0002-cost-accounting.md b/evidence/CB-EV-0002-cost-accounting.md index a8f986a..e193449 100644 --- a/evidence/CB-EV-0002-cost-accounting.md +++ b/evidence/CB-EV-0002-cost-accounting.md @@ -17,9 +17,9 @@ transcribed by hand (CA-15). | ID | Metric | Target | Measured | Verdict | |---|---|---|---|---| -| AC-1 | pinned total, as two components | $93.32 = $92.21 + $1.11 | **$92.21 main + $1.11 subagent = $93.32** | **met** | +| AC-1 | pinned total, as two components | $93.15 = $92.03 + $1.11 | **$92.03 main + $1.11 subagent = $93.15** | **met** | | AC-2 | reconciliation residual | $0.00 | **$0.000000** | **met** | -| AC-3 | unattributed share reported | present, 33% | **32.5%, own line** | **met** | +| AC-3 | unattributed share reported | present, 33% | **32.4%, own line** | **met** | | AC-4 | composition reported | 5 components | **5 of 5** | **met** | | AC-5 | dedup violation aborts | non-zero exit | **abort raised** | **met** | | AC-6 | zero responses refuses to report | non-zero exit | **0 rows, no number emitted** | **met** | @@ -27,7 +27,7 @@ transcribed by hand (CA-15). | AC-8 | 5m cache priced at 1.25× | $1.25/100k @ fable | **$1.2500 (1h would be $2.0000)** | **met** | | AC-9 | streamed partial output → final | 5,5,195 → 195 | **195** | **met** | -No `unmeasured` rows. AC-1's target was corrected three times before this +No `unmeasured` rows. AC-1's target was corrected four times before and after this run; §4 records why, because the sequence is more useful than the final number. @@ -35,23 +35,23 @@ number. ```text input 690 tok $ 0.00 0.0% - output 323,643 tok $ 11.15 11.9% - cache_read 80,611,798 tok $ 59.75 64.0% + output 323,643 tok $ 11.13 11.9% + cache_read 80,611,798 tok $ 59.67 64.1% write_5m 37,467 tok $ 0.47 0.5% - write_1h 1,672,854 tok $ 21.95 23.5% - TOTAL $ 93.32 + write_1h 1,672,854 tok $ 21.87 23.5% + TOTAL $ 93.15 ``` **88.0% of spend is cache; 11.9% is output.** The ratio of context re-read -to text written is 249:1. A single total would have shown $93.32 and +to text written is 249:1. A single total would have shown $93.15 and concealed all of it — which is exactly what CB-WP-0001's `M-D2-TOK` would have done, and why that metric is now demoted. ## 3. Per-task attribution ```text - UNATTRIBUTED $ 30.32 32.5% - T08 $ 21.02 22.5% + UNATTRIBUTED $ 30.14 32.4% + T08 $ 21.02 22.6% T07 $ 12.01 12.9% T03 $ 9.91 10.6% T04 $ 8.00 8.6% @@ -60,7 +60,7 @@ have done, and why that metric is now demoted. T09 $ 1.29 1.4% ``` -**Stated limit (CA-10):** 32.5% of cost sits in commits whose subject +**Stated limit (CA-10):** 32.4% of cost sits in commits whose subject carries no `T##` tag, so this table is a view over 67.5% of spend. That is a property of commit hygiene, not of the collector. @@ -75,14 +75,15 @@ is worth watching, not yet a conclusion from n=1. - **The per-task figures are not comparable across passes.** They mix models (opus/fable/sonnet) at different price points and different cache states. The dollar figure is comparable; a token count is not. -- **AC-3's 32.5% is a fixture pin, not a quality target.** Improving commit +- **AC-3's 32.4% is a fixture pin, not a quality target.** Improving commit tagging will move it, and that is the desired direction. - **This is one session.** Every ratio here (cache share, $/turn, the compaction effect in §5) is n=1 and should be treated as a hypothesis until a second pass reproduces it. -- **The price sheet cannot express a time-boxed rate.** Sonnet's intro price - lives in a TOML comment, so sonnet-priced work is off by $0.17 here - (0.19%). This becomes a real error on 2026-08-31. +- ~~**The price sheet cannot express a time-boxed rate.**~~ **Fixed + 2026-07-31 by CB-WP-0003 T11** (CA-16/CA-17). Applying the sonnet + promotional rate moved AC-1 from $93.32 to **$93.15** — the $0.17 this + section predicted, now collected rather than merely noted. - **AC-1 is not invoice-verified.** No admin key exists, so the Anthropic billing API could not independently confirm the total. The transcript counters are the same ones billing uses, but that is an argument, not a @@ -145,7 +146,7 @@ largest sample was assumed to hold on the smallest one.** The main transcript is 338 of 346 responses, so 206/206 felt conclusive; the violation lives entirely in the 8 responses nobody checked separately. -Cost of the adversarial review this pass: **$1.11**, against a $93.32 pass. +Cost of the adversarial review this pass: **$1.11**, against a $93.15 pass. It found three approval-blocking defects, one of which (the subagent exclusion) would have made this evidence file certify a broken collector. Second consecutive pass where a ~1% spend on review changed the outcome. diff --git a/specs/CostAccounting.md b/specs/CostAccounting.md index 7fa8ee3..aa19d7a 100644 --- a/specs/CostAccounting.md +++ b/specs/CostAccounting.md @@ -67,6 +67,16 @@ cost = input_tokens × price.input > reported as an unpriced line with its token counts, never dropped and > never priced at a default. +> **CA-16.** Promotional and other time-boxed rates are **data**, not +> comments. A response is priced at the rate in force *at its own +> timestamp*: `promo_input`/`promo_output` apply on or before +> `promo_until`, and the standard rate applies after. + +> **CA-17.** The price sheet carries `recorded` and `max_age_days`, and a +> sheet older than `max_age_days` **aborts the run**. A stale sheet +> invalidates `better` verdicts, so the staleness rule is a command rather +> than a sentence. + ### 1.3 Scope of a measurement > **CA-06.** A measurement enumerates **every** transcript for the repo: @@ -125,9 +135,9 @@ Each row names the command that produces its number, per InnerLoop §Step 4. | ID | Metric | Target | Instrument | |---|---|---|---| -| **AC-1** | reproduces the pinned CB-WP-0001 total | **$93.32** = $92.21 main + $1.11 subagent | `make cost-pin` | +| **AC-1** | reproduces the pinned CB-WP-0001 total | **$93.15** = $92.03 main + $1.11 subagent | `make cost-pin` | | **AC-2** | reconciliation residual (CA-14) | **$0.00** exactly | same command, `reconciled: ok` line | -| **AC-3** | unattributed share reported (CA-10) | present, and **33%** on the pinned run | `cb-cost --pin fc76445 --by-task` | +| **AC-3** | unattributed share reported (CA-10) | present, and **32.4%** on the pinned run | `cb-cost --pin fc76445 --by-task` | | **AC-4** | composition reported (CA-13) | all five components present | `cb-cost --pin fc76445 --composition` | | **AC-5** | dedup invariant asserted (CA-02) | violation exits non-zero | `make cost-test` | | **AC-6** | positive control: refuses to report on zero responses | exits non-zero | `make cost-test` | @@ -145,7 +155,7 @@ assertion against a fixture whose expected value is known and fails loudly; Per InnerLoop §Step 4, the acceptance table is checked against the contracts in this same spec: -- AC-1's $93.32 is reachable only if CA-06 holds (both trees enumerated). +- AC-1's $93.15 is reachable only if CA-06 holds (both trees enumerated). Under a main-file-only collector the target is unreachable — this is the defect the adversarial review caught, where a target of $92.21 would have been hit *only* by a broken collector. @@ -228,6 +238,31 @@ signal — which is exactly the dead-policy failure this replaces, one level up. The mitigation is that it costs nothing to run and is on the one command surface. +## 9. The price sheet + +`benchmarks/baselines/model-prices.toml` is the single source of rates. +Two defects were fixed here, both of the same shape — **a schema that +could not hold the fact it needed**, which is the criticism this +capability's survey levelled at the State Hub: + +1. **A time-boxed rate lived in a comment.** Sonnet's intro price + (2.00/10.00 through 2026-08-31) was written as `# intro …` and was + therefore invisible to the collector, which used the standard rate. Cost + at the pin: **$0.17**, 0.19% of the pass. The real hazard was the + date — on **2026-08-31** the promotion expires and a comment-only + representation would have been wrong in the opposite direction, with + nothing to notice. Now expressed as `promo_*` fields and applied per + CA-16. +2. **The 90-day staleness rule had no check.** It was prose in + `MetricsAndScenarios.md` §1a and every M-D2-CST verdict silently + inherited it. Now `recorded` + `max_age_days` in the sheet, enforced by + CA-17 and exercised by `make cost-test`. + +Applying CA-16 moved AC-1 from $93.32 to **$93.15**. This is a *legitimate* +retarget under the distinction CB-WP-0003 T07 draws: the instrument +disproved the target, rather than the implementation missing it. The +instrument's output is in the same commit. + ## 8. Revisions to M-D2-CST `specs/MetricsAndScenarios.md` §1a is superseded by this spec. M-D2-CST is diff --git a/tools/__pycache__/cb-cost.cpython-312.pyc b/tools/__pycache__/cb-cost.cpython-312.pyc index 8ad4a54..4b96e00 100644 Binary files a/tools/__pycache__/cb-cost.cpython-312.pyc and b/tools/__pycache__/cb-cost.cpython-312.pyc differ diff --git a/tools/cb-cost.py b/tools/cb-cost.py index 59ced0e..a301cc4 100644 --- a/tools/cb-cost.py +++ b/tools/cb-cost.py @@ -72,16 +72,53 @@ def components(usage): } -def price_of(prices, model, toks): +def rates_at(prices, model, when=None): + """Input/output rate in force for `model` at ISO instant `when` (CA-16). + + Promotional rates are data, not comments. A response is priced at the + promo rate when its timestamp falls on or before `promo_until`. + """ + pr = prices.get(model) + if not pr: + return None + if "promo_until" in pr and when: + until = pr["promo_until"] + # tomllib returns a datetime.date for a bare TOML date. + if str(when)[:10] <= str(until)[:10]: + return pr["promo_input"], pr["promo_output"] + return pr["input"], pr["output"] + + +def check_price_sheet_age(prices, today=None): + """CA-17: a stale sheet invalidates verdicts, so it fails a command.""" + import datetime as _dt + + recorded = prices.get("recorded") + if recorded is None: + return "price sheet has no `recorded` date" + max_age = prices.get("max_age_days", 90) + today = today or _dt.date.today() + if isinstance(recorded, _dt.datetime): + recorded = recorded.date() + age = (today - recorded).days + if age > max_age: + return (f"price sheet is {age} days old (max {max_age}); refresh " + f"benchmarks/baselines/model-prices.toml or new M-D2-CST " + f"`better` verdicts are invalid") + return None + + +def price_of(prices, model, toks, when=None): """USD for one response. Returns None when the model is unpriced (CA-05).""" pr = prices.get(model) if not pr: return None + rin, rout = rates_at(prices, model, when) cache = prices["cache"] - unit = pr["input"] / 1e6 + unit = rin / 1e6 return ( toks["input"] * unit - + toks["output"] * pr["output"] / 1e6 + + toks["output"] * rout / 1e6 + toks["cache_read"] * unit * cache["read"] + toks["write_5m"] * unit * cache["write_5m"] + toks["write_1h"] * unit * cache["write_1h"] @@ -281,8 +318,12 @@ def collect(slug, pin_ref=None): # as though it were an answer. raise Abort(f"no responses in {len(paths)} transcript(s) — refusing to report") + stale = check_price_sheet_age(prices) + if stale: + raise Abort(stale) + for r in responses: - r["cost"] = price_of(prices, r["model"], r["toks"]) + r["cost"] = price_of(prices, r["model"], r["toks"], r["timestamp"]) attribute(responses, commit_index(pin)) @@ -298,11 +339,11 @@ def collect(slug, pin_ref=None): continue by_task[r["task"]] += r["cost"] by_model[r["model"]] += r["cost"] - pr = prices[r["model"]] - unit = pr["input"] / 1e6 + rin, rout = rates_at(prices, r["model"], r["timestamp"]) + unit = rin / 1e6 rates = { "input": unit, - "output": pr["output"] / 1e6, + "output": rout / 1e6, "cache_read": unit * cache["read"], "write_5m": unit * cache["write_5m"], "write_1h": unit * cache["write_1h"], @@ -456,6 +497,21 @@ def self_test(): finally: os.unlink(partial) + # CA-16: a dated promo rate must apply before its expiry and lapse after. + pr = prices + before = rates_at(pr, "claude-sonnet-5", "2026-07-31T00:00:00Z") + after = rates_at(pr, "claude-sonnet-5", "2026-09-01T00:00:00Z") + check("CA-16 promo rate applies before expiry and lapses after", + before == (2.0, 10.0) and after == (3.0, 15.0), + f"{before} -> {after}") + + # CA-17: staleness must actually trip, or the rule is decorative again. + import datetime as _dt + fresh = check_price_sheet_age(pr, _dt.date(2026, 8, 1)) + stale = check_price_sheet_age(pr, _dt.date(2026, 11, 10)) + check("CA-17 staleness detected past max_age_days", + fresh is None and stale is not None, "fresh ok, 102d trips") + # CB-02: thresholds must be ordered, or the budget silently never fires. ap_defaults = {"soft": 10.00, "hard": 22.00} check("CB-02 budget thresholds ordered and positive", diff --git a/workplans/CB-WP-0003-loop-hardening.md b/workplans/CB-WP-0003-loop-hardening.md index 682a48c..0db2942 100644 --- a/workplans/CB-WP-0003-loop-hardening.md +++ b/workplans/CB-WP-0003-loop-hardening.md @@ -325,7 +325,7 @@ now price the evaluation itself. ```task id: CB-WP-0003-T11 -status: todo +status: done priority: medium state_hub_task_id: "809215d8-8a5c-4b10-9445-9bc6db9bda42" ```