T11: dated rates and staleness become data, ahead of the 2026-08-31 flip
The price sheet had two defects of one shape -- a schema that could not
hold the fact it needed, the same criticism the cost survey levelled at
the State Hub.
CA-16 time-boxed rates are DATA. Sonnet's intro price lived in a
`# intro ...` comment and was invisible to the collector that
reads the file. Now promo_input/promo_output/promo_until,
applied per response at its own timestamp.
CA-17 the 90-day staleness rule was prose in MetricsAndScenarios 1a
that every M-D2-CST verdict silently inherited. Now `recorded`
+ `max_age_days` in the sheet, and a stale sheet ABORTS.
Both are exercised by make cost-test: the promo rate must apply before
2026-08-31 and lapse after, and a 102-day-old sheet must trip.
Applying CA-16 moved AC-1 from $93.32 to $93.15 -- the $0.17 CB-EV-0002
predicted, now collected rather than noted. That is a legitimate
retarget under T07's distinction: the instrument disproved the target,
and its output is in this commit. The number has now been stated five
times ($248.46, $92.21, $92.87, $93.32, $93.15), each correction from a
different mechanism.
Evidence tables regenerated from the tool rather than hand-patched,
per CA-15 -- which is the rule that exists because hand-typed tables
were the only thing the adversarial review found wrong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
06628e83e1
commit
db731aa0bc
6 changed files with 138 additions and 32 deletions
|
|
@ -1,6 +1,15 @@
|
|||
# USD per million tokens; source: Anthropic pricing, recorded 2026-07-31
|
||||
# Governed by specs/MetricsAndScenarios.md §1a — refresh on price change or
|
||||
# after 90 days, or new M-D2-CST 'better' verdicts are invalid.
|
||||
# USD per million tokens. Source: Anthropic pricing.
|
||||
# Governed by specs/CostAccounting.md §9 — a stale sheet invalidates new
|
||||
# M-D2-CST `better` verdicts. Staleness is enforced by `make cost-test`,
|
||||
# not by this comment.
|
||||
#
|
||||
# Dated rates are DATA, not comments. A promotional rate written as a
|
||||
# comment is invisible to the collector that reads this file: the sonnet
|
||||
# intro price sat in a `#` remark and was silently ignored, costing $0.17
|
||||
# on a $93.32 pass and heading for a sign flip on 2026-08-31.
|
||||
|
||||
recorded = 2026-07-31
|
||||
max_age_days = 90
|
||||
|
||||
[claude-fable-5]
|
||||
input = 10.00
|
||||
|
|
@ -11,8 +20,13 @@ input = 5.00
|
|||
output = 25.00
|
||||
|
||||
[claude-sonnet-5]
|
||||
input = 3.00 # intro 2.00 through 2026-08-31
|
||||
output = 15.00 # intro 10.00 through 2026-08-31
|
||||
input = 3.00
|
||||
output = 15.00
|
||||
# Promotional rate in force through `until` (inclusive); the collector
|
||||
# applies it to any response whose timestamp falls on or before that date.
|
||||
promo_input = 2.00
|
||||
promo_output = 10.00
|
||||
promo_until = 2026-08-31
|
||||
|
||||
[claude-haiku-4-5]
|
||||
input = 1.00
|
||||
|
|
|
|||
|
|
@ -17,9 +17,9 @@ transcribed by hand (CA-15).
|
|||
|
||||
| ID | Metric | Target | Measured | Verdict |
|
||||
|---|---|---|---|---|
|
||||
| AC-1 | pinned total, as two components | $93.32 = $92.21 + $1.11 | **$92.21 main + $1.11 subagent = $93.32** | **met** |
|
||||
| AC-1 | pinned total, as two components | $93.15 = $92.03 + $1.11 | **$92.03 main + $1.11 subagent = $93.15** | **met** |
|
||||
| AC-2 | reconciliation residual | $0.00 | **$0.000000** | **met** |
|
||||
| AC-3 | unattributed share reported | present, 33% | **32.5%, own line** | **met** |
|
||||
| AC-3 | unattributed share reported | present, 33% | **32.4%, own line** | **met** |
|
||||
| AC-4 | composition reported | 5 components | **5 of 5** | **met** |
|
||||
| AC-5 | dedup violation aborts | non-zero exit | **abort raised** | **met** |
|
||||
| AC-6 | zero responses refuses to report | non-zero exit | **0 rows, no number emitted** | **met** |
|
||||
|
|
@ -27,7 +27,7 @@ transcribed by hand (CA-15).
|
|||
| AC-8 | 5m cache priced at 1.25× | $1.25/100k @ fable | **$1.2500 (1h would be $2.0000)** | **met** |
|
||||
| AC-9 | streamed partial output → final | 5,5,195 → 195 | **195** | **met** |
|
||||
|
||||
No `unmeasured` rows. AC-1's target was corrected three times before this
|
||||
No `unmeasured` rows. AC-1's target was corrected four times before and after this
|
||||
run; §4 records why, because the sequence is more useful than the final
|
||||
number.
|
||||
|
||||
|
|
@ -35,23 +35,23 @@ number.
|
|||
|
||||
```text
|
||||
input 690 tok $ 0.00 0.0%
|
||||
output 323,643 tok $ 11.15 11.9%
|
||||
cache_read 80,611,798 tok $ 59.75 64.0%
|
||||
output 323,643 tok $ 11.13 11.9%
|
||||
cache_read 80,611,798 tok $ 59.67 64.1%
|
||||
write_5m 37,467 tok $ 0.47 0.5%
|
||||
write_1h 1,672,854 tok $ 21.95 23.5%
|
||||
TOTAL $ 93.32
|
||||
write_1h 1,672,854 tok $ 21.87 23.5%
|
||||
TOTAL $ 93.15
|
||||
```
|
||||
|
||||
**88.0% of spend is cache; 11.9% is output.** The ratio of context re-read
|
||||
to text written is 249:1. A single total would have shown $93.32 and
|
||||
to text written is 249:1. A single total would have shown $93.15 and
|
||||
concealed all of it — which is exactly what CB-WP-0001's `M-D2-TOK` would
|
||||
have done, and why that metric is now demoted.
|
||||
|
||||
## 3. Per-task attribution
|
||||
|
||||
```text
|
||||
UNATTRIBUTED $ 30.32 32.5%
|
||||
T08 $ 21.02 22.5%
|
||||
UNATTRIBUTED $ 30.14 32.4%
|
||||
T08 $ 21.02 22.6%
|
||||
T07 $ 12.01 12.9%
|
||||
T03 $ 9.91 10.6%
|
||||
T04 $ 8.00 8.6%
|
||||
|
|
@ -60,7 +60,7 @@ have done, and why that metric is now demoted.
|
|||
T09 $ 1.29 1.4%
|
||||
```
|
||||
|
||||
**Stated limit (CA-10):** 32.5% of cost sits in commits whose subject
|
||||
**Stated limit (CA-10):** 32.4% of cost sits in commits whose subject
|
||||
carries no `T##` tag, so this table is a view over 67.5% of spend. That is
|
||||
a property of commit hygiene, not of the collector.
|
||||
|
||||
|
|
@ -75,14 +75,15 @@ is worth watching, not yet a conclusion from n=1.
|
|||
- **The per-task figures are not comparable across passes.** They mix
|
||||
models (opus/fable/sonnet) at different price points and different cache
|
||||
states. The dollar figure is comparable; a token count is not.
|
||||
- **AC-3's 32.5% is a fixture pin, not a quality target.** Improving commit
|
||||
- **AC-3's 32.4% is a fixture pin, not a quality target.** Improving commit
|
||||
tagging will move it, and that is the desired direction.
|
||||
- **This is one session.** Every ratio here (cache share, $/turn, the
|
||||
compaction effect in §5) is n=1 and should be treated as a hypothesis
|
||||
until a second pass reproduces it.
|
||||
- **The price sheet cannot express a time-boxed rate.** Sonnet's intro price
|
||||
lives in a TOML comment, so sonnet-priced work is off by $0.17 here
|
||||
(0.19%). This becomes a real error on 2026-08-31.
|
||||
- ~~**The price sheet cannot express a time-boxed rate.**~~ **Fixed
|
||||
2026-07-31 by CB-WP-0003 T11** (CA-16/CA-17). Applying the sonnet
|
||||
promotional rate moved AC-1 from $93.32 to **$93.15** — the $0.17 this
|
||||
section predicted, now collected rather than merely noted.
|
||||
- **AC-1 is not invoice-verified.** No admin key exists, so the Anthropic
|
||||
billing API could not independently confirm the total. The transcript
|
||||
counters are the same ones billing uses, but that is an argument, not a
|
||||
|
|
@ -145,7 +146,7 @@ largest sample was assumed to hold on the smallest one.** The main
|
|||
transcript is 338 of 346 responses, so 206/206 felt conclusive; the
|
||||
violation lives entirely in the 8 responses nobody checked separately.
|
||||
|
||||
Cost of the adversarial review this pass: **$1.11**, against a $93.32 pass.
|
||||
Cost of the adversarial review this pass: **$1.11**, against a $93.15 pass.
|
||||
It found three approval-blocking defects, one of which (the subagent
|
||||
exclusion) would have made this evidence file certify a broken collector.
|
||||
Second consecutive pass where a ~1% spend on review changed the outcome.
|
||||
|
|
|
|||
|
|
@ -67,6 +67,16 @@ cost = input_tokens × price.input
|
|||
> reported as an unpriced line with its token counts, never dropped and
|
||||
> never priced at a default.
|
||||
|
||||
> **CA-16.** Promotional and other time-boxed rates are **data**, not
|
||||
> comments. A response is priced at the rate in force *at its own
|
||||
> timestamp*: `promo_input`/`promo_output` apply on or before
|
||||
> `promo_until`, and the standard rate applies after.
|
||||
|
||||
> **CA-17.** The price sheet carries `recorded` and `max_age_days`, and a
|
||||
> sheet older than `max_age_days` **aborts the run**. A stale sheet
|
||||
> invalidates `better` verdicts, so the staleness rule is a command rather
|
||||
> than a sentence.
|
||||
|
||||
### 1.3 Scope of a measurement
|
||||
|
||||
> **CA-06.** A measurement enumerates **every** transcript for the repo:
|
||||
|
|
@ -125,9 +135,9 @@ Each row names the command that produces its number, per InnerLoop §Step 4.
|
|||
|
||||
| ID | Metric | Target | Instrument |
|
||||
|---|---|---|---|
|
||||
| **AC-1** | reproduces the pinned CB-WP-0001 total | **$93.32** = $92.21 main + $1.11 subagent | `make cost-pin` |
|
||||
| **AC-1** | reproduces the pinned CB-WP-0001 total | **$93.15** = $92.03 main + $1.11 subagent | `make cost-pin` |
|
||||
| **AC-2** | reconciliation residual (CA-14) | **$0.00** exactly | same command, `reconciled: ok` line |
|
||||
| **AC-3** | unattributed share reported (CA-10) | present, and **33%** on the pinned run | `cb-cost --pin fc76445 --by-task` |
|
||||
| **AC-3** | unattributed share reported (CA-10) | present, and **32.4%** on the pinned run | `cb-cost --pin fc76445 --by-task` |
|
||||
| **AC-4** | composition reported (CA-13) | all five components present | `cb-cost --pin fc76445 --composition` |
|
||||
| **AC-5** | dedup invariant asserted (CA-02) | violation exits non-zero | `make cost-test` |
|
||||
| **AC-6** | positive control: refuses to report on zero responses | exits non-zero | `make cost-test` |
|
||||
|
|
@ -145,7 +155,7 @@ assertion against a fixture whose expected value is known and fails loudly;
|
|||
Per InnerLoop §Step 4, the acceptance table is checked against the
|
||||
contracts in this same spec:
|
||||
|
||||
- AC-1's $93.32 is reachable only if CA-06 holds (both trees enumerated).
|
||||
- AC-1's $93.15 is reachable only if CA-06 holds (both trees enumerated).
|
||||
Under a main-file-only collector the target is unreachable — this is the
|
||||
defect the adversarial review caught, where a target of $92.21 would have
|
||||
been hit *only* by a broken collector.
|
||||
|
|
@ -228,6 +238,31 @@ signal — which is exactly the dead-policy failure this replaces, one level
|
|||
up. The mitigation is that it costs nothing to run and is on the one
|
||||
command surface.
|
||||
|
||||
## 9. The price sheet
|
||||
|
||||
`benchmarks/baselines/model-prices.toml` is the single source of rates.
|
||||
Two defects were fixed here, both of the same shape — **a schema that
|
||||
could not hold the fact it needed**, which is the criticism this
|
||||
capability's survey levelled at the State Hub:
|
||||
|
||||
1. **A time-boxed rate lived in a comment.** Sonnet's intro price
|
||||
(2.00/10.00 through 2026-08-31) was written as `# intro …` and was
|
||||
therefore invisible to the collector, which used the standard rate. Cost
|
||||
at the pin: **$0.17**, 0.19% of the pass. The real hazard was the
|
||||
date — on **2026-08-31** the promotion expires and a comment-only
|
||||
representation would have been wrong in the opposite direction, with
|
||||
nothing to notice. Now expressed as `promo_*` fields and applied per
|
||||
CA-16.
|
||||
2. **The 90-day staleness rule had no check.** It was prose in
|
||||
`MetricsAndScenarios.md` §1a and every M-D2-CST verdict silently
|
||||
inherited it. Now `recorded` + `max_age_days` in the sheet, enforced by
|
||||
CA-17 and exercised by `make cost-test`.
|
||||
|
||||
Applying CA-16 moved AC-1 from $93.32 to **$93.15**. This is a *legitimate*
|
||||
retarget under the distinction CB-WP-0003 T07 draws: the instrument
|
||||
disproved the target, rather than the implementation missing it. The
|
||||
instrument's output is in the same commit.
|
||||
|
||||
## 8. Revisions to M-D2-CST
|
||||
|
||||
`specs/MetricsAndScenarios.md` §1a is superseded by this spec. M-D2-CST is
|
||||
|
|
|
|||
Binary file not shown.
|
|
@ -72,16 +72,53 @@ def components(usage):
|
|||
}
|
||||
|
||||
|
||||
def price_of(prices, model, toks):
|
||||
def rates_at(prices, model, when=None):
|
||||
"""Input/output rate in force for `model` at ISO instant `when` (CA-16).
|
||||
|
||||
Promotional rates are data, not comments. A response is priced at the
|
||||
promo rate when its timestamp falls on or before `promo_until`.
|
||||
"""
|
||||
pr = prices.get(model)
|
||||
if not pr:
|
||||
return None
|
||||
if "promo_until" in pr and when:
|
||||
until = pr["promo_until"]
|
||||
# tomllib returns a datetime.date for a bare TOML date.
|
||||
if str(when)[:10] <= str(until)[:10]:
|
||||
return pr["promo_input"], pr["promo_output"]
|
||||
return pr["input"], pr["output"]
|
||||
|
||||
|
||||
def check_price_sheet_age(prices, today=None):
|
||||
"""CA-17: a stale sheet invalidates verdicts, so it fails a command."""
|
||||
import datetime as _dt
|
||||
|
||||
recorded = prices.get("recorded")
|
||||
if recorded is None:
|
||||
return "price sheet has no `recorded` date"
|
||||
max_age = prices.get("max_age_days", 90)
|
||||
today = today or _dt.date.today()
|
||||
if isinstance(recorded, _dt.datetime):
|
||||
recorded = recorded.date()
|
||||
age = (today - recorded).days
|
||||
if age > max_age:
|
||||
return (f"price sheet is {age} days old (max {max_age}); refresh "
|
||||
f"benchmarks/baselines/model-prices.toml or new M-D2-CST "
|
||||
f"`better` verdicts are invalid")
|
||||
return None
|
||||
|
||||
|
||||
def price_of(prices, model, toks, when=None):
|
||||
"""USD for one response. Returns None when the model is unpriced (CA-05)."""
|
||||
pr = prices.get(model)
|
||||
if not pr:
|
||||
return None
|
||||
rin, rout = rates_at(prices, model, when)
|
||||
cache = prices["cache"]
|
||||
unit = pr["input"] / 1e6
|
||||
unit = rin / 1e6
|
||||
return (
|
||||
toks["input"] * unit
|
||||
+ toks["output"] * pr["output"] / 1e6
|
||||
+ toks["output"] * rout / 1e6
|
||||
+ toks["cache_read"] * unit * cache["read"]
|
||||
+ toks["write_5m"] * unit * cache["write_5m"]
|
||||
+ toks["write_1h"] * unit * cache["write_1h"]
|
||||
|
|
@ -281,8 +318,12 @@ def collect(slug, pin_ref=None):
|
|||
# as though it were an answer.
|
||||
raise Abort(f"no responses in {len(paths)} transcript(s) — refusing to report")
|
||||
|
||||
stale = check_price_sheet_age(prices)
|
||||
if stale:
|
||||
raise Abort(stale)
|
||||
|
||||
for r in responses:
|
||||
r["cost"] = price_of(prices, r["model"], r["toks"])
|
||||
r["cost"] = price_of(prices, r["model"], r["toks"], r["timestamp"])
|
||||
|
||||
attribute(responses, commit_index(pin))
|
||||
|
||||
|
|
@ -298,11 +339,11 @@ def collect(slug, pin_ref=None):
|
|||
continue
|
||||
by_task[r["task"]] += r["cost"]
|
||||
by_model[r["model"]] += r["cost"]
|
||||
pr = prices[r["model"]]
|
||||
unit = pr["input"] / 1e6
|
||||
rin, rout = rates_at(prices, r["model"], r["timestamp"])
|
||||
unit = rin / 1e6
|
||||
rates = {
|
||||
"input": unit,
|
||||
"output": pr["output"] / 1e6,
|
||||
"output": rout / 1e6,
|
||||
"cache_read": unit * cache["read"],
|
||||
"write_5m": unit * cache["write_5m"],
|
||||
"write_1h": unit * cache["write_1h"],
|
||||
|
|
@ -456,6 +497,21 @@ def self_test():
|
|||
finally:
|
||||
os.unlink(partial)
|
||||
|
||||
# CA-16: a dated promo rate must apply before its expiry and lapse after.
|
||||
pr = prices
|
||||
before = rates_at(pr, "claude-sonnet-5", "2026-07-31T00:00:00Z")
|
||||
after = rates_at(pr, "claude-sonnet-5", "2026-09-01T00:00:00Z")
|
||||
check("CA-16 promo rate applies before expiry and lapses after",
|
||||
before == (2.0, 10.0) and after == (3.0, 15.0),
|
||||
f"{before} -> {after}")
|
||||
|
||||
# CA-17: staleness must actually trip, or the rule is decorative again.
|
||||
import datetime as _dt
|
||||
fresh = check_price_sheet_age(pr, _dt.date(2026, 8, 1))
|
||||
stale = check_price_sheet_age(pr, _dt.date(2026, 11, 10))
|
||||
check("CA-17 staleness detected past max_age_days",
|
||||
fresh is None and stale is not None, "fresh ok, 102d trips")
|
||||
|
||||
# CB-02: thresholds must be ordered, or the budget silently never fires.
|
||||
ap_defaults = {"soft": 10.00, "hard": 22.00}
|
||||
check("CB-02 budget thresholds ordered and positive",
|
||||
|
|
|
|||
|
|
@ -325,7 +325,7 @@ now price the evaluation itself.
|
|||
|
||||
```task
|
||||
id: CB-WP-0003-T11
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "809215d8-8a5c-4b10-9445-9bc6db9bda42"
|
||||
```
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue