T11: dated rates and staleness become data, ahead of the 2026-08-31 flip
The price sheet had two defects of one shape -- a schema that could not
hold the fact it needed, the same criticism the cost survey levelled at
the State Hub.
CA-16 time-boxed rates are DATA. Sonnet's intro price lived in a
`# intro ...` comment and was invisible to the collector that
reads the file. Now promo_input/promo_output/promo_until,
applied per response at its own timestamp.
CA-17 the 90-day staleness rule was prose in MetricsAndScenarios 1a
that every M-D2-CST verdict silently inherited. Now `recorded`
+ `max_age_days` in the sheet, and a stale sheet ABORTS.
Both are exercised by make cost-test: the promo rate must apply before
2026-08-31 and lapse after, and a 102-day-old sheet must trip.
Applying CA-16 moved AC-1 from $93.32 to $93.15 -- the $0.17 CB-EV-0002
predicted, now collected rather than noted. That is a legitimate
retarget under T07's distinction: the instrument disproved the target,
and its output is in this commit. The number has now been stated five
times ($248.46, $92.21, $92.87, $93.32, $93.15), each correction from a
different mechanism.
Evidence tables regenerated from the tool rather than hand-patched,
per CA-15 -- which is the rule that exists because hand-typed tables
were the only thing the adversarial review found wrong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
06628e83e1
commit
db731aa0bc
6 changed files with 138 additions and 32 deletions
|
|
@ -67,6 +67,16 @@ cost = input_tokens × price.input
|
|||
> reported as an unpriced line with its token counts, never dropped and
|
||||
> never priced at a default.
|
||||
|
||||
> **CA-16.** Promotional and other time-boxed rates are **data**, not
|
||||
> comments. A response is priced at the rate in force *at its own
|
||||
> timestamp*: `promo_input`/`promo_output` apply on or before
|
||||
> `promo_until`, and the standard rate applies after.
|
||||
|
||||
> **CA-17.** The price sheet carries `recorded` and `max_age_days`, and a
|
||||
> sheet older than `max_age_days` **aborts the run**. A stale sheet
|
||||
> invalidates `better` verdicts, so the staleness rule is a command rather
|
||||
> than a sentence.
|
||||
|
||||
### 1.3 Scope of a measurement
|
||||
|
||||
> **CA-06.** A measurement enumerates **every** transcript for the repo:
|
||||
|
|
@ -125,9 +135,9 @@ Each row names the command that produces its number, per InnerLoop §Step 4.
|
|||
|
||||
| ID | Metric | Target | Instrument |
|
||||
|---|---|---|---|
|
||||
| **AC-1** | reproduces the pinned CB-WP-0001 total | **$93.32** = $92.21 main + $1.11 subagent | `make cost-pin` |
|
||||
| **AC-1** | reproduces the pinned CB-WP-0001 total | **$93.15** = $92.03 main + $1.11 subagent | `make cost-pin` |
|
||||
| **AC-2** | reconciliation residual (CA-14) | **$0.00** exactly | same command, `reconciled: ok` line |
|
||||
| **AC-3** | unattributed share reported (CA-10) | present, and **33%** on the pinned run | `cb-cost --pin fc76445 --by-task` |
|
||||
| **AC-3** | unattributed share reported (CA-10) | present, and **32.4%** on the pinned run | `cb-cost --pin fc76445 --by-task` |
|
||||
| **AC-4** | composition reported (CA-13) | all five components present | `cb-cost --pin fc76445 --composition` |
|
||||
| **AC-5** | dedup invariant asserted (CA-02) | violation exits non-zero | `make cost-test` |
|
||||
| **AC-6** | positive control: refuses to report on zero responses | exits non-zero | `make cost-test` |
|
||||
|
|
@ -145,7 +155,7 @@ assertion against a fixture whose expected value is known and fails loudly;
|
|||
Per InnerLoop §Step 4, the acceptance table is checked against the
|
||||
contracts in this same spec:
|
||||
|
||||
- AC-1's $93.32 is reachable only if CA-06 holds (both trees enumerated).
|
||||
- AC-1's $93.15 is reachable only if CA-06 holds (both trees enumerated).
|
||||
Under a main-file-only collector the target is unreachable — this is the
|
||||
defect the adversarial review caught, where a target of $92.21 would have
|
||||
been hit *only* by a broken collector.
|
||||
|
|
@ -228,6 +238,31 @@ signal — which is exactly the dead-policy failure this replaces, one level
|
|||
up. The mitigation is that it costs nothing to run and is on the one
|
||||
command surface.
|
||||
|
||||
## 9. The price sheet
|
||||
|
||||
`benchmarks/baselines/model-prices.toml` is the single source of rates.
|
||||
Two defects were fixed here, both of the same shape — **a schema that
|
||||
could not hold the fact it needed**, which is the criticism this
|
||||
capability's survey levelled at the State Hub:
|
||||
|
||||
1. **A time-boxed rate lived in a comment.** Sonnet's intro price
|
||||
(2.00/10.00 through 2026-08-31) was written as `# intro …` and was
|
||||
therefore invisible to the collector, which used the standard rate. Cost
|
||||
at the pin: **$0.17**, 0.19% of the pass. The real hazard was the
|
||||
date — on **2026-08-31** the promotion expires and a comment-only
|
||||
representation would have been wrong in the opposite direction, with
|
||||
nothing to notice. Now expressed as `promo_*` fields and applied per
|
||||
CA-16.
|
||||
2. **The 90-day staleness rule had no check.** It was prose in
|
||||
`MetricsAndScenarios.md` §1a and every M-D2-CST verdict silently
|
||||
inherited it. Now `recorded` + `max_age_days` in the sheet, enforced by
|
||||
CA-17 and exercised by `make cost-test`.
|
||||
|
||||
Applying CA-16 moved AC-1 from $93.32 to **$93.15**. This is a *legitimate*
|
||||
retarget under the distinction CB-WP-0003 T07 draws: the instrument
|
||||
disproved the target, rather than the implementation missing it. The
|
||||
instrument's output is in the same commit.
|
||||
|
||||
## 8. Revisions to M-D2-CST
|
||||
|
||||
`specs/MetricsAndScenarios.md` §1a is superseded by this spec. M-D2-CST is
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue