# Metrics and Scenarios Status: **v0.1 draft** — instruments for the [InnerLoop](InnerLoop.md). Everything the loop calls "better" is measured through the three artifact kinds defined here: **scenarios** (correctness), **benchmarks** (speed and cost), and **evidence files** (the committed comparison record). --- ## 1. Metric selection is itself a loop pass Metrics are not invented ad hoc. Every metric used in an acceptance table carries **provenance** — a one-line answer to "what state-of-the-art practice does this metric derive from, and how does ours improve on it?" | Provenance tag | Meaning | |---|---| | `adopted:` | taken as-is from a named practice (e.g. `adopted:criterion` regression thresholds) | | `adapted:` | derived from a named practice, with the delta stated | | `novel` | no known precedent — requires a sentence justifying why nothing existing fits | A metric with no provenance line is invalid. This keeps the assimilate-and-surpass discipline applied to the measuring instruments, not only to the measured components. ### Standing metric set (v0.1) Selected for the four dimensions; capability specs pick from these first and add capability-specific rows only when these don't cover the claim. | ID | Dimension | Metric | Unit | Provenance | |---|---|---|---|---| | M-D1-COV | D1 | numbered spec rules covered by ≥1 passing scenario | % | adapted:requirements-traceability (per-rule, not per-feature) | | M-D1-LNK | D1 | claimed rules also **named in the aggregate source** | count | novel — closes the spec→code→scenario chain; instrument: `make coverage` (AM-1b) | | M-D1-SPL | D1 | spec lines per numbered rule | lines | novel — proxies statement simplicity; gameable, so paired with M-D1-COV | | M-D2-LOC | D2 | source LOC excluding tests (tokei) | lines | adopted:tokei | | M-D2-DEP | D2 | transitive dependency count (cargo tree) | crates | adopted:cargo-deny practice | | M-D2-BLD | D2 | clean build / incremental build time | s | adopted:cargo timing | | M-D2-TOK | D2 | tokens consumed per completed workplan task | tokens | novel — **demoted 2026-07-31**: an input to the cost model, not comparable across models or cache states | | M-D2-CST | D2 | **cost** per completed workplan task, attributed per CA-08 | USD | adapted:anthropic-pricing — instrument: `make cost`; normative spec [CostAccounting.md](CostAccounting.md) | | M-D3-THR | D3 | events applied per second, headless replay | events/s | adapted:criterion (throughput mode) | | M-D3-LAT | D3 | p99 command→state-applied latency | µs | adopted:criterion | | M-D3-MEM | D3 | peak resident memory during benchmark scenario | MB | adopted:/usr/bin/time -v | | M-D4-API | D4 | public API items (cargo doc item count) | items | adapted:cargo-public-api | | M-D4-LEAK | D4 | foreign types in canonical interfaces | count | novel — must be 0; enforced by grep/deny rule, the Clay-Borg hard rule | | M-D4-SWAP | D4 | capability has null + reference impls passing the same conformance suite | bool | adapted:hexagonal-architecture port testing | ### 1b. The coverage gate's numbers (M-D1-COV, M-D1-LNK) > **Widened 2026-07-31 (CB-WP-0005 T01).** Until then the instrument > matched `GR-` only, against `GroundRules.md` only, and linked against > one source file, so all **18** K-rules were outside it. > `58/58 (100%)` read as "all rules". > > `make coverage` now reports a second, separate denominator: > **18 of 18** K-rules are named across the source. > Unlinked: **(none)**. > K10 was unlinked until T06 implemented replay bundles; K14 and K18 until > T07 amended one and implemented the other. The link is complete — which > is a statement about **names**, not about assertions, and the gate says > so on every run. > > **Kernel rules are link-only, by design.** They are kernel invariants, > not game rules: there is no kernel aggregate, setup preset or command > vocabulary, so a `scenarios/kernel/*.yaml` carrying `covers: [K11]` > would be a tag in a directory the runner cannot dispatch. Claiming > scenario coverage for them would be the inflation this gate exists to > prevent. > > Per ADR-0005 §5 the kernel arm **reports without feeding the exit > code until 2026-08-31**, then binds. A newly widened denominator is not > a regression. The date lives in `tools/rule-coverage.py`, not here, and > the tool prints the days remaining on every run. **Both numbers count names.** Neither proves anything fails when a rule is violated. That is **M-D1-MUT** (`make mutation-check`, CB-WP-0005 T02) — four of the seven defects found by CB-RES-0004 were named in the source and inert, and therefore invisible to everything on this page. M-D1-COV counts tags. It proves no rule is unclaimed and no claimed rule is invented; it does **not** prove a scenario exercises what it names, and that limit is printed with the number every time (InnerLoop implementation rule 4). **M-D1-LNK** (CB-WP-0003 T09) closes one link of that chain mechanically: a rule a scenario claims should also appear in the aggregate source, or the claim rests on a tag and nothing else. Measured 2026-08-04: **50 of 59** claimed rules are named in `games/ground/src/lib.rs`. **Unmet**, target 59. The nine unlinked: ```text GR-D07 GR-F02 GR-L03 GR-O03 GR-P01 GR-P02 GR-P03 GR-P04 GR-T01 ``` The gate **reports** this rather than failing, deliberately. Closing the gap by adding rule IDs to comments would satisfy the check without establishing that each rule is implemented — which is the overclaim InnerLoop implementation rule 2 exists to prevent, and which CB-WP-0001 committed once already. Each of the nine needs its implementation confirmed before it is tagged. Promoting M-D1-LNK to a failing gate is correct **after** that, not before. ### 1a. Token cost accounting (M-D2-CST) > **Superseded 2026-07-31 by [CostAccounting.md](CostAccounting.md)**, which > is normative for the cost model, attribution, and acceptance metrics. > This section is retained for the price-sheet location and the > quality-gate rule; where the two disagree, CostAccounting.md wins. > > What changed and why: the definition below named no instrument and was > never computed, so CB-WP-0001 recorded M-D2-CST as *uncomputable* while > the data sat in the session transcripts. Three of its rules were also > wrong in ways that cost real money to discover — see the corrections > inline below. Token counts are only comparable at a single pricepoint. Since work moves between models (Fable for demanding passes, Sonnet/Opus for routine ones), every task's token record carries the **model** it ran on, and cost is computed against a committed price sheet: ```text benchmarks/baselines/model-prices.toml # the price sheet, updated when prices change ``` ```toml # The live sheet is benchmarks/baselines/model-prices.toml — read it, # do not copy it. An inlined copy here went stale within hours: it still # showed the sonnet intro rate as a `#` comment after CB-WP-0003 T11 made # dated rates into data (promo_input / promo_output / promo_until), and # added `recorded` + `max_age_days` for the staleness check (CA-16/CA-17). ``` Rules: - ~~`cost = (in_tokens × input + out_tokens × output) / 1e6` … If cache split is unknown, count all input at full price and note it — cost is then an upper bound.~~ **Corrected:** the cache split is never unknown; it is in every transcript. Treating it as unknown would have priced 80.5M cache reads at 10× their rate. See CostAccounting.md §1.2 (CA-03, CA-04) — cache writes bill at two different TTL rates and must not be aggregated. - ~~The state-hub task close (`update_task_status`) already records tokens and `model`.~~ **Corrected:** the hub schema has no cache fields and cannot represent 88% of spend, and the figures it recorded for CB-WP-0001 were estimates in error by ~100%. The hub is a **sink** for numbers computed by `make cost`, never a source. See CostAccounting.md §6. - **Cheaper is only better at equal quality**: M-D2-CST verdicts are valid only alongside passing scenarios/metrics from the same run — a cheap failed pass scores nothing. - Prices are `adopted:anthropic-pricing` with a recorded date; a stale sheet (> 90 days or known price change) invalidates new `better` verdicts on M-D2-CST until refreshed. **Now enforced** — CA-17 aborts the run rather than leaving this as prose every verdict silently inherited. Determinism is not a metric but an **invariant**: N replays of the same seed and command log must produce bit-identical state hashes. Invariant violations fail the run regardless of metric values. --- ## 2. Scenario format Scenarios are the correctness currency: executable, declarative, diffable. One file = one scenario. Location: `scenarios//.yaml`. ```yaml scenario: ground/darvo-interrupted-by-ground # id = path without extension description: GROUND practice interrupts a DARVO sequence at the Attack step. covers: [R-041, R-052, R-053] # numbered rules from the capability spec seed: 42 setup: players: 3 preset: standard-3p # named setup preset from the game spec patch: # optional explicit state overrides relationships: - {from: P1, to: P2, kind: rivalry, strength: 2} commands: # ordered; actor-tagged - {actor: P1, cmd: trigger_darvo, target: P2} - {actor: P2, cmd: play_ground, target: P1} expect: events: # ordered subsequence that must occur - {type: DarvoInterrupted, step: attack} state: # end-state assertions, dot-path = value darvo_sequences: [] relationships[P1->P2].strength: 1 rejects: [] # commands above that must be rejected, by index ``` Rules: - `covers` is what feeds M-D1-COV; a scenario without `covers` counts for nothing. - Assertions are **partial**: only listed paths are checked. Full-state golden comparison is opt-in via `expect.state_hash`. - Every scenario must be deterministic given `seed`; the runner executes each scenario twice and fails on hash divergence (cheap standing determinism check). - A failing run writes `replays/.cbreplay` (see §4). --- ## 3. Benchmarks and baselines Benchmarks live in `benchmarks/` as Criterion benches driving scenario files (a benchmark is a scenario run at scale — no separate workload format). Baselines are **committed numbers**, recorded once per approved survey and updated only by an explicit ADR: ```text benchmarks/baselines/.toml ``` ```toml [M-D3-THR] value = 120000 unit = "events/s" source = "boardgame.io v0.50, measured locally, 3-player synthetic log" recorded = 2026-07-31 machine = "bnt-lap001" [M-D2-LOC] value = 8400 unit = "lines" source = "boardgame.io core, cloc, cited from CB-RES-0001" ``` - Comparisons are same-machine where `machine` is set; cross-machine numbers are marked `provenance = cited` and treated as directional. Per the runnable-baseline option in [InnerLoop.md](InnerLoop.md) §Step 1, evidence rows compared only against cited numbers cap their verdict at `parity`; a `better` verdict requires a locally measured baseline from a fidelity-noted harness. - Regression rule (adopted:criterion): a merge-blocking regression is >3% on any D3 metric against **our own** last evidence file, independent of the SOTA baseline. --- ## 4. Replay bundle `*.cbreplay` is a directory (or tar) with exactly: ```text manifest.yaml # scenario id, seed, git commit, schema versions commands.log # the full ordered command stream (serialized events optional) initial.snapshot # starting state expected.yaml # the assertions that failed, with expected vs actual ``` Contract: `cb replay ` (until the CLI exists: the scenario runner's `--replay` flag) re-executes the bundle headless and must reproduce the failure bit-identically. A bug report without a replay bundle is information; with one, it is work an agent can start. --- ## 5. Evidence file `evidence/CB-EV-NNNN-.md` — the committed close-out of a loop pass: ```markdown # CB-EV-NNNN: research: CB-RES-NNNN adr: ADR-NNNN spec: specs/.md commit: | Metric | Baseline | Ours | Verdict | |---|---|---|---| | M-D3-THR | 120000 events/s (boardgame.io) | 410000 events/s | better | | ...every acceptance row, no `unmeasured`... | ## Task cost log | Task | Model | Tokens in/out | Cost (USD, per price sheet) | Iterations | ## Retrospective ``` Verdicts: `better / parity / worse`. A `worse` row does not necessarily fail the pass — the ADR's declared trade governs — but an undeclared `worse` does.