clay-borg/specs/MetricsAndScenarios.md

Metrics and Scenarios

Status: v0.1 draft — instruments for the InnerLoop. Everything the loop calls "better" is measured through the three artifact kinds defined here: scenarios (correctness), benchmarks (speed and cost), and evidence files (the committed comparison record).

---

1. Metric selection is itself a loop pass

Metrics are not invented ad hoc. Every metric used in an acceptance table carries provenance — a one-line answer to "what state-of-the-art practice does this metric derive from, and how does ours improve on it?"

Provenance tag	Meaning
adopted:<source>	taken as-is from a named practice (e.g. adopted:criterion regression thresholds)
adapted:<source>	derived from a named practice, with the delta stated
novel	no known precedent — requires a sentence justifying why nothing existing fits

A metric with no provenance line is invalid. This keeps the assimilate-and-surpass discipline applied to the measuring instruments, not only to the measured components.

Standing metric set (v0.1)

Selected for the four dimensions; capability specs pick from these first and add capability-specific rows only when these don't cover the claim.

ID	Dimension	Metric	Unit	Provenance
M-D1-COV	D1	numbered spec rules covered by ≥1 passing scenario	%	adapted:requirements-traceability (per-rule, not per-feature)
M-D1-LNK	D1	claimed rules also named in the aggregate source	count	novel — closes the spec→code→scenario chain; instrument: make coverage (AM-1b)
M-D1-SPL	D1	spec lines per numbered rule	lines	novel — proxies statement simplicity; gameable, so paired with M-D1-COV
M-D2-LOC	D2	source LOC excluding tests (tokei)	lines	adopted:tokei
M-D2-DEP	D2	transitive dependency count (cargo tree)	crates	adopted:cargo-deny practice
M-D2-BLD	D2	clean build / incremental build time	s	adopted:cargo timing
M-D2-TOK	D2	tokens consumed per completed workplan task	tokens	novel — demoted 2026-07-31: an input to the cost model, not comparable across models or cache states
M-D2-CST	D2	cost per completed workplan task, attributed per CA-08	USD	adapted:anthropic-pricing — instrument: make cost; normative spec CostAccounting.md
M-D3-THR	D3	events applied per second, headless replay	events/s	adapted:criterion (throughput mode)
M-D3-LAT	D3	p99 command→state-applied latency	µs	adopted:criterion
M-D3-MEM	D3	peak resident memory during benchmark scenario	MB	adopted:/usr/bin/time -v
M-D4-API	D4	public API items (cargo doc item count)	items	adapted:cargo-public-api
M-D4-LEAK	D4	foreign types in canonical interfaces	count	novel — must be 0; enforced by grep/deny rule, the Clay-Borg hard rule
M-D4-SWAP	D4	capability has null + reference impls passing the same conformance suite	bool	adapted:hexagonal-architecture port testing

1b. The coverage gate's two numbers (M-D1-COV, M-D1-LNK)

M-D1-COV counts tags. It proves no rule is unclaimed and no claimed rule is invented; it does not prove a scenario exercises what it names, and that limit is printed with the number every time (InnerLoop implementation rule 4).

M-D1-LNK (CB-WP-0003 T09) closes one link of that chain mechanically: a rule a scenario claims should also appear in the aggregate source, or the claim rests on a tag and nothing else.

Measured 2026-07-31: 49 of 58 claimed rules are named in games/ground/src/lib.rs. Unmet, target 58. The nine unlinked:

GR-D07 GR-F02 GR-L03 GR-O03 GR-P01 GR-P02 GR-P03 GR-P04 GR-T01

The gate reports this rather than failing, deliberately. Closing the gap by adding rule IDs to comments would satisfy the check without establishing that each rule is implemented — which is the overclaim InnerLoop implementation rule 2 exists to prevent, and which CB-WP-0001 committed once already. Each of the nine needs its implementation confirmed before it is tagged. Promoting M-D1-LNK to a failing gate is correct after that, not before.

1a. Token cost accounting (M-D2-CST)

Superseded 2026-07-31 by CostAccounting.md, which is normative for the cost model, attribution, and acceptance metrics. This section is retained for the price-sheet location and the quality-gate rule; where the two disagree, CostAccounting.md wins.

What changed and why: the definition below named no instrument and was never computed, so CB-WP-0001 recorded M-D2-CST as uncomputable while the data sat in the session transcripts. Three of its rules were also wrong in ways that cost real money to discover — see the corrections inline below.

Token counts are only comparable at a single pricepoint. Since work moves between models (Fable for demanding passes, Sonnet/Opus for routine ones), every task's token record carries the model it ran on, and cost is computed against a committed price sheet:

benchmarks/baselines/model-prices.toml   # the price sheet, updated when prices change

# The live sheet is benchmarks/baselines/model-prices.toml — read it,
# do not copy it. An inlined copy here went stale within hours: it still
# showed the sonnet intro rate as a `#` comment after CB-WP-0003 T11 made
# dated rates into data (promo_input / promo_output / promo_until), and
# added `recorded` + `max_age_days` for the staleness check (CA-16/CA-17).

Rules:

• cost = (in_tokens × input + out_tokens × output) / 1e6 … If cache split is unknown, count all input at full price and note it — cost is then an upper bound. Corrected: the cache split is never unknown; it is in every transcript. Treating it as unknown would have priced 80.5M cache reads at 10× their rate. See CostAccounting.md §1.2 (CA-03, CA-04) — cache writes bill at two different TTL rates and must not be aggregated.
• The state-hub task close (update_task_status) already records tokens and model. Corrected: the hub schema has no cache fields and cannot represent 88% of spend, and the figures it recorded for CB-WP-0001 were estimates in error by ~100%. The hub is a sink for numbers computed by make cost, never a source. See CostAccounting.md §6.
• Cheaper is only better at equal quality: M-D2-CST verdicts are valid only alongside passing scenarios/metrics from the same run — a cheap failed pass scores nothing.
• Prices are adopted:anthropic-pricing with a recorded date; a stale sheet (> 90 days or known price change) invalidates new better verdicts on M-D2-CST until refreshed. Now enforced — CA-17 aborts the run rather than leaving this as prose every verdict silently inherited.

Determinism is not a metric but an invariant: N replays of the same seed and command log must produce bit-identical state hashes. Invariant violations fail the run regardless of metric values.

---

2. Scenario format

Scenarios are the correctness currency: executable, declarative, diffable. One file = one scenario. Location: scenarios/<game-or-capability>/<slug>.yaml.

scenario: ground/darvo-interrupted-by-ground   # id = path without extension
description: GROUND practice interrupts a DARVO sequence at the Attack step.
covers: [R-041, R-052, R-053]    # numbered rules from the capability spec
seed: 42
setup:
  players: 3
  preset: standard-3p             # named setup preset from the game spec
  patch:                          # optional explicit state overrides
    relationships:
      - {from: P1, to: P2, kind: rivalry, strength: 2}
commands:                         # ordered; actor-tagged
  - {actor: P1, cmd: trigger_darvo, target: P2}
  - {actor: P2, cmd: play_ground, target: P1}
expect:
  events:                         # ordered subsequence that must occur
    - {type: DarvoInterrupted, step: attack}
  state:                          # end-state assertions, dot-path = value
    darvo_sequences: []
    relationships[P1->P2].strength: 1
  rejects: []                     # commands above that must be rejected, by index

Rules:

• covers is what feeds M-D1-COV; a scenario without covers counts for nothing.
• Assertions are partial: only listed paths are checked. Full-state golden comparison is opt-in via expect.state_hash.
• Every scenario must be deterministic given seed; the runner executes each scenario twice and fails on hash divergence (cheap standing determinism check).
• A failing run writes replays/<scenario-id>.cbreplay (see §4).

---

3. Benchmarks and baselines

Benchmarks live in benchmarks/ as Criterion benches driving scenario files (a benchmark is a scenario run at scale — no separate workload format).

Baselines are committed numbers, recorded once per approved survey and updated only by an explicit ADR:

benchmarks/baselines/<capability>.toml

[M-D3-THR]
value = 120000
unit = "events/s"
source = "boardgame.io v0.50, measured locally, 3-player synthetic log"
recorded = 2026-07-31
machine = "bnt-lap001"

[M-D2-LOC]
value = 8400
unit = "lines"
source = "boardgame.io core, cloc, cited from CB-RES-0001"

• Comparisons are same-machine where machine is set; cross-machine numbers are marked provenance = cited and treated as directional. Per the runnable-baseline option in InnerLoop.md §Step 1, evidence rows compared only against cited numbers cap their verdict at parity; a better verdict requires a locally measured baseline from a fidelity-noted harness.
• Regression rule (adopted:criterion): a merge-blocking regression is

  3% on any D3 metric against our own last evidence file, independent of the SOTA baseline.

---

4. Replay bundle

*.cbreplay is a directory (or tar) with exactly:

manifest.yaml     # scenario id, seed, git commit, schema versions
commands.log      # the full ordered command stream (serialized events optional)
initial.snapshot  # starting state
expected.yaml     # the assertions that failed, with expected vs actual

Contract: cb replay <bundle> (until the CLI exists: the scenario runner's --replay flag) re-executes the bundle headless and must reproduce the failure bit-identically. A bug report without a replay bundle is information; with one, it is work an agent can start.

---

5. Evidence file

evidence/CB-EV-NNNN-<slug>.md — the committed close-out of a loop pass:

# CB-EV-NNNN: <capability>
research: CB-RES-NNNN     adr: ADR-NNNN     spec: specs/<Capability>.md
commit: <sha of measured tree>

| Metric | Baseline | Ours | Verdict |
|---|---|---|---|
| M-D3-THR | 120000 events/s (boardgame.io) | 410000 events/s | better |
| ...every acceptance row, no `unmeasured`... |

## Task cost log
| Task | Model | Tokens in/out | Cost (USD, per price sheet) | Iterations |

## Retrospective
<what the loop itself should change — one paragraph minimum>

Verdicts: better / parity / worse. A worse row does not necessarily fail the pass — the ADR's declared trade governs — but an undeclared worse does.