2026-07-31 00:31:14 +02:00
# Metrics and Scenarios
Status: **v0.1 draft** — instruments for the [InnerLoop ](InnerLoop.md ).
Everything the loop calls "better" is measured through the three artifact
kinds defined here: **scenarios** (correctness), **benchmarks** (speed and
cost), and **evidence files** (the committed comparison record).
---
## 1. Metric selection is itself a loop pass
Metrics are not invented ad hoc. Every metric used in an acceptance table
carries **provenance** — a one-line answer to "what state-of-the-art
practice does this metric derive from, and how does ours improve on it?"
| Provenance tag | Meaning |
|---|---|
| `adopted:<source>` | taken as-is from a named practice (e.g. `adopted:criterion` regression thresholds) |
| `adapted:<source>` | derived from a named practice, with the delta stated |
| `novel` | no known precedent — requires a sentence justifying why nothing existing fits |
A metric with no provenance line is invalid. This keeps the
assimilate-and-surpass discipline applied to the measuring instruments,
not only to the measured components.
### Standing metric set (v0.1)
Selected for the four dimensions; capability specs pick from these first
and add capability-specific rows only when these don't cover the claim.
| ID | Dimension | Metric | Unit | Provenance |
|---|---|---|---|---|
| M-D1-COV | D1 | numbered spec rules covered by ≥1 passing scenario | % | adapted:requirements-traceability (per-rule, not per-feature) |
2026-07-31 09:27:14 +02:00
| M-D1-LNK | D1 | claimed rules also **named in the aggregate source** | count | novel — closes the spec→code→scenario chain; instrument: `make coverage` (AM-1b) |
2026-07-31 00:31:14 +02:00
| M-D1-SPL | D1 | spec lines per numbered rule | lines | novel — proxies statement simplicity; gameable, so paired with M-D1-COV |
| M-D2-LOC | D2 | source LOC excluding tests (tokei) | lines | adopted:tokei |
| M-D2-DEP | D2 | transitive dependency count (cargo tree) | crates | adopted:cargo-deny practice |
| M-D2-BLD | D2 | clean build / incremental build time | s | adopted:cargo timing |
T03: specs/CostAccounting.md — 15 contracts, 8 metrics, each with a command
Normative spec for the cost model (dedup by requestId, per-model and
per-TTL pricing), the attribution contract (ending-at-commit intervals
scoped by sessionId), and the reported shape (composition, not a total).
Every acceptance row names the command that produces its number, per
InnerLoop Step 4. AC-5..AC-8 are the positive control: cb-cost
--self-test must fail on a dedup violation, on zero responses, on a
missing subagent tree, and on 5m cache priced at the 1h rate.
The feasibility check earns its place here: AC-1's $92.87 is reachable
only if CA-06 holds. The earlier $92.21 target was reachable only by a
collector with the exact blind spot the survey documented.
MetricsAndScenarios 1a superseded. Two of its rules struck through
rather than deleted, because both were wrong in instructive ways: "if
the cache split is unknown, count all input at full price" would have
priced 80.5M cache reads at 10x, and "the hub already records tokens"
named as a source what is only ever a lossy sink. M-D2-TOK demoted --
tokens are not comparable across models or cache states.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:46:12 +02:00
| M-D2-TOK | D2 | tokens consumed per completed workplan task | tokens | novel — **demoted 2026-07-31** : an input to the cost model, not comparable across models or cache states |
| M-D2-CST | D2 | **cost** per completed workplan task, attributed per CA-08 | USD | adapted:anthropic-pricing — instrument: `make cost` ; normative spec [CostAccounting.md ](CostAccounting.md ) |
2026-07-31 00:31:14 +02:00
| M-D3-THR | D3 | events applied per second, headless replay | events/s | adapted:criterion (throughput mode) |
| M-D3-LAT | D3 | p99 command→state-applied latency | µs | adopted:criterion |
| M-D3-MEM | D3 | peak resident memory during benchmark scenario | MB | adopted:/usr/bin/time -v |
| M-D4-API | D4 | public API items (cargo doc item count) | items | adapted:cargo-public-api |
| M-D4-LEAK | D4 | foreign types in canonical interfaces | count | novel — must be 0; enforced by grep/deny rule, the Clay-Borg hard rule |
| M-D4-SWAP | D4 | capability has null + reference impls passing the same conformance suite | bool | adapted:hexagonal-architecture port testing |
CB-WP-0005 T01: spec->code link over every numbered spec and every crate
AGGREGATE becomes a list of source roots and rule patterns become
per-spec, so the link runs over every numbered spec x every crate rather
than GroundRules.md x games/ground/src/lib.rs.
The prediction held on the first run:
AM-1b kernel spec->code link: 15/18 (83%) across 10 source files
unlinked: K10 K14 K18
Kernel rules are link-only by design, and the output says so: they are
kernel invariants with no aggregate, setup preset or command vocabulary,
so scenarios/kernel/*.yaml with covers: [K11] would be a tag in a
directory the runner cannot dispatch. Claiming scenario coverage for them
is the inflation this gate exists to prevent.
Per ADR-0005 §5 the kernel arm reports without feeding the exit code
until 2026-08-31, then binds — the date in the tool, not in prose, with
days remaining printed every run, because open-ended "gate it later" is
how AM-4's targets went unratified for four workplans. The self-test
asserts the gate returns 0 before that date and 2 after.
The zero-rules positive control is replicated on the new denominator: a
kernel regex that stops matching aborts rather than printing 0/0 as
though it were 100%.
The self-test passed while the tool was completely broken. A print(
inside say() became say(), so every real `make coverage` died with
RecursionError while --self-test reported all-ok — it only ever called
kernel_arm(quiet=True) and never executed the reporting path. The control
named the behaviour and did not assert it, which is precisely what this
workplan is about. Fixed by exercising the loud path and asserting it
prints, then verified by re-breaking say() and confirming both new checks
go red. Seventh instance of the harness-does-nothing shape, in the tool
written to find that shape.
Also caught by its own gate: a self-test label that printed "0 K-ids"
beside a passing ">5" assertion, because the detail string rebuilt the
pattern with different escaping. A label that contradicts its own check
is worse than no label.
k_rules, k_linked and k_unlinked are registered facts under facts-check.
A limit of that checker is recorded rather than patched: it is
line-based, so a tagged value that prose-wraps fails.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 16:54:07 +02:00
### 1b. The coverage gate's numbers (M-D1-COV, M-D1-LNK)
> **Widened 2026-07-31 (CB-WP-0005 T01).** Until then the instrument
> matched `GR-` only, against `GroundRules.md` only, and linked against
> one source file, so all **18** K-rules were outside it. <!-- fact:k_rules -->
> `58/58 (100%)` read as "all rules".
>
> `make coverage` now reports a second, separate denominator:
CB-WP-0006 T07: implement K18, amend K14
Two rules, two different answers, which is the point of a task phrased
"implement, or amend and say why".
K18 is implemented. "Criterion benches driving the same scenario format at
scale" was false — the bench hardcoded its commands and never touched
ScenarioFile, while MetricsAndScenarios §3 pointed at a benchmarks/
directory containing only baselines/. benchmarks/synthetic-3p.yaml now
holds the workload and both the bench and bench_shape read it: the
workload is data, not code.
A second defect surfaced while fixing the first. After the bench switched
to the file, bench_shape still hardcoded the same sequence, so the
workload existed twice — deleting end_round from the YAML broke bench-test
while bench_shape kept passing. Duplicated-fact drift in executable form.
Both now read the same include_str! and deleting a command breaks both.
Explicitly not claimed: this does not unblock AM-3. AM-3's baseline is a
declarative game object — moves, turn order, rules. synthetic-3p.yaml is a
command list; the rules live in games/ground. Marking it as AM-3's subject
would compare a script to a game definition, which is the category error
AM-3 is blocked on. The file says so in its own header, where the next
person will be tempted.
K14 is amended. CommitWindow had zero non-test users and GROUND enforces
the same contract inline. Wiring GROUND through it was rejected: it would
change the serialized shape of `selections`, which four scenario files
assert by dot-path and every state hash depends on, for the sole benefit
of making a sentence literally true.
The deciding argument is INTENT's, not convenience: abstractions are
extracted from working games rather than invented in isolation, and no
concept becomes canonical until it survives a second concrete use.
CommitWindow was invented before any game needed it and has survived none.
Imposing it on GROUND would manufacture the first use rather than discover
it. So K14 states what is actually guaranteed, CommitWindow is marked
provisional in the source, and it carries a delete-by date of 2026-12-31.
Kernel spec->code link 16/18 -> 18/18, stated with the caveat the gate
prints every run: that is about names, not assertions.
Two self-tests broke and both broke correctly. rule-coverage's gate test
hardcoded "unlinked rules exist today" and failed when the last one was
linked; it now computes that and asserts the gate fails iff rules are
unlinked. facts' text check rejected k_unlinked once it became
legitimately empty; empty now renders as "(none)" and the check
distinguishes absent from empty.
M-D1-MUT: 8 of 14, unchanged — K14 and K18 are kernel rules, not
acceptance rows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 12:47:16 +02:00
> **18 of 18** K-rules are named across the source. <!-- fact:k_linked -->
> Unlinked: **(none)**. <!-- fact:k_unlinked -->
> K10 was unlinked until T06 implemented replay bundles; K14 and K18 until
> T07 amended one and implemented the other. The link is complete — which
> is a statement about **names**, not about assertions, and the gate says
> so on every run.
CB-WP-0005 T01: spec->code link over every numbered spec and every crate
AGGREGATE becomes a list of source roots and rule patterns become
per-spec, so the link runs over every numbered spec x every crate rather
than GroundRules.md x games/ground/src/lib.rs.
The prediction held on the first run:
AM-1b kernel spec->code link: 15/18 (83%) across 10 source files
unlinked: K10 K14 K18
Kernel rules are link-only by design, and the output says so: they are
kernel invariants with no aggregate, setup preset or command vocabulary,
so scenarios/kernel/*.yaml with covers: [K11] would be a tag in a
directory the runner cannot dispatch. Claiming scenario coverage for them
is the inflation this gate exists to prevent.
Per ADR-0005 §5 the kernel arm reports without feeding the exit code
until 2026-08-31, then binds — the date in the tool, not in prose, with
days remaining printed every run, because open-ended "gate it later" is
how AM-4's targets went unratified for four workplans. The self-test
asserts the gate returns 0 before that date and 2 after.
The zero-rules positive control is replicated on the new denominator: a
kernel regex that stops matching aborts rather than printing 0/0 as
though it were 100%.
The self-test passed while the tool was completely broken. A print(
inside say() became say(), so every real `make coverage` died with
RecursionError while --self-test reported all-ok — it only ever called
kernel_arm(quiet=True) and never executed the reporting path. The control
named the behaviour and did not assert it, which is precisely what this
workplan is about. Fixed by exercising the loud path and asserting it
prints, then verified by re-breaking say() and confirming both new checks
go red. Seventh instance of the harness-does-nothing shape, in the tool
written to find that shape.
Also caught by its own gate: a self-test label that printed "0 K-ids"
beside a passing ">5" assertion, because the detail string rebuilt the
pattern with different escaping. A label that contradicts its own check
is worse than no label.
k_rules, k_linked and k_unlinked are registered facts under facts-check.
A limit of that checker is recorded rather than patched: it is
line-based, so a tagged value that prose-wraps fails.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 16:54:07 +02:00
>
> **Kernel rules are link-only, by design.** They are kernel invariants,
> not game rules: there is no kernel aggregate, setup preset or command
> vocabulary, so a `scenarios/kernel/*.yaml` carrying `covers: [K11]`
> would be a tag in a directory the runner cannot dispatch. Claiming
> scenario coverage for them would be the inflation this gate exists to
> prevent.
>
> Per ADR-0005 §5 the kernel arm **reports without feeding the exit
> code until 2026-08-31**, then binds. A newly widened denominator is not
> a regression. The date lives in `tools/rule-coverage.py`, not here, and
> the tool prints the days remaining on every run.
**Both numbers count names.** Neither proves anything fails when a rule is
violated. That is **M-D1-MUT** (`make mutation-check` , CB-WP-0005 T02) —
four of the seven defects found by CB-RES-0004 were named in the source
and inert, and therefore invisible to everything on this page.
2026-07-31 09:27:14 +02:00
M-D1-COV counts tags. It proves no rule is unclaimed and no claimed rule
is invented; it does **not** prove a scenario exercises what it names, and
that limit is printed with the number every time (InnerLoop implementation
rule 4).
**M-D1-LNK** (CB-WP-0003 T09) closes one link of that chain mechanically:
a rule a scenario claims should also appear in the aggregate source, or
the claim rests on a tag and nothing else.
CB-WP-0023: SOLVE is legal only where it can do something
Implements ground-game's ruling of 2026-08-03. make all exits 0, 26
scenarios, rule coverage 59/59, and no scenario encoded the bug.
The rule ended up somewhere other than where I put it, and a gate moved
it. It went into legal_commands first; the AM-1 coverage gate then
demanded a scenario for the new GR-P05, and scenarios drive validate, not
the offer layer. A rule enforced only by the offer is enforced only for
clients that ask what is legal -- the browser would be filtered and a
scenario file would walk straight past it. Once GR-P05 moved into
validate, every condition in legal_commands was dead code, and the
layering test said so in those words.
And the reported case was not the one I reported. CB-WP-0018 and the
message to ground-game described SOLVE offered on a FACE-DOWN Problem.
Measured: validate already rejected face-down, so it never was offered.
Problem 1 is the Surface Problem, face-up from the deal -- the
maintainer's three inert SOLVEs were the HAND case, holding no Clarify
for a Clarify Problem. The ruling covers both so nothing is invalidated,
but the record was wrong.
Four conditions asserted separately, because one 'SOLVE is filtered' test
would pass with three of four implemented.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-04 00:18:40 +02:00
Measured 2026-08-04: **50 of 59** claimed rules are named in <!-- fact:gr_linked --><!-- fact:gr_rules -->
`games/ground/src/lib.rs` . **Unmet** , target 59. The nine unlinked:
2026-07-31 09:27:14 +02:00
```text
GR-D07 GR-F02 GR-L03 GR-O03 GR-P01 GR-P02 GR-P03 GR-P04 GR-T01
```
The gate **reports** this rather than failing, deliberately. Closing the
gap by adding rule IDs to comments would satisfy the check without
establishing that each rule is implemented — which is the overclaim
InnerLoop implementation rule 2 exists to prevent, and which CB-WP-0001
committed once already. Each of the nine needs its implementation
confirmed before it is tagged. Promoting M-D1-LNK to a failing gate is
correct **after** that, not before.
2026-07-31 00:53:22 +02:00
### 1a. Token cost accounting (M-D2-CST)
T03: specs/CostAccounting.md — 15 contracts, 8 metrics, each with a command
Normative spec for the cost model (dedup by requestId, per-model and
per-TTL pricing), the attribution contract (ending-at-commit intervals
scoped by sessionId), and the reported shape (composition, not a total).
Every acceptance row names the command that produces its number, per
InnerLoop Step 4. AC-5..AC-8 are the positive control: cb-cost
--self-test must fail on a dedup violation, on zero responses, on a
missing subagent tree, and on 5m cache priced at the 1h rate.
The feasibility check earns its place here: AC-1's $92.87 is reachable
only if CA-06 holds. The earlier $92.21 target was reachable only by a
collector with the exact blind spot the survey documented.
MetricsAndScenarios 1a superseded. Two of its rules struck through
rather than deleted, because both were wrong in instructive ways: "if
the cache split is unknown, count all input at full price" would have
priced 80.5M cache reads at 10x, and "the hub already records tokens"
named as a source what is only ever a lossy sink. M-D2-TOK demoted --
tokens are not comparable across models or cache states.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:46:12 +02:00
> **Superseded 2026-07-31 by [CostAccounting.md](CostAccounting.md)**, which
> is normative for the cost model, attribution, and acceptance metrics.
> This section is retained for the price-sheet location and the
> quality-gate rule; where the two disagree, CostAccounting.md wins.
>
> What changed and why: the definition below named no instrument and was
> never computed, so CB-WP-0001 recorded M-D2-CST as *uncomputable* while
> the data sat in the session transcripts. Three of its rules were also
> wrong in ways that cost real money to discover — see the corrections
> inline below.
2026-07-31 00:53:22 +02:00
Token counts are only comparable at a single pricepoint. Since work moves
between models (Fable for demanding passes, Sonnet/Opus for routine ones),
every task's token record carries the **model** it ran on, and cost is
computed against a committed price sheet:
```text
benchmarks/baselines/model-prices.toml # the price sheet, updated when prices change
```
```toml
2026-07-31 09:27:31 +02:00
# The live sheet is benchmarks/baselines/model-prices.toml — read it,
# do not copy it. An inlined copy here went stale within hours: it still
# showed the sonnet intro rate as a `#` comment after CB-WP-0003 T11 made
# dated rates into data (promo_input / promo_output / promo_until), and
# added `recorded` + `max_age_days` for the staleness check (CA-16/CA-17).
2026-07-31 00:53:22 +02:00
```
Rules:
T03: specs/CostAccounting.md — 15 contracts, 8 metrics, each with a command
Normative spec for the cost model (dedup by requestId, per-model and
per-TTL pricing), the attribution contract (ending-at-commit intervals
scoped by sessionId), and the reported shape (composition, not a total).
Every acceptance row names the command that produces its number, per
InnerLoop Step 4. AC-5..AC-8 are the positive control: cb-cost
--self-test must fail on a dedup violation, on zero responses, on a
missing subagent tree, and on 5m cache priced at the 1h rate.
The feasibility check earns its place here: AC-1's $92.87 is reachable
only if CA-06 holds. The earlier $92.21 target was reachable only by a
collector with the exact blind spot the survey documented.
MetricsAndScenarios 1a superseded. Two of its rules struck through
rather than deleted, because both were wrong in instructive ways: "if
the cache split is unknown, count all input at full price" would have
priced 80.5M cache reads at 10x, and "the hub already records tokens"
named as a source what is only ever a lossy sink. M-D2-TOK demoted --
tokens are not comparable across models or cache states.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-31 08:46:12 +02:00
- ~~`cost = (in_tokens × × … If cache
split is unknown, count all input at full price and note it — cost is then
an upper bound.~~ **Corrected:** the cache split is never unknown; it is
in every transcript. Treating it as unknown would have priced 80.5M cache
reads at 10×
writes bill at two different TTL rates and must not be aggregated.
- ~~The state-hub task close (`update_task_status` ) already records tokens
and `model` .~~ **Corrected:** the hub schema has no cache fields and
cannot represent 88% of spend, and the figures it recorded for CB-WP-0001
were estimates in error by ~100%. The hub is a **sink** for numbers
computed by `make cost` , never a source. See CostAccounting.md §6.
2026-07-31 00:53:22 +02:00
- **Cheaper is only better at equal quality**: M-D2-CST verdicts are valid
only alongside passing scenarios/metrics from the same run — a cheap
failed pass scores nothing.
- Prices are `adopted:anthropic-pricing` with a recorded date; a stale sheet
(> 90 days or known price change) invalidates new `better` verdicts on
2026-07-31 09:27:31 +02:00
M-D2-CST until refreshed. **Now enforced** — CA-17 aborts the run rather
than leaving this as prose every verdict silently inherited.
2026-07-31 00:53:22 +02:00
2026-07-31 00:31:14 +02:00
Determinism is not a metric but an **invariant** : N replays of the same
seed and command log must produce bit-identical state hashes. Invariant
violations fail the run regardless of metric values.
---
## 2. Scenario format
Scenarios are the correctness currency: executable, declarative, diffable.
One file = one scenario. Location: `scenarios/<game-or-capability>/<slug>.yaml` .
```yaml
scenario: ground/darvo-interrupted-by-ground # id = path without extension
description: GROUND practice interrupts a DARVO sequence at the Attack step.
covers: [R-041, R-052, R-053] # numbered rules from the capability spec
seed: 42
setup:
players: 3
preset: standard-3p # named setup preset from the game spec
patch: # optional explicit state overrides
relationships:
- {from: P1, to: P2, kind: rivalry, strength: 2}
commands: # ordered; actor-tagged
- {actor: P1, cmd: trigger_darvo, target: P2}
- {actor: P2, cmd: play_ground, target: P1}
expect:
events: # ordered subsequence that must occur
- {type: DarvoInterrupted, step: attack}
state: # end-state assertions, dot-path = value
darvo_sequences: []
relationships[P1->P2].strength: 1
rejects: [] # commands above that must be rejected, by index
```
Rules:
- `covers` is what feeds M-D1-COV; a scenario without `covers` counts for
nothing.
- Assertions are **partial** : only listed paths are checked. Full-state
golden comparison is opt-in via `expect.state_hash` .
- Every scenario must be deterministic given `seed` ; the runner executes
each scenario twice and fails on hash divergence (cheap standing
determinism check).
- A failing run writes `replays/<scenario-id>.cbreplay` (see §4).
---
## 3. Benchmarks and baselines
Benchmarks live in `benchmarks/` as Criterion benches driving scenario
files (a benchmark is a scenario run at scale — no separate workload
format).
Baselines are **committed numbers** , recorded once per approved survey and
updated only by an explicit ADR:
```text
benchmarks/baselines/< capability > .toml
```
```toml
[M-D3-THR]
value = 120000
unit = "events/s"
source = "boardgame.io v0.50, measured locally, 3-player synthetic log"
recorded = 2026-07-31
machine = "bnt-lap001"
[M-D2-LOC]
value = 8400
unit = "lines"
source = "boardgame.io core, cloc, cited from CB-RES-0001"
```
- Comparisons are same-machine where `machine` is set; cross-machine
numbers are marked `provenance = cited` and treated as directional.
2026-07-31 00:48:04 +02:00
Per the runnable-baseline option in [InnerLoop.md ](InnerLoop.md ) §Step 1,
evidence rows compared only against cited numbers cap their verdict at
`parity` ; a `better` verdict requires a locally measured baseline from a
fidelity-noted harness.
2026-07-31 00:31:14 +02:00
- Regression rule (adopted:criterion): a merge-blocking regression is
>3% on any D3 metric against **our own** last evidence file, independent
of the SOTA baseline.
---
## 4. Replay bundle
`*.cbreplay` is a directory (or tar) with exactly:
```text
manifest.yaml # scenario id, seed, git commit, schema versions
commands.log # the full ordered command stream (serialized events optional)
initial.snapshot # starting state
expected.yaml # the assertions that failed, with expected vs actual
```
Contract: `cb replay <bundle>` (until the CLI exists: the scenario runner's
`--replay` flag) re-executes the bundle headless and must reproduce the
failure bit-identically. A bug report without a replay bundle is
information; with one, it is work an agent can start.
---
## 5. Evidence file
`evidence/CB-EV-NNNN-<slug>.md` — the committed close-out of a loop pass:
```markdown
# CB-EV-NNNN: <capability>
research: CB-RES-NNNN adr: ADR-NNNN spec: specs/< Capability > .md
commit: < sha of measured tree >
| Metric | Baseline | Ours | Verdict |
|---|---|---|---|
| M-D3-THR | 120000 events/s (boardgame.io) | 410000 events/s | better |
| ...every acceptance row, no `unmeasured` ... |
2026-07-31 00:53:22 +02:00
## Task cost log
| Task | Model | Tokens in/out | Cost (USD, per price sheet) | Iterations |
2026-07-31 00:31:14 +02:00
## Retrospective
< what the loop itself should change — one paragraph minimum >
```
Verdicts: `better / parity / worse` . A `worse` row does not necessarily
fail the pass — the ADR's declared trade governs — but an undeclared
`worse` does.
