tegwick
bb35fcb168
AGGREGATE becomes a list of source roots and rule patterns become
per-spec, so the link runs over every numbered spec x every crate rather
than GroundRules.md x games/ground/src/lib.rs.
The prediction held on the first run:
AM-1b kernel spec->code link: 15/18 (83%) across 10 source files
unlinked: K10 K14 K18
Kernel rules are link-only by design, and the output says so: they are
kernel invariants with no aggregate, setup preset or command vocabulary,
so scenarios/kernel/*.yaml with covers: [K11] would be a tag in a
directory the runner cannot dispatch. Claiming scenario coverage for them
is the inflation this gate exists to prevent.
Per ADR-0005 §5 the kernel arm reports without feeding the exit code
until 2026-08-31, then binds — the date in the tool, not in prose, with
days remaining printed every run, because open-ended "gate it later" is
how AM-4's targets went unratified for four workplans. The self-test
asserts the gate returns 0 before that date and 2 after.
The zero-rules positive control is replicated on the new denominator: a
kernel regex that stops matching aborts rather than printing 0/0 as
though it were 100%.
The self-test passed while the tool was completely broken. A print(
inside say() became say(), so every real `make coverage` died with
RecursionError while --self-test reported all-ok — it only ever called
kernel_arm(quiet=True) and never executed the reporting path. The control
named the behaviour and did not assert it, which is precisely what this
workplan is about. Fixed by exercising the loud path and asserting it
prints, then verified by re-breaking say() and confirming both new checks
go red. Seventh instance of the harness-does-nothing shape, in the tool
written to find that shape.
Also caught by its own gate: a self-test label that printed "0 K-ids"
beside a passing ">5" assertion, because the detail string rebuilt the
pattern with different escaping. A label that contradicts its own check
is worse than no label.
k_rules, k_linked and k_unlinked are registered facts under facts-check.
A limit of that checker is recorded rather than patched: it is
line-based, so a tagged value that prose-wraps fails.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
279 lines
12 KiB
Markdown
279 lines
12 KiB
Markdown
# Metrics and Scenarios
|
||
|
||
Status: **v0.1 draft** — instruments for the [InnerLoop](InnerLoop.md).
|
||
Everything the loop calls "better" is measured through the three artifact
|
||
kinds defined here: **scenarios** (correctness), **benchmarks** (speed and
|
||
cost), and **evidence files** (the committed comparison record).
|
||
|
||
---
|
||
|
||
## 1. Metric selection is itself a loop pass
|
||
|
||
Metrics are not invented ad hoc. Every metric used in an acceptance table
|
||
carries **provenance** — a one-line answer to "what state-of-the-art
|
||
practice does this metric derive from, and how does ours improve on it?"
|
||
|
||
| Provenance tag | Meaning |
|
||
|---|---|
|
||
| `adopted:<source>` | taken as-is from a named practice (e.g. `adopted:criterion` regression thresholds) |
|
||
| `adapted:<source>` | derived from a named practice, with the delta stated |
|
||
| `novel` | no known precedent — requires a sentence justifying why nothing existing fits |
|
||
|
||
A metric with no provenance line is invalid. This keeps the
|
||
assimilate-and-surpass discipline applied to the measuring instruments,
|
||
not only to the measured components.
|
||
|
||
### Standing metric set (v0.1)
|
||
|
||
Selected for the four dimensions; capability specs pick from these first
|
||
and add capability-specific rows only when these don't cover the claim.
|
||
|
||
| ID | Dimension | Metric | Unit | Provenance |
|
||
|---|---|---|---|---|
|
||
| M-D1-COV | D1 | numbered spec rules covered by ≥1 passing scenario | % | adapted:requirements-traceability (per-rule, not per-feature) |
|
||
| M-D1-LNK | D1 | claimed rules also **named in the aggregate source** | count | novel — closes the spec→code→scenario chain; instrument: `make coverage` (AM-1b) |
|
||
| M-D1-SPL | D1 | spec lines per numbered rule | lines | novel — proxies statement simplicity; gameable, so paired with M-D1-COV |
|
||
| M-D2-LOC | D2 | source LOC excluding tests (tokei) | lines | adopted:tokei |
|
||
| M-D2-DEP | D2 | transitive dependency count (cargo tree) | crates | adopted:cargo-deny practice |
|
||
| M-D2-BLD | D2 | clean build / incremental build time | s | adopted:cargo timing |
|
||
| M-D2-TOK | D2 | tokens consumed per completed workplan task | tokens | novel — **demoted 2026-07-31**: an input to the cost model, not comparable across models or cache states |
|
||
| M-D2-CST | D2 | **cost** per completed workplan task, attributed per CA-08 | USD | adapted:anthropic-pricing — instrument: `make cost`; normative spec [CostAccounting.md](CostAccounting.md) |
|
||
| M-D3-THR | D3 | events applied per second, headless replay | events/s | adapted:criterion (throughput mode) |
|
||
| M-D3-LAT | D3 | p99 command→state-applied latency | µs | adopted:criterion |
|
||
| M-D3-MEM | D3 | peak resident memory during benchmark scenario | MB | adopted:/usr/bin/time -v |
|
||
| M-D4-API | D4 | public API items (cargo doc item count) | items | adapted:cargo-public-api |
|
||
| M-D4-LEAK | D4 | foreign types in canonical interfaces | count | novel — must be 0; enforced by grep/deny rule, the Clay-Borg hard rule |
|
||
| M-D4-SWAP | D4 | capability has null + reference impls passing the same conformance suite | bool | adapted:hexagonal-architecture port testing |
|
||
|
||
### 1b. The coverage gate's numbers (M-D1-COV, M-D1-LNK)
|
||
|
||
> **Widened 2026-07-31 (CB-WP-0005 T01).** Until then the instrument
|
||
> matched `GR-` only, against `GroundRules.md` only, and linked against
|
||
> one source file, so all **18** K-rules were outside it. <!-- fact:k_rules -->
|
||
> `58/58 (100%)` read as "all rules".
|
||
>
|
||
> `make coverage` now reports a second, separate denominator:
|
||
> **15 of 18** K-rules are named across the source. <!-- fact:k_linked -->
|
||
> Unlinked: **K10 K14 K18**. <!-- fact:k_unlinked -->
|
||
>
|
||
> **Kernel rules are link-only, by design.** They are kernel invariants,
|
||
> not game rules: there is no kernel aggregate, setup preset or command
|
||
> vocabulary, so a `scenarios/kernel/*.yaml` carrying `covers: [K11]`
|
||
> would be a tag in a directory the runner cannot dispatch. Claiming
|
||
> scenario coverage for them would be the inflation this gate exists to
|
||
> prevent.
|
||
>
|
||
> Per ADR-0005 §5 the kernel arm **reports without feeding the exit
|
||
> code until 2026-08-31**, then binds. A newly widened denominator is not
|
||
> a regression. The date lives in `tools/rule-coverage.py`, not here, and
|
||
> the tool prints the days remaining on every run.
|
||
|
||
**Both numbers count names.** Neither proves anything fails when a rule is
|
||
violated. That is **M-D1-MUT** (`make mutation-check`, CB-WP-0005 T02) —
|
||
four of the seven defects found by CB-RES-0004 were named in the source
|
||
and inert, and therefore invisible to everything on this page.
|
||
|
||
|
||
M-D1-COV counts tags. It proves no rule is unclaimed and no claimed rule
|
||
is invented; it does **not** prove a scenario exercises what it names, and
|
||
that limit is printed with the number every time (InnerLoop implementation
|
||
rule 4).
|
||
|
||
**M-D1-LNK** (CB-WP-0003 T09) closes one link of that chain mechanically:
|
||
a rule a scenario claims should also appear in the aggregate source, or
|
||
the claim rests on a tag and nothing else.
|
||
|
||
Measured 2026-07-31: **49 of 58** claimed rules are named in <!-- fact:gr_linked --><!-- fact:gr_rules -->
|
||
`games/ground/src/lib.rs`. **Unmet**, target 58. The nine unlinked:
|
||
|
||
```text
|
||
GR-D07 GR-F02 GR-L03 GR-O03 GR-P01 GR-P02 GR-P03 GR-P04 GR-T01
|
||
```
|
||
|
||
The gate **reports** this rather than failing, deliberately. Closing the
|
||
gap by adding rule IDs to comments would satisfy the check without
|
||
establishing that each rule is implemented — which is the overclaim
|
||
InnerLoop implementation rule 2 exists to prevent, and which CB-WP-0001
|
||
committed once already. Each of the nine needs its implementation
|
||
confirmed before it is tagged. Promoting M-D1-LNK to a failing gate is
|
||
correct **after** that, not before.
|
||
|
||
### 1a. Token cost accounting (M-D2-CST)
|
||
|
||
> **Superseded 2026-07-31 by [CostAccounting.md](CostAccounting.md)**, which
|
||
> is normative for the cost model, attribution, and acceptance metrics.
|
||
> This section is retained for the price-sheet location and the
|
||
> quality-gate rule; where the two disagree, CostAccounting.md wins.
|
||
>
|
||
> What changed and why: the definition below named no instrument and was
|
||
> never computed, so CB-WP-0001 recorded M-D2-CST as *uncomputable* while
|
||
> the data sat in the session transcripts. Three of its rules were also
|
||
> wrong in ways that cost real money to discover — see the corrections
|
||
> inline below.
|
||
|
||
Token counts are only comparable at a single pricepoint. Since work moves
|
||
between models (Fable for demanding passes, Sonnet/Opus for routine ones),
|
||
every task's token record carries the **model** it ran on, and cost is
|
||
computed against a committed price sheet:
|
||
|
||
```text
|
||
benchmarks/baselines/model-prices.toml # the price sheet, updated when prices change
|
||
```
|
||
|
||
```toml
|
||
# The live sheet is benchmarks/baselines/model-prices.toml — read it,
|
||
# do not copy it. An inlined copy here went stale within hours: it still
|
||
# showed the sonnet intro rate as a `#` comment after CB-WP-0003 T11 made
|
||
# dated rates into data (promo_input / promo_output / promo_until), and
|
||
# added `recorded` + `max_age_days` for the staleness check (CA-16/CA-17).
|
||
```
|
||
|
||
Rules:
|
||
|
||
- ~~`cost = (in_tokens × input + out_tokens × output) / 1e6` … If cache
|
||
split is unknown, count all input at full price and note it — cost is then
|
||
an upper bound.~~ **Corrected:** the cache split is never unknown; it is
|
||
in every transcript. Treating it as unknown would have priced 80.5M cache
|
||
reads at 10× their rate. See CostAccounting.md §1.2 (CA-03, CA-04) — cache
|
||
writes bill at two different TTL rates and must not be aggregated.
|
||
- ~~The state-hub task close (`update_task_status`) already records tokens
|
||
and `model`.~~ **Corrected:** the hub schema has no cache fields and
|
||
cannot represent 88% of spend, and the figures it recorded for CB-WP-0001
|
||
were estimates in error by ~100%. The hub is a **sink** for numbers
|
||
computed by `make cost`, never a source. See CostAccounting.md §6.
|
||
- **Cheaper is only better at equal quality**: M-D2-CST verdicts are valid
|
||
only alongside passing scenarios/metrics from the same run — a cheap
|
||
failed pass scores nothing.
|
||
- Prices are `adopted:anthropic-pricing` with a recorded date; a stale sheet
|
||
(> 90 days or known price change) invalidates new `better` verdicts on
|
||
M-D2-CST until refreshed. **Now enforced** — CA-17 aborts the run rather
|
||
than leaving this as prose every verdict silently inherited.
|
||
|
||
Determinism is not a metric but an **invariant**: N replays of the same
|
||
seed and command log must produce bit-identical state hashes. Invariant
|
||
violations fail the run regardless of metric values.
|
||
|
||
---
|
||
|
||
## 2. Scenario format
|
||
|
||
Scenarios are the correctness currency: executable, declarative, diffable.
|
||
One file = one scenario. Location: `scenarios/<game-or-capability>/<slug>.yaml`.
|
||
|
||
```yaml
|
||
scenario: ground/darvo-interrupted-by-ground # id = path without extension
|
||
description: GROUND practice interrupts a DARVO sequence at the Attack step.
|
||
covers: [R-041, R-052, R-053] # numbered rules from the capability spec
|
||
seed: 42
|
||
setup:
|
||
players: 3
|
||
preset: standard-3p # named setup preset from the game spec
|
||
patch: # optional explicit state overrides
|
||
relationships:
|
||
- {from: P1, to: P2, kind: rivalry, strength: 2}
|
||
commands: # ordered; actor-tagged
|
||
- {actor: P1, cmd: trigger_darvo, target: P2}
|
||
- {actor: P2, cmd: play_ground, target: P1}
|
||
expect:
|
||
events: # ordered subsequence that must occur
|
||
- {type: DarvoInterrupted, step: attack}
|
||
state: # end-state assertions, dot-path = value
|
||
darvo_sequences: []
|
||
relationships[P1->P2].strength: 1
|
||
rejects: [] # commands above that must be rejected, by index
|
||
```
|
||
|
||
Rules:
|
||
|
||
- `covers` is what feeds M-D1-COV; a scenario without `covers` counts for
|
||
nothing.
|
||
- Assertions are **partial**: only listed paths are checked. Full-state
|
||
golden comparison is opt-in via `expect.state_hash`.
|
||
- Every scenario must be deterministic given `seed`; the runner executes
|
||
each scenario twice and fails on hash divergence (cheap standing
|
||
determinism check).
|
||
- A failing run writes `replays/<scenario-id>.cbreplay` (see §4).
|
||
|
||
---
|
||
|
||
## 3. Benchmarks and baselines
|
||
|
||
Benchmarks live in `benchmarks/` as Criterion benches driving scenario
|
||
files (a benchmark is a scenario run at scale — no separate workload
|
||
format).
|
||
|
||
Baselines are **committed numbers**, recorded once per approved survey and
|
||
updated only by an explicit ADR:
|
||
|
||
```text
|
||
benchmarks/baselines/<capability>.toml
|
||
```
|
||
|
||
```toml
|
||
[M-D3-THR]
|
||
value = 120000
|
||
unit = "events/s"
|
||
source = "boardgame.io v0.50, measured locally, 3-player synthetic log"
|
||
recorded = 2026-07-31
|
||
machine = "bnt-lap001"
|
||
|
||
[M-D2-LOC]
|
||
value = 8400
|
||
unit = "lines"
|
||
source = "boardgame.io core, cloc, cited from CB-RES-0001"
|
||
```
|
||
|
||
- Comparisons are same-machine where `machine` is set; cross-machine
|
||
numbers are marked `provenance = cited` and treated as directional.
|
||
Per the runnable-baseline option in [InnerLoop.md](InnerLoop.md) §Step 1,
|
||
evidence rows compared only against cited numbers cap their verdict at
|
||
`parity`; a `better` verdict requires a locally measured baseline from a
|
||
fidelity-noted harness.
|
||
- Regression rule (adopted:criterion): a merge-blocking regression is
|
||
>3% on any D3 metric against **our own** last evidence file, independent
|
||
of the SOTA baseline.
|
||
|
||
---
|
||
|
||
## 4. Replay bundle
|
||
|
||
`*.cbreplay` is a directory (or tar) with exactly:
|
||
|
||
```text
|
||
manifest.yaml # scenario id, seed, git commit, schema versions
|
||
commands.log # the full ordered command stream (serialized events optional)
|
||
initial.snapshot # starting state
|
||
expected.yaml # the assertions that failed, with expected vs actual
|
||
```
|
||
|
||
Contract: `cb replay <bundle>` (until the CLI exists: the scenario runner's
|
||
`--replay` flag) re-executes the bundle headless and must reproduce the
|
||
failure bit-identically. A bug report without a replay bundle is
|
||
information; with one, it is work an agent can start.
|
||
|
||
---
|
||
|
||
## 5. Evidence file
|
||
|
||
`evidence/CB-EV-NNNN-<slug>.md` — the committed close-out of a loop pass:
|
||
|
||
```markdown
|
||
# CB-EV-NNNN: <capability>
|
||
research: CB-RES-NNNN adr: ADR-NNNN spec: specs/<Capability>.md
|
||
commit: <sha of measured tree>
|
||
|
||
| Metric | Baseline | Ours | Verdict |
|
||
|---|---|---|---|
|
||
| M-D3-THR | 120000 events/s (boardgame.io) | 410000 events/s | better |
|
||
| ...every acceptance row, no `unmeasured`... |
|
||
|
||
## Task cost log
|
||
| Task | Model | Tokens in/out | Cost (USD, per price sheet) | Iterations |
|
||
|
||
## Retrospective
|
||
<what the loop itself should change — one paragraph minimum>
|
||
```
|
||
|
||
Verdicts: `better / parity / worse`. A `worse` row does not necessarily
|
||
fail the pass — the ADR's declared trade governs — but an undeclared
|
||
`worse` does.
|