From 5d1c1bc86600cc83a50e2d35e477e620ebbf8762 Mon Sep 17 00:00:00 2001 From: tegwick Date: Fri, 31 Jul 2026 00:31:14 +0200 Subject: [PATCH] Phase A: codify inner loop (specs/InnerLoop.md) and metrics/scenario conventions --- specs/InnerLoop.md | 149 +++++++++++++++++++++++++ specs/MetricsAndScenarios.md | 169 +++++++++++++++++++++++++++++ workplans/CB-WP-0001-inner-loop.md | 4 +- 3 files changed, 320 insertions(+), 2 deletions(-) create mode 100644 specs/InnerLoop.md create mode 100644 specs/MetricsAndScenarios.md diff --git a/specs/InnerLoop.md b/specs/InnerLoop.md new file mode 100644 index 0000000..db6f24a --- /dev/null +++ b/specs/InnerLoop.md @@ -0,0 +1,149 @@ +# The Inner Loop — Assimilate and Surpass + +Status: **v0.1 draft** — becomes v1.0 only after surviving its first full +pass (CB-WP-0001-T09 retrospective). + +Normative process for building every Clay-Borg capability. Referenced by +all workplans. The loop's own optimization target is **agentic efficiency**: +every artifact it produces must be small enough to load whole, structured +enough to act on without interpretation, and falsifiable enough that an +agent can judge its own work without a human in the iteration. + +--- + +## The five steps + +```text +1 RESEARCH → research/CB-RES-NNNN-.md (survey, baselines) +2 APPROVE → decision recorded in the ADR (gate: survey complete?) +3 DECIDE → decisions/ADR-NNNN-.md (assimilate/reimplement/hybrid) +4 SPECIFY → specs/.md (contracts + acceptance metrics) +5 CODE LOOP → code + scenarios + benchmarks (iterate until metrics beat baseline) +``` + +**Hard gate: no implementation code for a capability exists before its ADR +(step 3) is committed.** Steps 1–3 may be compressed into one session for +small capabilities, but their artifacts are never skipped. + +### Step 1 — Research + +Identify the best implementation in existence for this capability. Produce +`research/CB-RES-NNNN-.md` following the survey template (below). +The survey is done when it can name, per dimension, a concrete +**benchmark-to-beat**: a number, a property, or a reproducible comparison — +not an impression. + +### Step 2 — Approve + +An explicit recorded judgment inside the ADR: the survey is complete, the +candidates were the right ones, the baselines are trustworthy enough to +measure against. If not approvable, the loop returns to step 1 with the +named gap. Approval is cheap to record and expensive to skip — it is the +point where "we looked at X" becomes contestable. + +### Step 3 — Decide + +`decisions/ADR-NNNN-.md`: assimilate behind a port, reimplement, or +hybrid — with the **expected advantage stated per dimension** (see rubric). +An honest "worse here, better there, and why that trade is right" beats a +claimed sweep of all four dimensions. + +### Step 4 — Specify + +`specs/.md`: the contracts, invariants, and — mandatory — the +**acceptance metrics table**, each row tied to a baseline from step 1. +A spec without measurable acceptance criteria is not done. Metrics follow +the conventions in [MetricsAndScenarios.md](MetricsAndScenarios.md), +including the rule that metric selection itself passes through a mini +research step (metric provenance). + +### Step 5 — Code loop + +Implement iteratively. Each iteration: + +```text +change → cb-check (fmt, clippy, tests) → scenarios → benchmarks + → compare against acceptance table → evidence row appended +``` + +Done when every acceptance metric meets or beats its baseline and the +comparison numbers are committed as an evidence file +(`evidence/CB-EV-NNNN-.md`). A failed scenario must yield a replay +artifact an agent can re-execute locally. + +--- + +## The four-dimension rubric + +Every survey, ADR, and acceptance table is organized by these dimensions: + +| Dimension | Question | Example measurable proxies | +|---|---|---| +| **D1 Ease of specification** | How simply can behavior be stated, tested, understood? | rules-to-scenario coverage %, spec lines per rule, time-to-first-correct-scenario for a fresh agent session | +| **D2 Efficiency of implementation** | How cheap to build and keep building? | source LOC, dependency count/weight, clean-build and incremental-build time, tokens-per-completed-task | +| **D3 Speed of execution** | How fast does it run? | benchmark wall-time vs baseline, events/sec, memory footprint, determinism overhead | +| **D4 Optionality** | How cleanly does it integrate, extend, get replaced? | public API surface size, count of leaked foreign types (must be 0), effort-to-swap measured by null/reference impl existence, WIT-expressibility | + +Scoring is always **relative to the step-1 baseline**, never absolute: +`better / parity / worse / unmeasured` per proxy, with the number attached. +`unmeasured` is legal in a survey, illegal in an evidence file. + +--- + +## Survey template (research/CB-RES-NNNN-.md) + +```markdown +# CB-RES-NNNN: +capability: +status: draft | approved + +## Candidates +Per candidate: origin, license, maturity, adoption; data model; mutation +mechanism; determinism/replay story; relevant performance (measured if +runnable locally, cited with source otherwise). + +## Baselines (benchmark-to-beat) +| Dimension | Baseline holder | Metric | Value | Provenance | +(one row minimum per dimension; provenance = measured / cited / estimated) + +## Verdict +Which candidate leads per dimension; what none of them do well +(the surpass opportunity); risks in the baselines themselves. +``` + +--- + +## Agentic-efficiency requirements + +The loop exists to be driven by agents. Therefore: + +1. **Whole-file loadability** — every loop artifact stays under ~400 lines; + split before exceeding, link with relative paths. +2. **Structured over prose** — tables and fenced blocks for anything a + later step must parse (baselines, acceptance metrics, evidence rows). +3. **One command surface** — all checks runnable through repo-root + commands (eventually `cb *`; until then, `make`/`cargo` aliases declared + in one place), each supporting deterministic, greppable output. +4. **Self-contained tasks** — a workplan task names its input artifacts and + output artifacts; a fresh session must be able to execute it from the + task text plus linked files alone. +5. **Evidence or it didn't happen** — claims of "better" live in committed + evidence files with numbers, never only in commit messages or chat. +6. **Token discipline** — per the global budget policy, a loop iteration + that exceeds its budget without measurable progress is stopped and + decomposed, not pushed through. + +--- + +## Definition of done — one loop pass + +A capability has completed the loop when all of the following are committed: + +- [ ] research/CB-RES-NNNN with approved status and full baseline table +- [ ] decisions/ADR-NNNN with per-dimension expected advantage +- [ ] specs/.md with acceptance-metrics table +- [ ] passing scenarios covering every numbered spec rule +- [ ] evidence/CB-EV-NNNN with final comparison vs baseline, no `unmeasured` +- [ ] retrospective note (may be one paragraph appended to the evidence + file): what the loop itself should change +``` diff --git a/specs/MetricsAndScenarios.md b/specs/MetricsAndScenarios.md new file mode 100644 index 0000000..45e9707 --- /dev/null +++ b/specs/MetricsAndScenarios.md @@ -0,0 +1,169 @@ +# Metrics and Scenarios + +Status: **v0.1 draft** — instruments for the [InnerLoop](InnerLoop.md). +Everything the loop calls "better" is measured through the three artifact +kinds defined here: **scenarios** (correctness), **benchmarks** (speed and +cost), and **evidence files** (the committed comparison record). + +--- + +## 1. Metric selection is itself a loop pass + +Metrics are not invented ad hoc. Every metric used in an acceptance table +carries **provenance** — a one-line answer to "what state-of-the-art +practice does this metric derive from, and how does ours improve on it?" + +| Provenance tag | Meaning | +|---|---| +| `adopted:` | taken as-is from a named practice (e.g. `adopted:criterion` regression thresholds) | +| `adapted:` | derived from a named practice, with the delta stated | +| `novel` | no known precedent — requires a sentence justifying why nothing existing fits | + +A metric with no provenance line is invalid. This keeps the +assimilate-and-surpass discipline applied to the measuring instruments, +not only to the measured components. + +### Standing metric set (v0.1) + +Selected for the four dimensions; capability specs pick from these first +and add capability-specific rows only when these don't cover the claim. + +| ID | Dimension | Metric | Unit | Provenance | +|---|---|---|---|---| +| M-D1-COV | D1 | numbered spec rules covered by ≥1 passing scenario | % | adapted:requirements-traceability (per-rule, not per-feature) | +| M-D1-SPL | D1 | spec lines per numbered rule | lines | novel — proxies statement simplicity; gameable, so paired with M-D1-COV | +| M-D2-LOC | D2 | source LOC excluding tests (tokei) | lines | adopted:tokei | +| M-D2-DEP | D2 | transitive dependency count (cargo tree) | crates | adopted:cargo-deny practice | +| M-D2-BLD | D2 | clean build / incremental build time | s | adopted:cargo timing | +| M-D2-TOK | D2 | tokens consumed per completed workplan task | tokens | novel — agentic-efficiency core metric; recorded per task in evidence | +| M-D3-THR | D3 | events applied per second, headless replay | events/s | adapted:criterion (throughput mode) | +| M-D3-LAT | D3 | p99 command→state-applied latency | µs | adopted:criterion | +| M-D3-MEM | D3 | peak resident memory during benchmark scenario | MB | adopted:/usr/bin/time -v | +| M-D4-API | D4 | public API items (cargo doc item count) | items | adapted:cargo-public-api | +| M-D4-LEAK | D4 | foreign types in canonical interfaces | count | novel — must be 0; enforced by grep/deny rule, the Clay-Borg hard rule | +| M-D4-SWAP | D4 | capability has null + reference impls passing the same conformance suite | bool | adapted:hexagonal-architecture port testing | + +Determinism is not a metric but an **invariant**: N replays of the same +seed and command log must produce bit-identical state hashes. Invariant +violations fail the run regardless of metric values. + +--- + +## 2. Scenario format + +Scenarios are the correctness currency: executable, declarative, diffable. +One file = one scenario. Location: `scenarios//.yaml`. + +```yaml +scenario: ground/darvo-interrupted-by-ground # id = path without extension +description: GROUND practice interrupts a DARVO sequence at the Attack step. +covers: [R-041, R-052, R-053] # numbered rules from the capability spec +seed: 42 +setup: + players: 3 + preset: standard-3p # named setup preset from the game spec + patch: # optional explicit state overrides + relationships: + - {from: P1, to: P2, kind: rivalry, strength: 2} +commands: # ordered; actor-tagged + - {actor: P1, cmd: trigger_darvo, target: P2} + - {actor: P2, cmd: play_ground, target: P1} +expect: + events: # ordered subsequence that must occur + - {type: DarvoInterrupted, step: attack} + state: # end-state assertions, dot-path = value + darvo_sequences: [] + relationships[P1->P2].strength: 1 + rejects: [] # commands above that must be rejected, by index +``` + +Rules: + +- `covers` is what feeds M-D1-COV; a scenario without `covers` counts for + nothing. +- Assertions are **partial**: only listed paths are checked. Full-state + golden comparison is opt-in via `expect.state_hash`. +- Every scenario must be deterministic given `seed`; the runner executes + each scenario twice and fails on hash divergence (cheap standing + determinism check). +- A failing run writes `replays/.cbreplay` (see §4). + +--- + +## 3. Benchmarks and baselines + +Benchmarks live in `benchmarks/` as Criterion benches driving scenario +files (a benchmark is a scenario run at scale — no separate workload +format). + +Baselines are **committed numbers**, recorded once per approved survey and +updated only by an explicit ADR: + +```text +benchmarks/baselines/.toml +``` + +```toml +[M-D3-THR] +value = 120000 +unit = "events/s" +source = "boardgame.io v0.50, measured locally, 3-player synthetic log" +recorded = 2026-07-31 +machine = "bnt-lap001" + +[M-D2-LOC] +value = 8400 +unit = "lines" +source = "boardgame.io core, cloc, cited from CB-RES-0001" +``` + +- Comparisons are same-machine where `machine` is set; cross-machine + numbers are marked `provenance = cited` and treated as directional. +- Regression rule (adopted:criterion): a merge-blocking regression is + >3% on any D3 metric against **our own** last evidence file, independent + of the SOTA baseline. + +--- + +## 4. Replay bundle + +`*.cbreplay` is a directory (or tar) with exactly: + +```text +manifest.yaml # scenario id, seed, git commit, schema versions +commands.log # the full ordered command stream (serialized events optional) +initial.snapshot # starting state +expected.yaml # the assertions that failed, with expected vs actual +``` + +Contract: `cb replay ` (until the CLI exists: the scenario runner's +`--replay` flag) re-executes the bundle headless and must reproduce the +failure bit-identically. A bug report without a replay bundle is +information; with one, it is work an agent can start. + +--- + +## 5. Evidence file + +`evidence/CB-EV-NNNN-.md` — the committed close-out of a loop pass: + +```markdown +# CB-EV-NNNN: +research: CB-RES-NNNN adr: ADR-NNNN spec: specs/.md +commit: + +| Metric | Baseline | Ours | Verdict | +|---|---|---|---| +| M-D3-THR | 120000 events/s (boardgame.io) | 410000 events/s | better | +| ...every acceptance row, no `unmeasured`... | + +## Task token log +| Task | Tokens (approx) | Iterations | + +## Retrospective + +``` + +Verdicts: `better / parity / worse`. A `worse` row does not necessarily +fail the pass — the ADR's declared trade governs — but an undeclared +`worse` does. diff --git a/workplans/CB-WP-0001-inner-loop.md b/workplans/CB-WP-0001-inner-loop.md index 1b792dd..947f0a6 100644 --- a/workplans/CB-WP-0001-inner-loop.md +++ b/workplans/CB-WP-0001-inner-loop.md @@ -45,7 +45,7 @@ boardgame.io (turn/phase game-state engines), Tabletop Simulator scripting ```task id: CB-WP-0001-T01 -status: todo +status: done priority: high state_hub_task_id: "7a0ff270-395c-4774-b0d1-332c10acbd8e" ``` @@ -60,7 +60,7 @@ loop iteration. This spec is what every later workplan references. ```task id: CB-WP-0001-T02 -status: todo +status: done priority: high state_hub_task_id: "04e2c44c-6db3-4799-94b0-e22ef395d7fe" ```