Phase A: codify inner loop (specs/InnerLoop.md) and metrics/scenario conventions
This commit is contained in:
parent
52b006580b
commit
5d1c1bc866
3 changed files with 320 additions and 2 deletions
149
specs/InnerLoop.md
Normal file
149
specs/InnerLoop.md
Normal file
|
|
@ -0,0 +1,149 @@
|
|||
# The Inner Loop — Assimilate and Surpass
|
||||
|
||||
Status: **v0.1 draft** — becomes v1.0 only after surviving its first full
|
||||
pass (CB-WP-0001-T09 retrospective).
|
||||
|
||||
Normative process for building every Clay-Borg capability. Referenced by
|
||||
all workplans. The loop's own optimization target is **agentic efficiency**:
|
||||
every artifact it produces must be small enough to load whole, structured
|
||||
enough to act on without interpretation, and falsifiable enough that an
|
||||
agent can judge its own work without a human in the iteration.
|
||||
|
||||
---
|
||||
|
||||
## The five steps
|
||||
|
||||
```text
|
||||
1 RESEARCH → research/CB-RES-NNNN-<slug>.md (survey, baselines)
|
||||
2 APPROVE → decision recorded in the ADR (gate: survey complete?)
|
||||
3 DECIDE → decisions/ADR-NNNN-<slug>.md (assimilate/reimplement/hybrid)
|
||||
4 SPECIFY → specs/<Capability>.md (contracts + acceptance metrics)
|
||||
5 CODE LOOP → code + scenarios + benchmarks (iterate until metrics beat baseline)
|
||||
```
|
||||
|
||||
**Hard gate: no implementation code for a capability exists before its ADR
|
||||
(step 3) is committed.** Steps 1–3 may be compressed into one session for
|
||||
small capabilities, but their artifacts are never skipped.
|
||||
|
||||
### Step 1 — Research
|
||||
|
||||
Identify the best implementation in existence for this capability. Produce
|
||||
`research/CB-RES-NNNN-<slug>.md` following the survey template (below).
|
||||
The survey is done when it can name, per dimension, a concrete
|
||||
**benchmark-to-beat**: a number, a property, or a reproducible comparison —
|
||||
not an impression.
|
||||
|
||||
### Step 2 — Approve
|
||||
|
||||
An explicit recorded judgment inside the ADR: the survey is complete, the
|
||||
candidates were the right ones, the baselines are trustworthy enough to
|
||||
measure against. If not approvable, the loop returns to step 1 with the
|
||||
named gap. Approval is cheap to record and expensive to skip — it is the
|
||||
point where "we looked at X" becomes contestable.
|
||||
|
||||
### Step 3 — Decide
|
||||
|
||||
`decisions/ADR-NNNN-<slug>.md`: assimilate behind a port, reimplement, or
|
||||
hybrid — with the **expected advantage stated per dimension** (see rubric).
|
||||
An honest "worse here, better there, and why that trade is right" beats a
|
||||
claimed sweep of all four dimensions.
|
||||
|
||||
### Step 4 — Specify
|
||||
|
||||
`specs/<Capability>.md`: the contracts, invariants, and — mandatory — the
|
||||
**acceptance metrics table**, each row tied to a baseline from step 1.
|
||||
A spec without measurable acceptance criteria is not done. Metrics follow
|
||||
the conventions in [MetricsAndScenarios.md](MetricsAndScenarios.md),
|
||||
including the rule that metric selection itself passes through a mini
|
||||
research step (metric provenance).
|
||||
|
||||
### Step 5 — Code loop
|
||||
|
||||
Implement iteratively. Each iteration:
|
||||
|
||||
```text
|
||||
change → cb-check (fmt, clippy, tests) → scenarios → benchmarks
|
||||
→ compare against acceptance table → evidence row appended
|
||||
```
|
||||
|
||||
Done when every acceptance metric meets or beats its baseline and the
|
||||
comparison numbers are committed as an evidence file
|
||||
(`evidence/CB-EV-NNNN-<slug>.md`). A failed scenario must yield a replay
|
||||
artifact an agent can re-execute locally.
|
||||
|
||||
---
|
||||
|
||||
## The four-dimension rubric
|
||||
|
||||
Every survey, ADR, and acceptance table is organized by these dimensions:
|
||||
|
||||
| Dimension | Question | Example measurable proxies |
|
||||
|---|---|---|
|
||||
| **D1 Ease of specification** | How simply can behavior be stated, tested, understood? | rules-to-scenario coverage %, spec lines per rule, time-to-first-correct-scenario for a fresh agent session |
|
||||
| **D2 Efficiency of implementation** | How cheap to build and keep building? | source LOC, dependency count/weight, clean-build and incremental-build time, tokens-per-completed-task |
|
||||
| **D3 Speed of execution** | How fast does it run? | benchmark wall-time vs baseline, events/sec, memory footprint, determinism overhead |
|
||||
| **D4 Optionality** | How cleanly does it integrate, extend, get replaced? | public API surface size, count of leaked foreign types (must be 0), effort-to-swap measured by null/reference impl existence, WIT-expressibility |
|
||||
|
||||
Scoring is always **relative to the step-1 baseline**, never absolute:
|
||||
`better / parity / worse / unmeasured` per proxy, with the number attached.
|
||||
`unmeasured` is legal in a survey, illegal in an evidence file.
|
||||
|
||||
---
|
||||
|
||||
## Survey template (research/CB-RES-NNNN-<slug>.md)
|
||||
|
||||
```markdown
|
||||
# CB-RES-NNNN: <capability>
|
||||
capability: <canonical.capability.id>
|
||||
status: draft | approved
|
||||
|
||||
## Candidates
|
||||
Per candidate: origin, license, maturity, adoption; data model; mutation
|
||||
mechanism; determinism/replay story; relevant performance (measured if
|
||||
runnable locally, cited with source otherwise).
|
||||
|
||||
## Baselines (benchmark-to-beat)
|
||||
| Dimension | Baseline holder | Metric | Value | Provenance |
|
||||
(one row minimum per dimension; provenance = measured / cited / estimated)
|
||||
|
||||
## Verdict
|
||||
Which candidate leads per dimension; what none of them do well
|
||||
(the surpass opportunity); risks in the baselines themselves.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Agentic-efficiency requirements
|
||||
|
||||
The loop exists to be driven by agents. Therefore:
|
||||
|
||||
1. **Whole-file loadability** — every loop artifact stays under ~400 lines;
|
||||
split before exceeding, link with relative paths.
|
||||
2. **Structured over prose** — tables and fenced blocks for anything a
|
||||
later step must parse (baselines, acceptance metrics, evidence rows).
|
||||
3. **One command surface** — all checks runnable through repo-root
|
||||
commands (eventually `cb *`; until then, `make`/`cargo` aliases declared
|
||||
in one place), each supporting deterministic, greppable output.
|
||||
4. **Self-contained tasks** — a workplan task names its input artifacts and
|
||||
output artifacts; a fresh session must be able to execute it from the
|
||||
task text plus linked files alone.
|
||||
5. **Evidence or it didn't happen** — claims of "better" live in committed
|
||||
evidence files with numbers, never only in commit messages or chat.
|
||||
6. **Token discipline** — per the global budget policy, a loop iteration
|
||||
that exceeds its budget without measurable progress is stopped and
|
||||
decomposed, not pushed through.
|
||||
|
||||
---
|
||||
|
||||
## Definition of done — one loop pass
|
||||
|
||||
A capability has completed the loop when all of the following are committed:
|
||||
|
||||
- [ ] research/CB-RES-NNNN with approved status and full baseline table
|
||||
- [ ] decisions/ADR-NNNN with per-dimension expected advantage
|
||||
- [ ] specs/<Capability>.md with acceptance-metrics table
|
||||
- [ ] passing scenarios covering every numbered spec rule
|
||||
- [ ] evidence/CB-EV-NNNN with final comparison vs baseline, no `unmeasured`
|
||||
- [ ] retrospective note (may be one paragraph appended to the evidence
|
||||
file): what the loop itself should change
|
||||
```
|
||||
169
specs/MetricsAndScenarios.md
Normal file
169
specs/MetricsAndScenarios.md
Normal file
|
|
@ -0,0 +1,169 @@
|
|||
# Metrics and Scenarios
|
||||
|
||||
Status: **v0.1 draft** — instruments for the [InnerLoop](InnerLoop.md).
|
||||
Everything the loop calls "better" is measured through the three artifact
|
||||
kinds defined here: **scenarios** (correctness), **benchmarks** (speed and
|
||||
cost), and **evidence files** (the committed comparison record).
|
||||
|
||||
---
|
||||
|
||||
## 1. Metric selection is itself a loop pass
|
||||
|
||||
Metrics are not invented ad hoc. Every metric used in an acceptance table
|
||||
carries **provenance** — a one-line answer to "what state-of-the-art
|
||||
practice does this metric derive from, and how does ours improve on it?"
|
||||
|
||||
| Provenance tag | Meaning |
|
||||
|---|---|
|
||||
| `adopted:<source>` | taken as-is from a named practice (e.g. `adopted:criterion` regression thresholds) |
|
||||
| `adapted:<source>` | derived from a named practice, with the delta stated |
|
||||
| `novel` | no known precedent — requires a sentence justifying why nothing existing fits |
|
||||
|
||||
A metric with no provenance line is invalid. This keeps the
|
||||
assimilate-and-surpass discipline applied to the measuring instruments,
|
||||
not only to the measured components.
|
||||
|
||||
### Standing metric set (v0.1)
|
||||
|
||||
Selected for the four dimensions; capability specs pick from these first
|
||||
and add capability-specific rows only when these don't cover the claim.
|
||||
|
||||
| ID | Dimension | Metric | Unit | Provenance |
|
||||
|---|---|---|---|---|
|
||||
| M-D1-COV | D1 | numbered spec rules covered by ≥1 passing scenario | % | adapted:requirements-traceability (per-rule, not per-feature) |
|
||||
| M-D1-SPL | D1 | spec lines per numbered rule | lines | novel — proxies statement simplicity; gameable, so paired with M-D1-COV |
|
||||
| M-D2-LOC | D2 | source LOC excluding tests (tokei) | lines | adopted:tokei |
|
||||
| M-D2-DEP | D2 | transitive dependency count (cargo tree) | crates | adopted:cargo-deny practice |
|
||||
| M-D2-BLD | D2 | clean build / incremental build time | s | adopted:cargo timing |
|
||||
| M-D2-TOK | D2 | tokens consumed per completed workplan task | tokens | novel — agentic-efficiency core metric; recorded per task in evidence |
|
||||
| M-D3-THR | D3 | events applied per second, headless replay | events/s | adapted:criterion (throughput mode) |
|
||||
| M-D3-LAT | D3 | p99 command→state-applied latency | µs | adopted:criterion |
|
||||
| M-D3-MEM | D3 | peak resident memory during benchmark scenario | MB | adopted:/usr/bin/time -v |
|
||||
| M-D4-API | D4 | public API items (cargo doc item count) | items | adapted:cargo-public-api |
|
||||
| M-D4-LEAK | D4 | foreign types in canonical interfaces | count | novel — must be 0; enforced by grep/deny rule, the Clay-Borg hard rule |
|
||||
| M-D4-SWAP | D4 | capability has null + reference impls passing the same conformance suite | bool | adapted:hexagonal-architecture port testing |
|
||||
|
||||
Determinism is not a metric but an **invariant**: N replays of the same
|
||||
seed and command log must produce bit-identical state hashes. Invariant
|
||||
violations fail the run regardless of metric values.
|
||||
|
||||
---
|
||||
|
||||
## 2. Scenario format
|
||||
|
||||
Scenarios are the correctness currency: executable, declarative, diffable.
|
||||
One file = one scenario. Location: `scenarios/<game-or-capability>/<slug>.yaml`.
|
||||
|
||||
```yaml
|
||||
scenario: ground/darvo-interrupted-by-ground # id = path without extension
|
||||
description: GROUND practice interrupts a DARVO sequence at the Attack step.
|
||||
covers: [R-041, R-052, R-053] # numbered rules from the capability spec
|
||||
seed: 42
|
||||
setup:
|
||||
players: 3
|
||||
preset: standard-3p # named setup preset from the game spec
|
||||
patch: # optional explicit state overrides
|
||||
relationships:
|
||||
- {from: P1, to: P2, kind: rivalry, strength: 2}
|
||||
commands: # ordered; actor-tagged
|
||||
- {actor: P1, cmd: trigger_darvo, target: P2}
|
||||
- {actor: P2, cmd: play_ground, target: P1}
|
||||
expect:
|
||||
events: # ordered subsequence that must occur
|
||||
- {type: DarvoInterrupted, step: attack}
|
||||
state: # end-state assertions, dot-path = value
|
||||
darvo_sequences: []
|
||||
relationships[P1->P2].strength: 1
|
||||
rejects: [] # commands above that must be rejected, by index
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- `covers` is what feeds M-D1-COV; a scenario without `covers` counts for
|
||||
nothing.
|
||||
- Assertions are **partial**: only listed paths are checked. Full-state
|
||||
golden comparison is opt-in via `expect.state_hash`.
|
||||
- Every scenario must be deterministic given `seed`; the runner executes
|
||||
each scenario twice and fails on hash divergence (cheap standing
|
||||
determinism check).
|
||||
- A failing run writes `replays/<scenario-id>.cbreplay` (see §4).
|
||||
|
||||
---
|
||||
|
||||
## 3. Benchmarks and baselines
|
||||
|
||||
Benchmarks live in `benchmarks/` as Criterion benches driving scenario
|
||||
files (a benchmark is a scenario run at scale — no separate workload
|
||||
format).
|
||||
|
||||
Baselines are **committed numbers**, recorded once per approved survey and
|
||||
updated only by an explicit ADR:
|
||||
|
||||
```text
|
||||
benchmarks/baselines/<capability>.toml
|
||||
```
|
||||
|
||||
```toml
|
||||
[M-D3-THR]
|
||||
value = 120000
|
||||
unit = "events/s"
|
||||
source = "boardgame.io v0.50, measured locally, 3-player synthetic log"
|
||||
recorded = 2026-07-31
|
||||
machine = "bnt-lap001"
|
||||
|
||||
[M-D2-LOC]
|
||||
value = 8400
|
||||
unit = "lines"
|
||||
source = "boardgame.io core, cloc, cited from CB-RES-0001"
|
||||
```
|
||||
|
||||
- Comparisons are same-machine where `machine` is set; cross-machine
|
||||
numbers are marked `provenance = cited` and treated as directional.
|
||||
- Regression rule (adopted:criterion): a merge-blocking regression is
|
||||
>3% on any D3 metric against **our own** last evidence file, independent
|
||||
of the SOTA baseline.
|
||||
|
||||
---
|
||||
|
||||
## 4. Replay bundle
|
||||
|
||||
`*.cbreplay` is a directory (or tar) with exactly:
|
||||
|
||||
```text
|
||||
manifest.yaml # scenario id, seed, git commit, schema versions
|
||||
commands.log # the full ordered command stream (serialized events optional)
|
||||
initial.snapshot # starting state
|
||||
expected.yaml # the assertions that failed, with expected vs actual
|
||||
```
|
||||
|
||||
Contract: `cb replay <bundle>` (until the CLI exists: the scenario runner's
|
||||
`--replay` flag) re-executes the bundle headless and must reproduce the
|
||||
failure bit-identically. A bug report without a replay bundle is
|
||||
information; with one, it is work an agent can start.
|
||||
|
||||
---
|
||||
|
||||
## 5. Evidence file
|
||||
|
||||
`evidence/CB-EV-NNNN-<slug>.md` — the committed close-out of a loop pass:
|
||||
|
||||
```markdown
|
||||
# CB-EV-NNNN: <capability>
|
||||
research: CB-RES-NNNN adr: ADR-NNNN spec: specs/<Capability>.md
|
||||
commit: <sha of measured tree>
|
||||
|
||||
| Metric | Baseline | Ours | Verdict |
|
||||
|---|---|---|---|
|
||||
| M-D3-THR | 120000 events/s (boardgame.io) | 410000 events/s | better |
|
||||
| ...every acceptance row, no `unmeasured`... |
|
||||
|
||||
## Task token log
|
||||
| Task | Tokens (approx) | Iterations |
|
||||
|
||||
## Retrospective
|
||||
<what the loop itself should change — one paragraph minimum>
|
||||
```
|
||||
|
||||
Verdicts: `better / parity / worse`. A `worse` row does not necessarily
|
||||
fail the pass — the ADR's declared trade governs — but an undeclared
|
||||
`worse` does.
|
||||
|
|
@ -45,7 +45,7 @@ boardgame.io (turn/phase game-state engines), Tabletop Simulator scripting
|
|||
|
||||
```task
|
||||
id: CB-WP-0001-T01
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "7a0ff270-395c-4774-b0d1-332c10acbd8e"
|
||||
```
|
||||
|
|
@ -60,7 +60,7 @@ loop iteration. This spec is what every later workplan references.
|
|||
|
||||
```task
|
||||
id: CB-WP-0001-T02
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "04e2c44c-6db3-4799-94b0-e22ef395d7fe"
|
||||
```
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue