clay-borg/history/260731-inner-loop-retrospective.md

Inner-loop retrospective — after CB-WP-0001 (first full pass)

Date: 2026-07-31 Subject: specs/InnerLoop.md v0.2 → v1.0 Evidence: evidence/CB-EV-0001-game-kernel.md

The loop has now survived one full pass, from SOTA survey through an adversarial review, an ADR, two specs, and six code iterations that ended in a committed acceptance run. This is what it got right, what it missed, and what changes in v1.0.

What earned its cost

The adversarial review (Step 2). On its single use it produced one blocking finding and three significant ones. The blocking one mattered: the survey claimed boardgame.io's degradation was "architectural, not a tuning artifact", and the challenge pointed at the disableUndo flag that the claim had not accounted for. The resolution was a rerun with undo disabled, not a rhetorical defence — degradation persisted (0.66× at 40k, still DNF at 100k), so the overclaim was retracted and two mechanisms were attributed separately. A survey that had gone straight to an ADR would have carried a false claim into the decision.

The parity-cap rule. It stopped the boardgame.io comparison from being written up as a verdict. The final evidence file says the ~450× command-rate ratio is a direction, not a verdict, and lists the four reasons it is not like-for-like. That paragraph exists because the rule required it.

Deferring rules that need a decision. Four rules (GR-L02, GR-A05, GR-A11, GR-A12) were left unimplemented across two iterations rather than given invented defaults, then implemented as explicit commands. No scenario claimed coverage of them in the interim. This is the single most useful habit the pass produced and v1.0 promotes it to a rule.

The provisional mechanism. Ten U-items with provisional: true scenarios meant an underdetermined rule cost a scenario tag, not a kernel decision. One further ambiguity (GR-E02's "successes") was found during implementation and captured the same way.

What the loop missed

Both serious errors in this pass were measurement errors, and the loop caught neither. Review caught claims; nothing caught numbers.

1. The boardgame.io harness v1 reported "8.4s for 100k moves". It was measuring rejected no-ops — every move was refused. Caught by noticing the error text in output, not by any gate.
2. The Rust benchmark reported 9.3M events/s on a flat curve and would have been published as a 93× beat of AM-6. The workload had a player selecting SUPPORT while parked at Stress 4, so GR-R03 rejected it, rounds never completed, and throughput was computed for rounds that never happened. Caught only because a probe test asserted a round produces events at all. The corrected number is 5.6× lower.

Both share a shape: the harness ran successfully while doing no work, and success was inferred from "it completed" rather than from "it did the thing". This is the dominant failure mode of an agent-driven loop, because an agent will happily report a fast number.

Scaffold code with unexercised paths. T07 produced a compiling, "green" workspace that contained two latent defects: relations keyed by a tuple, which JSON cannot serialize, so state_hash would have panicked on any state holding a relation; and setup.patch, which was parsed and silently discarded, so every scenario using it would have tested the wrong initial state. Both were invisible until real scenarios exercised them. Green gates on a scaffold mean very little.

Metrics were specified without checking they were reachable or instrumented. AM-4 (≤20 crates) is unreachable given the spec's own K5 and K7 contracts, which require ChaCha and SHA-256 and cost 12 crates between them — the metric was written before those contracts existed and never revisited. AM-12 (cost in USD) was specified in detail, with a price sheet and a formula, and then never instrumented, so it could not be computed at all. A metric with no named instrument is a wish.

The chaos roll never fired. Across the pass the d10 never selected a tier different from the structural one, so the mechanism is untested in practice. It is retained, but v1.0 requires the roll be recorded even when it changes nothing, so its absence is visible rather than assumed.

The coverage gate counts tags, not behaviour. make coverage reports 58/58 by comparing rule IDs in the spec against covers: lists. It catches invented IDs and outright gaps, which is real, but a scenario can name a rule it does not exercise. 100% on that gate is not proof AM-1 is met, and v1.0 says so where the number is reported.

Changes in v1.0

1. Measurement validity (new, Step 5). Every benchmark and harness must assert it performed the work it reports — a positive control. A number from a run that cannot prove it did the work is void.
2. Metric feasibility (Step 4). Each acceptance metric names its instrument and is checked reachable against the contracts in the same spec. Re-checked whenever a contract is added.
3. No silently-ignored input (new). A parsed-but-unused field is a defect. Inputs are honoured or rejected, never dropped.
4. Decisions get commands, not defaults (new). A rule requiring a participant's choice is implemented as a command; if it is not implemented yet, nothing claims coverage of it.
5. Scaffolds are exercised or marked (new). Scaffold code paths not reached by a test are marked as unexercised; a scaffold's green gates are not evidence.
6. Evidence states what it does not support (Step 5). Every cross-runtime comparison names its disanalogies explicitly.
7. The chaos roll is always recorded, including when it does not change the tier.

What did not need changing

The five steps, the four-dimension rubric, the tier system's structural triggers, the ADR gate ("no implementation code before the ADR is committed"), the survey template, and the agentic-efficiency requirements all held up. The gate in particular was never uncomfortable — by the time the ADR was written the decision was easy, which is what a good gate feels like.