tegwick
63dcffd9f1
Some checks failed
ci / check (push) Has been cancelled
The loop has now survived CB-WP-0001 end to end and is revised from what that pass actually taught, recorded in history/260731-inner-loop-retrospective.md. Held up: the adversarial review (one blocking and three significant findings on its single use, resolved with new measurements rather than argument), the parity-cap rule, the provisional U-item mechanism, and the ADR gate. The gap: both serious errors in the pass were measurement errors, and the loop caught neither. Review reads prose; these were claims about numbers. In both cases a harness ran successfully while doing no work — a JS harness timing rejected no-ops, and a Rust benchmark computing throughput for rounds that never completed. v1.0 adds: - Measurement validity: a harness must assert it performed the work it reports. A number from a run that cannot prove it did the work is void. - Metric feasibility: every metric names its instrument and is checked reachable against the contracts in its own spec. - No silently-ignored input; decisions get commands, not defaults; scaffolds are exercised or marked; coverage gates that count tags say so. - Evidence must state what a comparison does not support. - The chaos roll is recorded even when it changes nothing, so a mechanism that never fires is visible rather than assumed. CB-WP-0001 is complete: 9/9 tasks done. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
116 lines
6 KiB
Markdown
116 lines
6 KiB
Markdown
# Inner-loop retrospective — after CB-WP-0001 (first full pass)
|
||
|
||
Date: 2026-07-31
|
||
Subject: `specs/InnerLoop.md` v0.2 → v1.0
|
||
Evidence: `evidence/CB-EV-0001-game-kernel.md`
|
||
|
||
The loop has now survived one full pass, from SOTA survey through an
|
||
adversarial review, an ADR, two specs, and six code iterations that
|
||
ended in a committed acceptance run. This is what it got right, what it
|
||
missed, and what changes in v1.0.
|
||
|
||
## What earned its cost
|
||
|
||
**The adversarial review (Step 2).** On its single use it produced one
|
||
blocking finding and three significant ones. The blocking one mattered:
|
||
the survey claimed boardgame.io's degradation was "architectural, not a
|
||
tuning artifact", and the challenge pointed at the `disableUndo` flag
|
||
that the claim had not accounted for. The resolution was a rerun with
|
||
undo disabled, not a rhetorical defence — degradation persisted (0.66×
|
||
at 40k, still DNF at 100k), so the overclaim was retracted and two
|
||
mechanisms were attributed separately. A survey that had gone straight
|
||
to an ADR would have carried a false claim into the decision.
|
||
|
||
**The parity-cap rule.** It stopped the boardgame.io comparison from
|
||
being written up as a verdict. The final evidence file says the ~450×
|
||
command-rate ratio is a direction, not a verdict, and lists the four
|
||
reasons it is not like-for-like. That paragraph exists because the rule
|
||
required it.
|
||
|
||
**Deferring rules that need a decision.** Four rules (GR-L02, GR-A05,
|
||
GR-A11, GR-A12) were left unimplemented across two iterations rather
|
||
than given invented defaults, then implemented as explicit commands.
|
||
No scenario claimed coverage of them in the interim. This is the single
|
||
most useful habit the pass produced and v1.0 promotes it to a rule.
|
||
|
||
**The provisional mechanism.** Ten U-items with `provisional: true`
|
||
scenarios meant an underdetermined rule cost a scenario tag, not a
|
||
kernel decision. One further ambiguity (GR-E02's "successes") was found
|
||
during implementation and captured the same way.
|
||
|
||
## What the loop missed
|
||
|
||
**Both serious errors in this pass were measurement errors, and the
|
||
loop caught neither.** Review caught claims; nothing caught numbers.
|
||
|
||
1. The boardgame.io harness v1 reported "8.4s for 100k moves". It was
|
||
measuring rejected no-ops — every move was refused. Caught by
|
||
noticing the error text in output, not by any gate.
|
||
2. The Rust benchmark reported **9.3M events/s on a flat curve** and
|
||
would have been published as a 93× beat of AM-6. The workload had a
|
||
player selecting SUPPORT while parked at Stress 4, so GR-R03 rejected
|
||
it, rounds never completed, and throughput was computed for rounds
|
||
that never happened. Caught only because a probe test asserted a
|
||
round produces events at all. The corrected number is 5.6× lower.
|
||
|
||
Both share a shape: **the harness ran successfully while doing no
|
||
work**, and success was inferred from "it completed" rather than from
|
||
"it did the thing". This is the dominant failure mode of an agent-driven
|
||
loop, because an agent will happily report a fast number.
|
||
|
||
**Scaffold code with unexercised paths.** T07 produced a compiling,
|
||
"green" workspace that contained two latent defects: relations keyed by
|
||
a tuple, which JSON cannot serialize, so `state_hash` would have
|
||
panicked on any state holding a relation; and `setup.patch`, which was
|
||
parsed and silently discarded, so every scenario using it would have
|
||
tested the wrong initial state. Both were invisible until real scenarios
|
||
exercised them. Green gates on a scaffold mean very little.
|
||
|
||
**Metrics were specified without checking they were reachable or
|
||
instrumented.** AM-4 (≤20 crates) is unreachable given the spec's own
|
||
K5 and K7 contracts, which require ChaCha and SHA-256 and cost 12 crates
|
||
between them — the metric was written before those contracts existed and
|
||
never revisited. AM-12 (cost in USD) was specified in detail, with a
|
||
price sheet and a formula, and then never instrumented, so it could not
|
||
be computed at all. A metric with no named instrument is a wish.
|
||
|
||
**The chaos roll never fired.** Across the pass the d10 never selected
|
||
a tier different from the structural one, so the mechanism is untested
|
||
in practice. It is retained, but v1.0 requires the roll be recorded even
|
||
when it changes nothing, so its absence is visible rather than assumed.
|
||
|
||
**The coverage gate counts tags, not behaviour.** `make coverage`
|
||
reports 58/58 by comparing rule IDs in the spec against `covers:` lists.
|
||
It catches invented IDs and outright gaps, which is real, but a scenario
|
||
can name a rule it does not exercise. 100% on that gate is not proof
|
||
AM-1 is met, and v1.0 says so where the number is reported.
|
||
|
||
## Changes in v1.0
|
||
|
||
1. **Measurement validity (new, Step 5).** Every benchmark and harness
|
||
must assert it performed the work it reports — a positive control.
|
||
A number from a run that cannot prove it did the work is void.
|
||
2. **Metric feasibility (Step 4).** Each acceptance metric names its
|
||
instrument and is checked reachable against the contracts in the same
|
||
spec. Re-checked whenever a contract is added.
|
||
3. **No silently-ignored input (new).** A parsed-but-unused field is a
|
||
defect. Inputs are honoured or rejected, never dropped.
|
||
4. **Decisions get commands, not defaults (new).** A rule requiring a
|
||
participant's choice is implemented as a command; if it is not
|
||
implemented yet, nothing claims coverage of it.
|
||
5. **Scaffolds are exercised or marked (new).** Scaffold code paths not
|
||
reached by a test are marked as unexercised; a scaffold's green gates
|
||
are not evidence.
|
||
6. **Evidence states what it does not support (Step 5).** Every
|
||
cross-runtime comparison names its disanalogies explicitly.
|
||
7. **The chaos roll is always recorded**, including when it does not
|
||
change the tier.
|
||
|
||
## What did not need changing
|
||
|
||
The five steps, the four-dimension rubric, the tier system's structural
|
||
triggers, the ADR gate ("no implementation code before the ADR is
|
||
committed"), the survey template, and the agentic-efficiency
|
||
requirements all held up. The gate in particular was never
|
||
uncomfortable — by the time the ADR was written the decision was easy,
|
||
which is what a good gate feels like.
|