diff --git a/history/260731-inner-loop-retrospective.md b/history/260731-inner-loop-retrospective.md new file mode 100644 index 0000000..c6c6f8a --- /dev/null +++ b/history/260731-inner-loop-retrospective.md @@ -0,0 +1,116 @@ +# Inner-loop retrospective — after CB-WP-0001 (first full pass) + +Date: 2026-07-31 +Subject: `specs/InnerLoop.md` v0.2 → v1.0 +Evidence: `evidence/CB-EV-0001-game-kernel.md` + +The loop has now survived one full pass, from SOTA survey through an +adversarial review, an ADR, two specs, and six code iterations that +ended in a committed acceptance run. This is what it got right, what it +missed, and what changes in v1.0. + +## What earned its cost + +**The adversarial review (Step 2).** On its single use it produced one +blocking finding and three significant ones. The blocking one mattered: +the survey claimed boardgame.io's degradation was "architectural, not a +tuning artifact", and the challenge pointed at the `disableUndo` flag +that the claim had not accounted for. The resolution was a rerun with +undo disabled, not a rhetorical defence — degradation persisted (0.66× +at 40k, still DNF at 100k), so the overclaim was retracted and two +mechanisms were attributed separately. A survey that had gone straight +to an ADR would have carried a false claim into the decision. + +**The parity-cap rule.** It stopped the boardgame.io comparison from +being written up as a verdict. The final evidence file says the ~450× +command-rate ratio is a direction, not a verdict, and lists the four +reasons it is not like-for-like. That paragraph exists because the rule +required it. + +**Deferring rules that need a decision.** Four rules (GR-L02, GR-A05, +GR-A11, GR-A12) were left unimplemented across two iterations rather +than given invented defaults, then implemented as explicit commands. +No scenario claimed coverage of them in the interim. This is the single +most useful habit the pass produced and v1.0 promotes it to a rule. + +**The provisional mechanism.** Ten U-items with `provisional: true` +scenarios meant an underdetermined rule cost a scenario tag, not a +kernel decision. One further ambiguity (GR-E02's "successes") was found +during implementation and captured the same way. + +## What the loop missed + +**Both serious errors in this pass were measurement errors, and the +loop caught neither.** Review caught claims; nothing caught numbers. + +1. The boardgame.io harness v1 reported "8.4s for 100k moves". It was + measuring rejected no-ops — every move was refused. Caught by + noticing the error text in output, not by any gate. +2. The Rust benchmark reported **9.3M events/s on a flat curve** and + would have been published as a 93× beat of AM-6. The workload had a + player selecting SUPPORT while parked at Stress 4, so GR-R03 rejected + it, rounds never completed, and throughput was computed for rounds + that never happened. Caught only because a probe test asserted a + round produces events at all. The corrected number is 5.6× lower. + +Both share a shape: **the harness ran successfully while doing no +work**, and success was inferred from "it completed" rather than from +"it did the thing". This is the dominant failure mode of an agent-driven +loop, because an agent will happily report a fast number. + +**Scaffold code with unexercised paths.** T07 produced a compiling, +"green" workspace that contained two latent defects: relations keyed by +a tuple, which JSON cannot serialize, so `state_hash` would have +panicked on any state holding a relation; and `setup.patch`, which was +parsed and silently discarded, so every scenario using it would have +tested the wrong initial state. Both were invisible until real scenarios +exercised them. Green gates on a scaffold mean very little. + +**Metrics were specified without checking they were reachable or +instrumented.** AM-4 (≤20 crates) is unreachable given the spec's own +K5 and K7 contracts, which require ChaCha and SHA-256 and cost 12 crates +between them — the metric was written before those contracts existed and +never revisited. AM-12 (cost in USD) was specified in detail, with a +price sheet and a formula, and then never instrumented, so it could not +be computed at all. A metric with no named instrument is a wish. + +**The chaos roll never fired.** Across the pass the d10 never selected +a tier different from the structural one, so the mechanism is untested +in practice. It is retained, but v1.0 requires the roll be recorded even +when it changes nothing, so its absence is visible rather than assumed. + +**The coverage gate counts tags, not behaviour.** `make coverage` +reports 58/58 by comparing rule IDs in the spec against `covers:` lists. +It catches invented IDs and outright gaps, which is real, but a scenario +can name a rule it does not exercise. 100% on that gate is not proof +AM-1 is met, and v1.0 says so where the number is reported. + +## Changes in v1.0 + +1. **Measurement validity (new, Step 5).** Every benchmark and harness + must assert it performed the work it reports — a positive control. + A number from a run that cannot prove it did the work is void. +2. **Metric feasibility (Step 4).** Each acceptance metric names its + instrument and is checked reachable against the contracts in the same + spec. Re-checked whenever a contract is added. +3. **No silently-ignored input (new).** A parsed-but-unused field is a + defect. Inputs are honoured or rejected, never dropped. +4. **Decisions get commands, not defaults (new).** A rule requiring a + participant's choice is implemented as a command; if it is not + implemented yet, nothing claims coverage of it. +5. **Scaffolds are exercised or marked (new).** Scaffold code paths not + reached by a test are marked as unexercised; a scaffold's green gates + are not evidence. +6. **Evidence states what it does not support (Step 5).** Every + cross-runtime comparison names its disanalogies explicitly. +7. **The chaos roll is always recorded**, including when it does not + change the tier. + +## What did not need changing + +The five steps, the four-dimension rubric, the tier system's structural +triggers, the ADR gate ("no implementation code before the ADR is +committed"), the survey template, and the agentic-efficiency +requirements all held up. The gate in particular was never +uncomfortable — by the time the ADR was written the decision was easy, +which is what a good gate feels like. diff --git a/specs/InnerLoop.md b/specs/InnerLoop.md index 84b4591..e5c35a2 100644 --- a/specs/InnerLoop.md +++ b/specs/InnerLoop.md @@ -1,9 +1,11 @@ # The Inner Loop — Assimilate and Surpass -Status: **v0.2 draft** — becomes v1.0 only after surviving its first full -pass (CB-WP-0001-T09 retrospective). v0.2 adds loop tiers with the chaos -roll, adversarial survey review, and the runnable-baseline option -(maintainer decision, 2026-07-31). +Status: **v1.0** — survived its first full pass (CB-WP-0001, the GROUND +game kernel) and was corrected from it on 2026-07-31. Changes from v0.2: +measurement validity (the positive control), metric feasibility and +instrument naming, four implementation rules the pass earned, and the +requirement that evidence state what it does not support. Rationale and +the failures behind each: `history/260731-inner-loop-retrospective.md`. Normative process for building every Clay-Borg capability. Referenced by all workplans. The loop's own optimization target is **agentic efficiency**: @@ -42,7 +44,10 @@ are never skipped for code-producing work. (`shuf -i 1-10 -n 1`). On a **10**, the tier is instead picked uniformly at random (`shuf -e S M L -n 1`), overriding the structural derivation — up or down. Both rolls are recorded in the tier declaration -(`tier: M (structural L, chaos 10→M)`). Purpose: an occasional random +(`tier: M (structural L, chaos 10→M)`). **Record the roll every time, +including when it changes nothing** (`tier: L (structural L, chaos 4)`), +so a mechanism that never fires is visible rather than assumed. Purpose: +an occasional random reweighting keeps the classification honest — arguing everything into S stops paying off when audits can compare argued tiers against the random sample — and occasionally forces a deep look at something "obviously @@ -115,6 +120,18 @@ the conventions in [MetricsAndScenarios.md](MetricsAndScenarios.md), including the rule that metric selection itself passes through a mini research step (metric provenance). +**Every metric names its instrument, and is checked reachable.** A row +in the acceptance table carries the command that produces its number. +A metric with no named instrument is a wish, not a metric. A metric must +also be checked against the contracts in the *same spec*: if a contract +makes a target unreachable, one of the two is wrong and the conflict is +resolved when it is noticed, not at the acceptance run. Re-check the +table whenever a contract is added. + +*(v1.0, from CB-WP-0001: AM-4's ≤20-crate target was made unreachable by +the K5 and K7 contracts written after it, and AM-12's cost metric was +fully specified and never instrumented, so it could not be computed.)* + ### Step 5 — Code loop Implement iteratively. Each iteration: @@ -129,6 +146,40 @@ comparison numbers are committed as an evidence file (`evidence/CB-EV-NNNN-.md`). A failed scenario must yield a replay artifact an agent can re-execute locally. +#### Measurement validity — the positive control + +**Every benchmark and harness must assert that it performed the work it +reports.** Completing without error is not evidence of having done +anything: a loop whose commands are all rejected runs fast and reports a +throughput for work that never happened. + +Concretely, a measurement harness must, on every run: + +- assert the unit of work produced its expected effect (events applied, + rows written, moves accepted) — not merely that the call returned; +- fail loudly rather than report a number when that assertion fails; +- state the divisor used to convert raw timings into the metric's unit, + pinned by a test so a workload change cannot silently rescale it. + +**A number from a run that cannot prove it did the work is void** and +must not reach an evidence file. + +*(v1.0, from CB-WP-0001: both serious errors in the first pass were of +exactly this shape. A JS harness reported 8.4s for 100k moves while +every move was being rejected, and a Rust benchmark reported 9.3M +events/s — a 93× beat — while most rounds never completed because a +stress gate rejected one player's action. The corrected figure was 5.6× +lower. Adversarial review caught neither; both were claims about +numbers, and review reads prose.)* + +#### Evidence states what it does not support + +An evidence file that compares across runtimes, languages, or feature +sets **names the disanalogies explicitly**, in the same section as the +number. The reader must not have to infer that a ratio is not +like-for-like. This is the parity-cap rule applied to the write-up: +state the claim you will defend, and the claim you are not making. + --- ## The four-dimension rubric @@ -171,6 +222,33 @@ Which candidate leads per dimension; what none of them do well --- +## Implementation rules the first pass earned + +These are cheap, and each exists because its absence cost something in +CB-WP-0001. See `history/260731-inner-loop-retrospective.md`. + +1. **No silently-ignored input.** A field that is parsed and then unused + is a defect, not a stub. Inputs are honoured or rejected with an + error — never dropped. *(A scenario `setup.patch` was parsed and + discarded; every scenario using it would have tested the wrong + initial state while passing.)* +2. **Decisions get commands, not defaults.** A rule that requires a + participant's choice is implemented as a command carrying that + choice. Until it is, **nothing claims coverage of it** — no tag, no + scenario, no acceptance row. Inventing a default to make a rule + "done" is the failure this prevents. +3. **Scaffolds are exercised or marked.** A scaffold's green gates are + not evidence. Any scaffold path no test reaches is marked as + unexercised. *(A compiling, fully-green scaffold shipped a state-hash + that would panic on any state holding a relation.)* +4. **Coverage gates that count tags say so.** A gate comparing rule IDs + against `covers:` lists proves no rule is unclaimed and no claimed + rule is invented. It does **not** prove a scenario exercises what it + names. Wherever such a number is reported, that limit is reported + with it. + +--- + ## Agentic-efficiency requirements The loop exists to be driven by agents. Therefore: @@ -202,6 +280,12 @@ A capability has completed the loop when all of the following are committed: - [ ] specs/.md with acceptance-metrics table - [ ] passing scenarios covering every numbered spec rule - [ ] evidence/CB-EV-NNNN with final comparison vs baseline, no `unmeasured` +- [ ] every reported number produced by a harness with a positive + control; any metric that could not be instrumented is recorded as + uncomputable rather than estimated +- [ ] every unmet metric reported as unmet, with attribution and the + options for resolving it — a missed target is an output of the + loop, not a reason to move the target quietly - [ ] retrospective note (may be one paragraph appended to the evidence file): what the loop itself should change ``` diff --git a/workplans/CB-WP-0001-inner-loop.md b/workplans/CB-WP-0001-inner-loop.md index f1c9176..009b2a1 100644 --- a/workplans/CB-WP-0001-inner-loop.md +++ b/workplans/CB-WP-0001-inner-loop.md @@ -1,7 +1,7 @@ --- id: CB-WP-0001 title: "Establish the assimilate-and-surpass inner loop via the GROUND game kernel" -status: active +status: done state_hub_workstream_id: "a1b434dc-b1c6-46b5-bbd9-80a4e6b7620f" --- @@ -183,7 +183,7 @@ per-task token counts were never instrumented — also a T09 input. ```task id: CB-WP-0001-T09 -status: todo +status: done priority: low state_hub_task_id: "e99b107e-087e-49e1-96b5-67805deb242f" ``` @@ -193,3 +193,11 @@ heavy or too thin, what the survey template missed, what metrics turned out to matter. Output is InnerLoop v1.0 — the process the next capability workplan starts from. The loop is only "established" once it has survived its first full pass and been corrected. + +**Outcome:** specs/InnerLoop.md v1.0 and +history/260731-inner-loop-retrospective.md. The adversarial review, the +parity cap, the provisional mechanism and the ADR gate held. The gap the +pass exposed: both serious errors were *measurement* errors and review +caught neither, because review reads prose and these were claims about +numbers. v1.0 adds the positive-control rule, metric feasibility and +instrument naming, and four implementation rules.