T09: InnerLoop v1.0 — corrected from the first full pass
Some checks failed
ci / check (push) Has been cancelled

The loop has now survived CB-WP-0001 end to end and is revised from what
that pass actually taught, recorded in
history/260731-inner-loop-retrospective.md.

Held up: the adversarial review (one blocking and three significant
findings on its single use, resolved with new measurements rather than
argument), the parity-cap rule, the provisional U-item mechanism, and
the ADR gate.

The gap: both serious errors in the pass were measurement errors, and
the loop caught neither. Review reads prose; these were claims about
numbers. In both cases a harness ran successfully while doing no work —
a JS harness timing rejected no-ops, and a Rust benchmark computing
throughput for rounds that never completed.

v1.0 adds:
- Measurement validity: a harness must assert it performed the work it
  reports. A number from a run that cannot prove it did the work is void.
- Metric feasibility: every metric names its instrument and is checked
  reachable against the contracts in its own spec.
- No silently-ignored input; decisions get commands, not defaults;
  scaffolds are exercised or marked; coverage gates that count tags say
  so.
- Evidence must state what a comparison does not support.
- The chaos roll is recorded even when it changes nothing, so a
  mechanism that never fires is visible rather than assumed.

CB-WP-0001 is complete: 9/9 tasks done.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-31 03:11:19 +02:00
parent eb1378e667
commit 63dcffd9f1
3 changed files with 215 additions and 7 deletions

View file

@ -0,0 +1,116 @@
# Inner-loop retrospective — after CB-WP-0001 (first full pass)
Date: 2026-07-31
Subject: `specs/InnerLoop.md` v0.2 → v1.0
Evidence: `evidence/CB-EV-0001-game-kernel.md`
The loop has now survived one full pass, from SOTA survey through an
adversarial review, an ADR, two specs, and six code iterations that
ended in a committed acceptance run. This is what it got right, what it
missed, and what changes in v1.0.
## What earned its cost
**The adversarial review (Step 2).** On its single use it produced one
blocking finding and three significant ones. The blocking one mattered:
the survey claimed boardgame.io's degradation was "architectural, not a
tuning artifact", and the challenge pointed at the `disableUndo` flag
that the claim had not accounted for. The resolution was a rerun with
undo disabled, not a rhetorical defence — degradation persisted (0.66×
at 40k, still DNF at 100k), so the overclaim was retracted and two
mechanisms were attributed separately. A survey that had gone straight
to an ADR would have carried a false claim into the decision.
**The parity-cap rule.** It stopped the boardgame.io comparison from
being written up as a verdict. The final evidence file says the ~450×
command-rate ratio is a direction, not a verdict, and lists the four
reasons it is not like-for-like. That paragraph exists because the rule
required it.
**Deferring rules that need a decision.** Four rules (GR-L02, GR-A05,
GR-A11, GR-A12) were left unimplemented across two iterations rather
than given invented defaults, then implemented as explicit commands.
No scenario claimed coverage of them in the interim. This is the single
most useful habit the pass produced and v1.0 promotes it to a rule.
**The provisional mechanism.** Ten U-items with `provisional: true`
scenarios meant an underdetermined rule cost a scenario tag, not a
kernel decision. One further ambiguity (GR-E02's "successes") was found
during implementation and captured the same way.
## What the loop missed
**Both serious errors in this pass were measurement errors, and the
loop caught neither.** Review caught claims; nothing caught numbers.
1. The boardgame.io harness v1 reported "8.4s for 100k moves". It was
measuring rejected no-ops — every move was refused. Caught by
noticing the error text in output, not by any gate.
2. The Rust benchmark reported **9.3M events/s on a flat curve** and
would have been published as a 93× beat of AM-6. The workload had a
player selecting SUPPORT while parked at Stress 4, so GR-R03 rejected
it, rounds never completed, and throughput was computed for rounds
that never happened. Caught only because a probe test asserted a
round produces events at all. The corrected number is 5.6× lower.
Both share a shape: **the harness ran successfully while doing no
work**, and success was inferred from "it completed" rather than from
"it did the thing". This is the dominant failure mode of an agent-driven
loop, because an agent will happily report a fast number.
**Scaffold code with unexercised paths.** T07 produced a compiling,
"green" workspace that contained two latent defects: relations keyed by
a tuple, which JSON cannot serialize, so `state_hash` would have
panicked on any state holding a relation; and `setup.patch`, which was
parsed and silently discarded, so every scenario using it would have
tested the wrong initial state. Both were invisible until real scenarios
exercised them. Green gates on a scaffold mean very little.
**Metrics were specified without checking they were reachable or
instrumented.** AM-4 (≤20 crates) is unreachable given the spec's own
K5 and K7 contracts, which require ChaCha and SHA-256 and cost 12 crates
between them — the metric was written before those contracts existed and
never revisited. AM-12 (cost in USD) was specified in detail, with a
price sheet and a formula, and then never instrumented, so it could not
be computed at all. A metric with no named instrument is a wish.
**The chaos roll never fired.** Across the pass the d10 never selected
a tier different from the structural one, so the mechanism is untested
in practice. It is retained, but v1.0 requires the roll be recorded even
when it changes nothing, so its absence is visible rather than assumed.
**The coverage gate counts tags, not behaviour.** `make coverage`
reports 58/58 by comparing rule IDs in the spec against `covers:` lists.
It catches invented IDs and outright gaps, which is real, but a scenario
can name a rule it does not exercise. 100% on that gate is not proof
AM-1 is met, and v1.0 says so where the number is reported.
## Changes in v1.0
1. **Measurement validity (new, Step 5).** Every benchmark and harness
must assert it performed the work it reports — a positive control.
A number from a run that cannot prove it did the work is void.
2. **Metric feasibility (Step 4).** Each acceptance metric names its
instrument and is checked reachable against the contracts in the same
spec. Re-checked whenever a contract is added.
3. **No silently-ignored input (new).** A parsed-but-unused field is a
defect. Inputs are honoured or rejected, never dropped.
4. **Decisions get commands, not defaults (new).** A rule requiring a
participant's choice is implemented as a command; if it is not
implemented yet, nothing claims coverage of it.
5. **Scaffolds are exercised or marked (new).** Scaffold code paths not
reached by a test are marked as unexercised; a scaffold's green gates
are not evidence.
6. **Evidence states what it does not support (Step 5).** Every
cross-runtime comparison names its disanalogies explicitly.
7. **The chaos roll is always recorded**, including when it does not
change the tier.
## What did not need changing
The five steps, the four-dimension rubric, the tier system's structural
triggers, the ADR gate ("no implementation code before the ADR is
committed"), the survey template, and the agentic-efficiency
requirements all held up. The gate in particular was never
uncomfortable — by the time the ADR was written the decision was easy,
which is what a good gate feels like.

View file

@ -1,9 +1,11 @@
# The Inner Loop — Assimilate and Surpass
Status: **v0.2 draft** — becomes v1.0 only after surviving its first full
pass (CB-WP-0001-T09 retrospective). v0.2 adds loop tiers with the chaos
roll, adversarial survey review, and the runnable-baseline option
(maintainer decision, 2026-07-31).
Status: **v1.0** — survived its first full pass (CB-WP-0001, the GROUND
game kernel) and was corrected from it on 2026-07-31. Changes from v0.2:
measurement validity (the positive control), metric feasibility and
instrument naming, four implementation rules the pass earned, and the
requirement that evidence state what it does not support. Rationale and
the failures behind each: `history/260731-inner-loop-retrospective.md`.
Normative process for building every Clay-Borg capability. Referenced by
all workplans. The loop's own optimization target is **agentic efficiency**:
@ -42,7 +44,10 @@ are never skipped for code-producing work.
(`shuf -i 1-10 -n 1`). On a **10**, the tier is instead picked uniformly at
random (`shuf -e S M L -n 1`), overriding the structural derivation — up or
down. Both rolls are recorded in the tier declaration
(`tier: M (structural L, chaos 10→M)`). Purpose: an occasional random
(`tier: M (structural L, chaos 10→M)`). **Record the roll every time,
including when it changes nothing** (`tier: L (structural L, chaos 4)`),
so a mechanism that never fires is visible rather than assumed. Purpose:
an occasional random
reweighting keeps the classification honest — arguing everything into S
stops paying off when audits can compare argued tiers against the random
sample — and occasionally forces a deep look at something "obviously
@ -115,6 +120,18 @@ the conventions in [MetricsAndScenarios.md](MetricsAndScenarios.md),
including the rule that metric selection itself passes through a mini
research step (metric provenance).
**Every metric names its instrument, and is checked reachable.** A row
in the acceptance table carries the command that produces its number.
A metric with no named instrument is a wish, not a metric. A metric must
also be checked against the contracts in the *same spec*: if a contract
makes a target unreachable, one of the two is wrong and the conflict is
resolved when it is noticed, not at the acceptance run. Re-check the
table whenever a contract is added.
*(v1.0, from CB-WP-0001: AM-4's ≤20-crate target was made unreachable by
the K5 and K7 contracts written after it, and AM-12's cost metric was
fully specified and never instrumented, so it could not be computed.)*
### Step 5 — Code loop
Implement iteratively. Each iteration:
@ -129,6 +146,40 @@ comparison numbers are committed as an evidence file
(`evidence/CB-EV-NNNN-<slug>.md`). A failed scenario must yield a replay
artifact an agent can re-execute locally.
#### Measurement validity — the positive control
**Every benchmark and harness must assert that it performed the work it
reports.** Completing without error is not evidence of having done
anything: a loop whose commands are all rejected runs fast and reports a
throughput for work that never happened.
Concretely, a measurement harness must, on every run:
- assert the unit of work produced its expected effect (events applied,
rows written, moves accepted) — not merely that the call returned;
- fail loudly rather than report a number when that assertion fails;
- state the divisor used to convert raw timings into the metric's unit,
pinned by a test so a workload change cannot silently rescale it.
**A number from a run that cannot prove it did the work is void** and
must not reach an evidence file.
*(v1.0, from CB-WP-0001: both serious errors in the first pass were of
exactly this shape. A JS harness reported 8.4s for 100k moves while
every move was being rejected, and a Rust benchmark reported 9.3M
events/s — a 93× beat — while most rounds never completed because a
stress gate rejected one player's action. The corrected figure was 5.6×
lower. Adversarial review caught neither; both were claims about
numbers, and review reads prose.)*
#### Evidence states what it does not support
An evidence file that compares across runtimes, languages, or feature
sets **names the disanalogies explicitly**, in the same section as the
number. The reader must not have to infer that a ratio is not
like-for-like. This is the parity-cap rule applied to the write-up:
state the claim you will defend, and the claim you are not making.
---
## The four-dimension rubric
@ -171,6 +222,33 @@ Which candidate leads per dimension; what none of them do well
---
## Implementation rules the first pass earned
These are cheap, and each exists because its absence cost something in
CB-WP-0001. See `history/260731-inner-loop-retrospective.md`.
1. **No silently-ignored input.** A field that is parsed and then unused
is a defect, not a stub. Inputs are honoured or rejected with an
error — never dropped. *(A scenario `setup.patch` was parsed and
discarded; every scenario using it would have tested the wrong
initial state while passing.)*
2. **Decisions get commands, not defaults.** A rule that requires a
participant's choice is implemented as a command carrying that
choice. Until it is, **nothing claims coverage of it** — no tag, no
scenario, no acceptance row. Inventing a default to make a rule
"done" is the failure this prevents.
3. **Scaffolds are exercised or marked.** A scaffold's green gates are
not evidence. Any scaffold path no test reaches is marked as
unexercised. *(A compiling, fully-green scaffold shipped a state-hash
that would panic on any state holding a relation.)*
4. **Coverage gates that count tags say so.** A gate comparing rule IDs
against `covers:` lists proves no rule is unclaimed and no claimed
rule is invented. It does **not** prove a scenario exercises what it
names. Wherever such a number is reported, that limit is reported
with it.
---
## Agentic-efficiency requirements
The loop exists to be driven by agents. Therefore:
@ -202,6 +280,12 @@ A capability has completed the loop when all of the following are committed:
- [ ] specs/<Capability>.md with acceptance-metrics table
- [ ] passing scenarios covering every numbered spec rule
- [ ] evidence/CB-EV-NNNN with final comparison vs baseline, no `unmeasured`
- [ ] every reported number produced by a harness with a positive
control; any metric that could not be instrumented is recorded as
uncomputable rather than estimated
- [ ] every unmet metric reported as unmet, with attribution and the
options for resolving it — a missed target is an output of the
loop, not a reason to move the target quietly
- [ ] retrospective note (may be one paragraph appended to the evidence
file): what the loop itself should change
```

View file

@ -1,7 +1,7 @@
---
id: CB-WP-0001
title: "Establish the assimilate-and-surpass inner loop via the GROUND game kernel"
status: active
status: done
state_hub_workstream_id: "a1b434dc-b1c6-46b5-bbd9-80a4e6b7620f"
---
@ -183,7 +183,7 @@ per-task token counts were never instrumented — also a T09 input.
```task
id: CB-WP-0001-T09
status: todo
status: done
priority: low
state_hub_task_id: "e99b107e-087e-49e1-96b5-67805deb242f"
```
@ -193,3 +193,11 @@ heavy or too thin, what the survey template missed, what metrics turned out
to matter. Output is InnerLoop v1.0 — the process the next capability
workplan starts from. The loop is only "established" once it has survived
its first full pass and been corrected.
**Outcome:** specs/InnerLoop.md v1.0 and
history/260731-inner-loop-retrospective.md. The adversarial review, the
parity cap, the provisional mechanism and the ADR gate held. The gap the
pass exposed: both serious errors were *measurement* errors and review
caught neither, because review reads prose and these were claims about
numbers. v1.0 adds the positive-control rule, metric feasibility and
instrument naming, and four implementation rules.