T09: InnerLoop v1.0 — corrected from the first full pass
Some checks failed
ci / check (push) Has been cancelled
Some checks failed
ci / check (push) Has been cancelled
The loop has now survived CB-WP-0001 end to end and is revised from what that pass actually taught, recorded in history/260731-inner-loop-retrospective.md. Held up: the adversarial review (one blocking and three significant findings on its single use, resolved with new measurements rather than argument), the parity-cap rule, the provisional U-item mechanism, and the ADR gate. The gap: both serious errors in the pass were measurement errors, and the loop caught neither. Review reads prose; these were claims about numbers. In both cases a harness ran successfully while doing no work — a JS harness timing rejected no-ops, and a Rust benchmark computing throughput for rounds that never completed. v1.0 adds: - Measurement validity: a harness must assert it performed the work it reports. A number from a run that cannot prove it did the work is void. - Metric feasibility: every metric names its instrument and is checked reachable against the contracts in its own spec. - No silently-ignored input; decisions get commands, not defaults; scaffolds are exercised or marked; coverage gates that count tags say so. - Evidence must state what a comparison does not support. - The chaos roll is recorded even when it changes nothing, so a mechanism that never fires is visible rather than assumed. CB-WP-0001 is complete: 9/9 tasks done. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
eb1378e667
commit
63dcffd9f1
3 changed files with 215 additions and 7 deletions
116
history/260731-inner-loop-retrospective.md
Normal file
116
history/260731-inner-loop-retrospective.md
Normal file
|
|
@ -0,0 +1,116 @@
|
|||
# Inner-loop retrospective — after CB-WP-0001 (first full pass)
|
||||
|
||||
Date: 2026-07-31
|
||||
Subject: `specs/InnerLoop.md` v0.2 → v1.0
|
||||
Evidence: `evidence/CB-EV-0001-game-kernel.md`
|
||||
|
||||
The loop has now survived one full pass, from SOTA survey through an
|
||||
adversarial review, an ADR, two specs, and six code iterations that
|
||||
ended in a committed acceptance run. This is what it got right, what it
|
||||
missed, and what changes in v1.0.
|
||||
|
||||
## What earned its cost
|
||||
|
||||
**The adversarial review (Step 2).** On its single use it produced one
|
||||
blocking finding and three significant ones. The blocking one mattered:
|
||||
the survey claimed boardgame.io's degradation was "architectural, not a
|
||||
tuning artifact", and the challenge pointed at the `disableUndo` flag
|
||||
that the claim had not accounted for. The resolution was a rerun with
|
||||
undo disabled, not a rhetorical defence — degradation persisted (0.66×
|
||||
at 40k, still DNF at 100k), so the overclaim was retracted and two
|
||||
mechanisms were attributed separately. A survey that had gone straight
|
||||
to an ADR would have carried a false claim into the decision.
|
||||
|
||||
**The parity-cap rule.** It stopped the boardgame.io comparison from
|
||||
being written up as a verdict. The final evidence file says the ~450×
|
||||
command-rate ratio is a direction, not a verdict, and lists the four
|
||||
reasons it is not like-for-like. That paragraph exists because the rule
|
||||
required it.
|
||||
|
||||
**Deferring rules that need a decision.** Four rules (GR-L02, GR-A05,
|
||||
GR-A11, GR-A12) were left unimplemented across two iterations rather
|
||||
than given invented defaults, then implemented as explicit commands.
|
||||
No scenario claimed coverage of them in the interim. This is the single
|
||||
most useful habit the pass produced and v1.0 promotes it to a rule.
|
||||
|
||||
**The provisional mechanism.** Ten U-items with `provisional: true`
|
||||
scenarios meant an underdetermined rule cost a scenario tag, not a
|
||||
kernel decision. One further ambiguity (GR-E02's "successes") was found
|
||||
during implementation and captured the same way.
|
||||
|
||||
## What the loop missed
|
||||
|
||||
**Both serious errors in this pass were measurement errors, and the
|
||||
loop caught neither.** Review caught claims; nothing caught numbers.
|
||||
|
||||
1. The boardgame.io harness v1 reported "8.4s for 100k moves". It was
|
||||
measuring rejected no-ops — every move was refused. Caught by
|
||||
noticing the error text in output, not by any gate.
|
||||
2. The Rust benchmark reported **9.3M events/s on a flat curve** and
|
||||
would have been published as a 93× beat of AM-6. The workload had a
|
||||
player selecting SUPPORT while parked at Stress 4, so GR-R03 rejected
|
||||
it, rounds never completed, and throughput was computed for rounds
|
||||
that never happened. Caught only because a probe test asserted a
|
||||
round produces events at all. The corrected number is 5.6× lower.
|
||||
|
||||
Both share a shape: **the harness ran successfully while doing no
|
||||
work**, and success was inferred from "it completed" rather than from
|
||||
"it did the thing". This is the dominant failure mode of an agent-driven
|
||||
loop, because an agent will happily report a fast number.
|
||||
|
||||
**Scaffold code with unexercised paths.** T07 produced a compiling,
|
||||
"green" workspace that contained two latent defects: relations keyed by
|
||||
a tuple, which JSON cannot serialize, so `state_hash` would have
|
||||
panicked on any state holding a relation; and `setup.patch`, which was
|
||||
parsed and silently discarded, so every scenario using it would have
|
||||
tested the wrong initial state. Both were invisible until real scenarios
|
||||
exercised them. Green gates on a scaffold mean very little.
|
||||
|
||||
**Metrics were specified without checking they were reachable or
|
||||
instrumented.** AM-4 (≤20 crates) is unreachable given the spec's own
|
||||
K5 and K7 contracts, which require ChaCha and SHA-256 and cost 12 crates
|
||||
between them — the metric was written before those contracts existed and
|
||||
never revisited. AM-12 (cost in USD) was specified in detail, with a
|
||||
price sheet and a formula, and then never instrumented, so it could not
|
||||
be computed at all. A metric with no named instrument is a wish.
|
||||
|
||||
**The chaos roll never fired.** Across the pass the d10 never selected
|
||||
a tier different from the structural one, so the mechanism is untested
|
||||
in practice. It is retained, but v1.0 requires the roll be recorded even
|
||||
when it changes nothing, so its absence is visible rather than assumed.
|
||||
|
||||
**The coverage gate counts tags, not behaviour.** `make coverage`
|
||||
reports 58/58 by comparing rule IDs in the spec against `covers:` lists.
|
||||
It catches invented IDs and outright gaps, which is real, but a scenario
|
||||
can name a rule it does not exercise. 100% on that gate is not proof
|
||||
AM-1 is met, and v1.0 says so where the number is reported.
|
||||
|
||||
## Changes in v1.0
|
||||
|
||||
1. **Measurement validity (new, Step 5).** Every benchmark and harness
|
||||
must assert it performed the work it reports — a positive control.
|
||||
A number from a run that cannot prove it did the work is void.
|
||||
2. **Metric feasibility (Step 4).** Each acceptance metric names its
|
||||
instrument and is checked reachable against the contracts in the same
|
||||
spec. Re-checked whenever a contract is added.
|
||||
3. **No silently-ignored input (new).** A parsed-but-unused field is a
|
||||
defect. Inputs are honoured or rejected, never dropped.
|
||||
4. **Decisions get commands, not defaults (new).** A rule requiring a
|
||||
participant's choice is implemented as a command; if it is not
|
||||
implemented yet, nothing claims coverage of it.
|
||||
5. **Scaffolds are exercised or marked (new).** Scaffold code paths not
|
||||
reached by a test are marked as unexercised; a scaffold's green gates
|
||||
are not evidence.
|
||||
6. **Evidence states what it does not support (Step 5).** Every
|
||||
cross-runtime comparison names its disanalogies explicitly.
|
||||
7. **The chaos roll is always recorded**, including when it does not
|
||||
change the tier.
|
||||
|
||||
## What did not need changing
|
||||
|
||||
The five steps, the four-dimension rubric, the tier system's structural
|
||||
triggers, the ADR gate ("no implementation code before the ADR is
|
||||
committed"), the survey template, and the agentic-efficiency
|
||||
requirements all held up. The gate in particular was never
|
||||
uncomfortable — by the time the ADR was written the decision was easy,
|
||||
which is what a good gate feels like.
|
||||
|
|
@ -1,9 +1,11 @@
|
|||
# The Inner Loop — Assimilate and Surpass
|
||||
|
||||
Status: **v0.2 draft** — becomes v1.0 only after surviving its first full
|
||||
pass (CB-WP-0001-T09 retrospective). v0.2 adds loop tiers with the chaos
|
||||
roll, adversarial survey review, and the runnable-baseline option
|
||||
(maintainer decision, 2026-07-31).
|
||||
Status: **v1.0** — survived its first full pass (CB-WP-0001, the GROUND
|
||||
game kernel) and was corrected from it on 2026-07-31. Changes from v0.2:
|
||||
measurement validity (the positive control), metric feasibility and
|
||||
instrument naming, four implementation rules the pass earned, and the
|
||||
requirement that evidence state what it does not support. Rationale and
|
||||
the failures behind each: `history/260731-inner-loop-retrospective.md`.
|
||||
|
||||
Normative process for building every Clay-Borg capability. Referenced by
|
||||
all workplans. The loop's own optimization target is **agentic efficiency**:
|
||||
|
|
@ -42,7 +44,10 @@ are never skipped for code-producing work.
|
|||
(`shuf -i 1-10 -n 1`). On a **10**, the tier is instead picked uniformly at
|
||||
random (`shuf -e S M L -n 1`), overriding the structural derivation — up or
|
||||
down. Both rolls are recorded in the tier declaration
|
||||
(`tier: M (structural L, chaos 10→M)`). Purpose: an occasional random
|
||||
(`tier: M (structural L, chaos 10→M)`). **Record the roll every time,
|
||||
including when it changes nothing** (`tier: L (structural L, chaos 4)`),
|
||||
so a mechanism that never fires is visible rather than assumed. Purpose:
|
||||
an occasional random
|
||||
reweighting keeps the classification honest — arguing everything into S
|
||||
stops paying off when audits can compare argued tiers against the random
|
||||
sample — and occasionally forces a deep look at something "obviously
|
||||
|
|
@ -115,6 +120,18 @@ the conventions in [MetricsAndScenarios.md](MetricsAndScenarios.md),
|
|||
including the rule that metric selection itself passes through a mini
|
||||
research step (metric provenance).
|
||||
|
||||
**Every metric names its instrument, and is checked reachable.** A row
|
||||
in the acceptance table carries the command that produces its number.
|
||||
A metric with no named instrument is a wish, not a metric. A metric must
|
||||
also be checked against the contracts in the *same spec*: if a contract
|
||||
makes a target unreachable, one of the two is wrong and the conflict is
|
||||
resolved when it is noticed, not at the acceptance run. Re-check the
|
||||
table whenever a contract is added.
|
||||
|
||||
*(v1.0, from CB-WP-0001: AM-4's ≤20-crate target was made unreachable by
|
||||
the K5 and K7 contracts written after it, and AM-12's cost metric was
|
||||
fully specified and never instrumented, so it could not be computed.)*
|
||||
|
||||
### Step 5 — Code loop
|
||||
|
||||
Implement iteratively. Each iteration:
|
||||
|
|
@ -129,6 +146,40 @@ comparison numbers are committed as an evidence file
|
|||
(`evidence/CB-EV-NNNN-<slug>.md`). A failed scenario must yield a replay
|
||||
artifact an agent can re-execute locally.
|
||||
|
||||
#### Measurement validity — the positive control
|
||||
|
||||
**Every benchmark and harness must assert that it performed the work it
|
||||
reports.** Completing without error is not evidence of having done
|
||||
anything: a loop whose commands are all rejected runs fast and reports a
|
||||
throughput for work that never happened.
|
||||
|
||||
Concretely, a measurement harness must, on every run:
|
||||
|
||||
- assert the unit of work produced its expected effect (events applied,
|
||||
rows written, moves accepted) — not merely that the call returned;
|
||||
- fail loudly rather than report a number when that assertion fails;
|
||||
- state the divisor used to convert raw timings into the metric's unit,
|
||||
pinned by a test so a workload change cannot silently rescale it.
|
||||
|
||||
**A number from a run that cannot prove it did the work is void** and
|
||||
must not reach an evidence file.
|
||||
|
||||
*(v1.0, from CB-WP-0001: both serious errors in the first pass were of
|
||||
exactly this shape. A JS harness reported 8.4s for 100k moves while
|
||||
every move was being rejected, and a Rust benchmark reported 9.3M
|
||||
events/s — a 93× beat — while most rounds never completed because a
|
||||
stress gate rejected one player's action. The corrected figure was 5.6×
|
||||
lower. Adversarial review caught neither; both were claims about
|
||||
numbers, and review reads prose.)*
|
||||
|
||||
#### Evidence states what it does not support
|
||||
|
||||
An evidence file that compares across runtimes, languages, or feature
|
||||
sets **names the disanalogies explicitly**, in the same section as the
|
||||
number. The reader must not have to infer that a ratio is not
|
||||
like-for-like. This is the parity-cap rule applied to the write-up:
|
||||
state the claim you will defend, and the claim you are not making.
|
||||
|
||||
---
|
||||
|
||||
## The four-dimension rubric
|
||||
|
|
@ -171,6 +222,33 @@ Which candidate leads per dimension; what none of them do well
|
|||
|
||||
---
|
||||
|
||||
## Implementation rules the first pass earned
|
||||
|
||||
These are cheap, and each exists because its absence cost something in
|
||||
CB-WP-0001. See `history/260731-inner-loop-retrospective.md`.
|
||||
|
||||
1. **No silently-ignored input.** A field that is parsed and then unused
|
||||
is a defect, not a stub. Inputs are honoured or rejected with an
|
||||
error — never dropped. *(A scenario `setup.patch` was parsed and
|
||||
discarded; every scenario using it would have tested the wrong
|
||||
initial state while passing.)*
|
||||
2. **Decisions get commands, not defaults.** A rule that requires a
|
||||
participant's choice is implemented as a command carrying that
|
||||
choice. Until it is, **nothing claims coverage of it** — no tag, no
|
||||
scenario, no acceptance row. Inventing a default to make a rule
|
||||
"done" is the failure this prevents.
|
||||
3. **Scaffolds are exercised or marked.** A scaffold's green gates are
|
||||
not evidence. Any scaffold path no test reaches is marked as
|
||||
unexercised. *(A compiling, fully-green scaffold shipped a state-hash
|
||||
that would panic on any state holding a relation.)*
|
||||
4. **Coverage gates that count tags say so.** A gate comparing rule IDs
|
||||
against `covers:` lists proves no rule is unclaimed and no claimed
|
||||
rule is invented. It does **not** prove a scenario exercises what it
|
||||
names. Wherever such a number is reported, that limit is reported
|
||||
with it.
|
||||
|
||||
---
|
||||
|
||||
## Agentic-efficiency requirements
|
||||
|
||||
The loop exists to be driven by agents. Therefore:
|
||||
|
|
@ -202,6 +280,12 @@ A capability has completed the loop when all of the following are committed:
|
|||
- [ ] specs/<Capability>.md with acceptance-metrics table
|
||||
- [ ] passing scenarios covering every numbered spec rule
|
||||
- [ ] evidence/CB-EV-NNNN with final comparison vs baseline, no `unmeasured`
|
||||
- [ ] every reported number produced by a harness with a positive
|
||||
control; any metric that could not be instrumented is recorded as
|
||||
uncomputable rather than estimated
|
||||
- [ ] every unmet metric reported as unmet, with attribution and the
|
||||
options for resolving it — a missed target is an output of the
|
||||
loop, not a reason to move the target quietly
|
||||
- [ ] retrospective note (may be one paragraph appended to the evidence
|
||||
file): what the loop itself should change
|
||||
```
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
id: CB-WP-0001
|
||||
title: "Establish the assimilate-and-surpass inner loop via the GROUND game kernel"
|
||||
status: active
|
||||
status: done
|
||||
state_hub_workstream_id: "a1b434dc-b1c6-46b5-bbd9-80a4e6b7620f"
|
||||
---
|
||||
|
||||
|
|
@ -183,7 +183,7 @@ per-task token counts were never instrumented — also a T09 input.
|
|||
|
||||
```task
|
||||
id: CB-WP-0001-T09
|
||||
status: todo
|
||||
status: done
|
||||
priority: low
|
||||
state_hub_task_id: "e99b107e-087e-49e1-96b5-67805deb242f"
|
||||
```
|
||||
|
|
@ -193,3 +193,11 @@ heavy or too thin, what the survey template missed, what metrics turned out
|
|||
to matter. Output is InnerLoop v1.0 — the process the next capability
|
||||
workplan starts from. The loop is only "established" once it has survived
|
||||
its first full pass and been corrected.
|
||||
|
||||
**Outcome:** specs/InnerLoop.md v1.0 and
|
||||
history/260731-inner-loop-retrospective.md. The adversarial review, the
|
||||
parity cap, the provisional mechanism and the ADR gate held. The gap the
|
||||
pass exposed: both serious errors were *measurement* errors and review
|
||||
caught neither, because review reads prose and these were claims about
|
||||
numbers. v1.0 adds the positive-control rule, metric feasibility and
|
||||
instrument naming, and four implementation rules.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue