clay-borg/evidence/CB-EV-0021-the-design-instrument.md

CB-EV-0021 — the design instrument

CB-WP-0022 T07. Tier L, declaration 5 of chaos window 2 (structural L, chaos d8=6 → no override). Closed 2026-08-05.

Delivered: INTENT property 4 (Instrument), ADR-0012 (nine decisions), specs/GameDesign.md v1.0, the register in GroundRules.md (14 rows), tools/design.py + make design, and a report file in ground-game.

---

1. The headline: a separate reviewer found the survey's showcase was false

This is the first adversarial review in this repo run by a genuinely separate agent. CB-RES-0006's reviewer opened by conceding it could not be, and wrote:

"Treat the challenges below as a lower bound on what a genuinely separate reviewer would find."

That was a testable claim. This is the test. The separate reviewer ran git log against the survey's central example and found our own commit had falsified it four days earlier — while the author, who wrote that commit, quoted the dead number twice in the same workplan.

CB-WP-0022:61 justified the whole proposal with "GR-E01 is admissible because 4/6/9 against 5/7/9 is a computation anyone can rerun." 2da19a4 had rerun it: 6/9/12, scenario renamed -unreachable- → -reachable-, conclusion inverted. And GR-E01 was one of the two findings that passed the reproduction rule.

Verdict on same-session review: it is not a weak form of the control, it is a different control. Four of the seven challenges rest on facts a same-session reviewer had already seen and not questioned. The cost was ~70k subagent tokens; it caught a defect in flight (§2) and rewrote the pass's central rule.

2. What it caught in flight

T06's payload, status todo, still read "the deal count puts 4/6/9 points in play … no dataset reconciles them" and was queued for transmission to ground-game.

It would have been the fourth wrong premise sent downstream, after "12 in the file" (a sum with no deal table) and "SOLVE offered on a face-down Problem" (the wrong condition named). It is the only one stopped before sending.

wrong premise	reached ground-game?	passed an existence test?
"12 in the file"	yes	—
"SOLVE on a face-down Problem"	yes	—
"4/6/9, no dataset reconciles them"	stopped 2026-08-05	yes

3. The rule the pass came to build was one clause short

The declaration's load-bearing rule was an existence test. Three of three wrong premises would have passed it, and one did.

What shipped instead (GameDesign §1): a reproduction must exist, have the shape ground-game ruled (row-level deal table, never a sum — promoted from a T04 addendum to part of the rule), and be able to fail.

Clause 3 is the one this pass bought. GR-E01's scenario went green when the edition landed and the finding stayed admissible and stayed queued, because nothing treated a passing artifact as a signal. A green reproduction is an alarm, not a reassurance.

4. Building it produced a distinction the ADR does not have

make design's first run alarmed on U2 — and was wrong to.

U2's scenario is green because the provisional default it documents is implemented, which says nothing about whether ground-game agrees. GR-E01's was a counterexample that went green. Same colour, opposite meaning.

So the register gained a role column — counterexample | default — and only a green counterexample alarms. A register that cannot tell them apart either alarms constantly or never, and this was invisible until the metric ran against real rows. Folded back into GameDesign §1.3.

5. Did backfilling change the design?

T07 asked to be suspicious if all six findings fit the first taxonomy. They did — five kinds absorbed everything, no sixth invented. Two reasons that is less reassuring than it looks and one reason it is fine:

• the taxonomy was derived from these six, so fitting them is nearly circular. Its real test is the seventh finding;
• one of the six was deleted during backfill. "Six provisional defaults" was not entered: C3 showed it double-counted GR-E01, and provisional scenarios are reproductions for underdetermined items, not a finding. So the taxonomy absorbed five, not six;
• what it did do is force the note tier to earn itself — GR-A13 and GR-E03/E04 have no artifact and became notes, which is the first time the distinction cost anything.

6. The numbers, and why none of the survey's survive

CB-RES-0007's measurements are withdrawn. C2 dismantled the instrument: design-baseline.py is a hand-maintained dict where n = len(FINDINGS); has_reproduction is startswith("scenarios/") and never stats the file, so its own YES-control was green against a path 2da19a4 deleted; the 4-day figure is two date literals subtracted (mutated to 2020-01-01 → 2406 days, all checks still green); and Makefile:127 ran only --self-test, so the reporting path had no CI.

It is superseded, not deleted — it is the evidence for how a wrong number got into a survey, and the header now says so.

Measured by make design, 2026-08-05, over rows that name real files:

findings                       12   (+2 notes)
with a resolving reproduction  3/12 = 25%   target 100%
open, lacking a reproduction   9    target 0
reproductions green while open 0    target 0
notes past 30 days             0    target 0
closed (log)                   2    [F11, F13]

Two survey claims died against measurement:

survey said	measured
"six of the ten already have provisional scenarios"	one. grep -lE "\bU<n>\b" scenarios/ground/*.yaml → only U2
"2/6 = 33% reproduced"	withdrawn; GR-E01 was double-counted and the detector was broken

The first is the same defect class as the wrong premises — an uncheckable link asserted as a fact — found inside the survey that proposed the fix.

7. The register's first run found ten answers nobody had collected

The most useful thing make design did was not a metric.

GROUND-WP-0002 T05 ruled all ten U-items on 2026-08-03 — every one confirmed. T03 confirmed five of the six provisional scenarios. The workplan is finished.

CB-RES-0007 reported "0 of 10 ruled" on 2026-08-03, and the register built this session initially recorded them as reported. Both were two days stale on the day they were written.

This is the unread-inbox failure running in the opposite direction. The pass was designed around we send findings and nobody reads them. The symmetric failure — they answer and we do not collect it — was not anticipated by the declaration, the survey, or the review, and the first run of the instrument surfaced it immediately.

The items are ruled, not applied: the rulings confirmed our defaults, so the rules did not move, but the scenarios still carry provisional: true for settled choices. Lifting those flags is owed and is not done — make design shows them open until it is, which is the register working.

8. The engine-evolution question, as the review left it

Declared judgment: no second register, because engine evolution already lives in evidence/, decisions/, gates.toml and workplans.

The conclusion survives; the argument did not. C5 found the "third thing" the maintainer meant is visible in specs/InnerLoopReference.md (383 lines, two sections literally headed "moved from InnerLoop v1.6") and 769 lines of history/ retrospectives — neither of which the inventory named. So the content exists and found homes somewhat by accident.

ADR-0012 D7 records it as argued and narrowed, not settled, with the revisit condition: if InnerLoopReference keeps absorbing material that is neither a decision nor a finding, build the register.

9. What tier L cost against what it caught

Second full-weight L pass in this window, third in the project.

caught	by
the central example was false	only the separate reviewer
a wrong premise in flight to ground-game	the same
the baseline instrument counts itself	the same
the incumbent register was never evaluated	the same
role: green default ≠ green counterexample	building it, not reviewing it
ten uncollected rulings	running it, not building it

Two of six were caught by execution, not by process — the pattern InnerLoop §Design goal predicts (cheap correction beats exhaustive prevention), and an argument against front-loading more review rather than less.

The one number that matters for the loop's own calibration: the review step is what distinguished this pass from a competent one. Without it the pass ships a rule justified by a false example, sends a fourth bad premise downstream, and builds make design on a dict that counts itself. The correct reading of CB-RES-0006's fidelity note is not "our reviews were somewhat weaker" — it is "our reviews were not performing this function at all."

10. Cost

CB-WP-0021's cost, by re-running the instrument: see make cost / tools/cb-cost.py. Not quoted here as a literal — §Single source of fact, and this pass has just spent itself proving what typed numbers do.

Open after this pass

• U1–U10 → applied: lift provisional: true where a ruling confirmed the default, and record each ruling. Nine items of bookkeeping.
• The U-item ↔ scenario mapping is unwritten for 9 of 10. Cheap, and it is what makes the register checkable.
• F12 and F14 are notes and expire 2026-08-31. GROUND-WP-0003's playtest closes F14 and produces its artifact via cb-play --record.
• The taxonomy's real test is the seventh finding, not these six.