clay-borg/evidence/CB-EV-0021-the-design-instrument.md

# CB-EV-0021 — the design instrument

CB-WP-0022 T07. Tier L, declaration 5 of chaos window 2 (structural L,
chaos d8=6 → no override). Closed 2026-08-05.

**Delivered:** INTENT property 4 (*Instrument*),
[ADR-0012](../decisions/ADR-0012-the-design-instrument.md) (nine
decisions), [specs/GameDesign.md](../specs/GameDesign.md) v1.0, the
register in [GroundRules.md](../specs/GroundRules.md) (14 rows),
`tools/design.py` + `make design`, and a report file in `ground-game`.

---

## 1. The headline: a separate reviewer found the survey's showcase was false

This is the first adversarial review in this repo **run by a genuinely
separate agent**. CB-RES-0006's reviewer opened by conceding it could not
be, and wrote:

> *"Treat the challenges below as a lower bound on what a genuinely
> separate reviewer would find."*

That was a testable claim. **This is the test.** The separate reviewer ran
`git log` against the survey's central example and found our own commit
had falsified it four days earlier — while the author, who wrote that
commit, quoted the dead number twice in the same workplan.

`CB-WP-0022:61` justified the whole proposal with *"GR-E01 is admissible
because 4/6/9 against 5/7/9 is a computation anyone can rerun."*
`2da19a4` had rerun it: **6/9/12**, scenario renamed `-unreachable-` →
`-reachable-`, conclusion inverted. And GR-E01 was **one of the two
findings that passed the reproduction rule**.

**Verdict on same-session review: it is not a weak form of the control, it
is a different control.** Four of the seven challenges rest on facts a
same-session reviewer had already seen and not questioned. The cost was
~70k subagent tokens; it caught a defect in flight (§2) and rewrote the
pass's central rule.

## 2. What it caught in flight

T06's payload, status `todo`, still read *"the deal count puts 4/6/9
points in play … no dataset reconciles them"* and was queued for
transmission to `ground-game`.

It would have been the **fourth** wrong premise sent downstream, after
*"12 in the file"* (a sum with no deal table) and *"SOLVE offered on a
face-down Problem"* (the wrong condition named). **It is the only one
stopped before sending.**

| wrong premise | reached ground-game? | passed an existence test? |
|---|---|---|
| *"12 in the file"* | yes | — |
| *"SOLVE on a face-down Problem"* | yes | — |
| *"4/6/9, no dataset reconciles them"* | **stopped 2026-08-05** | **yes** |

## 3. The rule the pass came to build was one clause short

The declaration's load-bearing rule was an **existence** test. Three of
three wrong premises would have passed it, and one did.

What shipped instead (GameDesign §1): a reproduction must **exist**, have
the **shape ground-game ruled** (row-level deal table, never a sum —
promoted from a T04 addendum to part of the rule), and **be able to
fail**.

**Clause 3 is the one this pass bought.** GR-E01's scenario went green
when the edition landed and the finding stayed admissible and stayed
queued, because nothing treated a passing artifact as a signal. *A green
reproduction is an alarm, not a reassurance.*

## 4. Building it produced a distinction the ADR does not have

`make design`'s first run alarmed on **U2** — and was wrong to.

U2's scenario is green **because the provisional default it documents is
implemented**, which says nothing about whether ground-game agrees.
GR-E01's was a *counterexample* that went green. **Same colour, opposite
meaning.**

So the register gained a `role` column — `counterexample` | `default` —
and only a green counterexample alarms. **A register that cannot tell them
apart either alarms constantly or never**, and this was invisible until
the metric ran against real rows. Folded back into GameDesign §1.3.

## 5. Did backfilling change the design?

**T07 asked to be suspicious if all six findings fit the first taxonomy.**
They did — five kinds absorbed everything, no sixth invented. Two reasons
that is less reassuring than it looks and one reason it is fine:

- the taxonomy was **derived from these six**, so fitting them is nearly
  circular. Its real test is the seventh finding;
- **one of the six was deleted during backfill.** *"Six provisional
  defaults"* was not entered: C3 showed it double-counted GR-E01, and
  provisional scenarios are *reproductions for* underdetermined items, not
  a finding. So the taxonomy absorbed five, not six;
- what it did do is **force the note tier to earn itself** — GR-A13 and
  GR-E03/E04 have no artifact and became notes, which is the first time
  the distinction cost anything.

## 6. The numbers, and why none of the survey's survive

**CB-RES-0007's measurements are withdrawn.** C2 dismantled the
instrument: `design-baseline.py` is a hand-maintained dict where
`n = len(FINDINGS)`; `has_reproduction` is `startswith("scenarios/")` and
never stats the file, so its own YES-control was **green against a path
`2da19a4` deleted**; the 4-day figure is two date literals subtracted
(mutated to 2020-01-01 → 2406 days, all checks still green); and
`Makefile:127` ran only `--self-test`, so the reporting path had no CI.

It is **superseded, not deleted** — it is the evidence for how a wrong
number got into a survey, and the header now says so.

Measured by `make design`, 2026-08-05, over rows that name real files:

```
findings                       12   (+2 notes)
with a resolving reproduction  3/12 = 25%   target 100%
open, lacking a reproduction   9    target 0
reproductions green while open 0    target 0
notes past 30 days             0    target 0
closed (log)                   2    [F11, F13]
```

**Two survey claims died against measurement:**

| survey said | measured |
|---|---|
| *"six of the ten already have provisional scenarios"* | **one.** `grep -lE "\bU<n>\b" scenarios/ground/*.yaml` → only U2 |
| *"2/6 = 33% reproduced"* | withdrawn; GR-E01 was double-counted and the detector was broken |

The first is the same defect class as the wrong premises — **an
uncheckable link asserted as a fact** — found inside the survey that
proposed the fix.

## 7. The register's first run found ten answers nobody had collected

The most useful thing `make design` did was not a metric.

**GROUND-WP-0002 T05 ruled all ten U-items on 2026-08-03 — every one
confirmed.** T03 confirmed five of the six provisional scenarios. The
workplan is `finished`.

CB-RES-0007 reported *"0 of 10 ruled"* on 2026-08-03, and the register
built this session initially recorded them as `reported`. **Both were two
days stale on the day they were written.**

**This is the unread-inbox failure running in the opposite direction.**
The pass was designed around *we send findings and nobody reads them*. The
symmetric failure — *they answer and we do not collect it* — was not
anticipated by the declaration, the survey, or the review, and the first
run of the instrument surfaced it immediately.

The items are `ruled`, not `applied`: the rulings confirmed our defaults,
so the rules did not move, but the scenarios still carry `provisional:
true` for settled choices. **Lifting those flags is owed and is not done**
— `make design` shows them open until it is, which is the register
working.

## 8. The engine-evolution question, as the review left it

Declared judgment: no second register, because engine evolution already
lives in `evidence/`, `decisions/`, `gates.toml` and workplans.

**The conclusion survives; the argument did not.** C5 found the "third
thing" the maintainer meant is visible in `specs/InnerLoopReference.md`
(383 lines, two sections literally headed *"moved from InnerLoop v1.6"*)
and 769 lines of `history/` retrospectives — **neither of which the
inventory named**. So the content exists and found homes somewhat by
accident.

ADR-0012 D7 records it as **argued and narrowed, not settled**, with the
revisit condition: if InnerLoopReference keeps absorbing material that is
neither a decision nor a finding, build the register.

## 9. What tier L cost against what it caught

Second full-weight L pass in this window, third in the project.

| caught | by |
|---|---|
| the central example was false | **only** the separate reviewer |
| a wrong premise in flight to ground-game | the same |
| the baseline instrument counts itself | the same |
| the incumbent register was never evaluated | the same |
| `role`: green default ≠ green counterexample | **building it**, not reviewing it |
| ten uncollected rulings | **running it**, not building it |

**Two of six were caught by execution, not by process** — the pattern
InnerLoop §Design goal predicts (cheap correction beats exhaustive
prevention), and an argument against front-loading more review rather than
less.

**The one number that matters for the loop's own calibration:** the review
step is what distinguished this pass from a competent one. Without it the
pass ships a rule justified by a false example, sends a fourth bad premise
downstream, and builds `make design` on a dict that counts itself. The
correct reading of CB-RES-0006's fidelity note is **not** *"our reviews
were somewhat weaker"* — it is *"our reviews were not performing this
function at all."*

## 10. Cost

CB-WP-0021's cost, by re-running the instrument: see `make cost` /
`tools/cb-cost.py`. **Not quoted here as a literal** — §Single source of
fact, and this pass has just spent itself proving what typed numbers do.

## Open after this pass

- **U1–U10 → `applied`**: lift `provisional: true` where a ruling confirmed
  the default, and record each ruling. Nine items of bookkeeping.
- **The U-item ↔ scenario mapping is unwritten** for 9 of 10. Cheap, and
  it is what makes the register checkable.
- **F12 and F14 are notes** and expire 2026-08-31. GROUND-WP-0003's
  playtest closes F14 and produces its artifact via `cb-play --record`.
- **The taxonomy's real test is the seventh finding**, not these six.