clay-borg/history/260805-design-instrument-challenge.md

# 260805 — challenge to CB-RES-0007

Adversarial review, one round, per InnerLoop §Step 2. Target: the survey
`research/CB-RES-0007-design-instrument.md`, the harness
`tools/design-baseline.py`, and the §judgment in `CB-WP-0022`.

**Fidelity note, first.** Unlike `260802-render-port-challenge.md`, this
review *was* run in a separate agent session with no access to the author's
reasoning trail — only the files. So it has the property §Step 2 asks for.
It does **not** have a different sample: there is exactly one baseline
harness and one repo, so every number below was re-derived from the
author's instrument. Per §Step 2 that is stated rather than reported as a
clean verify. What this review substitutes for a second sample is
**mutation**: each quoted number was traced to the assertion that produces
it and the assertion was inverted. Four of the five headline numbers do not
move when the world they claim to measure moves.

Ranked. C1 and C2 land hardest. C6 and C7 are marked weak — do not spend
the one response round on them.

---

## C1 — The rule's own showcase example is a false finding that shipped a green reproduction

**This is the challenge that lands, and it is worse than the two cases the
task description already knows about.**

`CB-WP-0022:61` states the admissibility rule and then gives its canonical
example:

> *"**GR-E01 is admissible because 4/6/9 against 5/7/9 is a computation
> anyone can rerun**; the SOLVE inertness is admissible because a recorded
> session shows three no-ops."*

Both halves of that sentence are now known to be wrong.

- The SOLVE half: `CB-WP-0023:87` — *"the reported case was not the one I
  reported… `validate` already rejected face-down, so it never was
  offered."* Already conceded by the author.
- **The GR-E01 half has not been conceded, and it is the more damaging
  one.** `4/6/9` is stale. Commit `2da19a4` (CB-WP-0021, 2026-08-04)
  measured **6/9/12** against thresholds 5/7/9, `ADR-0011:110` records the
  ruled 6/9/12, and the scenario was renamed
  `gr-e01-threshold-**unreachable**-2p.yaml` →
  `gr-e01-threshold-**reachable**-2p.yaml`. The finding did not get
  refined; **its conclusion inverted**.

Now the part that breaks the proposal's central claim. GR-E01 is one of the
**two** findings in the survey's `2/6 = 33% with a runnable reproduction`
(`design-baseline.py:26-29`). It is a *success case* of the reproduction
rule. It carried a scenario. The scenario was green. And the scenario
asserted the wrong conclusion for four days, was reported to ground-game on
that basis, and was renamed only when someone re-did the arithmetic by
hand.

So the count is not two bad premises. **It is three, and the third one is
the exemplar the workplan uses to define what a good finding looks like.**

The survey's §3 argument — *"in formal methods a claim without a trace is
not a result, it is a hypothesis"* — is true and irrelevant here. A model
checker's counterexample is generated *by* the checker from the property;
it cannot disagree with the property, because it is the property's witness.
Our scenarios are **hand-authored assertions of an expected value**. A
hand-written scenario that encodes a wrong expectation is a trace that
witnesses the author's belief, not the system's behaviour. §3 imports the
prestige of counterexample traces onto an artifact class that does not have
their guarantee, and it does so in the section that supplies the whole
justification for the rule.

**Required:** the survey cannot claim the reproduction rule is the load-
bearing control while its own two positive instances of that rule both
described the wrong thing. Either (a) name the *additional* control that
would have caught GR-E01 — the ground-game row-level deal table
requirement is the obvious candidate and it is a **shape** requirement, not
an existence one — and demote "carries its reproduction" from *the* rule to
*half* the rule, or (b) withdraw §3's claim that a reproduction is
sufficient to make a finding admissible.

Secondary, and checkable: `4/6/9` is still live in three places —
`workplans/CB-WP-0022:61`, `workplans/CB-WP-0022:283` (task T06, still
`todo`, still planning to report the stale number to ground-game), and
`evidence/CB-EV-0018:153`. **The pass that exists to stop wrong premises
reaching ground-game currently has a wrong premise queued for transmission
to ground-game.** `grep -rn "4/6/9" --include=*.md .` settles it.

## C2 — The baseline is a hand-maintained list reporting on itself, and four of its five numbers are mutation-proof

The survey's §1 heading is *"The baseline is us, and **it is measured**"*.
It is not measured. It is a dict literal at `tools/design-baseline.py:16-36`
with one globbed row, printed back with arithmetic.

Per §Step 2 (v1.4), each number was traced to its assertion and mutated.

**(a) `findings 6`** is `len(FINDINGS)` (`:89`) — the number of keys typed
into a dict at `:16-36`. It is a count of hardcoded rows, not of findings.
The self-test that "pins" it (`:58`) asserts `len(FINDINGS) >= 6` against
the same literal. Nothing in the repo can change this number; only editing
line 16-36 can.

**(b) `2/6 = 33%`** rests on `has_reproduction()` (`:38-43`), which is a
**string prefix test on `p.startswith("scenarios/")`**. It never touches
the filesystem. Mutation:

```
has_reproduction(['scenarios/COMPLETELY-MADE-UP.yaml']) -> True
has_reproduction(['scenarios/'])                        -> True
has_reproduction(['crates/cb-engine/tests/foo.rs'])     -> False
```

Two consequences. First, the self-test's YES-control at `:63` cites
`scenarios/ground/gr-e01-threshold-unreachable-2p.yaml` — **a file that no
longer exists** (renamed in `2da19a4`; `ls scenarios/ground/ | grep e01`).
It passes anyway, because the detector never looks. This is exactly the
render-port C6 class: a control that cannot distinguish "found a
reproduction" from "found a string". The `every cited location exists`
check at `:57` iterates `FINDINGS.values()` only and never sees the
self-test's own literal, so the one guard that would have caught it is
pointed at the wrong list.

Second, the detector **contradicts the rule it is measuring**. `CB-WP-0022:55-56`
admits *"a scenario that fails, an arithmetic check that prints the
contradiction, a recorded game the reader can replay, or **a named test**."*
`has_reproduction` recognises exactly one of those four. A finding backed by
a named Rust test scores 0. The 33% is therefore not the baseline for the
proposed rule; it is the baseline for a narrower rule nobody proposed, and
it is biased **downward**, which flatters the "nobody was ever asked for
one" story in §1.

**(c) `4 days`** is `datetime.date(2026, 8, 3) - datetime.date(2026, 7, 30)`
— two literals on `:96-97`, subtracted on `:98`. No data source. Mutated
`:96` to `2020-01-01`:

```
U1..U10: raised 2020-01-01, first READ 2026-08-03 — 2406 days
--self-test: all 5 checks [ok]
```

The headline number can be set to anything and every control stays green.
Worse, **it disagrees with the repo**: `specs/GroundRules.md:224` states
*"Owner: ground-game. **Raised: 2026-07-31**"*, and
`scenarios/ground/gr-a04-bond-support.yaml:9` carries
`provisional_raised: 2026-07-31`. Under the spec's own date the interval is
**3 days, not 4**. Two sources of the same fact, disagreeing, with zero
`<!-- fact: -->` tags in either the survey or the workplan
(`grep -c "fact:"` returns 0 for both) — the drift surface InnerLoop
§Single-source-of-fact names explicitly.

`4 days` is also the row the survey's §5 calls *"the number this exists to
fix; any mechanism that does not move it has failed"*. It is the least
defensible number in the document.

**(d) `11 files`** is `len(places)` — the deduplicated union of the
hardcoded paths. It is derived, so it is the healthiest of the four, but it
measures **citation sites the author typed**, not files holding findings.
Add a citation, the "findability" benchmark worsens; nothing about the
repo changed.

**(e)** The one number with real provenance is the 5-element glob at
`:30-34` — and see C3, it disagrees with the label above it.

**What the harness would report if the work silently stopped** (§Step 2
requires this be stated): the reporting path, `:80-99`, performs **no
existence check at all**. Delete every file in `FINDINGS` and it still
prints `6 findings, 2/6 = 33%, 11 files, 4 days`. The existence guard lives
only in `--self-test`, and `Makefile:127` runs **only** `--self-test` —
the numbers quoted in the survey come from a code path CI never executes.

**Required:** either the survey stops calling §1 "measured" and labels the
baseline what it is — a hand-curated snapshot, which is a legitimate thing
to have but caps at the same directional status the survey correctly
assigns to the external rows — or the harness derives its finding set from
the repo (the `provisional:` marks, `evidence/` frontmatter, the U-list)
so that mutating the repo moves the number.

## C3 — "six provisional defaults" is five, and the survey's own instrument says so

`research/CB-RES-0007:46` lists `six provisional defaults | 6 | yes`.

```
$ python3 tools/design-baseline.py
  repro  5 location(s)  six provisional defaults
$ make coverage
provisional U-item defaults: 5
```

Six became five in `2da19a4` when `gr-e01-threshold-unreachable-2p.yaml`
lost its `provisional: true` (verified:
`git grep -l "provisional: true" 2da19a4^ -- scenarios/ground/ | wc -l` →
**6**; at HEAD → **5**). The author edited `design-baseline.py:22-29` in
that same window to add the `[RESOLVED]` comment on the GR-E01 row, and did
not notice that the *next* row's glob had silently dropped a member and its
hardcoded label `"six provisional defaults"` had gone stale.

Two things follow.

1. **The survey's table row 5 (`locations: 6`) is a number the survey's own
   tool contradicts on stdout**, and no self-test compares the word "six"
   to `len()`. §4 of the survey argues the register *"must reuse [the
   provisional machinery], not compete with it"* — and the very row that
   reads the provisional machinery is the one that drifted from it.
2. **GR-E01 was double-counted.** It is row 4 *and* was a member of row 5's
   six. The denominator of `2/6` contained the same finding twice, and one
   of the two numerator hits was that finding.

**Required:** correct the row, and state whether the `2/6` survives the
double-count. If rows 4 and 5 overlap, the honest denominator at survey
time was 5 distinct findings, and the headline rate changes.

## C4 — `GroundRules.md §Underdetermined` was never evaluated as the incumbent, and it already implements four of the five benchmarks

T02 requires an attempt at *"whether a register is needed at all, rather
than one more section in `GroundRules.md §Underdetermined`."* The survey
does not attempt it. §4 mentions the provisional machinery only as
*machinery to reuse*; §5's benchmark table has a `today` column that never
names §Underdetermined; and §6's "did not settle" list does not include it.
Per §Step 2's row for surveys, **that is an omitted candidate, and it is
the strongest one, because it is the incumbent.**

Read against the survey's own five benchmarks (`specs/GroundRules.md:222-277`):

| benchmark (§5) | §Underdetermined today |
|---|---|
| findability: *one register with an index* | **has it** — one section, ten numbered items, `grep U7` |
| reproducibility: *runnable artifact* | **has the obligation** — *"every U-item's default must be covered by a scenario tagged `provisional: true`"* (`:274-277`) |
| closure: *closes when the source changed* | **has it** — *"A ruling flips the **scenario**, not the kernel; a correction is a data change"* (`:231-233`). This is the Magic Oracle property the survey went to Magic to discover. It has been in the spec since 2026-07-31. |
| raised → read, with age | **has it, executable** — owner + raise date per item, `make coverage` prints `age=5d` per item and warns past 30 days |
| taxonomy fits reality | not addressed — genuinely absent |

The survey travels to Magic: the Gathering to extract *"a finding closes
when the source changes, not when an annotation is added"* and presents it
as a corrected assumption (§2, and `CB-WP-0022:136-139` treats it as the
pass's main external yield). **The repo already states that property in
those terms, one directory over.** That is not a small omission: it is the
survey's headline external finding being rediscovered from the incumbent it
failed to survey.

**Where this challenge stops, honestly.** §Underdetermined has a real gap
and the review found it: it obliges ten U-items to carry provisional
scenarios and **five exist**. `make coverage` prints `provisional U-item
defaults: 5` and does *not* flag the five missing — it reports what exists,
never what is owed. That is a genuine argument for something with a
denominator, and it is stronger than any argument the survey actually
makes. It is also an argument for *one gate in `rule-coverage.py`*, not for
a new register.

**Required:** evaluate §Underdetermined as a candidate row with a `today`
column, and state what a register does that §Underdetermined + one
completeness check cannot. If the answer is "holds findings that are not
U-items" — say which of the six, and note that four of the six are
single-file prose notes in `evidence/`.

## C5 — The engine-evolution "third thing" exists, and its homelessness is visible in the filesystem

`CB-WP-0022:73-84` argues the engine-evolution register is redundant
because *"`evidence/CB-EV-*` records what each pass found, `decisions/ADR-*`
records what was decided…, `gates.toml` records what every control has
caught…, and workplans record what was attempted."*

The list is incomplete, and the omission is the whole question. The third
thing — *what we learned about building engines* — is
**`specs/InnerLoopReference.md`, 383 lines**, and its section list is
nothing but that:

```
§ Implementation rules the first pass earned
§ Why review is adversarial execution, not re-derivation
§ Numbers you inherit — two cases from CB-WP-0002
§ Quoting a cost — six for six, never high
§ Chaos roll — window 1's verdict and the d4 → d8 change
§ Retargeting — the case, and the four corrections it must not forbid
```

None of those is a decision (ADR), a per-pass finding (evidence), or a
control (gates.toml). They are cross-pass lessons about method. Plus six
`history/*-retrospective.md` files, 769 lines, in the same category.

**And the homelessness is on the record in the headings themselves.** Two
sections are literally titled *"(moved from InnerLoop v1.6)"* — the content
grew inside the normative spec until it had to be evicted into an
appendix-shaped file that is not in the author's list of homes. That is
precisely the signature T02 asked the reviewer to look for: *content that
had nowhere to go*.

**Where this challenge stops.** It does not establish that a *register* is
needed. It establishes that the author's redundancy argument is made
against the wrong inventory. `InnerLoopReference.md` and `history/` **are**
a home, so the conclusion "no second register" probably survives — but it
survives for a different reason than the one given, and the difference
matters for T03: the asymmetry claimed at `:83` (*"engine evolution has a
home and game design does not"*) is real, but the home is an unindexed
383-line spec appendix plus six retrospectives, i.e. **exactly the
`11 files, no index` condition the survey calls a defect when it is game
findings**. The survey applies one standard to game findings and another to
engine lessons and does not state the distinction.

**Required:** name `InnerLoopReference.md` and `history/*-retrospective.md`
in the redundancy argument, and say why "several files, no index" is
disqualifying for one subject and acceptable for the other.

## C6 — (weak) The survey's own §2 conclusion contradicts the harness's stated design

§2: *"A register that accumulates permanent rulings is a system that has
given up on fixing its rules. Ours should be a **queue that empties**, not
an archive that grows."*

`design-baseline.py:23-25`: *"Kept in the baseline because… a register that
drops findings when they close cannot report a close rate."*

Both are defensible and they are not strictly contradictory — a queue can
retain closed items out-of-band. But T03 has to pick, and the two artifacts
of this pass currently point opposite ways, with the tool's comment being
the more considered of the two. Marked **weak**: this is a clarification,
not a defect.

## C7 — (weak) `n=6`, all engine-surfaced, and §6 already concedes it

§6 concedes the sample is six and all engine-surfaced, and flags the
play-sourced finding (*"DARVO feels punishing at 2 players"*) as unsettled.
That concession is correct and I have nothing to add to it, so this is not
a challenge — it is noted only so the response does not spend a round
defending something already conceded. Marked **weak**.

---

## Verdict

**Not approvable as written.** C1, C2 and C3 each require a change to the
survey, not a clarification. C4 requires an added candidate row. C5
requires a corrected inventory in the workplan's §judgment.

| # | verdict |
|---|---|
| **C1** | **lands hardest.** Requires demoting "carries its reproduction" from sufficient to necessary-but-not-sufficient, and adding the shape requirement. GR-E01 is a rule-satisfying finding that was wrong. |
| **C2** | **lands.** Four of five headline numbers do not move under mutation; `4 days` is a literal and disagrees with `GroundRules.md:224` (3 days). §1's word "measured" must go or the harness must derive. |
| **C3** | **lands.** "six" is five; the survey's table contradicts its own tool's stdout; GR-E01 double-counted in the `2/6` denominator. |
| **C4** | **lands, moderate.** The incumbent was never surveyed, and it already holds the Magic Oracle property the survey travelled to Magic to find. Weakened by a real gap the review found for the author: 10 U-items owe scenarios, 5 exist. |
| **C5** | **partial.** The third thing exists and is visibly homeless; the redundancy *conclusion* probably survives, the *argument* does not. |
| **C6** | **weak.** Clarification for T03. |
| **C7** | **weak / already conceded.** |

**What survives.** Two claims attacked and not broken:

- **That the reproduction rule is affordable.** §1's four cheap
  reproductions were checked and the estimates are plausible; `bot::play`
  exists, the scenario harness exists, and the SOLVE case in fact got its
  reproduction in CB-WP-0023 T01 for a fraction of a pass. What would have
  falsified it: a finding among the four whose reproduction needed engine
  work that does not exist. There is none. **Affordability is not the
  problem; sufficiency is (C1).**
- **That the register must reuse the provisional machinery rather than
  compete with it** (§4). Checked against `rule-coverage.py:98-108,285-308`
  — owner, raise date, age and a warn threshold all already exist and are
  executable. What would have falsified it: a second age-tracking mechanism
  already in the repo. There is none.

**The single challenge that forces a change to the design: C1.** The
proposal's load-bearing rule is stated as an existence test on a
reproduction. Three findings have now reached the record on wrong premises,
and **the third one satisfied the existence test**. The rule as written
does not catch its own worst observed failure. T03 cannot adopt it in the
form `CB-WP-0022:53` states it.