clay-borg/history/260805-design-instrument-challenge.md

260805 — challenge to CB-RES-0007

Adversarial review, one round, per InnerLoop §Step 2. Target: the survey research/CB-RES-0007-design-instrument.md, the harness tools/design-baseline.py, and the §judgment in CB-WP-0022.

Fidelity note, first. Unlike 260802-render-port-challenge.md, this review was run in a separate agent session with no access to the author's reasoning trail — only the files. So it has the property §Step 2 asks for. It does not have a different sample: there is exactly one baseline harness and one repo, so every number below was re-derived from the author's instrument. Per §Step 2 that is stated rather than reported as a clean verify. What this review substitutes for a second sample is mutation: each quoted number was traced to the assertion that produces it and the assertion was inverted. Four of the five headline numbers do not move when the world they claim to measure moves.

Ranked. C1 and C2 land hardest. C6 and C7 are marked weak — do not spend the one response round on them.

---

C1 — The rule's own showcase example is a false finding that shipped a green reproduction

This is the challenge that lands, and it is worse than the two cases the task description already knows about.

CB-WP-0022:61 states the admissibility rule and then gives its canonical example:

"GR-E01 is admissible because 4/6/9 against 5/7/9 is a computation anyone can rerun; the SOLVE inertness is admissible because a recorded session shows three no-ops."

Both halves of that sentence are now known to be wrong.

• The SOLVE half: CB-WP-0023:87 — "the reported case was not the one I reported… validate already rejected face-down, so it never was offered." Already conceded by the author.
• The GR-E01 half has not been conceded, and it is the more damaging one. 4/6/9 is stale. Commit 2da19a4 (CB-WP-0021, 2026-08-04) measured 6/9/12 against thresholds 5/7/9, ADR-0011:110 records the ruled 6/9/12, and the scenario was renamed gr-e01-threshold-**unreachable**-2p.yaml → gr-e01-threshold-**reachable**-2p.yaml. The finding did not get refined; its conclusion inverted.

Now the part that breaks the proposal's central claim. GR-E01 is one of the two findings in the survey's 2/6 = 33% with a runnable reproduction (design-baseline.py:26-29). It is a success case of the reproduction rule. It carried a scenario. The scenario was green. And the scenario asserted the wrong conclusion for four days, was reported to ground-game on that basis, and was renamed only when someone re-did the arithmetic by hand.

So the count is not two bad premises. It is three, and the third one is the exemplar the workplan uses to define what a good finding looks like.

The survey's §3 argument — "in formal methods a claim without a trace is not a result, it is a hypothesis" — is true and irrelevant here. A model checker's counterexample is generated by the checker from the property; it cannot disagree with the property, because it is the property's witness. Our scenarios are hand-authored assertions of an expected value. A hand-written scenario that encodes a wrong expectation is a trace that witnesses the author's belief, not the system's behaviour. §3 imports the prestige of counterexample traces onto an artifact class that does not have their guarantee, and it does so in the section that supplies the whole justification for the rule.

Required: the survey cannot claim the reproduction rule is the load- bearing control while its own two positive instances of that rule both described the wrong thing. Either (a) name the additional control that would have caught GR-E01 — the ground-game row-level deal table requirement is the obvious candidate and it is a shape requirement, not an existence one — and demote "carries its reproduction" from the rule to half the rule, or (b) withdraw §3's claim that a reproduction is sufficient to make a finding admissible.

Secondary, and checkable: 4/6/9 is still live in three places — workplans/CB-WP-0022:61, workplans/CB-WP-0022:283 (task T06, still todo, still planning to report the stale number to ground-game), and evidence/CB-EV-0018:153. The pass that exists to stop wrong premises reaching ground-game currently has a wrong premise queued for transmission to ground-game. grep -rn "4/6/9" --include=*.md . settles it.

C2 — The baseline is a hand-maintained list reporting on itself, and four of its five numbers are mutation-proof

The survey's §1 heading is "The baseline is us, and it is measured". It is not measured. It is a dict literal at tools/design-baseline.py:16-36 with one globbed row, printed back with arithmetic.

Per §Step 2 (v1.4), each number was traced to its assertion and mutated.

(a) findings 6 is len(FINDINGS) (:89) — the number of keys typed into a dict at :16-36. It is a count of hardcoded rows, not of findings. The self-test that "pins" it (:58) asserts len(FINDINGS) >= 6 against the same literal. Nothing in the repo can change this number; only editing line 16-36 can.

(b) 2/6 = 33% rests on has_reproduction() (:38-43), which is a string prefix test on p.startswith("scenarios/"). It never touches the filesystem. Mutation:

has_reproduction(['scenarios/COMPLETELY-MADE-UP.yaml']) -> True
has_reproduction(['scenarios/'])                        -> True
has_reproduction(['crates/cb-engine/tests/foo.rs'])     -> False

Two consequences. First, the self-test's YES-control at :63 cites scenarios/ground/gr-e01-threshold-unreachable-2p.yaml — a file that no longer exists (renamed in 2da19a4; ls scenarios/ground/ | grep e01). It passes anyway, because the detector never looks. This is exactly the render-port C6 class: a control that cannot distinguish "found a reproduction" from "found a string". The every cited location exists check at :57 iterates FINDINGS.values() only and never sees the self-test's own literal, so the one guard that would have caught it is pointed at the wrong list.

Second, the detector contradicts the rule it is measuring. CB-WP-0022:55-56 admits "a scenario that fails, an arithmetic check that prints the contradiction, a recorded game the reader can replay, or a named test." has_reproduction recognises exactly one of those four. A finding backed by a named Rust test scores 0. The 33% is therefore not the baseline for the proposed rule; it is the baseline for a narrower rule nobody proposed, and it is biased downward, which flatters the "nobody was ever asked for one" story in §1.

(c) 4 days is datetime.date(2026, 8, 3) - datetime.date(2026, 7, 30) — two literals on :96-97, subtracted on :98. No data source. Mutated :96 to 2020-01-01:

U1..U10: raised 2020-01-01, first READ 2026-08-03 — 2406 days
--self-test: all 5 checks [ok]

The headline number can be set to anything and every control stays green. Worse, it disagrees with the repo: specs/GroundRules.md:224 states "Owner: ground-game. Raised: 2026-07-31", and scenarios/ground/gr-a04-bond-support.yaml:9 carries provisional_raised: 2026-07-31. Under the spec's own date the interval is 3 days, not 4. Two sources of the same fact, disagreeing, with zero <!-- fact: --> tags in either the survey or the workplan (grep -c "fact:" returns 0 for both) — the drift surface InnerLoop §Single-source-of-fact names explicitly.

4 days is also the row the survey's §5 calls "the number this exists to fix; any mechanism that does not move it has failed". It is the least defensible number in the document.

(d) 11 files is len(places) — the deduplicated union of the hardcoded paths. It is derived, so it is the healthiest of the four, but it measures citation sites the author typed, not files holding findings. Add a citation, the "findability" benchmark worsens; nothing about the repo changed.

(e) The one number with real provenance is the 5-element glob at :30-34 — and see C3, it disagrees with the label above it.

What the harness would report if the work silently stopped (§Step 2 requires this be stated): the reporting path, :80-99, performs no existence check at all. Delete every file in FINDINGS and it still prints 6 findings, 2/6 = 33%, 11 files, 4 days. The existence guard lives only in --self-test, and Makefile:127 runs only --self-test — the numbers quoted in the survey come from a code path CI never executes.

Required: either the survey stops calling §1 "measured" and labels the baseline what it is — a hand-curated snapshot, which is a legitimate thing to have but caps at the same directional status the survey correctly assigns to the external rows — or the harness derives its finding set from the repo (the provisional: marks, evidence/ frontmatter, the U-list) so that mutating the repo moves the number.

C3 — "six provisional defaults" is five, and the survey's own instrument says so

research/CB-RES-0007:46 lists six provisional defaults | 6 | yes.

$ python3 tools/design-baseline.py
  repro  5 location(s)  six provisional defaults
$ make coverage
provisional U-item defaults: 5

Six became five in 2da19a4 when gr-e01-threshold-unreachable-2p.yaml lost its provisional: true (verified: git grep -l "provisional: true" 2da19a4^ -- scenarios/ground/ | wc -l → 6; at HEAD → 5). The author edited design-baseline.py:22-29 in that same window to add the [RESOLVED] comment on the GR-E01 row, and did not notice that the next row's glob had silently dropped a member and its hardcoded label "six provisional defaults" had gone stale.

Two things follow.

1. The survey's table row 5 (locations: 6) is a number the survey's own tool contradicts on stdout, and no self-test compares the word "six" to len(). §4 of the survey argues the register "must reuse [the provisional machinery], not compete with it" — and the very row that reads the provisional machinery is the one that drifted from it.
2. GR-E01 was double-counted. It is row 4 and was a member of row 5's six. The denominator of 2/6 contained the same finding twice, and one of the two numerator hits was that finding.

Required: correct the row, and state whether the 2/6 survives the double-count. If rows 4 and 5 overlap, the honest denominator at survey time was 5 distinct findings, and the headline rate changes.

C4 — GroundRules.md §Underdetermined was never evaluated as the incumbent, and it already implements four of the five benchmarks

T02 requires an attempt at "whether a register is needed at all, rather than one more section in GroundRules.md §Underdetermined." The survey does not attempt it. §4 mentions the provisional machinery only as machinery to reuse; §5's benchmark table has a today column that never names §Underdetermined; and §6's "did not settle" list does not include it. Per §Step 2's row for surveys, that is an omitted candidate, and it is the strongest one, because it is the incumbent.

Read against the survey's own five benchmarks (specs/GroundRules.md:222-277):

benchmark (§5)	§Underdetermined today
findability: one register with an index	has it — one section, ten numbered items, grep U7
reproducibility: runnable artifact	has the obligation — "every U-item's default must be covered by a scenario tagged provisional: true" (:274-277)
closure: closes when the source changed	has it — "A ruling flips the scenario, not the kernel; a correction is a data change" (:231-233). This is the Magic Oracle property the survey went to Magic to discover. It has been in the spec since 2026-07-31.
raised → read, with age	has it, executable — owner + raise date per item, make coverage prints age=5d per item and warns past 30 days
taxonomy fits reality	not addressed — genuinely absent

The survey travels to Magic: the Gathering to extract "a finding closes when the source changes, not when an annotation is added" and presents it as a corrected assumption (§2, and CB-WP-0022:136-139 treats it as the pass's main external yield). The repo already states that property in those terms, one directory over. That is not a small omission: it is the survey's headline external finding being rediscovered from the incumbent it failed to survey.

Where this challenge stops, honestly. §Underdetermined has a real gap and the review found it: it obliges ten U-items to carry provisional scenarios and five exist. make coverage prints provisional U-item defaults: 5 and does not flag the five missing — it reports what exists, never what is owed. That is a genuine argument for something with a denominator, and it is stronger than any argument the survey actually makes. It is also an argument for one gate in rule-coverage.py, not for a new register.

Required: evaluate §Underdetermined as a candidate row with a today column, and state what a register does that §Underdetermined + one completeness check cannot. If the answer is "holds findings that are not U-items" — say which of the six, and note that four of the six are single-file prose notes in evidence/.

C5 — The engine-evolution "third thing" exists, and its homelessness is visible in the filesystem

CB-WP-0022:73-84 argues the engine-evolution register is redundant because "evidence/CB-EV-* records what each pass found, decisions/ADR-* records what was decided…, gates.toml records what every control has caught…, and workplans record what was attempted."

The list is incomplete, and the omission is the whole question. The third thing — what we learned about building engines — is specs/InnerLoopReference.md, 383 lines, and its section list is nothing but that:

§ Implementation rules the first pass earned
§ Why review is adversarial execution, not re-derivation
§ Numbers you inherit — two cases from CB-WP-0002
§ Quoting a cost — six for six, never high
§ Chaos roll — window 1's verdict and the d4 → d8 change
§ Retargeting — the case, and the four corrections it must not forbid

None of those is a decision (ADR), a per-pass finding (evidence), or a control (gates.toml). They are cross-pass lessons about method. Plus six history/*-retrospective.md files, 769 lines, in the same category.

And the homelessness is on the record in the headings themselves. Two sections are literally titled "(moved from InnerLoop v1.6)" — the content grew inside the normative spec until it had to be evicted into an appendix-shaped file that is not in the author's list of homes. That is precisely the signature T02 asked the reviewer to look for: content that had nowhere to go.

Where this challenge stops. It does not establish that a register is needed. It establishes that the author's redundancy argument is made against the wrong inventory. InnerLoopReference.md and history/ are a home, so the conclusion "no second register" probably survives — but it survives for a different reason than the one given, and the difference matters for T03: the asymmetry claimed at :83 ("engine evolution has a home and game design does not") is real, but the home is an unindexed 383-line spec appendix plus six retrospectives, i.e. exactly the 11 files, no index condition the survey calls a defect when it is game findings. The survey applies one standard to game findings and another to engine lessons and does not state the distinction.

Required: name InnerLoopReference.md and history/*-retrospective.md in the redundancy argument, and say why "several files, no index" is disqualifying for one subject and acceptable for the other.

C6 — (weak) The survey's own §2 conclusion contradicts the harness's stated design

§2: "A register that accumulates permanent rulings is a system that has given up on fixing its rules. Ours should be a queue that empties, not an archive that grows."

design-baseline.py:23-25: "Kept in the baseline because… a register that drops findings when they close cannot report a close rate."

Both are defensible and they are not strictly contradictory — a queue can retain closed items out-of-band. But T03 has to pick, and the two artifacts of this pass currently point opposite ways, with the tool's comment being the more considered of the two. Marked weak: this is a clarification, not a defect.

C7 — (weak) n=6, all engine-surfaced, and §6 already concedes it

§6 concedes the sample is six and all engine-surfaced, and flags the play-sourced finding ("DARVO feels punishing at 2 players") as unsettled. That concession is correct and I have nothing to add to it, so this is not a challenge — it is noted only so the response does not spend a round defending something already conceded. Marked weak.

---

Verdict

Not approvable as written. C1, C2 and C3 each require a change to the survey, not a clarification. C4 requires an added candidate row. C5 requires a corrected inventory in the workplan's §judgment.

#	verdict
C1	lands hardest. Requires demoting "carries its reproduction" from sufficient to necessary-but-not-sufficient, and adding the shape requirement. GR-E01 is a rule-satisfying finding that was wrong.
C2	lands. Four of five headline numbers do not move under mutation; 4 days is a literal and disagrees with GroundRules.md:224 (3 days). §1's word "measured" must go or the harness must derive.
C3	lands. "six" is five; the survey's table contradicts its own tool's stdout; GR-E01 double-counted in the 2/6 denominator.
C4	lands, moderate. The incumbent was never surveyed, and it already holds the Magic Oracle property the survey travelled to Magic to find. Weakened by a real gap the review found for the author: 10 U-items owe scenarios, 5 exist.
C5	partial. The third thing exists and is visibly homeless; the redundancy conclusion probably survives, the argument does not.
C6	weak. Clarification for T03.
C7	weak / already conceded.

What survives. Two claims attacked and not broken:

• That the reproduction rule is affordable. §1's four cheap reproductions were checked and the estimates are plausible; bot::play exists, the scenario harness exists, and the SOLVE case in fact got its reproduction in CB-WP-0023 T01 for a fraction of a pass. What would have falsified it: a finding among the four whose reproduction needed engine work that does not exist. There is none. Affordability is not the problem; sufficiency is (C1).
• That the register must reuse the provisional machinery rather than compete with it (§4). Checked against rule-coverage.py:98-108,285-308 — owner, raise date, age and a warn threshold all already exist and are executable. What would have falsified it: a second age-tracking mechanism already in the repo. There is none.

The single challenge that forces a change to the design: C1. The proposal's load-bearing rule is stated as an existence test on a reproduction. Three findings have now reached the record on wrong premises, and the third one satisfied the existence test. The rule as written does not catch its own worst observed failure. T03 cannot adopt it in the form CB-WP-0022:53 states it.