clay-borg/research/CB-RES-0007-design-instrument.md

---
id: CB-RES-0007
capability: design.findings
status: draft — awaiting adversarial review (CB-WP-0022 T02)
tier: L
chaos: d8 = 6 → no override
---

# CB-RES-0007 — how rigorous rule systems record the ambiguity they find

CB-WP-0022 T01. Surveyed 2026-08-03.

**Not a survey of issue trackers.** The question is narrower: when a
system is formal enough to *notice* that its rules do not decide
something, what does it do with that observation? Three practices, and
ourselves.

---

## 1. The baseline is us, and it is measured

The external candidates are **practices, not runnable software**, so per
InnerLoop Step 1 their rows are **directional and cap at `parity`**. The
row that *can* be run is our own, and it is the one that matters, because
the register has to beat what we already do.

`tools/` harness output, 2026-08-03:

```
  findings                      6
  with a runnable reproduction  2/6 = 33%
  distinct files holding them   11
  single register?              NO — 11 files, no index

  U1..U10: raised 2026-07-30, first READ 2026-08-03 — 4 days
  U1..U10: answered? NO — 4+ days open, 0 of 10 ruled
```

| finding | locations | reproduction |
|---|---:|---|
| U1–U10 underdetermined points | 1 | — |
| SOLVE on a face-down Problem | 2 | — |
| GR-A13 wasted SOLVE | 1 | — |
| GR-E01 unreachable below 5 seats | 3 | **yes** (scenario) |
| six provisional defaults | 6 | **yes** (scenarios) |
| GR-E03/GR-E04 never played | 1 | — |

### The uncomfortable number, stated before anyone else finds it

**The rule the declaration proposed — *no finding without its
reproduction* — would reject four of our six existing findings.**

That is exactly the objection T02 is instructed to press, and the survey's
job is to answer it rather than route around it. The answer is that
**none of the four is expensive to reproduce**:

- *SOLVE on a face-down Problem* — a scenario selecting SOLVE on a
  face-down Problem and asserting no claim follows. The engine already
  produced the behaviour; nobody wrote it down.
- *GR-A13 wasted SOLVE* — the same shape.
- *GR-E03/E04 never played* — a bot game to completion in each mode,
  which `bot::play` already does.
- *U1–U10* — six of the ten already have provisional scenarios; the
  remaining four need the same treatment.

So the 33% is not evidence that reproductions are costly. **It is evidence
that nobody was ever asked for one**, which is the defect the register
exists to fix and not an argument against the rule.

## 2. Magic: the correction lands in the source, and the ruling is not authoritative

The most instructive practice, and it **corrected an assumption this pass
was about to build on**. I expected rulings to be the authoritative
resolution of an ambiguity. They are not.

The system is three layers, not two:

| layer | what it is | authoritative? |
|---|---|---|
| Comprehensive Rules | the general rules | yes |
| **Oracle** | the current authoritative text of each *card* — errata folded **in** | **yes** |
| Rulings (via Gatherer) | judge annotations attached to a card | **no** — *"reminder information with no actual weight or rules meaning"* |

When a card *"causes confusion, has outdated terminology, or doesn't work
as intended, the Oracle text may be updated"* — the fix goes **into the
authoritative text**, and the ruling layer stays explanatory.

**The property to steal: a finding closes when the source changes, not
when an annotation is added.** A register that accumulates permanent
rulings is a system that has given up on fixing its rules. Ours should be
a **queue that empties**, not an archive that grows.

This directly shapes the lifecycle in T03: `applied` must mean *the rules
text or the dataset changed and our provisional default was deleted* — not
*a ruling was recorded*.

*Directional, cited-only: the practice is described, not benchmarked.*

## 3. Model checkers: the counterexample IS the finding

A model checker does not report *"this property may not hold."* It emits a
**counterexample trace** — a concrete execution that violates the claim,
replayable by the tool that produced it.

That is precisely the shape the declaration proposed, arrived at
independently, and it is the strongest argument for the reproduction rule:
in formal methods a claim without a trace is not a result, it is a
hypothesis. Our `gr-e01-threshold-unreachable-2p` scenario is a
counterexample in exactly this sense — it is a failing-in-fact artifact
rather than a paragraph, and CB-EV-0005 already refused to delete it for
that reason.

**Property to beat: 100% of findings carry a replayable artifact.**

*Directional, cited-only.*

## 4. W3C / WHATWG: naming the undecided so it cannot be silently decided

Specifications mark points as *implementation-defined* or
*implementation-dependent*, which does two things at once: it tells an
implementer they may choose, and it tells the *spec* that a choice is
outstanding. The mark is machine-findable and survives revisions.

We already have this and it works: `provisional: true` +
`provisional_owner` on a scenario, surfaced by `make coverage` with an
age. It is the one piece of the machinery that is not missing.

**The register must reuse it, not compete with it.** Six of our findings
already live there; a second mechanism would immediately disagree with the
first.

*Directional, cited-only.*

## 5. Benchmarks to beat

| dimension | today | benchmark |
|---|---|---|
| **findability** | 11 files, no index | **one register with an index**; every finding reachable from one command |
| **reproducibility** | **2/6 = 33%** | **100%**, with withdrawn findings kept in the denominator |
| **closure** | 0 of 10 U-items ruled in 4+ days | a finding closes only when **the source changed** (Magic's Oracle property), and the register says which commit |
| **time raised → read** | **4 days** | the number this exists to fix; any mechanism that does not move it has failed |
| **taxonomy fits reality** | untested | all six existing findings expressible **without** a new kind being invented during backfill |

## 6. What the survey did not settle

- **Whether a finding without a reproduction is rejected or admitted as a
  note.** §1 shows the rule is affordable for our six, but that is n=6 and
  all six are *engine-surfaced*. A finding from **play** — *"the DARVO
  sequence feels punishing at 2 players"* — may be real and have no cheap
  artifact. T03 must decide, and the honest options are a `note` tier that
  can never be reported as a finding, or refusal.
- **Where the register lives.** A finding is about *ground-game's* rules
  but is produced by *clay-borg*. Putting it only in clay-borg repeats the
  unread-inbox failure; putting it only in ground-game separates it from
  its reproduction. Not decided here.
- **Whether the engine-evolution register is redundant.** The
  declaration's judgment is that it is. This survey found nothing that
  bears on it either way, which is itself worth saying: the practices
  above are all about *rules*, and none of them is about how the tool that
  found the problem evolved.