CB-WP-0022-T01: survey — how rule systems record the ambiguity they find

CB-RES-0007 plus a runnable baseline harness. Tier L invokes the
runnable-baseline option; the external candidates are practices rather
than software, so their rows are directional and cap at parity, and the
row that CAN be run is our own.

Measured: 6 findings across 11 files with no index, 2 of 6 (33%) with a
runnable reproduction, U1-U10 raised 2026-07-30 and first READ
2026-08-03 -- 4 days, 0 of 10 ruled.

The uncomfortable number is stated before the review can find it: the
proposed 'no finding without its reproduction' rule would reject four of
our six existing findings. The survey answers rather than routes around
it -- none of the four is expensive to reproduce, so 33% is evidence
nobody was ever asked for one.

Magic corrected an assumption this pass was about to build on. Rulings
are NOT authoritative -- they are 'reminder information with no actual
weight or rules meaning' -- and the authoritative fix folds into the
Oracle card text. So a finding closes when the SOURCE changes, not when
an annotation is added, and the register must be a queue that empties
rather than an archive that grows. That is now a constraint on the ADR's
lifecycle.

Model checkers supply the reproduction rule independently: a
counterexample trace IS the finding. W3C's implementation-defined mark is
the machinery we already have in provisional: scenarios and must reuse.

The loop-lint gate caught the new tool with no --self-test; it has one,
pinning the 2-of-6 baseline so a later edit cannot move it silently.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Makefile

@@ -119,6 +119,7 @@ self-tests:
 	$(PY) $(TOOLS)/size-metrics.py --self-test
 	$(PY) $(TOOLS)/runtime-metrics.py --self-test
 	$(PY) $(TOOLS)/replay-test.py --self-test
+	$(PY) $(TOOLS)/design-baseline.py --self-test
 
 # T01 positive control: prove the environment fix, do not assume it. Runs
 # every tool from a foreign working directory with a PATH that has no

research/CB-RES-0007-design-instrument.md

@@ -0,0 +1,160 @@
+---
+id: CB-RES-0007
+capability: design.findings
+status: draft — awaiting adversarial review (CB-WP-0022 T02)
+tier: L
+chaos: d8 = 6 → no override
+---
+
+# CB-RES-0007 — how rigorous rule systems record the ambiguity they find
+
+CB-WP-0022 T01. Surveyed 2026-08-03.
+
+**Not a survey of issue trackers.** The question is narrower: when a
+system is formal enough to *notice* that its rules do not decide
+something, what does it do with that observation? Three practices, and
+ourselves.
+
+---
+
+## 1. The baseline is us, and it is measured
+
+The external candidates are **practices, not runnable software**, so per
+InnerLoop Step 1 their rows are **directional and cap at `parity`**. The
+row that *can* be run is our own, and it is the one that matters, because
+the register has to beat what we already do.
+
+`tools/` harness output, 2026-08-03:
+
+```
+  findings                      6
+  with a runnable reproduction  2/6 = 33%
+  distinct files holding them   11
+  single register?              NO — 11 files, no index
+
+  U1..U10: raised 2026-07-30, first READ 2026-08-03 — 4 days
+  U1..U10: answered? NO — 4+ days open, 0 of 10 ruled
+```
+
+| finding | locations | reproduction |
+|---|---:|---|
+| U1–U10 underdetermined points | 1 | — |
+| SOLVE on a face-down Problem | 2 | — |
+| GR-A13 wasted SOLVE | 1 | — |
+| GR-E01 unreachable below 5 seats | 3 | **yes** (scenario) |
+| six provisional defaults | 6 | **yes** (scenarios) |
+| GR-E03/GR-E04 never played | 1 | — |
+
+### The uncomfortable number, stated before anyone else finds it
+
+**The rule the declaration proposed — *no finding without its
+reproduction* — would reject four of our six existing findings.**
+
+That is exactly the objection T02 is instructed to press, and the survey's
+job is to answer it rather than route around it. The answer is that
+**none of the four is expensive to reproduce**:
+
+- *SOLVE on a face-down Problem* — a scenario selecting SOLVE on a
+  face-down Problem and asserting no claim follows. The engine already
+  produced the behaviour; nobody wrote it down.
+- *GR-A13 wasted SOLVE* — the same shape.
+- *GR-E03/E04 never played* — a bot game to completion in each mode,
+  which `bot::play` already does.
+- *U1–U10* — six of the ten already have provisional scenarios; the
+  remaining four need the same treatment.
+
+So the 33% is not evidence that reproductions are costly. **It is evidence
+that nobody was ever asked for one**, which is the defect the register
+exists to fix and not an argument against the rule.
+
+## 2. Magic: the correction lands in the source, and the ruling is not authoritative
+
+The most instructive practice, and it **corrected an assumption this pass
+was about to build on**. I expected rulings to be the authoritative
+resolution of an ambiguity. They are not.
+
+The system is three layers, not two:
+
+| layer | what it is | authoritative? |
+|---|---|---|
+| Comprehensive Rules | the general rules | yes |
+| **Oracle** | the current authoritative text of each *card* — errata folded **in** | **yes** |
+| Rulings (via Gatherer) | judge annotations attached to a card | **no** — *"reminder information with no actual weight or rules meaning"* |
+
+When a card *"causes confusion, has outdated terminology, or doesn't work
+as intended, the Oracle text may be updated"* — the fix goes **into the
+authoritative text**, and the ruling layer stays explanatory.
+
+**The property to steal: a finding closes when the source changes, not
+when an annotation is added.** A register that accumulates permanent
+rulings is a system that has given up on fixing its rules. Ours should be
+a **queue that empties**, not an archive that grows.
+
+This directly shapes the lifecycle in T03: `applied` must mean *the rules
+text or the dataset changed and our provisional default was deleted* — not
+*a ruling was recorded*.
+
+*Directional, cited-only: the practice is described, not benchmarked.*
+
+## 3. Model checkers: the counterexample IS the finding
+
+A model checker does not report *"this property may not hold."* It emits a
+**counterexample trace** — a concrete execution that violates the claim,
+replayable by the tool that produced it.
+
+That is precisely the shape the declaration proposed, arrived at
+independently, and it is the strongest argument for the reproduction rule:
+in formal methods a claim without a trace is not a result, it is a
+hypothesis. Our `gr-e01-threshold-unreachable-2p` scenario is a
+counterexample in exactly this sense — it is a failing-in-fact artifact
+rather than a paragraph, and CB-EV-0005 already refused to delete it for
+that reason.
+
+**Property to beat: 100% of findings carry a replayable artifact.**
+
+*Directional, cited-only.*
+
+## 4. W3C / WHATWG: naming the undecided so it cannot be silently decided
+
+Specifications mark points as *implementation-defined* or
+*implementation-dependent*, which does two things at once: it tells an
+implementer they may choose, and it tells the *spec* that a choice is
+outstanding. The mark is machine-findable and survives revisions.
+
+We already have this and it works: `provisional: true` +
+`provisional_owner` on a scenario, surfaced by `make coverage` with an
+age. It is the one piece of the machinery that is not missing.
+
+**The register must reuse it, not compete with it.** Six of our findings
+already live there; a second mechanism would immediately disagree with the
+first.
+
+*Directional, cited-only.*
+
+## 5. Benchmarks to beat
+
+| dimension | today | benchmark |
+|---|---|---|
+| **findability** | 11 files, no index | **one register with an index**; every finding reachable from one command |
+| **reproducibility** | **2/6 = 33%** | **100%**, with withdrawn findings kept in the denominator |
+| **closure** | 0 of 10 U-items ruled in 4+ days | a finding closes only when **the source changed** (Magic's Oracle property), and the register says which commit |
+| **time raised → read** | **4 days** | the number this exists to fix; any mechanism that does not move it has failed |
+| **taxonomy fits reality** | untested | all six existing findings expressible **without** a new kind being invented during backfill |
+
+## 6. What the survey did not settle
+
+- **Whether a finding without a reproduction is rejected or admitted as a
+  note.** §1 shows the rule is affordable for our six, but that is n=6 and
+  all six are *engine-surfaced*. A finding from **play** — *"the DARVO
+  sequence feels punishing at 2 players"* — may be real and have no cheap
+  artifact. T03 must decide, and the honest options are a `note` tier that
+  can never be reported as a finding, or refusal.
+- **Where the register lives.** A finding is about *ground-game's* rules
+  but is produced by *clay-borg*. Putting it only in clay-borg repeats the
+  unread-inbox failure; putting it only in ground-game separates it from
+  its reproduction. Not decided here.
+- **Whether the engine-evolution register is redundant.** The
+  declaration's judgment is that it is. This survey found nothing that
+  bears on it either way, which is itself worth saying: the practices
+  above are all about *rules*, and none of them is about how the tool that
+  found the problem evolved.

tools/design-baseline.py

@@ -0,0 +1,93 @@
+#!/usr/bin/env python3
+"""Baseline harness: how findable, reproducible and answered are the
+design findings this project has already produced?
+
+The comparator is US, today. The external candidates (rulings databases,
+model-checker traces, W3C provisional marks) are practices rather than
+runnable software, so per InnerLoop Step 1 their rows are DIRECTIONAL and
+cap at `parity`. This is the row that can be measured.
+"""
+import os, re, subprocess, sys, datetime
+
+ROOT = "/home/worsch/clay-borg"
+os.chdir(ROOT)
+
+# The findings this project has actually produced, and where each lives.
+FINDINGS = {
+    "U1..U10 underdetermined points": ["specs/GroundRules.md"],
+    "SOLVE on a face-down Problem": ["workplans/CB-WP-0018-the-browser-is-a-client.md",
+                                     "evidence/CB-EV-0016-the-browser-is-a-client.md"],
+    "GR-A13 wasted SOLVE": ["evidence/CB-EV-0007-stage-0.md"],
+    "GR-E01 unreachable below 5 seats": ["evidence/CB-EV-0007-stage-0.md",
+                                         "scenarios/ground/gr-e01-threshold-unreachable-2p.yaml",
+                                         "workplans/CB-WP-0021-import-the-edition.md"],
+    "six provisional defaults": sorted(
+        os.path.join("scenarios/ground", f)
+        for f in os.listdir("scenarios/ground")
+        if f.endswith(".yaml")
+        and "provisional: true" in open(os.path.join("scenarios/ground", f)).read()),
+    "GR-E03/GR-E04 never played": ["evidence/CB-EV-0007-stage-0.md"],
+}
+
+def has_reproduction(paths):
+    """A runnable thing: a scenario file, or a named test/command."""
+    for p in paths:
+        if p.startswith("scenarios/"):
+            return True
+    return False
+
+def self_test():
+    """The control that matters: a harness that read nothing must not
+    report a clean baseline. Every path this survey cites must exist, and
+    the reproduction test must be able to say NO — one that answered yes
+    for everything would report 100% and look excellent."""
+    results = []
+
+    def check(name, ok, detail=""):
+        results.append((name, ok, detail))
+
+    missing = [p for paths in FINDINGS.values() for p in paths
+               if not os.path.exists(p)]
+    check("every cited location exists", not missing, ", ".join(missing[:3]))
+    check("the finding set is not empty", len(FINDINGS) >= 6, f"{len(FINDINGS)}")
+    check("reproduction detection can say NO",
+          not has_reproduction(["evidence/CB-EV-0007-stage-0.md"]),
+          "a detector that always says yes would report 100%")
+    check("reproduction detection can say YES",
+          has_reproduction(["scenarios/ground/gr-e01-threshold-unreachable-2p.yaml"]))
+    # The number this survey turns on, pinned so a later edit cannot move
+    # it silently: 2 of 6 today.
+    repro_now = sum(has_reproduction(v) for v in FINDINGS.values())
+    check("the measured baseline is 2 of 6", repro_now == 2, f"{repro_now}/6")
+
+    print("design-baseline self-test (positive control)")
+    ok = True
+    for name, passed, det in results:
+        print(f"  [{'ok ' if passed else 'FAIL'}] {name}" + (f"  — {det}" if det else ""))
+        ok &= passed
+    return 0 if ok else 1
+
+
+if "--self-test" in sys.argv:
+    raise SystemExit(self_test())
+
+print("BASELINE — design findings as they stand, 2026-08-03\n")
+places = set()
+repro = 0
+for name, paths in FINDINGS.items():
+    places.update(paths)
+    r = has_reproduction(paths)
+    repro += r
+    print(f"  {'repro' if r else '  -  '}  {len(paths)} location(s)  {name}")
+
+n = len(FINDINGS)
+print(f"\n  findings                      {n}")
+print(f"  with a runnable reproduction  {repro}/{n} = {100*repro//n}%")
+print(f"  distinct files holding them   {len(places)}")
+print(f"  single register?              NO — {len(places)} files, no index")
+
+# Time from raised to READ, for the one finding with a timestamp trail.
+raised = datetime.date(2026, 7, 30)   # hub message from clay-borg-custodian
+read   = datetime.date(2026, 8, 3)    # marked read this session
+print(f"\n  U1..U10: raised {raised}, first READ {read} — {(read-raised).days} days")
+print(f"  U1..U10: answered? NO — {(read-raised).days}+ days open, 0 of 10 ruled")

workplans/CB-WP-0022-the-design-instrument.md

@@ -2,7 +2,7 @@
 id: CB-WP-0022
 kind: product
 title: "The design instrument: findings about the game, with their reproductions"
-status: ready
+status: active
 ---
 
 # Purpose
@@ -88,7 +88,7 @@ tier L exists to surface.
 
 ```task
 id: CB-WP-0022-T01
-status: todo
+status: done
 priority: high
 ```
 
@@ -114,6 +114,33 @@ found*? Candidates worth a benchmark-to-beat:
 Name, per dimension, the property to beat — findability, reproducibility,
 and whether a ruling can *close* a finding mechanically.
 
+**Done 2026-08-03.**
+[CB-RES-0007](../research/CB-RES-0007-design-instrument.md), with a
+runnable baseline (`tools/design-baseline.py`).
+
+**The baseline is us, and it is measured**: 6 findings across **11 files**
+with no index, **2 of 6 (33%) with a runnable reproduction**, and U1–U10
+raised 2026-07-30, first *read* 2026-08-03 — **4 days**, 0 of 10 ruled.
+
+**The uncomfortable number is stated up front rather than left for the
+review to find: the reproduction rule would reject four of our six
+existing findings.** The survey answers it — none of the four is
+expensive to reproduce, so the 33% is evidence that nobody was ever asked
+for one, not that the rule is unaffordable.
+
+**Magic corrected an assumption this pass was about to build on.** I
+expected a ruling to be the authoritative resolution. It is not: rulings
+are *"reminder information with no actual weight or rules meaning"*, and
+the authoritative fix folds into the **Oracle** card text. **A finding
+closes when the source changes, not when an annotation is added** — so
+the register must be a queue that empties, not an archive that grows.
+That is now a constraint on T03's lifecycle.
+
+Model checkers supplied the reproduction rule independently (a
+counterexample trace *is* the finding), and W3C's *implementation-defined*
+mark is the one piece of machinery we already have and must reuse rather
+than duplicate.
+
 ## Task: adversarial review
 
 ```task