tegwick
a86efba4c3
CB-RES-0007 plus a runnable baseline harness. Tier L invokes the runnable-baseline option; the external candidates are practices rather than software, so their rows are directional and cap at parity, and the row that CAN be run is our own. Measured: 6 findings across 11 files with no index, 2 of 6 (33%) with a runnable reproduction, U1-U10 raised 2026-07-30 and first READ 2026-08-03 -- 4 days, 0 of 10 ruled. The uncomfortable number is stated before the review can find it: the proposed 'no finding without its reproduction' rule would reject four of our six existing findings. The survey answers rather than routes around it -- none of the four is expensive to reproduce, so 33% is evidence nobody was ever asked for one. Magic corrected an assumption this pass was about to build on. Rulings are NOT authoritative -- they are 'reminder information with no actual weight or rules meaning' -- and the authoritative fix folds into the Oracle card text. So a finding closes when the SOURCE changes, not when an annotation is added, and the register must be a queue that empties rather than an archive that grows. That is now a constraint on the ADR's lifecycle. Model checkers supply the reproduction rule independently: a counterexample trace IS the finding. W3C's implementation-defined mark is the machinery we already have in provisional: scenarios and must reuse. The loop-lint gate caught the new tool with no --self-test; it has one, pinning the 2-of-6 baseline so a later edit cannot move it silently. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
7 KiB
| id | capability | status | tier | chaos |
|---|---|---|---|---|
| CB-RES-0007 | design.findings | draft — awaiting adversarial review (CB-WP-0022 T02) | L | d8 = 6 → no override |
CB-RES-0007 — how rigorous rule systems record the ambiguity they find
CB-WP-0022 T01. Surveyed 2026-08-03.
Not a survey of issue trackers. The question is narrower: when a system is formal enough to notice that its rules do not decide something, what does it do with that observation? Three practices, and ourselves.
1. The baseline is us, and it is measured
The external candidates are practices, not runnable software, so per
InnerLoop Step 1 their rows are directional and cap at parity. The
row that can be run is our own, and it is the one that matters, because
the register has to beat what we already do.
tools/ harness output, 2026-08-03:
findings 6
with a runnable reproduction 2/6 = 33%
distinct files holding them 11
single register? NO — 11 files, no index
U1..U10: raised 2026-07-30, first READ 2026-08-03 — 4 days
U1..U10: answered? NO — 4+ days open, 0 of 10 ruled
| finding | locations | reproduction |
|---|---|---|
| U1–U10 underdetermined points | 1 | — |
| SOLVE on a face-down Problem | 2 | — |
| GR-A13 wasted SOLVE | 1 | — |
| GR-E01 unreachable below 5 seats | 3 | yes (scenario) |
| six provisional defaults | 6 | yes (scenarios) |
| GR-E03/GR-E04 never played | 1 | — |
The uncomfortable number, stated before anyone else finds it
The rule the declaration proposed — no finding without its reproduction — would reject four of our six existing findings.
That is exactly the objection T02 is instructed to press, and the survey's job is to answer it rather than route around it. The answer is that none of the four is expensive to reproduce:
- SOLVE on a face-down Problem — a scenario selecting SOLVE on a face-down Problem and asserting no claim follows. The engine already produced the behaviour; nobody wrote it down.
- GR-A13 wasted SOLVE — the same shape.
- GR-E03/E04 never played — a bot game to completion in each mode,
which
bot::playalready does. - U1–U10 — six of the ten already have provisional scenarios; the remaining four need the same treatment.
So the 33% is not evidence that reproductions are costly. It is evidence that nobody was ever asked for one, which is the defect the register exists to fix and not an argument against the rule.
2. Magic: the correction lands in the source, and the ruling is not authoritative
The most instructive practice, and it corrected an assumption this pass was about to build on. I expected rulings to be the authoritative resolution of an ambiguity. They are not.
The system is three layers, not two:
| layer | what it is | authoritative? |
|---|---|---|
| Comprehensive Rules | the general rules | yes |
| Oracle | the current authoritative text of each card — errata folded in | yes |
| Rulings (via Gatherer) | judge annotations attached to a card | no — "reminder information with no actual weight or rules meaning" |
When a card "causes confusion, has outdated terminology, or doesn't work as intended, the Oracle text may be updated" — the fix goes into the authoritative text, and the ruling layer stays explanatory.
The property to steal: a finding closes when the source changes, not when an annotation is added. A register that accumulates permanent rulings is a system that has given up on fixing its rules. Ours should be a queue that empties, not an archive that grows.
This directly shapes the lifecycle in T03: applied must mean the rules
text or the dataset changed and our provisional default was deleted — not
a ruling was recorded.
Directional, cited-only: the practice is described, not benchmarked.
3. Model checkers: the counterexample IS the finding
A model checker does not report "this property may not hold." It emits a counterexample trace — a concrete execution that violates the claim, replayable by the tool that produced it.
That is precisely the shape the declaration proposed, arrived at
independently, and it is the strongest argument for the reproduction rule:
in formal methods a claim without a trace is not a result, it is a
hypothesis. Our gr-e01-threshold-unreachable-2p scenario is a
counterexample in exactly this sense — it is a failing-in-fact artifact
rather than a paragraph, and CB-EV-0005 already refused to delete it for
that reason.
Property to beat: 100% of findings carry a replayable artifact.
Directional, cited-only.
4. W3C / WHATWG: naming the undecided so it cannot be silently decided
Specifications mark points as implementation-defined or implementation-dependent, which does two things at once: it tells an implementer they may choose, and it tells the spec that a choice is outstanding. The mark is machine-findable and survives revisions.
We already have this and it works: provisional: true +
provisional_owner on a scenario, surfaced by make coverage with an
age. It is the one piece of the machinery that is not missing.
The register must reuse it, not compete with it. Six of our findings already live there; a second mechanism would immediately disagree with the first.
Directional, cited-only.
5. Benchmarks to beat
| dimension | today | benchmark |
|---|---|---|
| findability | 11 files, no index | one register with an index; every finding reachable from one command |
| reproducibility | 2/6 = 33% | 100%, with withdrawn findings kept in the denominator |
| closure | 0 of 10 U-items ruled in 4+ days | a finding closes only when the source changed (Magic's Oracle property), and the register says which commit |
| time raised → read | 4 days | the number this exists to fix; any mechanism that does not move it has failed |
| taxonomy fits reality | untested | all six existing findings expressible without a new kind being invented during backfill |
6. What the survey did not settle
- Whether a finding without a reproduction is rejected or admitted as a
note. §1 shows the rule is affordable for our six, but that is n=6 and
all six are engine-surfaced. A finding from play — "the DARVO
sequence feels punishing at 2 players" — may be real and have no cheap
artifact. T03 must decide, and the honest options are a
notetier that can never be reported as a finding, or refusal. - Where the register lives. A finding is about ground-game's rules but is produced by clay-borg. Putting it only in clay-borg repeats the unread-inbox failure; putting it only in ground-game separates it from its reproduction. Not decided here.
- Whether the engine-evolution register is redundant. The declaration's judgment is that it is. This survey found nothing that bears on it either way, which is itself worth saying: the practices above are all about rules, and none of them is about how the tool that found the problem evolved.