tegwick
a86efba4c3
CB-RES-0007 plus a runnable baseline harness. Tier L invokes the runnable-baseline option; the external candidates are practices rather than software, so their rows are directional and cap at parity, and the row that CAN be run is our own. Measured: 6 findings across 11 files with no index, 2 of 6 (33%) with a runnable reproduction, U1-U10 raised 2026-07-30 and first READ 2026-08-03 -- 4 days, 0 of 10 ruled. The uncomfortable number is stated before the review can find it: the proposed 'no finding without its reproduction' rule would reject four of our six existing findings. The survey answers rather than routes around it -- none of the four is expensive to reproduce, so 33% is evidence nobody was ever asked for one. Magic corrected an assumption this pass was about to build on. Rulings are NOT authoritative -- they are 'reminder information with no actual weight or rules meaning' -- and the authoritative fix folds into the Oracle card text. So a finding closes when the SOURCE changes, not when an annotation is added, and the register must be a queue that empties rather than an archive that grows. That is now a constraint on the ADR's lifecycle. Model checkers supply the reproduction rule independently: a counterexample trace IS the finding. W3C's implementation-defined mark is the machinery we already have in provisional: scenarios and must reuse. The loop-lint gate caught the new tool with no --self-test; it has one, pinning the 2-of-6 baseline so a later edit cannot move it silently. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
160 lines
7 KiB
Markdown
160 lines
7 KiB
Markdown
---
|
||
id: CB-RES-0007
|
||
capability: design.findings
|
||
status: draft — awaiting adversarial review (CB-WP-0022 T02)
|
||
tier: L
|
||
chaos: d8 = 6 → no override
|
||
---
|
||
|
||
# CB-RES-0007 — how rigorous rule systems record the ambiguity they find
|
||
|
||
CB-WP-0022 T01. Surveyed 2026-08-03.
|
||
|
||
**Not a survey of issue trackers.** The question is narrower: when a
|
||
system is formal enough to *notice* that its rules do not decide
|
||
something, what does it do with that observation? Three practices, and
|
||
ourselves.
|
||
|
||
---
|
||
|
||
## 1. The baseline is us, and it is measured
|
||
|
||
The external candidates are **practices, not runnable software**, so per
|
||
InnerLoop Step 1 their rows are **directional and cap at `parity`**. The
|
||
row that *can* be run is our own, and it is the one that matters, because
|
||
the register has to beat what we already do.
|
||
|
||
`tools/` harness output, 2026-08-03:
|
||
|
||
```
|
||
findings 6
|
||
with a runnable reproduction 2/6 = 33%
|
||
distinct files holding them 11
|
||
single register? NO — 11 files, no index
|
||
|
||
U1..U10: raised 2026-07-30, first READ 2026-08-03 — 4 days
|
||
U1..U10: answered? NO — 4+ days open, 0 of 10 ruled
|
||
```
|
||
|
||
| finding | locations | reproduction |
|
||
|---|---:|---|
|
||
| U1–U10 underdetermined points | 1 | — |
|
||
| SOLVE on a face-down Problem | 2 | — |
|
||
| GR-A13 wasted SOLVE | 1 | — |
|
||
| GR-E01 unreachable below 5 seats | 3 | **yes** (scenario) |
|
||
| six provisional defaults | 6 | **yes** (scenarios) |
|
||
| GR-E03/GR-E04 never played | 1 | — |
|
||
|
||
### The uncomfortable number, stated before anyone else finds it
|
||
|
||
**The rule the declaration proposed — *no finding without its
|
||
reproduction* — would reject four of our six existing findings.**
|
||
|
||
That is exactly the objection T02 is instructed to press, and the survey's
|
||
job is to answer it rather than route around it. The answer is that
|
||
**none of the four is expensive to reproduce**:
|
||
|
||
- *SOLVE on a face-down Problem* — a scenario selecting SOLVE on a
|
||
face-down Problem and asserting no claim follows. The engine already
|
||
produced the behaviour; nobody wrote it down.
|
||
- *GR-A13 wasted SOLVE* — the same shape.
|
||
- *GR-E03/E04 never played* — a bot game to completion in each mode,
|
||
which `bot::play` already does.
|
||
- *U1–U10* — six of the ten already have provisional scenarios; the
|
||
remaining four need the same treatment.
|
||
|
||
So the 33% is not evidence that reproductions are costly. **It is evidence
|
||
that nobody was ever asked for one**, which is the defect the register
|
||
exists to fix and not an argument against the rule.
|
||
|
||
## 2. Magic: the correction lands in the source, and the ruling is not authoritative
|
||
|
||
The most instructive practice, and it **corrected an assumption this pass
|
||
was about to build on**. I expected rulings to be the authoritative
|
||
resolution of an ambiguity. They are not.
|
||
|
||
The system is three layers, not two:
|
||
|
||
| layer | what it is | authoritative? |
|
||
|---|---|---|
|
||
| Comprehensive Rules | the general rules | yes |
|
||
| **Oracle** | the current authoritative text of each *card* — errata folded **in** | **yes** |
|
||
| Rulings (via Gatherer) | judge annotations attached to a card | **no** — *"reminder information with no actual weight or rules meaning"* |
|
||
|
||
When a card *"causes confusion, has outdated terminology, or doesn't work
|
||
as intended, the Oracle text may be updated"* — the fix goes **into the
|
||
authoritative text**, and the ruling layer stays explanatory.
|
||
|
||
**The property to steal: a finding closes when the source changes, not
|
||
when an annotation is added.** A register that accumulates permanent
|
||
rulings is a system that has given up on fixing its rules. Ours should be
|
||
a **queue that empties**, not an archive that grows.
|
||
|
||
This directly shapes the lifecycle in T03: `applied` must mean *the rules
|
||
text or the dataset changed and our provisional default was deleted* — not
|
||
*a ruling was recorded*.
|
||
|
||
*Directional, cited-only: the practice is described, not benchmarked.*
|
||
|
||
## 3. Model checkers: the counterexample IS the finding
|
||
|
||
A model checker does not report *"this property may not hold."* It emits a
|
||
**counterexample trace** — a concrete execution that violates the claim,
|
||
replayable by the tool that produced it.
|
||
|
||
That is precisely the shape the declaration proposed, arrived at
|
||
independently, and it is the strongest argument for the reproduction rule:
|
||
in formal methods a claim without a trace is not a result, it is a
|
||
hypothesis. Our `gr-e01-threshold-unreachable-2p` scenario is a
|
||
counterexample in exactly this sense — it is a failing-in-fact artifact
|
||
rather than a paragraph, and CB-EV-0005 already refused to delete it for
|
||
that reason.
|
||
|
||
**Property to beat: 100% of findings carry a replayable artifact.**
|
||
|
||
*Directional, cited-only.*
|
||
|
||
## 4. W3C / WHATWG: naming the undecided so it cannot be silently decided
|
||
|
||
Specifications mark points as *implementation-defined* or
|
||
*implementation-dependent*, which does two things at once: it tells an
|
||
implementer they may choose, and it tells the *spec* that a choice is
|
||
outstanding. The mark is machine-findable and survives revisions.
|
||
|
||
We already have this and it works: `provisional: true` +
|
||
`provisional_owner` on a scenario, surfaced by `make coverage` with an
|
||
age. It is the one piece of the machinery that is not missing.
|
||
|
||
**The register must reuse it, not compete with it.** Six of our findings
|
||
already live there; a second mechanism would immediately disagree with the
|
||
first.
|
||
|
||
*Directional, cited-only.*
|
||
|
||
## 5. Benchmarks to beat
|
||
|
||
| dimension | today | benchmark |
|
||
|---|---|---|
|
||
| **findability** | 11 files, no index | **one register with an index**; every finding reachable from one command |
|
||
| **reproducibility** | **2/6 = 33%** | **100%**, with withdrawn findings kept in the denominator |
|
||
| **closure** | 0 of 10 U-items ruled in 4+ days | a finding closes only when **the source changed** (Magic's Oracle property), and the register says which commit |
|
||
| **time raised → read** | **4 days** | the number this exists to fix; any mechanism that does not move it has failed |
|
||
| **taxonomy fits reality** | untested | all six existing findings expressible **without** a new kind being invented during backfill |
|
||
|
||
## 6. What the survey did not settle
|
||
|
||
- **Whether a finding without a reproduction is rejected or admitted as a
|
||
note.** §1 shows the rule is affordable for our six, but that is n=6 and
|
||
all six are *engine-surfaced*. A finding from **play** — *"the DARVO
|
||
sequence feels punishing at 2 players"* — may be real and have no cheap
|
||
artifact. T03 must decide, and the honest options are a `note` tier that
|
||
can never be reported as a finding, or refusal.
|
||
- **Where the register lives.** A finding is about *ground-game's* rules
|
||
but is produced by *clay-borg*. Putting it only in clay-borg repeats the
|
||
unread-inbox failure; putting it only in ground-game separates it from
|
||
its reproduction. Not decided here.
|
||
- **Whether the engine-evolution register is redundant.** The
|
||
declaration's judgment is that it is. This survey found nothing that
|
||
bears on it either way, which is itself worth saying: the practices
|
||
above are all about *rules*, and none of them is about how the tool that
|
||
found the problem evolved.
|