coulomb/clay-borg
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

CB-WP-0022-T01: survey — how rule systems record the ambiguity they find

Browse source 
CB-RES-0007 plus a runnable baseline harness. Tier L invokes the
runnable-baseline option; the external candidates are practices rather
than software, so their rows are directional and cap at parity, and the
row that CAN be run is our own.

Measured: 6 findings across 11 files with no index, 2 of 6 (33%) with a
runnable reproduction, U1-U10 raised 2026-07-30 and first READ
2026-08-03 -- 4 days, 0 of 10 ruled.

The uncomfortable number is stated before the review can find it: the
proposed 'no finding without its reproduction' rule would reject four of
our six existing findings. The survey answers rather than routes around
it -- none of the four is expensive to reproduce, so 33% is evidence
nobody was ever asked for one.

Magic corrected an assumption this pass was about to build on. Rulings
are NOT authoritative -- they are 'reminder information with no actual
weight or rules meaning' -- and the authoritative fix folds into the
Oracle card text. So a finding closes when the SOURCE changes, not when
an annotation is added, and the register must be a queue that empties
rather than an archive that grows. That is now a constraint on the ADR's
lifecycle.

Model checkers supply the reproduction rule independently: a
counterexample trace IS the finding. W3C's implementation-defined mark is
the machinery we already have in provisional: scenarios and must reuse.

The loop-lint gate caught the new tool with no --self-test; it has one,
pinning the 2-of-6 baseline so a later edit cannot move it silently.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-08-03 21:19:12 +02:00
parent bb426b3e9b
commit a86efba4c3
4 changed files with 283 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

1
Makefile
View file

@ -119,6 +119,7 @@ self-tests:
$(PY) $(TOOLS)/size-metrics.py --self-test
$(PY) $(TOOLS)/runtime-metrics.py --self-test
$(PY) $(TOOLS)/replay-test.py --self-test
$(PY) $(TOOLS)/design-baseline.py --self-test
# T01 positive control: prove the environment fix, do not assume it. Runs
# every tool from a foreign working directory with a PATH that has no

160
research/CB-RES-0007-design-instrument.md Normal file
View file

@ -0,0 +1,160 @@
---
id: CB-RES-0007
capability: design.findings
status: draft — awaiting adversarial review (CB-WP-0022 T02)
tier: L
chaos: d8 = 6 → no override
---
# CB-RES-0007 — how rigorous rule systems record the ambiguity they find
CB-WP-0022 T01. Surveyed 2026-08-03.
**Not a survey of issue trackers.** The question is narrower: when a
system is formal enough to *notice* that its rules do not decide
something, what does it do with that observation? Three practices, and
ourselves.
---
## 1. The baseline is us, and it is measured
The external candidates are **practices, not runnable software**, so per
InnerLoop Step 1 their rows are **directional and cap at `parity`**. The
row that *can* be run is our own, and it is the one that matters, because
the register has to beat what we already do.
`tools/` harness output, 2026-08-03:
```
findings 6
with a runnable reproduction 2/6 = 33%
distinct files holding them 11
single register? NO — 11 files, no index
U1..U10: raised 2026-07-30, first READ 2026-08-03 — 4 days
U1..U10: answered? NO — 4+ days open, 0 of 10 ruled
```
| finding | locations | reproduction |
|---|---:|---|
| U1–U10 underdetermined points | 1 | — |
| SOLVE on a face-down Problem | 2 | — |
| GR-A13 wasted SOLVE | 1 | — |
| GR-E01 unreachable below 5 seats | 3 | **yes** (scenario) |
| six provisional defaults | 6 | **yes** (scenarios) |
| GR-E03/GR-E04 never played | 1 | — |
### The uncomfortable number, stated before anyone else finds it
**The rule the declaration proposed — *no finding without its
reproduction* — would reject four of our six existing findings.**
That is exactly the objection T02 is instructed to press, and the survey's
job is to answer it rather than route around it. The answer is that
**none of the four is expensive to reproduce**:
- *SOLVE on a face-down Problem* — a scenario selecting SOLVE on a
face-down Problem and asserting no claim follows. The engine already
produced the behaviour; nobody wrote it down.
- *GR-A13 wasted SOLVE* — the same shape.
- *GR-E03/E04 never played* — a bot game to completion in each mode,
which `bot::play` already does.
- *U1–U10* — six of the ten already have provisional scenarios; the
remaining four need the same treatment.
So the 33% is not evidence that reproductions are costly. **It is evidence
that nobody was ever asked for one**, which is the defect the register
exists to fix and not an argument against the rule.
## 2. Magic: the correction lands in the source, and the ruling is not authoritative
The most instructive practice, and it **corrected an assumption this pass
was about to build on**. I expected rulings to be the authoritative
resolution of an ambiguity. They are not.
The system is three layers, not two:
| layer | what it is | authoritative? |
|---|---|---|
| Comprehensive Rules | the general rules | yes |
| **Oracle** | the current authoritative text of each *card* — errata folded **in** | **yes** |
| Rulings (via Gatherer) | judge annotations attached to a card | **no** — *"reminder information with no actual weight or rules meaning"* |
When a card *"causes confusion, has outdated terminology, or doesn't work
as intended, the Oracle text may be updated"* — the fix goes **into the
authoritative text**, and the ruling layer stays explanatory.
**The property to steal: a finding closes when the source changes, not
when an annotation is added.** A register that accumulates permanent
rulings is a system that has given up on fixing its rules. Ours should be
a **queue that empties**, not an archive that grows.
This directly shapes the lifecycle in T03: `applied` must mean *the rules
text or the dataset changed and our provisional default was deleted* — not
*a ruling was recorded*.
*Directional, cited-only: the practice is described, not benchmarked.*
## 3. Model checkers: the counterexample IS the finding
A model checker does not report *"this property may not hold."* It emits a
**counterexample trace** — a concrete execution that violates the claim,
replayable by the tool that produced it.
That is precisely the shape the declaration proposed, arrived at
independently, and it is the strongest argument for the reproduction rule:
in formal methods a claim without a trace is not a result, it is a
hypothesis. Our `gr-e01-threshold-unreachable-2p` scenario is a
counterexample in exactly this sense — it is a failing-in-fact artifact
rather than a paragraph, and CB-EV-0005 already refused to delete it for
that reason.
**Property to beat: 100% of findings carry a replayable artifact.**
*Directional, cited-only.*
## 4. W3C / WHATWG: naming the undecided so it cannot be silently decided
Specifications mark points as *implementation-defined* or
*implementation-dependent*, which does two things at once: it tells an
implementer they may choose, and it tells the *spec* that a choice is
outstanding. The mark is machine-findable and survives revisions.
We already have this and it works: `provisional: true` +
`provisional_owner` on a scenario, surfaced by `make coverage` with an
age. It is the one piece of the machinery that is not missing.
**The register must reuse it, not compete with it.** Six of our findings
already live there; a second mechanism would immediately disagree with the
first.
*Directional, cited-only.*
## 5. Benchmarks to beat
| dimension | today | benchmark |
|---|---|---|
| **findability** | 11 files, no index | **one register with an index**; every finding reachable from one command |
| **reproducibility** | **2/6 = 33%** | **100%**, with withdrawn findings kept in the denominator |
| **closure** | 0 of 10 U-items ruled in 4+ days | a finding closes only when **the source changed** (Magic's Oracle property), and the register says which commit |
| **time raised → read** | **4 days** | the number this exists to fix; any mechanism that does not move it has failed |
| **taxonomy fits reality** | untested | all six existing findings expressible **without** a new kind being invented during backfill |
## 6. What the survey did not settle
- **Whether a finding without a reproduction is rejected or admitted as a
note.** §1 shows the rule is affordable for our six, but that is n=6 and
all six are *engine-surfaced*. A finding from **play** — *"the DARVO
sequence feels punishing at 2 players"* — may be real and have no cheap
artifact. T03 must decide, and the honest options are a `note` tier that
can never be reported as a finding, or refusal.
- **Where the register lives.** A finding is about *ground-game's* rules
but is produced by *clay-borg*. Putting it only in clay-borg repeats the
unread-inbox failure; putting it only in ground-game separates it from
its reproduction. Not decided here.
- **Whether the engine-evolution register is redundant.** The
declaration's judgment is that it is. This survey found nothing that
bears on it either way, which is itself worth saying: the practices
above are all about *rules*, and none of them is about how the tool that
found the problem evolved.

93
tools/design-baseline.py Executable file
View file

@ -0,0 +1,93 @@
#!/usr/bin/env python3
"""Baseline harness: how findable, reproducible and answered are the
design findings this project has already produced?
The comparator is US, today. The external candidates (rulings databases,
model-checker traces, W3C provisional marks) are practices rather than
runnable software, so per InnerLoop Step 1 their rows are DIRECTIONAL and
cap at `parity`. This is the row that can be measured.
"""
import os, re, subprocess, sys, datetime
ROOT = "/home/worsch/clay-borg"
os.chdir(ROOT)
# The findings this project has actually produced, and where each lives.
FINDINGS = {
"U1..U10 underdetermined points": ["specs/GroundRules.md"],
"SOLVE on a face-down Problem": ["workplans/CB-WP-0018-the-browser-is-a-client.md",
"evidence/CB-EV-0016-the-browser-is-a-client.md"],
"GR-A13 wasted SOLVE": ["evidence/CB-EV-0007-stage-0.md"],
"GR-E01 unreachable below 5 seats": ["evidence/CB-EV-0007-stage-0.md",
"scenarios/ground/gr-e01-threshold-unreachable-2p.yaml",
"workplans/CB-WP-0021-import-the-edition.md"],
"six provisional defaults": sorted(
os.path.join("scenarios/ground", f)
for f in os.listdir("scenarios/ground")
if f.endswith(".yaml")
and "provisional: true" in open(os.path.join("scenarios/ground", f)).read()),
"GR-E03/GR-E04 never played": ["evidence/CB-EV-0007-stage-0.md"],
}
def has_reproduction(paths):
"""A runnable thing: a scenario file, or a named test/command."""
for p in paths:
if p.startswith("scenarios/"):
return True
return False
def self_test():
"""The control that matters: a harness that read nothing must not
report a clean baseline. Every path this survey cites must exist, and
the reproduction test must be able to say NO — one that answered yes
for everything would report 100% and look excellent."""
results = []
def check(name, ok, detail=""):
results.append((name, ok, detail))
missing = [p for paths in FINDINGS.values() for p in paths
if not os.path.exists(p)]
check("every cited location exists", not missing, ", ".join(missing[:3]))
check("the finding set is not empty", len(FINDINGS) >= 6, f"{len(FINDINGS)}")
check("reproduction detection can say NO",
not has_reproduction(["evidence/CB-EV-0007-stage-0.md"]),
"a detector that always says yes would report 100%")
check("reproduction detection can say YES",
has_reproduction(["scenarios/ground/gr-e01-threshold-unreachable-2p.yaml"]))
# The number this survey turns on, pinned so a later edit cannot move
# it silently: 2 of 6 today.
repro_now = sum(has_reproduction(v) for v in FINDINGS.values())
check("the measured baseline is 2 of 6", repro_now == 2, f"{repro_now}/6")
print("design-baseline self-test (positive control)")
ok = True
for name, passed, det in results:
print(f" [{'ok ' if passed else 'FAIL'}] {name}" + (f" — {det}" if det else ""))
ok &= passed
return 0 if ok else 1
if "--self-test" in sys.argv:
raise SystemExit(self_test())
print("BASELINE — design findings as they stand, 2026-08-03\n")
places = set()
repro = 0
for name, paths in FINDINGS.items():
places.update(paths)
r = has_reproduction(paths)
repro += r
print(f" {'repro' if r else ' - '} {len(paths)} location(s) {name}")
n = len(FINDINGS)
print(f"\n findings {n}")
print(f" with a runnable reproduction {repro}/{n} = {100*repro//n}%")
print(f" distinct files holding them {len(places)}")
print(f" single register? NO — {len(places)} files, no index")
# Time from raised to READ, for the one finding with a timestamp trail.
raised = datetime.date(2026, 7, 30) # hub message from clay-borg-custodian
read = datetime.date(2026, 8, 3) # marked read this session
print(f"\n U1..U10: raised {raised}, first READ {read} — {(read-raised).days} days")
print(f" U1..U10: answered? NO — {(read-raised).days}+ days open, 0 of 10 ruled")

31
workplans/CB-WP-0022-the-design-instrument.md
View file

@ -2,7 +2,7 @@
id: CB-WP-0022
kind: product
title: "The design instrument: findings about the game, with their reproductions"
status: ready
status: active
---
# Purpose
@ -88,7 +88,7 @@ tier L exists to surface.
```task
id: CB-WP-0022-T01
status: todo
status: done
priority: high
```
@ -114,6 +114,33 @@ found*? Candidates worth a benchmark-to-beat:
Name, per dimension, the property to beat — findability, reproducibility,
and whether a ruling can *close* a finding mechanically.
**Done 2026-08-03.**
[CB-RES-0007](../research/CB-RES-0007-design-instrument.md), with a
runnable baseline (`tools/design-baseline.py`).
**The baseline is us, and it is measured**: 6 findings across **11 files**
with no index, **2 of 6 (33%) with a runnable reproduction**, and U1–U10
raised 2026-07-30, first *read* 2026-08-03 — **4 days**, 0 of 10 ruled.
**The uncomfortable number is stated up front rather than left for the
review to find: the reproduction rule would reject four of our six
existing findings.** The survey answers it — none of the four is
expensive to reproduce, so the 33% is evidence that nobody was ever asked
for one, not that the rule is unaffordable.
**Magic corrected an assumption this pass was about to build on.** I
expected a ruling to be the authoritative resolution. It is not: rulings
are *"reminder information with no actual weight or rules meaning"*, and
the authoritative fix folds into the **Oracle** card text. **A finding
closes when the source changes, not when an annotation is added** — so
the register must be a queue that empties, not an archive that grows.
That is now a constraint on T03's lifecycle.
Model checkers supplied the reproduction rule independently (a
counterexample trace *is* the finding), and W3C's *implementation-defined*
mark is the one piece of machinery we already have and must reuse rather
than duplicate.
## Task: adversarial review
```task