From a86efba4c30af503567e7a2535f537ccd282d004 Mon Sep 17 00:00:00 2001 From: tegwick Date: Mon, 3 Aug 2026 21:19:12 +0200 Subject: [PATCH] =?UTF-8?q?CB-WP-0022-T01:=20survey=20=E2=80=94=20how=20ru?= =?UTF-8?q?le=20systems=20record=20the=20ambiguity=20they=20find?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CB-RES-0007 plus a runnable baseline harness. Tier L invokes the runnable-baseline option; the external candidates are practices rather than software, so their rows are directional and cap at parity, and the row that CAN be run is our own. Measured: 6 findings across 11 files with no index, 2 of 6 (33%) with a runnable reproduction, U1-U10 raised 2026-07-30 and first READ 2026-08-03 -- 4 days, 0 of 10 ruled. The uncomfortable number is stated before the review can find it: the proposed 'no finding without its reproduction' rule would reject four of our six existing findings. The survey answers rather than routes around it -- none of the four is expensive to reproduce, so 33% is evidence nobody was ever asked for one. Magic corrected an assumption this pass was about to build on. Rulings are NOT authoritative -- they are 'reminder information with no actual weight or rules meaning' -- and the authoritative fix folds into the Oracle card text. So a finding closes when the SOURCE changes, not when an annotation is added, and the register must be a queue that empties rather than an archive that grows. That is now a constraint on the ADR's lifecycle. Model checkers supply the reproduction rule independently: a counterexample trace IS the finding. W3C's implementation-defined mark is the machinery we already have in provisional: scenarios and must reuse. The loop-lint gate caught the new tool with no --self-test; it has one, pinning the 2-of-6 baseline so a later edit cannot move it silently. Co-Authored-By: Claude Opus 5 --- Makefile | 1 + research/CB-RES-0007-design-instrument.md | 160 ++++++++++++++++++ tools/design-baseline.py | 93 ++++++++++ workplans/CB-WP-0022-the-design-instrument.md | 31 +++- 4 files changed, 283 insertions(+), 2 deletions(-) create mode 100644 research/CB-RES-0007-design-instrument.md create mode 100755 tools/design-baseline.py diff --git a/Makefile b/Makefile index 1d77698..e0ddc1d 100644 --- a/Makefile +++ b/Makefile @@ -119,6 +119,7 @@ self-tests: $(PY) $(TOOLS)/size-metrics.py --self-test $(PY) $(TOOLS)/runtime-metrics.py --self-test $(PY) $(TOOLS)/replay-test.py --self-test + $(PY) $(TOOLS)/design-baseline.py --self-test # T01 positive control: prove the environment fix, do not assume it. Runs # every tool from a foreign working directory with a PATH that has no diff --git a/research/CB-RES-0007-design-instrument.md b/research/CB-RES-0007-design-instrument.md new file mode 100644 index 0000000..b709f1d --- /dev/null +++ b/research/CB-RES-0007-design-instrument.md @@ -0,0 +1,160 @@ +--- +id: CB-RES-0007 +capability: design.findings +status: draft — awaiting adversarial review (CB-WP-0022 T02) +tier: L +chaos: d8 = 6 → no override +--- + +# CB-RES-0007 — how rigorous rule systems record the ambiguity they find + +CB-WP-0022 T01. Surveyed 2026-08-03. + +**Not a survey of issue trackers.** The question is narrower: when a +system is formal enough to *notice* that its rules do not decide +something, what does it do with that observation? Three practices, and +ourselves. + +--- + +## 1. The baseline is us, and it is measured + +The external candidates are **practices, not runnable software**, so per +InnerLoop Step 1 their rows are **directional and cap at `parity`**. The +row that *can* be run is our own, and it is the one that matters, because +the register has to beat what we already do. + +`tools/` harness output, 2026-08-03: + +``` + findings 6 + with a runnable reproduction 2/6 = 33% + distinct files holding them 11 + single register? NO — 11 files, no index + + U1..U10: raised 2026-07-30, first READ 2026-08-03 — 4 days + U1..U10: answered? NO — 4+ days open, 0 of 10 ruled +``` + +| finding | locations | reproduction | +|---|---:|---| +| U1–U10 underdetermined points | 1 | — | +| SOLVE on a face-down Problem | 2 | — | +| GR-A13 wasted SOLVE | 1 | — | +| GR-E01 unreachable below 5 seats | 3 | **yes** (scenario) | +| six provisional defaults | 6 | **yes** (scenarios) | +| GR-E03/GR-E04 never played | 1 | — | + +### The uncomfortable number, stated before anyone else finds it + +**The rule the declaration proposed — *no finding without its +reproduction* — would reject four of our six existing findings.** + +That is exactly the objection T02 is instructed to press, and the survey's +job is to answer it rather than route around it. The answer is that +**none of the four is expensive to reproduce**: + +- *SOLVE on a face-down Problem* — a scenario selecting SOLVE on a + face-down Problem and asserting no claim follows. The engine already + produced the behaviour; nobody wrote it down. +- *GR-A13 wasted SOLVE* — the same shape. +- *GR-E03/E04 never played* — a bot game to completion in each mode, + which `bot::play` already does. +- *U1–U10* — six of the ten already have provisional scenarios; the + remaining four need the same treatment. + +So the 33% is not evidence that reproductions are costly. **It is evidence +that nobody was ever asked for one**, which is the defect the register +exists to fix and not an argument against the rule. + +## 2. Magic: the correction lands in the source, and the ruling is not authoritative + +The most instructive practice, and it **corrected an assumption this pass +was about to build on**. I expected rulings to be the authoritative +resolution of an ambiguity. They are not. + +The system is three layers, not two: + +| layer | what it is | authoritative? | +|---|---|---| +| Comprehensive Rules | the general rules | yes | +| **Oracle** | the current authoritative text of each *card* — errata folded **in** | **yes** | +| Rulings (via Gatherer) | judge annotations attached to a card | **no** — *"reminder information with no actual weight or rules meaning"* | + +When a card *"causes confusion, has outdated terminology, or doesn't work +as intended, the Oracle text may be updated"* — the fix goes **into the +authoritative text**, and the ruling layer stays explanatory. + +**The property to steal: a finding closes when the source changes, not +when an annotation is added.** A register that accumulates permanent +rulings is a system that has given up on fixing its rules. Ours should be +a **queue that empties**, not an archive that grows. + +This directly shapes the lifecycle in T03: `applied` must mean *the rules +text or the dataset changed and our provisional default was deleted* — not +*a ruling was recorded*. + +*Directional, cited-only: the practice is described, not benchmarked.* + +## 3. Model checkers: the counterexample IS the finding + +A model checker does not report *"this property may not hold."* It emits a +**counterexample trace** — a concrete execution that violates the claim, +replayable by the tool that produced it. + +That is precisely the shape the declaration proposed, arrived at +independently, and it is the strongest argument for the reproduction rule: +in formal methods a claim without a trace is not a result, it is a +hypothesis. Our `gr-e01-threshold-unreachable-2p` scenario is a +counterexample in exactly this sense — it is a failing-in-fact artifact +rather than a paragraph, and CB-EV-0005 already refused to delete it for +that reason. + +**Property to beat: 100% of findings carry a replayable artifact.** + +*Directional, cited-only.* + +## 4. W3C / WHATWG: naming the undecided so it cannot be silently decided + +Specifications mark points as *implementation-defined* or +*implementation-dependent*, which does two things at once: it tells an +implementer they may choose, and it tells the *spec* that a choice is +outstanding. The mark is machine-findable and survives revisions. + +We already have this and it works: `provisional: true` + +`provisional_owner` on a scenario, surfaced by `make coverage` with an +age. It is the one piece of the machinery that is not missing. + +**The register must reuse it, not compete with it.** Six of our findings +already live there; a second mechanism would immediately disagree with the +first. + +*Directional, cited-only.* + +## 5. Benchmarks to beat + +| dimension | today | benchmark | +|---|---|---| +| **findability** | 11 files, no index | **one register with an index**; every finding reachable from one command | +| **reproducibility** | **2/6 = 33%** | **100%**, with withdrawn findings kept in the denominator | +| **closure** | 0 of 10 U-items ruled in 4+ days | a finding closes only when **the source changed** (Magic's Oracle property), and the register says which commit | +| **time raised → read** | **4 days** | the number this exists to fix; any mechanism that does not move it has failed | +| **taxonomy fits reality** | untested | all six existing findings expressible **without** a new kind being invented during backfill | + +## 6. What the survey did not settle + +- **Whether a finding without a reproduction is rejected or admitted as a + note.** §1 shows the rule is affordable for our six, but that is n=6 and + all six are *engine-surfaced*. A finding from **play** — *"the DARVO + sequence feels punishing at 2 players"* — may be real and have no cheap + artifact. T03 must decide, and the honest options are a `note` tier that + can never be reported as a finding, or refusal. +- **Where the register lives.** A finding is about *ground-game's* rules + but is produced by *clay-borg*. Putting it only in clay-borg repeats the + unread-inbox failure; putting it only in ground-game separates it from + its reproduction. Not decided here. +- **Whether the engine-evolution register is redundant.** The + declaration's judgment is that it is. This survey found nothing that + bears on it either way, which is itself worth saying: the practices + above are all about *rules*, and none of them is about how the tool that + found the problem evolved. diff --git a/tools/design-baseline.py b/tools/design-baseline.py new file mode 100755 index 0000000..6e3f83c --- /dev/null +++ b/tools/design-baseline.py @@ -0,0 +1,93 @@ +#!/usr/bin/env python3 +"""Baseline harness: how findable, reproducible and answered are the +design findings this project has already produced? + +The comparator is US, today. The external candidates (rulings databases, +model-checker traces, W3C provisional marks) are practices rather than +runnable software, so per InnerLoop Step 1 their rows are DIRECTIONAL and +cap at `parity`. This is the row that can be measured. +""" +import os, re, subprocess, sys, datetime + +ROOT = "/home/worsch/clay-borg" +os.chdir(ROOT) + +# The findings this project has actually produced, and where each lives. +FINDINGS = { + "U1..U10 underdetermined points": ["specs/GroundRules.md"], + "SOLVE on a face-down Problem": ["workplans/CB-WP-0018-the-browser-is-a-client.md", + "evidence/CB-EV-0016-the-browser-is-a-client.md"], + "GR-A13 wasted SOLVE": ["evidence/CB-EV-0007-stage-0.md"], + "GR-E01 unreachable below 5 seats": ["evidence/CB-EV-0007-stage-0.md", + "scenarios/ground/gr-e01-threshold-unreachable-2p.yaml", + "workplans/CB-WP-0021-import-the-edition.md"], + "six provisional defaults": sorted( + os.path.join("scenarios/ground", f) + for f in os.listdir("scenarios/ground") + if f.endswith(".yaml") + and "provisional: true" in open(os.path.join("scenarios/ground", f)).read()), + "GR-E03/GR-E04 never played": ["evidence/CB-EV-0007-stage-0.md"], +} + +def has_reproduction(paths): + """A runnable thing: a scenario file, or a named test/command.""" + for p in paths: + if p.startswith("scenarios/"): + return True + return False + +def self_test(): + """The control that matters: a harness that read nothing must not + report a clean baseline. Every path this survey cites must exist, and + the reproduction test must be able to say NO — one that answered yes + for everything would report 100% and look excellent.""" + results = [] + + def check(name, ok, detail=""): + results.append((name, ok, detail)) + + missing = [p for paths in FINDINGS.values() for p in paths + if not os.path.exists(p)] + check("every cited location exists", not missing, ", ".join(missing[:3])) + check("the finding set is not empty", len(FINDINGS) >= 6, f"{len(FINDINGS)}") + check("reproduction detection can say NO", + not has_reproduction(["evidence/CB-EV-0007-stage-0.md"]), + "a detector that always says yes would report 100%") + check("reproduction detection can say YES", + has_reproduction(["scenarios/ground/gr-e01-threshold-unreachable-2p.yaml"])) + # The number this survey turns on, pinned so a later edit cannot move + # it silently: 2 of 6 today. + repro_now = sum(has_reproduction(v) for v in FINDINGS.values()) + check("the measured baseline is 2 of 6", repro_now == 2, f"{repro_now}/6") + + print("design-baseline self-test (positive control)") + ok = True + for name, passed, det in results: + print(f" [{'ok ' if passed else 'FAIL'}] {name}" + (f" — {det}" if det else "")) + ok &= passed + return 0 if ok else 1 + + +if "--self-test" in sys.argv: + raise SystemExit(self_test()) + +print("BASELINE — design findings as they stand, 2026-08-03\n") +places = set() +repro = 0 +for name, paths in FINDINGS.items(): + places.update(paths) + r = has_reproduction(paths) + repro += r + print(f" {'repro' if r else ' - '} {len(paths)} location(s) {name}") + +n = len(FINDINGS) +print(f"\n findings {n}") +print(f" with a runnable reproduction {repro}/{n} = {100*repro//n}%") +print(f" distinct files holding them {len(places)}") +print(f" single register? NO — {len(places)} files, no index") + +# Time from raised to READ, for the one finding with a timestamp trail. +raised = datetime.date(2026, 7, 30) # hub message from clay-borg-custodian +read = datetime.date(2026, 8, 3) # marked read this session +print(f"\n U1..U10: raised {raised}, first READ {read} — {(read-raised).days} days") +print(f" U1..U10: answered? NO — {(read-raised).days}+ days open, 0 of 10 ruled") diff --git a/workplans/CB-WP-0022-the-design-instrument.md b/workplans/CB-WP-0022-the-design-instrument.md index 5462d63..d91ddd5 100644 --- a/workplans/CB-WP-0022-the-design-instrument.md +++ b/workplans/CB-WP-0022-the-design-instrument.md @@ -2,7 +2,7 @@ id: CB-WP-0022 kind: product title: "The design instrument: findings about the game, with their reproductions" -status: ready +status: active --- # Purpose @@ -88,7 +88,7 @@ tier L exists to surface. ```task id: CB-WP-0022-T01 -status: todo +status: done priority: high ``` @@ -114,6 +114,33 @@ found*? Candidates worth a benchmark-to-beat: Name, per dimension, the property to beat — findability, reproducibility, and whether a ruling can *close* a finding mechanically. +**Done 2026-08-03.** +[CB-RES-0007](../research/CB-RES-0007-design-instrument.md), with a +runnable baseline (`tools/design-baseline.py`). + +**The baseline is us, and it is measured**: 6 findings across **11 files** +with no index, **2 of 6 (33%) with a runnable reproduction**, and U1–U10 +raised 2026-07-30, first *read* 2026-08-03 — **4 days**, 0 of 10 ruled. + +**The uncomfortable number is stated up front rather than left for the +review to find: the reproduction rule would reject four of our six +existing findings.** The survey answers it — none of the four is +expensive to reproduce, so the 33% is evidence that nobody was ever asked +for one, not that the rule is unaffordable. + +**Magic corrected an assumption this pass was about to build on.** I +expected a ruling to be the authoritative resolution. It is not: rulings +are *"reminder information with no actual weight or rules meaning"*, and +the authoritative fix folds into the **Oracle** card text. **A finding +closes when the source changes, not when an annotation is added** — so +the register must be a queue that empties, not an archive that grows. +That is now a constraint on T03's lifecycle. + +Model checkers supplied the reproduction rule independently (a +counterexample trace *is* the finding), and W3C's *implementation-defined* +mark is the one piece of machinery we already have and must reuse rather +than duplicate. + ## Task: adversarial review ```task