diff --git a/Makefile b/Makefile index 8661f7f..63d0495 100644 --- a/Makefile +++ b/Makefile @@ -125,6 +125,7 @@ self-tests: $(PY) $(TOOLS)/runtime-metrics.py --self-test $(PY) $(TOOLS)/replay-test.py --self-test $(PY) $(TOOLS)/design-baseline.py --self-test + $(PY) $(TOOLS)/design.py --self-test $(PY) $(TOOLS)/edition-check.py --self-test # T01 positive control: prove the environment fix, do not assume it. Runs @@ -160,6 +161,13 @@ facts-check: facts-gen: $(PY) $(TOOLS)/facts.py --gen +# CB-WP-0022 T05: the design-finding register, reported over +# specs/GroundRules.md. Shows the QUEUE by default; the log of closed +# findings is a line, not a listing, because a default view that mixes +# them loses the queue property (ADR-0012 D5). +design: + @$(PY) $(TOOLS)/design.py + # T03: one-shot orientation — workplans, next task, spend, fast gates. # Cheap by design: no build. Start a session with this instead of grepping. status: diff --git a/evidence/CB-EV-0021-the-design-instrument.md b/evidence/CB-EV-0021-the-design-instrument.md new file mode 100644 index 0000000..d1aa71c --- /dev/null +++ b/evidence/CB-EV-0021-the-design-instrument.md @@ -0,0 +1,217 @@ +# CB-EV-0021 — the design instrument + +CB-WP-0022 T07. Tier L, declaration 5 of chaos window 2 (structural L, +chaos d8=6 → no override). Closed 2026-08-05. + +**Delivered:** INTENT property 4 (*Instrument*), +[ADR-0012](../decisions/ADR-0012-the-design-instrument.md) (nine +decisions), [specs/GameDesign.md](../specs/GameDesign.md) v1.0, the +register in [GroundRules.md](../specs/GroundRules.md) (14 rows), +`tools/design.py` + `make design`, and a report file in `ground-game`. + +--- + +## 1. The headline: a separate reviewer found the survey's showcase was false + +This is the first adversarial review in this repo **run by a genuinely +separate agent**. CB-RES-0006's reviewer opened by conceding it could not +be, and wrote: + +> *"Treat the challenges below as a lower bound on what a genuinely +> separate reviewer would find."* + +That was a testable claim. **This is the test.** The separate reviewer ran +`git log` against the survey's central example and found our own commit +had falsified it four days earlier — while the author, who wrote that +commit, quoted the dead number twice in the same workplan. + +`CB-WP-0022:61` justified the whole proposal with *"GR-E01 is admissible +because 4/6/9 against 5/7/9 is a computation anyone can rerun."* +`2da19a4` had rerun it: **6/9/12**, scenario renamed `-unreachable-` → +`-reachable-`, conclusion inverted. And GR-E01 was **one of the two +findings that passed the reproduction rule**. + +**Verdict on same-session review: it is not a weak form of the control, it +is a different control.** Four of the seven challenges rest on facts a +same-session reviewer had already seen and not questioned. The cost was +~70k subagent tokens; it caught a defect in flight (§2) and rewrote the +pass's central rule. + +## 2. What it caught in flight + +T06's payload, status `todo`, still read *"the deal count puts 4/6/9 +points in play … no dataset reconciles them"* and was queued for +transmission to `ground-game`. + +It would have been the **fourth** wrong premise sent downstream, after +*"12 in the file"* (a sum with no deal table) and *"SOLVE offered on a +face-down Problem"* (the wrong condition named). **It is the only one +stopped before sending.** + +| wrong premise | reached ground-game? | passed an existence test? | +|---|---|---| +| *"12 in the file"* | yes | — | +| *"SOLVE on a face-down Problem"* | yes | — | +| *"4/6/9, no dataset reconciles them"* | **stopped 2026-08-05** | **yes** | + +## 3. The rule the pass came to build was one clause short + +The declaration's load-bearing rule was an **existence** test. Three of +three wrong premises would have passed it, and one did. + +What shipped instead (GameDesign §1): a reproduction must **exist**, have +the **shape ground-game ruled** (row-level deal table, never a sum — +promoted from a T04 addendum to part of the rule), and **be able to +fail**. + +**Clause 3 is the one this pass bought.** GR-E01's scenario went green +when the edition landed and the finding stayed admissible and stayed +queued, because nothing treated a passing artifact as a signal. *A green +reproduction is an alarm, not a reassurance.* + +## 4. Building it produced a distinction the ADR does not have + +`make design`'s first run alarmed on **U2** — and was wrong to. + +U2's scenario is green **because the provisional default it documents is +implemented**, which says nothing about whether ground-game agrees. +GR-E01's was a *counterexample* that went green. **Same colour, opposite +meaning.** + +So the register gained a `role` column — `counterexample` | `default` — +and only a green counterexample alarms. **A register that cannot tell them +apart either alarms constantly or never**, and this was invisible until +the metric ran against real rows. Folded back into GameDesign §1.3. + +## 5. Did backfilling change the design? + +**T07 asked to be suspicious if all six findings fit the first taxonomy.** +They did — five kinds absorbed everything, no sixth invented. Two reasons +that is less reassuring than it looks and one reason it is fine: + +- the taxonomy was **derived from these six**, so fitting them is nearly + circular. Its real test is the seventh finding; +- **one of the six was deleted during backfill.** *"Six provisional + defaults"* was not entered: C3 showed it double-counted GR-E01, and + provisional scenarios are *reproductions for* underdetermined items, not + a finding. So the taxonomy absorbed five, not six; +- what it did do is **force the note tier to earn itself** — GR-A13 and + GR-E03/E04 have no artifact and became notes, which is the first time + the distinction cost anything. + +## 6. The numbers, and why none of the survey's survive + +**CB-RES-0007's measurements are withdrawn.** C2 dismantled the +instrument: `design-baseline.py` is a hand-maintained dict where +`n = len(FINDINGS)`; `has_reproduction` is `startswith("scenarios/")` and +never stats the file, so its own YES-control was **green against a path +`2da19a4` deleted**; the 4-day figure is two date literals subtracted +(mutated to 2020-01-01 → 2406 days, all checks still green); and +`Makefile:127` ran only `--self-test`, so the reporting path had no CI. + +It is **superseded, not deleted** — it is the evidence for how a wrong +number got into a survey, and the header now says so. + +Measured by `make design`, 2026-08-05, over rows that name real files: + +``` +findings 12 (+2 notes) +with a resolving reproduction 3/12 = 25% target 100% +open, lacking a reproduction 9 target 0 +reproductions green while open 0 target 0 +notes past 30 days 0 target 0 +closed (log) 2 [F11, F13] +``` + +**Two survey claims died against measurement:** + +| survey said | measured | +|---|---| +| *"six of the ten already have provisional scenarios"* | **one.** `grep -lE "\bU\b" scenarios/ground/*.yaml` → only U2 | +| *"2/6 = 33% reproduced"* | withdrawn; GR-E01 was double-counted and the detector was broken | + +The first is the same defect class as the wrong premises — **an +uncheckable link asserted as a fact** — found inside the survey that +proposed the fix. + +## 7. The register's first run found ten answers nobody had collected + +The most useful thing `make design` did was not a metric. + +**GROUND-WP-0002 T05 ruled all ten U-items on 2026-08-03 — every one +confirmed.** T03 confirmed five of the six provisional scenarios. The +workplan is `finished`. + +CB-RES-0007 reported *"0 of 10 ruled"* on 2026-08-03, and the register +built this session initially recorded them as `reported`. **Both were two +days stale on the day they were written.** + +**This is the unread-inbox failure running in the opposite direction.** +The pass was designed around *we send findings and nobody reads them*. The +symmetric failure — *they answer and we do not collect it* — was not +anticipated by the declaration, the survey, or the review, and the first +run of the instrument surfaced it immediately. + +The items are `ruled`, not `applied`: the rulings confirmed our defaults, +so the rules did not move, but the scenarios still carry `provisional: +true` for settled choices. **Lifting those flags is owed and is not done** +— `make design` shows them open until it is, which is the register +working. + +## 8. The engine-evolution question, as the review left it + +Declared judgment: no second register, because engine evolution already +lives in `evidence/`, `decisions/`, `gates.toml` and workplans. + +**The conclusion survives; the argument did not.** C5 found the "third +thing" the maintainer meant is visible in `specs/InnerLoopReference.md` +(383 lines, two sections literally headed *"moved from InnerLoop v1.6"*) +and 769 lines of `history/` retrospectives — **neither of which the +inventory named**. So the content exists and found homes somewhat by +accident. + +ADR-0012 D7 records it as **argued and narrowed, not settled**, with the +revisit condition: if InnerLoopReference keeps absorbing material that is +neither a decision nor a finding, build the register. + +## 9. What tier L cost against what it caught + +Second full-weight L pass in this window, third in the project. + +| caught | by | +|---|---| +| the central example was false | **only** the separate reviewer | +| a wrong premise in flight to ground-game | the same | +| the baseline instrument counts itself | the same | +| the incumbent register was never evaluated | the same | +| `role`: green default ≠ green counterexample | **building it**, not reviewing it | +| ten uncollected rulings | **running it**, not building it | + +**Two of six were caught by execution, not by process** — the pattern +InnerLoop §Design goal predicts (cheap correction beats exhaustive +prevention), and an argument against front-loading more review rather than +less. + +**The one number that matters for the loop's own calibration:** the review +step is what distinguished this pass from a competent one. Without it the +pass ships a rule justified by a false example, sends a fourth bad premise +downstream, and builds `make design` on a dict that counts itself. The +correct reading of CB-RES-0006's fidelity note is **not** *"our reviews +were somewhat weaker"* — it is *"our reviews were not performing this +function at all."* + +## 10. Cost + +CB-WP-0021's cost, by re-running the instrument: see `make cost` / +`tools/cb-cost.py`. **Not quoted here as a literal** — §Single source of +fact, and this pass has just spent itself proving what typed numbers do. + +## Open after this pass + +- **U1–U10 → `applied`**: lift `provisional: true` where a ruling confirmed + the default, and record each ruling. Nine items of bookkeeping. +- **The U-item ↔ scenario mapping is unwritten** for 9 of 10. Cheap, and + it is what makes the register checkable. +- **F12 and F14 are notes** and expire 2026-08-31. GROUND-WP-0003's + playtest closes F14 and produces its artifact via `cb-play --record`. +- **The taxonomy's real test is the seventh finding**, not these six. diff --git a/specs/GameDesign.md b/specs/GameDesign.md index 9dd605f..4aeea8c 100644 --- a/specs/GameDesign.md +++ b/specs/GameDesign.md @@ -61,6 +61,20 @@ The artifact must be capable of going red, and the register records its finding** — it is a finding whose claim is now in question, and §4 reports it as needing resolution to `applied` or `withdrawn`. +**A reproduction has a `role`, and only one of them alarms.** This +distinction was not in ADR-0012; **T05's backfill produced it**, when the +first report cried wolf over U2. + +| role | what it is | green means | +|---|---|---| +| `counterexample` | demonstrates the defect — the model-checker shape | **alarm.** It should be red while the finding is open | +| `default` | encodes a provisional choice so it is testable and visible | **expected.** It passes because our default is implemented, which says nothing about whether the owner agrees | + +U2's scenario is green because the default it documents is implemented. +GR-E01's was a counterexample that went green when the edition landed, and +sat that way for four days. **Same colour, opposite meaning** — a register +that cannot tell them apart either alarms constantly or never. + **Rationale, and it is the expensive one.** `gr-e01-threshold-…-2p.yaml` went green when the edition landed on 2026-08-04. The finding remained admissible, remained in the register, and remained queued for transmission diff --git a/specs/GroundRules.md b/specs/GroundRules.md index 2ffad75..b053a6f 100644 --- a/specs/GroundRules.md +++ b/specs/GroundRules.md @@ -219,6 +219,103 @@ terms. --- +## The finding register + +**This section is the design-finding register** (ADR-0012 D2). It was the +register for dataset ambiguities already; CB-WP-0022 extended it to all +five kinds rather than building a second one beside it. Admissibility, +kinds, states and metrics: [`GameDesign.md`](GameDesign.md). Reported by +`make design`. + + + +| id | kind | state | reproduction | role | raised | owner | +|---|---|---|---|---|---|---| +| U1 | underdetermined | ruled | — | default | 2026-07-31 | ground-game | +| U2 | underdetermined | ruled | scenarios/ground/gr-d01-darvo-trigger.yaml | default | 2026-07-31 | ground-game | +| U3 | underdetermined | ruled | — | default | 2026-07-31 | ground-game | +| U4 | underdetermined | ruled | — | default | 2026-07-31 | ground-game | +| U5 | underdetermined | ruled | — | default | 2026-07-31 | ground-game | +| U6 | underdetermined | ruled | — | default | 2026-07-31 | ground-game | +| U7 | underdetermined | ruled | — | default | 2026-07-31 | ground-game | +| U8 | underdetermined | ruled | — | default | 2026-07-31 | ground-game | +| U9 | underdetermined | ruled | — | default | 2026-07-31 | ground-game | +| U10 | underdetermined | ruled | — | default | 2026-07-31 | ground-game | +| F11 | inert | applied | scenarios/ground/gr-p05-solve-legality.yaml | counterexample | 2026-08-02 | clay-borg | +| F12 | degenerate | note | — | — | 2026-08-01 | clay-borg | +| F13 | inconsistent | withdrawn | scenarios/ground/gr-e01-threshold-reachable-2p.yaml | counterexample | 2026-08-01 | clay-borg | +| F14 | unplayed | note | — | — | 2026-08-01 | clay-borg | + + + +- **F11 — SOLVE offered where it cannot act.** Offered on a face-down + Problem, or with no matching suit in hand; inert every time. Ruled + GROUND-WP-0002 T02, implemented CB-WP-0023 as GR-P05. `applied` — the + rule changed, not just the annotation. **The case we reported was not the + case that fired**: `validate` already rejected face-down, and the + maintainer's three inert SOLVEs were the *hand* case. +- **F12 — GR-A13 "wasted SOLVE" on an already-claimed Problem.** A + scenario had to pick a default and did. `note`: no artifact isolates the + degenerate line, so under GameDesign §3.1 it may not be reported until + one exists. +- **F13 — GR-E01 vs GR-S01, withdrawn 2026-08-05.** Raised as *"4/6/9 + against 5/7/9, no dataset reconciles them."* `2da19a4` measured **6/9/12 + against 5/7/9** and the scenario was renamed `-unreachable-` → + `-reachable-`. **Its reproduction is green**, which under GameDesign §1.3 + is the alarm that forced the resolution. Withdrawn rather than deleted, + and the withdrawal is reported (ADR-0012 D5). +- **F14 — GR-E03/GR-E04 never played to the end.** Nineteen passes, never + played out. `note` until a trial game exists; GROUND-WP-0003 is the + playtest that would close it, and GameDesign §5's protocol makes the + recording the artifact. + +### The register's first run found ten answers nobody had collected + +**U1–U10 are `ruled`, not `reported`.** GROUND-WP-0002 T05 answered all +ten on **2026-08-03** — every one **confirmed** as the default clay-borg +already simulates — and GROUND-WP-0002 T03 confirmed five of the six +provisional scenarios, voiding `gr-e01` as a rules gap. The workplan is +`finished`. + +CB-RES-0007 reported *"0 of 10 ruled"* and this register was built saying +`reported`. **Both were two days stale on the day they were written.** The +answers had arrived and nothing propagated them — the same failure as the +unread inbox, in the opposite direction. + +**They are `ruled`, not `applied`, and the difference is work we owe.** +Per ADR-0012 D5, `applied` means the source changed and the provisional +default was deleted. The rulings *confirmed* our defaults, so the rules did +not move — but the scenarios still carry `provisional: true` for choices +that are now settled. **Lifting those flags and recording each ruling is +what closes U1–U10**, and it is not done. `make design` shows them open +until it is. + +### What the backfill measured, and what it contradicted + +**Only U2 names its U-item in a scenario.** Measured, not estimated: + +``` +for u in U1 .. U10; do grep -lE "\b$u\b" scenarios/ground/*.yaml; done +``` + +CB-RES-0007 asserted *"six of the ten already have provisional +scenarios."* Five provisional scenarios exist and **one** cites the item +it stands for. The other four may well encode U-item defaults — the +mapping is simply **not written down**, so it is not checkable, and an +uncheckable link is the defect this register exists to fix. The register +records what is citable; the rest is debt, visible in `make design`. + +**No sixth kind was needed** — the five kinds absorbed all four non-U +findings. And the survey's *"six provisional defaults"* was **not** entered +as a finding: C3 showed it double-counted GR-E01, and the provisional +scenarios are *reproductions for* underdetermined items, not a finding of +their own. + +**U1–U10 are `reported` while lacking reproductions**, which GameDesign +§3.1 would now forbid. They were reported on 2026-07-30, before the rule +existed. They are grandfathered rather than rewritten, and the debt is a +reported metric with a target of zero. + ## Underdetermined in dataset 0.1 — PROVISIONAL defaults (flag to ground-game) **Owner: ground-game. Raised: 2026-07-31.** Every U-item carries an owner diff --git a/tools/design-baseline.py b/tools/design-baseline.py index 7102d2a..46b4551 100755 --- a/tools/design-baseline.py +++ b/tools/design-baseline.py @@ -7,6 +7,26 @@ model-checker traces, W3C provisional marks) are practices rather than runnable software, so per InnerLoop Step 1 their rows are DIRECTIONAL and cap at `parity`. This is the row that can be measured. """ +# +# ── SUPERSEDED 2026-08-05 by tools/design.py (ADR-0012 D8) ────────────── +# +# Kept as CB-RES-0007's dated snapshot, NOT as a live instrument. Deleting +# it would erase the evidence for how a wrong number got into a survey. +# +# The adversarial review (C2) dismantled it and every point is true: +# * FINDINGS below is a hand-maintained dict; `n = len(FINDINGS)` counts +# the rows someone typed, not findings that exist. +# * `has_reproduction` is `startswith("scenarios/")` and never stats the +# file, so the self-test's YES-control is GREEN against a path that +# 2da19a4 deleted. A positive control that controls nothing. +# * It rejects a named test, which the admissibility rule admits, so its +# 33% is biased downward. +# * The 4-day figure is two date literals subtracted. Mutated to +# 2020-01-01 it reports 2406 days with every check still green. +# +# **Its 33% is withdrawn as a measured rate.** What survived: 11 files, +# no index, 0 of 10 ruled — all checkable without this tool. +# import os, re, subprocess, sys, datetime ROOT = "/home/worsch/clay-borg" diff --git a/tools/design.py b/tools/design.py new file mode 100644 index 0000000..730a653 --- /dev/null +++ b/tools/design.py @@ -0,0 +1,194 @@ +#!/usr/bin/env python3 +"""design — report the finding register (CB-WP-0022 T05). + +Implements the metrics in `specs/GameDesign.md` §4 over the register in +`specs/GroundRules.md`. ADR-0012 D8 retired `design-baseline.py`, which +was a hand-maintained dict counting itself; the difference that matters is +that **every number here is computed over rows that name real files, and +the reproduction check stats the file.** + +`design-baseline.py`'s `has_reproduction` was `p.startswith("scenarios/")` +and nothing else, so its own positive control was green against a path +that had been deleted. The self-test below asserts the opposite property +directly: a row citing a nonexistent file must NOT count as reproduced. +""" + +import os, re, sys, subprocess, datetime + +ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +REGISTER = os.path.join(ROOT, "specs", "GroundRules.md") + +BEGIN = "" +END = "" + +KINDS = {"underdetermined", "inconsistent", "inert", "degenerate", "unplayed"} +OPEN_STATES = {"raised", "reported", "ruled"} # the queue: not yet applied +CLOSED_STATES = {"applied", "withdrawn"} # the log +NOTE = "note" +NOTE_EXPIRY_DAYS = 30 # GameDesign §3.1, same figure as rule-coverage.py + + +def parse(text): + """Rows between the register markers. Raises if the block is absent — + a register that silently reports zero findings is worse than one that + fails.""" + try: + block = text.split(BEGIN)[1].split(END)[0] + except IndexError: + raise SystemExit(f"no design-register block in {REGISTER}") + rows = [] + for line in block.splitlines(): + line = line.strip() + if not line.startswith("|") or line.startswith("|---"): + continue + cells = [c.strip() for c in line.strip("|").split("|")] + if len(cells) != 7 or cells[0] in ("id",): + continue + rows.append(dict(zip( + ("id", "kind", "state", "repro", "role", "raised", "owner"), cells))) + return rows + + +def reproduced(row, root=ROOT): + """GameDesign §1.1: the artifact must resolve. A named test is admitted + by its `crate::module::name` shape; anything else must be a real path + on disk, and that is checked by stat, not by prefix.""" + p = row["repro"] + if p in ("", "—", "-"): + return False + if "::" in p: # a named test + return True + return os.path.exists(os.path.join(root, p)) + + +_SIM = None + + +def sim_passes(root=ROOT): + """`make sim` is the authority on whether scenarios pass. Run once and + cache — asking per row costs a full suite per finding.""" + global _SIM + if _SIM is None: + r = subprocess.run(["make", "sim"], cwd=root, capture_output=True, text=True) + _SIM = r.returncode == 0 + return _SIM + + +def alarming(row, root=ROOT): + """§1.3, and the distinction the backfill discovered: **only a + counterexample alarms when green.** + + A `default` reproduction encodes a provisional choice and is SUPPOSED + to pass — U2's scenario is green because the default it documents is + implemented, which says nothing about whether ground-game agrees. A + `counterexample` is the model-checker shape: it demonstrates the + defect, so it must be red while the finding is open. GR-E01's went + green when the edition landed and nothing noticed for four days. + """ + if row["role"] != "counterexample": + return False + if not row["repro"].startswith("scenarios/"): + return False + return sim_passes(root) + + +def report(root=ROOT, today=None): + today = today or datetime.date.today() + rows = parse(open(os.path.join(root, "specs", "GroundRules.md")).read()) + + findings = [r for r in rows if r["state"] != NOTE] + notes = [r for r in rows if r["state"] == NOTE] + queue = [r for r in findings if r["state"] in OPEN_STATES] + closed = [r for r in findings if r["state"] in CLOSED_STATES] + + print("design — the finding register\n") + print(" QUEUE (open findings)") + for r in sorted(queue, key=lambda r: r["raised"]): + age = (today - datetime.date.fromisoformat(r["raised"])).days + mark = "repro" if reproduced(r, root) else " - " + print(f" {mark} {r['id']:<4} {r['kind']:<16} {r['state']:<9} {age:>4}d {r['owner']}") + + if notes: + print("\n NOTES (not reportable — GameDesign §3.1)") + for r in sorted(notes, key=lambda r: r["raised"]): + age = (today - datetime.date.fromisoformat(r["raised"])).days + flag = " EXPIRED" if age > NOTE_EXPIRY_DAYS else "" + print(f" {r['id']:<4} {r['kind']:<16} {age:>4}d{flag}") + + # ---- §4 metrics. The log is reported on request, not by default, + # because a default view mixing open and closed loses the queue + # property (ADR-0012 D5). + n = len(findings) + repro = sum(1 for r in findings if reproduced(r, root)) + debt = [r for r in queue if not reproduced(r, root)] + expired = [r for r in notes + if (today - datetime.date.fromisoformat(r["raised"])).days > NOTE_EXPIRY_DAYS] + unresolved_green = [r for r in queue if reproduced(r, root) and alarming(r, root)] + + print(f"\n findings {n} (+{len(notes)} note(s))") + print(f" with a resolving reproduction {repro}/{n}" + f"{'' if not n else f' = {100*repro//n}%'} target 100%") + print(f" open, lacking a reproduction {len(debt)} target 0" + + (f" [{', '.join(r['id'] for r in debt)}]" if debt else "")) + print(f" reproductions green while open {len(unresolved_green)} target 0" + + (" <-- ALARM, GameDesign §1.3" if unresolved_green else "")) + print(f" notes past {NOTE_EXPIRY_DAYS} days {len(expired)} target 0") + print(f" closed (log) {len(closed)}" + f" [{', '.join(r['id'] for r in closed)}]" if closed else "") + + bad = [r for r in rows if r["kind"] not in KINDS] + if bad: + print(f"\n UNKNOWN KIND: {', '.join(r['id'] for r in bad)}" + " — a sixth kind means the taxonomy was invented (ADR-0012 D4)") + return 0 + + +def self_test(): + ok = True + + def check(name, cond, detail=""): + nonlocal ok + ok = ok and bool(cond) + print(f" {'ok ' if cond else 'FAIL'} {name}{' ' + detail if detail else ''}") + + rows = parse(open(REGISTER).read()) + check("the register parses", len(rows) >= 14, f"{len(rows)} row(s)") + check("every kind is one of the five", + all(r["kind"] in KINDS for r in rows), + "a sixth kind means the taxonomy was invented") + check("every state is known", + all(r["state"] in OPEN_STATES | CLOSED_STATES | {NOTE} for r in rows)) + + # THE control design-baseline.py did not have. Its YES-control passed a + # path that had been deleted and still returned True. + check("a nonexistent reproduction does NOT count", + not reproduced({"repro": "scenarios/ground/gr-e01-threshold-unreachable-2p.yaml"}), + "this exact path was deleted by 2da19a4 and the old tool said yes") + check("a real reproduction DOES count", + reproduced({"repro": "scenarios/ground/gr-p05-solve-legality.yaml"})) + check("a named test counts", reproduced({"repro": "games_ground::view::tests::a_spectator_sees_no_hands"})) + check("an em-dash does not count", not reproduced({"repro": "—"})) + + # The distinction the backfill discovered: a green DEFAULT is expected, + # a green COUNTEREXAMPLE is the alarm. Without this the report cried + # wolf over U2, whose scenario is green precisely because the + # provisional default it documents is implemented. + check("a green default does not alarm", + not alarming({"role": "default", "repro": "scenarios/ground/gr-d01-darvo-trigger.yaml"})) + check("role is recorded for every reproduced row", + all(r["role"] in ("counterexample", "default") + for r in rows if r["repro"] not in ("—", "-", ""))) + + # The measured claim in the backfill: only U2 cites its U-item. + cited = [u for u in [f"U{i}" for i in range(1, 11)] + if subprocess.run(["grep", "-rlE", rf"\b{u}\b"] + + [os.path.join(ROOT, "scenarios", "ground")], + capture_output=True, text=True, cwd=ROOT).stdout.strip()] + check("exactly one U-item is cited by a scenario", cited == ["U2"], f"{cited}") + + print("design self-test (positive control)") + return 0 if ok else 1 + + +if __name__ == "__main__": + sys.exit(self_test() if "--self-test" in sys.argv else report()) diff --git a/workplans/CB-WP-0022-the-design-instrument.md b/workplans/CB-WP-0022-the-design-instrument.md index c1ba327..fa67fa0 100644 --- a/workplans/CB-WP-0022-the-design-instrument.md +++ b/workplans/CB-WP-0022-the-design-instrument.md @@ -2,7 +2,7 @@ id: CB-WP-0022 kind: product title: "The design instrument: findings about the game, with their reproductions" -status: active +status: done state_hub_workstream_id: "fda16340-0049-4acf-884b-a5cfdbde47c0" --- @@ -36,8 +36,8 @@ has produced, as a by-product of being rigorous: | ten underdetermined rules points (U1–U10) | formalizing the dataset into testable rules | | SOLVE offered on a face-down Problem, always inert | a human dragging it three rounds running | | GR-A13 "wasted SOLVE" on a claimed Problem | a scenario that had to pick a default | -| GR-E01 unreachable below 5 seats | arithmetic over the deal count | -| six provisional scenario defaults | scenarios that could not be written without deciding something | +| ~~GR-E01 unreachable below 5 seats~~ | arithmetic over the deal count — **withdrawn 2026-08-05, it was wrong (C1)** | +| ~~six provisional scenario defaults~~ | **five, and double-counted with GR-E01 (C3)** — they are reproductions, not a finding | | GR-E03 / GR-E04 never played to the end | nobody noticed for nineteen passes | Every one was found by *building the simulator*, not by playing. That is @@ -79,19 +79,19 @@ The maintainer asked whether this should extend to *"a meta about the clay-borg engine evolution itself."* **My answer is no, and it should be argued rather than accepted.** That -register already exists and is load-bearing: `evidence/CB-EV-*` records -what each pass found, `decisions/ADR-*` records what was decided and what -was rejected, `gates.toml` records what every control has caught and what -would retire it, and workplans record what was attempted. Nineteen passes -of engine evolution are already captured, with dates, costs and -falsifiers. +register already exists and is load-bearing: `evidence/CB-EV-*`, +`decisions/ADR-*`, `gates.toml` and the workplans capture nineteen passes +with dates, costs and falsifiers. A second register for the same subject +would be ceremony. The asymmetry is the point: engine evolution has a home +and game design does not. -**Building a second register for the same subject would be ceremony**, and -this project has a standing rule that a gate must cash out. The asymmetry -is the point: engine evolution has a home and game design does not. - -If the adversarial review disagrees, that is exactly the kind of thing -tier L exists to surface. +> **Settled by [ADR-0012](../decisions/ADR-0012-the-design-instrument.md) +> D7: no register — but the argument above did not survive.** C5 found the +> "third thing" the maintainer meant is visible in +> `specs/InnerLoopReference.md` and `history/`'s retrospectives, **neither +> of which this inventory names.** Conclusion narrowed, not settled: if +> InnerLoopReference keeps absorbing material that is neither a decision +> nor a finding, revisit. ## Task: survey how this is done elsewhere, and what we already have @@ -128,28 +128,21 @@ and whether a ruling can *close* a finding mechanically. [CB-RES-0007](../research/CB-RES-0007-design-instrument.md), with a runnable baseline (`tools/design-baseline.py`). -**The baseline is us, and it is measured**: 6 findings across **11 files** -with no index, **2 of 6 (33%) with a runnable reproduction**, and U1–U10 -raised 2026-07-30, first *read* 2026-08-03 — **4 days**, 0 of 10 ruled. +**Its numbers were withdrawn by T02 and must not be quoted from here.** +The survey reported *6 findings, 2/6 = 33% reproduced, 11 files, 4 days*. +C2 showed the instrument counted itself and its reproduction check never +stat'd the file; C3 showed *"six provisional defaults"* is five and GR-E01 +was double-counted; T05's backfill contradicted *"six of the ten have +provisional scenarios"* — **one** does. What survives is direction: many +files, no index, 0 of 10 ruled. The first honest figures are T05's. -**The uncomfortable number is stated up front rather than left for the -review to find: the reproduction rule would reject four of our six -existing findings.** The survey answers it — none of the four is -expensive to reproduce, so the 33% is evidence that nobody was ever asked -for one, not that the rule is unaffordable. - -**Magic corrected an assumption this pass was about to build on.** I -expected a ruling to be the authoritative resolution. It is not: rulings -are *"reminder information with no actual weight or rules meaning"*, and -the authoritative fix folds into the **Oracle** card text. **A finding -closes when the source changes, not when an annotation is added** — so -the register must be a queue that empties, not an archive that grows. -That is now a constraint on T03's lifecycle. - -Model checkers supplied the reproduction rule independently (a -counterexample trace *is* the finding), and W3C's *implementation-defined* -mark is the one piece of machinery we already have and must reuse rather -than duplicate. +**Magic corrected an assumption this pass was about to build on.** Rulings +are *"reminder information with no actual weight or rules meaning"*; the +authoritative fix folds into the **Oracle** card text. **A finding closes +when the source changes, not when an annotation is added** — the register +is a queue that empties. Model checkers supplied the reproduction rule +independently, and W3C's *implementation-defined* mark is machinery we +already have and must reuse rather than duplicate. ## Task: adversarial review @@ -169,12 +162,9 @@ above, and require an attempt at: third thing; - **that "carries its reproduction" is affordable** — if half the real findings cannot be reproduced cheaply, the rule will be quietly dropped - and the register becomes a suggestion box anyway. *(Since this task was - written the answer has hardened: two findings reached ground-game on - wrong premises, and ground-game has itself ruled that reproductions are - required. The reviewer should now press the opposite way — whether the - rule is **sufficient**, given both bad premises would have shipped a - passing test alongside a wrong description.)* + and the register becomes a suggestion box anyway. *(Hardened before the review ran: two findings had already reached + ground-game on wrong premises, so the reviewer was told to press whether + the rule is **sufficient**, not whether it is affordable.)* - **that a register is needed at all**, rather than one more section in `GroundRules.md §Underdetermined`, which already exists and already works. @@ -194,24 +184,14 @@ had falsified it four days earlier, while the author — who wrote that commit — quoted the dead number twice. **Seven challenges: four conceded, two conceded in part, one answered.** - -- **C1 lands hardest and changed the design.** The rule's showcase finding - was false and had *passed* the rule, so existence is not the missing - property — **shape** and **falsifiability** are. It also **caught a - defect in flight**: T06's payload, `todo`, still named the dead number. - Both folded in above. -- **C2 withdrew the baseline's precision.** `design-baseline.py` counts - itself, its reproduction check never stats the file, and `Makefile:127` - runs only `--self-test`. Direction stands; **33% is not a measured rate.** -- **C3**: "six provisional defaults" is five; GR-E01 is double-counted. -- **C4**: `§Underdetermined` was never evaluated as a candidate and already - delivers four of five benchmarks. **T03's burden flips: argue extension, - not replacement.** -- **C5**: the engine-evolution "third thing" is visible in - `specs/InnerLoopReference.md` and `history/`'s retrospectives, neither of - which my inventory named. Narrowed, not settled. -- **Survived**: the rule's *affordability*, and reuse of the provisional - machinery — both with stated falsifiers. +**C1 changed the design** — the rule's showcase finding was false and had +*passed* the rule, so existence is not what was missing — and **caught a +defect in flight**, T06's payload still naming the dead number. C2 +withdrew the baseline's precision, C3 its arithmetic, C4 flipped T03's +burden toward extending `§Underdetermined`, C5 corrected the redundancy +inventory. Survived: affordability, and reuse of the provisional +machinery. Full account: +[CB-EV-0021](../evidence/CB-EV-0021-the-design-instrument.md) §1–§3. ## Task: decide @@ -284,56 +264,40 @@ Candidate measures, to be argued not adopted: **ground-game has ruled on what a finding must carry** (GROUND-WP-0004 T02, 2026-08-04), and it is stricter than this pass proposed. Adopt it: -> 1. Arithmetic findings must ship a **runnable reproduction** *and* a -> **row-level deal table** listing Surface and each hidden priority -> separately — **never only "sum of file" or "deal depth N"**. -> 2. ground-game rulings that depend on arithmetic **should cite** that -> reproduction id or path, so a later reader can re-run it. +> Arithmetic findings ship a **runnable reproduction** *and* a +> **row-level deal table** — never only "sum of file" or "deal depth N"; +> and ground-game's arithmetic rulings cite that reproduction by path. -The second half is theirs to keep. The first is ours, and it is a direct -consequence of both premises we got wrong: *"12 in the file"* was a sum -without a deal table, and *"offered on a face-down Problem"* was a -condition named without checking which one actually fired. +The second half is theirs to keep. **So the reproduction rule gains a +shape requirement, not just an existence one** — a finding that ships a +passing test but describes the wrong quantity is still a bad finding, +which is exactly what happened twice. -**So the reproduction rule gains a shape requirement, not just an -existence one.** A finding that ships a passing test but describes the -wrong quantity is still a bad finding — which is exactly what happened -twice. The spec must say what a reproduction has to *show*, not only that -one exists. +Also specify the **trial protocol**: a trial game is a `--record`ed +session plus an observation log, so *"we played it and X happened"* is +replayable rather than remembered. It must cost almost nothing or it will +not be done. -Also specify the **trial protocol**, which is the part with no precedent -here: a trial game is a `--record`ed session plus an observation log, so -*"we played it and X happened"* is replayable rather than remembered. The -engine already records sessions as scenarios; a trial is that plus notes, -and it must cost almost nothing or it will not be done. - -**Done 2026-08-05.** [specs/GameDesign.md](../specs/GameDesign.md) v1.0. - -**It is not a register, and says so in its second line** — ADR-0012 D2 put -that in `§Underdetermined`. +**Done 2026-08-05.** [specs/GameDesign.md](../specs/GameDesign.md) v1.0 — +not a register (ADR-0012 D2 put that in `§Underdetermined`). **§1.2 is written against evidence rather than principle**: a finding must -print the rows behind any number it claims, and the spec carries the table -of what shipped instead — a sum, a green scenario, and a condition named -without checking which one fired. *"12" was arithmetically defensible and -still wrong about the game.* +print the rows behind any number it claims. *"12" was arithmetically +defensible and still wrong about the game.* **§1.3's target is `0` +reproductions gone green while open** — what GR-E01 would have tripped +four days before a human caught it. **No baseline rate is quoted.** -**§1.3's target is `0` reproductions gone green while open** — what GR-E01 -would have tripped four days before a human caught it. **No baseline rate -is quoted**; the first honest denominator is T05's backfill. - -**The trial protocol costs one flag.** `cb-play --record` already writes a -finished game as a scenario, so a trial is that plus a sibling `.md` in -the player's own words. An observation is a **note** until it has a -reproduction — the maintainer's *"I felt it was too easy but then we -lost"* is the case it is shaped around, and forcing that into a schema at -the moment of observation would lose it. +**The trial protocol costs one flag**: `cb-play --record` plus a sibling +`.md` in the player's own words. An observation is a **note** until it has +a reproduction — *"I felt it was too easy but then we lost"* is the case +it is shaped around, and a schema at the moment of observation would lose +it. ## Task: build it, and backfill what is already known ```task id: CB-WP-0022-T05 -status: todo +status: done priority: high state_hub_task_id: "469413ba-ed47-4840-a60e-ee7f95dc06ff" ``` @@ -346,44 +310,63 @@ designing it is the point of doing it in this order. `make design` (or equivalent) must report: open findings by kind, those without a reproduction, and those never reported to their owner. +**Done 2026-08-05.** `tools/design.py`, `make design`, and the register in +[`GroundRules.md`](../specs/GroundRules.md) — **14 rows, no new file.** + +Backfill was the test. The taxonomy held (five kinds, no sixth), and it +**produced a `role` column ADR-0012 does not have**: the first report +alarmed on U2, wrongly — a green *default* is expected, a green +*counterexample* is the alarm. Folded into GameDesign §1.3. It also +contradicted the survey: **one** U-item names itself in a scenario, not +six. Detail and figures: +[CB-EV-0021](../evidence/CB-EV-0021-the-design-instrument.md) §4, §6. + ## Task: report to ground-game, mechanically ```task id: CB-WP-0022-T06 -status: todo +status: done priority: high state_hub_task_id: "c6a659bd-c495-4754-9332-baadad51012a" ``` Generate the report and send it. **The message that sat unread for four -days is the baseline to beat**, and the failure was not the message — it -was that nothing pointed at it and nothing tracked whether it was -answered. - -So the report must land somewhere that persists: a file in `ground-game` -under its own workplan, not only an inbox entry. GROUND-WP-0002 already -holds the ten U-items; this should extend it rather than duplicate it. +days is the baseline to beat** — the failure was not the message, it was +that nothing pointed at it. So the report lands as a file in `ground-game` +under its own workplan, extending GROUND-WP-0002 rather than duplicating +it. Include the findings this pass has sharpened: - **SOLVE's legality** against a face-down Problem or an unmatchable suit — and note that the case we *reported* was not the case that fired (CB-WP-0023 T01). -- ~~**GR-E01 vs GR-S01** — 4/6/9 against 5/7/9, no dataset reconciles - them~~ — **withdrawn 2026-08-05, before sending.** The adversarial - review (C1) found `2da19a4` had already measured **6/9/12 against - 5/7/9**: the dataset reconciles them and the scenario is now - `-reachable-`. Sending this would have been the **fourth** wrong premise - to reach `ground-game`, and the only one caught before transmission. - **Report the withdrawal, not the finding** — GROUND-WP-0002 holds the - original, and a claim retracted silently is how the first three - survived. +- ~~**GR-E01 vs GR-S01** — 4/6/9 against 5/7/9~~ — **withdrawn + 2026-08-05, before sending.** C1 found `2da19a4` had measured **6/9/12**: + the dataset reconciles them. It would have been the **fourth** wrong + premise to reach `ground-game` and is the only one caught before + transmission. **Report the withdrawal** — a claim retracted silently is + how the first three survived. + +**Done 2026-08-05.** +[`ground-game/workplans/GROUND-WP-0002-clay-borg-report-260805.md`](../../ground-game/workplans/GROUND-WP-0002-clay-borg-report-260805.md), +committed there, with a hub message that only *points at* the file. + +**The report asks for no ruling.** It carries GR-E01's withdrawal, our own +reproduction debt, and two notes that are explicitly not findings. + +**And it acknowledged something the pass did not expect.** +GROUND-WP-0002 is `finished`: **all ten U-items were ruled 2026-08-03**, +every one confirmed as the default we simulate. CB-RES-0007 said *"0 of 10 +ruled"* two days later. **The unread-inbox failure running in the opposite +direction** — they answered and we did not collect it. The instrument's +first run surfaced it. ## Task: evidence ```task id: CB-WP-0022-T07 -status: todo +status: done priority: high state_hub_task_id: "4fb77911-486d-4d9e-a53d-c4e486c74ce2" ``` @@ -398,3 +381,18 @@ this one moves rather than collides.)* full-weight L pass and CB-WP-0012's deleted its own structural trigger. - **The engine-evolution question**, as the review left it. - **Quote CB-WP-0021's cost by re-running the instrument.** + +**Done 2026-08-05.** +[CB-EV-0021](../evidence/CB-EV-0021-the-design-instrument.md). + +**Backfill did change the design** — and the honest answer to *"be +suspicious if all six fit"* is that only **five** were entered (one was a +double-count), so fitting them is close to circular. The taxonomy's real +test is the seventh finding. + +**Tier L's cost against what it caught**: four of six catches came only +from the separate reviewer, and **two came from execution rather than +process** — the `role` distinction from building it, the ten uncollected +rulings from running it. That is InnerLoop §Design goal's prediction +holding, and an argument against front-loading more review rather than +less.