clay-borg/workplans/CB-WP-0022-the-design-instrument.md

id	kind	title	status	state_hub_workstream_id
CB-WP-0022	product	The design instrument: findings about the game, with their reproductions	active	fda16340-0049-4acf-884b-a5cfdbde47c0

Purpose

structural tier  L   (named a high-leverage pass by the maintainer, and it
                      amends INTENT — clay-borg gains a stated aspect)
chaos            d8 = 6  → no override
declared tier    L

Declaration 5 of chaos window 2. Tier L: separate survey, adversarial review, ADR, then spec, then code.

The insight, in the maintainer's words

"We should consider ourselves testing the game and document inconsistencies to report them back to the ground-game repo, so that the game designer can improve the rules accordingly… We should have the rigorous game simulation engine and a meta scope to capture notes about game design flaws, questions, results and protocols about trial games… This will provide clay-borg an additional aspect as a valuable game design tool."

This is already happening and has no home. In five passes the engine has produced, as a by-product of being rigorous:

finding	how it surfaced
ten underdetermined rules points (U1–U10)	formalizing the dataset into testable rules
SOLVE offered on a face-down Problem, always inert	a human dragging it three rounds running
GR-A13 "wasted SOLVE" on a claimed Problem	a scenario that had to pick a default
GR-E01 unreachable below 5 seats	arithmetic over the deal count
six provisional scenario defaults	scenarios that could not be written without deciding something
GR-E03 / GR-E04 never played to the end	nobody noticed for nineteen passes

Every one was found by building the simulator, not by playing. That is the thing worth naming: a simulator rigorous enough to refuse ambiguity is a design instrument, because it cannot proceed past a rule that does not decide.

And every one of them has been carried in prose, in six different places, and one sat unread in an inbox for four days.

The load-bearing rule this must have

The project's standing failure is unexecuted verification. A design register that collects opinions would reproduce it in a new medium.

A design finding is not admissible without its reproduction.

Concretely: a scenario that fails, an arithmetic check that prints the contradiction, a recorded game the reader can replay, or a named test. "This feels unbalanced" is a note, not a finding. The SOLVE inertness is admissible because a recorded session shows three no-ops.

The example that stood here was GR-E01, and the review killed it (C1). "4/6/9 against 5/7/9 is a computation anyone can rerun" had already been rerun: 2da19a4 measured 6/9/12, and the scenario was renamed -unreachable- → -reachable-. The conclusion inverted — and it was one of the two findings that passed this rule. So existence is not what was missing. See ADR-0012 D3: the rule gains shape, and a reproduction must be able to fail. Ours went green and stayed admissible.

This is what would make clay-borg a design tool rather than a suggestion box, and it is the one part of this proposal that must not be traded away for convenience.

The judgment I want reviewed, not assumed

The maintainer asked whether this should extend to "a meta about the clay-borg engine evolution itself."

My answer is no, and it should be argued rather than accepted. That register already exists and is load-bearing: evidence/CB-EV-* records what each pass found, decisions/ADR-* records what was decided and what was rejected, gates.toml records what every control has caught and what would retire it, and workplans record what was attempted. Nineteen passes of engine evolution are already captured, with dates, costs and falsifiers.

Building a second register for the same subject would be ceremony, and this project has a standing rule that a gate must cash out. The asymmetry is the point: engine evolution has a home and game design does not.

If the adversarial review disagrees, that is exactly the kind of thing tier L exists to surface.

Task: survey how this is done elsewhere, and what we already have

id: CB-WP-0022-T01
status: done
priority: high
state_hub_task_id: "6b8663b8-4143-4361-ad9e-7df0c4f4a19d"

research/CB-RES-0007-*.md.

Do not survey issue trackers. The question is narrower and more interesting: how do rigorous rule systems record the ambiguity they found? Candidates worth a benchmark-to-beat:

• Errata and rulings practice in published games (Magic's comprehensive-rules + rulings split, Netrunner's NAPD card rulings) — what makes a ruling findable years later.
• Formal-methods counterexample traces — a model checker's output is precisely a reproduction attached to a claim, which is the shape wanted here.
• Conformance-suite provisional behaviour — how W3C/WHATWG mark "implementation-defined" and how a spec later absorbs it.
• What this repo already has: provisional: true scenarios, §Underdetermined, gates.toml's caught/retire_if shape, and the hub message that went unread. The register must reuse the provisional machinery rather than compete with it.

Name, per dimension, the property to beat — findability, reproducibility, and whether a ruling can close a finding mechanically.

Done 2026-08-03. CB-RES-0007, with a runnable baseline (tools/design-baseline.py).

The baseline is us, and it is measured: 6 findings across 11 files with no index, 2 of 6 (33%) with a runnable reproduction, and U1–U10 raised 2026-07-30, first read 2026-08-03 — 4 days, 0 of 10 ruled.

The uncomfortable number is stated up front rather than left for the review to find: the reproduction rule would reject four of our six existing findings. The survey answers it — none of the four is expensive to reproduce, so the 33% is evidence that nobody was ever asked for one, not that the rule is unaffordable.

Magic corrected an assumption this pass was about to build on. I expected a ruling to be the authoritative resolution. It is not: rulings are "reminder information with no actual weight or rules meaning", and the authoritative fix folds into the Oracle card text. A finding closes when the source changes, not when an annotation is added — so the register must be a queue that empties, not an archive that grows. That is now a constraint on T03's lifecycle.

Model checkers supplied the reproduction rule independently (a counterexample trace is the finding), and W3C's implementation-defined mark is the one piece of machinery we already have and must reuse rather than duplicate.

Task: adversarial review

id: CB-WP-0022-T02
status: done
priority: high
state_hub_task_id: "d2597895-fe2e-4f1e-a3db-a5fb8833434c"

Tier L requires it. Give the reviewer the survey and the §judgment above, and require an attempt at:

• that the engine-evolution register is redundant — the strongest counter is that ADRs record decisions and evidence records findings, but nothing records what we learned about building engines, which is a third thing;
• that "carries its reproduction" is affordable — if half the real findings cannot be reproduced cheaply, the rule will be quietly dropped and the register becomes a suggestion box anyway. (Since this task was written the answer has hardened: two findings reached ground-game on wrong premises, and ground-game has itself ruled that reproductions are required. The reviewer should now press the opposite way — whether the rule is sufficient, given both bad premises would have shipped a passing test alongside a wrong description.)
• that a register is needed at all, rather than one more section in GroundRules.md §Underdetermined, which already exists and already works.

Record the trail in history/, unpolished.

Done 2026-08-05. Trail: challenge, response.

Run by a separate agent — the first in this repo that was. CB-RES-0006's review opened by conceding it could not be, and called its own findings "a lower bound on what a genuinely separate reviewer would find." That was measurable, and this is the measurement: the separate reviewer ran git log against the survey's central example and found our own commit had falsified it four days earlier, while the author — who wrote that commit — quoted the dead number twice.

Seven challenges: four conceded, two conceded in part, one answered.

• C1 lands hardest and changed the design. The rule's showcase finding was false and had passed the rule, so existence is not the missing property — shape and falsifiability are. It also caught a defect in flight: T06's payload, todo, still named the dead number. Both folded in above.
• C2 withdrew the baseline's precision. design-baseline.py counts itself, its reproduction check never stats the file, and Makefile:127 runs only --self-test. Direction stands; 33% is not a measured rate.
• C3: "six provisional defaults" is five; GR-E01 is double-counted.
• C4: §Underdetermined was never evaluated as a candidate and already delivers four of five benchmarks. T03's burden flips: argue extension, not replacement.
• C5: the engine-evolution "third thing" is visible in specs/InnerLoopReference.md and history/'s retrospectives, neither of which my inventory named. Narrowed, not settled.
• Survived: the rule's affordability, and reuse of the provisional machinery — both with stated falsifiers.

Task: decide

id: CB-WP-0022-T03
status: done
priority: high
state_hub_task_id: "a2e85810-e949-4ea0-81fc-36b912af326c"

decisions/ADR-0012-*.md. At minimum:

• whether clay-borg's INTENT gains a stated aspect as a design instrument, and in what words — this is the change with the longest half-life in the pass;
• the finding taxonomy, and it should be grounded in the six findings above rather than invented: underdetermined (rules do not say), inconsistent (rules disagree with each other or the data), inert (a rule that cannot fire), degenerate (fires, but collapses play), unplayed (implemented, never played);
• the lifecycle and who owns each state: raised → reported → ruled → applied, or withdrawn;
• whether a finding without a reproduction is rejected or admitted as a note — and if admitted, how it is prevented from aging into an apparent finding.

Done 2026-08-05. ADR-0012, nine decisions. The two not on this list are the two the review forced:

• D2 — §Underdetermined is the register; nothing parallel is built. Against the survey's own five benchmarks the incumbent already delivers four, including the Oracle property the survey went to Magic to find and we had written ourselves eight days earlier (GroundRules.md:231-233). What it lacks is reproductions. So this pass extends a section — no new file, no new schema.
• D3 — admissibility is three clauses. Exists, has the ruled shape (row-level table, never a sum), and can fail. GR-E01's artifact went green and the finding stayed admissible and stayed queued, because nothing said a passing artifact was a signal. A green reproduction is an alarm.

The rest, in one line each: D1 INTENT gains property 4, Instrument, applied with its falsifier. D4 five kinds, each forced by an existing finding. D5 applied means the source changed; withdrawals are reported, not deleted. D6 notes admitted but never reportable, 30-day expiry. D7 no engine-evolution register, on an inventory C5 corrected. D8 design-baseline.py retired. D9 the artifact stays here, ground-game gets a generated file under its own workplan.

Task: specify

id: CB-WP-0022-T04
status: done
priority: high
state_hub_task_id: "60ddfeab-81b1-45e4-9ec9-91cc8ea7fb72"

specs/GameDesign.md, with metrics, because a spec without them is prose.

Candidate measures, to be argued not adopted:

• findings with a runnable reproduction — target 100%, and the denominator includes withdrawn ones;
• time from raised to reported — the U-items took four days to be read; that is the number this exists to fix;
• findings closed by a ruling vs findings still open, with age.

ground-game has ruled on what a finding must carry (GROUND-WP-0004 T02, 2026-08-04), and it is stricter than this pass proposed. Adopt it:

1. Arithmetic findings must ship a runnable reproduction and a row-level deal table listing Surface and each hidden priority separately — never only "sum of file" or "deal depth N".
2. ground-game rulings that depend on arithmetic should cite that reproduction id or path, so a later reader can re-run it.

The second half is theirs to keep. The first is ours, and it is a direct consequence of both premises we got wrong: "12 in the file" was a sum without a deal table, and "offered on a face-down Problem" was a condition named without checking which one actually fired.

So the reproduction rule gains a shape requirement, not just an existence one. A finding that ships a passing test but describes the wrong quantity is still a bad finding — which is exactly what happened twice. The spec must say what a reproduction has to show, not only that one exists.

Also specify the trial protocol, which is the part with no precedent here: a trial game is a --recorded session plus an observation log, so "we played it and X happened" is replayable rather than remembered. The engine already records sessions as scenarios; a trial is that plus notes, and it must cost almost nothing or it will not be done.

Done 2026-08-05. specs/GameDesign.md v1.0.

It is not a register, and says so in its second line — ADR-0012 D2 put that in §Underdetermined.

§1.2 is written against evidence rather than principle: a finding must print the rows behind any number it claims, and the spec carries the table of what shipped instead — a sum, a green scenario, and a condition named without checking which one fired. "12" was arithmetically defensible and still wrong about the game.

§1.3's target is 0 reproductions gone green while open — what GR-E01 would have tripped four days before a human caught it. No baseline rate is quoted; the first honest denominator is T05's backfill.

The trial protocol costs one flag. cb-play --record already writes a finished game as a scenario, so a trial is that plus a sibling .md in the player's own words. An observation is a note until it has a reproduction — the maintainer's "I felt it was too easy but then we lost" is the case it is shaped around, and forcing that into a schema at the moment of observation would lose it.

Task: build it, and backfill what is already known

id: CB-WP-0022-T05
status: todo
priority: high
state_hub_task_id: "469413ba-ed47-4840-a60e-ee7f95dc06ff"

The register, the tool, and then the six findings above entered into it — backfilling is the test. A register that cannot express findings the project already has is the wrong register, and discovering that after designing it is the point of doing it in this order.

make design (or equivalent) must report: open findings by kind, those without a reproduction, and those never reported to their owner.

Task: report to ground-game, mechanically

id: CB-WP-0022-T06
status: todo
priority: high
state_hub_task_id: "c6a659bd-c495-4754-9332-baadad51012a"

Generate the report and send it. The message that sat unread for four days is the baseline to beat, and the failure was not the message — it was that nothing pointed at it and nothing tracked whether it was answered.

So the report must land somewhere that persists: a file in ground-game under its own workplan, not only an inbox entry. GROUND-WP-0002 already holds the ten U-items; this should extend it rather than duplicate it.

Include the findings this pass has sharpened:

• SOLVE's legality against a face-down Problem or an unmatchable suit — and note that the case we reported was not the case that fired (CB-WP-0023 T01).
• GR-E01 vs GR-S01 — 4/6/9 against 5/7/9, no dataset reconciles them — withdrawn 2026-08-05, before sending. The adversarial review (C1) found 2da19a4 had already measured 6/9/12 against 5/7/9: the dataset reconciles them and the scenario is now -reachable-. Sending this would have been the fourth wrong premise to reach ground-game, and the only one caught before transmission. Report the withdrawal, not the finding — GROUND-WP-0002 holds the original, and a claim retracted silently is how the first three survived.

Task: evidence

id: CB-WP-0022-T07
status: todo
priority: high
state_hub_task_id: "4fb77911-486d-4d9e-a53d-c4e486c74ce2"

evidence/CB-EV-0021-*.md. (Was CB-EV-0020 when written; CB-WP-0023 shipped that number first — evidence/CB-EV-0020-solve-legality.md — so this one moves rather than collides.)

• Whether backfilling changed the design — if all six findings fit the first taxonomy, say so and be suspicious of it.
• What tier L cost against what it caught, since this is the second full-weight L pass and CB-WP-0012's deleted its own structural trigger.
• The engine-evolution question, as the review left it.
• Quote CB-WP-0021's cost by re-running the instrument.