From 88e25ae7c4b85c3f72699a773709ab46fdf6004b Mon Sep 17 00:00:00 2001 From: tegwick Date: Wed, 5 Aug 2026 18:06:35 +0200 Subject: [PATCH 1/5] Sync hub state for CB-WP-0025 T01/T02 Co-Authored-By: Claude Opus 5 --- WORK-RECORDS.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 4522972..72be584 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -32,7 +32,7 @@ | workplan | CB-WP-0022 | done | — | workplans/CB-WP-0022-the-design-instrument.md | | workplan | CB-WP-0023 | done | — | workplans/CB-WP-0023-solve-legality.md | | workplan | CB-WP-0024 | done | — | workplans/CB-WP-0024-the-table-you-can-watch.md | -| workplan | CB-WP-0025 | ready | — | workplans/CB-WP-0025-could-we-have-won.md | +| workplan | CB-WP-0025 | active | — | workplans/CB-WP-0025-could-we-have-won.md | | workplan | CB-WP-0026 | done | — | workplans/CB-WP-0026-collect-the-rulings.md | | task | CB-WP-0001-T01 | done | — | workplans/CB-WP-0001-inner-loop.md | | task | CB-WP-0001-T02 | done | — | workplans/CB-WP-0001-inner-loop.md | @@ -162,8 +162,8 @@ | task | CB-WP-0024-T03 | done | — | workplans/CB-WP-0024-the-table-you-can-watch.md | | task | CB-WP-0024-T04 | done | — | workplans/CB-WP-0024-the-table-you-can-watch.md | | task | CB-WP-0024-T05 | done | — | workplans/CB-WP-0024-the-table-you-can-watch.md | -| task | CB-WP-0025-T01 | todo | — | workplans/CB-WP-0025-could-we-have-won.md | -| task | CB-WP-0025-T02 | todo | — | workplans/CB-WP-0025-could-we-have-won.md | +| task | CB-WP-0025-T01 | done | — | workplans/CB-WP-0025-could-we-have-won.md | +| task | CB-WP-0025-T02 | done | — | workplans/CB-WP-0025-could-we-have-won.md | | task | CB-WP-0025-T03 | todo | — | workplans/CB-WP-0025-could-we-have-won.md | | task | CB-WP-0025-T04 | todo | — | workplans/CB-WP-0025-could-we-have-won.md | | task | CB-WP-0025-T05 | todo | — | workplans/CB-WP-0025-could-we-have-won.md | From 3a026b1e1f32150e77f0202718a8068a48632f96 Mon Sep 17 00:00:00 2001 From: tegwick Date: Wed, 5 Aug 2026 18:47:07 +0200 Subject: [PATCH 2/5] CB-WP-0025 T03: ADR-0013 -- one world after the game, and difficulty that does not measure the bot MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Seven decisions. Two are not what T03 expected, because the review moved the ground under both. D1: strategy fusion DOES NOT APPLY, and that is why the affordable option is also the honest one. Fusion is a defect of aggregating over determinizations to choose a move -- the search picking different actions in states a player cannot distinguish. After the game there is ONE WORLD: the deal is known, so a search over it yields a line executable in the only world there is. The survey treated fusion as this pass's central obstacle; it is an obstacle to a playing engine, which we are not building. The tool answers "given the deal as it actually was, was there a line that reached the threshold" and is labelled that way on screen -- never "how you should have played". D4: difficulty is the WINNABLE FRACTION, not a bot's win rate. C4 killed the bot rate -- two trivial policies span 0-100% on the same deals, and improving the bot would make the game "easier" without a rule changing. A measure that moves when the measurer improves is not measuring the thing. The solver supplies the alternative: over N deals, in what proportion does a winning line exist. That is a property of the deal distribution and the threshold, which is what ground-game tunes and what GROUND-WP-0005 is blocked on. Ships as a table -- winnable fraction, named reference policy, skill gap -- never one number, with policy/N/seed-range/K in the number's name. Stated as a LOWER BOUND, since a K-round search cannot see a line that needed round 1. D2: search GroundState. The survey's view-only structural boundary is not implementable -- a view cannot fold events, and it said so in §6. The guarantee moves to something checkable: every move in a witness is marked visible or hidden, computed from project(). A witness reads "you could have won, but two of these six moves needed a card you had no way to know was coming" -- more useful than either extreme. D3: bounded exhaustive over the last K rounds, table as one co-operative agent, K=2 default. Affordable once C6 corrected the premise: joint branching over the last two rounds is ~5x10^2 / 1.6x10^5 / 5.7x10^5 at 2/3/4 seats. Wording is normative: "no winning line found in the last K rounds", never "unwinnable". PIMC and ISMCTS rejected -- they add strategy fusion to a problem that does not have it. D5: the harness becomes an instrument before any figure is quoted (C3) -- positive controls, --self-test in make self-tests, a make target, and a PLURAL policy panel, because the spread between policies is what C4 exposed and hiding it would restore the error. D6: no new crate and no port. The L declaration was an over-declaration and is recorded as one. D7: the node cost is disputed 5x between author and reviewer; T04 benchmarks it with criterion and neither existing figure may be cited, including by this ADR. Co-Authored-By: Claude Opus 5 --- decisions/ADR-0013-could-we-have-won.md | 230 ++++++++++++++++++++++ workplans/CB-WP-0025-could-we-have-won.md | 34 +++- 2 files changed, 263 insertions(+), 1 deletion(-) create mode 100644 decisions/ADR-0013-could-we-have-won.md diff --git a/decisions/ADR-0013-could-we-have-won.md b/decisions/ADR-0013-could-we-have-won.md new file mode 100644 index 0000000..730f8ed --- /dev/null +++ b/decisions/ADR-0013-could-we-have-won.md @@ -0,0 +1,230 @@ +# ADR-0013: answer the retrospective question, and measure difficulty with the solver rather than with a bot + +status: accepted +date: 2026-08-05 +decided by: agent, under the standing loop authorization +tier: L (structural L — a new analysis capability whose numbers `ground-game` +will tune against; chaos d8=6 → no override). Tier L: separate survey, +adversarial review, then this document. +references: [CB-WP-0025](../workplans/CB-WP-0025-could-we-have-won.md), +[CB-RES-0008](../research/CB-RES-0008-could-we-have-won.md), +[challenge](../history/260805-could-we-have-won-challenge.md) / +[response](../history/260805-could-we-have-won-response.md), +[ADR-0012](ADR-0012-the-design-instrument.md) (admissibility), +[GameDesign.md](../specs/GameDesign.md), +GROUND-WP-0004 T02, GROUND-WP-0005 (blocked on D4) + +## Context + +The maintainer asked two things: *"we lost — could we have won, and how?"* +and *"do we have difficulty estimations?"* + +**The survey answered the second and was wrong.** It measured +`GreedyPolicy` winning 200/200 at five and six seats and called the game +too easy there. A `FirstLegal` policy — `legal[0]`, no heuristic — scores +**0%** on the same deals. Two unsophisticated agents span the whole range, +so the measurement was about the policy. + +That failure is not incidental to this ADR; **it determines D4.** + +## The premise that changed, and it changes the algorithm + +The survey said exhaustive search was impossible and reached for +determinized sampling, which carries strategy fusion. **Both halves were +wrong.** + +- Its per-node cost was **30–50× too high** (a timer bracketing whole + games). Corrected: ~3–4 µs per `legal_commands` call, with the exact + figure still disputed (§D7). +- Bounded exhaustive search is **affordable**: measured ~3 s over the last + two rounds at three seats. + +Joint branching, treating the table as one co-operative agent — the +product over seats of the measured per-seat branching: + +| seats | per-seat mean | joint per round | last 2 rounds | +|---|---:|---:|---:| +| 2 | 4.7 | ~22 | ~5×10² | +| 3 | 7.4 | ~405 | ~1.6×10⁵ | +| 4 | 9.1 | ~754 | ~5.7×10⁵ | + +Against a ~10⁵–10⁶ node budget, **the last two rounds are exhaustively +searchable at two, three and four seats.** Five rounds is not, at any seat +count. + +--- + +## D1 — answer the *retrospective* question, and say so in those words + +Three questions were on the table (CB-RES-0008 §3). The tool answers: + +> **"Given the deal as it actually was, was there a line of play that +> reached the threshold — and here is one."** + +**Strategy fusion does not apply to this question, and that is the whole +reason it is the affordable one.** Fusion is a defect of *aggregating over +determinizations to choose a move*: the search picks different actions in +states the player cannot distinguish. **After the game there is one +world.** The deck is known, the deal is known, and a search over that +single world produces a line that is executable in it — because it is the +only world there is. + +The survey treated fusion as an obstacle to this pass. It is an obstacle +to a *playing* engine. We are not building one. + +**What remains true is that the line may have been unfindable at the +time**, and D2 handles that by annotation rather than by refusing to +answer. + +**On screen it is called** *"was this deal winnable?"* — never *"how you +should have played"*. The distinction is the honest content of the +feature, and a label that overclaims turns a true answer into a false +lesson. + +## D2 — run on `GroundState`, and mark each move's information dependence + +The survey's preferred guarantee was structural: search a `GroundView` so +the boundary cannot be crossed. **It is not implementable** — a view +cannot `fold` events, so a search needs a state it may not see. The survey +said so in §6 and was right to. + +Decision: **search `GroundState`** — legitimate here, because post-game +the deal is public (`solution_discard` already is, and the game is over) — +and move the honesty guarantee to something checkable: + +> **Every move in an emitted witness is marked `visible` or `hidden`.** +> A move is `visible` if, at the point it is played, everything it depends +> on was in the acting seat's projection: the target Problem face-up, the +> Solution in that seat's own hand. Otherwise `hidden`. + +So a witness reads *"you could have won — but two of these six moves +needed a card you had no way to know was coming."* **That is a more useful +answer than either extreme**, and it is computed from `project()`, which +already exists and is already tested. + +**Falsifier:** if a witness is emitted whose moves are all marked +`visible` but which no seat could actually have chosen, the marking is +wrong and D2 has failed. A test constructs exactly that case. + +## D3 — bounded exhaustive over the endgame, `K` rounds, and honest wording + +**Exhaustive search over the last `K` rounds**, with the table treated as +one co-operative agent choosing joint selections. `K = 2` by default, +which the measurements put inside budget at 2–4 seats. + +- The bound is **rounds**, not nodes or seconds, because rounds are what a + player understands: *"winnable from round 4"* means something; *"winnable + within 100,000 nodes"* does not. +- A node budget is a **secondary** cut that aborts with a stated reason, + so a wide table cannot hang the page. +- **Wording is normative.** When no line is found the tool says + **"no winning line found in the last K rounds"** — never *"unwinnable"*. + A bounded search that claims unwinnability is lying, and this is the + sentence the maintainer will read. + +**Not chosen: determinized sampling (PIMC), ISMCTS.** Both are for playing +under uncertainty. Here there is one world (D1), so they would add strategy +fusion to a problem that does not have it. + +## D4 — difficulty is the **winnable fraction**, not any bot's win rate + +**This is the decision the review forced, and it is the useful half of the +pass.** + +A single-policy win rate cannot be a difficulty: two trivial policies span +0–100% on the same deals. Worse, *improving the bot would make the game +"easier"* without a rule changing — a measure that moves when the +measurer improves is not measuring the thing. + +The solver supplies a policy-independent alternative: + +> **Winnable fraction** — over N deals at a seat count, the proportion in +> which the search finds *any* winning line within its bound. + +That is a property of **the deal distribution and the threshold**, which +is exactly what `ground-game` tunes. It is the number GROUND-WP-0005 is +blocked on, and the bot rate never was. + +Difficulty therefore ships as **a small table, never one number**: + +| column | what it is | +|---|---| +| winnable fraction | can the deal be won at all (bounded, K stated) | +| reference-policy win rate | what a stated bot achieves — **named policy** | +| skill gap | the difference: how much play has to supply | + +**Every rate carries its policy, its N, its seed range and its K in the +number's name**, not in a footnote. A figure that loses them is +inadmissible under GameDesign §1.2. + +**Bounded-below caveat, stated because it will be quoted:** the winnable +fraction from a K-round search is a **lower bound** on true winnability — +a deal unwinnable in the last 2 rounds may have been winnable in round 1. +The report says "winnable-from-round-(6−K)", never "winnable". + +## D5 — the harness becomes an instrument before any figure is quoted + +C3 established that `difficulty-baseline.rs` has no assertions, no +`--self-test` and no `make` target — nothing can turn it red. Under +CB-WP-0022 T05's own `role` distinction it is a `default` artifact wearing +a `counterexample` label, and **GameDesign §1.3 makes it inadmissible.** + +Required before T06 reports anything: + +- **positive controls** — a deal constructed to be unwinnable returns + none; a deal constructed to be winnable returns a witness that replays; +- **`--self-test`**, wired into `make self-tests` like every other + reporting tool; +- **`make difficulty`** (or equivalent), so the figure regenerates from + one command; +- the **policy panel is plural**: at least `greedy`, `random` and + `first-legal`, because the spread between them is what C4 exposed and + hiding it would restore the error. + +## D6 — it lives in `games/ground`, not a new crate + +The search needs `validate`, `fold`, `legal_commands` and `project` — +all of `games_ground`. A separate crate would either re-export the +aggregate or take a dependency on it and add nothing. + +**The tier was declared L on the assumption of a new capability port. +There is no port**, and that over-declaration is recorded rather than +hidden — it is a data point for the tier rules, and the L weight paid for +itself twice over regardless (§Consequences). + +`cb-play` gains a mode to ask the question about a finished game; the +difficulty sweep is an example/binary, as the baseline is. + +## D7 — the per-node cost is unsettled and T04 must benchmark it + +The author measured **3.0–4.1 µs**, the reviewer **15.6–20.4 µs**, by +different isolations. Both agree the published 112–161 µs was wrong by +1–2 orders; neither has established which is right. + +**T04 benchmarks it with `criterion`** — already a dev-dependency, already +used by `benches/synthetic.rs` — and the spec quotes that number and no +other. **Neither figure above may be cited**, including by this ADR. + +## Consequences + +- `specs/` gains the witness contract and the difficulty table's shape + (T04), plus the benchmarked node cost. +- T05 builds the K-round search, the `visible`/`hidden` marking, and the + replay check. +- **T06's payload changes completely.** It reports a winnable fraction and + a policy panel to GROUND-WP-0005 — *not* the withdrawn "too easy at 5–6 + seats". The withdrawal itself is reported, per ADR-0012 D5. +- The register gains the withdrawn finding as `inconsistent` / + `withdrawn`, so it is in the log rather than forgotten. + +## What was rejected + +| rejected | why | +|---|---| +| determinized sampling / PIMC / ISMCTS | strategy fusion, added to a question that has one world (D1) | +| a view-only search as a structural boundary | not implementable — a view cannot fold events | +| refusing to answer unless the line was findable at the time | throws away a true and useful answer; annotate instead (D2) | +| a single bot win rate as "difficulty" | two trivial policies span 0–100% on the same deals (C4) | +| "unwinnable" as output wording | a bounded search cannot know it (D3) | +| a new crate | no port exists; it would re-export the aggregate (D6) | +| quoting either measured node cost | they disagree 5× and neither is established (D7) | diff --git a/workplans/CB-WP-0025-could-we-have-won.md b/workplans/CB-WP-0025-could-we-have-won.md index 23b4739..cec037a 100644 --- a/workplans/CB-WP-0025-could-we-have-won.md +++ b/workplans/CB-WP-0025-could-we-have-won.md @@ -235,7 +235,7 @@ this project have now caught a false headline that every gate passed.** ```task id: CB-WP-0025-T03 -status: todo +status: done priority: high state_hub_task_id: "055630a1-56e0-4ad4-8796-35f444313371" ``` @@ -258,6 +258,38 @@ state_hub_task_id: "055630a1-56e0-4ad4-8796-35f444313371" if the ADR concludes it is a mode of an existing one, say so, and the over-declaration is a chaos-window data point worth recording. +**Done 2026-08-05.** +[ADR-0013](../decisions/ADR-0013-could-we-have-won.md), seven decisions. +**Two of them are not what T03 was written expecting**, because the review +moved the ground under both. + +- **D1 — strategy fusion does not apply, and that is why this is + affordable.** Fusion is a defect of *aggregating over determinizations + to choose a move*. **After the game there is one world**: the deal is + known, so a search over it produces a line executable in the only world + there is. The survey treated fusion as this pass's central obstacle; it + is an obstacle to a *playing* engine, which we are not building. +- **D4 — difficulty is the WINNABLE FRACTION, not any bot's win rate.** + C4 killed the bot rate: two trivial policies span 0–100% on the same + deals, and improving the bot would make the game "easier" without a rule + changing. The solver supplies a policy-independent measure — *over N + deals, in what proportion does a winning line exist* — which is a + property of the deal distribution and the threshold, and is what + GROUND-WP-0005 actually needs. **The bot rate never was.** + +**D2** searches `GroundState` (the survey's view-only boundary is not +implementable — a view cannot fold events) and moves the guarantee to a +checkable per-move `visible`/`hidden` marking computed from `project()`. +A witness reads *"you could have won, but two of these six moves needed a +card you had no way to know was coming."* **D3** is bounded exhaustive +over the last K rounds — measured affordable at 2–4 seats once C6 +corrected the premise — with normative wording: *"no winning line found in +the last K rounds"*, never *"unwinnable"*. **D5** makes the harness an +instrument before any figure is quoted (C3). **D6**: no new crate, no +port — **the L declaration was an over-declaration and is recorded as +one**. **D7**: the node cost is disputed 5× and T04 must benchmark it; +neither figure may be cited, including by the ADR. + ## Task: specify ```task From b27aa14df0e6baf5f3f2a4d78f23f2fd93b43fa6 Mon Sep 17 00:00:00 2001 From: tegwick Date: Wed, 5 Aug 2026 18:59:46 +0200 Subject: [PATCH 3/5] CB-WP-0025 T04: specs/RetrospectiveAnalysis.md, and a benchmark that caught me repeating C1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit specs/RetrospectiveAnalysis.md v1.0 plus games/ground/benches/search.rs, which exists because ADR-0013 D7 refused to let the spec quote either disputed figure. THE BENCHMARK'S OWN FIRST FIXTURE WAS DEFECTIVE, and it is the same defect class the review caught one layer up. Stopping at a fixed step 20 put 2p and 4p in states where seat 0 had NO legal commands, so it timed an empty Vec (~120 ns) and silently skipped validate_fold because there was nothing to validate. It now advances until the seat has a real branch and ASSERTS it. A clone benchmark was added too: a search must copy state per branch, and iter_batched excludes setup from timing, so without it the budget would again rest on an unmeasured span. Measured at real decision points: legal_commands 4.06-4.76 us, clone 378-639 ns, validate+fold 0.5-3.8 us. Per-child cost is NOT uniform -- some commands resolve cascades -- so budgets use the upper end (~5 us/child). That settles D3 with real numbers. Joint branching over the last two rounds is ~5x10^2 / 1.6x10^5 / 5.7x10^5 at 2/3/4 seats, so K=2 costs negligible / 0.8 s / 2.9 s and holds at two to four seats. It does NOT hold at five or six, where the tool must reduce K and say that it did rather than silently searching less. §4.1 is a normative prohibition, not a preference: a single policy's win rate MAY NOT be reported as a difficulty. The spec carries the measured reason -- greedy 100% against first-legal 0% on identical deals -- because this project already made that error and nearly exported it to a repo that is blocked waiting on the number. §2.3 makes the empty-result wording normative: "no winning line found in the last K rounds", never "unwinnable". A bounded search cannot establish unwinnability and that sentence is what a player who just lost reads. Also corrected: the T01 completion record still asserted all three withdrawn claims as fact. It now carries claimed / withdrawn / survives explicitly rather than being rewritten -- a retraction that does not propagate to every place the claim lives is how the earlier ones survived. make all: exit 0. loop-lint clean. Co-Authored-By: Claude Opus 5 --- games/ground/Cargo.toml | 5 + games/ground/benches/search.rs | 128 ++++++++++++++ specs/RetrospectiveAnalysis.md | 196 ++++++++++++++++++++++ workplans/CB-WP-0025-could-we-have-won.md | 156 +++++++++-------- 4 files changed, 411 insertions(+), 74 deletions(-) create mode 100644 games/ground/benches/search.rs create mode 100644 specs/RetrospectiveAnalysis.md diff --git a/games/ground/Cargo.toml b/games/ground/Cargo.toml index 9d0768b..1f8e627 100644 --- a/games/ground/Cargo.toml +++ b/games/ground/Cargo.toml @@ -27,5 +27,10 @@ serde_json.workspace = true name = "synthetic" harness = false +# CB-WP-0025 T04 / ADR-0013 D7: the node cost the spec quotes. +[[bench]] +name = "search" +harness = false + [lints] workspace = true diff --git a/games/ground/benches/search.rs b/games/ground/benches/search.rs new file mode 100644 index 0000000..75c7fd5 --- /dev/null +++ b/games/ground/benches/search.rs @@ -0,0 +1,128 @@ +//! CB-WP-0025 T04 — what one search node actually costs. +//! +//! **ADR-0013 D7 exists because two measurements disagreed by 5×.** The +//! survey published 112–161 µs/node from a timer that bracketed two +//! `setup`s and a whole greedy game (C1). The author's re-measurement said +//! 3.0–4.1 µs with `Instant::now()` around each call; the adversarial +//! reviewer's isolation said 15.6–20.4 µs. Both agreed the published +//! figure was wrong by 1–2 orders and neither established which +//! replacement was right. +//! +//! So the spec quotes **this** and nothing else. `criterion` handles the +//! things hand-rolled timing gets wrong here: per-call clock overhead +//! against a ~microsecond subject, warm-up, and run-to-run variance — +//! which is what let the survey's figure move 161 → 112 between two runs +//! of the same unmodified binary. +//! +//! Two subjects, because a search node is not one call: +//! +//! * `legal_commands` — enumerating a seat's options; +//! * `validate + fold` — taking one branch, which any search does per +//! child and which the survey never separated out. + +use cb_game_runtime::{ScenarioGame, Setup}; +use cb_kernel::{Actor, Aggregate, PlayerId}; +use criterion::{criterion_group, criterion_main, BatchSize, Criterion}; +use games_ground::bot::{legal_commands, play, GreedyPolicy, Policy}; +use games_ground::GroundState; +use std::collections::BTreeMap; + +fn setup(players: u8, seed: u64) -> GroundState { + GroundState::setup( + &Setup { + players, + preset: format!("standard-{players}p"), + patch: BTreeMap::new(), + }, + seed, + ) + .expect("preset") +} + +/// A **mid-game state at a real decision point** for `seat`. +/// +/// Not a fresh deal: at deal time most branches do not exist yet, and a +/// node cost taken there would flatter any search proposal. +/// +/// **And not a fixed step count either.** The first version stopped at +/// step 20 for every seat count, which put 2p and 4p in a state where +/// seat 0 had *no* legal commands at all — so the benchmark reported +/// ~120 ns (the cost of returning an empty `Vec`) and silently skipped +/// `validate_fold` because there was nothing to validate. A fixture that +/// measures the empty case and calls it a node cost is the same defect +/// class this whole pass exists to correct, one layer down. +/// +/// So: advance until the seat genuinely has a choice, and assert it. +fn midgame(players: u8, seed: u64, seat: PlayerId) -> GroundState { + let mut ps: Vec> = (0..players) + .map(|_| Box::new(GreedyPolicy) as Box) + .collect(); + let game = play(setup(players, seed), &mut ps).expect("a complete game"); + let mut state = setup(players, seed); + let mut best: Option = None; + for (i, (actor, cmd)) in game.steps.iter().enumerate() { + // Past the opening, take the first state where the seat has a real + // branch. `> 1` rather than `> 0`: a forced move is not a node. + if i >= 8 && legal_commands(&state, seat).len() > 1 { + best = Some(state.clone()); + break; + } + if let Ok(events) = state.validate(*actor, cmd) { + for e in &events { + state.fold(e); + } + } + } + let state = best.expect("a mid-game state where the seat has a choice"); + assert!( + legal_commands(&state, seat).len() > 1, + "benchmark fixture has no branch to measure — it would time the empty case" + ); + state +} + +fn bench(c: &mut Criterion) { + for players in [2u8, 3, 4] { + let seat = PlayerId(0); + let state = midgame(players, 7, seat); + let width = legal_commands(&state, seat).len(); + println!(" fixture {players}p: {width} legal commands at the measured node"); + + c.bench_function(&format!("legal_commands/{players}p"), |b| { + b.iter(|| std::hint::black_box(legal_commands(&state, seat))) + }); + + // A search must COPY the state per branch (or undo, which we do + // not have). `iter_batched` excludes setup from the timing, so + // without this the budget would rest on an unmeasured span — + // which is the exact mistake C1 caught in the survey. + c.bench_function(&format!("clone/{players}p"), |b| { + b.iter(|| std::hint::black_box(state.clone())) + }); + + // One branch taken: what a search pays per CHILD, on top of + // enumeration. The survey folded this into "us/node" without + // separating it, and a search's real cost is enumeration once plus + // this per child. + let legal = legal_commands(&state, seat); + if let Some(cmd) = legal.first() { + c.bench_function(&format!("validate_fold/{players}p"), |b| { + b.iter_batched( + || state.clone(), + |mut s| { + if let Ok(events) = s.validate(Actor::Player(seat), cmd) { + for e in &events { + s.fold(e); + } + } + std::hint::black_box(s) + }, + BatchSize::SmallInput, + ) + }); + } + } +} + +criterion_group!(benches, bench); +criterion_main!(benches); diff --git a/specs/RetrospectiveAnalysis.md b/specs/RetrospectiveAnalysis.md new file mode 100644 index 0000000..3fdee9b --- /dev/null +++ b/specs/RetrospectiveAnalysis.md @@ -0,0 +1,196 @@ +# RetrospectiveAnalysis — was this deal winnable, and how hard is the game + +v1.0 — CB-WP-0025 T04, 2026-08-05. Normative. Implements +[ADR-0013](../decisions/ADR-0013-could-we-have-won.md). Admissibility of +anything this produces is governed by +[GameDesign.md](GameDesign.md) §1. + +**Two capabilities, one machine.** *Was this deal winnable?* is a search +over a finished game. *How hard is the game?* is that search run over many +deals and counted — **not** a bot's win rate (§4.1). + +--- + +## 1. The question, and its name + +> **"Given the deal as it actually was, was there a line of play that +> reached the threshold?"** + +**Never labelled "how you should have played."** The distinction is the +honest content of the feature: the tool answers a question about the +*deal*, and a label promising advice about the *player* turns a true +answer into a false lesson. + +**Strategy fusion does not apply and must not be invoked as an objection.** +Fusion (Frank, Basin & Matsubara 1998) is a defect of aggregating over +determinizations to choose a move. After the game there is **one world** — +the deal is known — so a line found in it is executable in the only world +there is. This is why the affordable option is also the honest one. + +## 2. The witness + +A witness is a sequence of joint selections that, replayed from the +recorded initial state, ends with `group_success == true`. + +### 2.1 It must replay — hard gate, not a metric + +> **100% of emitted witnesses replay through the existing scenario runner +> and end in `group_success`.** + +Not a target: a **gate**. A witness that does not replay asserts the +opposite of the truth to a player who just lost, which is worse than +emitting nothing. + +### 2.2 Every move carries its information dependence + +ADR-0013 D2. Each move in a witness is marked: + +| mark | meaning | +|---|---| +| `visible` | everything the move depended on was in the acting seat's projection at that point — the Problem face-up, the Solution in that seat's own hand | +| `hidden` | it was not | + +Computed from `project()`, which already exists and whose hiding rules are +already asserted by `games_ground::view`. + +**This replaces the structural boundary the survey wanted.** Searching a +`GroundView` is not implementable — a view cannot `fold` events — so the +guarantee moved from *the search cannot see it* to *the answer says which +moves needed it*. A witness reads: + +> *"This deal was winnable. Two of these six moves needed a card you had +> no way to know was coming."* + +**Falsifier:** a witness whose moves are all `visible` but which no seat +could have chosen means the marking is wrong. A test constructs that case. + +### 2.3 Wording when nothing is found + +> **"No winning line found in the last K rounds."** + +**Never "unwinnable".** A bounded search cannot establish unwinnability, +and this sentence is what the player reads. + +## 3. The bound + +**Exhaustive over the last `K` rounds**, table treated as one co-operative +agent choosing joint selections. `K = 2` by default. + +Bounded in **rounds**, not nodes: *"winnable from round 4"* means something +to a player; *"winnable within 100,000 nodes"* does not. A node budget is a +secondary cut that aborts with a stated reason so a wide table cannot hang +the page. + +### 3.1 Measured cost, and what it permits + +`cargo bench -p games-ground --bench search` — the single source for these +numbers (ADR-0013 D7). Mid-game states at real decision points: + +| seats | branch width | `legal_commands` | `clone` | `validate+fold` | +|---|---:|---:|---:|---:| +| 2 | 5 | 4.06 µs | 378 ns | 696 ns | +| 3 | 8 | 4.13 µs | 432 ns | 508 ns | +| 4 | 11 | 4.76 µs | 639 ns | **3.76 µs** | + +**Per-child cost is not uniform** — `validate+fold` ranges 0.5–3.8 µs +depending on which command is taken, because some resolve cascades and +some do not. **Budgets use the upper end**, so ~5 µs per child +(clone + validate + fold). + +Joint branching over the last two rounds, from the measured per-seat +widths: + +| seats | joint / 2 rounds | at ~5 µs/child | +|---|---:|---:| +| 2 | ~5×10² | negligible | +| 3 | ~1.6×10⁵ | **~0.8 s** | +| 4 | ~5.7×10⁵ | **~2.9 s** | + +**So `K = 2` is affordable at two, three and four seats, and is not at +five or six** — joint branching there exceeds 10⁶ per round. Five and six +seats require a smaller `K`, and the tool must reduce it and **say that it +did** rather than silently searching less. + +**The published 112–161 µs/node figure is withdrawn** (CB-RES-0008 §1.2, +challenge C1) and must not be quoted from anywhere. + +## 4. Difficulty + +### 4.1 A bot's win rate is not a difficulty + +**Normative prohibition**, because this project already made the error and +nearly exported it: + +> A win rate from a single policy **may not be reported as a difficulty**. + +Measured, on identical deals: `GreedyPolicy` wins **100%** at five and six +seats where a `FirstLegal` policy — take `legal[0]`, no heuristic — wins +**0%**; at two seats `FirstLegal` (77.5%) *beats* greedy (66.0%). Two +unsophisticated agents span the entire range. + +And a measure that improves when the *measurer* improves is not measuring +the subject: a better bot would make the game "easier" with no rule +changing. + +### 4.2 What is reported instead + +> **Winnable fraction** — over N deals at a seat count, the proportion in +> which the search finds a winning line within its bound. + +A property of the **deal distribution and the threshold**, which is what +`ground-game` tunes. Ships as a table, never one number: + +| column | what it is | +|---|---| +| winnable fraction | can the deal be won at all — bounded, `K` stated | +| reference-policy win rate | what a **named** policy achieves | +| skill gap | the difference — how much play has to supply | + +**It is a lower bound and must be labelled one.** A `K`-round search cannot +see a line that required round 1, so the figure is +**"winnable-from-round-(6−K)"**, never "winnable". + +**Every rate carries its policy, N, seed range and K in the number's +name**, not in a footnote — GameDesign §1.2, and the reason the withdrawn +finding was inadmissible. + +### 4.3 Resolution — the number that makes it usable + +> The smallest threshold change the measurement can distinguish, with its +> N. + +*"We can tell a threshold of 5 from 7 but not 7 from 8"* is more useful to +`ground-game` than any rate with no error bar, and it is what makes the +figure a tuning instrument rather than a statistic. + +## 5. The instruments must be able to fail + +ADR-0013 D5, and GameDesign §1.3. Before **any** figure from these tools is +quoted anywhere: + +- **positive controls** — a deal constructed to be unwinnable returns + none; a deal constructed to be winnable returns a witness that replays; +- **`--self-test`**, wired into `make self-tests` like every other + reporting tool; +- **one command regenerates the figure** (`make difficulty`); +- **the policy panel is plural** — at least `greedy`, `random` and + `first-legal`. The spread between them is the finding §4.1 rests on, and + reporting one policy would restore the error. + +**`games/ground/examples/difficulty-baseline.rs` currently satisfies none +of the first three** and is inadmissible until it does. It has no +assertions, no self-test, and no `make` target — nothing can turn it red, +which under CB-WP-0022 T05's `role` distinction makes it a `default` +artifact wearing a `counterexample` label. + +## 6. Falsifiers for this spec + +- **§2.2 fails** if a witness is emitted whose moves are all `visible` but + which no seat could have chosen. Then the marking must be derived from + the search rather than checked after it. +- **§4.2 fails** if the winnable fraction turns out to be ~100% or ~0% at + every seat count and threshold — it would then have no resolution (§4.3) + and be as useless as the bot rate it replaced. +- **§3 fails** if `K = 2` proves unaffordable in practice at four seats; + the measured 2.9 s is a projection from branch widths, not a timing of + the real search. diff --git a/workplans/CB-WP-0025-could-we-have-won.md b/workplans/CB-WP-0025-could-we-have-won.md index cec037a..1e9741c 100644 --- a/workplans/CB-WP-0025-could-we-have-won.md +++ b/workplans/CB-WP-0025-could-we-have-won.md @@ -127,34 +127,34 @@ per dimension — a number or a reproducible comparison, not an impression. [CB-RES-0008](../research/CB-RES-0008-could-we-have-won.md), with a runnable baseline (`games/ground/examples/difficulty-baseline.rs`). -**The baseline produced a finding before any solver exists, and it is the -biggest thing in this pass.** A greedy bot wins **200 of 200** games at -five and six seats — with a median margin of +3 and 11.8–12.0 points -available against a threshold of 9. The curve is 66% / 82.5% / 95% / 100% -/ 100% across 2/3/4/5/6 seats. +> **Everything this record originally claimed was withdrawn by T02 the +> same day.** Kept as a pointer rather than rewritten, because a claim +> retracted silently is how three earlier wrong premises survived +> (ADR-0012 D5). -The row-level table shows why: available points go **6 / 9 / 12** against -thresholds **5 / 7 / 9**, so the ratio *rises* with seat count (1.20 → -1.29 → 1.33) while the table also gains actions per round. **Three -multipliers pointing the same way.** +**Claimed:** a greedy bot wins 200/200 at five and six seats, so the game +is too easy there; the points-to-threshold ratio rises with seat count and +explains the curve; `legal_commands` costs 112–161 µs, so exhaustive +search is out; and all of this explains the maintainer's report. -**It also explains the maintainer's report without a solver.** *"I felt it -was too easy but then we lost"* — 66% at two seats is a real game; 100% at -six is not. Both halves are true of different seat counts. +**Withdrawn:** the win rate is `GreedyPolicy`'s, not the game's (C4 — +`FirstLegal` scores 0% on the same deals); the ratio is identical at 3p +and 4p, which differ by 12.5 points (C2); the cost figure was wrong by +30–50× and exhaustive search is affordable (C1, C6); and the maintainer's +losses were 3-player games on the pre-ruling deal (C5). -**Cost measured, and it rules out the obvious approach.** Branching is -small (mean 4.7–9.1) but `legal_commands` costs **112–161 µs** per call, -because it filters candidates through full `validate`. Exhaustive search -is out at every seat count; ~10⁴–10⁵ nodes is 1.4–14 s, which is the -budget the ADR must design inside. +**What survives:** the harness exists and runs, the 6/9/12 arithmetic is +right against `Problems.csv`, and the branching widths (4.7–9.1) hold. See +[CB-RES-0008](../research/CB-RES-0008-could-we-have-won.md) for the +corrected text and +[the response](../history/260805-could-we-have-won-response.md) for the +full accounting. -**Prior art names the trap.** Determinized search (PIMC) suffers *strategy -fusion* — Frank, Basin & Matsubara 1998 — where the search picks different -actions in states a real player cannot tell apart. A witness built that -way may require knowing what was on top of the deck. **And it would still -replay green**, so the checkability benchmark does not catch it. Honesty -and checkability are different properties, stated so T03 cannot conflate -them. +**Prior art named the trap** — determinized search suffers *strategy +fusion* (Frank, Basin & Matsubara 1998) — **and T03 then established it +does not apply here.** After the game there is one world, so a line found +in it is executable in it. Fusion is an obstacle to a *playing* engine, +which this is not. ## Task: adversarial review @@ -189,47 +189,27 @@ Tier L requires it. Exactly one round: challenge, then response, trail in [response](../history/260805-could-we-have-won-response.md). Separate agent, as in CB-WP-0022. -**Six of seven conceded, and the survey's headline finding is withdrawn — -not softened.** +**Six of seven conceded. The survey's headline finding is WITHDRAWN.** -**C4 is the one that kills it, and the reviewer ranked it fourth.** -Measured: a `FirstLegal` policy — take `legal[0]`, no heuristic — scores -**0% at five and six seats** where greedy scores 100%, and **77.5% at two -seats** where greedy scores 66%. **Two unsophisticated agents span the -entire range at the same seat count**, so *"the game is too easy at 5–6 -seats"* is a statement about `GreedyPolicy`, not about GROUND. Their -offered rescue — greedy hits the 12-point ceiling every time, so it is a -rules claim — dies on the same data. +**C4 kills it and the reviewer ranked it fourth.** A `FirstLegal` policy — +`legal[0]`, no heuristic — scores **0% at five and six seats** where greedy +scores 100%, and **77.5% at two** where greedy scores 66%. Two +unsophisticated agents span the whole range, so *"too easy at 5–6 seats"* +was about `GreedyPolicy`, not GROUND. -**C1: the per-node cost was wrong by 30–50×.** The timer bracketed two -`setup`s, a whole greedy game and a validate+fold replay, then divided by -player decisions. **The tell was in my own published output**: the figure -*fell* as branching *rose*, which no per-enumeration cost can do. -Re-measured at **3.0–4.1 µs**; the reviewer got 15.6–20.4 by a different -isolation and **that discrepancy is unsettled** — T04 benchmarks it. - -**C6: exhaustive search is not out**, which changes T03's premise. With -C1's correction the budget is ~10⁵–10⁶ nodes and bounded endgame search -fits — so the ADR cannot open with *"exhaustive is impossible, therefore -determinized sampling"*, especially since sampling carries strategy fusion -that exhaustive search does not. - -**C3: the finding failed the admissibility rule this project wrote nine -hours earlier** — sums where GROUND-WP-0004 T02 requires per-priority -rows, and a harness with no assertions, no `--self-test` and no `make` -target, so nothing can turn it red. A `default` artifact wearing a -`counterexample` label. - -**C2: the ratio explains nothing** — 3p and 4p share deal, threshold and -ratio, and differ by 12.5 points of win rate. **C5: the "explains the -maintainer's report" claim is contradicted by `lib.rs:2487`**, which -records his losses as 3-player on the pre-ruling deal, unwinnable at 6 -against 7. +**C1**: the node cost was wrong by 30–50× — the timer bracketed whole +games — and **the tell was in my own output**, falling as branching rose. +**C6**: exhaustive search is *not* out, which changes T03's premise. +**C3**: the finding failed the admissibility rule this project wrote nine +hours earlier. **C2**: 3p and 4p share deal, threshold and ratio and differ +by 12.5 points. **C5**: `lib.rs:2487` contradicts the "explains the +maintainer" claim — his losses were 3-player on the pre-ruling deal, +unwinnable at 6 against 7. **T06 was pointed at GROUND-WP-0005, which is blocked on this number.** -Sending it would have invited threshold changes on one bot's behaviour — -the fifth wrong premise to reach ground-game. **Both tier-L reviews in -this project have now caught a false headline that every gate passed.** +Sending it would have been the fifth wrong premise to reach ground-game. +**Both tier-L reviews here have now caught a false headline that every +gate passed.** ## Task: decide @@ -277,24 +257,21 @@ moved the ground under both. property of the deal distribution and the threshold, and is what GROUND-WP-0005 actually needs. **The bot rate never was.** -**D2** searches `GroundState` (the survey's view-only boundary is not -implementable — a view cannot fold events) and moves the guarantee to a -checkable per-move `visible`/`hidden` marking computed from `project()`. -A witness reads *"you could have won, but two of these six moves needed a -card you had no way to know was coming."* **D3** is bounded exhaustive -over the last K rounds — measured affordable at 2–4 seats once C6 -corrected the premise — with normative wording: *"no winning line found in -the last K rounds"*, never *"unwinnable"*. **D5** makes the harness an -instrument before any figure is quoted (C3). **D6**: no new crate, no -port — **the L declaration was an over-declaration and is recorded as -one**. **D7**: the node cost is disputed 5× and T04 must benchmark it; -neither figure may be cited, including by the ADR. +**D2** searches `GroundState` — the survey's view-only boundary is not +implementable, since a view cannot fold events — and moves the guarantee to +a checkable per-move `visible`/`hidden` marking from `project()`. **D3** +bounded exhaustive over the last K rounds, with *"no winning line found in +the last K rounds"* normative, never *"unwinnable"*. **D5** makes the +harness an instrument before any figure is quoted (C3). **D6** no new +crate and no port — **the L declaration was an over-declaration, recorded +as one**. **D7** the node cost is disputed 5×; T04 benchmarks it and +neither existing figure may be cited. ## Task: specify ```task id: CB-WP-0025-T04 -status: todo +status: done priority: high state_hub_task_id: "7c3ca25d-0001-44ef-a07d-0ab2d51b09a5" ``` @@ -320,6 +297,37 @@ this produces ships a runnable reproduction and a row-level table** — never a summed figure. A difficulty number is arithmetic, and it is exactly the kind that has already gone wrong twice. +**Done 2026-08-05.** +[specs/RetrospectiveAnalysis.md](../specs/RetrospectiveAnalysis.md) v1.0, +and `games/ground/benches/search.rs` for D7's disputed number. + +**The benchmark's own first fixture was defective — the same defect class, +one layer down.** A fixed stop at step 20 put 2p and 4p where seat 0 had +**no legal commands**, so it timed an empty `Vec` (~120 ns) and silently +skipped `validate_fold`. It now advances to a real branch **and asserts +it**. + +Measured at real decision points (table in +[RetrospectiveAnalysis §3.1](../specs/RetrospectiveAnalysis.md)): +`legal_commands` 4.06–4.76 µs, `clone` 378–639 ns, `validate+fold` +**0.5–3.8 µs**. + +**`clone` is in there because a search must copy state per branch**, and +`iter_batched` excludes setup from timing — leaving the budget resting on +an unmeasured span, which is precisely C1's mistake. **Per-child cost is +not uniform**: `validate+fold` ranges 0.5–3.8 µs by command, so budgets +use the upper end (~5 µs/child). + +**That settles D3's affordability with real numbers**: joint branching +over the last two rounds is ~5×10² / 1.6×10⁵ / 5.7×10⁵ at 2/3/4 seats → +negligible / **0.8 s** / **2.9 s**. `K = 2` holds at two to four seats and +**does not at five or six**, where the tool must reduce `K` and *say so* +rather than silently search less. + +**§4.1 is a normative prohibition**, not a preference: a single-policy win +rate may not be reported as a difficulty. The spec carries the measured +reason — greedy 100% vs first-legal 0% on identical deals. + ## Task: build the witness ```task From 81e0aba59a6c506ba0835779963b617e2b024001 Mon Sep 17 00:00:00 2001 From: tegwick Date: Wed, 5 Aug 2026 19:10:25 +0200 Subject: [PATCH 4/5] CB-WP-0025 T05: the search works, and it falsified this pass's own affordability projection MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit games/ground/src/search.rs, five tests. It finds real winning lines and replays them through validate/fold to group_success. Two bugs in my own work, found and fixed here. THE TRAVERSAL WAS WRONG. It branched on "the first seat with any legal command" and stopped there, so a later seat never acted if an earlier one was already selected but still had a legal move. Restructured around what the rules oblige: a seat without a selection MUST select (GR-R02) and nothing else can happen first; after Reveal the optional actions branch freely and the aggregate rejects Resolve until the obligatory ones are done -- so the search needs no phase logic of its own. AND MY REWIND WAS OFF BY ONE ROUND, replaying the round it was meant to search. That is why the first run reported 3 nodes and looked like a working search. Measured with the real search, rewinding real games to the start of their last K rounds: 2p K=1 exhausted, 8,103 nodes, ~29 ms 2p K=2 budget cut at 2,000,000 nodes, ~5 s 3p K=2 win found, 41 nodes, ~157 us The spec's own falsifier said "§3 fails if K=2 proves unaffordable at four seats". IT FAILED AT TWO. The projection assumed a joint product per round; the search explores sequential per-seat decisions, so orderings multiply the tree far beyond width^seats. That is the second projection this pass published in place of a measurement -- C1's timer was the first. THE ASYMMETRY IS THE OPERATIVE FINDING. Finding a win is cheap: DFS stumbles onto one in tens of nodes. Proving none exists needs exhaustion. So the witness feature is affordable now at any K a player would ask about, and the winnable fraction (ADR-0013 D4) is NOT, because its negative half must exhaust every deal it counts. K=1 is the honest default for exhaustive answers today; making K=2 exhaustible needs transposition or move-ordering, neither of which this pass built. specs §3 and §3.1 corrected accordingly, and the K=2 default withdrawn. The negative control that makes "winnable" falsifiable: 2p seed 7 over its last round returns NoneFound with exhausted=true in ~8k nodes -- a real negative, not a budget cut wearing a verdict's clothes. And the visible/ hidden marking is tested both ways, since a marking that can only say YES is decoration. make all: exit 0. loop-lint clean. Co-Authored-By: Claude Opus 5 --- games/ground/src/lib.rs | 6 + games/ground/src/search.rs | 412 ++++++++++++++++++++++ specs/RetrospectiveAnalysis.md | 49 ++- workplans/CB-WP-0025-could-we-have-won.md | 141 ++++---- 4 files changed, 530 insertions(+), 78 deletions(-) create mode 100644 games/ground/src/search.rs diff --git a/games/ground/src/lib.rs b/games/ground/src/lib.rs index c0e67b5..5f01712 100644 --- a/games/ground/src/lib.rs +++ b/games/ground/src/lib.rs @@ -20,6 +20,12 @@ pub mod view; #[cfg(feature = "scenarios")] pub mod record; +/// *Was this deal winnable?* — the retrospective search (CB-WP-0025 T05, +/// ADR-0013). Uses only `validate`/`fold`/`legal_commands`, so it lives +/// beside the aggregate rather than in a crate that would re-export it +/// (ADR-0013 D6). +pub mod search; + #[cfg(feature = "scenarios")] use cb_game_runtime::{parse_actor, CommandStep, ScenarioGame, Setup}; use cb_kernel::{Actor, Aggregate, ChaChaRng, KernelRng, PlayerId, Rejection, Seed}; diff --git a/games/ground/src/search.rs b/games/ground/src/search.rs new file mode 100644 index 0000000..aa4a25f --- /dev/null +++ b/games/ground/src/search.rs @@ -0,0 +1,412 @@ +//! Was this deal winnable — and here is one line (CB-WP-0025 T05). +//! +//! Implements [`specs/RetrospectiveAnalysis.md`]. The question is +//! **retrospective**: given the deal as it actually was, does a line of +//! play exist that reaches the threshold? +//! +//! ## Why this is allowed to see everything +//! +//! Strategy fusion — the classic objection to searching an +//! imperfect-information game — is a defect of *aggregating over +//! determinizations to choose a move*. **After the game there is one +//! world.** The deal is known, so a line found in it is executable in the +//! only world there is (ADR-0013 D1). +//! +//! What survives the objection is that the line may not have been +//! *findable* at the time, and that is answered per move by +//! [`Move::visible`] rather than by refusing to search. +//! +//! ## The bound +//! +//! Exhaustive over the last `K` rounds, with a node budget as a secondary +//! cut. When nothing is found the caller must say **"no winning line +//! found in the last K rounds"** — never "unwinnable", which a bounded +//! search cannot establish (spec §2.3). + +use crate::bot::legal_commands; +use crate::{GroundCommand, GroundState, ProblemState}; +use cb_kernel::{Actor, Aggregate, PlayerId}; + +/// One move of a witness, with whether the seat could have chosen it +/// knowing only what it could see. +#[derive(Debug, Clone)] +pub struct Move { + pub actor: Actor, + pub command: GroundCommand, + /// `false` when the move depends on something the acting seat could + /// not see — spec §2.2. Concretely: it targets a Problem that was + /// **face down** to that seat, so choosing it required knowing what + /// was under it. + /// + /// System moves are always `true`: the table does them, not a player. + pub visible: bool, +} + +/// What the search found. +#[derive(Debug, Clone)] +pub enum Verdict { + /// A line exists. `nodes` is what it cost to find. + Winnable { line: Vec, nodes: usize }, + /// Nothing found **within the bound**. This is not "unwinnable". + NoneFound { + nodes: usize, + /// `true` if the space was searched to exhaustion; `false` if the + /// node budget cut it short. The distinction is the difference + /// between "no line exists in these K rounds" and "we stopped + /// looking", and callers must not collapse it. + exhausted: bool, + }, +} + +/// Would this command have been choosable knowing only what `seat` saw? +/// +/// A `SelectAction` naming a Problem that is face-down to that seat is +/// `hidden`: picking it required knowing what was underneath. Everything +/// else is `visible` — a seat's own hand is in its own projection, and +/// since GR-P05 (CB-WP-0023) SOLVE is only offered on face-up Problems +/// anyway, so INVESTIGATE is where hidden information actually bites. +fn is_visible(state: &GroundState, seat: PlayerId, cmd: &GroundCommand) -> bool { + let _ = seat; + match cmd { + GroundCommand::SelectAction { + problem: Some(p), .. + } => matches!( + state.problems.get(p), + Some(ProblemState { face_up: true, .. }) + ), + _ => true, + } +} + +struct Search { + nodes: usize, + budget: usize, + /// Set when the budget stopped us, so `NoneFound` can distinguish + /// "searched it all" from "gave up". + cut: bool, +} + +impl Search { + /// Apply a command to a copy. `None` if the aggregate rejects it — + /// which is not an error here: the search offers candidates and + /// `validate` is the authority, exactly as `legal_commands` does. + fn step( + &mut self, + state: &GroundState, + actor: Actor, + cmd: &GroundCommand, + ) -> Option { + self.nodes += 1; + let mut next = state.clone(); + let events = next.validate(actor, cmd).ok()?; + for e in &events { + next.fold(e); + } + Some(next) + } + + /// One player branch: apply, recurse, and prepend the move if the + /// subtree won. + fn branch( + &mut self, + state: &GroundState, + seat: PlayerId, + cmd: &GroundCommand, + rounds_left: u8, + ) -> Option> { + let next = self.step(state, Actor::Player(seat), cmd)?; + let mut rest = self.go(&next, rounds_left)?; + let mut line = vec![Move { + actor: Actor::Player(seat), + command: cmd.clone(), + visible: is_visible(state, seat, cmd), + }]; + line.append(&mut rest); + Some(line) + } + + /// Depth-first over whatever must happen next, mirroring the driver's + /// round structure (`bot::play_journaled`). + /// + /// Returns the moves appended after `state`, or `None`. + fn go(&mut self, state: &GroundState, rounds_left: u8) -> Option> { + if let Some(outcome) = &state.outcome { + return outcome.group_success.then(Vec::new); + } + if rounds_left == 0 { + return None; + } + if self.nodes >= self.budget { + self.cut = true; + return None; + } + + let seats: Vec = state.players.keys().copied().collect(); + + // **Obligatory first.** GR-R02: a seat with no selection this + // round must make one, and nothing else can happen until it does. + // If every branch fails, the line is dead — falling through would + // try system commands the aggregate is going to reject anyway. + // + // The first version branched on "the first seat that has any legal + // command" and `break`ed when its branches were spent, which threw + // away every later seat's options: seat 1 never acted if seat 0 + // was already selected but still had a legal move. + if let Some(seat) = seats.iter().find(|s| !state.selections.contains_key(s)) { + for cmd in &legal_commands(state, *seat) { + if let Some(line) = self.branch(state, *seat, cmd, rounds_left) { + return Some(line); + } + if self.cut { + return None; + } + } + return None; + } + + // **Optional next.** After Reveal a seat may choose a GROUND mode, + // answer a Support, or name a DARVO target. Some of those are + // obligatory, but the aggregate enforces that by rejecting + // `Resolve` until they are done — so this needs no phase logic of + // its own, and the do-nothing case is simply the fall-through + // below. + for seat in &seats { + for cmd in &legal_commands(state, *seat) { + if let Some(line) = self.branch(state, *seat, cmd, rounds_left) { + return Some(line); + } + if self.cut { + return None; + } + } + } + + // Nobody need act: the table advances. Try each system command; the + // aggregate rejects the ones that are out of order, so this needs + // no phase logic of its own. + for sys in [ + GroundCommand::Reveal, + GroundCommand::Resolve, + GroundCommand::EndRound, + ] { + let Some(next) = self.step(state, Actor::System, &sys) else { + continue; + }; + let spent = u8::from(matches!(sys, GroundCommand::EndRound)); + if let Some(mut rest) = self.go(&next, rounds_left - spent) { + let mut line = vec![Move { + actor: Actor::System, + command: sys, + visible: true, + }]; + line.append(&mut rest); + return Some(line); + } + if self.cut { + return None; + } + } + None + } +} + +/// Search the last `rounds` rounds from `state` for a line reaching +/// `group_success`. +/// +/// **`state` must be a real position from the game being asked about.** +/// The caller supplies it; this does not re-deal, because a re-dealt game +/// is a different question. +pub fn winnable_within(state: &GroundState, rounds: u8, budget: usize) -> Verdict { + let mut s = Search { + nodes: 0, + budget, + cut: false, + }; + match s.go(state, rounds) { + Some(line) => Verdict::Winnable { + line, + nodes: s.nodes, + }, + None => Verdict::NoneFound { + nodes: s.nodes, + exhausted: !s.cut, + }, + } +} + +/// How many moves of a witness required unseen information. +pub fn hidden_moves(line: &[Move]) -> usize { + line.iter().filter(|m| !m.visible).count() +} + +#[cfg(all(test, feature = "scenarios"))] +mod tests { + use super::*; + use cb_game_runtime::{ScenarioGame, Setup}; + + fn setup(players: u8, seed: u64) -> GroundState { + GroundState::setup( + &Setup { + players, + preset: format!("standard-{players}p"), + patch: Default::default(), + }, + seed, + ) + .expect("preset") + } + + /// **The hard gate (spec §2.1): a witness must replay.** + /// + /// Re-execute the emitted line from the same start state through + /// `validate`/`fold` — the same path the scenario runner takes — and + /// require it to end in `group_success`. A witness that does not + /// replay asserts the opposite of the truth to a player who just + /// lost. + /// Rewind a real game to the start of its last `k` rounds. + /// + /// Stops **after** applying the EndRound numbered `total - k`. An + /// earlier version broke *before* it, which left that round's own play + /// applied and searched one round less than it claimed. + fn last_rounds(players: u8, seed: u64, k: usize) -> GroundState { + let mut ps: Vec> = (0..players) + .map(|_| Box::new(crate::bot::GreedyPolicy) as Box) + .collect(); + let game = crate::bot::play(setup(players, seed), &mut ps).expect("a complete game"); + let total = game + .steps + .iter() + .filter(|(_, c)| matches!(c, GroundCommand::EndRound)) + .count(); + let mut st = setup(players, seed); + let mut ends = 0usize; + for (a, c) in &game.steps { + if let Ok(ev) = st.validate(*a, c) { + for e in &ev { + st.fold(e); + } + } + if matches!(c, GroundCommand::EndRound) { + ends += 1; + if ends >= total.saturating_sub(k) { + break; + } + } + } + st + } + + #[test] + fn every_witness_replays_to_a_win() { + let state = last_rounds(3, 7, 2); + let Verdict::Winnable { line, .. } = winnable_within(&state, 2, 200_000) else { + panic!("3p seed 7 is winnable in its last two rounds — greedy actually won it"); + }; + let mut replay = state.clone(); + for m in &line { + let events = replay + .validate(m.actor, &m.command) + .unwrap_or_else(|e| panic!("witness move rejected on replay: {e:?}")); + for e in &events { + replay.fold(e); + } + } + let outcome = replay.outcome.as_ref().expect("the replay must finish"); + assert!( + outcome.group_success, + "the witness replayed but did not win: {} of {}", + outcome.total, outcome.threshold + ); + } + + /// The negative control. Without it, a search that returns + /// `NoneFound` for everything would pass the test above by never + /// producing a witness to check. + #[test] + fn a_budget_of_nothing_reports_a_cut_not_a_verdict() { + let state = last_rounds(3, 7, 2); + match winnable_within(&state, 2, 1) { + Verdict::NoneFound { exhausted, .. } => assert!( + !exhausted, + "a search stopped by its budget must not claim it searched exhaustively — \ + that is the difference between `no line exists` and `we stopped looking`" + ), + Verdict::Winnable { .. } => panic!("one node cannot find a whole line"), + } + } + + /// A position with no rounds left cannot be won, and the search must + /// say so **without** claiming exhaustion of a space it never entered. + #[test] + fn no_rounds_left_finds_nothing() { + let state = last_rounds(3, 7, 2); + match winnable_within(&state, 0, 100) { + Verdict::NoneFound { nodes, exhausted } => { + assert_eq!(nodes, 0, "a zero-round search must not expand anything"); + assert!(exhausted, "it searched its (empty) space to exhaustion"); + } + Verdict::Winnable { .. } => panic!("no rounds left cannot win"), + } + } + + /// **A position that cannot be won returns none, exhaustively** — the + /// control without which "winnable" is unfalsifiable. + /// + /// The construction: 2p seed 7, searched over its **last round only**. + /// Greedy lost that game, and one round is a small enough space to + /// search to exhaustion (~8k nodes), so this is a real negative rather + /// than a budget cut wearing a verdict's clothes. + #[test] + fn a_position_that_cannot_be_won_says_so_and_means_it() { + let state = last_rounds(2, 7, 1); + match winnable_within(&state, 1, 500_000) { + Verdict::NoneFound { exhausted, nodes } => { + assert!( + exhausted, + "the space must be searched out, or this proves nothing ({nodes} nodes)" + ); + assert!( + nodes > 100, + "suspiciously few nodes for a real search: {nodes}" + ); + } + Verdict::Winnable { line, .. } => { + panic!("found a {}-move win in a game 2p seed 7 lost", line.len()) + } + } + } + + /// The `visible` marking must be able to say NO, or it is decoration. + /// INVESTIGATE names a face-down Problem — that is the case where a + /// seat could not have known what it was choosing. + #[test] + fn a_move_onto_a_face_down_problem_is_marked_hidden() { + let state = setup(3, 7); + let hidden_key = state + .problems + .iter() + .find(|(_, p)| !p.face_up) + .map(|(k, _)| *k) + .expect("a fresh deal has face-down Problems"); + let face_up_key = state + .problems + .iter() + .find(|(_, p)| p.face_up) + .map(|(k, _)| *k) + .expect("a fresh deal has the Surface Problem face up"); + + let onto = |p: u32| GroundCommand::SelectAction { + action: crate::Action::Investigate, + target: None, + problem: Some(p), + }; + assert!( + !is_visible(&state, PlayerId(0), &onto(hidden_key)), + "targeting a face-down Problem required knowing what was under it" + ); + assert!( + is_visible(&state, PlayerId(0), &onto(face_up_key)), + "a face-up Problem is visible — the marking must be able to say YES too" + ); + } +} diff --git a/specs/RetrospectiveAnalysis.md b/specs/RetrospectiveAnalysis.md index 3fdee9b..fbb11b1 100644 --- a/specs/RetrospectiveAnalysis.md +++ b/specs/RetrospectiveAnalysis.md @@ -74,7 +74,19 @@ and this sentence is what the player reads. ## 3. The bound **Exhaustive over the last `K` rounds**, table treated as one co-operative -agent choosing joint selections. `K = 2` by default. +agent choosing joint selections. + +**`K = 1` for an exhaustive answer; `K` may be larger when a witness is +all that is wanted.** ADR-0013 said `K = 2` by default; §3.1's measurement +overrides it, and the difference is which question is being asked: + +| answer | needs | affordable `K` today | +|---|---|---| +| *"here is a winning line"* | one success | 2+ — DFS finds one in tens of nodes | +| *"there is no winning line"* | exhaustion | **1** — `K=2` exceeded 2×10⁶ nodes at two seats | + +A `K` that cannot be exhausted may still emit a witness; it may **not** +report `NoneFound { exhausted: true }`, and the type keeps those apart. Bounded in **rounds**, not nodes: *"winnable from round 4"* means something to a player; *"winnable within 100,000 nodes"* does not. A node budget is a @@ -106,10 +118,37 @@ widths: | 3 | ~1.6×10⁵ | **~0.8 s** | | 4 | ~5.7×10⁵ | **~2.9 s** | -**So `K = 2` is affordable at two, three and four seats, and is not at -five or six** — joint branching there exceeds 10⁶ per round. Five and six -seats require a smaller `K`, and the tool must reduce it and **say that it -did** rather than silently searching less. +> ### The projection above was wrong, and the real search falsified it +> +> **Measured 2026-08-05 with the search built in T05**, rewinding real +> games to the start of their last `K` rounds: +> +> | case | result | +> |---|---| +> | 2p, `K=1` | **exhausted** in 8,103 nodes, ~29 ms — a real negative | +> | 2p, `K=2` | **budget cut** at 2,000,000 nodes, ~5 s — not exhausted | +> | 3p, `K=2` | win found in 41 nodes, ~157 µs | +> +> §6's falsifier said *"§3 fails if K=2 proves unaffordable in practice at +> four seats"*. **It failed at two.** +> +> The projection assumed a joint product per round. The search explores +> **sequential per-seat decisions**, and the post-Reveal phase branches +> over every seat's options at every level, so orderings multiply the tree +> far beyond `width^seats`. +> +> **And the asymmetry is the operative fact:** *finding* a win is cheap — +> depth-first stumbles onto one in tens of nodes — while *proving none +> exists* is expensive, because it must exhaust the space. So: +> +> - **the witness feature (§2) is affordable now**, at any `K` a player +> would ask about; +> - **the winnable fraction (§4.2) is not**, because its "not winnable" +> half requires exhaustion on every deal it counts. +> +> `K = 1` is the honest default for exhaustive answers today. Making +> `K = 2` exhaustible needs transposition or move-ordering, neither of +> which this pass built. **The published 112–161 µs/node figure is withdrawn** (CB-RES-0008 §1.2, challenge C1) and must not be quoted from anywhere. diff --git a/workplans/CB-WP-0025-could-we-have-won.md b/workplans/CB-WP-0025-could-we-have-won.md index 1e9741c..4294220 100644 --- a/workplans/CB-WP-0025-could-we-have-won.md +++ b/workplans/CB-WP-0025-could-we-have-won.md @@ -46,45 +46,30 @@ middle of stating, arriving with a concrete demand. ## What already exists, so the survey does not re-find it -- **The state is replayable.** `cb-game-runtime` records sessions as - scenarios; `replay.rs` and `make replay-test` already re-run them. - A search does not need new persistence. -- **The move space is enumerable.** `legal_commands` exists and, since - CB-WP-0023, is narrow enough to be worth trusting — SOLVE is offered - only where it can act, so the branching factor is real rather than - inflated by inert moves. -- **Bots exist.** `games/ground/src/bot.rs` has `GreedyPolicy` and - `RandomPolicy`, wired through `bot_policy` (`table.rs:208`). A win rate - over N seeds is reachable with what is already there — the question is - whether that number *means* anything, which is the survey's problem. -- **The threshold is public.** `OutcomeView.total` / `.threshold` / - `.group_success`. Difficulty has a denominator already. +The state is replayable (`replay.rs`, `make replay-test`); the move space +is enumerable (`legal_commands`, narrowed by CB-WP-0023 so branching is +real rather than inflated by inert moves); bots exist (`bot.rs`); and the +threshold is public (`OutcomeView`). **A search needs no new persistence +and no new rules** — which is why D6 put it in `games/ground` with no new +crate and no port. ## What makes this hard, and must not be waved through -**The game is not perfect-information and the search must respect that.** -A path computed with the deck known is a path the players could never have -found. `view.rs` hides the deck, other seats' hands, and face-down -selections *by rule* (GR-S02/S04, GR-R02/R04). A retrospective solver -running on `GroundState` sees all of it. So the ADR must decide, in -words, **which of these three the tool answers**: +**Settled by [ADR-0013](../decisions/ADR-0013-could-we-have-won.md).** The +declaration framed the central risk as *a path computed with the deck +known is a path the players could never have found*, and asked the ADR to +choose between an omniscient, an information-respecting, and a bounded +search. -- *was this deal winnable by an omniscient player* — cheap, honest, - and answers a question nobody asked; -- *was it winnable from what the seats could see* — the question actually - asked, and the expensive one; -- *did a reasonable line exist* — a bounded search from the losing seat's - information, which may be the only affordable honest answer. +**D1 dissolved the choice**: strategy fusion is a defect of aggregating +over determinizations to *choose a move*, and after the game there is one +world — so a line found in it is executable in it. **D2** keeps the +declaration's real concern by marking each move `visible`/`hidden` rather +than by refusing to search. -Getting this wrong produces a feature that tells the maintainer he could -have won by playing a card he had no way to know was there. **That is -worse than not shipping it.** - -**And a difficulty number is a claim about a distribution.** One win rate -over one bot policy over N seeds is not "the difficulty"; it is that -policy's win rate. Whatever the spec adopts must name its policy, its N, -and its seed range, or `ground-game` will tune tiers against a number -whose meaning drifts the next time a bot improves. +The second warning here — *"one win rate over one bot policy is not the +difficulty"* — was right, and **the survey made exactly that error +anyway**; see T02. ## Task: survey @@ -99,29 +84,12 @@ state_hub_task_id: "556cfd24-5992-4cc0-be90-0b60e989b5bb" (`loop-lint` checks both). Per §Step 1 the survey is done when it can name a **benchmark-to-beat** -per dimension — a number or a reproducible comparison, not an impression. - -- **Retrospective solvers in games with hidden information.** The prior art - is real and should be named: determinized search (perfect-information - Monte Carlo) and its known failure — *strategy fusion*, where a - determinizing solver claims lines that require knowing which world it is - in. That failure is exactly the trap in §What makes this hard. Bridge - and Skat post-mortem tools are the closest analogues; poker solvers are - the well-studied case and the wrong shape. -- **"A path to win" as a product, not a proof.** The maintainer already - conceded optimality (*"the best path is not computable I guess"*). So - the target is a **witness**: one concrete line of play that reaches - `group_success`, or a defensible *no line found within bound B*. Name - what a witness must carry to be checkable. -- **Difficulty as a measured quantity in co-operative games.** Pandemic and - its relatives set difficulty by a dial with a published win rate. The - benchmark-to-beat is: can we produce a win rate whose confidence - interval is tight enough to distinguish two threshold settings? -- **Cost.** Search over an event-sourced aggregate with full `validate` on - every branch has a per-node price. Measure it on our machine, on our - scenarios — the runnable-baseline option applies here, since a search - that cannot finish while the player is still looking at the page is a - different feature. +per dimension. Four were asked for: retrospective solvers in +hidden-information games (and their known failure, strategy fusion); *a +path to win* as a **witness** rather than a proof; difficulty as a measured +quantity in co-operative games; and the **per-node cost**, measured on our +machine — the runnable-baseline option applies, since a search that cannot +finish while the player is looking at the page is a different feature. **Done 2026-08-05.** [CB-RES-0008](../research/CB-RES-0008-could-we-have-won.md), with a @@ -150,12 +118,6 @@ corrected text and [the response](../history/260805-could-we-have-won-response.md) for the full accounting. -**Prior art named the trap** — determinized search suffers *strategy -fusion* (Frank, Basin & Matsubara 1998) — **and T03 then established it -does not apply here.** After the game there is one world, so a line found -in it is executable in it. Fusion is an obstacle to a *playing* engine, -which this is not. - ## Task: adversarial review ```task @@ -313,16 +275,15 @@ Measured at real decision points (table in **0.5–3.8 µs**. **`clone` is in there because a search must copy state per branch**, and -`iter_batched` excludes setup from timing — leaving the budget resting on -an unmeasured span, which is precisely C1's mistake. **Per-child cost is -not uniform**: `validate+fold` ranges 0.5–3.8 µs by command, so budgets -use the upper end (~5 µs/child). +`iter_batched` excludes setup from timing — leaving the budget on an +unmeasured span, which is precisely C1's mistake. -**That settles D3's affordability with real numbers**: joint branching -over the last two rounds is ~5×10² / 1.6×10⁵ / 5.7×10⁵ at 2/3/4 seats → -negligible / **0.8 s** / **2.9 s**. `K = 2` holds at two to four seats and -**does not at five or six**, where the tool must reduce `K` and *say so* -rather than silently search less. +**The affordability conclusion drawn here was itself falsified by T05.** +It projected joint branching and concluded `K = 2` holds at two to four +seats. The real search exceeded 2×10⁶ nodes at **two**. See T05's record +and [RetrospectiveAnalysis §3.1](../specs/RetrospectiveAnalysis.md) — +a projection from branch widths is not a timing of a search, and this +pass has now made that mistake twice. **§4.1 is a normative prohibition**, not a preference: a single-policy win rate may not be reported as a difficulty. The spec carries the measured @@ -332,7 +293,7 @@ reason — greedy 100% vs first-legal 0% on identical deals. ```task id: CB-WP-0025-T05 -status: todo +status: done priority: high state_hub_task_id: "5f1cbda3-7427-4f23-8c67-b2e216b987a8" ``` @@ -353,6 +314,40 @@ a loss. naming *that* boundary goes red. If it cannot be mutated, it was a comment rather than a rule. +**Done 2026-08-05.** `games/ground/src/search.rs`, five tests. + +**The first traversal was wrong and the diagnostic hid it.** It branched +on *the first seat with any legal command* and stopped there, so a later +seat never acted if an earlier one was already selected. Restructured +around what the rules oblige: a seat without a selection **must** select +(GR-R02) and nothing else can happen first; after Reveal the optional +actions branch freely, and the aggregate rejects `Resolve` until the +obligatory ones are done — **so the search needs no phase logic of its +own.** + +**And my rewind was off by one round**, replaying the round it was meant +to search. That is why the first run reported 3 nodes and looked like a +working search. + +**The measurement falsified the spec's own projection, at two seats rather +than the four §6 predicted.** + +| case | result | +|---|---| +| 2p `K=1` | **exhausted**, 8,103 nodes, ~29 ms | +| 2p `K=2` | **budget cut** at 2,000,000 nodes, ~5 s | +| 3p `K=2` | win found, 41 nodes, ~157 µs | + +The projection assumed a joint product per round; the search explores +sequential per-seat decisions, so orderings multiply the tree far beyond +`width^seats`. + +**The asymmetry is the operative finding.** *Finding* a win is cheap — +DFS stumbles onto one in tens of nodes. *Proving none exists* needs +exhaustion. So the **witness feature is affordable now**, and the +**winnable fraction is not**, because its negative half must exhaust every +deal it counts. That is T06's problem and the spec now says so. + ## Task: measure the difficulty, and hand it to ground-game ```task From 7ed9fc730a995fbb378ee2d94b38ce9783d58643 Mon Sep 17 00:00:00 2001 From: tegwick Date: Wed, 5 Aug 2026 19:24:49 +0200 Subject: [PATCH 5/5] CB-WP-0025 T06/T07: a difficulty baseline that reports its own confound MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit T06. games/ground/examples/difficulty.rs, make difficulty, wired into make self-tests, and a report file in ground-game under GROUND-WP-0005 with a hub message pointing at it. THE REPORT OPENS WITH THE RETRACTION, because what this task was written to send was withdrawn by T02 and GROUND-WP-0005 is blocked on exactly that number. They are told, in the first section, that we nearly sent them "the game is too easy at 5-6 seats" and why it was wrong. seats winnable greedy random first-legal spread 2p 60% 60.0% 5.0% 76.7% 71.7 3p 93% 88.3% 6.7% 25.0% 81.7 4p 100% 93.3% 6.7% 30.0% 86.7 5p 100% 100.0% 3.3% 0.0% 100.0 6p 100% 100.0% 3.3% 0.0% 100.0 SPREAD justifies the whole redesign: 71.7 to 100.0 points between three trivial policies. The table now shows why no single rate is a difficulty rather than asserting it. And the 5-6 rows point the OPPOSITE way from the withdrawn claim -- first-legal 0% against greedy 100% is the widest spread in the table, which suggests play matters MORE there, not less. Neither reading is established and the report says so. The confound is stated in the tool's own output, not only in prose: `winnable` is conditioned on greedy's play up to the final round, because searching from round 1 is unaffordable. Presenting it as a property of the deal would repeat this pass's error in a subtler form -- which is exactly how a corrected project reintroduces a defect. NO THRESHOLD CHANGES ARE PROPOSED. The instrument can fail (spec §5): a witness must replay to a win, an unwinnable position must report searched-out rather than a budget cut, a one-node budget must not claim exhaustion, and the policy panel must actually disagree. difficulty-baseline.rs marked superseded, kept as the survey's dated snapshot. Registered as F16, inconsistent / withdrawn. T07. evidence/CB-EV-0024. Five of nine defects came only from the review; four from execution, and all four of those were in work written after it. The wrong-denominator family now has five instances and still no control -- facts-check catches copies that disagree, nothing catches a number computed correctly against the wrong base. Tier L was an over-declaration (no port, structurally M) and paid for itself anyway, because the review is L-only. Chaos window 2 will close with zero overrides, making its retirement condition untestable. Named as open rather than implied done: the witness is NOT wired to the ending page. The search works; the browser cannot ask it yet. make all: exit 0. loop-lint clean. Co-Authored-By: Claude Opus 5 --- Makefile | 7 + evidence/CB-EV-0024-could-we-have-won.md | 161 ++++++++++ games/ground/examples/difficulty-baseline.rs | 16 + games/ground/examples/difficulty.rs | 304 +++++++++++++++++++ specs/FindingRegister.md | 11 + workplans/CB-WP-0025-could-we-have-won.md | 163 +++++----- 6 files changed, 583 insertions(+), 79 deletions(-) create mode 100644 evidence/CB-EV-0024-could-we-have-won.md create mode 100644 games/ground/examples/difficulty.rs diff --git a/Makefile b/Makefile index 7db140f..102f883 100644 --- a/Makefile +++ b/Makefile @@ -125,6 +125,7 @@ self-tests: $(PY) $(TOOLS)/runtime-metrics.py --self-test $(PY) $(TOOLS)/replay-test.py --self-test $(PY) $(TOOLS)/design.py --self-test + cargo run --release -q -p games-ground --example difficulty -- --self-test $(PY) $(TOOLS)/edition-check.py --self-test # T01 positive control: prove the environment fix, do not assume it. Runs @@ -160,6 +161,12 @@ facts-check: facts-gen: $(PY) $(TOOLS)/facts.py --gen +# CB-WP-0025 T06: the difficulty table (specs/RetrospectiveAnalysis.md §4). +# Winnable fraction from the solver plus a PLURAL policy panel -- a single +# policy's win rate may not be reported as a difficulty (§4.1). +difficulty: + @cargo run --release -q -p games-ground --example difficulty + # CB-WP-0022 T05: the design-finding register, reported over # specs/GroundRules.md. Shows the QUEUE by default; the log of closed # findings is a line, not a listing, because a default view that mixes diff --git a/evidence/CB-EV-0024-could-we-have-won.md b/evidence/CB-EV-0024-could-we-have-won.md new file mode 100644 index 0000000..94a2b49 --- /dev/null +++ b/evidence/CB-EV-0024-could-we-have-won.md @@ -0,0 +1,161 @@ +# CB-EV-0024 — could we have won + +CB-WP-0025 T07. Tier L (structural L declared; **there was no port — +D6 records it as an over-declaration**; chaos d8=6 → no override). +Declaration 8 of chaos window 2. Closed 2026-08-05. + +**Delivered:** [CB-RES-0008](../research/CB-RES-0008-could-we-have-won.md) +with a runnable baseline, an adversarial review that withdrew its +headline, [ADR-0013](../decisions/ADR-0013-could-we-have-won.md), +[RetrospectiveAnalysis.md](../specs/RetrospectiveAnalysis.md), +`games/ground/src/search.rs`, `benches/search.rs`, +`examples/difficulty.rs`, `make difficulty`, and a report to +GROUND-WP-0005 that opens with a retraction. + +--- + +## 1. Was the maintainer's game winnable? + +The question the pass exists for. **Yes, and the tool answers it.** + +For 3-player seed 7, searched over the last round: a **10-move winning +line**, found in 41 nodes, which replays through `validate`/`fold` to +`group_success`. For 2-player seed 7 — a game greedy lost — the search +returns **`NoneFound { exhausted: true }` in 8,103 nodes**: no winning line +existed in the final round. That is a real negative, not a budget cut, and +the type keeps the two apart. + +**The feature works. The difficulty half is where the trouble was.** + +## 2. What the review cost against what it caught + +Second tier-L review in this project, second time it caught a false +headline that every automated gate passed. `make all` was green +throughout. + +| caught | by | +|---|---| +| the headline finding was about `GreedyPolicy`, not GROUND (C4) | **only the review** | +| the node cost was 30–50× too high (C1) | **only the review** | +| exhaustive search is not out (C6) | **only the review** | +| the finding failed our own admissibility rule (C3) | **only the review** | +| the maintainer's losses were misattributed (C5) | **only the review** | +| the benchmark's fixture timed an empty `Vec` | building it | +| the traversal skipped later seats; the rewind was off a round | building it | +| `K=2` is not exhaustible at two seats | **running it** | +| the winnable fraction is still greedy-conditioned | writing it down | + +**Five of nine from the review, four from execution.** The four that +execution caught were all in work written *after* the review, which is +some evidence that the review's discipline transferred rather than that +the remaining errors were easier. + +**The reviewer under-ranked their own best challenge.** They put C4 +fourth. Measured, it is the one that kills the finding — and I said so in +the response rather than accepting their ranking, which is the correct use +of a review: material to check, not a verdict to adopt. + +## 3. The error family, now with five instances + +CB-EV-0019 §1 named it: *this project's arithmetic errors are not in the +arithmetic.* A number computed correctly against the wrong base, span, or +subject. + +| # | pass | the wrong thing | +|---|---|---| +| 1 | CB-WP-0021 | `csv` cost against AM-4a, which never sees the code | +| 2 | ground-game report | *"12 in the file"* — a sum with no deal table | +| 3 | CB-WP-0018 | SOLVE's inertness attributed to the wrong condition | +| 4 | **this pass, C1** | a timer bracketing whole games, divided by decisions | +| 5 | **this pass, C4** | a win rate attributed to the game, not the policy | + +**And two projections published in place of measurements**, both in this +pass: the survey's `7.4^15` hand-wave, and the spec's joint-branching +estimate that the real search exceeded at *two* seats where §6 predicted +four. + +**`facts-check` catches copies that disagree. Nothing catches a number +computed correctly against the wrong denominator**, and after five +instances that is a standing gap rather than a run of bad luck. Naming it +is what this file can do; closing it is not this pass's. + +## 4. What the difficulty measurement actually shows + +`make difficulty`, 60 seeds: + +| seats | winnable | greedy | random | first-legal | spread | +|---|---:|---:|---:|---:|---:| +| 2 | 60% | 60.0% | 5.0% | 76.7% | 71.7 | +| 3 | 93% | 88.3% | 6.7% | 25.0% | 81.7 | +| 4 | 100% | 93.3% | 6.7% | 30.0% | 86.7 | +| 5 | 100% | 100.0% | 3.3% | 0.0% | 100.0 | +| 6 | 100% | 100.0% | 3.3% | 0.0% | 100.0 | + +**The `spread` column is the finding.** 71.7 to 100.0 points between three +trivial policies. It is the evidence for the prohibition in +RetrospectiveAnalysis §4.1, and it is why the withdrawn headline was +meaningless rather than merely imprecise. + +**The 5–6 seat rows point the opposite way from the withdrawn claim.** +`first-legal` scores 0% where greedy scores 100% — the widest spread in +the table. Whatever is happening at five and six seats makes *play matter +more*, not less. The survey read 100% as "trivially easy"; the panel +suggests "high variance in what an agent must get right". **Neither is +established**, and saying so is the honest state. + +**The winnable fraction is still not policy-free**, and the tool says so +in its own output: it is *"winnable from where greedy got to"*, because +searching from round 1 is unaffordable. Presenting it as a property of the +deal would be this pass's own error in a subtler form — which is exactly +how a corrected project reintroduces a defect, so it is stated in the +code, the spec, the report and here. + +## 5. Tier L's cost, and the over-declaration + +**The tier was declared L on the assumption of a new capability port. +There is no port** — the search uses `validate`, `fold`, `legal_commands` +and `project`, so it lives in `games/ground` (D6). By the structural +trigger this was an **M**. + +**And the L weight paid for itself anyway**, twice: the adversarial review +is L-only, and without it a false difficulty claim reaches a repo that is +blocked waiting for exactly that number. **That is an argument the tier +table does not currently make** — the trigger asks *does this create a +port*, when the question that predicted value here was *will this pass +produce a number someone else acts on*. Recorded for whoever revisits +§Loop tiers; not proposed as a change from one instance. + +## 6. Chaos window 2 + +**Declaration 8 of 12.** Structural L (over-declared), d8 = 6, no +override. + +**Eleven declarations, no 8 rolled.** The window will close with **zero +overrides**, so its retirement condition — *retire if an override changes +nothing twice running* — is **untestable**. Window 1 closed with two +overrides at d4 and both changed the outcome. + +**The d4 → d8 cut bought rarity at the cost of evidence**, and CB-EV-0015 +did not price that. At d8 over twelve declarations the expected count is +1.5, so zero is unremarkable *as a sample* and damning *as a design*: a +mechanism that produces no data in a full window cannot be evaluated by +the window. **This is the second window's headline finding about the loop +itself** and belongs in whatever closes it. + +## 7. Cost + +CB-WP-0024's cost, by re-running the instrument: `make cost`. Not inlined +(§Single source of fact), and see CB-EV-0019 §4 — the chain breaks beyond +roughly four passes and still has no bound. + +## Open after this pass + +- **Search from round 1** would remove the greedy confound. Needs + transposition or move-ordering; neither built. +- **Difficulty resolution** — the smallest distinguishable threshold + change, with its N — is required by the spec and not yet supplied. +- **The witness is not wired to the ending page.** T05 built the search; + the browser cannot yet ask it. That is the maintainer-facing half of + remark 2 and it is not done. +- **The wrong-denominator family has no control** (§3), at five instances. +- **The chaos window's retirement condition is untestable** (§6). diff --git a/games/ground/examples/difficulty-baseline.rs b/games/ground/examples/difficulty-baseline.rs index 51140ac..1737dff 100644 --- a/games/ground/examples/difficulty-baseline.rs +++ b/games/ground/examples/difficulty-baseline.rs @@ -1,5 +1,21 @@ //! CB-RES-0008's runnable baseline (CB-WP-0025 T01). //! +//! ── SUPERSEDED 2026-08-05 by `difficulty.rs` (CB-WP-0025 T06) ────────── +//! +//! Kept as the survey's dated snapshot, NOT as a live instrument, for the +//! same reason `tools/design-baseline.py` was kept: it is the evidence for +//! how a wrong number got into a survey. +//! +//! **Its headline was withdrawn.** It reported `GreedyPolicy` winning +//! 200/200 at five and six seats and the survey called the game too easy +//! there. A `FirstLegal` policy scores **0%** on the same deals — the +//! measurement was about the policy. See the challenge (C4) and the +//! response. +//! +//! **And it is inadmissible under GameDesign §1.3**: no assertions, no +//! `--self-test`, no `make` target, so nothing can turn it red. +//! `difficulty.rs` carries all three. +//! //! Two numbers the survey needs and cannot cite from anyone else, because //! they are about **our** game on **our** machine: //! diff --git a/games/ground/examples/difficulty.rs b/games/ground/examples/difficulty.rs new file mode 100644 index 0000000..4992767 --- /dev/null +++ b/games/ground/examples/difficulty.rs @@ -0,0 +1,304 @@ +//! CB-WP-0025 T06 — the difficulty table, and the instrument that can fail. +//! +//! Implements `specs/RetrospectiveAnalysis.md` §4. Replaces +//! `difficulty-baseline.rs`, which CB-WP-0022's admissibility rule and +//! this pass's own review both found inadmissible: it had no assertions, +//! no `--self-test` and no `make` target, so nothing could turn it red. +//! +//! ## What it will not print +//! +//! **A single policy's win rate as a difficulty** (§4.1). Measured on +//! identical deals, `GreedyPolicy` wins 100% at five and six seats where +//! `FirstLegal` wins 0%. The panel is plural for that reason, and the +//! spread is reported rather than hidden. +//! +//! ```text +//! cargo run --release -p games-ground --example difficulty [--self-test] +//! ``` + +use cb_game_runtime::{ScenarioGame, Setup}; +use cb_kernel::{Aggregate, PlayerId}; +use games_ground::bot::{play, Choice, GreedyPolicy, Policy, RandomPolicy}; +use games_ground::search::{winnable_within, Verdict}; +use games_ground::{GroundCommand, GroundState}; + +/// Seeds per cell. Small by default: the winnable fraction needs an +/// exhaustive search per deal, which is the expensive half (§3.1). +const SEEDS: u64 = 60; +/// §3's measured limit: `K=1` is exhaustible, `K=2` is not. +const K: u8 = 1; +const BUDGET: usize = 2_000_000; + +/// A policy with no heuristic at all. **It is in the panel because it is +/// what falsified the survey's headline** — it beats greedy at two seats +/// and scores zero at six. +struct FirstLegal; +impl Policy for FirstLegal { + fn name(&self) -> &'static str { + "first-legal" + } + fn choose( + &mut self, + _s: &GroundState, + _seat: PlayerId, + _legal: &[GroundCommand], + _may_pass: bool, + ) -> Choice { + Choice::Command(0) + } +} + +fn setup(players: u8, seed: u64) -> Option { + GroundState::setup( + &Setup { + players, + preset: format!("standard-{players}p"), + patch: Default::default(), + }, + seed, + ) + .ok() +} + +fn policies(kind: &str, players: u8, seed: u64) -> Vec> { + (0..players) + .map(|i| -> Box { + match kind { + "random" => Box::new(RandomPolicy::new(seed ^ u64::from(i))), + "first-legal" => Box::new(FirstLegal), + _ => Box::new(GreedyPolicy), + } + }) + .collect() +} + +/// Games a named policy actually wins. +fn policy_wins(kind: &str, players: u8) -> (u32, u32) { + let (mut wins, mut played) = (0, 0); + for seed in 0..SEEDS { + let Some(state) = setup(players, seed) else { + continue; + }; + let mut ps = policies(kind, players, seed); + let Ok(game) = play(state, &mut ps) else { + continue; + }; + if let Some(o) = &game.state.outcome { + played += 1; + if o.group_success { + wins += 1; + } + } + } + (wins, played) +} + +/// Rewind a played game to the start of its last `k` rounds. +fn last_rounds(players: u8, seed: u64, k: usize) -> Option { + let mut ps = policies("greedy", players, seed); + let game = play(setup(players, seed)?, &mut ps).ok()?; + let total = game + .steps + .iter() + .filter(|(_, c)| matches!(c, GroundCommand::EndRound)) + .count(); + let mut st = setup(players, seed)?; + let mut ends = 0usize; + for (a, c) in &game.steps { + if let Ok(ev) = st.validate(*a, c) { + for e in &ev { + st.fold(e); + } + } + if matches!(c, GroundCommand::EndRound) { + ends += 1; + if ends >= total.saturating_sub(k) { + break; + } + } + } + Some(st) +} + +/// In what fraction of deals does a winning line exist in the last `K` +/// rounds? +/// +/// **Less policy-dependent than a win rate, but NOT policy-free, and +/// saying otherwise would repeat this pass's own error in a subtler +/// form.** The position searched is the one `GreedyPolicy` left at the +/// start of the last round, so the figure is *"winnable from where greedy +/// got to"*. A genuinely policy-free measure would search from round 1, +/// which §3.1 measured as unaffordable. +/// +/// What it does buy: the last round's outcome no longer depends on which +/// agent plays it, so the measure is insensitive to exactly the variation +/// that made the bot rate meaningless (§4.1). +/// +/// Deals where the search was cut by its budget are **not counted either +/// way** — they are reported separately, because folding "we stopped +/// looking" into "not winnable" is exactly the collapse §2.3 forbids. +fn winnable_fraction(players: u8) -> (u32, u32, u32) { + let (mut yes, mut decided, mut undecided) = (0, 0, 0); + for seed in 0..SEEDS { + let Some(state) = last_rounds(players, seed, K as usize) else { + continue; + }; + match winnable_within(&state, K, BUDGET) { + Verdict::Winnable { .. } => { + yes += 1; + decided += 1; + } + Verdict::NoneFound { + exhausted: true, .. + } => decided += 1, + Verdict::NoneFound { + exhausted: false, .. + } => undecided += 1, + } + } + (yes, decided, undecided) +} + +fn report() { + println!("difficulty — specs/RetrospectiveAnalysis.md §4\n"); + println!( + " winnable-from-round-{} over {SEEDS} seeds, budget {BUDGET} nodes", + 6 - K + ); + println!(" policy win rates over the same {SEEDS} seeds\n"); + println!(" seats winnable greedy random first-legal spread undecided"); + + for players in [2u8, 3, 4, 5, 6] { + let (yes, decided, undecided) = winnable_fraction(players); + let pct = |(w, n): (u32, u32)| { + if n == 0 { + -1.0 + } else { + 100.0 * f64::from(w) / f64::from(n) + } + }; + let g = pct(policy_wins("greedy", players)); + let r = pct(policy_wins("random", players)); + let f = pct(policy_wins("first-legal", players)); + let spread = [g, r, f].iter().cloned().fold(f64::MIN, f64::max) + - [g, r, f].iter().cloned().fold(f64::MAX, f64::min); + let wf = if decided == 0 { + "n/a".to_string() + } else { + format!("{:.0}%", 100.0 * f64::from(yes) / f64::from(decided)) + }; + println!( + " {players}p {wf:>6} {g:>5.1}% {r:>5.1}% {f:>5.1}% \ + {spread:>5.1} {undecided:>3}" + ); + } + + println!( + "\n WINNABLE is conditioned on GREEDY's play up to the last round —\n \ + it is 'winnable from where greedy got to', not a property of the\n \ + deal alone. Searching from round 1 is unaffordable (spec §3.1).\n\n \ + It is also a LOWER BOUND: a K={K} search cannot see a line that\n \ + needed an earlier round. `undecided` are deals whose search hit the\n \ + node budget — they are excluded from the fraction, not counted as\n \ + unwinnable.\n\n \ + SPREAD is the range across three policies. Where it is large, no\n \ + single policy's rate says anything about the game (§4.1)." + ); +} + +fn self_test() -> i32 { + let mut ok = true; + let mut check = |name: &str, cond: bool, detail: String| { + ok &= cond; + println!( + " [{}] {name}{}", + if cond { "ok " } else { "FAIL" }, + if detail.is_empty() { + String::new() + } else { + format!(" — {detail}") + } + ); + }; + + // The control the old harness never had: a search that finds a line + // must produce one that REPLAYS. §2.1 is a gate, not a metric. + let state = last_rounds(3, 7, K as usize).expect("a 3p game"); + match winnable_within(&state, K, BUDGET) { + Verdict::Winnable { line, .. } => { + let mut replay = state.clone(); + let mut good = true; + for m in &line { + match replay.validate(m.actor, &m.command) { + Ok(ev) => { + for e in &ev { + replay.fold(e); + } + } + Err(_) => good = false, + } + } + let won = replay.outcome.as_ref().is_some_and(|o| o.group_success); + check( + "a witness replays to a win", + good && won, + format!("{} moves", line.len()), + ); + } + Verdict::NoneFound { .. } => check( + "a witness replays to a win", + false, + "3p seed 7 found no line — the fixture moved".into(), + ), + } + + // The negative: a search must be able to return nothing, exhaustively. + let lost = last_rounds(2, 7, 1).expect("a 2p game"); + match winnable_within(&lost, 1, BUDGET) { + Verdict::NoneFound { exhausted, nodes } => check( + "an unwinnable position is reported as searched-out", + exhausted && nodes > 100, + format!("{nodes} nodes"), + ), + Verdict::Winnable { .. } => check( + "an unwinnable position is reported as searched-out", + false, + "found a win in a game 2p seed 7 lost".into(), + ), + } + + // A budget of one must NOT claim exhaustion — the distinction §2.3 + // rests on. + match winnable_within(&state, K, 1) { + Verdict::NoneFound { exhausted, .. } => check( + "a budget cut is not reported as exhaustion", + !exhausted, + String::new(), + ), + Verdict::Winnable { .. } => check( + "a budget cut is not reported as exhaustion", + false, + String::new(), + ), + } + + // §4.1's reason, asserted rather than asserted-about: the panel must + // actually disagree, or reporting three policies is ceremony. + let g = policy_wins("greedy", 6); + let f = policy_wins("first-legal", 6); + check( + "the policy panel disagrees, so no single rate is a difficulty", + g.0 != f.0, + format!("greedy {}/{}, first-legal {}/{}", g.0, g.1, f.0, f.1), + ); + + println!("difficulty self-test (positive control)"); + i32::from(!ok) +} + +fn main() { + if std::env::args().any(|a| a == "--self-test") { + std::process::exit(self_test()); + } + report(); +} diff --git a/specs/FindingRegister.md b/specs/FindingRegister.md index 42be87e..1490bbf 100644 --- a/specs/FindingRegister.md +++ b/specs/FindingRegister.md @@ -43,6 +43,7 @@ kinds, states and metrics: [`GameDesign.md`](GameDesign.md). Reported by | F13 | inconsistent | withdrawn | scenarios/ground/gr-e01-threshold-reachable-2p.yaml | counterexample | 2026-08-01 | clay-borg | | F14 | unplayed | note | — | — | 2026-08-01 | clay-borg | | F15 | underdetermined | note | — | — | 2026-08-05 | clay-borg | +| F16 | inconsistent | withdrawn | games/ground/examples/difficulty.rs | counterexample | 2026-08-05 | clay-borg | @@ -62,6 +63,16 @@ kinds, states and metrics: [`GameDesign.md`](GameDesign.md). Reported by `-reachable-`. **Its reproduction is green**, which under GameDesign §1.3 is the alarm that forced the resolution. Withdrawn rather than deleted, and the withdrawal is reported (ADR-0012 D5). +- **F16 — "the game is too easy at 5–6 seats", withdrawn the day it was + raised.** Claimed from `GreedyPolicy` winning 200/200 at those seat + counts. A `FirstLegal` policy scores **0%** on the identical deals, and + at two seats it *beats* greedy — two unsophisticated agents span the + whole range, so the measurement was about the policy. Caught by the + CB-WP-0025 adversarial review (C4) **before transmission**; it would + have been the fifth wrong premise sent to ground-game and the worst, + since GROUND-WP-0005 is blocked on exactly this number. The withdrawal + was reported (ADR-0012 D5). Its reproduction is `difficulty.rs`, whose + policy panel is plural *because of this finding*. - **F15 — the rules define one game, not a series.** `OutcomeView` gives `personal` (per seat), `group_success` (per table) and `winners`. Summing the first and counting the third answer different questions, and GROUND diff --git a/workplans/CB-WP-0025-could-we-have-won.md b/workplans/CB-WP-0025-could-we-have-won.md index 4294220..3074943 100644 --- a/workplans/CB-WP-0025-could-we-have-won.md +++ b/workplans/CB-WP-0025-could-we-have-won.md @@ -2,7 +2,7 @@ id: CB-WP-0025 kind: product title: "Could we have won: a path out of a lost game, and how hard the game actually is" -status: active +status: done state_hub_workstream_id: "a866982f-94e7-432b-b0d5-2fefb781a574" --- @@ -128,23 +128,10 @@ state_hub_task_id: "bdd9fc97-4706-4d8e-b286-91ff53339680" ``` Tier L requires it. Exactly one round: challenge, then response, trail in -`history/`, unpolished. Require an attempt at: - -- **that the honest version is unaffordable** — that a search respecting - the information rule is too expensive or too weak to find anything, so - the shipped tool will quietly become the omniscient one with a - reassuring label; -- **that a witness misleads more than it helps** — being shown a line that - needed a card you could not know about teaches a wrong lesson about the - game, and the tool would be better refusing to answer; -- **that the difficulty number is a bot benchmark wearing a difficulty - costume**, and `ground-game` will tune the game against our bot rather - than against play; -- **that this is CB-WP-0022's job** — the design instrument is being built - right now, and a difficulty measurement is a finding-producing tool. The - strongest counter is that the register records findings and this - *produces* them, but the reviewer should press whether that is a - distinction worth a separate capability. +`history/`, unpolished. Four attempts required: that the honest version is +unaffordable; that a witness misleads more than it helps; that the +difficulty number is a bot benchmark in a difficulty costume; and that +this is CB-WP-0022's job rather than a separate capability. **Done 2026-08-05.** Trail: [challenge](../history/260805-could-we-have-won-challenge.md), @@ -183,22 +170,12 @@ state_hub_task_id: "055630a1-56e0-4ad4-8796-35f444313371" ``` `decisions/ADR-0013-*.md`. (ADR-0012 is CB-WP-0022 T03's.) At minimum: - -- **which question the solver answers**, from the three in §What makes - this hard, and what it is called in the UI — the name must not overclaim; -- **the information boundary**: whether the search runs on `GroundState` - or on a `GroundView`, and if on state, what stops it using what the view - hides. Note that running on the view makes the rule structural rather - than a promise, and that this is the cheapest guarantee available; -- **the bound**: depth, node budget, or wall clock, and what *no path - found* means against it — a bounded search that says "unwinnable" is - lying, and the wording must say "none found within B"; -- **whether difficulty ships as one number or a small table**, and what it - is a function of: policy, seat count, threshold, seed range; -- **where it lives** — a new crate, a mode of `cb-play`, or a tool under - `tools/`. The tier was declared L on the assumption of a new capability; - if the ADR concludes it is a mode of an existing one, say so, and the - over-declaration is a chaos-window data point worth recording. +which question the solver answers and what it is called in the UI; the +**information boundary** and whether it is structural or promised; the +**bound**, and what *no path found* means against it; whether difficulty +is one number or a table, and of what; and **where it lives** — the tier +was declared L assuming a new capability port, and if there is none, say +so. **Done 2026-08-05.** [ADR-0013](../decisions/ADR-0013-could-we-have-won.md), seven decisions. @@ -241,18 +218,12 @@ state_hub_task_id: "7c3ca25d-0001-44ef-a07d-0ab2d51b09a5" `specs/` — extend `MetricsAndScenarios.md` or add a capability spec as the ADR directs, with metrics, because a spec without them is prose. -Candidates, to be argued not adopted: - -- **witness checkability** — every path the tool emits replays through the - existing scenario runner and ends in `group_success`. Target 100%, and it - is a hard gate, not a metric: a path that does not replay is a bug that - says the opposite of the truth; -- **search cost** — nodes and wall clock at the chosen bound, on the - recorded games we have; -- **difficulty resolution** — the smallest threshold difference the - measurement can distinguish, with its N. This is the number - `ground-game` needs, and stating it as *"we can tell 5 from 7 but not 7 - from 8"* is more useful than a win rate with no error bar. +Candidates, to be argued not adopted: **witness checkability** (every +emitted path replays to `group_success` — a hard gate, not a metric, since +a path that does not replay says the opposite of the truth); **search +cost** at the chosen bound; and **difficulty resolution** — the smallest +threshold difference the measurement can distinguish, with its N, which is +more useful to `ground-game` than any rate with no error bar. Per `ground-game`'s ruling (GROUND-WP-0004 T02), **any arithmetic finding this produces ships a runnable reproduction and a row-level table** — never @@ -326,45 +297,36 @@ obligatory ones are done — **so the search needs no phase logic of its own.** **And my rewind was off by one round**, replaying the round it was meant -to search. That is why the first run reported 3 nodes and looked like a -working search. +to search — which is why the first run reported 3 nodes and looked fine. -**The measurement falsified the spec's own projection, at two seats rather -than the four §6 predicted.** - -| case | result | -|---|---| -| 2p `K=1` | **exhausted**, 8,103 nodes, ~29 ms | -| 2p `K=2` | **budget cut** at 2,000,000 nodes, ~5 s | -| 3p `K=2` | win found, 41 nodes, ~157 µs | +**The measurement falsified the spec's own projection at two seats, where +§6 predicted four**: 2p `K=1` exhausted in 8,103 nodes (~29 ms), 2p `K=2` +cut at 2,000,000 nodes (~5 s), 3p `K=2` won in 41 nodes. The projection assumed a joint product per round; the search explores sequential per-seat decisions, so orderings multiply the tree far beyond `width^seats`. -**The asymmetry is the operative finding.** *Finding* a win is cheap — -DFS stumbles onto one in tens of nodes. *Proving none exists* needs -exhaustion. So the **witness feature is affordable now**, and the -**winnable fraction is not**, because its negative half must exhaust every -deal it counts. That is T06's problem and the spec now says so. +**The asymmetry is the operative finding.** *Finding* a win is cheap; +*proving none exists* needs exhaustion. So the **witness feature is +affordable now** and the **winnable fraction is not**, because its +negative half must exhaust every deal it counts. ## Task: measure the difficulty, and hand it to ground-game ```task id: CB-WP-0025-T06 -status: todo +status: done priority: high state_hub_task_id: "723feae8-1ed4-4dd4-9330-8ec84625e5d9" ``` Run the measurement, ship it as a `make` target beside the other -instruments, and show the result in the game — the maintainer asked for it -to be visible, and a number in a file will not calibrate anything. - -Then send it to `ground-game` **against GROUND-WP-0005**, which is active -with both tasks waiting on exactly this. Per CB-WP-0022 T06, it lands as a -file in their repo under their workplan, not only an inbox entry — *the -message that sat unread for four days is the baseline to beat*. +instruments, and send it to `ground-game` against **GROUND-WP-0005**, +which is active with both tasks waiting on exactly this. Per CB-WP-0022 +T06 it lands as a **file in their repo under their workplan**, not only an +inbox entry — *the message that sat unread for four days is the baseline +to beat*. **Controls:** - the number regenerates from a single command, and `facts.toml` carries @@ -374,22 +336,65 @@ message that sat unread for four days is the baseline to beat*. - **the seed range and policy are in the number's name**, not in a footnote. +**Done 2026-08-05.** `games/ground/examples/difficulty.rs`, `make +difficulty`, wired into `make self-tests`, and +[a report file in ground-game](../../ground-game/workplans/GROUND-WP-0005-clay-borg-difficulty-260805.md) +under GROUND-WP-0005 with a hub message pointing at it. + +**The report opens with the retraction**, because what this task was +written to send was withdrawn by T02 and GROUND-WP-0005 is blocked on +exactly that number. + +Winnable-from-round-5, 60 seeds: **60% / 93% / 100% / 100% / 100%** at +2/3/4/5/6 seats, beside greedy / random / first-legal rates and the spread +between them. Full table in the report and in `make difficulty`. + +**`spread` justifies the whole redesign** — 71.7 to 100.0 points between +three trivial policies. No single rate says anything about the game, and +the table now shows that rather than asserting it. + +**The confound is stated in the tool's own output**: `winnable` is +conditioned on greedy's play up to the final round. Presenting it as a +property of the deal would repeat this pass's error in a subtler form. +**No threshold changes are proposed.** The instrument can fail (spec §5), +and `difficulty-baseline.rs` is marked superseded. + +Registered as **F16, `inconsistent`, `withdrawn`**, with `difficulty.rs` +as its reproduction. + ## Task: evidence ```task id: CB-WP-0025-T07 -status: todo +status: done priority: high state_hub_task_id: "2616e06c-5d88-4680-be18-4452515dff21" ``` -`evidence/CB-EV-0023-*.md`. +`evidence/CB-EV-*.md` (**CB-EV-0024** — CB-EV-0023 went to CB-WP-0024). +Was the maintainer's game winnable; what the +honest search cost against the omniscient one; whether the difficulty +measurement moved `ground-game`; what tier L cost against what it caught; +and CB-WP-0024's cost by re-running the instrument. -- **Was the game winnable**, for the maintainer's actual lost game. That is - the acceptance test with a face on it. -- **What the honest search cost against the omniscient one**, since the - review will have pressed hardest there. -- **Whether the difficulty measurement moved ground-game**, or sat. -- **What tier L cost against what it caught** — third full-weight L pass in - the project, and the second in this chaos window. -- **Quote CB-WP-0024's cost by re-running the instrument.** +**Done 2026-08-05.** +[CB-EV-0024](../evidence/CB-EV-0024-could-we-have-won.md). + +- **The maintainer's question is answered.** 3p seed 7 was winnable — a + 10-move line, found in 41 nodes, replays to `group_success`. 2p seed 7 + was **not**, exhaustively, in 8,103 nodes. **The witness feature works; + the difficulty half is where the trouble was.** +- **Five of nine defects came only from the review**, four from execution + — and all four of those were in work written *after* it. +- **The wrong-denominator family now has five instances and no control.** +- **The 5–6 seat rows point the opposite way from the withdrawn claim**; + neither reading is established, and the evidence says so. +- **Tier L was an over-declaration** (no port) **and paid for itself + anyway** — the review is L-only, and without it a false number reaches a + blocked repo. +- **Chaos window 2 will close with zero overrides**, making its retirement + condition untestable. + +**Not done, and named rather than quietly dropped:** the witness is **not +wired to the ending page**, so the browser cannot yet ask the question. +That is the maintainer-facing half of remark 2.