diff --git a/Makefile b/Makefile index 7db140f..102f883 100644 --- a/Makefile +++ b/Makefile @@ -125,6 +125,7 @@ self-tests: $(PY) $(TOOLS)/runtime-metrics.py --self-test $(PY) $(TOOLS)/replay-test.py --self-test $(PY) $(TOOLS)/design.py --self-test + cargo run --release -q -p games-ground --example difficulty -- --self-test $(PY) $(TOOLS)/edition-check.py --self-test # T01 positive control: prove the environment fix, do not assume it. Runs @@ -160,6 +161,12 @@ facts-check: facts-gen: $(PY) $(TOOLS)/facts.py --gen +# CB-WP-0025 T06: the difficulty table (specs/RetrospectiveAnalysis.md §4). +# Winnable fraction from the solver plus a PLURAL policy panel -- a single +# policy's win rate may not be reported as a difficulty (§4.1). +difficulty: + @cargo run --release -q -p games-ground --example difficulty + # CB-WP-0022 T05: the design-finding register, reported over # specs/GroundRules.md. Shows the QUEUE by default; the log of closed # findings is a line, not a listing, because a default view that mixes diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 4522972..72be584 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -32,7 +32,7 @@ | workplan | CB-WP-0022 | done | — | workplans/CB-WP-0022-the-design-instrument.md | | workplan | CB-WP-0023 | done | — | workplans/CB-WP-0023-solve-legality.md | | workplan | CB-WP-0024 | done | — | workplans/CB-WP-0024-the-table-you-can-watch.md | -| workplan | CB-WP-0025 | ready | — | workplans/CB-WP-0025-could-we-have-won.md | +| workplan | CB-WP-0025 | active | — | workplans/CB-WP-0025-could-we-have-won.md | | workplan | CB-WP-0026 | done | — | workplans/CB-WP-0026-collect-the-rulings.md | | task | CB-WP-0001-T01 | done | — | workplans/CB-WP-0001-inner-loop.md | | task | CB-WP-0001-T02 | done | — | workplans/CB-WP-0001-inner-loop.md | @@ -162,8 +162,8 @@ | task | CB-WP-0024-T03 | done | — | workplans/CB-WP-0024-the-table-you-can-watch.md | | task | CB-WP-0024-T04 | done | — | workplans/CB-WP-0024-the-table-you-can-watch.md | | task | CB-WP-0024-T05 | done | — | workplans/CB-WP-0024-the-table-you-can-watch.md | -| task | CB-WP-0025-T01 | todo | — | workplans/CB-WP-0025-could-we-have-won.md | -| task | CB-WP-0025-T02 | todo | — | workplans/CB-WP-0025-could-we-have-won.md | +| task | CB-WP-0025-T01 | done | — | workplans/CB-WP-0025-could-we-have-won.md | +| task | CB-WP-0025-T02 | done | — | workplans/CB-WP-0025-could-we-have-won.md | | task | CB-WP-0025-T03 | todo | — | workplans/CB-WP-0025-could-we-have-won.md | | task | CB-WP-0025-T04 | todo | — | workplans/CB-WP-0025-could-we-have-won.md | | task | CB-WP-0025-T05 | todo | — | workplans/CB-WP-0025-could-we-have-won.md | diff --git a/decisions/ADR-0013-could-we-have-won.md b/decisions/ADR-0013-could-we-have-won.md new file mode 100644 index 0000000..730f8ed --- /dev/null +++ b/decisions/ADR-0013-could-we-have-won.md @@ -0,0 +1,230 @@ +# ADR-0013: answer the retrospective question, and measure difficulty with the solver rather than with a bot + +status: accepted +date: 2026-08-05 +decided by: agent, under the standing loop authorization +tier: L (structural L — a new analysis capability whose numbers `ground-game` +will tune against; chaos d8=6 → no override). Tier L: separate survey, +adversarial review, then this document. +references: [CB-WP-0025](../workplans/CB-WP-0025-could-we-have-won.md), +[CB-RES-0008](../research/CB-RES-0008-could-we-have-won.md), +[challenge](../history/260805-could-we-have-won-challenge.md) / +[response](../history/260805-could-we-have-won-response.md), +[ADR-0012](ADR-0012-the-design-instrument.md) (admissibility), +[GameDesign.md](../specs/GameDesign.md), +GROUND-WP-0004 T02, GROUND-WP-0005 (blocked on D4) + +## Context + +The maintainer asked two things: *"we lost — could we have won, and how?"* +and *"do we have difficulty estimations?"* + +**The survey answered the second and was wrong.** It measured +`GreedyPolicy` winning 200/200 at five and six seats and called the game +too easy there. A `FirstLegal` policy — `legal[0]`, no heuristic — scores +**0%** on the same deals. Two unsophisticated agents span the whole range, +so the measurement was about the policy. + +That failure is not incidental to this ADR; **it determines D4.** + +## The premise that changed, and it changes the algorithm + +The survey said exhaustive search was impossible and reached for +determinized sampling, which carries strategy fusion. **Both halves were +wrong.** + +- Its per-node cost was **30–50× too high** (a timer bracketing whole + games). Corrected: ~3–4 µs per `legal_commands` call, with the exact + figure still disputed (§D7). +- Bounded exhaustive search is **affordable**: measured ~3 s over the last + two rounds at three seats. + +Joint branching, treating the table as one co-operative agent — the +product over seats of the measured per-seat branching: + +| seats | per-seat mean | joint per round | last 2 rounds | +|---|---:|---:|---:| +| 2 | 4.7 | ~22 | ~5×10² | +| 3 | 7.4 | ~405 | ~1.6×10⁵ | +| 4 | 9.1 | ~754 | ~5.7×10⁵ | + +Against a ~10⁵–10⁶ node budget, **the last two rounds are exhaustively +searchable at two, three and four seats.** Five rounds is not, at any seat +count. + +--- + +## D1 — answer the *retrospective* question, and say so in those words + +Three questions were on the table (CB-RES-0008 §3). The tool answers: + +> **"Given the deal as it actually was, was there a line of play that +> reached the threshold — and here is one."** + +**Strategy fusion does not apply to this question, and that is the whole +reason it is the affordable one.** Fusion is a defect of *aggregating over +determinizations to choose a move*: the search picks different actions in +states the player cannot distinguish. **After the game there is one +world.** The deck is known, the deal is known, and a search over that +single world produces a line that is executable in it — because it is the +only world there is. + +The survey treated fusion as an obstacle to this pass. It is an obstacle +to a *playing* engine. We are not building one. + +**What remains true is that the line may have been unfindable at the +time**, and D2 handles that by annotation rather than by refusing to +answer. + +**On screen it is called** *"was this deal winnable?"* — never *"how you +should have played"*. The distinction is the honest content of the +feature, and a label that overclaims turns a true answer into a false +lesson. + +## D2 — run on `GroundState`, and mark each move's information dependence + +The survey's preferred guarantee was structural: search a `GroundView` so +the boundary cannot be crossed. **It is not implementable** — a view +cannot `fold` events, so a search needs a state it may not see. The survey +said so in §6 and was right to. + +Decision: **search `GroundState`** — legitimate here, because post-game +the deal is public (`solution_discard` already is, and the game is over) — +and move the honesty guarantee to something checkable: + +> **Every move in an emitted witness is marked `visible` or `hidden`.** +> A move is `visible` if, at the point it is played, everything it depends +> on was in the acting seat's projection: the target Problem face-up, the +> Solution in that seat's own hand. Otherwise `hidden`. + +So a witness reads *"you could have won — but two of these six moves +needed a card you had no way to know was coming."* **That is a more useful +answer than either extreme**, and it is computed from `project()`, which +already exists and is already tested. + +**Falsifier:** if a witness is emitted whose moves are all marked +`visible` but which no seat could actually have chosen, the marking is +wrong and D2 has failed. A test constructs exactly that case. + +## D3 — bounded exhaustive over the endgame, `K` rounds, and honest wording + +**Exhaustive search over the last `K` rounds**, with the table treated as +one co-operative agent choosing joint selections. `K = 2` by default, +which the measurements put inside budget at 2–4 seats. + +- The bound is **rounds**, not nodes or seconds, because rounds are what a + player understands: *"winnable from round 4"* means something; *"winnable + within 100,000 nodes"* does not. +- A node budget is a **secondary** cut that aborts with a stated reason, + so a wide table cannot hang the page. +- **Wording is normative.** When no line is found the tool says + **"no winning line found in the last K rounds"** — never *"unwinnable"*. + A bounded search that claims unwinnability is lying, and this is the + sentence the maintainer will read. + +**Not chosen: determinized sampling (PIMC), ISMCTS.** Both are for playing +under uncertainty. Here there is one world (D1), so they would add strategy +fusion to a problem that does not have it. + +## D4 — difficulty is the **winnable fraction**, not any bot's win rate + +**This is the decision the review forced, and it is the useful half of the +pass.** + +A single-policy win rate cannot be a difficulty: two trivial policies span +0–100% on the same deals. Worse, *improving the bot would make the game +"easier"* without a rule changing — a measure that moves when the +measurer improves is not measuring the thing. + +The solver supplies a policy-independent alternative: + +> **Winnable fraction** — over N deals at a seat count, the proportion in +> which the search finds *any* winning line within its bound. + +That is a property of **the deal distribution and the threshold**, which +is exactly what `ground-game` tunes. It is the number GROUND-WP-0005 is +blocked on, and the bot rate never was. + +Difficulty therefore ships as **a small table, never one number**: + +| column | what it is | +|---|---| +| winnable fraction | can the deal be won at all (bounded, K stated) | +| reference-policy win rate | what a stated bot achieves — **named policy** | +| skill gap | the difference: how much play has to supply | + +**Every rate carries its policy, its N, its seed range and its K in the +number's name**, not in a footnote. A figure that loses them is +inadmissible under GameDesign §1.2. + +**Bounded-below caveat, stated because it will be quoted:** the winnable +fraction from a K-round search is a **lower bound** on true winnability — +a deal unwinnable in the last 2 rounds may have been winnable in round 1. +The report says "winnable-from-round-(6−K)", never "winnable". + +## D5 — the harness becomes an instrument before any figure is quoted + +C3 established that `difficulty-baseline.rs` has no assertions, no +`--self-test` and no `make` target — nothing can turn it red. Under +CB-WP-0022 T05's own `role` distinction it is a `default` artifact wearing +a `counterexample` label, and **GameDesign §1.3 makes it inadmissible.** + +Required before T06 reports anything: + +- **positive controls** — a deal constructed to be unwinnable returns + none; a deal constructed to be winnable returns a witness that replays; +- **`--self-test`**, wired into `make self-tests` like every other + reporting tool; +- **`make difficulty`** (or equivalent), so the figure regenerates from + one command; +- the **policy panel is plural**: at least `greedy`, `random` and + `first-legal`, because the spread between them is what C4 exposed and + hiding it would restore the error. + +## D6 — it lives in `games/ground`, not a new crate + +The search needs `validate`, `fold`, `legal_commands` and `project` — +all of `games_ground`. A separate crate would either re-export the +aggregate or take a dependency on it and add nothing. + +**The tier was declared L on the assumption of a new capability port. +There is no port**, and that over-declaration is recorded rather than +hidden — it is a data point for the tier rules, and the L weight paid for +itself twice over regardless (§Consequences). + +`cb-play` gains a mode to ask the question about a finished game; the +difficulty sweep is an example/binary, as the baseline is. + +## D7 — the per-node cost is unsettled and T04 must benchmark it + +The author measured **3.0–4.1 µs**, the reviewer **15.6–20.4 µs**, by +different isolations. Both agree the published 112–161 µs was wrong by +1–2 orders; neither has established which is right. + +**T04 benchmarks it with `criterion`** — already a dev-dependency, already +used by `benches/synthetic.rs` — and the spec quotes that number and no +other. **Neither figure above may be cited**, including by this ADR. + +## Consequences + +- `specs/` gains the witness contract and the difficulty table's shape + (T04), plus the benchmarked node cost. +- T05 builds the K-round search, the `visible`/`hidden` marking, and the + replay check. +- **T06's payload changes completely.** It reports a winnable fraction and + a policy panel to GROUND-WP-0005 — *not* the withdrawn "too easy at 5–6 + seats". The withdrawal itself is reported, per ADR-0012 D5. +- The register gains the withdrawn finding as `inconsistent` / + `withdrawn`, so it is in the log rather than forgotten. + +## What was rejected + +| rejected | why | +|---|---| +| determinized sampling / PIMC / ISMCTS | strategy fusion, added to a question that has one world (D1) | +| a view-only search as a structural boundary | not implementable — a view cannot fold events | +| refusing to answer unless the line was findable at the time | throws away a true and useful answer; annotate instead (D2) | +| a single bot win rate as "difficulty" | two trivial policies span 0–100% on the same deals (C4) | +| "unwinnable" as output wording | a bounded search cannot know it (D3) | +| a new crate | no port exists; it would re-export the aggregate (D6) | +| quoting either measured node cost | they disagree 5× and neither is established (D7) | diff --git a/evidence/CB-EV-0024-could-we-have-won.md b/evidence/CB-EV-0024-could-we-have-won.md new file mode 100644 index 0000000..94a2b49 --- /dev/null +++ b/evidence/CB-EV-0024-could-we-have-won.md @@ -0,0 +1,161 @@ +# CB-EV-0024 — could we have won + +CB-WP-0025 T07. Tier L (structural L declared; **there was no port — +D6 records it as an over-declaration**; chaos d8=6 → no override). +Declaration 8 of chaos window 2. Closed 2026-08-05. + +**Delivered:** [CB-RES-0008](../research/CB-RES-0008-could-we-have-won.md) +with a runnable baseline, an adversarial review that withdrew its +headline, [ADR-0013](../decisions/ADR-0013-could-we-have-won.md), +[RetrospectiveAnalysis.md](../specs/RetrospectiveAnalysis.md), +`games/ground/src/search.rs`, `benches/search.rs`, +`examples/difficulty.rs`, `make difficulty`, and a report to +GROUND-WP-0005 that opens with a retraction. + +--- + +## 1. Was the maintainer's game winnable? + +The question the pass exists for. **Yes, and the tool answers it.** + +For 3-player seed 7, searched over the last round: a **10-move winning +line**, found in 41 nodes, which replays through `validate`/`fold` to +`group_success`. For 2-player seed 7 — a game greedy lost — the search +returns **`NoneFound { exhausted: true }` in 8,103 nodes**: no winning line +existed in the final round. That is a real negative, not a budget cut, and +the type keeps the two apart. + +**The feature works. The difficulty half is where the trouble was.** + +## 2. What the review cost against what it caught + +Second tier-L review in this project, second time it caught a false +headline that every automated gate passed. `make all` was green +throughout. + +| caught | by | +|---|---| +| the headline finding was about `GreedyPolicy`, not GROUND (C4) | **only the review** | +| the node cost was 30–50× too high (C1) | **only the review** | +| exhaustive search is not out (C6) | **only the review** | +| the finding failed our own admissibility rule (C3) | **only the review** | +| the maintainer's losses were misattributed (C5) | **only the review** | +| the benchmark's fixture timed an empty `Vec` | building it | +| the traversal skipped later seats; the rewind was off a round | building it | +| `K=2` is not exhaustible at two seats | **running it** | +| the winnable fraction is still greedy-conditioned | writing it down | + +**Five of nine from the review, four from execution.** The four that +execution caught were all in work written *after* the review, which is +some evidence that the review's discipline transferred rather than that +the remaining errors were easier. + +**The reviewer under-ranked their own best challenge.** They put C4 +fourth. Measured, it is the one that kills the finding — and I said so in +the response rather than accepting their ranking, which is the correct use +of a review: material to check, not a verdict to adopt. + +## 3. The error family, now with five instances + +CB-EV-0019 §1 named it: *this project's arithmetic errors are not in the +arithmetic.* A number computed correctly against the wrong base, span, or +subject. + +| # | pass | the wrong thing | +|---|---|---| +| 1 | CB-WP-0021 | `csv` cost against AM-4a, which never sees the code | +| 2 | ground-game report | *"12 in the file"* — a sum with no deal table | +| 3 | CB-WP-0018 | SOLVE's inertness attributed to the wrong condition | +| 4 | **this pass, C1** | a timer bracketing whole games, divided by decisions | +| 5 | **this pass, C4** | a win rate attributed to the game, not the policy | + +**And two projections published in place of measurements**, both in this +pass: the survey's `7.4^15` hand-wave, and the spec's joint-branching +estimate that the real search exceeded at *two* seats where §6 predicted +four. + +**`facts-check` catches copies that disagree. Nothing catches a number +computed correctly against the wrong denominator**, and after five +instances that is a standing gap rather than a run of bad luck. Naming it +is what this file can do; closing it is not this pass's. + +## 4. What the difficulty measurement actually shows + +`make difficulty`, 60 seeds: + +| seats | winnable | greedy | random | first-legal | spread | +|---|---:|---:|---:|---:|---:| +| 2 | 60% | 60.0% | 5.0% | 76.7% | 71.7 | +| 3 | 93% | 88.3% | 6.7% | 25.0% | 81.7 | +| 4 | 100% | 93.3% | 6.7% | 30.0% | 86.7 | +| 5 | 100% | 100.0% | 3.3% | 0.0% | 100.0 | +| 6 | 100% | 100.0% | 3.3% | 0.0% | 100.0 | + +**The `spread` column is the finding.** 71.7 to 100.0 points between three +trivial policies. It is the evidence for the prohibition in +RetrospectiveAnalysis §4.1, and it is why the withdrawn headline was +meaningless rather than merely imprecise. + +**The 5–6 seat rows point the opposite way from the withdrawn claim.** +`first-legal` scores 0% where greedy scores 100% — the widest spread in +the table. Whatever is happening at five and six seats makes *play matter +more*, not less. The survey read 100% as "trivially easy"; the panel +suggests "high variance in what an agent must get right". **Neither is +established**, and saying so is the honest state. + +**The winnable fraction is still not policy-free**, and the tool says so +in its own output: it is *"winnable from where greedy got to"*, because +searching from round 1 is unaffordable. Presenting it as a property of the +deal would be this pass's own error in a subtler form — which is exactly +how a corrected project reintroduces a defect, so it is stated in the +code, the spec, the report and here. + +## 5. Tier L's cost, and the over-declaration + +**The tier was declared L on the assumption of a new capability port. +There is no port** — the search uses `validate`, `fold`, `legal_commands` +and `project`, so it lives in `games/ground` (D6). By the structural +trigger this was an **M**. + +**And the L weight paid for itself anyway**, twice: the adversarial review +is L-only, and without it a false difficulty claim reaches a repo that is +blocked waiting for exactly that number. **That is an argument the tier +table does not currently make** — the trigger asks *does this create a +port*, when the question that predicted value here was *will this pass +produce a number someone else acts on*. Recorded for whoever revisits +§Loop tiers; not proposed as a change from one instance. + +## 6. Chaos window 2 + +**Declaration 8 of 12.** Structural L (over-declared), d8 = 6, no +override. + +**Eleven declarations, no 8 rolled.** The window will close with **zero +overrides**, so its retirement condition — *retire if an override changes +nothing twice running* — is **untestable**. Window 1 closed with two +overrides at d4 and both changed the outcome. + +**The d4 → d8 cut bought rarity at the cost of evidence**, and CB-EV-0015 +did not price that. At d8 over twelve declarations the expected count is +1.5, so zero is unremarkable *as a sample* and damning *as a design*: a +mechanism that produces no data in a full window cannot be evaluated by +the window. **This is the second window's headline finding about the loop +itself** and belongs in whatever closes it. + +## 7. Cost + +CB-WP-0024's cost, by re-running the instrument: `make cost`. Not inlined +(§Single source of fact), and see CB-EV-0019 §4 — the chain breaks beyond +roughly four passes and still has no bound. + +## Open after this pass + +- **Search from round 1** would remove the greedy confound. Needs + transposition or move-ordering; neither built. +- **Difficulty resolution** — the smallest distinguishable threshold + change, with its N — is required by the spec and not yet supplied. +- **The witness is not wired to the ending page.** T05 built the search; + the browser cannot yet ask it. That is the maintainer-facing half of + remark 2 and it is not done. +- **The wrong-denominator family has no control** (§3), at five instances. +- **The chaos window's retirement condition is untestable** (§6). diff --git a/games/ground/Cargo.toml b/games/ground/Cargo.toml index 9d0768b..1f8e627 100644 --- a/games/ground/Cargo.toml +++ b/games/ground/Cargo.toml @@ -27,5 +27,10 @@ serde_json.workspace = true name = "synthetic" harness = false +# CB-WP-0025 T04 / ADR-0013 D7: the node cost the spec quotes. +[[bench]] +name = "search" +harness = false + [lints] workspace = true diff --git a/games/ground/benches/search.rs b/games/ground/benches/search.rs new file mode 100644 index 0000000..75c7fd5 --- /dev/null +++ b/games/ground/benches/search.rs @@ -0,0 +1,128 @@ +//! CB-WP-0025 T04 — what one search node actually costs. +//! +//! **ADR-0013 D7 exists because two measurements disagreed by 5×.** The +//! survey published 112–161 µs/node from a timer that bracketed two +//! `setup`s and a whole greedy game (C1). The author's re-measurement said +//! 3.0–4.1 µs with `Instant::now()` around each call; the adversarial +//! reviewer's isolation said 15.6–20.4 µs. Both agreed the published +//! figure was wrong by 1–2 orders and neither established which +//! replacement was right. +//! +//! So the spec quotes **this** and nothing else. `criterion` handles the +//! things hand-rolled timing gets wrong here: per-call clock overhead +//! against a ~microsecond subject, warm-up, and run-to-run variance — +//! which is what let the survey's figure move 161 → 112 between two runs +//! of the same unmodified binary. +//! +//! Two subjects, because a search node is not one call: +//! +//! * `legal_commands` — enumerating a seat's options; +//! * `validate + fold` — taking one branch, which any search does per +//! child and which the survey never separated out. + +use cb_game_runtime::{ScenarioGame, Setup}; +use cb_kernel::{Actor, Aggregate, PlayerId}; +use criterion::{criterion_group, criterion_main, BatchSize, Criterion}; +use games_ground::bot::{legal_commands, play, GreedyPolicy, Policy}; +use games_ground::GroundState; +use std::collections::BTreeMap; + +fn setup(players: u8, seed: u64) -> GroundState { + GroundState::setup( + &Setup { + players, + preset: format!("standard-{players}p"), + patch: BTreeMap::new(), + }, + seed, + ) + .expect("preset") +} + +/// A **mid-game state at a real decision point** for `seat`. +/// +/// Not a fresh deal: at deal time most branches do not exist yet, and a +/// node cost taken there would flatter any search proposal. +/// +/// **And not a fixed step count either.** The first version stopped at +/// step 20 for every seat count, which put 2p and 4p in a state where +/// seat 0 had *no* legal commands at all — so the benchmark reported +/// ~120 ns (the cost of returning an empty `Vec`) and silently skipped +/// `validate_fold` because there was nothing to validate. A fixture that +/// measures the empty case and calls it a node cost is the same defect +/// class this whole pass exists to correct, one layer down. +/// +/// So: advance until the seat genuinely has a choice, and assert it. +fn midgame(players: u8, seed: u64, seat: PlayerId) -> GroundState { + let mut ps: Vec> = (0..players) + .map(|_| Box::new(GreedyPolicy) as Box) + .collect(); + let game = play(setup(players, seed), &mut ps).expect("a complete game"); + let mut state = setup(players, seed); + let mut best: Option = None; + for (i, (actor, cmd)) in game.steps.iter().enumerate() { + // Past the opening, take the first state where the seat has a real + // branch. `> 1` rather than `> 0`: a forced move is not a node. + if i >= 8 && legal_commands(&state, seat).len() > 1 { + best = Some(state.clone()); + break; + } + if let Ok(events) = state.validate(*actor, cmd) { + for e in &events { + state.fold(e); + } + } + } + let state = best.expect("a mid-game state where the seat has a choice"); + assert!( + legal_commands(&state, seat).len() > 1, + "benchmark fixture has no branch to measure — it would time the empty case" + ); + state +} + +fn bench(c: &mut Criterion) { + for players in [2u8, 3, 4] { + let seat = PlayerId(0); + let state = midgame(players, 7, seat); + let width = legal_commands(&state, seat).len(); + println!(" fixture {players}p: {width} legal commands at the measured node"); + + c.bench_function(&format!("legal_commands/{players}p"), |b| { + b.iter(|| std::hint::black_box(legal_commands(&state, seat))) + }); + + // A search must COPY the state per branch (or undo, which we do + // not have). `iter_batched` excludes setup from the timing, so + // without this the budget would rest on an unmeasured span — + // which is the exact mistake C1 caught in the survey. + c.bench_function(&format!("clone/{players}p"), |b| { + b.iter(|| std::hint::black_box(state.clone())) + }); + + // One branch taken: what a search pays per CHILD, on top of + // enumeration. The survey folded this into "us/node" without + // separating it, and a search's real cost is enumeration once plus + // this per child. + let legal = legal_commands(&state, seat); + if let Some(cmd) = legal.first() { + c.bench_function(&format!("validate_fold/{players}p"), |b| { + b.iter_batched( + || state.clone(), + |mut s| { + if let Ok(events) = s.validate(Actor::Player(seat), cmd) { + for e in &events { + s.fold(e); + } + } + std::hint::black_box(s) + }, + BatchSize::SmallInput, + ) + }); + } + } +} + +criterion_group!(benches, bench); +criterion_main!(benches); diff --git a/games/ground/examples/difficulty-baseline.rs b/games/ground/examples/difficulty-baseline.rs index 51140ac..1737dff 100644 --- a/games/ground/examples/difficulty-baseline.rs +++ b/games/ground/examples/difficulty-baseline.rs @@ -1,5 +1,21 @@ //! CB-RES-0008's runnable baseline (CB-WP-0025 T01). //! +//! ── SUPERSEDED 2026-08-05 by `difficulty.rs` (CB-WP-0025 T06) ────────── +//! +//! Kept as the survey's dated snapshot, NOT as a live instrument, for the +//! same reason `tools/design-baseline.py` was kept: it is the evidence for +//! how a wrong number got into a survey. +//! +//! **Its headline was withdrawn.** It reported `GreedyPolicy` winning +//! 200/200 at five and six seats and the survey called the game too easy +//! there. A `FirstLegal` policy scores **0%** on the same deals — the +//! measurement was about the policy. See the challenge (C4) and the +//! response. +//! +//! **And it is inadmissible under GameDesign §1.3**: no assertions, no +//! `--self-test`, no `make` target, so nothing can turn it red. +//! `difficulty.rs` carries all three. +//! //! Two numbers the survey needs and cannot cite from anyone else, because //! they are about **our** game on **our** machine: //! diff --git a/games/ground/examples/difficulty.rs b/games/ground/examples/difficulty.rs new file mode 100644 index 0000000..4992767 --- /dev/null +++ b/games/ground/examples/difficulty.rs @@ -0,0 +1,304 @@ +//! CB-WP-0025 T06 — the difficulty table, and the instrument that can fail. +//! +//! Implements `specs/RetrospectiveAnalysis.md` §4. Replaces +//! `difficulty-baseline.rs`, which CB-WP-0022's admissibility rule and +//! this pass's own review both found inadmissible: it had no assertions, +//! no `--self-test` and no `make` target, so nothing could turn it red. +//! +//! ## What it will not print +//! +//! **A single policy's win rate as a difficulty** (§4.1). Measured on +//! identical deals, `GreedyPolicy` wins 100% at five and six seats where +//! `FirstLegal` wins 0%. The panel is plural for that reason, and the +//! spread is reported rather than hidden. +//! +//! ```text +//! cargo run --release -p games-ground --example difficulty [--self-test] +//! ``` + +use cb_game_runtime::{ScenarioGame, Setup}; +use cb_kernel::{Aggregate, PlayerId}; +use games_ground::bot::{play, Choice, GreedyPolicy, Policy, RandomPolicy}; +use games_ground::search::{winnable_within, Verdict}; +use games_ground::{GroundCommand, GroundState}; + +/// Seeds per cell. Small by default: the winnable fraction needs an +/// exhaustive search per deal, which is the expensive half (§3.1). +const SEEDS: u64 = 60; +/// §3's measured limit: `K=1` is exhaustible, `K=2` is not. +const K: u8 = 1; +const BUDGET: usize = 2_000_000; + +/// A policy with no heuristic at all. **It is in the panel because it is +/// what falsified the survey's headline** — it beats greedy at two seats +/// and scores zero at six. +struct FirstLegal; +impl Policy for FirstLegal { + fn name(&self) -> &'static str { + "first-legal" + } + fn choose( + &mut self, + _s: &GroundState, + _seat: PlayerId, + _legal: &[GroundCommand], + _may_pass: bool, + ) -> Choice { + Choice::Command(0) + } +} + +fn setup(players: u8, seed: u64) -> Option { + GroundState::setup( + &Setup { + players, + preset: format!("standard-{players}p"), + patch: Default::default(), + }, + seed, + ) + .ok() +} + +fn policies(kind: &str, players: u8, seed: u64) -> Vec> { + (0..players) + .map(|i| -> Box { + match kind { + "random" => Box::new(RandomPolicy::new(seed ^ u64::from(i))), + "first-legal" => Box::new(FirstLegal), + _ => Box::new(GreedyPolicy), + } + }) + .collect() +} + +/// Games a named policy actually wins. +fn policy_wins(kind: &str, players: u8) -> (u32, u32) { + let (mut wins, mut played) = (0, 0); + for seed in 0..SEEDS { + let Some(state) = setup(players, seed) else { + continue; + }; + let mut ps = policies(kind, players, seed); + let Ok(game) = play(state, &mut ps) else { + continue; + }; + if let Some(o) = &game.state.outcome { + played += 1; + if o.group_success { + wins += 1; + } + } + } + (wins, played) +} + +/// Rewind a played game to the start of its last `k` rounds. +fn last_rounds(players: u8, seed: u64, k: usize) -> Option { + let mut ps = policies("greedy", players, seed); + let game = play(setup(players, seed)?, &mut ps).ok()?; + let total = game + .steps + .iter() + .filter(|(_, c)| matches!(c, GroundCommand::EndRound)) + .count(); + let mut st = setup(players, seed)?; + let mut ends = 0usize; + for (a, c) in &game.steps { + if let Ok(ev) = st.validate(*a, c) { + for e in &ev { + st.fold(e); + } + } + if matches!(c, GroundCommand::EndRound) { + ends += 1; + if ends >= total.saturating_sub(k) { + break; + } + } + } + Some(st) +} + +/// In what fraction of deals does a winning line exist in the last `K` +/// rounds? +/// +/// **Less policy-dependent than a win rate, but NOT policy-free, and +/// saying otherwise would repeat this pass's own error in a subtler +/// form.** The position searched is the one `GreedyPolicy` left at the +/// start of the last round, so the figure is *"winnable from where greedy +/// got to"*. A genuinely policy-free measure would search from round 1, +/// which §3.1 measured as unaffordable. +/// +/// What it does buy: the last round's outcome no longer depends on which +/// agent plays it, so the measure is insensitive to exactly the variation +/// that made the bot rate meaningless (§4.1). +/// +/// Deals where the search was cut by its budget are **not counted either +/// way** — they are reported separately, because folding "we stopped +/// looking" into "not winnable" is exactly the collapse §2.3 forbids. +fn winnable_fraction(players: u8) -> (u32, u32, u32) { + let (mut yes, mut decided, mut undecided) = (0, 0, 0); + for seed in 0..SEEDS { + let Some(state) = last_rounds(players, seed, K as usize) else { + continue; + }; + match winnable_within(&state, K, BUDGET) { + Verdict::Winnable { .. } => { + yes += 1; + decided += 1; + } + Verdict::NoneFound { + exhausted: true, .. + } => decided += 1, + Verdict::NoneFound { + exhausted: false, .. + } => undecided += 1, + } + } + (yes, decided, undecided) +} + +fn report() { + println!("difficulty — specs/RetrospectiveAnalysis.md §4\n"); + println!( + " winnable-from-round-{} over {SEEDS} seeds, budget {BUDGET} nodes", + 6 - K + ); + println!(" policy win rates over the same {SEEDS} seeds\n"); + println!(" seats winnable greedy random first-legal spread undecided"); + + for players in [2u8, 3, 4, 5, 6] { + let (yes, decided, undecided) = winnable_fraction(players); + let pct = |(w, n): (u32, u32)| { + if n == 0 { + -1.0 + } else { + 100.0 * f64::from(w) / f64::from(n) + } + }; + let g = pct(policy_wins("greedy", players)); + let r = pct(policy_wins("random", players)); + let f = pct(policy_wins("first-legal", players)); + let spread = [g, r, f].iter().cloned().fold(f64::MIN, f64::max) + - [g, r, f].iter().cloned().fold(f64::MAX, f64::min); + let wf = if decided == 0 { + "n/a".to_string() + } else { + format!("{:.0}%", 100.0 * f64::from(yes) / f64::from(decided)) + }; + println!( + " {players}p {wf:>6} {g:>5.1}% {r:>5.1}% {f:>5.1}% \ + {spread:>5.1} {undecided:>3}" + ); + } + + println!( + "\n WINNABLE is conditioned on GREEDY's play up to the last round —\n \ + it is 'winnable from where greedy got to', not a property of the\n \ + deal alone. Searching from round 1 is unaffordable (spec §3.1).\n\n \ + It is also a LOWER BOUND: a K={K} search cannot see a line that\n \ + needed an earlier round. `undecided` are deals whose search hit the\n \ + node budget — they are excluded from the fraction, not counted as\n \ + unwinnable.\n\n \ + SPREAD is the range across three policies. Where it is large, no\n \ + single policy's rate says anything about the game (§4.1)." + ); +} + +fn self_test() -> i32 { + let mut ok = true; + let mut check = |name: &str, cond: bool, detail: String| { + ok &= cond; + println!( + " [{}] {name}{}", + if cond { "ok " } else { "FAIL" }, + if detail.is_empty() { + String::new() + } else { + format!(" — {detail}") + } + ); + }; + + // The control the old harness never had: a search that finds a line + // must produce one that REPLAYS. §2.1 is a gate, not a metric. + let state = last_rounds(3, 7, K as usize).expect("a 3p game"); + match winnable_within(&state, K, BUDGET) { + Verdict::Winnable { line, .. } => { + let mut replay = state.clone(); + let mut good = true; + for m in &line { + match replay.validate(m.actor, &m.command) { + Ok(ev) => { + for e in &ev { + replay.fold(e); + } + } + Err(_) => good = false, + } + } + let won = replay.outcome.as_ref().is_some_and(|o| o.group_success); + check( + "a witness replays to a win", + good && won, + format!("{} moves", line.len()), + ); + } + Verdict::NoneFound { .. } => check( + "a witness replays to a win", + false, + "3p seed 7 found no line — the fixture moved".into(), + ), + } + + // The negative: a search must be able to return nothing, exhaustively. + let lost = last_rounds(2, 7, 1).expect("a 2p game"); + match winnable_within(&lost, 1, BUDGET) { + Verdict::NoneFound { exhausted, nodes } => check( + "an unwinnable position is reported as searched-out", + exhausted && nodes > 100, + format!("{nodes} nodes"), + ), + Verdict::Winnable { .. } => check( + "an unwinnable position is reported as searched-out", + false, + "found a win in a game 2p seed 7 lost".into(), + ), + } + + // A budget of one must NOT claim exhaustion — the distinction §2.3 + // rests on. + match winnable_within(&state, K, 1) { + Verdict::NoneFound { exhausted, .. } => check( + "a budget cut is not reported as exhaustion", + !exhausted, + String::new(), + ), + Verdict::Winnable { .. } => check( + "a budget cut is not reported as exhaustion", + false, + String::new(), + ), + } + + // §4.1's reason, asserted rather than asserted-about: the panel must + // actually disagree, or reporting three policies is ceremony. + let g = policy_wins("greedy", 6); + let f = policy_wins("first-legal", 6); + check( + "the policy panel disagrees, so no single rate is a difficulty", + g.0 != f.0, + format!("greedy {}/{}, first-legal {}/{}", g.0, g.1, f.0, f.1), + ); + + println!("difficulty self-test (positive control)"); + i32::from(!ok) +} + +fn main() { + if std::env::args().any(|a| a == "--self-test") { + std::process::exit(self_test()); + } + report(); +} diff --git a/games/ground/src/lib.rs b/games/ground/src/lib.rs index c0e67b5..5f01712 100644 --- a/games/ground/src/lib.rs +++ b/games/ground/src/lib.rs @@ -20,6 +20,12 @@ pub mod view; #[cfg(feature = "scenarios")] pub mod record; +/// *Was this deal winnable?* — the retrospective search (CB-WP-0025 T05, +/// ADR-0013). Uses only `validate`/`fold`/`legal_commands`, so it lives +/// beside the aggregate rather than in a crate that would re-export it +/// (ADR-0013 D6). +pub mod search; + #[cfg(feature = "scenarios")] use cb_game_runtime::{parse_actor, CommandStep, ScenarioGame, Setup}; use cb_kernel::{Actor, Aggregate, ChaChaRng, KernelRng, PlayerId, Rejection, Seed}; diff --git a/games/ground/src/search.rs b/games/ground/src/search.rs new file mode 100644 index 0000000..aa4a25f --- /dev/null +++ b/games/ground/src/search.rs @@ -0,0 +1,412 @@ +//! Was this deal winnable — and here is one line (CB-WP-0025 T05). +//! +//! Implements [`specs/RetrospectiveAnalysis.md`]. The question is +//! **retrospective**: given the deal as it actually was, does a line of +//! play exist that reaches the threshold? +//! +//! ## Why this is allowed to see everything +//! +//! Strategy fusion — the classic objection to searching an +//! imperfect-information game — is a defect of *aggregating over +//! determinizations to choose a move*. **After the game there is one +//! world.** The deal is known, so a line found in it is executable in the +//! only world there is (ADR-0013 D1). +//! +//! What survives the objection is that the line may not have been +//! *findable* at the time, and that is answered per move by +//! [`Move::visible`] rather than by refusing to search. +//! +//! ## The bound +//! +//! Exhaustive over the last `K` rounds, with a node budget as a secondary +//! cut. When nothing is found the caller must say **"no winning line +//! found in the last K rounds"** — never "unwinnable", which a bounded +//! search cannot establish (spec §2.3). + +use crate::bot::legal_commands; +use crate::{GroundCommand, GroundState, ProblemState}; +use cb_kernel::{Actor, Aggregate, PlayerId}; + +/// One move of a witness, with whether the seat could have chosen it +/// knowing only what it could see. +#[derive(Debug, Clone)] +pub struct Move { + pub actor: Actor, + pub command: GroundCommand, + /// `false` when the move depends on something the acting seat could + /// not see — spec §2.2. Concretely: it targets a Problem that was + /// **face down** to that seat, so choosing it required knowing what + /// was under it. + /// + /// System moves are always `true`: the table does them, not a player. + pub visible: bool, +} + +/// What the search found. +#[derive(Debug, Clone)] +pub enum Verdict { + /// A line exists. `nodes` is what it cost to find. + Winnable { line: Vec, nodes: usize }, + /// Nothing found **within the bound**. This is not "unwinnable". + NoneFound { + nodes: usize, + /// `true` if the space was searched to exhaustion; `false` if the + /// node budget cut it short. The distinction is the difference + /// between "no line exists in these K rounds" and "we stopped + /// looking", and callers must not collapse it. + exhausted: bool, + }, +} + +/// Would this command have been choosable knowing only what `seat` saw? +/// +/// A `SelectAction` naming a Problem that is face-down to that seat is +/// `hidden`: picking it required knowing what was underneath. Everything +/// else is `visible` — a seat's own hand is in its own projection, and +/// since GR-P05 (CB-WP-0023) SOLVE is only offered on face-up Problems +/// anyway, so INVESTIGATE is where hidden information actually bites. +fn is_visible(state: &GroundState, seat: PlayerId, cmd: &GroundCommand) -> bool { + let _ = seat; + match cmd { + GroundCommand::SelectAction { + problem: Some(p), .. + } => matches!( + state.problems.get(p), + Some(ProblemState { face_up: true, .. }) + ), + _ => true, + } +} + +struct Search { + nodes: usize, + budget: usize, + /// Set when the budget stopped us, so `NoneFound` can distinguish + /// "searched it all" from "gave up". + cut: bool, +} + +impl Search { + /// Apply a command to a copy. `None` if the aggregate rejects it — + /// which is not an error here: the search offers candidates and + /// `validate` is the authority, exactly as `legal_commands` does. + fn step( + &mut self, + state: &GroundState, + actor: Actor, + cmd: &GroundCommand, + ) -> Option { + self.nodes += 1; + let mut next = state.clone(); + let events = next.validate(actor, cmd).ok()?; + for e in &events { + next.fold(e); + } + Some(next) + } + + /// One player branch: apply, recurse, and prepend the move if the + /// subtree won. + fn branch( + &mut self, + state: &GroundState, + seat: PlayerId, + cmd: &GroundCommand, + rounds_left: u8, + ) -> Option> { + let next = self.step(state, Actor::Player(seat), cmd)?; + let mut rest = self.go(&next, rounds_left)?; + let mut line = vec![Move { + actor: Actor::Player(seat), + command: cmd.clone(), + visible: is_visible(state, seat, cmd), + }]; + line.append(&mut rest); + Some(line) + } + + /// Depth-first over whatever must happen next, mirroring the driver's + /// round structure (`bot::play_journaled`). + /// + /// Returns the moves appended after `state`, or `None`. + fn go(&mut self, state: &GroundState, rounds_left: u8) -> Option> { + if let Some(outcome) = &state.outcome { + return outcome.group_success.then(Vec::new); + } + if rounds_left == 0 { + return None; + } + if self.nodes >= self.budget { + self.cut = true; + return None; + } + + let seats: Vec = state.players.keys().copied().collect(); + + // **Obligatory first.** GR-R02: a seat with no selection this + // round must make one, and nothing else can happen until it does. + // If every branch fails, the line is dead — falling through would + // try system commands the aggregate is going to reject anyway. + // + // The first version branched on "the first seat that has any legal + // command" and `break`ed when its branches were spent, which threw + // away every later seat's options: seat 1 never acted if seat 0 + // was already selected but still had a legal move. + if let Some(seat) = seats.iter().find(|s| !state.selections.contains_key(s)) { + for cmd in &legal_commands(state, *seat) { + if let Some(line) = self.branch(state, *seat, cmd, rounds_left) { + return Some(line); + } + if self.cut { + return None; + } + } + return None; + } + + // **Optional next.** After Reveal a seat may choose a GROUND mode, + // answer a Support, or name a DARVO target. Some of those are + // obligatory, but the aggregate enforces that by rejecting + // `Resolve` until they are done — so this needs no phase logic of + // its own, and the do-nothing case is simply the fall-through + // below. + for seat in &seats { + for cmd in &legal_commands(state, *seat) { + if let Some(line) = self.branch(state, *seat, cmd, rounds_left) { + return Some(line); + } + if self.cut { + return None; + } + } + } + + // Nobody need act: the table advances. Try each system command; the + // aggregate rejects the ones that are out of order, so this needs + // no phase logic of its own. + for sys in [ + GroundCommand::Reveal, + GroundCommand::Resolve, + GroundCommand::EndRound, + ] { + let Some(next) = self.step(state, Actor::System, &sys) else { + continue; + }; + let spent = u8::from(matches!(sys, GroundCommand::EndRound)); + if let Some(mut rest) = self.go(&next, rounds_left - spent) { + let mut line = vec![Move { + actor: Actor::System, + command: sys, + visible: true, + }]; + line.append(&mut rest); + return Some(line); + } + if self.cut { + return None; + } + } + None + } +} + +/// Search the last `rounds` rounds from `state` for a line reaching +/// `group_success`. +/// +/// **`state` must be a real position from the game being asked about.** +/// The caller supplies it; this does not re-deal, because a re-dealt game +/// is a different question. +pub fn winnable_within(state: &GroundState, rounds: u8, budget: usize) -> Verdict { + let mut s = Search { + nodes: 0, + budget, + cut: false, + }; + match s.go(state, rounds) { + Some(line) => Verdict::Winnable { + line, + nodes: s.nodes, + }, + None => Verdict::NoneFound { + nodes: s.nodes, + exhausted: !s.cut, + }, + } +} + +/// How many moves of a witness required unseen information. +pub fn hidden_moves(line: &[Move]) -> usize { + line.iter().filter(|m| !m.visible).count() +} + +#[cfg(all(test, feature = "scenarios"))] +mod tests { + use super::*; + use cb_game_runtime::{ScenarioGame, Setup}; + + fn setup(players: u8, seed: u64) -> GroundState { + GroundState::setup( + &Setup { + players, + preset: format!("standard-{players}p"), + patch: Default::default(), + }, + seed, + ) + .expect("preset") + } + + /// **The hard gate (spec §2.1): a witness must replay.** + /// + /// Re-execute the emitted line from the same start state through + /// `validate`/`fold` — the same path the scenario runner takes — and + /// require it to end in `group_success`. A witness that does not + /// replay asserts the opposite of the truth to a player who just + /// lost. + /// Rewind a real game to the start of its last `k` rounds. + /// + /// Stops **after** applying the EndRound numbered `total - k`. An + /// earlier version broke *before* it, which left that round's own play + /// applied and searched one round less than it claimed. + fn last_rounds(players: u8, seed: u64, k: usize) -> GroundState { + let mut ps: Vec> = (0..players) + .map(|_| Box::new(crate::bot::GreedyPolicy) as Box) + .collect(); + let game = crate::bot::play(setup(players, seed), &mut ps).expect("a complete game"); + let total = game + .steps + .iter() + .filter(|(_, c)| matches!(c, GroundCommand::EndRound)) + .count(); + let mut st = setup(players, seed); + let mut ends = 0usize; + for (a, c) in &game.steps { + if let Ok(ev) = st.validate(*a, c) { + for e in &ev { + st.fold(e); + } + } + if matches!(c, GroundCommand::EndRound) { + ends += 1; + if ends >= total.saturating_sub(k) { + break; + } + } + } + st + } + + #[test] + fn every_witness_replays_to_a_win() { + let state = last_rounds(3, 7, 2); + let Verdict::Winnable { line, .. } = winnable_within(&state, 2, 200_000) else { + panic!("3p seed 7 is winnable in its last two rounds — greedy actually won it"); + }; + let mut replay = state.clone(); + for m in &line { + let events = replay + .validate(m.actor, &m.command) + .unwrap_or_else(|e| panic!("witness move rejected on replay: {e:?}")); + for e in &events { + replay.fold(e); + } + } + let outcome = replay.outcome.as_ref().expect("the replay must finish"); + assert!( + outcome.group_success, + "the witness replayed but did not win: {} of {}", + outcome.total, outcome.threshold + ); + } + + /// The negative control. Without it, a search that returns + /// `NoneFound` for everything would pass the test above by never + /// producing a witness to check. + #[test] + fn a_budget_of_nothing_reports_a_cut_not_a_verdict() { + let state = last_rounds(3, 7, 2); + match winnable_within(&state, 2, 1) { + Verdict::NoneFound { exhausted, .. } => assert!( + !exhausted, + "a search stopped by its budget must not claim it searched exhaustively — \ + that is the difference between `no line exists` and `we stopped looking`" + ), + Verdict::Winnable { .. } => panic!("one node cannot find a whole line"), + } + } + + /// A position with no rounds left cannot be won, and the search must + /// say so **without** claiming exhaustion of a space it never entered. + #[test] + fn no_rounds_left_finds_nothing() { + let state = last_rounds(3, 7, 2); + match winnable_within(&state, 0, 100) { + Verdict::NoneFound { nodes, exhausted } => { + assert_eq!(nodes, 0, "a zero-round search must not expand anything"); + assert!(exhausted, "it searched its (empty) space to exhaustion"); + } + Verdict::Winnable { .. } => panic!("no rounds left cannot win"), + } + } + + /// **A position that cannot be won returns none, exhaustively** — the + /// control without which "winnable" is unfalsifiable. + /// + /// The construction: 2p seed 7, searched over its **last round only**. + /// Greedy lost that game, and one round is a small enough space to + /// search to exhaustion (~8k nodes), so this is a real negative rather + /// than a budget cut wearing a verdict's clothes. + #[test] + fn a_position_that_cannot_be_won_says_so_and_means_it() { + let state = last_rounds(2, 7, 1); + match winnable_within(&state, 1, 500_000) { + Verdict::NoneFound { exhausted, nodes } => { + assert!( + exhausted, + "the space must be searched out, or this proves nothing ({nodes} nodes)" + ); + assert!( + nodes > 100, + "suspiciously few nodes for a real search: {nodes}" + ); + } + Verdict::Winnable { line, .. } => { + panic!("found a {}-move win in a game 2p seed 7 lost", line.len()) + } + } + } + + /// The `visible` marking must be able to say NO, or it is decoration. + /// INVESTIGATE names a face-down Problem — that is the case where a + /// seat could not have known what it was choosing. + #[test] + fn a_move_onto_a_face_down_problem_is_marked_hidden() { + let state = setup(3, 7); + let hidden_key = state + .problems + .iter() + .find(|(_, p)| !p.face_up) + .map(|(k, _)| *k) + .expect("a fresh deal has face-down Problems"); + let face_up_key = state + .problems + .iter() + .find(|(_, p)| p.face_up) + .map(|(k, _)| *k) + .expect("a fresh deal has the Surface Problem face up"); + + let onto = |p: u32| GroundCommand::SelectAction { + action: crate::Action::Investigate, + target: None, + problem: Some(p), + }; + assert!( + !is_visible(&state, PlayerId(0), &onto(hidden_key)), + "targeting a face-down Problem required knowing what was under it" + ); + assert!( + is_visible(&state, PlayerId(0), &onto(face_up_key)), + "a face-up Problem is visible — the marking must be able to say YES too" + ); + } +} diff --git a/specs/FindingRegister.md b/specs/FindingRegister.md index 42be87e..1490bbf 100644 --- a/specs/FindingRegister.md +++ b/specs/FindingRegister.md @@ -43,6 +43,7 @@ kinds, states and metrics: [`GameDesign.md`](GameDesign.md). Reported by | F13 | inconsistent | withdrawn | scenarios/ground/gr-e01-threshold-reachable-2p.yaml | counterexample | 2026-08-01 | clay-borg | | F14 | unplayed | note | — | — | 2026-08-01 | clay-borg | | F15 | underdetermined | note | — | — | 2026-08-05 | clay-borg | +| F16 | inconsistent | withdrawn | games/ground/examples/difficulty.rs | counterexample | 2026-08-05 | clay-borg | @@ -62,6 +63,16 @@ kinds, states and metrics: [`GameDesign.md`](GameDesign.md). Reported by `-reachable-`. **Its reproduction is green**, which under GameDesign §1.3 is the alarm that forced the resolution. Withdrawn rather than deleted, and the withdrawal is reported (ADR-0012 D5). +- **F16 — "the game is too easy at 5–6 seats", withdrawn the day it was + raised.** Claimed from `GreedyPolicy` winning 200/200 at those seat + counts. A `FirstLegal` policy scores **0%** on the identical deals, and + at two seats it *beats* greedy — two unsophisticated agents span the + whole range, so the measurement was about the policy. Caught by the + CB-WP-0025 adversarial review (C4) **before transmission**; it would + have been the fifth wrong premise sent to ground-game and the worst, + since GROUND-WP-0005 is blocked on exactly this number. The withdrawal + was reported (ADR-0012 D5). Its reproduction is `difficulty.rs`, whose + policy panel is plural *because of this finding*. - **F15 — the rules define one game, not a series.** `OutcomeView` gives `personal` (per seat), `group_success` (per table) and `winners`. Summing the first and counting the third answer different questions, and GROUND diff --git a/specs/RetrospectiveAnalysis.md b/specs/RetrospectiveAnalysis.md new file mode 100644 index 0000000..fbb11b1 --- /dev/null +++ b/specs/RetrospectiveAnalysis.md @@ -0,0 +1,235 @@ +# RetrospectiveAnalysis — was this deal winnable, and how hard is the game + +v1.0 — CB-WP-0025 T04, 2026-08-05. Normative. Implements +[ADR-0013](../decisions/ADR-0013-could-we-have-won.md). Admissibility of +anything this produces is governed by +[GameDesign.md](GameDesign.md) §1. + +**Two capabilities, one machine.** *Was this deal winnable?* is a search +over a finished game. *How hard is the game?* is that search run over many +deals and counted — **not** a bot's win rate (§4.1). + +--- + +## 1. The question, and its name + +> **"Given the deal as it actually was, was there a line of play that +> reached the threshold?"** + +**Never labelled "how you should have played."** The distinction is the +honest content of the feature: the tool answers a question about the +*deal*, and a label promising advice about the *player* turns a true +answer into a false lesson. + +**Strategy fusion does not apply and must not be invoked as an objection.** +Fusion (Frank, Basin & Matsubara 1998) is a defect of aggregating over +determinizations to choose a move. After the game there is **one world** — +the deal is known — so a line found in it is executable in the only world +there is. This is why the affordable option is also the honest one. + +## 2. The witness + +A witness is a sequence of joint selections that, replayed from the +recorded initial state, ends with `group_success == true`. + +### 2.1 It must replay — hard gate, not a metric + +> **100% of emitted witnesses replay through the existing scenario runner +> and end in `group_success`.** + +Not a target: a **gate**. A witness that does not replay asserts the +opposite of the truth to a player who just lost, which is worse than +emitting nothing. + +### 2.2 Every move carries its information dependence + +ADR-0013 D2. Each move in a witness is marked: + +| mark | meaning | +|---|---| +| `visible` | everything the move depended on was in the acting seat's projection at that point — the Problem face-up, the Solution in that seat's own hand | +| `hidden` | it was not | + +Computed from `project()`, which already exists and whose hiding rules are +already asserted by `games_ground::view`. + +**This replaces the structural boundary the survey wanted.** Searching a +`GroundView` is not implementable — a view cannot `fold` events — so the +guarantee moved from *the search cannot see it* to *the answer says which +moves needed it*. A witness reads: + +> *"This deal was winnable. Two of these six moves needed a card you had +> no way to know was coming."* + +**Falsifier:** a witness whose moves are all `visible` but which no seat +could have chosen means the marking is wrong. A test constructs that case. + +### 2.3 Wording when nothing is found + +> **"No winning line found in the last K rounds."** + +**Never "unwinnable".** A bounded search cannot establish unwinnability, +and this sentence is what the player reads. + +## 3. The bound + +**Exhaustive over the last `K` rounds**, table treated as one co-operative +agent choosing joint selections. + +**`K = 1` for an exhaustive answer; `K` may be larger when a witness is +all that is wanted.** ADR-0013 said `K = 2` by default; §3.1's measurement +overrides it, and the difference is which question is being asked: + +| answer | needs | affordable `K` today | +|---|---|---| +| *"here is a winning line"* | one success | 2+ — DFS finds one in tens of nodes | +| *"there is no winning line"* | exhaustion | **1** — `K=2` exceeded 2×10⁶ nodes at two seats | + +A `K` that cannot be exhausted may still emit a witness; it may **not** +report `NoneFound { exhausted: true }`, and the type keeps those apart. + +Bounded in **rounds**, not nodes: *"winnable from round 4"* means something +to a player; *"winnable within 100,000 nodes"* does not. A node budget is a +secondary cut that aborts with a stated reason so a wide table cannot hang +the page. + +### 3.1 Measured cost, and what it permits + +`cargo bench -p games-ground --bench search` — the single source for these +numbers (ADR-0013 D7). Mid-game states at real decision points: + +| seats | branch width | `legal_commands` | `clone` | `validate+fold` | +|---|---:|---:|---:|---:| +| 2 | 5 | 4.06 µs | 378 ns | 696 ns | +| 3 | 8 | 4.13 µs | 432 ns | 508 ns | +| 4 | 11 | 4.76 µs | 639 ns | **3.76 µs** | + +**Per-child cost is not uniform** — `validate+fold` ranges 0.5–3.8 µs +depending on which command is taken, because some resolve cascades and +some do not. **Budgets use the upper end**, so ~5 µs per child +(clone + validate + fold). + +Joint branching over the last two rounds, from the measured per-seat +widths: + +| seats | joint / 2 rounds | at ~5 µs/child | +|---|---:|---:| +| 2 | ~5×10² | negligible | +| 3 | ~1.6×10⁵ | **~0.8 s** | +| 4 | ~5.7×10⁵ | **~2.9 s** | + +> ### The projection above was wrong, and the real search falsified it +> +> **Measured 2026-08-05 with the search built in T05**, rewinding real +> games to the start of their last `K` rounds: +> +> | case | result | +> |---|---| +> | 2p, `K=1` | **exhausted** in 8,103 nodes, ~29 ms — a real negative | +> | 2p, `K=2` | **budget cut** at 2,000,000 nodes, ~5 s — not exhausted | +> | 3p, `K=2` | win found in 41 nodes, ~157 µs | +> +> §6's falsifier said *"§3 fails if K=2 proves unaffordable in practice at +> four seats"*. **It failed at two.** +> +> The projection assumed a joint product per round. The search explores +> **sequential per-seat decisions**, and the post-Reveal phase branches +> over every seat's options at every level, so orderings multiply the tree +> far beyond `width^seats`. +> +> **And the asymmetry is the operative fact:** *finding* a win is cheap — +> depth-first stumbles onto one in tens of nodes — while *proving none +> exists* is expensive, because it must exhaust the space. So: +> +> - **the witness feature (§2) is affordable now**, at any `K` a player +> would ask about; +> - **the winnable fraction (§4.2) is not**, because its "not winnable" +> half requires exhaustion on every deal it counts. +> +> `K = 1` is the honest default for exhaustive answers today. Making +> `K = 2` exhaustible needs transposition or move-ordering, neither of +> which this pass built. + +**The published 112–161 µs/node figure is withdrawn** (CB-RES-0008 §1.2, +challenge C1) and must not be quoted from anywhere. + +## 4. Difficulty + +### 4.1 A bot's win rate is not a difficulty + +**Normative prohibition**, because this project already made the error and +nearly exported it: + +> A win rate from a single policy **may not be reported as a difficulty**. + +Measured, on identical deals: `GreedyPolicy` wins **100%** at five and six +seats where a `FirstLegal` policy — take `legal[0]`, no heuristic — wins +**0%**; at two seats `FirstLegal` (77.5%) *beats* greedy (66.0%). Two +unsophisticated agents span the entire range. + +And a measure that improves when the *measurer* improves is not measuring +the subject: a better bot would make the game "easier" with no rule +changing. + +### 4.2 What is reported instead + +> **Winnable fraction** — over N deals at a seat count, the proportion in +> which the search finds a winning line within its bound. + +A property of the **deal distribution and the threshold**, which is what +`ground-game` tunes. Ships as a table, never one number: + +| column | what it is | +|---|---| +| winnable fraction | can the deal be won at all — bounded, `K` stated | +| reference-policy win rate | what a **named** policy achieves | +| skill gap | the difference — how much play has to supply | + +**It is a lower bound and must be labelled one.** A `K`-round search cannot +see a line that required round 1, so the figure is +**"winnable-from-round-(6−K)"**, never "winnable". + +**Every rate carries its policy, N, seed range and K in the number's +name**, not in a footnote — GameDesign §1.2, and the reason the withdrawn +finding was inadmissible. + +### 4.3 Resolution — the number that makes it usable + +> The smallest threshold change the measurement can distinguish, with its +> N. + +*"We can tell a threshold of 5 from 7 but not 7 from 8"* is more useful to +`ground-game` than any rate with no error bar, and it is what makes the +figure a tuning instrument rather than a statistic. + +## 5. The instruments must be able to fail + +ADR-0013 D5, and GameDesign §1.3. Before **any** figure from these tools is +quoted anywhere: + +- **positive controls** — a deal constructed to be unwinnable returns + none; a deal constructed to be winnable returns a witness that replays; +- **`--self-test`**, wired into `make self-tests` like every other + reporting tool; +- **one command regenerates the figure** (`make difficulty`); +- **the policy panel is plural** — at least `greedy`, `random` and + `first-legal`. The spread between them is the finding §4.1 rests on, and + reporting one policy would restore the error. + +**`games/ground/examples/difficulty-baseline.rs` currently satisfies none +of the first three** and is inadmissible until it does. It has no +assertions, no self-test, and no `make` target — nothing can turn it red, +which under CB-WP-0022 T05's `role` distinction makes it a `default` +artifact wearing a `counterexample` label. + +## 6. Falsifiers for this spec + +- **§2.2 fails** if a witness is emitted whose moves are all `visible` but + which no seat could have chosen. Then the marking must be derived from + the search rather than checked after it. +- **§4.2 fails** if the winnable fraction turns out to be ~100% or ~0% at + every seat count and threshold — it would then have no resolution (§4.3) + and be as useless as the bot rate it replaced. +- **§3 fails** if `K = 2` proves unaffordable in practice at four seats; + the measured 2.9 s is a projection from branch widths, not a timing of + the real search. diff --git a/workplans/CB-WP-0025-could-we-have-won.md b/workplans/CB-WP-0025-could-we-have-won.md index 23b4739..3074943 100644 --- a/workplans/CB-WP-0025-could-we-have-won.md +++ b/workplans/CB-WP-0025-could-we-have-won.md @@ -2,7 +2,7 @@ id: CB-WP-0025 kind: product title: "Could we have won: a path out of a lost game, and how hard the game actually is" -status: active +status: done state_hub_workstream_id: "a866982f-94e7-432b-b0d5-2fefb781a574" --- @@ -46,45 +46,30 @@ middle of stating, arriving with a concrete demand. ## What already exists, so the survey does not re-find it -- **The state is replayable.** `cb-game-runtime` records sessions as - scenarios; `replay.rs` and `make replay-test` already re-run them. - A search does not need new persistence. -- **The move space is enumerable.** `legal_commands` exists and, since - CB-WP-0023, is narrow enough to be worth trusting — SOLVE is offered - only where it can act, so the branching factor is real rather than - inflated by inert moves. -- **Bots exist.** `games/ground/src/bot.rs` has `GreedyPolicy` and - `RandomPolicy`, wired through `bot_policy` (`table.rs:208`). A win rate - over N seeds is reachable with what is already there — the question is - whether that number *means* anything, which is the survey's problem. -- **The threshold is public.** `OutcomeView.total` / `.threshold` / - `.group_success`. Difficulty has a denominator already. +The state is replayable (`replay.rs`, `make replay-test`); the move space +is enumerable (`legal_commands`, narrowed by CB-WP-0023 so branching is +real rather than inflated by inert moves); bots exist (`bot.rs`); and the +threshold is public (`OutcomeView`). **A search needs no new persistence +and no new rules** — which is why D6 put it in `games/ground` with no new +crate and no port. ## What makes this hard, and must not be waved through -**The game is not perfect-information and the search must respect that.** -A path computed with the deck known is a path the players could never have -found. `view.rs` hides the deck, other seats' hands, and face-down -selections *by rule* (GR-S02/S04, GR-R02/R04). A retrospective solver -running on `GroundState` sees all of it. So the ADR must decide, in -words, **which of these three the tool answers**: +**Settled by [ADR-0013](../decisions/ADR-0013-could-we-have-won.md).** The +declaration framed the central risk as *a path computed with the deck +known is a path the players could never have found*, and asked the ADR to +choose between an omniscient, an information-respecting, and a bounded +search. -- *was this deal winnable by an omniscient player* — cheap, honest, - and answers a question nobody asked; -- *was it winnable from what the seats could see* — the question actually - asked, and the expensive one; -- *did a reasonable line exist* — a bounded search from the losing seat's - information, which may be the only affordable honest answer. +**D1 dissolved the choice**: strategy fusion is a defect of aggregating +over determinizations to *choose a move*, and after the game there is one +world — so a line found in it is executable in it. **D2** keeps the +declaration's real concern by marking each move `visible`/`hidden` rather +than by refusing to search. -Getting this wrong produces a feature that tells the maintainer he could -have won by playing a card he had no way to know was there. **That is -worse than not shipping it.** - -**And a difficulty number is a claim about a distribution.** One win rate -over one bot policy over N seeds is not "the difficulty"; it is that -policy's win rate. Whatever the spec adopts must name its policy, its N, -and its seed range, or `ground-game` will tune tiers against a number -whose meaning drifts the next time a bot improves. +The second warning here — *"one win rate over one bot policy is not the +difficulty"* — was right, and **the survey made exactly that error +anyway**; see T02. ## Task: survey @@ -99,62 +84,39 @@ state_hub_task_id: "556cfd24-5992-4cc0-be90-0b60e989b5bb" (`loop-lint` checks both). Per §Step 1 the survey is done when it can name a **benchmark-to-beat** -per dimension — a number or a reproducible comparison, not an impression. - -- **Retrospective solvers in games with hidden information.** The prior art - is real and should be named: determinized search (perfect-information - Monte Carlo) and its known failure — *strategy fusion*, where a - determinizing solver claims lines that require knowing which world it is - in. That failure is exactly the trap in §What makes this hard. Bridge - and Skat post-mortem tools are the closest analogues; poker solvers are - the well-studied case and the wrong shape. -- **"A path to win" as a product, not a proof.** The maintainer already - conceded optimality (*"the best path is not computable I guess"*). So - the target is a **witness**: one concrete line of play that reaches - `group_success`, or a defensible *no line found within bound B*. Name - what a witness must carry to be checkable. -- **Difficulty as a measured quantity in co-operative games.** Pandemic and - its relatives set difficulty by a dial with a published win rate. The - benchmark-to-beat is: can we produce a win rate whose confidence - interval is tight enough to distinguish two threshold settings? -- **Cost.** Search over an event-sourced aggregate with full `validate` on - every branch has a per-node price. Measure it on our machine, on our - scenarios — the runnable-baseline option applies here, since a search - that cannot finish while the player is still looking at the page is a - different feature. +per dimension. Four were asked for: retrospective solvers in +hidden-information games (and their known failure, strategy fusion); *a +path to win* as a **witness** rather than a proof; difficulty as a measured +quantity in co-operative games; and the **per-node cost**, measured on our +machine — the runnable-baseline option applies, since a search that cannot +finish while the player is looking at the page is a different feature. **Done 2026-08-05.** [CB-RES-0008](../research/CB-RES-0008-could-we-have-won.md), with a runnable baseline (`games/ground/examples/difficulty-baseline.rs`). -**The baseline produced a finding before any solver exists, and it is the -biggest thing in this pass.** A greedy bot wins **200 of 200** games at -five and six seats — with a median margin of +3 and 11.8–12.0 points -available against a threshold of 9. The curve is 66% / 82.5% / 95% / 100% -/ 100% across 2/3/4/5/6 seats. +> **Everything this record originally claimed was withdrawn by T02 the +> same day.** Kept as a pointer rather than rewritten, because a claim +> retracted silently is how three earlier wrong premises survived +> (ADR-0012 D5). -The row-level table shows why: available points go **6 / 9 / 12** against -thresholds **5 / 7 / 9**, so the ratio *rises* with seat count (1.20 → -1.29 → 1.33) while the table also gains actions per round. **Three -multipliers pointing the same way.** +**Claimed:** a greedy bot wins 200/200 at five and six seats, so the game +is too easy there; the points-to-threshold ratio rises with seat count and +explains the curve; `legal_commands` costs 112–161 µs, so exhaustive +search is out; and all of this explains the maintainer's report. -**It also explains the maintainer's report without a solver.** *"I felt it -was too easy but then we lost"* — 66% at two seats is a real game; 100% at -six is not. Both halves are true of different seat counts. +**Withdrawn:** the win rate is `GreedyPolicy`'s, not the game's (C4 — +`FirstLegal` scores 0% on the same deals); the ratio is identical at 3p +and 4p, which differ by 12.5 points (C2); the cost figure was wrong by +30–50× and exhaustive search is affordable (C1, C6); and the maintainer's +losses were 3-player games on the pre-ruling deal (C5). -**Cost measured, and it rules out the obvious approach.** Branching is -small (mean 4.7–9.1) but `legal_commands` costs **112–161 µs** per call, -because it filters candidates through full `validate`. Exhaustive search -is out at every seat count; ~10⁴–10⁵ nodes is 1.4–14 s, which is the -budget the ADR must design inside. - -**Prior art names the trap.** Determinized search (PIMC) suffers *strategy -fusion* — Frank, Basin & Matsubara 1998 — where the search picks different -actions in states a real player cannot tell apart. A witness built that -way may require knowing what was on top of the deck. **And it would still -replay green**, so the checkability benchmark does not catch it. Honesty -and checkability are different properties, stated so T03 cannot conflate -them. +**What survives:** the harness exists and runs, the 6/9/12 arithmetic is +right against `Problems.csv`, and the branching widths (4.7–9.1) hold. See +[CB-RES-0008](../research/CB-RES-0008-could-we-have-won.md) for the +corrected text and +[the response](../history/260805-could-we-have-won-response.md) for the +full accounting. ## Task: adversarial review @@ -166,103 +128,89 @@ state_hub_task_id: "bdd9fc97-4706-4d8e-b286-91ff53339680" ``` Tier L requires it. Exactly one round: challenge, then response, trail in -`history/`, unpolished. Require an attempt at: - -- **that the honest version is unaffordable** — that a search respecting - the information rule is too expensive or too weak to find anything, so - the shipped tool will quietly become the omniscient one with a - reassuring label; -- **that a witness misleads more than it helps** — being shown a line that - needed a card you could not know about teaches a wrong lesson about the - game, and the tool would be better refusing to answer; -- **that the difficulty number is a bot benchmark wearing a difficulty - costume**, and `ground-game` will tune the game against our bot rather - than against play; -- **that this is CB-WP-0022's job** — the design instrument is being built - right now, and a difficulty measurement is a finding-producing tool. The - strongest counter is that the register records findings and this - *produces* them, but the reviewer should press whether that is a - distinction worth a separate capability. +`history/`, unpolished. Four attempts required: that the honest version is +unaffordable; that a witness misleads more than it helps; that the +difficulty number is a bot benchmark in a difficulty costume; and that +this is CB-WP-0022's job rather than a separate capability. **Done 2026-08-05.** Trail: [challenge](../history/260805-could-we-have-won-challenge.md), [response](../history/260805-could-we-have-won-response.md). Separate agent, as in CB-WP-0022. -**Six of seven conceded, and the survey's headline finding is withdrawn — -not softened.** +**Six of seven conceded. The survey's headline finding is WITHDRAWN.** -**C4 is the one that kills it, and the reviewer ranked it fourth.** -Measured: a `FirstLegal` policy — take `legal[0]`, no heuristic — scores -**0% at five and six seats** where greedy scores 100%, and **77.5% at two -seats** where greedy scores 66%. **Two unsophisticated agents span the -entire range at the same seat count**, so *"the game is too easy at 5–6 -seats"* is a statement about `GreedyPolicy`, not about GROUND. Their -offered rescue — greedy hits the 12-point ceiling every time, so it is a -rules claim — dies on the same data. +**C4 kills it and the reviewer ranked it fourth.** A `FirstLegal` policy — +`legal[0]`, no heuristic — scores **0% at five and six seats** where greedy +scores 100%, and **77.5% at two** where greedy scores 66%. Two +unsophisticated agents span the whole range, so *"too easy at 5–6 seats"* +was about `GreedyPolicy`, not GROUND. -**C1: the per-node cost was wrong by 30–50×.** The timer bracketed two -`setup`s, a whole greedy game and a validate+fold replay, then divided by -player decisions. **The tell was in my own published output**: the figure -*fell* as branching *rose*, which no per-enumeration cost can do. -Re-measured at **3.0–4.1 µs**; the reviewer got 15.6–20.4 by a different -isolation and **that discrepancy is unsettled** — T04 benchmarks it. - -**C6: exhaustive search is not out**, which changes T03's premise. With -C1's correction the budget is ~10⁵–10⁶ nodes and bounded endgame search -fits — so the ADR cannot open with *"exhaustive is impossible, therefore -determinized sampling"*, especially since sampling carries strategy fusion -that exhaustive search does not. - -**C3: the finding failed the admissibility rule this project wrote nine -hours earlier** — sums where GROUND-WP-0004 T02 requires per-priority -rows, and a harness with no assertions, no `--self-test` and no `make` -target, so nothing can turn it red. A `default` artifact wearing a -`counterexample` label. - -**C2: the ratio explains nothing** — 3p and 4p share deal, threshold and -ratio, and differ by 12.5 points of win rate. **C5: the "explains the -maintainer's report" claim is contradicted by `lib.rs:2487`**, which -records his losses as 3-player on the pre-ruling deal, unwinnable at 6 -against 7. +**C1**: the node cost was wrong by 30–50× — the timer bracketed whole +games — and **the tell was in my own output**, falling as branching rose. +**C6**: exhaustive search is *not* out, which changes T03's premise. +**C3**: the finding failed the admissibility rule this project wrote nine +hours earlier. **C2**: 3p and 4p share deal, threshold and ratio and differ +by 12.5 points. **C5**: `lib.rs:2487` contradicts the "explains the +maintainer" claim — his losses were 3-player on the pre-ruling deal, +unwinnable at 6 against 7. **T06 was pointed at GROUND-WP-0005, which is blocked on this number.** -Sending it would have invited threshold changes on one bot's behaviour — -the fifth wrong premise to reach ground-game. **Both tier-L reviews in -this project have now caught a false headline that every gate passed.** +Sending it would have been the fifth wrong premise to reach ground-game. +**Both tier-L reviews here have now caught a false headline that every +gate passed.** ## Task: decide ```task id: CB-WP-0025-T03 -status: todo +status: done priority: high state_hub_task_id: "055630a1-56e0-4ad4-8796-35f444313371" ``` `decisions/ADR-0013-*.md`. (ADR-0012 is CB-WP-0022 T03's.) At minimum: +which question the solver answers and what it is called in the UI; the +**information boundary** and whether it is structural or promised; the +**bound**, and what *no path found* means against it; whether difficulty +is one number or a table, and of what; and **where it lives** — the tier +was declared L assuming a new capability port, and if there is none, say +so. -- **which question the solver answers**, from the three in §What makes - this hard, and what it is called in the UI — the name must not overclaim; -- **the information boundary**: whether the search runs on `GroundState` - or on a `GroundView`, and if on state, what stops it using what the view - hides. Note that running on the view makes the rule structural rather - than a promise, and that this is the cheapest guarantee available; -- **the bound**: depth, node budget, or wall clock, and what *no path - found* means against it — a bounded search that says "unwinnable" is - lying, and the wording must say "none found within B"; -- **whether difficulty ships as one number or a small table**, and what it - is a function of: policy, seat count, threshold, seed range; -- **where it lives** — a new crate, a mode of `cb-play`, or a tool under - `tools/`. The tier was declared L on the assumption of a new capability; - if the ADR concludes it is a mode of an existing one, say so, and the - over-declaration is a chaos-window data point worth recording. +**Done 2026-08-05.** +[ADR-0013](../decisions/ADR-0013-could-we-have-won.md), seven decisions. +**Two of them are not what T03 was written expecting**, because the review +moved the ground under both. + +- **D1 — strategy fusion does not apply, and that is why this is + affordable.** Fusion is a defect of *aggregating over determinizations + to choose a move*. **After the game there is one world**: the deal is + known, so a search over it produces a line executable in the only world + there is. The survey treated fusion as this pass's central obstacle; it + is an obstacle to a *playing* engine, which we are not building. +- **D4 — difficulty is the WINNABLE FRACTION, not any bot's win rate.** + C4 killed the bot rate: two trivial policies span 0–100% on the same + deals, and improving the bot would make the game "easier" without a rule + changing. The solver supplies a policy-independent measure — *over N + deals, in what proportion does a winning line exist* — which is a + property of the deal distribution and the threshold, and is what + GROUND-WP-0005 actually needs. **The bot rate never was.** + +**D2** searches `GroundState` — the survey's view-only boundary is not +implementable, since a view cannot fold events — and moves the guarantee to +a checkable per-move `visible`/`hidden` marking from `project()`. **D3** +bounded exhaustive over the last K rounds, with *"no winning line found in +the last K rounds"* normative, never *"unwinnable"*. **D5** makes the +harness an instrument before any figure is quoted (C3). **D6** no new +crate and no port — **the L declaration was an over-declaration, recorded +as one**. **D7** the node cost is disputed 5×; T04 benchmarks it and +neither existing figure may be cited. ## Task: specify ```task id: CB-WP-0025-T04 -status: todo +status: done priority: high state_hub_task_id: "7c3ca25d-0001-44ef-a07d-0ab2d51b09a5" ``` @@ -270,29 +218,53 @@ state_hub_task_id: "7c3ca25d-0001-44ef-a07d-0ab2d51b09a5" `specs/` — extend `MetricsAndScenarios.md` or add a capability spec as the ADR directs, with metrics, because a spec without them is prose. -Candidates, to be argued not adopted: - -- **witness checkability** — every path the tool emits replays through the - existing scenario runner and ends in `group_success`. Target 100%, and it - is a hard gate, not a metric: a path that does not replay is a bug that - says the opposite of the truth; -- **search cost** — nodes and wall clock at the chosen bound, on the - recorded games we have; -- **difficulty resolution** — the smallest threshold difference the - measurement can distinguish, with its N. This is the number - `ground-game` needs, and stating it as *"we can tell 5 from 7 but not 7 - from 8"* is more useful than a win rate with no error bar. +Candidates, to be argued not adopted: **witness checkability** (every +emitted path replays to `group_success` — a hard gate, not a metric, since +a path that does not replay says the opposite of the truth); **search +cost** at the chosen bound; and **difficulty resolution** — the smallest +threshold difference the measurement can distinguish, with its N, which is +more useful to `ground-game` than any rate with no error bar. Per `ground-game`'s ruling (GROUND-WP-0004 T02), **any arithmetic finding this produces ships a runnable reproduction and a row-level table** — never a summed figure. A difficulty number is arithmetic, and it is exactly the kind that has already gone wrong twice. +**Done 2026-08-05.** +[specs/RetrospectiveAnalysis.md](../specs/RetrospectiveAnalysis.md) v1.0, +and `games/ground/benches/search.rs` for D7's disputed number. + +**The benchmark's own first fixture was defective — the same defect class, +one layer down.** A fixed stop at step 20 put 2p and 4p where seat 0 had +**no legal commands**, so it timed an empty `Vec` (~120 ns) and silently +skipped `validate_fold`. It now advances to a real branch **and asserts +it**. + +Measured at real decision points (table in +[RetrospectiveAnalysis §3.1](../specs/RetrospectiveAnalysis.md)): +`legal_commands` 4.06–4.76 µs, `clone` 378–639 ns, `validate+fold` +**0.5–3.8 µs**. + +**`clone` is in there because a search must copy state per branch**, and +`iter_batched` excludes setup from timing — leaving the budget on an +unmeasured span, which is precisely C1's mistake. + +**The affordability conclusion drawn here was itself falsified by T05.** +It projected joint branching and concluded `K = 2` holds at two to four +seats. The real search exceeded 2×10⁶ nodes at **two**. See T05's record +and [RetrospectiveAnalysis §3.1](../specs/RetrospectiveAnalysis.md) — +a projection from branch widths is not a timing of a search, and this +pass has now made that mistake twice. + +**§4.1 is a normative prohibition**, not a preference: a single-policy win +rate may not be reported as a difficulty. The spec carries the measured +reason — greedy 100% vs first-legal 0% on identical deals. + ## Task: build the witness ```task id: CB-WP-0025-T05 -status: todo +status: done priority: high state_hub_task_id: "5f1cbda3-7427-4f23-8c67-b2e216b987a8" ``` @@ -313,23 +285,48 @@ a loss. naming *that* boundary goes red. If it cannot be mutated, it was a comment rather than a rule. +**Done 2026-08-05.** `games/ground/src/search.rs`, five tests. + +**The first traversal was wrong and the diagnostic hid it.** It branched +on *the first seat with any legal command* and stopped there, so a later +seat never acted if an earlier one was already selected. Restructured +around what the rules oblige: a seat without a selection **must** select +(GR-R02) and nothing else can happen first; after Reveal the optional +actions branch freely, and the aggregate rejects `Resolve` until the +obligatory ones are done — **so the search needs no phase logic of its +own.** + +**And my rewind was off by one round**, replaying the round it was meant +to search — which is why the first run reported 3 nodes and looked fine. + +**The measurement falsified the spec's own projection at two seats, where +§6 predicted four**: 2p `K=1` exhausted in 8,103 nodes (~29 ms), 2p `K=2` +cut at 2,000,000 nodes (~5 s), 3p `K=2` won in 41 nodes. + +The projection assumed a joint product per round; the search explores +sequential per-seat decisions, so orderings multiply the tree far beyond +`width^seats`. + +**The asymmetry is the operative finding.** *Finding* a win is cheap; +*proving none exists* needs exhaustion. So the **witness feature is +affordable now** and the **winnable fraction is not**, because its +negative half must exhaust every deal it counts. + ## Task: measure the difficulty, and hand it to ground-game ```task id: CB-WP-0025-T06 -status: todo +status: done priority: high state_hub_task_id: "723feae8-1ed4-4dd4-9330-8ec84625e5d9" ``` Run the measurement, ship it as a `make` target beside the other -instruments, and show the result in the game — the maintainer asked for it -to be visible, and a number in a file will not calibrate anything. - -Then send it to `ground-game` **against GROUND-WP-0005**, which is active -with both tasks waiting on exactly this. Per CB-WP-0022 T06, it lands as a -file in their repo under their workplan, not only an inbox entry — *the -message that sat unread for four days is the baseline to beat*. +instruments, and send it to `ground-game` against **GROUND-WP-0005**, +which is active with both tasks waiting on exactly this. Per CB-WP-0022 +T06 it lands as a **file in their repo under their workplan**, not only an +inbox entry — *the message that sat unread for four days is the baseline +to beat*. **Controls:** - the number regenerates from a single command, and `facts.toml` carries @@ -339,22 +336,65 @@ message that sat unread for four days is the baseline to beat*. - **the seed range and policy are in the number's name**, not in a footnote. +**Done 2026-08-05.** `games/ground/examples/difficulty.rs`, `make +difficulty`, wired into `make self-tests`, and +[a report file in ground-game](../../ground-game/workplans/GROUND-WP-0005-clay-borg-difficulty-260805.md) +under GROUND-WP-0005 with a hub message pointing at it. + +**The report opens with the retraction**, because what this task was +written to send was withdrawn by T02 and GROUND-WP-0005 is blocked on +exactly that number. + +Winnable-from-round-5, 60 seeds: **60% / 93% / 100% / 100% / 100%** at +2/3/4/5/6 seats, beside greedy / random / first-legal rates and the spread +between them. Full table in the report and in `make difficulty`. + +**`spread` justifies the whole redesign** — 71.7 to 100.0 points between +three trivial policies. No single rate says anything about the game, and +the table now shows that rather than asserting it. + +**The confound is stated in the tool's own output**: `winnable` is +conditioned on greedy's play up to the final round. Presenting it as a +property of the deal would repeat this pass's error in a subtler form. +**No threshold changes are proposed.** The instrument can fail (spec §5), +and `difficulty-baseline.rs` is marked superseded. + +Registered as **F16, `inconsistent`, `withdrawn`**, with `difficulty.rs` +as its reproduction. + ## Task: evidence ```task id: CB-WP-0025-T07 -status: todo +status: done priority: high state_hub_task_id: "2616e06c-5d88-4680-be18-4452515dff21" ``` -`evidence/CB-EV-0023-*.md`. +`evidence/CB-EV-*.md` (**CB-EV-0024** — CB-EV-0023 went to CB-WP-0024). +Was the maintainer's game winnable; what the +honest search cost against the omniscient one; whether the difficulty +measurement moved `ground-game`; what tier L cost against what it caught; +and CB-WP-0024's cost by re-running the instrument. -- **Was the game winnable**, for the maintainer's actual lost game. That is - the acceptance test with a face on it. -- **What the honest search cost against the omniscient one**, since the - review will have pressed hardest there. -- **Whether the difficulty measurement moved ground-game**, or sat. -- **What tier L cost against what it caught** — third full-weight L pass in - the project, and the second in this chaos window. -- **Quote CB-WP-0024's cost by re-running the instrument.** +**Done 2026-08-05.** +[CB-EV-0024](../evidence/CB-EV-0024-could-we-have-won.md). + +- **The maintainer's question is answered.** 3p seed 7 was winnable — a + 10-move line, found in 41 nodes, replays to `group_success`. 2p seed 7 + was **not**, exhaustively, in 8,103 nodes. **The witness feature works; + the difficulty half is where the trouble was.** +- **Five of nine defects came only from the review**, four from execution + — and all four of those were in work written *after* it. +- **The wrong-denominator family now has five instances and no control.** +- **The 5–6 seat rows point the opposite way from the withdrawn claim**; + neither reading is established, and the evidence says so. +- **Tier L was an over-declaration** (no port) **and paid for itself + anyway** — the review is L-only, and without it a false number reaches a + blocked repo. +- **Chaos window 2 will close with zero overrides**, making its retirement + condition untestable. + +**Not done, and named rather than quietly dropped:** the witness is **not +wired to the ending page**, so the browser cannot yet ask the question. +That is the maintainer-facing half of remark 2.