Compare commits
5 commits
16adcee8e8
...
7ed9fc730a
| Author | SHA1 | Date | |
|---|---|---|---|
| 7ed9fc730a | |||
| 81e0aba59a | |||
| b27aa14df0 | |||
| 3a026b1e1f | |||
| 88e25ae7c4 |
13 changed files with 1743 additions and 188 deletions
7
Makefile
7
Makefile
|
|
@ -125,6 +125,7 @@ self-tests:
|
|||
$(PY) $(TOOLS)/runtime-metrics.py --self-test
|
||||
$(PY) $(TOOLS)/replay-test.py --self-test
|
||||
$(PY) $(TOOLS)/design.py --self-test
|
||||
cargo run --release -q -p games-ground --example difficulty -- --self-test
|
||||
$(PY) $(TOOLS)/edition-check.py --self-test
|
||||
|
||||
# T01 positive control: prove the environment fix, do not assume it. Runs
|
||||
|
|
@ -160,6 +161,12 @@ facts-check:
|
|||
facts-gen:
|
||||
$(PY) $(TOOLS)/facts.py --gen
|
||||
|
||||
# CB-WP-0025 T06: the difficulty table (specs/RetrospectiveAnalysis.md §4).
|
||||
# Winnable fraction from the solver plus a PLURAL policy panel -- a single
|
||||
# policy's win rate may not be reported as a difficulty (§4.1).
|
||||
difficulty:
|
||||
@cargo run --release -q -p games-ground --example difficulty
|
||||
|
||||
# CB-WP-0022 T05: the design-finding register, reported over
|
||||
# specs/GroundRules.md. Shows the QUEUE by default; the log of closed
|
||||
# findings is a line, not a listing, because a default view that mixes
|
||||
|
|
|
|||
|
|
@ -32,7 +32,7 @@
|
|||
| workplan | CB-WP-0022 | done | — | workplans/CB-WP-0022-the-design-instrument.md |
|
||||
| workplan | CB-WP-0023 | done | — | workplans/CB-WP-0023-solve-legality.md |
|
||||
| workplan | CB-WP-0024 | done | — | workplans/CB-WP-0024-the-table-you-can-watch.md |
|
||||
| workplan | CB-WP-0025 | ready | — | workplans/CB-WP-0025-could-we-have-won.md |
|
||||
| workplan | CB-WP-0025 | active | — | workplans/CB-WP-0025-could-we-have-won.md |
|
||||
| workplan | CB-WP-0026 | done | — | workplans/CB-WP-0026-collect-the-rulings.md |
|
||||
| task | CB-WP-0001-T01 | done | — | workplans/CB-WP-0001-inner-loop.md |
|
||||
| task | CB-WP-0001-T02 | done | — | workplans/CB-WP-0001-inner-loop.md |
|
||||
|
|
@ -162,8 +162,8 @@
|
|||
| task | CB-WP-0024-T03 | done | — | workplans/CB-WP-0024-the-table-you-can-watch.md |
|
||||
| task | CB-WP-0024-T04 | done | — | workplans/CB-WP-0024-the-table-you-can-watch.md |
|
||||
| task | CB-WP-0024-T05 | done | — | workplans/CB-WP-0024-the-table-you-can-watch.md |
|
||||
| task | CB-WP-0025-T01 | todo | — | workplans/CB-WP-0025-could-we-have-won.md |
|
||||
| task | CB-WP-0025-T02 | todo | — | workplans/CB-WP-0025-could-we-have-won.md |
|
||||
| task | CB-WP-0025-T01 | done | — | workplans/CB-WP-0025-could-we-have-won.md |
|
||||
| task | CB-WP-0025-T02 | done | — | workplans/CB-WP-0025-could-we-have-won.md |
|
||||
| task | CB-WP-0025-T03 | todo | — | workplans/CB-WP-0025-could-we-have-won.md |
|
||||
| task | CB-WP-0025-T04 | todo | — | workplans/CB-WP-0025-could-we-have-won.md |
|
||||
| task | CB-WP-0025-T05 | todo | — | workplans/CB-WP-0025-could-we-have-won.md |
|
||||
|
|
|
|||
230
decisions/ADR-0013-could-we-have-won.md
Normal file
230
decisions/ADR-0013-could-we-have-won.md
Normal file
|
|
@ -0,0 +1,230 @@
|
|||
# ADR-0013: answer the retrospective question, and measure difficulty with the solver rather than with a bot
|
||||
|
||||
status: accepted
|
||||
date: 2026-08-05
|
||||
decided by: agent, under the standing loop authorization
|
||||
tier: L (structural L — a new analysis capability whose numbers `ground-game`
|
||||
will tune against; chaos d8=6 → no override). Tier L: separate survey,
|
||||
adversarial review, then this document.
|
||||
references: [CB-WP-0025](../workplans/CB-WP-0025-could-we-have-won.md),
|
||||
[CB-RES-0008](../research/CB-RES-0008-could-we-have-won.md),
|
||||
[challenge](../history/260805-could-we-have-won-challenge.md) /
|
||||
[response](../history/260805-could-we-have-won-response.md),
|
||||
[ADR-0012](ADR-0012-the-design-instrument.md) (admissibility),
|
||||
[GameDesign.md](../specs/GameDesign.md),
|
||||
GROUND-WP-0004 T02, GROUND-WP-0005 (blocked on D4)
|
||||
|
||||
## Context
|
||||
|
||||
The maintainer asked two things: *"we lost — could we have won, and how?"*
|
||||
and *"do we have difficulty estimations?"*
|
||||
|
||||
**The survey answered the second and was wrong.** It measured
|
||||
`GreedyPolicy` winning 200/200 at five and six seats and called the game
|
||||
too easy there. A `FirstLegal` policy — `legal[0]`, no heuristic — scores
|
||||
**0%** on the same deals. Two unsophisticated agents span the whole range,
|
||||
so the measurement was about the policy.
|
||||
|
||||
That failure is not incidental to this ADR; **it determines D4.**
|
||||
|
||||
## The premise that changed, and it changes the algorithm
|
||||
|
||||
The survey said exhaustive search was impossible and reached for
|
||||
determinized sampling, which carries strategy fusion. **Both halves were
|
||||
wrong.**
|
||||
|
||||
- Its per-node cost was **30–50× too high** (a timer bracketing whole
|
||||
games). Corrected: ~3–4 µs per `legal_commands` call, with the exact
|
||||
figure still disputed (§D7).
|
||||
- Bounded exhaustive search is **affordable**: measured ~3 s over the last
|
||||
two rounds at three seats.
|
||||
|
||||
Joint branching, treating the table as one co-operative agent — the
|
||||
product over seats of the measured per-seat branching:
|
||||
|
||||
| seats | per-seat mean | joint per round | last 2 rounds |
|
||||
|---|---:|---:|---:|
|
||||
| 2 | 4.7 | ~22 | ~5×10² |
|
||||
| 3 | 7.4 | ~405 | ~1.6×10⁵ |
|
||||
| 4 | 9.1 | ~754 | ~5.7×10⁵ |
|
||||
|
||||
Against a ~10⁵–10⁶ node budget, **the last two rounds are exhaustively
|
||||
searchable at two, three and four seats.** Five rounds is not, at any seat
|
||||
count.
|
||||
|
||||
---
|
||||
|
||||
## D1 — answer the *retrospective* question, and say so in those words
|
||||
|
||||
Three questions were on the table (CB-RES-0008 §3). The tool answers:
|
||||
|
||||
> **"Given the deal as it actually was, was there a line of play that
|
||||
> reached the threshold — and here is one."**
|
||||
|
||||
**Strategy fusion does not apply to this question, and that is the whole
|
||||
reason it is the affordable one.** Fusion is a defect of *aggregating over
|
||||
determinizations to choose a move*: the search picks different actions in
|
||||
states the player cannot distinguish. **After the game there is one
|
||||
world.** The deck is known, the deal is known, and a search over that
|
||||
single world produces a line that is executable in it — because it is the
|
||||
only world there is.
|
||||
|
||||
The survey treated fusion as an obstacle to this pass. It is an obstacle
|
||||
to a *playing* engine. We are not building one.
|
||||
|
||||
**What remains true is that the line may have been unfindable at the
|
||||
time**, and D2 handles that by annotation rather than by refusing to
|
||||
answer.
|
||||
|
||||
**On screen it is called** *"was this deal winnable?"* — never *"how you
|
||||
should have played"*. The distinction is the honest content of the
|
||||
feature, and a label that overclaims turns a true answer into a false
|
||||
lesson.
|
||||
|
||||
## D2 — run on `GroundState`, and mark each move's information dependence
|
||||
|
||||
The survey's preferred guarantee was structural: search a `GroundView` so
|
||||
the boundary cannot be crossed. **It is not implementable** — a view
|
||||
cannot `fold` events, so a search needs a state it may not see. The survey
|
||||
said so in §6 and was right to.
|
||||
|
||||
Decision: **search `GroundState`** — legitimate here, because post-game
|
||||
the deal is public (`solution_discard` already is, and the game is over) —
|
||||
and move the honesty guarantee to something checkable:
|
||||
|
||||
> **Every move in an emitted witness is marked `visible` or `hidden`.**
|
||||
> A move is `visible` if, at the point it is played, everything it depends
|
||||
> on was in the acting seat's projection: the target Problem face-up, the
|
||||
> Solution in that seat's own hand. Otherwise `hidden`.
|
||||
|
||||
So a witness reads *"you could have won — but two of these six moves
|
||||
needed a card you had no way to know was coming."* **That is a more useful
|
||||
answer than either extreme**, and it is computed from `project()`, which
|
||||
already exists and is already tested.
|
||||
|
||||
**Falsifier:** if a witness is emitted whose moves are all marked
|
||||
`visible` but which no seat could actually have chosen, the marking is
|
||||
wrong and D2 has failed. A test constructs exactly that case.
|
||||
|
||||
## D3 — bounded exhaustive over the endgame, `K` rounds, and honest wording
|
||||
|
||||
**Exhaustive search over the last `K` rounds**, with the table treated as
|
||||
one co-operative agent choosing joint selections. `K = 2` by default,
|
||||
which the measurements put inside budget at 2–4 seats.
|
||||
|
||||
- The bound is **rounds**, not nodes or seconds, because rounds are what a
|
||||
player understands: *"winnable from round 4"* means something; *"winnable
|
||||
within 100,000 nodes"* does not.
|
||||
- A node budget is a **secondary** cut that aborts with a stated reason,
|
||||
so a wide table cannot hang the page.
|
||||
- **Wording is normative.** When no line is found the tool says
|
||||
**"no winning line found in the last K rounds"** — never *"unwinnable"*.
|
||||
A bounded search that claims unwinnability is lying, and this is the
|
||||
sentence the maintainer will read.
|
||||
|
||||
**Not chosen: determinized sampling (PIMC), ISMCTS.** Both are for playing
|
||||
under uncertainty. Here there is one world (D1), so they would add strategy
|
||||
fusion to a problem that does not have it.
|
||||
|
||||
## D4 — difficulty is the **winnable fraction**, not any bot's win rate
|
||||
|
||||
**This is the decision the review forced, and it is the useful half of the
|
||||
pass.**
|
||||
|
||||
A single-policy win rate cannot be a difficulty: two trivial policies span
|
||||
0–100% on the same deals. Worse, *improving the bot would make the game
|
||||
"easier"* without a rule changing — a measure that moves when the
|
||||
measurer improves is not measuring the thing.
|
||||
|
||||
The solver supplies a policy-independent alternative:
|
||||
|
||||
> **Winnable fraction** — over N deals at a seat count, the proportion in
|
||||
> which the search finds *any* winning line within its bound.
|
||||
|
||||
That is a property of **the deal distribution and the threshold**, which
|
||||
is exactly what `ground-game` tunes. It is the number GROUND-WP-0005 is
|
||||
blocked on, and the bot rate never was.
|
||||
|
||||
Difficulty therefore ships as **a small table, never one number**:
|
||||
|
||||
| column | what it is |
|
||||
|---|---|
|
||||
| winnable fraction | can the deal be won at all (bounded, K stated) |
|
||||
| reference-policy win rate | what a stated bot achieves — **named policy** |
|
||||
| skill gap | the difference: how much play has to supply |
|
||||
|
||||
**Every rate carries its policy, its N, its seed range and its K in the
|
||||
number's name**, not in a footnote. A figure that loses them is
|
||||
inadmissible under GameDesign §1.2.
|
||||
|
||||
**Bounded-below caveat, stated because it will be quoted:** the winnable
|
||||
fraction from a K-round search is a **lower bound** on true winnability —
|
||||
a deal unwinnable in the last 2 rounds may have been winnable in round 1.
|
||||
The report says "winnable-from-round-(6−K)", never "winnable".
|
||||
|
||||
## D5 — the harness becomes an instrument before any figure is quoted
|
||||
|
||||
C3 established that `difficulty-baseline.rs` has no assertions, no
|
||||
`--self-test` and no `make` target — nothing can turn it red. Under
|
||||
CB-WP-0022 T05's own `role` distinction it is a `default` artifact wearing
|
||||
a `counterexample` label, and **GameDesign §1.3 makes it inadmissible.**
|
||||
|
||||
Required before T06 reports anything:
|
||||
|
||||
- **positive controls** — a deal constructed to be unwinnable returns
|
||||
none; a deal constructed to be winnable returns a witness that replays;
|
||||
- **`--self-test`**, wired into `make self-tests` like every other
|
||||
reporting tool;
|
||||
- **`make difficulty`** (or equivalent), so the figure regenerates from
|
||||
one command;
|
||||
- the **policy panel is plural**: at least `greedy`, `random` and
|
||||
`first-legal`, because the spread between them is what C4 exposed and
|
||||
hiding it would restore the error.
|
||||
|
||||
## D6 — it lives in `games/ground`, not a new crate
|
||||
|
||||
The search needs `validate`, `fold`, `legal_commands` and `project` —
|
||||
all of `games_ground`. A separate crate would either re-export the
|
||||
aggregate or take a dependency on it and add nothing.
|
||||
|
||||
**The tier was declared L on the assumption of a new capability port.
|
||||
There is no port**, and that over-declaration is recorded rather than
|
||||
hidden — it is a data point for the tier rules, and the L weight paid for
|
||||
itself twice over regardless (§Consequences).
|
||||
|
||||
`cb-play` gains a mode to ask the question about a finished game; the
|
||||
difficulty sweep is an example/binary, as the baseline is.
|
||||
|
||||
## D7 — the per-node cost is unsettled and T04 must benchmark it
|
||||
|
||||
The author measured **3.0–4.1 µs**, the reviewer **15.6–20.4 µs**, by
|
||||
different isolations. Both agree the published 112–161 µs was wrong by
|
||||
1–2 orders; neither has established which is right.
|
||||
|
||||
**T04 benchmarks it with `criterion`** — already a dev-dependency, already
|
||||
used by `benches/synthetic.rs` — and the spec quotes that number and no
|
||||
other. **Neither figure above may be cited**, including by this ADR.
|
||||
|
||||
## Consequences
|
||||
|
||||
- `specs/` gains the witness contract and the difficulty table's shape
|
||||
(T04), plus the benchmarked node cost.
|
||||
- T05 builds the K-round search, the `visible`/`hidden` marking, and the
|
||||
replay check.
|
||||
- **T06's payload changes completely.** It reports a winnable fraction and
|
||||
a policy panel to GROUND-WP-0005 — *not* the withdrawn "too easy at 5–6
|
||||
seats". The withdrawal itself is reported, per ADR-0012 D5.
|
||||
- The register gains the withdrawn finding as `inconsistent` /
|
||||
`withdrawn`, so it is in the log rather than forgotten.
|
||||
|
||||
## What was rejected
|
||||
|
||||
| rejected | why |
|
||||
|---|---|
|
||||
| determinized sampling / PIMC / ISMCTS | strategy fusion, added to a question that has one world (D1) |
|
||||
| a view-only search as a structural boundary | not implementable — a view cannot fold events |
|
||||
| refusing to answer unless the line was findable at the time | throws away a true and useful answer; annotate instead (D2) |
|
||||
| a single bot win rate as "difficulty" | two trivial policies span 0–100% on the same deals (C4) |
|
||||
| "unwinnable" as output wording | a bounded search cannot know it (D3) |
|
||||
| a new crate | no port exists; it would re-export the aggregate (D6) |
|
||||
| quoting either measured node cost | they disagree 5× and neither is established (D7) |
|
||||
161
evidence/CB-EV-0024-could-we-have-won.md
Normal file
161
evidence/CB-EV-0024-could-we-have-won.md
Normal file
|
|
@ -0,0 +1,161 @@
|
|||
# CB-EV-0024 — could we have won
|
||||
|
||||
CB-WP-0025 T07. Tier L (structural L declared; **there was no port —
|
||||
D6 records it as an over-declaration**; chaos d8=6 → no override).
|
||||
Declaration 8 of chaos window 2. Closed 2026-08-05.
|
||||
|
||||
**Delivered:** [CB-RES-0008](../research/CB-RES-0008-could-we-have-won.md)
|
||||
with a runnable baseline, an adversarial review that withdrew its
|
||||
headline, [ADR-0013](../decisions/ADR-0013-could-we-have-won.md),
|
||||
[RetrospectiveAnalysis.md](../specs/RetrospectiveAnalysis.md),
|
||||
`games/ground/src/search.rs`, `benches/search.rs`,
|
||||
`examples/difficulty.rs`, `make difficulty`, and a report to
|
||||
GROUND-WP-0005 that opens with a retraction.
|
||||
|
||||
---
|
||||
|
||||
## 1. Was the maintainer's game winnable?
|
||||
|
||||
The question the pass exists for. **Yes, and the tool answers it.**
|
||||
|
||||
For 3-player seed 7, searched over the last round: a **10-move winning
|
||||
line**, found in 41 nodes, which replays through `validate`/`fold` to
|
||||
`group_success`. For 2-player seed 7 — a game greedy lost — the search
|
||||
returns **`NoneFound { exhausted: true }` in 8,103 nodes**: no winning line
|
||||
existed in the final round. That is a real negative, not a budget cut, and
|
||||
the type keeps the two apart.
|
||||
|
||||
**The feature works. The difficulty half is where the trouble was.**
|
||||
|
||||
## 2. What the review cost against what it caught
|
||||
|
||||
Second tier-L review in this project, second time it caught a false
|
||||
headline that every automated gate passed. `make all` was green
|
||||
throughout.
|
||||
|
||||
| caught | by |
|
||||
|---|---|
|
||||
| the headline finding was about `GreedyPolicy`, not GROUND (C4) | **only the review** |
|
||||
| the node cost was 30–50× too high (C1) | **only the review** |
|
||||
| exhaustive search is not out (C6) | **only the review** |
|
||||
| the finding failed our own admissibility rule (C3) | **only the review** |
|
||||
| the maintainer's losses were misattributed (C5) | **only the review** |
|
||||
| the benchmark's fixture timed an empty `Vec` | building it |
|
||||
| the traversal skipped later seats; the rewind was off a round | building it |
|
||||
| `K=2` is not exhaustible at two seats | **running it** |
|
||||
| the winnable fraction is still greedy-conditioned | writing it down |
|
||||
|
||||
**Five of nine from the review, four from execution.** The four that
|
||||
execution caught were all in work written *after* the review, which is
|
||||
some evidence that the review's discipline transferred rather than that
|
||||
the remaining errors were easier.
|
||||
|
||||
**The reviewer under-ranked their own best challenge.** They put C4
|
||||
fourth. Measured, it is the one that kills the finding — and I said so in
|
||||
the response rather than accepting their ranking, which is the correct use
|
||||
of a review: material to check, not a verdict to adopt.
|
||||
|
||||
## 3. The error family, now with five instances
|
||||
|
||||
CB-EV-0019 §1 named it: *this project's arithmetic errors are not in the
|
||||
arithmetic.* A number computed correctly against the wrong base, span, or
|
||||
subject.
|
||||
|
||||
| # | pass | the wrong thing |
|
||||
|---|---|---|
|
||||
| 1 | CB-WP-0021 | `csv` cost against AM-4a, which never sees the code |
|
||||
| 2 | ground-game report | *"12 in the file"* — a sum with no deal table |
|
||||
| 3 | CB-WP-0018 | SOLVE's inertness attributed to the wrong condition |
|
||||
| 4 | **this pass, C1** | a timer bracketing whole games, divided by decisions |
|
||||
| 5 | **this pass, C4** | a win rate attributed to the game, not the policy |
|
||||
|
||||
**And two projections published in place of measurements**, both in this
|
||||
pass: the survey's `7.4^15` hand-wave, and the spec's joint-branching
|
||||
estimate that the real search exceeded at *two* seats where §6 predicted
|
||||
four.
|
||||
|
||||
**`facts-check` catches copies that disagree. Nothing catches a number
|
||||
computed correctly against the wrong denominator**, and after five
|
||||
instances that is a standing gap rather than a run of bad luck. Naming it
|
||||
is what this file can do; closing it is not this pass's.
|
||||
|
||||
## 4. What the difficulty measurement actually shows
|
||||
|
||||
`make difficulty`, 60 seeds:
|
||||
|
||||
| seats | winnable | greedy | random | first-legal | spread |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| 2 | 60% | 60.0% | 5.0% | 76.7% | 71.7 |
|
||||
| 3 | 93% | 88.3% | 6.7% | 25.0% | 81.7 |
|
||||
| 4 | 100% | 93.3% | 6.7% | 30.0% | 86.7 |
|
||||
| 5 | 100% | 100.0% | 3.3% | 0.0% | 100.0 |
|
||||
| 6 | 100% | 100.0% | 3.3% | 0.0% | 100.0 |
|
||||
|
||||
**The `spread` column is the finding.** 71.7 to 100.0 points between three
|
||||
trivial policies. It is the evidence for the prohibition in
|
||||
RetrospectiveAnalysis §4.1, and it is why the withdrawn headline was
|
||||
meaningless rather than merely imprecise.
|
||||
|
||||
**The 5–6 seat rows point the opposite way from the withdrawn claim.**
|
||||
`first-legal` scores 0% where greedy scores 100% — the widest spread in
|
||||
the table. Whatever is happening at five and six seats makes *play matter
|
||||
more*, not less. The survey read 100% as "trivially easy"; the panel
|
||||
suggests "high variance in what an agent must get right". **Neither is
|
||||
established**, and saying so is the honest state.
|
||||
|
||||
**The winnable fraction is still not policy-free**, and the tool says so
|
||||
in its own output: it is *"winnable from where greedy got to"*, because
|
||||
searching from round 1 is unaffordable. Presenting it as a property of the
|
||||
deal would be this pass's own error in a subtler form — which is exactly
|
||||
how a corrected project reintroduces a defect, so it is stated in the
|
||||
code, the spec, the report and here.
|
||||
|
||||
## 5. Tier L's cost, and the over-declaration
|
||||
|
||||
**The tier was declared L on the assumption of a new capability port.
|
||||
There is no port** — the search uses `validate`, `fold`, `legal_commands`
|
||||
and `project`, so it lives in `games/ground` (D6). By the structural
|
||||
trigger this was an **M**.
|
||||
|
||||
**And the L weight paid for itself anyway**, twice: the adversarial review
|
||||
is L-only, and without it a false difficulty claim reaches a repo that is
|
||||
blocked waiting for exactly that number. **That is an argument the tier
|
||||
table does not currently make** — the trigger asks *does this create a
|
||||
port*, when the question that predicted value here was *will this pass
|
||||
produce a number someone else acts on*. Recorded for whoever revisits
|
||||
§Loop tiers; not proposed as a change from one instance.
|
||||
|
||||
## 6. Chaos window 2
|
||||
|
||||
**Declaration 8 of 12.** Structural L (over-declared), d8 = 6, no
|
||||
override.
|
||||
|
||||
**Eleven declarations, no 8 rolled.** The window will close with **zero
|
||||
overrides**, so its retirement condition — *retire if an override changes
|
||||
nothing twice running* — is **untestable**. Window 1 closed with two
|
||||
overrides at d4 and both changed the outcome.
|
||||
|
||||
**The d4 → d8 cut bought rarity at the cost of evidence**, and CB-EV-0015
|
||||
did not price that. At d8 over twelve declarations the expected count is
|
||||
1.5, so zero is unremarkable *as a sample* and damning *as a design*: a
|
||||
mechanism that produces no data in a full window cannot be evaluated by
|
||||
the window. **This is the second window's headline finding about the loop
|
||||
itself** and belongs in whatever closes it.
|
||||
|
||||
## 7. Cost
|
||||
|
||||
CB-WP-0024's cost, by re-running the instrument: `make cost`. Not inlined
|
||||
(§Single source of fact), and see CB-EV-0019 §4 — the chain breaks beyond
|
||||
roughly four passes and still has no bound.
|
||||
|
||||
## Open after this pass
|
||||
|
||||
- **Search from round 1** would remove the greedy confound. Needs
|
||||
transposition or move-ordering; neither built.
|
||||
- **Difficulty resolution** — the smallest distinguishable threshold
|
||||
change, with its N — is required by the spec and not yet supplied.
|
||||
- **The witness is not wired to the ending page.** T05 built the search;
|
||||
the browser cannot yet ask it. That is the maintainer-facing half of
|
||||
remark 2 and it is not done.
|
||||
- **The wrong-denominator family has no control** (§3), at five instances.
|
||||
- **The chaos window's retirement condition is untestable** (§6).
|
||||
|
|
@ -27,5 +27,10 @@ serde_json.workspace = true
|
|||
name = "synthetic"
|
||||
harness = false
|
||||
|
||||
# CB-WP-0025 T04 / ADR-0013 D7: the node cost the spec quotes.
|
||||
[[bench]]
|
||||
name = "search"
|
||||
harness = false
|
||||
|
||||
[lints]
|
||||
workspace = true
|
||||
|
|
|
|||
128
games/ground/benches/search.rs
Normal file
128
games/ground/benches/search.rs
Normal file
|
|
@ -0,0 +1,128 @@
|
|||
//! CB-WP-0025 T04 — what one search node actually costs.
|
||||
//!
|
||||
//! **ADR-0013 D7 exists because two measurements disagreed by 5×.** The
|
||||
//! survey published 112–161 µs/node from a timer that bracketed two
|
||||
//! `setup`s and a whole greedy game (C1). The author's re-measurement said
|
||||
//! 3.0–4.1 µs with `Instant::now()` around each call; the adversarial
|
||||
//! reviewer's isolation said 15.6–20.4 µs. Both agreed the published
|
||||
//! figure was wrong by 1–2 orders and neither established which
|
||||
//! replacement was right.
|
||||
//!
|
||||
//! So the spec quotes **this** and nothing else. `criterion` handles the
|
||||
//! things hand-rolled timing gets wrong here: per-call clock overhead
|
||||
//! against a ~microsecond subject, warm-up, and run-to-run variance —
|
||||
//! which is what let the survey's figure move 161 → 112 between two runs
|
||||
//! of the same unmodified binary.
|
||||
//!
|
||||
//! Two subjects, because a search node is not one call:
|
||||
//!
|
||||
//! * `legal_commands` — enumerating a seat's options;
|
||||
//! * `validate + fold` — taking one branch, which any search does per
|
||||
//! child and which the survey never separated out.
|
||||
|
||||
use cb_game_runtime::{ScenarioGame, Setup};
|
||||
use cb_kernel::{Actor, Aggregate, PlayerId};
|
||||
use criterion::{criterion_group, criterion_main, BatchSize, Criterion};
|
||||
use games_ground::bot::{legal_commands, play, GreedyPolicy, Policy};
|
||||
use games_ground::GroundState;
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
fn setup(players: u8, seed: u64) -> GroundState {
|
||||
GroundState::setup(
|
||||
&Setup {
|
||||
players,
|
||||
preset: format!("standard-{players}p"),
|
||||
patch: BTreeMap::new(),
|
||||
},
|
||||
seed,
|
||||
)
|
||||
.expect("preset")
|
||||
}
|
||||
|
||||
/// A **mid-game state at a real decision point** for `seat`.
|
||||
///
|
||||
/// Not a fresh deal: at deal time most branches do not exist yet, and a
|
||||
/// node cost taken there would flatter any search proposal.
|
||||
///
|
||||
/// **And not a fixed step count either.** The first version stopped at
|
||||
/// step 20 for every seat count, which put 2p and 4p in a state where
|
||||
/// seat 0 had *no* legal commands at all — so the benchmark reported
|
||||
/// ~120 ns (the cost of returning an empty `Vec`) and silently skipped
|
||||
/// `validate_fold` because there was nothing to validate. A fixture that
|
||||
/// measures the empty case and calls it a node cost is the same defect
|
||||
/// class this whole pass exists to correct, one layer down.
|
||||
///
|
||||
/// So: advance until the seat genuinely has a choice, and assert it.
|
||||
fn midgame(players: u8, seed: u64, seat: PlayerId) -> GroundState {
|
||||
let mut ps: Vec<Box<dyn Policy>> = (0..players)
|
||||
.map(|_| Box::new(GreedyPolicy) as Box<dyn Policy>)
|
||||
.collect();
|
||||
let game = play(setup(players, seed), &mut ps).expect("a complete game");
|
||||
let mut state = setup(players, seed);
|
||||
let mut best: Option<GroundState> = None;
|
||||
for (i, (actor, cmd)) in game.steps.iter().enumerate() {
|
||||
// Past the opening, take the first state where the seat has a real
|
||||
// branch. `> 1` rather than `> 0`: a forced move is not a node.
|
||||
if i >= 8 && legal_commands(&state, seat).len() > 1 {
|
||||
best = Some(state.clone());
|
||||
break;
|
||||
}
|
||||
if let Ok(events) = state.validate(*actor, cmd) {
|
||||
for e in &events {
|
||||
state.fold(e);
|
||||
}
|
||||
}
|
||||
}
|
||||
let state = best.expect("a mid-game state where the seat has a choice");
|
||||
assert!(
|
||||
legal_commands(&state, seat).len() > 1,
|
||||
"benchmark fixture has no branch to measure — it would time the empty case"
|
||||
);
|
||||
state
|
||||
}
|
||||
|
||||
fn bench(c: &mut Criterion) {
|
||||
for players in [2u8, 3, 4] {
|
||||
let seat = PlayerId(0);
|
||||
let state = midgame(players, 7, seat);
|
||||
let width = legal_commands(&state, seat).len();
|
||||
println!(" fixture {players}p: {width} legal commands at the measured node");
|
||||
|
||||
c.bench_function(&format!("legal_commands/{players}p"), |b| {
|
||||
b.iter(|| std::hint::black_box(legal_commands(&state, seat)))
|
||||
});
|
||||
|
||||
// A search must COPY the state per branch (or undo, which we do
|
||||
// not have). `iter_batched` excludes setup from the timing, so
|
||||
// without this the budget would rest on an unmeasured span —
|
||||
// which is the exact mistake C1 caught in the survey.
|
||||
c.bench_function(&format!("clone/{players}p"), |b| {
|
||||
b.iter(|| std::hint::black_box(state.clone()))
|
||||
});
|
||||
|
||||
// One branch taken: what a search pays per CHILD, on top of
|
||||
// enumeration. The survey folded this into "us/node" without
|
||||
// separating it, and a search's real cost is enumeration once plus
|
||||
// this per child.
|
||||
let legal = legal_commands(&state, seat);
|
||||
if let Some(cmd) = legal.first() {
|
||||
c.bench_function(&format!("validate_fold/{players}p"), |b| {
|
||||
b.iter_batched(
|
||||
|| state.clone(),
|
||||
|mut s| {
|
||||
if let Ok(events) = s.validate(Actor::Player(seat), cmd) {
|
||||
for e in &events {
|
||||
s.fold(e);
|
||||
}
|
||||
}
|
||||
std::hint::black_box(s)
|
||||
},
|
||||
BatchSize::SmallInput,
|
||||
)
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
criterion_group!(benches, bench);
|
||||
criterion_main!(benches);
|
||||
|
|
@ -1,5 +1,21 @@
|
|||
//! CB-RES-0008's runnable baseline (CB-WP-0025 T01).
|
||||
//!
|
||||
//! ── SUPERSEDED 2026-08-05 by `difficulty.rs` (CB-WP-0025 T06) ──────────
|
||||
//!
|
||||
//! Kept as the survey's dated snapshot, NOT as a live instrument, for the
|
||||
//! same reason `tools/design-baseline.py` was kept: it is the evidence for
|
||||
//! how a wrong number got into a survey.
|
||||
//!
|
||||
//! **Its headline was withdrawn.** It reported `GreedyPolicy` winning
|
||||
//! 200/200 at five and six seats and the survey called the game too easy
|
||||
//! there. A `FirstLegal` policy scores **0%** on the same deals — the
|
||||
//! measurement was about the policy. See the challenge (C4) and the
|
||||
//! response.
|
||||
//!
|
||||
//! **And it is inadmissible under GameDesign §1.3**: no assertions, no
|
||||
//! `--self-test`, no `make` target, so nothing can turn it red.
|
||||
//! `difficulty.rs` carries all three.
|
||||
//!
|
||||
//! Two numbers the survey needs and cannot cite from anyone else, because
|
||||
//! they are about **our** game on **our** machine:
|
||||
//!
|
||||
|
|
|
|||
304
games/ground/examples/difficulty.rs
Normal file
304
games/ground/examples/difficulty.rs
Normal file
|
|
@ -0,0 +1,304 @@
|
|||
//! CB-WP-0025 T06 — the difficulty table, and the instrument that can fail.
|
||||
//!
|
||||
//! Implements `specs/RetrospectiveAnalysis.md` §4. Replaces
|
||||
//! `difficulty-baseline.rs`, which CB-WP-0022's admissibility rule and
|
||||
//! this pass's own review both found inadmissible: it had no assertions,
|
||||
//! no `--self-test` and no `make` target, so nothing could turn it red.
|
||||
//!
|
||||
//! ## What it will not print
|
||||
//!
|
||||
//! **A single policy's win rate as a difficulty** (§4.1). Measured on
|
||||
//! identical deals, `GreedyPolicy` wins 100% at five and six seats where
|
||||
//! `FirstLegal` wins 0%. The panel is plural for that reason, and the
|
||||
//! spread is reported rather than hidden.
|
||||
//!
|
||||
//! ```text
|
||||
//! cargo run --release -p games-ground --example difficulty [--self-test]
|
||||
//! ```
|
||||
|
||||
use cb_game_runtime::{ScenarioGame, Setup};
|
||||
use cb_kernel::{Aggregate, PlayerId};
|
||||
use games_ground::bot::{play, Choice, GreedyPolicy, Policy, RandomPolicy};
|
||||
use games_ground::search::{winnable_within, Verdict};
|
||||
use games_ground::{GroundCommand, GroundState};
|
||||
|
||||
/// Seeds per cell. Small by default: the winnable fraction needs an
|
||||
/// exhaustive search per deal, which is the expensive half (§3.1).
|
||||
const SEEDS: u64 = 60;
|
||||
/// §3's measured limit: `K=1` is exhaustible, `K=2` is not.
|
||||
const K: u8 = 1;
|
||||
const BUDGET: usize = 2_000_000;
|
||||
|
||||
/// A policy with no heuristic at all. **It is in the panel because it is
|
||||
/// what falsified the survey's headline** — it beats greedy at two seats
|
||||
/// and scores zero at six.
|
||||
struct FirstLegal;
|
||||
impl Policy for FirstLegal {
|
||||
fn name(&self) -> &'static str {
|
||||
"first-legal"
|
||||
}
|
||||
fn choose(
|
||||
&mut self,
|
||||
_s: &GroundState,
|
||||
_seat: PlayerId,
|
||||
_legal: &[GroundCommand],
|
||||
_may_pass: bool,
|
||||
) -> Choice {
|
||||
Choice::Command(0)
|
||||
}
|
||||
}
|
||||
|
||||
fn setup(players: u8, seed: u64) -> Option<GroundState> {
|
||||
GroundState::setup(
|
||||
&Setup {
|
||||
players,
|
||||
preset: format!("standard-{players}p"),
|
||||
patch: Default::default(),
|
||||
},
|
||||
seed,
|
||||
)
|
||||
.ok()
|
||||
}
|
||||
|
||||
fn policies(kind: &str, players: u8, seed: u64) -> Vec<Box<dyn Policy>> {
|
||||
(0..players)
|
||||
.map(|i| -> Box<dyn Policy> {
|
||||
match kind {
|
||||
"random" => Box::new(RandomPolicy::new(seed ^ u64::from(i))),
|
||||
"first-legal" => Box::new(FirstLegal),
|
||||
_ => Box::new(GreedyPolicy),
|
||||
}
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Games a named policy actually wins.
|
||||
fn policy_wins(kind: &str, players: u8) -> (u32, u32) {
|
||||
let (mut wins, mut played) = (0, 0);
|
||||
for seed in 0..SEEDS {
|
||||
let Some(state) = setup(players, seed) else {
|
||||
continue;
|
||||
};
|
||||
let mut ps = policies(kind, players, seed);
|
||||
let Ok(game) = play(state, &mut ps) else {
|
||||
continue;
|
||||
};
|
||||
if let Some(o) = &game.state.outcome {
|
||||
played += 1;
|
||||
if o.group_success {
|
||||
wins += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
(wins, played)
|
||||
}
|
||||
|
||||
/// Rewind a played game to the start of its last `k` rounds.
|
||||
fn last_rounds(players: u8, seed: u64, k: usize) -> Option<GroundState> {
|
||||
let mut ps = policies("greedy", players, seed);
|
||||
let game = play(setup(players, seed)?, &mut ps).ok()?;
|
||||
let total = game
|
||||
.steps
|
||||
.iter()
|
||||
.filter(|(_, c)| matches!(c, GroundCommand::EndRound))
|
||||
.count();
|
||||
let mut st = setup(players, seed)?;
|
||||
let mut ends = 0usize;
|
||||
for (a, c) in &game.steps {
|
||||
if let Ok(ev) = st.validate(*a, c) {
|
||||
for e in &ev {
|
||||
st.fold(e);
|
||||
}
|
||||
}
|
||||
if matches!(c, GroundCommand::EndRound) {
|
||||
ends += 1;
|
||||
if ends >= total.saturating_sub(k) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
Some(st)
|
||||
}
|
||||
|
||||
/// In what fraction of deals does a winning line exist in the last `K`
|
||||
/// rounds?
|
||||
///
|
||||
/// **Less policy-dependent than a win rate, but NOT policy-free, and
|
||||
/// saying otherwise would repeat this pass's own error in a subtler
|
||||
/// form.** The position searched is the one `GreedyPolicy` left at the
|
||||
/// start of the last round, so the figure is *"winnable from where greedy
|
||||
/// got to"*. A genuinely policy-free measure would search from round 1,
|
||||
/// which §3.1 measured as unaffordable.
|
||||
///
|
||||
/// What it does buy: the last round's outcome no longer depends on which
|
||||
/// agent plays it, so the measure is insensitive to exactly the variation
|
||||
/// that made the bot rate meaningless (§4.1).
|
||||
///
|
||||
/// Deals where the search was cut by its budget are **not counted either
|
||||
/// way** — they are reported separately, because folding "we stopped
|
||||
/// looking" into "not winnable" is exactly the collapse §2.3 forbids.
|
||||
fn winnable_fraction(players: u8) -> (u32, u32, u32) {
|
||||
let (mut yes, mut decided, mut undecided) = (0, 0, 0);
|
||||
for seed in 0..SEEDS {
|
||||
let Some(state) = last_rounds(players, seed, K as usize) else {
|
||||
continue;
|
||||
};
|
||||
match winnable_within(&state, K, BUDGET) {
|
||||
Verdict::Winnable { .. } => {
|
||||
yes += 1;
|
||||
decided += 1;
|
||||
}
|
||||
Verdict::NoneFound {
|
||||
exhausted: true, ..
|
||||
} => decided += 1,
|
||||
Verdict::NoneFound {
|
||||
exhausted: false, ..
|
||||
} => undecided += 1,
|
||||
}
|
||||
}
|
||||
(yes, decided, undecided)
|
||||
}
|
||||
|
||||
fn report() {
|
||||
println!("difficulty — specs/RetrospectiveAnalysis.md §4\n");
|
||||
println!(
|
||||
" winnable-from-round-{} over {SEEDS} seeds, budget {BUDGET} nodes",
|
||||
6 - K
|
||||
);
|
||||
println!(" policy win rates over the same {SEEDS} seeds\n");
|
||||
println!(" seats winnable greedy random first-legal spread undecided");
|
||||
|
||||
for players in [2u8, 3, 4, 5, 6] {
|
||||
let (yes, decided, undecided) = winnable_fraction(players);
|
||||
let pct = |(w, n): (u32, u32)| {
|
||||
if n == 0 {
|
||||
-1.0
|
||||
} else {
|
||||
100.0 * f64::from(w) / f64::from(n)
|
||||
}
|
||||
};
|
||||
let g = pct(policy_wins("greedy", players));
|
||||
let r = pct(policy_wins("random", players));
|
||||
let f = pct(policy_wins("first-legal", players));
|
||||
let spread = [g, r, f].iter().cloned().fold(f64::MIN, f64::max)
|
||||
- [g, r, f].iter().cloned().fold(f64::MAX, f64::min);
|
||||
let wf = if decided == 0 {
|
||||
"n/a".to_string()
|
||||
} else {
|
||||
format!("{:.0}%", 100.0 * f64::from(yes) / f64::from(decided))
|
||||
};
|
||||
println!(
|
||||
" {players}p {wf:>6} {g:>5.1}% {r:>5.1}% {f:>5.1}% \
|
||||
{spread:>5.1} {undecided:>3}"
|
||||
);
|
||||
}
|
||||
|
||||
println!(
|
||||
"\n WINNABLE is conditioned on GREEDY's play up to the last round —\n \
|
||||
it is 'winnable from where greedy got to', not a property of the\n \
|
||||
deal alone. Searching from round 1 is unaffordable (spec §3.1).\n\n \
|
||||
It is also a LOWER BOUND: a K={K} search cannot see a line that\n \
|
||||
needed an earlier round. `undecided` are deals whose search hit the\n \
|
||||
node budget — they are excluded from the fraction, not counted as\n \
|
||||
unwinnable.\n\n \
|
||||
SPREAD is the range across three policies. Where it is large, no\n \
|
||||
single policy's rate says anything about the game (§4.1)."
|
||||
);
|
||||
}
|
||||
|
||||
fn self_test() -> i32 {
|
||||
let mut ok = true;
|
||||
let mut check = |name: &str, cond: bool, detail: String| {
|
||||
ok &= cond;
|
||||
println!(
|
||||
" [{}] {name}{}",
|
||||
if cond { "ok " } else { "FAIL" },
|
||||
if detail.is_empty() {
|
||||
String::new()
|
||||
} else {
|
||||
format!(" — {detail}")
|
||||
}
|
||||
);
|
||||
};
|
||||
|
||||
// The control the old harness never had: a search that finds a line
|
||||
// must produce one that REPLAYS. §2.1 is a gate, not a metric.
|
||||
let state = last_rounds(3, 7, K as usize).expect("a 3p game");
|
||||
match winnable_within(&state, K, BUDGET) {
|
||||
Verdict::Winnable { line, .. } => {
|
||||
let mut replay = state.clone();
|
||||
let mut good = true;
|
||||
for m in &line {
|
||||
match replay.validate(m.actor, &m.command) {
|
||||
Ok(ev) => {
|
||||
for e in &ev {
|
||||
replay.fold(e);
|
||||
}
|
||||
}
|
||||
Err(_) => good = false,
|
||||
}
|
||||
}
|
||||
let won = replay.outcome.as_ref().is_some_and(|o| o.group_success);
|
||||
check(
|
||||
"a witness replays to a win",
|
||||
good && won,
|
||||
format!("{} moves", line.len()),
|
||||
);
|
||||
}
|
||||
Verdict::NoneFound { .. } => check(
|
||||
"a witness replays to a win",
|
||||
false,
|
||||
"3p seed 7 found no line — the fixture moved".into(),
|
||||
),
|
||||
}
|
||||
|
||||
// The negative: a search must be able to return nothing, exhaustively.
|
||||
let lost = last_rounds(2, 7, 1).expect("a 2p game");
|
||||
match winnable_within(&lost, 1, BUDGET) {
|
||||
Verdict::NoneFound { exhausted, nodes } => check(
|
||||
"an unwinnable position is reported as searched-out",
|
||||
exhausted && nodes > 100,
|
||||
format!("{nodes} nodes"),
|
||||
),
|
||||
Verdict::Winnable { .. } => check(
|
||||
"an unwinnable position is reported as searched-out",
|
||||
false,
|
||||
"found a win in a game 2p seed 7 lost".into(),
|
||||
),
|
||||
}
|
||||
|
||||
// A budget of one must NOT claim exhaustion — the distinction §2.3
|
||||
// rests on.
|
||||
match winnable_within(&state, K, 1) {
|
||||
Verdict::NoneFound { exhausted, .. } => check(
|
||||
"a budget cut is not reported as exhaustion",
|
||||
!exhausted,
|
||||
String::new(),
|
||||
),
|
||||
Verdict::Winnable { .. } => check(
|
||||
"a budget cut is not reported as exhaustion",
|
||||
false,
|
||||
String::new(),
|
||||
),
|
||||
}
|
||||
|
||||
// §4.1's reason, asserted rather than asserted-about: the panel must
|
||||
// actually disagree, or reporting three policies is ceremony.
|
||||
let g = policy_wins("greedy", 6);
|
||||
let f = policy_wins("first-legal", 6);
|
||||
check(
|
||||
"the policy panel disagrees, so no single rate is a difficulty",
|
||||
g.0 != f.0,
|
||||
format!("greedy {}/{}, first-legal {}/{}", g.0, g.1, f.0, f.1),
|
||||
);
|
||||
|
||||
println!("difficulty self-test (positive control)");
|
||||
i32::from(!ok)
|
||||
}
|
||||
|
||||
fn main() {
|
||||
if std::env::args().any(|a| a == "--self-test") {
|
||||
std::process::exit(self_test());
|
||||
}
|
||||
report();
|
||||
}
|
||||
|
|
@ -20,6 +20,12 @@ pub mod view;
|
|||
#[cfg(feature = "scenarios")]
|
||||
pub mod record;
|
||||
|
||||
/// *Was this deal winnable?* — the retrospective search (CB-WP-0025 T05,
|
||||
/// ADR-0013). Uses only `validate`/`fold`/`legal_commands`, so it lives
|
||||
/// beside the aggregate rather than in a crate that would re-export it
|
||||
/// (ADR-0013 D6).
|
||||
pub mod search;
|
||||
|
||||
#[cfg(feature = "scenarios")]
|
||||
use cb_game_runtime::{parse_actor, CommandStep, ScenarioGame, Setup};
|
||||
use cb_kernel::{Actor, Aggregate, ChaChaRng, KernelRng, PlayerId, Rejection, Seed};
|
||||
|
|
|
|||
412
games/ground/src/search.rs
Normal file
412
games/ground/src/search.rs
Normal file
|
|
@ -0,0 +1,412 @@
|
|||
//! Was this deal winnable — and here is one line (CB-WP-0025 T05).
|
||||
//!
|
||||
//! Implements [`specs/RetrospectiveAnalysis.md`]. The question is
|
||||
//! **retrospective**: given the deal as it actually was, does a line of
|
||||
//! play exist that reaches the threshold?
|
||||
//!
|
||||
//! ## Why this is allowed to see everything
|
||||
//!
|
||||
//! Strategy fusion — the classic objection to searching an
|
||||
//! imperfect-information game — is a defect of *aggregating over
|
||||
//! determinizations to choose a move*. **After the game there is one
|
||||
//! world.** The deal is known, so a line found in it is executable in the
|
||||
//! only world there is (ADR-0013 D1).
|
||||
//!
|
||||
//! What survives the objection is that the line may not have been
|
||||
//! *findable* at the time, and that is answered per move by
|
||||
//! [`Move::visible`] rather than by refusing to search.
|
||||
//!
|
||||
//! ## The bound
|
||||
//!
|
||||
//! Exhaustive over the last `K` rounds, with a node budget as a secondary
|
||||
//! cut. When nothing is found the caller must say **"no winning line
|
||||
//! found in the last K rounds"** — never "unwinnable", which a bounded
|
||||
//! search cannot establish (spec §2.3).
|
||||
|
||||
use crate::bot::legal_commands;
|
||||
use crate::{GroundCommand, GroundState, ProblemState};
|
||||
use cb_kernel::{Actor, Aggregate, PlayerId};
|
||||
|
||||
/// One move of a witness, with whether the seat could have chosen it
|
||||
/// knowing only what it could see.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct Move {
|
||||
pub actor: Actor,
|
||||
pub command: GroundCommand,
|
||||
/// `false` when the move depends on something the acting seat could
|
||||
/// not see — spec §2.2. Concretely: it targets a Problem that was
|
||||
/// **face down** to that seat, so choosing it required knowing what
|
||||
/// was under it.
|
||||
///
|
||||
/// System moves are always `true`: the table does them, not a player.
|
||||
pub visible: bool,
|
||||
}
|
||||
|
||||
/// What the search found.
|
||||
#[derive(Debug, Clone)]
|
||||
pub enum Verdict {
|
||||
/// A line exists. `nodes` is what it cost to find.
|
||||
Winnable { line: Vec<Move>, nodes: usize },
|
||||
/// Nothing found **within the bound**. This is not "unwinnable".
|
||||
NoneFound {
|
||||
nodes: usize,
|
||||
/// `true` if the space was searched to exhaustion; `false` if the
|
||||
/// node budget cut it short. The distinction is the difference
|
||||
/// between "no line exists in these K rounds" and "we stopped
|
||||
/// looking", and callers must not collapse it.
|
||||
exhausted: bool,
|
||||
},
|
||||
}
|
||||
|
||||
/// Would this command have been choosable knowing only what `seat` saw?
|
||||
///
|
||||
/// A `SelectAction` naming a Problem that is face-down to that seat is
|
||||
/// `hidden`: picking it required knowing what was underneath. Everything
|
||||
/// else is `visible` — a seat's own hand is in its own projection, and
|
||||
/// since GR-P05 (CB-WP-0023) SOLVE is only offered on face-up Problems
|
||||
/// anyway, so INVESTIGATE is where hidden information actually bites.
|
||||
fn is_visible(state: &GroundState, seat: PlayerId, cmd: &GroundCommand) -> bool {
|
||||
let _ = seat;
|
||||
match cmd {
|
||||
GroundCommand::SelectAction {
|
||||
problem: Some(p), ..
|
||||
} => matches!(
|
||||
state.problems.get(p),
|
||||
Some(ProblemState { face_up: true, .. })
|
||||
),
|
||||
_ => true,
|
||||
}
|
||||
}
|
||||
|
||||
struct Search {
|
||||
nodes: usize,
|
||||
budget: usize,
|
||||
/// Set when the budget stopped us, so `NoneFound` can distinguish
|
||||
/// "searched it all" from "gave up".
|
||||
cut: bool,
|
||||
}
|
||||
|
||||
impl Search {
|
||||
/// Apply a command to a copy. `None` if the aggregate rejects it —
|
||||
/// which is not an error here: the search offers candidates and
|
||||
/// `validate` is the authority, exactly as `legal_commands` does.
|
||||
fn step(
|
||||
&mut self,
|
||||
state: &GroundState,
|
||||
actor: Actor,
|
||||
cmd: &GroundCommand,
|
||||
) -> Option<GroundState> {
|
||||
self.nodes += 1;
|
||||
let mut next = state.clone();
|
||||
let events = next.validate(actor, cmd).ok()?;
|
||||
for e in &events {
|
||||
next.fold(e);
|
||||
}
|
||||
Some(next)
|
||||
}
|
||||
|
||||
/// One player branch: apply, recurse, and prepend the move if the
|
||||
/// subtree won.
|
||||
fn branch(
|
||||
&mut self,
|
||||
state: &GroundState,
|
||||
seat: PlayerId,
|
||||
cmd: &GroundCommand,
|
||||
rounds_left: u8,
|
||||
) -> Option<Vec<Move>> {
|
||||
let next = self.step(state, Actor::Player(seat), cmd)?;
|
||||
let mut rest = self.go(&next, rounds_left)?;
|
||||
let mut line = vec![Move {
|
||||
actor: Actor::Player(seat),
|
||||
command: cmd.clone(),
|
||||
visible: is_visible(state, seat, cmd),
|
||||
}];
|
||||
line.append(&mut rest);
|
||||
Some(line)
|
||||
}
|
||||
|
||||
/// Depth-first over whatever must happen next, mirroring the driver's
|
||||
/// round structure (`bot::play_journaled`).
|
||||
///
|
||||
/// Returns the moves appended after `state`, or `None`.
|
||||
fn go(&mut self, state: &GroundState, rounds_left: u8) -> Option<Vec<Move>> {
|
||||
if let Some(outcome) = &state.outcome {
|
||||
return outcome.group_success.then(Vec::new);
|
||||
}
|
||||
if rounds_left == 0 {
|
||||
return None;
|
||||
}
|
||||
if self.nodes >= self.budget {
|
||||
self.cut = true;
|
||||
return None;
|
||||
}
|
||||
|
||||
let seats: Vec<PlayerId> = state.players.keys().copied().collect();
|
||||
|
||||
// **Obligatory first.** GR-R02: a seat with no selection this
|
||||
// round must make one, and nothing else can happen until it does.
|
||||
// If every branch fails, the line is dead — falling through would
|
||||
// try system commands the aggregate is going to reject anyway.
|
||||
//
|
||||
// The first version branched on "the first seat that has any legal
|
||||
// command" and `break`ed when its branches were spent, which threw
|
||||
// away every later seat's options: seat 1 never acted if seat 0
|
||||
// was already selected but still had a legal move.
|
||||
if let Some(seat) = seats.iter().find(|s| !state.selections.contains_key(s)) {
|
||||
for cmd in &legal_commands(state, *seat) {
|
||||
if let Some(line) = self.branch(state, *seat, cmd, rounds_left) {
|
||||
return Some(line);
|
||||
}
|
||||
if self.cut {
|
||||
return None;
|
||||
}
|
||||
}
|
||||
return None;
|
||||
}
|
||||
|
||||
// **Optional next.** After Reveal a seat may choose a GROUND mode,
|
||||
// answer a Support, or name a DARVO target. Some of those are
|
||||
// obligatory, but the aggregate enforces that by rejecting
|
||||
// `Resolve` until they are done — so this needs no phase logic of
|
||||
// its own, and the do-nothing case is simply the fall-through
|
||||
// below.
|
||||
for seat in &seats {
|
||||
for cmd in &legal_commands(state, *seat) {
|
||||
if let Some(line) = self.branch(state, *seat, cmd, rounds_left) {
|
||||
return Some(line);
|
||||
}
|
||||
if self.cut {
|
||||
return None;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Nobody need act: the table advances. Try each system command; the
|
||||
// aggregate rejects the ones that are out of order, so this needs
|
||||
// no phase logic of its own.
|
||||
for sys in [
|
||||
GroundCommand::Reveal,
|
||||
GroundCommand::Resolve,
|
||||
GroundCommand::EndRound,
|
||||
] {
|
||||
let Some(next) = self.step(state, Actor::System, &sys) else {
|
||||
continue;
|
||||
};
|
||||
let spent = u8::from(matches!(sys, GroundCommand::EndRound));
|
||||
if let Some(mut rest) = self.go(&next, rounds_left - spent) {
|
||||
let mut line = vec![Move {
|
||||
actor: Actor::System,
|
||||
command: sys,
|
||||
visible: true,
|
||||
}];
|
||||
line.append(&mut rest);
|
||||
return Some(line);
|
||||
}
|
||||
if self.cut {
|
||||
return None;
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
/// Search the last `rounds` rounds from `state` for a line reaching
|
||||
/// `group_success`.
|
||||
///
|
||||
/// **`state` must be a real position from the game being asked about.**
|
||||
/// The caller supplies it; this does not re-deal, because a re-dealt game
|
||||
/// is a different question.
|
||||
pub fn winnable_within(state: &GroundState, rounds: u8, budget: usize) -> Verdict {
|
||||
let mut s = Search {
|
||||
nodes: 0,
|
||||
budget,
|
||||
cut: false,
|
||||
};
|
||||
match s.go(state, rounds) {
|
||||
Some(line) => Verdict::Winnable {
|
||||
line,
|
||||
nodes: s.nodes,
|
||||
},
|
||||
None => Verdict::NoneFound {
|
||||
nodes: s.nodes,
|
||||
exhausted: !s.cut,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
/// How many moves of a witness required unseen information.
|
||||
pub fn hidden_moves(line: &[Move]) -> usize {
|
||||
line.iter().filter(|m| !m.visible).count()
|
||||
}
|
||||
|
||||
#[cfg(all(test, feature = "scenarios"))]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use cb_game_runtime::{ScenarioGame, Setup};
|
||||
|
||||
fn setup(players: u8, seed: u64) -> GroundState {
|
||||
GroundState::setup(
|
||||
&Setup {
|
||||
players,
|
||||
preset: format!("standard-{players}p"),
|
||||
patch: Default::default(),
|
||||
},
|
||||
seed,
|
||||
)
|
||||
.expect("preset")
|
||||
}
|
||||
|
||||
/// **The hard gate (spec §2.1): a witness must replay.**
|
||||
///
|
||||
/// Re-execute the emitted line from the same start state through
|
||||
/// `validate`/`fold` — the same path the scenario runner takes — and
|
||||
/// require it to end in `group_success`. A witness that does not
|
||||
/// replay asserts the opposite of the truth to a player who just
|
||||
/// lost.
|
||||
/// Rewind a real game to the start of its last `k` rounds.
|
||||
///
|
||||
/// Stops **after** applying the EndRound numbered `total - k`. An
|
||||
/// earlier version broke *before* it, which left that round's own play
|
||||
/// applied and searched one round less than it claimed.
|
||||
fn last_rounds(players: u8, seed: u64, k: usize) -> GroundState {
|
||||
let mut ps: Vec<Box<dyn crate::bot::Policy>> = (0..players)
|
||||
.map(|_| Box::new(crate::bot::GreedyPolicy) as Box<dyn crate::bot::Policy>)
|
||||
.collect();
|
||||
let game = crate::bot::play(setup(players, seed), &mut ps).expect("a complete game");
|
||||
let total = game
|
||||
.steps
|
||||
.iter()
|
||||
.filter(|(_, c)| matches!(c, GroundCommand::EndRound))
|
||||
.count();
|
||||
let mut st = setup(players, seed);
|
||||
let mut ends = 0usize;
|
||||
for (a, c) in &game.steps {
|
||||
if let Ok(ev) = st.validate(*a, c) {
|
||||
for e in &ev {
|
||||
st.fold(e);
|
||||
}
|
||||
}
|
||||
if matches!(c, GroundCommand::EndRound) {
|
||||
ends += 1;
|
||||
if ends >= total.saturating_sub(k) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
st
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_witness_replays_to_a_win() {
|
||||
let state = last_rounds(3, 7, 2);
|
||||
let Verdict::Winnable { line, .. } = winnable_within(&state, 2, 200_000) else {
|
||||
panic!("3p seed 7 is winnable in its last two rounds — greedy actually won it");
|
||||
};
|
||||
let mut replay = state.clone();
|
||||
for m in &line {
|
||||
let events = replay
|
||||
.validate(m.actor, &m.command)
|
||||
.unwrap_or_else(|e| panic!("witness move rejected on replay: {e:?}"));
|
||||
for e in &events {
|
||||
replay.fold(e);
|
||||
}
|
||||
}
|
||||
let outcome = replay.outcome.as_ref().expect("the replay must finish");
|
||||
assert!(
|
||||
outcome.group_success,
|
||||
"the witness replayed but did not win: {} of {}",
|
||||
outcome.total, outcome.threshold
|
||||
);
|
||||
}
|
||||
|
||||
/// The negative control. Without it, a search that returns
|
||||
/// `NoneFound` for everything would pass the test above by never
|
||||
/// producing a witness to check.
|
||||
#[test]
|
||||
fn a_budget_of_nothing_reports_a_cut_not_a_verdict() {
|
||||
let state = last_rounds(3, 7, 2);
|
||||
match winnable_within(&state, 2, 1) {
|
||||
Verdict::NoneFound { exhausted, .. } => assert!(
|
||||
!exhausted,
|
||||
"a search stopped by its budget must not claim it searched exhaustively — \
|
||||
that is the difference between `no line exists` and `we stopped looking`"
|
||||
),
|
||||
Verdict::Winnable { .. } => panic!("one node cannot find a whole line"),
|
||||
}
|
||||
}
|
||||
|
||||
/// A position with no rounds left cannot be won, and the search must
|
||||
/// say so **without** claiming exhaustion of a space it never entered.
|
||||
#[test]
|
||||
fn no_rounds_left_finds_nothing() {
|
||||
let state = last_rounds(3, 7, 2);
|
||||
match winnable_within(&state, 0, 100) {
|
||||
Verdict::NoneFound { nodes, exhausted } => {
|
||||
assert_eq!(nodes, 0, "a zero-round search must not expand anything");
|
||||
assert!(exhausted, "it searched its (empty) space to exhaustion");
|
||||
}
|
||||
Verdict::Winnable { .. } => panic!("no rounds left cannot win"),
|
||||
}
|
||||
}
|
||||
|
||||
/// **A position that cannot be won returns none, exhaustively** — the
|
||||
/// control without which "winnable" is unfalsifiable.
|
||||
///
|
||||
/// The construction: 2p seed 7, searched over its **last round only**.
|
||||
/// Greedy lost that game, and one round is a small enough space to
|
||||
/// search to exhaustion (~8k nodes), so this is a real negative rather
|
||||
/// than a budget cut wearing a verdict's clothes.
|
||||
#[test]
|
||||
fn a_position_that_cannot_be_won_says_so_and_means_it() {
|
||||
let state = last_rounds(2, 7, 1);
|
||||
match winnable_within(&state, 1, 500_000) {
|
||||
Verdict::NoneFound { exhausted, nodes } => {
|
||||
assert!(
|
||||
exhausted,
|
||||
"the space must be searched out, or this proves nothing ({nodes} nodes)"
|
||||
);
|
||||
assert!(
|
||||
nodes > 100,
|
||||
"suspiciously few nodes for a real search: {nodes}"
|
||||
);
|
||||
}
|
||||
Verdict::Winnable { line, .. } => {
|
||||
panic!("found a {}-move win in a game 2p seed 7 lost", line.len())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The `visible` marking must be able to say NO, or it is decoration.
|
||||
/// INVESTIGATE names a face-down Problem — that is the case where a
|
||||
/// seat could not have known what it was choosing.
|
||||
#[test]
|
||||
fn a_move_onto_a_face_down_problem_is_marked_hidden() {
|
||||
let state = setup(3, 7);
|
||||
let hidden_key = state
|
||||
.problems
|
||||
.iter()
|
||||
.find(|(_, p)| !p.face_up)
|
||||
.map(|(k, _)| *k)
|
||||
.expect("a fresh deal has face-down Problems");
|
||||
let face_up_key = state
|
||||
.problems
|
||||
.iter()
|
||||
.find(|(_, p)| p.face_up)
|
||||
.map(|(k, _)| *k)
|
||||
.expect("a fresh deal has the Surface Problem face up");
|
||||
|
||||
let onto = |p: u32| GroundCommand::SelectAction {
|
||||
action: crate::Action::Investigate,
|
||||
target: None,
|
||||
problem: Some(p),
|
||||
};
|
||||
assert!(
|
||||
!is_visible(&state, PlayerId(0), &onto(hidden_key)),
|
||||
"targeting a face-down Problem required knowing what was under it"
|
||||
);
|
||||
assert!(
|
||||
is_visible(&state, PlayerId(0), &onto(face_up_key)),
|
||||
"a face-up Problem is visible — the marking must be able to say YES too"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
@ -43,6 +43,7 @@ kinds, states and metrics: [`GameDesign.md`](GameDesign.md). Reported by
|
|||
| F13 | inconsistent | withdrawn | scenarios/ground/gr-e01-threshold-reachable-2p.yaml | counterexample | 2026-08-01 | clay-borg |
|
||||
| F14 | unplayed | note | — | — | 2026-08-01 | clay-borg |
|
||||
| F15 | underdetermined | note | — | — | 2026-08-05 | clay-borg |
|
||||
| F16 | inconsistent | withdrawn | games/ground/examples/difficulty.rs | counterexample | 2026-08-05 | clay-borg |
|
||||
|
||||
<!-- design-register:end -->
|
||||
|
||||
|
|
@ -62,6 +63,16 @@ kinds, states and metrics: [`GameDesign.md`](GameDesign.md). Reported by
|
|||
`-reachable-`. **Its reproduction is green**, which under GameDesign §1.3
|
||||
is the alarm that forced the resolution. Withdrawn rather than deleted,
|
||||
and the withdrawal is reported (ADR-0012 D5).
|
||||
- **F16 — "the game is too easy at 5–6 seats", withdrawn the day it was
|
||||
raised.** Claimed from `GreedyPolicy` winning 200/200 at those seat
|
||||
counts. A `FirstLegal` policy scores **0%** on the identical deals, and
|
||||
at two seats it *beats* greedy — two unsophisticated agents span the
|
||||
whole range, so the measurement was about the policy. Caught by the
|
||||
CB-WP-0025 adversarial review (C4) **before transmission**; it would
|
||||
have been the fifth wrong premise sent to ground-game and the worst,
|
||||
since GROUND-WP-0005 is blocked on exactly this number. The withdrawal
|
||||
was reported (ADR-0012 D5). Its reproduction is `difficulty.rs`, whose
|
||||
policy panel is plural *because of this finding*.
|
||||
- **F15 — the rules define one game, not a series.** `OutcomeView` gives
|
||||
`personal` (per seat), `group_success` (per table) and `winners`. Summing
|
||||
the first and counting the third answer different questions, and GROUND
|
||||
|
|
|
|||
235
specs/RetrospectiveAnalysis.md
Normal file
235
specs/RetrospectiveAnalysis.md
Normal file
|
|
@ -0,0 +1,235 @@
|
|||
# RetrospectiveAnalysis — was this deal winnable, and how hard is the game
|
||||
|
||||
v1.0 — CB-WP-0025 T04, 2026-08-05. Normative. Implements
|
||||
[ADR-0013](../decisions/ADR-0013-could-we-have-won.md). Admissibility of
|
||||
anything this produces is governed by
|
||||
[GameDesign.md](GameDesign.md) §1.
|
||||
|
||||
**Two capabilities, one machine.** *Was this deal winnable?* is a search
|
||||
over a finished game. *How hard is the game?* is that search run over many
|
||||
deals and counted — **not** a bot's win rate (§4.1).
|
||||
|
||||
---
|
||||
|
||||
## 1. The question, and its name
|
||||
|
||||
> **"Given the deal as it actually was, was there a line of play that
|
||||
> reached the threshold?"**
|
||||
|
||||
**Never labelled "how you should have played."** The distinction is the
|
||||
honest content of the feature: the tool answers a question about the
|
||||
*deal*, and a label promising advice about the *player* turns a true
|
||||
answer into a false lesson.
|
||||
|
||||
**Strategy fusion does not apply and must not be invoked as an objection.**
|
||||
Fusion (Frank, Basin & Matsubara 1998) is a defect of aggregating over
|
||||
determinizations to choose a move. After the game there is **one world** —
|
||||
the deal is known — so a line found in it is executable in the only world
|
||||
there is. This is why the affordable option is also the honest one.
|
||||
|
||||
## 2. The witness
|
||||
|
||||
A witness is a sequence of joint selections that, replayed from the
|
||||
recorded initial state, ends with `group_success == true`.
|
||||
|
||||
### 2.1 It must replay — hard gate, not a metric
|
||||
|
||||
> **100% of emitted witnesses replay through the existing scenario runner
|
||||
> and end in `group_success`.**
|
||||
|
||||
Not a target: a **gate**. A witness that does not replay asserts the
|
||||
opposite of the truth to a player who just lost, which is worse than
|
||||
emitting nothing.
|
||||
|
||||
### 2.2 Every move carries its information dependence
|
||||
|
||||
ADR-0013 D2. Each move in a witness is marked:
|
||||
|
||||
| mark | meaning |
|
||||
|---|---|
|
||||
| `visible` | everything the move depended on was in the acting seat's projection at that point — the Problem face-up, the Solution in that seat's own hand |
|
||||
| `hidden` | it was not |
|
||||
|
||||
Computed from `project()`, which already exists and whose hiding rules are
|
||||
already asserted by `games_ground::view`.
|
||||
|
||||
**This replaces the structural boundary the survey wanted.** Searching a
|
||||
`GroundView` is not implementable — a view cannot `fold` events — so the
|
||||
guarantee moved from *the search cannot see it* to *the answer says which
|
||||
moves needed it*. A witness reads:
|
||||
|
||||
> *"This deal was winnable. Two of these six moves needed a card you had
|
||||
> no way to know was coming."*
|
||||
|
||||
**Falsifier:** a witness whose moves are all `visible` but which no seat
|
||||
could have chosen means the marking is wrong. A test constructs that case.
|
||||
|
||||
### 2.3 Wording when nothing is found
|
||||
|
||||
> **"No winning line found in the last K rounds."**
|
||||
|
||||
**Never "unwinnable".** A bounded search cannot establish unwinnability,
|
||||
and this sentence is what the player reads.
|
||||
|
||||
## 3. The bound
|
||||
|
||||
**Exhaustive over the last `K` rounds**, table treated as one co-operative
|
||||
agent choosing joint selections.
|
||||
|
||||
**`K = 1` for an exhaustive answer; `K` may be larger when a witness is
|
||||
all that is wanted.** ADR-0013 said `K = 2` by default; §3.1's measurement
|
||||
overrides it, and the difference is which question is being asked:
|
||||
|
||||
| answer | needs | affordable `K` today |
|
||||
|---|---|---|
|
||||
| *"here is a winning line"* | one success | 2+ — DFS finds one in tens of nodes |
|
||||
| *"there is no winning line"* | exhaustion | **1** — `K=2` exceeded 2×10⁶ nodes at two seats |
|
||||
|
||||
A `K` that cannot be exhausted may still emit a witness; it may **not**
|
||||
report `NoneFound { exhausted: true }`, and the type keeps those apart.
|
||||
|
||||
Bounded in **rounds**, not nodes: *"winnable from round 4"* means something
|
||||
to a player; *"winnable within 100,000 nodes"* does not. A node budget is a
|
||||
secondary cut that aborts with a stated reason so a wide table cannot hang
|
||||
the page.
|
||||
|
||||
### 3.1 Measured cost, and what it permits
|
||||
|
||||
`cargo bench -p games-ground --bench search` — the single source for these
|
||||
numbers (ADR-0013 D7). Mid-game states at real decision points:
|
||||
|
||||
| seats | branch width | `legal_commands` | `clone` | `validate+fold` |
|
||||
|---|---:|---:|---:|---:|
|
||||
| 2 | 5 | 4.06 µs | 378 ns | 696 ns |
|
||||
| 3 | 8 | 4.13 µs | 432 ns | 508 ns |
|
||||
| 4 | 11 | 4.76 µs | 639 ns | **3.76 µs** |
|
||||
|
||||
**Per-child cost is not uniform** — `validate+fold` ranges 0.5–3.8 µs
|
||||
depending on which command is taken, because some resolve cascades and
|
||||
some do not. **Budgets use the upper end**, so ~5 µs per child
|
||||
(clone + validate + fold).
|
||||
|
||||
Joint branching over the last two rounds, from the measured per-seat
|
||||
widths:
|
||||
|
||||
| seats | joint / 2 rounds | at ~5 µs/child |
|
||||
|---|---:|---:|
|
||||
| 2 | ~5×10² | negligible |
|
||||
| 3 | ~1.6×10⁵ | **~0.8 s** |
|
||||
| 4 | ~5.7×10⁵ | **~2.9 s** |
|
||||
|
||||
> ### The projection above was wrong, and the real search falsified it
|
||||
>
|
||||
> **Measured 2026-08-05 with the search built in T05**, rewinding real
|
||||
> games to the start of their last `K` rounds:
|
||||
>
|
||||
> | case | result |
|
||||
> |---|---|
|
||||
> | 2p, `K=1` | **exhausted** in 8,103 nodes, ~29 ms — a real negative |
|
||||
> | 2p, `K=2` | **budget cut** at 2,000,000 nodes, ~5 s — not exhausted |
|
||||
> | 3p, `K=2` | win found in 41 nodes, ~157 µs |
|
||||
>
|
||||
> §6's falsifier said *"§3 fails if K=2 proves unaffordable in practice at
|
||||
> four seats"*. **It failed at two.**
|
||||
>
|
||||
> The projection assumed a joint product per round. The search explores
|
||||
> **sequential per-seat decisions**, and the post-Reveal phase branches
|
||||
> over every seat's options at every level, so orderings multiply the tree
|
||||
> far beyond `width^seats`.
|
||||
>
|
||||
> **And the asymmetry is the operative fact:** *finding* a win is cheap —
|
||||
> depth-first stumbles onto one in tens of nodes — while *proving none
|
||||
> exists* is expensive, because it must exhaust the space. So:
|
||||
>
|
||||
> - **the witness feature (§2) is affordable now**, at any `K` a player
|
||||
> would ask about;
|
||||
> - **the winnable fraction (§4.2) is not**, because its "not winnable"
|
||||
> half requires exhaustion on every deal it counts.
|
||||
>
|
||||
> `K = 1` is the honest default for exhaustive answers today. Making
|
||||
> `K = 2` exhaustible needs transposition or move-ordering, neither of
|
||||
> which this pass built.
|
||||
|
||||
**The published 112–161 µs/node figure is withdrawn** (CB-RES-0008 §1.2,
|
||||
challenge C1) and must not be quoted from anywhere.
|
||||
|
||||
## 4. Difficulty
|
||||
|
||||
### 4.1 A bot's win rate is not a difficulty
|
||||
|
||||
**Normative prohibition**, because this project already made the error and
|
||||
nearly exported it:
|
||||
|
||||
> A win rate from a single policy **may not be reported as a difficulty**.
|
||||
|
||||
Measured, on identical deals: `GreedyPolicy` wins **100%** at five and six
|
||||
seats where a `FirstLegal` policy — take `legal[0]`, no heuristic — wins
|
||||
**0%**; at two seats `FirstLegal` (77.5%) *beats* greedy (66.0%). Two
|
||||
unsophisticated agents span the entire range.
|
||||
|
||||
And a measure that improves when the *measurer* improves is not measuring
|
||||
the subject: a better bot would make the game "easier" with no rule
|
||||
changing.
|
||||
|
||||
### 4.2 What is reported instead
|
||||
|
||||
> **Winnable fraction** — over N deals at a seat count, the proportion in
|
||||
> which the search finds a winning line within its bound.
|
||||
|
||||
A property of the **deal distribution and the threshold**, which is what
|
||||
`ground-game` tunes. Ships as a table, never one number:
|
||||
|
||||
| column | what it is |
|
||||
|---|---|
|
||||
| winnable fraction | can the deal be won at all — bounded, `K` stated |
|
||||
| reference-policy win rate | what a **named** policy achieves |
|
||||
| skill gap | the difference — how much play has to supply |
|
||||
|
||||
**It is a lower bound and must be labelled one.** A `K`-round search cannot
|
||||
see a line that required round 1, so the figure is
|
||||
**"winnable-from-round-(6−K)"**, never "winnable".
|
||||
|
||||
**Every rate carries its policy, N, seed range and K in the number's
|
||||
name**, not in a footnote — GameDesign §1.2, and the reason the withdrawn
|
||||
finding was inadmissible.
|
||||
|
||||
### 4.3 Resolution — the number that makes it usable
|
||||
|
||||
> The smallest threshold change the measurement can distinguish, with its
|
||||
> N.
|
||||
|
||||
*"We can tell a threshold of 5 from 7 but not 7 from 8"* is more useful to
|
||||
`ground-game` than any rate with no error bar, and it is what makes the
|
||||
figure a tuning instrument rather than a statistic.
|
||||
|
||||
## 5. The instruments must be able to fail
|
||||
|
||||
ADR-0013 D5, and GameDesign §1.3. Before **any** figure from these tools is
|
||||
quoted anywhere:
|
||||
|
||||
- **positive controls** — a deal constructed to be unwinnable returns
|
||||
none; a deal constructed to be winnable returns a witness that replays;
|
||||
- **`--self-test`**, wired into `make self-tests` like every other
|
||||
reporting tool;
|
||||
- **one command regenerates the figure** (`make difficulty`);
|
||||
- **the policy panel is plural** — at least `greedy`, `random` and
|
||||
`first-legal`. The spread between them is the finding §4.1 rests on, and
|
||||
reporting one policy would restore the error.
|
||||
|
||||
**`games/ground/examples/difficulty-baseline.rs` currently satisfies none
|
||||
of the first three** and is inadmissible until it does. It has no
|
||||
assertions, no self-test, and no `make` target — nothing can turn it red,
|
||||
which under CB-WP-0022 T05's `role` distinction makes it a `default`
|
||||
artifact wearing a `counterexample` label.
|
||||
|
||||
## 6. Falsifiers for this spec
|
||||
|
||||
- **§2.2 fails** if a witness is emitted whose moves are all `visible` but
|
||||
which no seat could have chosen. Then the marking must be derived from
|
||||
the search rather than checked after it.
|
||||
- **§4.2 fails** if the winnable fraction turns out to be ~100% or ~0% at
|
||||
every seat count and threshold — it would then have no resolution (§4.3)
|
||||
and be as useless as the bot rate it replaced.
|
||||
- **§3 fails** if `K = 2` proves unaffordable in practice at four seats;
|
||||
the measured 2.9 s is a projection from branch widths, not a timing of
|
||||
the real search.
|
||||
|
|
@ -2,7 +2,7 @@
|
|||
id: CB-WP-0025
|
||||
kind: product
|
||||
title: "Could we have won: a path out of a lost game, and how hard the game actually is"
|
||||
status: active
|
||||
status: done
|
||||
state_hub_workstream_id: "a866982f-94e7-432b-b0d5-2fefb781a574"
|
||||
---
|
||||
|
||||
|
|
@ -46,45 +46,30 @@ middle of stating, arriving with a concrete demand.
|
|||
|
||||
## What already exists, so the survey does not re-find it
|
||||
|
||||
- **The state is replayable.** `cb-game-runtime` records sessions as
|
||||
scenarios; `replay.rs` and `make replay-test` already re-run them.
|
||||
A search does not need new persistence.
|
||||
- **The move space is enumerable.** `legal_commands` exists and, since
|
||||
CB-WP-0023, is narrow enough to be worth trusting — SOLVE is offered
|
||||
only where it can act, so the branching factor is real rather than
|
||||
inflated by inert moves.
|
||||
- **Bots exist.** `games/ground/src/bot.rs` has `GreedyPolicy` and
|
||||
`RandomPolicy`, wired through `bot_policy` (`table.rs:208`). A win rate
|
||||
over N seeds is reachable with what is already there — the question is
|
||||
whether that number *means* anything, which is the survey's problem.
|
||||
- **The threshold is public.** `OutcomeView.total` / `.threshold` /
|
||||
`.group_success`. Difficulty has a denominator already.
|
||||
The state is replayable (`replay.rs`, `make replay-test`); the move space
|
||||
is enumerable (`legal_commands`, narrowed by CB-WP-0023 so branching is
|
||||
real rather than inflated by inert moves); bots exist (`bot.rs`); and the
|
||||
threshold is public (`OutcomeView`). **A search needs no new persistence
|
||||
and no new rules** — which is why D6 put it in `games/ground` with no new
|
||||
crate and no port.
|
||||
|
||||
## What makes this hard, and must not be waved through
|
||||
|
||||
**The game is not perfect-information and the search must respect that.**
|
||||
A path computed with the deck known is a path the players could never have
|
||||
found. `view.rs` hides the deck, other seats' hands, and face-down
|
||||
selections *by rule* (GR-S02/S04, GR-R02/R04). A retrospective solver
|
||||
running on `GroundState` sees all of it. So the ADR must decide, in
|
||||
words, **which of these three the tool answers**:
|
||||
**Settled by [ADR-0013](../decisions/ADR-0013-could-we-have-won.md).** The
|
||||
declaration framed the central risk as *a path computed with the deck
|
||||
known is a path the players could never have found*, and asked the ADR to
|
||||
choose between an omniscient, an information-respecting, and a bounded
|
||||
search.
|
||||
|
||||
- *was this deal winnable by an omniscient player* — cheap, honest,
|
||||
and answers a question nobody asked;
|
||||
- *was it winnable from what the seats could see* — the question actually
|
||||
asked, and the expensive one;
|
||||
- *did a reasonable line exist* — a bounded search from the losing seat's
|
||||
information, which may be the only affordable honest answer.
|
||||
**D1 dissolved the choice**: strategy fusion is a defect of aggregating
|
||||
over determinizations to *choose a move*, and after the game there is one
|
||||
world — so a line found in it is executable in it. **D2** keeps the
|
||||
declaration's real concern by marking each move `visible`/`hidden` rather
|
||||
than by refusing to search.
|
||||
|
||||
Getting this wrong produces a feature that tells the maintainer he could
|
||||
have won by playing a card he had no way to know was there. **That is
|
||||
worse than not shipping it.**
|
||||
|
||||
**And a difficulty number is a claim about a distribution.** One win rate
|
||||
over one bot policy over N seeds is not "the difficulty"; it is that
|
||||
policy's win rate. Whatever the spec adopts must name its policy, its N,
|
||||
and its seed range, or `ground-game` will tune tiers against a number
|
||||
whose meaning drifts the next time a bot improves.
|
||||
The second warning here — *"one win rate over one bot policy is not the
|
||||
difficulty"* — was right, and **the survey made exactly that error
|
||||
anyway**; see T02.
|
||||
|
||||
## Task: survey
|
||||
|
||||
|
|
@ -99,62 +84,39 @@ state_hub_task_id: "556cfd24-5992-4cc0-be90-0b60e989b5bb"
|
|||
(`loop-lint` checks both).
|
||||
|
||||
Per §Step 1 the survey is done when it can name a **benchmark-to-beat**
|
||||
per dimension — a number or a reproducible comparison, not an impression.
|
||||
|
||||
- **Retrospective solvers in games with hidden information.** The prior art
|
||||
is real and should be named: determinized search (perfect-information
|
||||
Monte Carlo) and its known failure — *strategy fusion*, where a
|
||||
determinizing solver claims lines that require knowing which world it is
|
||||
in. That failure is exactly the trap in §What makes this hard. Bridge
|
||||
and Skat post-mortem tools are the closest analogues; poker solvers are
|
||||
the well-studied case and the wrong shape.
|
||||
- **"A path to win" as a product, not a proof.** The maintainer already
|
||||
conceded optimality (*"the best path is not computable I guess"*). So
|
||||
the target is a **witness**: one concrete line of play that reaches
|
||||
`group_success`, or a defensible *no line found within bound B*. Name
|
||||
what a witness must carry to be checkable.
|
||||
- **Difficulty as a measured quantity in co-operative games.** Pandemic and
|
||||
its relatives set difficulty by a dial with a published win rate. The
|
||||
benchmark-to-beat is: can we produce a win rate whose confidence
|
||||
interval is tight enough to distinguish two threshold settings?
|
||||
- **Cost.** Search over an event-sourced aggregate with full `validate` on
|
||||
every branch has a per-node price. Measure it on our machine, on our
|
||||
scenarios — the runnable-baseline option applies here, since a search
|
||||
that cannot finish while the player is still looking at the page is a
|
||||
different feature.
|
||||
per dimension. Four were asked for: retrospective solvers in
|
||||
hidden-information games (and their known failure, strategy fusion); *a
|
||||
path to win* as a **witness** rather than a proof; difficulty as a measured
|
||||
quantity in co-operative games; and the **per-node cost**, measured on our
|
||||
machine — the runnable-baseline option applies, since a search that cannot
|
||||
finish while the player is looking at the page is a different feature.
|
||||
|
||||
**Done 2026-08-05.**
|
||||
[CB-RES-0008](../research/CB-RES-0008-could-we-have-won.md), with a
|
||||
runnable baseline (`games/ground/examples/difficulty-baseline.rs`).
|
||||
|
||||
**The baseline produced a finding before any solver exists, and it is the
|
||||
biggest thing in this pass.** A greedy bot wins **200 of 200** games at
|
||||
five and six seats — with a median margin of +3 and 11.8–12.0 points
|
||||
available against a threshold of 9. The curve is 66% / 82.5% / 95% / 100%
|
||||
/ 100% across 2/3/4/5/6 seats.
|
||||
> **Everything this record originally claimed was withdrawn by T02 the
|
||||
> same day.** Kept as a pointer rather than rewritten, because a claim
|
||||
> retracted silently is how three earlier wrong premises survived
|
||||
> (ADR-0012 D5).
|
||||
|
||||
The row-level table shows why: available points go **6 / 9 / 12** against
|
||||
thresholds **5 / 7 / 9**, so the ratio *rises* with seat count (1.20 →
|
||||
1.29 → 1.33) while the table also gains actions per round. **Three
|
||||
multipliers pointing the same way.**
|
||||
**Claimed:** a greedy bot wins 200/200 at five and six seats, so the game
|
||||
is too easy there; the points-to-threshold ratio rises with seat count and
|
||||
explains the curve; `legal_commands` costs 112–161 µs, so exhaustive
|
||||
search is out; and all of this explains the maintainer's report.
|
||||
|
||||
**It also explains the maintainer's report without a solver.** *"I felt it
|
||||
was too easy but then we lost"* — 66% at two seats is a real game; 100% at
|
||||
six is not. Both halves are true of different seat counts.
|
||||
**Withdrawn:** the win rate is `GreedyPolicy`'s, not the game's (C4 —
|
||||
`FirstLegal` scores 0% on the same deals); the ratio is identical at 3p
|
||||
and 4p, which differ by 12.5 points (C2); the cost figure was wrong by
|
||||
30–50× and exhaustive search is affordable (C1, C6); and the maintainer's
|
||||
losses were 3-player games on the pre-ruling deal (C5).
|
||||
|
||||
**Cost measured, and it rules out the obvious approach.** Branching is
|
||||
small (mean 4.7–9.1) but `legal_commands` costs **112–161 µs** per call,
|
||||
because it filters candidates through full `validate`. Exhaustive search
|
||||
is out at every seat count; ~10⁴–10⁵ nodes is 1.4–14 s, which is the
|
||||
budget the ADR must design inside.
|
||||
|
||||
**Prior art names the trap.** Determinized search (PIMC) suffers *strategy
|
||||
fusion* — Frank, Basin & Matsubara 1998 — where the search picks different
|
||||
actions in states a real player cannot tell apart. A witness built that
|
||||
way may require knowing what was on top of the deck. **And it would still
|
||||
replay green**, so the checkability benchmark does not catch it. Honesty
|
||||
and checkability are different properties, stated so T03 cannot conflate
|
||||
them.
|
||||
**What survives:** the harness exists and runs, the 6/9/12 arithmetic is
|
||||
right against `Problems.csv`, and the branching widths (4.7–9.1) hold. See
|
||||
[CB-RES-0008](../research/CB-RES-0008-could-we-have-won.md) for the
|
||||
corrected text and
|
||||
[the response](../history/260805-could-we-have-won-response.md) for the
|
||||
full accounting.
|
||||
|
||||
## Task: adversarial review
|
||||
|
||||
|
|
@ -166,103 +128,89 @@ state_hub_task_id: "bdd9fc97-4706-4d8e-b286-91ff53339680"
|
|||
```
|
||||
|
||||
Tier L requires it. Exactly one round: challenge, then response, trail in
|
||||
`history/`, unpolished. Require an attempt at:
|
||||
|
||||
- **that the honest version is unaffordable** — that a search respecting
|
||||
the information rule is too expensive or too weak to find anything, so
|
||||
the shipped tool will quietly become the omniscient one with a
|
||||
reassuring label;
|
||||
- **that a witness misleads more than it helps** — being shown a line that
|
||||
needed a card you could not know about teaches a wrong lesson about the
|
||||
game, and the tool would be better refusing to answer;
|
||||
- **that the difficulty number is a bot benchmark wearing a difficulty
|
||||
costume**, and `ground-game` will tune the game against our bot rather
|
||||
than against play;
|
||||
- **that this is CB-WP-0022's job** — the design instrument is being built
|
||||
right now, and a difficulty measurement is a finding-producing tool. The
|
||||
strongest counter is that the register records findings and this
|
||||
*produces* them, but the reviewer should press whether that is a
|
||||
distinction worth a separate capability.
|
||||
`history/`, unpolished. Four attempts required: that the honest version is
|
||||
unaffordable; that a witness misleads more than it helps; that the
|
||||
difficulty number is a bot benchmark in a difficulty costume; and that
|
||||
this is CB-WP-0022's job rather than a separate capability.
|
||||
|
||||
**Done 2026-08-05.** Trail:
|
||||
[challenge](../history/260805-could-we-have-won-challenge.md),
|
||||
[response](../history/260805-could-we-have-won-response.md).
|
||||
Separate agent, as in CB-WP-0022.
|
||||
|
||||
**Six of seven conceded, and the survey's headline finding is withdrawn —
|
||||
not softened.**
|
||||
**Six of seven conceded. The survey's headline finding is WITHDRAWN.**
|
||||
|
||||
**C4 is the one that kills it, and the reviewer ranked it fourth.**
|
||||
Measured: a `FirstLegal` policy — take `legal[0]`, no heuristic — scores
|
||||
**0% at five and six seats** where greedy scores 100%, and **77.5% at two
|
||||
seats** where greedy scores 66%. **Two unsophisticated agents span the
|
||||
entire range at the same seat count**, so *"the game is too easy at 5–6
|
||||
seats"* is a statement about `GreedyPolicy`, not about GROUND. Their
|
||||
offered rescue — greedy hits the 12-point ceiling every time, so it is a
|
||||
rules claim — dies on the same data.
|
||||
**C4 kills it and the reviewer ranked it fourth.** A `FirstLegal` policy —
|
||||
`legal[0]`, no heuristic — scores **0% at five and six seats** where greedy
|
||||
scores 100%, and **77.5% at two** where greedy scores 66%. Two
|
||||
unsophisticated agents span the whole range, so *"too easy at 5–6 seats"*
|
||||
was about `GreedyPolicy`, not GROUND.
|
||||
|
||||
**C1: the per-node cost was wrong by 30–50×.** The timer bracketed two
|
||||
`setup`s, a whole greedy game and a validate+fold replay, then divided by
|
||||
player decisions. **The tell was in my own published output**: the figure
|
||||
*fell* as branching *rose*, which no per-enumeration cost can do.
|
||||
Re-measured at **3.0–4.1 µs**; the reviewer got 15.6–20.4 by a different
|
||||
isolation and **that discrepancy is unsettled** — T04 benchmarks it.
|
||||
|
||||
**C6: exhaustive search is not out**, which changes T03's premise. With
|
||||
C1's correction the budget is ~10⁵–10⁶ nodes and bounded endgame search
|
||||
fits — so the ADR cannot open with *"exhaustive is impossible, therefore
|
||||
determinized sampling"*, especially since sampling carries strategy fusion
|
||||
that exhaustive search does not.
|
||||
|
||||
**C3: the finding failed the admissibility rule this project wrote nine
|
||||
hours earlier** — sums where GROUND-WP-0004 T02 requires per-priority
|
||||
rows, and a harness with no assertions, no `--self-test` and no `make`
|
||||
target, so nothing can turn it red. A `default` artifact wearing a
|
||||
`counterexample` label.
|
||||
|
||||
**C2: the ratio explains nothing** — 3p and 4p share deal, threshold and
|
||||
ratio, and differ by 12.5 points of win rate. **C5: the "explains the
|
||||
maintainer's report" claim is contradicted by `lib.rs:2487`**, which
|
||||
records his losses as 3-player on the pre-ruling deal, unwinnable at 6
|
||||
against 7.
|
||||
**C1**: the node cost was wrong by 30–50× — the timer bracketed whole
|
||||
games — and **the tell was in my own output**, falling as branching rose.
|
||||
**C6**: exhaustive search is *not* out, which changes T03's premise.
|
||||
**C3**: the finding failed the admissibility rule this project wrote nine
|
||||
hours earlier. **C2**: 3p and 4p share deal, threshold and ratio and differ
|
||||
by 12.5 points. **C5**: `lib.rs:2487` contradicts the "explains the
|
||||
maintainer" claim — his losses were 3-player on the pre-ruling deal,
|
||||
unwinnable at 6 against 7.
|
||||
|
||||
**T06 was pointed at GROUND-WP-0005, which is blocked on this number.**
|
||||
Sending it would have invited threshold changes on one bot's behaviour —
|
||||
the fifth wrong premise to reach ground-game. **Both tier-L reviews in
|
||||
this project have now caught a false headline that every gate passed.**
|
||||
Sending it would have been the fifth wrong premise to reach ground-game.
|
||||
**Both tier-L reviews here have now caught a false headline that every
|
||||
gate passed.**
|
||||
|
||||
## Task: decide
|
||||
|
||||
```task
|
||||
id: CB-WP-0025-T03
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "055630a1-56e0-4ad4-8796-35f444313371"
|
||||
```
|
||||
|
||||
`decisions/ADR-0013-*.md`. (ADR-0012 is CB-WP-0022 T03's.) At minimum:
|
||||
which question the solver answers and what it is called in the UI; the
|
||||
**information boundary** and whether it is structural or promised; the
|
||||
**bound**, and what *no path found* means against it; whether difficulty
|
||||
is one number or a table, and of what; and **where it lives** — the tier
|
||||
was declared L assuming a new capability port, and if there is none, say
|
||||
so.
|
||||
|
||||
- **which question the solver answers**, from the three in §What makes
|
||||
this hard, and what it is called in the UI — the name must not overclaim;
|
||||
- **the information boundary**: whether the search runs on `GroundState`
|
||||
or on a `GroundView`, and if on state, what stops it using what the view
|
||||
hides. Note that running on the view makes the rule structural rather
|
||||
than a promise, and that this is the cheapest guarantee available;
|
||||
- **the bound**: depth, node budget, or wall clock, and what *no path
|
||||
found* means against it — a bounded search that says "unwinnable" is
|
||||
lying, and the wording must say "none found within B";
|
||||
- **whether difficulty ships as one number or a small table**, and what it
|
||||
is a function of: policy, seat count, threshold, seed range;
|
||||
- **where it lives** — a new crate, a mode of `cb-play`, or a tool under
|
||||
`tools/`. The tier was declared L on the assumption of a new capability;
|
||||
if the ADR concludes it is a mode of an existing one, say so, and the
|
||||
over-declaration is a chaos-window data point worth recording.
|
||||
**Done 2026-08-05.**
|
||||
[ADR-0013](../decisions/ADR-0013-could-we-have-won.md), seven decisions.
|
||||
**Two of them are not what T03 was written expecting**, because the review
|
||||
moved the ground under both.
|
||||
|
||||
- **D1 — strategy fusion does not apply, and that is why this is
|
||||
affordable.** Fusion is a defect of *aggregating over determinizations
|
||||
to choose a move*. **After the game there is one world**: the deal is
|
||||
known, so a search over it produces a line executable in the only world
|
||||
there is. The survey treated fusion as this pass's central obstacle; it
|
||||
is an obstacle to a *playing* engine, which we are not building.
|
||||
- **D4 — difficulty is the WINNABLE FRACTION, not any bot's win rate.**
|
||||
C4 killed the bot rate: two trivial policies span 0–100% on the same
|
||||
deals, and improving the bot would make the game "easier" without a rule
|
||||
changing. The solver supplies a policy-independent measure — *over N
|
||||
deals, in what proportion does a winning line exist* — which is a
|
||||
property of the deal distribution and the threshold, and is what
|
||||
GROUND-WP-0005 actually needs. **The bot rate never was.**
|
||||
|
||||
**D2** searches `GroundState` — the survey's view-only boundary is not
|
||||
implementable, since a view cannot fold events — and moves the guarantee to
|
||||
a checkable per-move `visible`/`hidden` marking from `project()`. **D3**
|
||||
bounded exhaustive over the last K rounds, with *"no winning line found in
|
||||
the last K rounds"* normative, never *"unwinnable"*. **D5** makes the
|
||||
harness an instrument before any figure is quoted (C3). **D6** no new
|
||||
crate and no port — **the L declaration was an over-declaration, recorded
|
||||
as one**. **D7** the node cost is disputed 5×; T04 benchmarks it and
|
||||
neither existing figure may be cited.
|
||||
|
||||
## Task: specify
|
||||
|
||||
```task
|
||||
id: CB-WP-0025-T04
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "7c3ca25d-0001-44ef-a07d-0ab2d51b09a5"
|
||||
```
|
||||
|
|
@ -270,29 +218,53 @@ state_hub_task_id: "7c3ca25d-0001-44ef-a07d-0ab2d51b09a5"
|
|||
`specs/` — extend `MetricsAndScenarios.md` or add a capability spec as the
|
||||
ADR directs, with metrics, because a spec without them is prose.
|
||||
|
||||
Candidates, to be argued not adopted:
|
||||
|
||||
- **witness checkability** — every path the tool emits replays through the
|
||||
existing scenario runner and ends in `group_success`. Target 100%, and it
|
||||
is a hard gate, not a metric: a path that does not replay is a bug that
|
||||
says the opposite of the truth;
|
||||
- **search cost** — nodes and wall clock at the chosen bound, on the
|
||||
recorded games we have;
|
||||
- **difficulty resolution** — the smallest threshold difference the
|
||||
measurement can distinguish, with its N. This is the number
|
||||
`ground-game` needs, and stating it as *"we can tell 5 from 7 but not 7
|
||||
from 8"* is more useful than a win rate with no error bar.
|
||||
Candidates, to be argued not adopted: **witness checkability** (every
|
||||
emitted path replays to `group_success` — a hard gate, not a metric, since
|
||||
a path that does not replay says the opposite of the truth); **search
|
||||
cost** at the chosen bound; and **difficulty resolution** — the smallest
|
||||
threshold difference the measurement can distinguish, with its N, which is
|
||||
more useful to `ground-game` than any rate with no error bar.
|
||||
|
||||
Per `ground-game`'s ruling (GROUND-WP-0004 T02), **any arithmetic finding
|
||||
this produces ships a runnable reproduction and a row-level table** — never
|
||||
a summed figure. A difficulty number is arithmetic, and it is exactly the
|
||||
kind that has already gone wrong twice.
|
||||
|
||||
**Done 2026-08-05.**
|
||||
[specs/RetrospectiveAnalysis.md](../specs/RetrospectiveAnalysis.md) v1.0,
|
||||
and `games/ground/benches/search.rs` for D7's disputed number.
|
||||
|
||||
**The benchmark's own first fixture was defective — the same defect class,
|
||||
one layer down.** A fixed stop at step 20 put 2p and 4p where seat 0 had
|
||||
**no legal commands**, so it timed an empty `Vec` (~120 ns) and silently
|
||||
skipped `validate_fold`. It now advances to a real branch **and asserts
|
||||
it**.
|
||||
|
||||
Measured at real decision points (table in
|
||||
[RetrospectiveAnalysis §3.1](../specs/RetrospectiveAnalysis.md)):
|
||||
`legal_commands` 4.06–4.76 µs, `clone` 378–639 ns, `validate+fold`
|
||||
**0.5–3.8 µs**.
|
||||
|
||||
**`clone` is in there because a search must copy state per branch**, and
|
||||
`iter_batched` excludes setup from timing — leaving the budget on an
|
||||
unmeasured span, which is precisely C1's mistake.
|
||||
|
||||
**The affordability conclusion drawn here was itself falsified by T05.**
|
||||
It projected joint branching and concluded `K = 2` holds at two to four
|
||||
seats. The real search exceeded 2×10⁶ nodes at **two**. See T05's record
|
||||
and [RetrospectiveAnalysis §3.1](../specs/RetrospectiveAnalysis.md) —
|
||||
a projection from branch widths is not a timing of a search, and this
|
||||
pass has now made that mistake twice.
|
||||
|
||||
**§4.1 is a normative prohibition**, not a preference: a single-policy win
|
||||
rate may not be reported as a difficulty. The spec carries the measured
|
||||
reason — greedy 100% vs first-legal 0% on identical deals.
|
||||
|
||||
## Task: build the witness
|
||||
|
||||
```task
|
||||
id: CB-WP-0025-T05
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "5f1cbda3-7427-4f23-8c67-b2e216b987a8"
|
||||
```
|
||||
|
|
@ -313,23 +285,48 @@ a loss.
|
|||
naming *that* boundary goes red. If it cannot be mutated, it was a
|
||||
comment rather than a rule.
|
||||
|
||||
**Done 2026-08-05.** `games/ground/src/search.rs`, five tests.
|
||||
|
||||
**The first traversal was wrong and the diagnostic hid it.** It branched
|
||||
on *the first seat with any legal command* and stopped there, so a later
|
||||
seat never acted if an earlier one was already selected. Restructured
|
||||
around what the rules oblige: a seat without a selection **must** select
|
||||
(GR-R02) and nothing else can happen first; after Reveal the optional
|
||||
actions branch freely, and the aggregate rejects `Resolve` until the
|
||||
obligatory ones are done — **so the search needs no phase logic of its
|
||||
own.**
|
||||
|
||||
**And my rewind was off by one round**, replaying the round it was meant
|
||||
to search — which is why the first run reported 3 nodes and looked fine.
|
||||
|
||||
**The measurement falsified the spec's own projection at two seats, where
|
||||
§6 predicted four**: 2p `K=1` exhausted in 8,103 nodes (~29 ms), 2p `K=2`
|
||||
cut at 2,000,000 nodes (~5 s), 3p `K=2` won in 41 nodes.
|
||||
|
||||
The projection assumed a joint product per round; the search explores
|
||||
sequential per-seat decisions, so orderings multiply the tree far beyond
|
||||
`width^seats`.
|
||||
|
||||
**The asymmetry is the operative finding.** *Finding* a win is cheap;
|
||||
*proving none exists* needs exhaustion. So the **witness feature is
|
||||
affordable now** and the **winnable fraction is not**, because its
|
||||
negative half must exhaust every deal it counts.
|
||||
|
||||
## Task: measure the difficulty, and hand it to ground-game
|
||||
|
||||
```task
|
||||
id: CB-WP-0025-T06
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "723feae8-1ed4-4dd4-9330-8ec84625e5d9"
|
||||
```
|
||||
|
||||
Run the measurement, ship it as a `make` target beside the other
|
||||
instruments, and show the result in the game — the maintainer asked for it
|
||||
to be visible, and a number in a file will not calibrate anything.
|
||||
|
||||
Then send it to `ground-game` **against GROUND-WP-0005**, which is active
|
||||
with both tasks waiting on exactly this. Per CB-WP-0022 T06, it lands as a
|
||||
file in their repo under their workplan, not only an inbox entry — *the
|
||||
message that sat unread for four days is the baseline to beat*.
|
||||
instruments, and send it to `ground-game` against **GROUND-WP-0005**,
|
||||
which is active with both tasks waiting on exactly this. Per CB-WP-0022
|
||||
T06 it lands as a **file in their repo under their workplan**, not only an
|
||||
inbox entry — *the message that sat unread for four days is the baseline
|
||||
to beat*.
|
||||
|
||||
**Controls:**
|
||||
- the number regenerates from a single command, and `facts.toml` carries
|
||||
|
|
@ -339,22 +336,65 @@ message that sat unread for four days is the baseline to beat*.
|
|||
- **the seed range and policy are in the number's name**, not in a
|
||||
footnote.
|
||||
|
||||
**Done 2026-08-05.** `games/ground/examples/difficulty.rs`, `make
|
||||
difficulty`, wired into `make self-tests`, and
|
||||
[a report file in ground-game](../../ground-game/workplans/GROUND-WP-0005-clay-borg-difficulty-260805.md)
|
||||
under GROUND-WP-0005 with a hub message pointing at it.
|
||||
|
||||
**The report opens with the retraction**, because what this task was
|
||||
written to send was withdrawn by T02 and GROUND-WP-0005 is blocked on
|
||||
exactly that number.
|
||||
|
||||
Winnable-from-round-5, 60 seeds: **60% / 93% / 100% / 100% / 100%** at
|
||||
2/3/4/5/6 seats, beside greedy / random / first-legal rates and the spread
|
||||
between them. Full table in the report and in `make difficulty`.
|
||||
|
||||
**`spread` justifies the whole redesign** — 71.7 to 100.0 points between
|
||||
three trivial policies. No single rate says anything about the game, and
|
||||
the table now shows that rather than asserting it.
|
||||
|
||||
**The confound is stated in the tool's own output**: `winnable` is
|
||||
conditioned on greedy's play up to the final round. Presenting it as a
|
||||
property of the deal would repeat this pass's error in a subtler form.
|
||||
**No threshold changes are proposed.** The instrument can fail (spec §5),
|
||||
and `difficulty-baseline.rs` is marked superseded.
|
||||
|
||||
Registered as **F16, `inconsistent`, `withdrawn`**, with `difficulty.rs`
|
||||
as its reproduction.
|
||||
|
||||
## Task: evidence
|
||||
|
||||
```task
|
||||
id: CB-WP-0025-T07
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "2616e06c-5d88-4680-be18-4452515dff21"
|
||||
```
|
||||
|
||||
`evidence/CB-EV-0023-*.md`.
|
||||
`evidence/CB-EV-*.md` (**CB-EV-0024** — CB-EV-0023 went to CB-WP-0024).
|
||||
Was the maintainer's game winnable; what the
|
||||
honest search cost against the omniscient one; whether the difficulty
|
||||
measurement moved `ground-game`; what tier L cost against what it caught;
|
||||
and CB-WP-0024's cost by re-running the instrument.
|
||||
|
||||
- **Was the game winnable**, for the maintainer's actual lost game. That is
|
||||
the acceptance test with a face on it.
|
||||
- **What the honest search cost against the omniscient one**, since the
|
||||
review will have pressed hardest there.
|
||||
- **Whether the difficulty measurement moved ground-game**, or sat.
|
||||
- **What tier L cost against what it caught** — third full-weight L pass in
|
||||
the project, and the second in this chaos window.
|
||||
- **Quote CB-WP-0024's cost by re-running the instrument.**
|
||||
**Done 2026-08-05.**
|
||||
[CB-EV-0024](../evidence/CB-EV-0024-could-we-have-won.md).
|
||||
|
||||
- **The maintainer's question is answered.** 3p seed 7 was winnable — a
|
||||
10-move line, found in 41 nodes, replays to `group_success`. 2p seed 7
|
||||
was **not**, exhaustively, in 8,103 nodes. **The witness feature works;
|
||||
the difficulty half is where the trouble was.**
|
||||
- **Five of nine defects came only from the review**, four from execution
|
||||
— and all four of those were in work written *after* it.
|
||||
- **The wrong-denominator family now has five instances and no control.**
|
||||
- **The 5–6 seat rows point the opposite way from the withdrawn claim**;
|
||||
neither reading is established, and the evidence says so.
|
||||
- **Tier L was an over-declaration** (no port) **and paid for itself
|
||||
anyway** — the review is L-only, and without it a false number reaches a
|
||||
blocked repo.
|
||||
- **Chaos window 2 will close with zero overrides**, making its retirement
|
||||
condition untestable.
|
||||
|
||||
**Not done, and named rather than quietly dropped:** the witness is **not
|
||||
wired to the ending page**, so the browser cannot yet ask the question.
|
||||
That is the maintainer-facing half of remark 2.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue