# 260805 — challenge to CB-RES-0008 Adversarial review, one round, per InnerLoop §Step 2. Target: the survey `research/CB-RES-0008-could-we-have-won.md`, the harness `games/ground/examples/difficulty-baseline.rs`, and the T01 §judgment in `CB-WP-0025:126-157`. **Fidelity note, first.** Run in a separate agent session with only the files. There is one harness and one repo, so per §Step 2 this review **inherits the author's sampling** and does not report a clean verify on that basis. What it substitutes is **mutation**: every quoted number was traced to the expression that produces it, and the expression was changed. A mutant copy of the harness (`examples/zz-review-mutant.rs`) was written, run, and deleted; `git status` is clean and no file in the repo was modified. Every number below is reproducible by re-creating that mutant from the diffs quoted inline. Three of the four headline claims move under mutation. One of them moves by a factor of seven. Ranked. **C1 and C2 land hardest.** C7 is marked weak. --- ## C1 — `legal_commands` does not cost 112–161 µs. It costs 15.6–20.4 µs. The timer is around the wrong thing. The survey, §1.2: > *"**`legal_commands` costs 112–161 µs per call**, because it constructs > candidates and filters them through full `validate`. … it is the number > that decides this pass."* Read the timer's scope. `difficulty-baseline.rs:100` starts the clock: ```rust 100: let start = std::time::Instant::now(); 102: for seed in 0..40u64 { 103: let Some(state) = setup(players, seed) else { continue }; // deal #1 108: let Ok(game) = play(state, &mut ps) else { continue }; // A WHOLE GREEDY GAME 112: let Some(mut replay) = setup(players, seed) else { continue };// deal #2 115: for (actor, cmd) in &game.steps { 118: let legal = legal_commands(&replay, *seat); // the thing being claimed 121: if let Ok(events) = replay.validate(*actor, cmd) { // + full replay 122: for e in &events { replay.fold(e); } 129: let elapsed = start.elapsed(); 143: per = elapsed.as_micros() as f64 / nodes as f64, ``` `elapsed` is **two `setup`s, a complete five-round greedy game (which itself calls `legal_commands` on every step and runs the whole ranking loop), a second deal, and a full validate+fold replay of every command, player and system alike** — divided by the count of *player* decision points only. It is not a per-node search cost. It is a per-game cost with a per-node denominator. **Mutation.** Bracket `legal_commands` alone (`Instant::now()` immediately before line 118, accumulate on the next line), and separately time the `GroundState::clone` a real search must also pay per node: ``` 2p 462 nodes mean width 4.7 ORIGINAL 112 us legal_commands ALONE 15.6 us clone 1.3 us 3p 649 nodes mean width 7.4 ORIGINAL 115 us legal_commands ALONE 18.0 us clone 1.4 us 4p 870 nodes mean width 9.1 ORIGINAL 116 us legal_commands ALONE 20.4 us clone 1.4 us ``` **The claimed number is 6–8× the measured one.** And the mutant supplies the diagnostic that should have caught it in the survey: the survey's figures **fall** with seat count (161 → 139 → 112) while branching **rises** (4.7 → 7.4 → 9.1). If the number were the cost of constructing and validating candidates, it would rise with the number of candidates. `legal_commands` alone does exactly that — 15.6 → 18.0 → 20.4. The survey's number falls because the fixed per-game overhead is being amortised over more nodes at higher seat counts. **The reported quantity varies inversely with the mechanism the prose gives for it**, in the survey's own printed table, and the survey did not notice. **Secondary, and it is enough on its own.** The number is not stable. Three runs of the *unmodified* example on this machine: ``` run 1 (survey) 2p 161 3p 139 4p 112 run 2 2p 132 3p 128 4p 139 run 3 (mutant) 2p 112 3p 115 4p 116 ``` The 2p figure moved 161 → 112 between runs, and the seat-count ordering inverted. The survey quotes `112–161 µs/node` in three places (`:44`, `:83`, `:189`) as a **measured range across seat counts**. It is a range across *runs*, of a quantity that is mostly loop overhead. Anyone can settle this: run the example twice. **What this changes in the design, which is why it is C1.** §1.2 concludes *"at ~140 µs/node: a bounded search of ~10⁴–10⁵ nodes costs 1.4–14 seconds. **That is the budget the ADR has to design inside.**"* At the measured ~17 µs for `legal_commands` plus ~1.4 µs for the clone, ~19 µs/node, the same 1.4–14 s buys **~10⁵–10⁶ nodes**. T03 is about to pick a node bound one order of magnitude too small, and T04 is about to write it into an acceptance table. **Required:** re-scope the timer to the call the prose names, re-quote, and state the run-to-run spread rather than a single range. If the intent was "the cost of one node of a search that replays from a scenario", say so and price the clone-and-fold node separately — but then it is not `legal_commands`'s cost and §1.2's causal sentence must go. ## C2 — The ratio does not explain the curve, and the survey's own table proves it: two rows with an identical ratio are 12.5 points apart §1.1 is the pass's headline finding: > *"**The ratio moves the wrong way.** … Three multipliers all pointing the > same direction, which is why the curve is not gentle — it is 66% → 100% > across four seat counts."* | seats | ratio (survey) | greedy win rate | |---|---|---| | 2 | 1.20 | 66.0% | | 3 | 1.29 | 82.5% | | 4 | **1.29** | **95.0%** | | 5 | 1.33 | 100% | | 6 | **1.33** | **100%** | **3p and 4p have the same deal, the same threshold, and the same ratio, and differ by 12.5 points of win rate** — a jump as large as either of the two between-ratio jumps. The ratio takes three distinct values across five rows; seat count takes five. The explanatory variable and the confound are not separated anywhere in the survey, and the one comparison that separates them (3p vs 4p) points at seat count, not at the ratio. It gets worse for the ratio when the arithmetic is done at row level, which §1.1 claims to have done and has not (see C3). Claimable point values are `{2, 2, 2}` at 2p, `{2, 2, 2, 3}` at 3–4p, `{2, 2, 2, 3, 3}` at 5–6p (`editions/ground-darvo-r0/Problems.csv`, priorities 0–4 of every scenario; all four scenarios carry the same value vector, checked). So the **achievable** totals are not continuous, and the useful quantity is *how much of the board must be claimed*: | seats | achievable totals | threshold | Problems that must be claimed | effective slack | |---|---|---:|---|---:| | 2 | 0, 2, 4, 6 | 5 | **3 of 3** — a full clear | **1.00** | | 3–4 | 0, 2, 3, 4, 5, 6, 7, 9 | 7 | 3 of 4, incl. the 3-pointer | 1.29 | | 5–6 | 0 … 12 | 9 | 4 of 5 | 1.20 | A threshold of 5 at 2p is **identical to a threshold of 6** — nothing sums to 5. The survey's `1.20` is not slack; the real slack at 2p is 1.00, and `games/ground/src/lib.rs:2504` already says so in as many words (*"2+2+2 against a threshold of 5 means a full clear"*). And on this measure the sequence is **1.00 / 1.29 / 1.20 — not monotone**, so "the ratio moves the wrong way" reverses at the seat count the finding is loudest about. **Confirmed directly.** Instrumenting the harness to record whether the group claimed *every* Problem on the board: ``` 2p greedy 132/200 won cleared-the-board 132/200 AVAIL 6 3p greedy 165/200 won cleared-the-board 161/200 AVAIL 9 4p greedy 190/200 won cleared-the-board 190/200 AVAIL 9 5p greedy 200/200 won cleared-the-board 186/200 AVAIL 12 6p greedy 200/200 won cleared-the-board 200/200 AVAIL 12 ``` At 2p, wins == board-clears **exactly** (132 = 132): the 2-player game is not "a real game at 66%", it is pass/fail on a full clear. The mechanism driving the curve is the third item the survey lists last and never quantifies — more seats means more hands, which means more matching Solutions, which means a higher fraction of a *fixed* pool gets claimed. More seats add **no points**; they add claimants for the same 5 Problems. **Required:** withdraw "three multipliers all pointing the same direction" or measure it. The separating experiment is cheap and was not run: hold seats fixed and move the threshold, or hold the threshold fixed and move `hidden_depth` (`games/ground/src/edition.rs:123-130`). Until one of those runs, the finding handed to GROUND-WP-0005 tells them to tune the wrong dial — and T06 ships it into another repo. ## C3 — The finding is inadmissible under GameDesign §1 on two of the three clauses, and the survey asserts all three §1.1: > *"**This is admissible under GameDesign §1**: the reproduction exists > (`examples/difficulty-baseline.rs`), it has the ruled shape (row-level, > no sums), and it can fail — change a threshold and the numbers move."* **Clause 3, "can fail" — fails.** `difficulty-baseline.rs` contains **zero assertions**: ``` $ grep -c "assert" games/ground/examples/difficulty-baseline.rs 0 $ grep -rn "difficulty" Makefile gates.toml facts.toml (no output) ``` It prints and exits 0. Delete the win-rate line, change a threshold, break the deal — it still exits 0. *"The numbers move"* is **sensitivity**, not failability; GameDesign §1.3 requires *"the artifact must be capable of going red, and the register records its current colour."* An artifact with no assertion has no colour. It is not a `counterexample` (nothing alarms) and it is not a `default` (it encodes no choice) — it is a third thing, an **observation**, and CB-WP-0022 T05's role table has no row for it. That is the distinction this repo paid for four days ago. This also violates InnerLoop §Measurement validity twice over: *"every tool that reports a number exposes `--self-test`, and that self-test runs before the number is produced"* — there is none, and the example is not wired into `make` at all — and *"state the divisor used to convert raw timings into the metric's unit, pinned by a test"* — `nodes` at `:143` is unpinned. **Mutation:** run the win-rate loop over 40 seeds instead of 200 (`const SEEDS: u64 = 40`): ``` 6p greedy 40/40 = 100.0% mean total 12.0 of 9.0 median margin +3 ``` Nothing in the harness objects to a denominator that shrank by 5×. Three `continue`s (`:57`, `:62`, `:65`) can silently drop games out of `played` and there is no `assert_eq!(played, SEEDS)`. *(To the author's credit, `played` is printed, and on the current run it is genuinely 200/200 — see §What survives. The control is missing, not the work.)* **Clause 2, "the ruled shape" — fails.** GameDesign §1.2, quoting GROUND-WP-0004 T02: *"an arithmetic finding ships a row-level table — Surface and each hidden priority listed **separately** — never 'sum of file'"*, and *"the artifact **prints the rows** it came from. A finding stating a total without its rows is inadmissible even if the total is right."* The survey's table is: | seats | Surface | hidden dealt | points available | threshold | ratio | |---|---|---|---:|---:|---| | 2 | priority 1 | 2 | **6** | 5 | 1.20 | `6`, `9`, `12` are **sums**. The hidden priorities are collapsed into a count (`hidden dealt: 2`), which is the exact shape the ruling forbids. The `Surface` column says *"priority 1"* in all three rows, and Surface is `hidden_priority` **0** in `Problems.csv` — the cell is either wrong or meaningless, and it is the cell the ruled shape is about. And the named artifact prints **none of this**. `difficulty-baseline.rs` never prints available points, thresholds by row, or priorities; it prints win rates and timings. The reproduction cited for the 6/9/12 finding does not compute 6/9/12. The thing that does is `games_ground::gd0001_group_success_is_reachable_at_every_seat_count` (`lib.rs:2508`) — which the survey does not name, and which *also* sums (`lib.rs:2512`: `.map(|p| p.value).sum()`). **Required:** either name `gd0001` and give it the row-level print the ruling requires, or drop the admissibility claim. This is the third time a correct total has shipped with the wrong shape (`"12 in the file"`, `"4/6/9"`, and now this), and it is the failure GameDesign §1.2 was written this week to stop. ## C4 — 66% at two seats is a `GreedyPolicy` defect, not a difficulty. A policy with no heuristic at all scores 77.5%. The task the survey sets itself in §4 and defers in §6 — *"whether a bot win rate is a difficulty at all"* — is settled against it by one mutation. Add two zero-knowledge policies: `FirstLegal` (always `Choice::Command(0)`) and `LastLegal` (always the last offered command), 200 seeds each: ``` 2p greedy 132/200 = 66.0% 2p first 155/200 = 77.5% 2p last 0/200 2p random 10/200 = 5.0% 3p greedy 165/200 = 82.5% 3p first 50/200 = 25.0% 3p last 0/200 3p random 19/200 = 9.5% 4p greedy 190/200 = 95.0% 4p first 64/200 = 32.0% 4p last 0/200 4p random 16/200 = 8.0% 5p greedy 200/200 = 100.0% 5p first 0/200 = 0.0% 5p last 0/200 5p random 6/200 = 3.0% 6p greedy 200/200 = 100.0% 6p first 0/200 = 0.0% 6p last 0/200 6p random 7/200 = 3.5% ``` **At two seats, taking the first command in canonical order beats the stated heuristic by 11.5 points.** So: 1. **The 66% is not the game's 2-player difficulty.** It is the point at which `GreedyPolicy`'s ranking (`bot.rs:326-380`) becomes *worse than no ranking*. Somebody improving the bot next week moves this row to ~78% and "the 2-player game got easier" without a rule changing — which is the objection §6 says is the strongest and leaves open. It is not open; it is demonstrated, on the row the survey leans on hardest. 2. **The greedy/random pair brackets nothing.** The task asked whether the two figures bracket anything meaningful. They do not: a third trivial policy escapes the bracket from above at 2p (77.5% > 66%) and falls below `random` at 5–6p (0% < 3%). The interval `[random, greedy]` is not a range of achievable play; it is two arbitrary points. 3. **`GreedyPolicy` never passes.** `choose` ignores `may_pass` (`bot.rs:388-405`) and always returns a `Command`. At Reveal the driver loops while anything is legal (`bot.rs:501-510`), so greedy takes *every* available Reveal action — Bonds, GROUND modes, DARVO targets — until the offer set empties. That is not "the obvious action"; it is maximal action. The survey's honest reading at §4 (*"a bot that takes the obvious action"*) understates what is being measured. **Where this challenge stops, and it stops in the author's favour.** The 6-seat row survives, for a reason the survey never gives. At 6p available points are 12 and greedy's mean total is **12.0 with 200/200 board clears** — greedy attains the **theoretical maximum in every single deal**. No policy can beat it, so *"a greedy bot wins 200/200 at six seats"* is not a bot claim at 6p: it is the claim that the threshold (9) sits below a ceiling (12) that ordinary play reaches every time. **That argument is available in the harness's own output and the survey does not make it** — it concedes the ground at §4 and §6 instead. Make it, and the 6-seat finding is defensible on rules grounds. The 2p, 3p and 4p rows are not, and 5p (186/200 clears) is intermediate. **Required:** restate §1.1 as a claim about seat counts 5–6 only, supported by the ceiling argument, and withdraw the 66%/82.5%/95% figures as difficulty statements — or ship the bracket (≥3 policies) and name the number `greedy-200seed-win-rate`, which §5's own benchmark row already demands and §1.1 does not do. ## C5 — "It explains the maintainer's report" — the repo already contains a better explanation, and it is in a doc comment the survey did not read §1.1: > *"It also **explains the maintainer's report** … He plays at low seat > counts, where 66% is a real game, and had been feeling the 5–6 seat > experience from elsewhere in the same session."* `games/ground/src/lib.rs:2487-2498`: > *"This test used to assert the opposite, and it was right to: **the > maintainer played several 3-player games on 2026-08-03 and could not win > any of them**, because GR-S01 dealt 2/3/4 Problems worth 3/6/10 against > thresholds of 5/7/9."* At 3 players on the pre-ruling deal there were **6 points on the table against a threshold of 7**. The games he lost were not 66%-likely; they were **arithmetically unwinnable**, at 3 seats, and the deal was ruled the next day (`2da19a4`, 2026-08-04). *"I felt it was too easy but then we lost, so who knows"* is fully explained by: the pre-ruling deal made losing certain, and the ruling that fixed it landed after he played. The survey's explanation requires (a) that he plays at 2 seats, (b) that 66% is the relevant rate, and (c) an unevidenced claim that he *"had been feeling the 5–6 seat experience from elsewhere in the same session"*. None of the three is sourced anywhere in the repo. The competing explanation is sourced, dated, and sitting in the test that was inverted because of it. This matters beyond tidiness: §1.1's claim to *explain the report* is what elevates the finding from "a bot measurement" to "the answer to the maintainer's question", and it is the sentence CB-WP-0025:141-143 repeats. If the report is already explained by a bug that is already fixed, then the 5–6 seat finding is a **new, separate** finding and should be reported as one — which is a better outcome for the pass, not a worse one. **Required:** delete the explanation, or ask him. §6 already concedes the one-question experiment was not run (*"Nobody has asked him"*) — that concession applies to this sentence too, and §1.1 states as settled what §6 lists as unsettled. ## C6 — "Exhaustive search is out at any seat count" is false at two seats, and it is answering a question T05 does not ask §1.2: *"An exhaustive search from round 1 at 3 seats is roughly `7.4^15` — not a number worth writing down. **Exhaustive search is out at any seat count**, and this was measured rather than assumed."* The exponent survives (see §What survives). Three things about the conclusion do not. **(a) Two seats.** Measured decisions per game at 2p: 462/40 = 11.6, mean width 4.7. `4.7^11.6 ≈ 5.8 × 10⁷` nodes. At C1's corrected ~17 µs that is **~16 minutes**, single-threaded, no pruning, no transposition. Slow, and plainly not "out". The survey computed the 3-seat figure and generalised to "any seat count" without computing the 2-seat one — at the seat count §1.1 says the maintainer plays. **(b) The wrong root.** T05's feature runs on **a recorded lost game**. Nobody asks "could we have won from round 1"; they ask it after the loss, and the useful witness starts at the divergence, typically the last one or two rounds. Depth 2 rounds × 3 seats = 6: `7.4^6 ≈ 1.6 × 10⁵` nodes × ~19 µs = **~3 seconds**. Exhaustive search over the final two rounds is affordable *today*, at 3 seats, with no algorithm at all. That is a materially different ADR than "bounded search, PIMC or ISMCTS, 10⁴ nodes". **(c) `branching^depth` is the tree, not the state space.** GROUND is co-operative — all seats share one objective, so this is single-agent planning, not adversarial search, and single-agent planning transposes. The state that determines the answer is roughly (round, claimed-set, hands, stress): at 5–6 seats the claimed-set is a subset of **five** Problems. `2^5 × 5 rounds = 160` scoring-relevant classes. A search that memoises collapses `7.4^15` to something that does not need a bound at all. The survey does not mention memoisation, transposition, or the co-operative structure once, and it rules out the class of search that would benefit most from all three. **Required:** compute the exponent for the question T05 asks (search from a recorded state, not from round 1), state the 2-seat figure, and either argue that transposition does not help here or stop concluding "exhaustive is out **at any seat count**". §5's `search cost` benchmark row inherits the wrong bound as written. ## C7 — (weak) The replay loop discards rejections silently, but currently has none `difficulty-baseline.rs:121`: `if let Ok(events) = replay.validate(...)`. A rejection is dropped on the floor: the replay state would stop advancing, `legal_commands` would then be sampled on a **stale state**, and the branching figures would be measured against a game that had diverged from the one `play` produced. Nothing reports it. **Mutation, and it clears the author.** Counting the two arms: ``` 2p replay validate: 1062 ok / 0 REJECTED 3p replay validate: 1249 ok / 0 REJECTED 4p replay validate: 1470 ok / 0 REJECTED ``` The replay is faithful today, so no quoted number is contaminated. Marked **weak**: this is a missing control, not a wrong number, and it costs one `else { panic! }` to close. Do not spend the response round on it beyond adding the arm. --- ## Verdict **Not approvable as written.** C1, C2, C3 and C4 each require a change to the survey, not a clarification. C5 requires a deletion. C6 requires a recomputation. | # | verdict | |---|---| | **C1** | **lands hardest.** `legal_commands` is 15.6–20.4 µs, not 112–161. The timer brackets two deals, a whole greedy game and a full replay, over a player-decision denominator. The claimed number varies *inversely* with the mechanism given for it, and moves 161→112 between runs of the unmodified example. The ADR's node budget is off by ~10×. | | **C2** | **lands, equally hard on the finding.** 3p and 4p share a ratio and differ by 12.5 points of win rate — the survey's own table falsifies "the ratio explains the curve". Row-level slack is 1.00/1.29/1.20, not monotone. The separating experiment was never run, and T06 ships this to another repo. | | **C3** | **lands.** Zero assertions, no `--self-test`, not in `make`; "can fail" is asserted about an artifact that cannot go red. And 6/9/12 are sums, printed by nothing — the exact shape GameDesign §1.2 forbids, for the third time. | | **C4** | **lands.** A policy with no heuristic scores 77.5% at 2p against greedy's 66%. §6's "strongest counter" is not open, it is demonstrated. Partly self-repairing: the 6-seat row is rescuable by a ceiling argument the survey has the data for and does not make. | | **C5** | **lands, narrow.** The maintainer's lost games are already explained by the pre-ruling deal (`lib.rs:2489`, 3p, 6 points against a threshold of 7). §1.1 states as settled what §6 lists as unasked. | | **C6** | **lands, moderate.** "Out at any seat count" is ~16 min at 2p and ~3 s over the last two rounds at 3p — and the co-operative, small-state-space structure that makes memoisation work is never mentioned. | | **C7** | **weak.** Missing control, currently clean (0 rejections at all three seat counts). | **What survives.** Four things were attacked and held: - **`nodes` and `widths.len()` are the same denominator.** Both are incremented inside the same `if let Actor::Player` arm (`:118-120`), and the printed `n` equals the divisor on every run (462/649/870). What would have falsified it: `nodes` counting system steps too, which would have deflated the per-node figure by a further ~2.3×. It does not. - **No games were dropped.** Instrumenting all three `continue` arms gives `setup 0, play 0, outcome 0` at every seat count — the quoted `/200` denominators are genuinely 200. What would have falsified it: any non-zero drop, which would have made "200 of 200" a survivor-biased rate. There is none. *(The control is still absent — C3.)* - **`GreedyPolicy` is a real heuristic, not first-legal.** C4's `FirstLegal` mutant diverges from it at every seat count (0% vs 100% at 6p). What would have falsified it: the two policies producing identical rates, which would have meant the ranking at `bot.rs:326` was inert. - **The 6/9/12 arithmetic itself, and the 5×N exponent.** Surface 2 + hidden {2,2}/{2,2,3}/{2,2,3,3} = 6/9/12 against 5/7/9 checks out against `Problems.csv` (all four scenarios carry the identical value vector) and `edition.rs:119-130`. Measured decisions per game — 11.6/16.2/21.8 vs the survey's 5×N of 10/15/20 — are within 10%, so C6 attacks the conclusion, not the exponent. What would have falsified either: a scenario in the edition with different values, or a decision count far from 5N. Neither exists. **The single challenge that forces a change to the design: C1.** Every other challenge changes what the survey *says*. C1 changes what T03 will *decide*: the affordable budget is ~10⁵–10⁶ nodes, not ~10⁴–10⁵, and at that budget C6's exhaustive-over-the-last-two-rounds search comes into range — which is a different ADR, with a different honesty story, than a bounded PIMC/ISMCTS design chosen because exhaustive was ruled out. The number that "decides this pass" was measured around the wrong brackets.