clay-borg/history/260805-could-we-have-won-challenge.md

260805 — challenge to CB-RES-0008

Adversarial review, one round, per InnerLoop §Step 2. Target: the survey research/CB-RES-0008-could-we-have-won.md, the harness games/ground/examples/difficulty-baseline.rs, and the T01 §judgment in CB-WP-0025:126-157.

Fidelity note, first. Run in a separate agent session with only the files. There is one harness and one repo, so per §Step 2 this review inherits the author's sampling and does not report a clean verify on that basis. What it substitutes is mutation: every quoted number was traced to the expression that produces it, and the expression was changed. A mutant copy of the harness (examples/zz-review-mutant.rs) was written, run, and deleted; git status is clean and no file in the repo was modified. Every number below is reproducible by re-creating that mutant from the diffs quoted inline.

Three of the four headline claims move under mutation. One of them moves by a factor of seven.

Ranked. C1 and C2 land hardest. C7 is marked weak.

---

C1 — legal_commands does not cost 112–161 µs. It costs 15.6–20.4 µs. The timer is around the wrong thing.

The survey, §1.2:

"legal_commands costs 112–161 µs per call, because it constructs candidates and filters them through full validate. … it is the number that decides this pass."

Read the timer's scope. difficulty-baseline.rs:100 starts the clock:

100:    let start = std::time::Instant::now();
102:    for seed in 0..40u64 {
103:        let Some(state) = setup(players, seed) else { continue };   // deal #1
108:        let Ok(game) = play(state, &mut ps) else { continue };      // A WHOLE GREEDY GAME
112:        let Some(mut replay) = setup(players, seed) else { continue };// deal #2
115:        for (actor, cmd) in &game.steps {
118:                let legal = legal_commands(&replay, *seat);          // the thing being claimed
121:            if let Ok(events) = replay.validate(*actor, cmd) {       // + full replay
122:                for e in &events { replay.fold(e); }
129:    let elapsed = start.elapsed();
143:        per = elapsed.as_micros() as f64 / nodes as f64,

elapsed is two setups, a complete five-round greedy game (which itself calls legal_commands on every step and runs the whole ranking loop), a second deal, and a full validate+fold replay of every command, player and system alike — divided by the count of player decision points only. It is not a per-node search cost. It is a per-game cost with a per-node denominator.

Mutation. Bracket legal_commands alone (Instant::now() immediately before line 118, accumulate on the next line), and separately time the GroundState::clone a real search must also pay per node:

  2p  462 nodes  mean width 4.7   ORIGINAL 112 us   legal_commands ALONE 15.6 us   clone 1.3 us
  3p  649 nodes  mean width 7.4   ORIGINAL 115 us   legal_commands ALONE 18.0 us   clone 1.4 us
  4p  870 nodes  mean width 9.1   ORIGINAL 116 us   legal_commands ALONE 20.4 us   clone 1.4 us

The claimed number is 6–8× the measured one. And the mutant supplies the diagnostic that should have caught it in the survey: the survey's figures fall with seat count (161 → 139 → 112) while branching rises (4.7 → 7.4 → 9.1). If the number were the cost of constructing and validating candidates, it would rise with the number of candidates. legal_commands alone does exactly that — 15.6 → 18.0 → 20.4. The survey's number falls because the fixed per-game overhead is being amortised over more nodes at higher seat counts. The reported quantity varies inversely with the mechanism the prose gives for it, in the survey's own printed table, and the survey did not notice.

Secondary, and it is enough on its own. The number is not stable. Three runs of the unmodified example on this machine:

run 1 (survey)  2p 161   3p 139   4p 112
run 2           2p 132   3p 128   4p 139
run 3 (mutant)  2p 112   3p 115   4p 116

The 2p figure moved 161 → 112 between runs, and the seat-count ordering inverted. The survey quotes 112–161 µs/node in three places (:44, :83, :189) as a measured range across seat counts. It is a range across runs, of a quantity that is mostly loop overhead. Anyone can settle this: run the example twice.

What this changes in the design, which is why it is C1. §1.2 concludes "at ~140 µs/node: a bounded search of ~10⁴–10⁵ nodes costs 1.4–14 seconds. That is the budget the ADR has to design inside." At the measured ~17 µs for legal_commands plus ~1.4 µs for the clone, ~19 µs/node, the same 1.4–14 s buys ~10⁵–10⁶ nodes. T03 is about to pick a node bound one order of magnitude too small, and T04 is about to write it into an acceptance table.

Required: re-scope the timer to the call the prose names, re-quote, and state the run-to-run spread rather than a single range. If the intent was "the cost of one node of a search that replays from a scenario", say so and price the clone-and-fold node separately — but then it is not legal_commands's cost and §1.2's causal sentence must go.

C2 — The ratio does not explain the curve, and the survey's own table proves it: two rows with an identical ratio are 12.5 points apart

§1.1 is the pass's headline finding:

"The ratio moves the wrong way. … Three multipliers all pointing the same direction, which is why the curve is not gentle — it is 66% → 100% across four seat counts."

seats	ratio (survey)	greedy win rate
2	1.20	66.0%
3	1.29	82.5%
4	1.29	95.0%
5	1.33	100%
6	1.33	100%

3p and 4p have the same deal, the same threshold, and the same ratio, and differ by 12.5 points of win rate — a jump as large as either of the two between-ratio jumps. The ratio takes three distinct values across five rows; seat count takes five. The explanatory variable and the confound are not separated anywhere in the survey, and the one comparison that separates them (3p vs 4p) points at seat count, not at the ratio.

It gets worse for the ratio when the arithmetic is done at row level, which §1.1 claims to have done and has not (see C3). Claimable point values are {2, 2, 2} at 2p, {2, 2, 2, 3} at 3–4p, {2, 2, 2, 3, 3} at 5–6p (editions/ground-darvo-r0/Problems.csv, priorities 0–4 of every scenario; all four scenarios carry the same value vector, checked). So the achievable totals are not continuous, and the useful quantity is how much of the board must be claimed:

seats	achievable totals	threshold	Problems that must be claimed	effective slack
2	0, 2, 4, 6	5	3 of 3 — a full clear	1.00
3–4	0, 2, 3, 4, 5, 6, 7, 9	7	3 of 4, incl. the 3-pointer	1.29
5–6	0 … 12	9	4 of 5	1.20

A threshold of 5 at 2p is identical to a threshold of 6 — nothing sums to 5. The survey's 1.20 is not slack; the real slack at 2p is 1.00, and games/ground/src/lib.rs:2504 already says so in as many words ("2+2+2 against a threshold of 5 means a full clear"). And on this measure the sequence is 1.00 / 1.29 / 1.20 — not monotone, so "the ratio moves the wrong way" reverses at the seat count the finding is loudest about.

Confirmed directly. Instrumenting the harness to record whether the group claimed every Problem on the board:

  2p greedy  132/200 won   cleared-the-board 132/200   AVAIL 6
  3p greedy  165/200 won   cleared-the-board 161/200   AVAIL 9
  4p greedy  190/200 won   cleared-the-board 190/200   AVAIL 9
  5p greedy  200/200 won   cleared-the-board 186/200   AVAIL 12
  6p greedy  200/200 won   cleared-the-board 200/200   AVAIL 12

At 2p, wins == board-clears exactly (132 = 132): the 2-player game is not "a real game at 66%", it is pass/fail on a full clear. The mechanism driving the curve is the third item the survey lists last and never quantifies — more seats means more hands, which means more matching Solutions, which means a higher fraction of a fixed pool gets claimed. More seats add no points; they add claimants for the same 5 Problems.

Required: withdraw "three multipliers all pointing the same direction" or measure it. The separating experiment is cheap and was not run: hold seats fixed and move the threshold, or hold the threshold fixed and move hidden_depth (games/ground/src/edition.rs:123-130). Until one of those runs, the finding handed to GROUND-WP-0005 tells them to tune the wrong dial — and T06 ships it into another repo.

C3 — The finding is inadmissible under GameDesign §1 on two of the three clauses, and the survey asserts all three

§1.1:

"This is admissible under GameDesign §1: the reproduction exists (examples/difficulty-baseline.rs), it has the ruled shape (row-level, no sums), and it can fail — change a threshold and the numbers move."

Clause 3, "can fail" — fails. difficulty-baseline.rs contains zero assertions:

$ grep -c "assert" games/ground/examples/difficulty-baseline.rs
0
$ grep -rn "difficulty" Makefile gates.toml facts.toml
(no output)

It prints and exits 0. Delete the win-rate line, change a threshold, break the deal — it still exits 0. "The numbers move" is sensitivity, not failability; GameDesign §1.3 requires "the artifact must be capable of going red, and the register records its current colour." An artifact with no assertion has no colour. It is not a counterexample (nothing alarms) and it is not a default (it encodes no choice) — it is a third thing, an observation, and CB-WP-0022 T05's role table has no row for it. That is the distinction this repo paid for four days ago.

This also violates InnerLoop §Measurement validity twice over: "every tool that reports a number exposes --self-test, and that self-test runs before the number is produced" — there is none, and the example is not wired into make at all — and "state the divisor used to convert raw timings into the metric's unit, pinned by a test" — nodes at :143 is unpinned. Mutation: run the win-rate loop over 40 seeds instead of 200 (const SEEDS: u64 = 40):

  6p greedy   40/40  = 100.0%   mean total 12.0 of  9.0   median margin +3

Nothing in the harness objects to a denominator that shrank by 5×. Three continues (:57, :62, :65) can silently drop games out of played and there is no assert_eq!(played, SEEDS). (To the author's credit, played is printed, and on the current run it is genuinely 200/200 — see §What survives. The control is missing, not the work.)

Clause 2, "the ruled shape" — fails. GameDesign §1.2, quoting GROUND-WP-0004 T02: "an arithmetic finding ships a row-level table — Surface and each hidden priority listed separately — never 'sum of file'", and "the artifact prints the rows it came from. A finding stating a total without its rows is inadmissible even if the total is right."

The survey's table is:

seats	Surface	hidden dealt	points available	threshold	ratio
2	priority 1	2	6	5	1.20

6, 9, 12 are sums. The hidden priorities are collapsed into a count (hidden dealt: 2), which is the exact shape the ruling forbids. The Surface column says "priority 1" in all three rows, and Surface is hidden_priority 0 in Problems.csv — the cell is either wrong or meaningless, and it is the cell the ruled shape is about.

And the named artifact prints none of this. difficulty-baseline.rs never prints available points, thresholds by row, or priorities; it prints win rates and timings. The reproduction cited for the 6/9/12 finding does not compute 6/9/12. The thing that does is games_ground::gd0001_group_success_is_reachable_at_every_seat_count (lib.rs:2508) — which the survey does not name, and which also sums (lib.rs:2512: .map(|p| p.value).sum()).

Required: either name gd0001 and give it the row-level print the ruling requires, or drop the admissibility claim. This is the third time a correct total has shipped with the wrong shape ("12 in the file", "4/6/9", and now this), and it is the failure GameDesign §1.2 was written this week to stop.

C4 — 66% at two seats is a GreedyPolicy defect, not a difficulty. A policy with no heuristic at all scores 77.5%.

The task the survey sets itself in §4 and defers in §6 — "whether a bot win rate is a difficulty at all" — is settled against it by one mutation. Add two zero-knowledge policies: FirstLegal (always Choice::Command(0)) and LastLegal (always the last offered command), 200 seeds each:

  2p greedy 132/200 = 66.0%    2p first 155/200 = 77.5%   2p last 0/200   2p random  10/200 = 5.0%
  3p greedy 165/200 = 82.5%    3p first  50/200 = 25.0%   3p last 0/200   3p random  19/200 = 9.5%
  4p greedy 190/200 = 95.0%    4p first  64/200 = 32.0%   4p last 0/200   4p random  16/200 = 8.0%
  5p greedy 200/200 = 100.0%   5p first   0/200 =  0.0%   5p last 0/200   5p random   6/200 = 3.0%
  6p greedy 200/200 = 100.0%   6p first   0/200 =  0.0%   6p last 0/200   6p random   7/200 = 3.5%

At two seats, taking the first command in canonical order beats the stated heuristic by 11.5 points. So:

1. The 66% is not the game's 2-player difficulty. It is the point at which GreedyPolicy's ranking (bot.rs:326-380) becomes worse than no ranking. Somebody improving the bot next week moves this row to ~78% and "the 2-player game got easier" without a rule changing — which is the objection §6 says is the strongest and leaves open. It is not open; it is demonstrated, on the row the survey leans on hardest.
2. The greedy/random pair brackets nothing. The task asked whether the two figures bracket anything meaningful. They do not: a third trivial policy escapes the bracket from above at 2p (77.5% > 66%) and falls below random at 5–6p (0% < 3%). The interval [random, greedy] is not a range of achievable play; it is two arbitrary points.
3. GreedyPolicy never passes. choose ignores may_pass (bot.rs:388-405) and always returns a Command. At Reveal the driver loops while anything is legal (bot.rs:501-510), so greedy takes every available Reveal action — Bonds, GROUND modes, DARVO targets — until the offer set empties. That is not "the obvious action"; it is maximal action. The survey's honest reading at §4 ("a bot that takes the obvious action") understates what is being measured.

Where this challenge stops, and it stops in the author's favour. The 6-seat row survives, for a reason the survey never gives. At 6p available points are 12 and greedy's mean total is 12.0 with 200/200 board clears — greedy attains the theoretical maximum in every single deal. No policy can beat it, so "a greedy bot wins 200/200 at six seats" is not a bot claim at 6p: it is the claim that the threshold (9) sits below a ceiling (12) that ordinary play reaches every time. That argument is available in the harness's own output and the survey does not make it — it concedes the ground at §4 and §6 instead. Make it, and the 6-seat finding is defensible on rules grounds. The 2p, 3p and 4p rows are not, and 5p (186/200 clears) is intermediate.

Required: restate §1.1 as a claim about seat counts 5–6 only, supported by the ceiling argument, and withdraw the 66%/82.5%/95% figures as difficulty statements — or ship the bracket (≥3 policies) and name the number greedy-200seed-win-rate, which §5's own benchmark row already demands and §1.1 does not do.

C5 — "It explains the maintainer's report" — the repo already contains a better explanation, and it is in a doc comment the survey did not read

§1.1:

"It also explains the maintainer's report … He plays at low seat counts, where 66% is a real game, and had been feeling the 5–6 seat experience from elsewhere in the same session."

games/ground/src/lib.rs:2487-2498:

"This test used to assert the opposite, and it was right to: the maintainer played several 3-player games on 2026-08-03 and could not win any of them, because GR-S01 dealt 2/3/4 Problems worth 3/6/10 against thresholds of 5/7/9."

At 3 players on the pre-ruling deal there were 6 points on the table against a threshold of 7. The games he lost were not 66%-likely; they were arithmetically unwinnable, at 3 seats, and the deal was ruled the next day (2da19a4, 2026-08-04). "I felt it was too easy but then we lost, so who knows" is fully explained by: the pre-ruling deal made losing certain, and the ruling that fixed it landed after he played.

The survey's explanation requires (a) that he plays at 2 seats, (b) that 66% is the relevant rate, and (c) an unevidenced claim that he "had been feeling the 5–6 seat experience from elsewhere in the same session". None of the three is sourced anywhere in the repo. The competing explanation is sourced, dated, and sitting in the test that was inverted because of it.

This matters beyond tidiness: §1.1's claim to explain the report is what elevates the finding from "a bot measurement" to "the answer to the maintainer's question", and it is the sentence CB-WP-0025:141-143 repeats. If the report is already explained by a bug that is already fixed, then the 5–6 seat finding is a new, separate finding and should be reported as one — which is a better outcome for the pass, not a worse one.

Required: delete the explanation, or ask him. §6 already concedes the one-question experiment was not run ("Nobody has asked him") — that concession applies to this sentence too, and §1.1 states as settled what §6 lists as unsettled.

C6 — "Exhaustive search is out at any seat count" is false at two seats, and it is answering a question T05 does not ask

§1.2: "An exhaustive search from round 1 at 3 seats is roughly 7.4^15 — not a number worth writing down. Exhaustive search is out at any seat count, and this was measured rather than assumed."

The exponent survives (see §What survives). Three things about the conclusion do not.

(a) Two seats. Measured decisions per game at 2p: 462/40 = 11.6, mean width 4.7. 4.7^11.6 ≈ 5.8 × 10⁷ nodes. At C1's corrected ~17 µs that is ~16 minutes, single-threaded, no pruning, no transposition. Slow, and plainly not "out". The survey computed the 3-seat figure and generalised to "any seat count" without computing the 2-seat one — at the seat count §1.1 says the maintainer plays.

(b) The wrong root. T05's feature runs on a recorded lost game. Nobody asks "could we have won from round 1"; they ask it after the loss, and the useful witness starts at the divergence, typically the last one or two rounds. Depth 2 rounds × 3 seats = 6: 7.4^6 ≈ 1.6 × 10⁵ nodes × ~19 µs = ~3 seconds. Exhaustive search over the final two rounds is affordable today, at 3 seats, with no algorithm at all. That is a materially different ADR than "bounded search, PIMC or ISMCTS, 10⁴ nodes".

(c) branching^depth is the tree, not the state space. GROUND is co-operative — all seats share one objective, so this is single-agent planning, not adversarial search, and single-agent planning transposes. The state that determines the answer is roughly (round, claimed-set, hands, stress): at 5–6 seats the claimed-set is a subset of five Problems. 2^5 × 5 rounds = 160 scoring-relevant classes. A search that memoises collapses 7.4^15 to something that does not need a bound at all. The survey does not mention memoisation, transposition, or the co-operative structure once, and it rules out the class of search that would benefit most from all three.

Required: compute the exponent for the question T05 asks (search from a recorded state, not from round 1), state the 2-seat figure, and either argue that transposition does not help here or stop concluding "exhaustive is out at any seat count". §5's search cost benchmark row inherits the wrong bound as written.

C7 — (weak) The replay loop discards rejections silently, but currently has none

difficulty-baseline.rs:121: if let Ok(events) = replay.validate(...). A rejection is dropped on the floor: the replay state would stop advancing, legal_commands would then be sampled on a stale state, and the branching figures would be measured against a game that had diverged from the one play produced. Nothing reports it.

Mutation, and it clears the author. Counting the two arms:

  2p  replay validate: 1062 ok / 0 REJECTED
  3p  replay validate: 1249 ok / 0 REJECTED
  4p  replay validate: 1470 ok / 0 REJECTED

The replay is faithful today, so no quoted number is contaminated. Marked weak: this is a missing control, not a wrong number, and it costs one else { panic! } to close. Do not spend the response round on it beyond adding the arm.

---

Verdict

Not approvable as written. C1, C2, C3 and C4 each require a change to the survey, not a clarification. C5 requires a deletion. C6 requires a recomputation.

#	verdict
C1	lands hardest. legal_commands is 15.6–20.4 µs, not 112–161. The timer brackets two deals, a whole greedy game and a full replay, over a player-decision denominator. The claimed number varies inversely with the mechanism given for it, and moves 161→112 between runs of the unmodified example. The ADR's node budget is off by ~10×.
C2	lands, equally hard on the finding. 3p and 4p share a ratio and differ by 12.5 points of win rate — the survey's own table falsifies "the ratio explains the curve". Row-level slack is 1.00/1.29/1.20, not monotone. The separating experiment was never run, and T06 ships this to another repo.
C3	lands. Zero assertions, no --self-test, not in make; "can fail" is asserted about an artifact that cannot go red. And 6/9/12 are sums, printed by nothing — the exact shape GameDesign §1.2 forbids, for the third time.
C4	lands. A policy with no heuristic scores 77.5% at 2p against greedy's 66%. §6's "strongest counter" is not open, it is demonstrated. Partly self-repairing: the 6-seat row is rescuable by a ceiling argument the survey has the data for and does not make.
C5	lands, narrow. The maintainer's lost games are already explained by the pre-ruling deal (lib.rs:2489, 3p, 6 points against a threshold of 7). §1.1 states as settled what §6 lists as unasked.
C6	lands, moderate. "Out at any seat count" is ~16 min at 2p and ~3 s over the last two rounds at 3p — and the co-operative, small-state-space structure that makes memoisation work is never mentioned.
C7	weak. Missing control, currently clean (0 rejections at all three seat counts).

What survives. Four things were attacked and held:

• nodes and widths.len() are the same denominator. Both are incremented inside the same if let Actor::Player arm (:118-120), and the printed n equals the divisor on every run (462/649/870). What would have falsified it: nodes counting system steps too, which would have deflated the per-node figure by a further ~2.3×. It does not.
• No games were dropped. Instrumenting all three continue arms gives setup 0, play 0, outcome 0 at every seat count — the quoted /200 denominators are genuinely 200. What would have falsified it: any non-zero drop, which would have made "200 of 200" a survivor-biased rate. There is none. (The control is still absent — C3.)
• GreedyPolicy is a real heuristic, not first-legal. C4's FirstLegal mutant diverges from it at every seat count (0% vs 100% at 6p). What would have falsified it: the two policies producing identical rates, which would have meant the ranking at bot.rs:326 was inert.
• The 6/9/12 arithmetic itself, and the 5×N exponent. Surface 2 + hidden {2,2}/{2,2,3}/{2,2,3,3} = 6/9/12 against 5/7/9 checks out against Problems.csv (all four scenarios carry the identical value vector) and edition.rs:119-130. Measured decisions per game — 11.6/16.2/21.8 vs the survey's 5×N of 10/15/20 — are within 10%, so C6 attacks the conclusion, not the exponent. What would have falsified either: a scenario in the edition with different values, or a decision count far from 5N. Neither exists.

The single challenge that forces a change to the design: C1. Every other challenge changes what the survey says. C1 changes what T03 will decide: the affordable budget is ~10⁵–10⁶ nodes, not ~10⁴–10⁵, and at that budget C6's exhaustive-over-the-last-two-rounds search comes into range — which is a different ADR, with a different honesty story, than a bounded PIMC/ISMCTS design chosen because exhaustive was ruled out. The number that "decides this pass" was measured around the wrong brackets.