clay-borg/history/260805-could-we-have-won-response.md
tegwick 1f0f652920
Some checks are pending
ci / check (push) Waiting to run
CB-WP-0025 T02: the review withdrew the finding, and the fifth wrong
premise never left the repo

Separate agent, second tier-L review in this project. Six of seven
challenges conceded. The survey's headline finding is WITHDRAWN, not
softened.

C4 kills it, and the reviewer ranked it fourth. A FirstLegal policy --
take legal[0], no heuristic at all -- scores 0% at five and six seats
where GreedyPolicy scores 100%, and 77.5% at two seats where greedy scores
66%. Two unsophisticated agents span the entire range at the same seat
count. "The game is too easy at 5-6 seats" is therefore a statement about
GreedyPolicy, not about GROUND. The rescue the reviewer offered -- greedy
hits the 12-point ceiling in 200/200 deals, so the 6p row is a rules claim
-- dies on the same data: FirstLegal reaches that ceiling never.

C1: the per-node cost was wrong by 30-50x. The timer started before the
seed loop, so "us/node" included two setups, an entire greedy game and a
full validate+fold replay, divided by player-decision count. The tell was
in my own published output and I did not look at it: the figure FELL
(161/139/112) as branching ROSE (4.7/7.4/9.1), which no per-enumeration
cost can do. Re-measured with the clock around legal_commands alone:
3.0/3.5/4.1 us, now rising with branching. The reviewer measured 15.6-20.4
by a different isolation; we disagree by ~5x and neither has established
which is right, so T04 must benchmark it with criterion rather than adopt
either number.

C6: "exhaustive search is out at any seat count" is false -- ~3 seconds
over the last two rounds at 3p. With C1's correction the budget is
~10^5-10^6 nodes and bounded endgame search fits, so ADR-0013 cannot open
with "exhaustive is impossible, therefore determinized sampling" --
especially as sampling carries strategy fusion that exhaustive search does
not.

C3: the finding failed the admissibility rule this project wrote nine
hours earlier. 6/9/12 are sums where GROUND-WP-0004 T02 requires
per-priority rows, and the harness has no assertions, no --self-test and
no make target, so nothing can turn it red -- a `default` artifact wearing
a `counterexample` label, by CB-WP-0022 T05's own distinction.

C2: the ratio story explains nothing; 3p and 4p share deal, threshold and
ratio and differ by 12.5 points of win rate. C5: "explains the
maintainer's report" is contradicted by lib.rs:2487, which records his
losses as 3-player games on the pre-ruling deal, arithmetically unwinnable
at 6 against 7.

T06 exists to report to GROUND-WP-0005, which is BLOCKED waiting on a
difficulty baseline. Had this proceeded they would have been invited to
move thresholds on the strength of one bot's behaviour. That is the fifth
wrong premise this project would have sent them, and the second stopped by
an adversarial review rather than by a control. Both tier-L reviews here
have now caught a false headline that every gate passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 18:06:03 +02:00

9.2 KiB
Raw Blame History

260805 — response to the challenge on CB-RES-0008

One round, per InnerLoop §Step 2. Separate agent, as with CB-RES-0007.

Six challenges conceded, one noted. The survey's headline finding is withdrawn. And one challenge lands harder than the reviewer scored it — C4, which they ranked fourth, is the one that kills the claim.


C1 — the timer measured almost everything except the thing it named — conceded

difficulty-baseline.rs started the clock before the seed loop, so "µs/node" included two setups, an entire greedy game, and a full validate+fold replay — then divided by the number of player decisions.

The tell was in my own published output and I did not look at it. The figure fell as seat count rose — 161 → 139 → 112 — while branching rose 4.7 → 7.4 → 9.1. A per-enumeration cost cannot do that. The denominator was growing faster than the numerator because the numerator was mostly fixed per-game work.

Re-measured with the clock bracketing legal_commands and nothing else:

  2p  462 decisions   branching mean 4.7  median 5  max 7    3.0 us/call
  3p  649 decisions   branching mean 7.4  median 8  max 10   3.5 us/call
  4p  870 decisions   branching mean 9.1  median 10  max 12  4.1 us/call

Now it rises with branching, which is the direction the mechanism predicts.

A discrepancy I am not going to paper over. The reviewer measured 15.620.4 µs; I measure 3.04.1 µs bracketing each call with Instant::now(). We disagree by ~5×, which is small against the 3050× correction we agree on but is not nothing. Two different isolation methods gave two different answers and neither of us has established which is right — so T04 must settle the per-node cost with a proper benchmark (this repo has criterion already), not with either of these numbers.

This is the wrong-denominator family again — CB-EV-0019 §1 named it with three instances, and this is a fourth, in the numerator this time: a number computed correctly over the wrong span.

C4 — conceded, and the reviewer under-ranked it. This is the one that kills the finding.

They ranked it fourth. Measured, it is first.

A FirstLegal policy — take legal[0], no heuristic at all — against GreedyPolicy, 200 seeds each:

seats greedy firstlegal
2 66.0% 77.5%
3 82.5% 25.0%
4 95.0% 32.0%
5 100.0% 0.0%
6 100.0% 0.0%

At five and six seats, one unsophisticated policy wins every game and another wins none. The spread between two dumb agents at the same seat count is the entire range.

The survey said "a greedy bot wins 200 of 200 at five and six seats" and concluded the game is too easy there. That conclusion does not follow from a measurement whose value ranges over 0100% depending on which trivial agent you pick. It is a statement about GreedyPolicy, not about GROUND.

The reviewer offered me a rescue and it does not survive either. They suggested that greedy hitting the 12-point ceiling in 200/200 deals makes the 6p row a rules claim — the ceiling is reachable, therefore the game is clearable. FirstLegal's 0% at 6p refutes it: reachable-by-greedy is not reachable-by-an-agent, and "the ceiling can be hit" was never in doubt anyway. What was claimed is that it is easy to hit, and nothing here supports that.

And at two seats greedy is worse than no heuristic at all (66% vs 77.5%), so the 66% figure — which I used as "a real game" — is partly a GreedyPolicy defect, not a property of the deal.

The headline finding is withdrawn. Not softened: withdrawn.

C2 — the ratio does not explain the curve — conceded

I wrote "three multipliers all pointing the same direction" over a table of ratios 1.20 / 1.29 / 1.33.

3-player and 4-player share the same deal, the same threshold and therefore the same ratio (9 points against 7, 1.29) — and differ by 12.5 points of win rate (82.5% vs 95%). My own harness prints of 7.0 for both. A quantity identical across two rows cannot explain a difference between them.

The mechanism the reviewer identifies — more hands claiming a fixed five-Problem pool — is consistent with the data and mine was not. I constructed a story from a table I had built and did not test it against the rows sitting next to each other in my own output.

C3 — the finding is inadmissible under GameDesign §1 — conceded

The survey claimed its finding met the admissibility rule this project wrote nine hours earlier. It fails two of three clauses:

  • Clause 2 (ruled shape). 6/9/12 are sums. GROUND-WP-0004 T02 requires Surface and each hidden priority listed separately, and explicitly forbids "deal depth N". My table has a hidden dealt column containing a count. That is the exact shape the ruling exists to prohibit, and I reproduced it while citing the ruling.
  • Clause 3 (can fail). The harness has no assertions, no --self-test, and is not in any make target. Nothing can turn it red. Under CB-WP-0022 T05's own distinction it is a default-role artifact — it prints what the code does — wearing a counterexample label.

A reproduction that cannot fail is a printout. The rule was one day old and I broke it in the first pass that used it.

C5 — the "explains the maintainer's report" claim is contradicted by this repo — conceded

I wrote that 66%-at-2p and 100%-at-6p together explain "I felt it was too easy but then we lost."

games/ground/src/lib.rs:2487-2493, in a test comment I did not read:

"the maintainer played several 3-player games on 2026-08-03 and could not win any of them, because GR-S01 dealt 2/3/4 Problems worth 3/6/10 against thresholds of 5/7/9."

Three-player games, on the pre-ruling deal, arithmetically unwinnable at 6 against 7. Not 2-player, not a 66% coin-flip, and nothing to do with the curve I fitted to it.

This is the same defect as CB-WP-0022's C1, four days later: a claim about the maintainer's own experience, contradicted by a record in this repo, written by a pass that had the record available. The remedy there was to check git log; the remedy here was to read the test comment attached to the very test that inverted.

C6 — "exhaustive search is out at any seat count" is false — conceded, and it changes the ADR

I asserted 7.4^15 and stopped thinking. The reviewer measured ~16 min for a full 2p game and ~3 s over the last two rounds at 3p.

Combined with C1's correction, the affordable budget is ~10⁵10⁶ nodes, not ~10⁴10⁵, and bounded exhaustive search over the endgame is inside it. That is a different starting point for T03: the ADR cannot open with "exhaustive is impossible, therefore determinized sampling", because the premise is false and the alternative carries strategy fusion that exhaustive search does not.

Neither I nor the reviewer considered transposition or the co-operative single-agent framing, which cut the exponent further.

C7 — the replay loop discards rejections silently — noted

Correct, and the reviewer measured zero rejections, so no published number is contaminated. It is still a silent if let Ok(...), and it will lie the moment the replay diverges. Fixed alongside the rest.

What survives, and what would have falsified it

  • nodes == widths.len() — the denominator is the one the label says.
  • No games were dropped: all 200 seeds completed at every seat count, so the win-rate denominators are real.
  • GreedyPolicy is a genuine heuristic, not first-legal in disguise — which is what makes C4's comparison meaningful rather than trivial.
  • The 6/9/12 arithmetic itself is right against Problems.csv and GR-S01. It is the shape that fails C3 and the interpretation that fails C2 and C4, not the arithmetic.
  • The 5×N exponent was measured at 11.6/16.2/21.8 against my claimed 10/15/20 — close enough that the exponent was not where C6's error was.

What the review changed

before after
headline finding the game is too easy at 56 seats withdrawn — one dumb policy wins 100%, another 0%
per-node cost 112161 µs 3.04.1 µs (mine) vs 15.620.4 (theirs); unsettled, T04 must benchmark
exhaustive search ruled out available for bounded endgame; ADR's premise changes
the ratio story three multipliers 3p and 4p share the ratio and differ by 12.5 points
admissibility claimed fails clauses 2 and 3 of a rule one day old
explains the maintainer asserted contradicted by lib.rs:2487

The number was going to ground-game. T06 exists to report to GROUND-WP-0005, which is blocked waiting on a difficulty baseline. Had this pass proceeded, they would have been told the game is too easy at five and six seats and invited to move thresholds — on the strength of one bot's behaviour, when a second bot of the same sophistication scores zero.

That would have been the fifth wrong premise this project has sent them, and the second stopped by an adversarial review rather than by a control. Two for two: both tier-L reviews in this project have caught a false headline that every gate passed.