CB-WP-0025 T02: the review withdrew the finding, and the fifth wrong
Some checks are pending
ci / check (push) Waiting to run

premise never left the repo

Separate agent, second tier-L review in this project. Six of seven
challenges conceded. The survey's headline finding is WITHDRAWN, not
softened.

C4 kills it, and the reviewer ranked it fourth. A FirstLegal policy --
take legal[0], no heuristic at all -- scores 0% at five and six seats
where GreedyPolicy scores 100%, and 77.5% at two seats where greedy scores
66%. Two unsophisticated agents span the entire range at the same seat
count. "The game is too easy at 5-6 seats" is therefore a statement about
GreedyPolicy, not about GROUND. The rescue the reviewer offered -- greedy
hits the 12-point ceiling in 200/200 deals, so the 6p row is a rules claim
-- dies on the same data: FirstLegal reaches that ceiling never.

C1: the per-node cost was wrong by 30-50x. The timer started before the
seed loop, so "us/node" included two setups, an entire greedy game and a
full validate+fold replay, divided by player-decision count. The tell was
in my own published output and I did not look at it: the figure FELL
(161/139/112) as branching ROSE (4.7/7.4/9.1), which no per-enumeration
cost can do. Re-measured with the clock around legal_commands alone:
3.0/3.5/4.1 us, now rising with branching. The reviewer measured 15.6-20.4
by a different isolation; we disagree by ~5x and neither has established
which is right, so T04 must benchmark it with criterion rather than adopt
either number.

C6: "exhaustive search is out at any seat count" is false -- ~3 seconds
over the last two rounds at 3p. With C1's correction the budget is
~10^5-10^6 nodes and bounded endgame search fits, so ADR-0013 cannot open
with "exhaustive is impossible, therefore determinized sampling" --
especially as sampling carries strategy fusion that exhaustive search does
not.

C3: the finding failed the admissibility rule this project wrote nine
hours earlier. 6/9/12 are sums where GROUND-WP-0004 T02 requires
per-priority rows, and the harness has no assertions, no --self-test and
no make target, so nothing can turn it red -- a `default` artifact wearing
a `counterexample` label, by CB-WP-0022 T05's own distinction.

C2: the ratio story explains nothing; 3p and 4p share deal, threshold and
ratio and differ by 12.5 points of win rate. C5: "explains the
maintainer's report" is contradicted by lib.rs:2487, which records his
losses as 3-player games on the pre-ruling deal, arithmetically unwinnable
at 6 against 7.

T06 exists to report to GROUND-WP-0005, which is BLOCKED waiting on a
difficulty baseline. Had this proceeded they would have been invited to
move thresholds on the strength of one bot's behaviour. That is the fifth
wrong premise this project would have sent them, and the second stopped by
an adversarial review rather than by a control. Both tier-L reviews here
have now caught a false headline that every gate passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-08-05 18:06:03 +02:00
parent 469d00d679
commit 1f0f652920
5 changed files with 804 additions and 31 deletions

View file

@ -1,7 +1,7 @@
---
id: CB-RES-0008
capability: analysis.witness-and-difficulty
status: draft — awaiting adversarial review (CB-WP-0025 T02)
status: reviewed 2026-08-05 — headline finding WITHDRAWN (C4); numbers corrected (C1)
tier: L
chaos: d8 = 6 → no override
---
@ -39,14 +39,39 @@ bot win rate over 200 seeds (GR-E01 group success):
4p random 16/200 won = 8.0% mean total 3.2 of 7.0 median margin -4
search cost — legal_commands at every real decision point:
2p 462 decisions branching mean 4.7 median 5 max 7 161 us/node
3p 649 decisions branching mean 7.4 median 8 max 10 139 us/node
4p 870 decisions branching mean 9.1 median 10 max 12 112 us/node
2p 462 decisions branching mean 4.7 median 5 max 7 3.0 us/call
3p 649 decisions branching mean 7.4 median 8 max 10 3.5 us/call
4p 870 decisions branching mean 9.1 median 10 max 12 4.1 us/call
```
### 1.1 The finding this produced before any solver exists
*(Search-cost row re-measured after C1. The first published figures —
161/139/112 "us/node" — timed two `setup`s, a whole greedy game and a
validate+fold replay. See §1.2.)*
**A greedy bot wins 200 of 200 games at five and six seats.** Not 95%,
### 1.1 WITHDRAWN — the finding this claimed, and why it is not one
> **Withdrawn 2026-08-05 by the adversarial review (C4), before it left
> the repo.** The section is kept, struck through, because a claim
> retracted silently is how three earlier wrong premises survived
> (ADR-0012 D5).
>
> A `FirstLegal` policy — take `legal[0]`, no heuristic — scores
> **0% at five and six seats** where `GreedyPolicy` scores 100%, and
> **77.5% at two seats** where greedy scores 66%. Two unsophisticated
> agents span the entire range at the same seat count. **A measurement
> that does that is about the policy, not about the game.**
>
> | seats | greedy | firstlegal |
> |---|---:|---:|
> | 2 | 66.0% | **77.5%** |
> | 5 | **100.0%** | **0.0%** |
> | 6 | **100.0%** | **0.0%** |
>
> The arithmetic below is right; the **interpretation** is not, and the
> table also fails GameDesign §1.2 by reporting **sums** where
> GROUND-WP-0004 T02 requires per-priority rows.
~~**A greedy bot wins 200 of 200 games at five and six seats.**~~ Not 95%,
not 99% — every game, with a median margin of **+3** and mean available
points of **11.812.0 against a threshold of 9**.
@ -64,36 +89,69 @@ table *and* a proportionally lower bar *and* more actions per round to
clear it with. Three multipliers all pointing the same direction, which is
why the curve is not gentle — it is 66% → 100% across four seat counts.
**This is admissible under GameDesign §1**: the reproduction exists
(`examples/difficulty-baseline.rs`), it has the ruled shape (row-level, no
sums), and it can fail — change a threshold and the numbers move.
~~**This is admissible under GameDesign §1.**~~ **It is not (C3), and the
rule it fails was one day old.**
It also **explains the maintainer's report** — *"I felt it was too easy
- **Clause 2, ruled shape.** 6/9/12 are **sums**, and the `hidden dealt`
column is a *count*. GROUND-WP-0004 T02 requires Surface and each hidden
priority listed separately and explicitly forbids "deal depth N". The
table reproduces the prohibited shape while citing the ruling.
- **Clause 3, can fail.** The harness has **no assertions, no
`--self-test`, and is in no `make` target**. Nothing can turn it red.
Under CB-WP-0022 T05's own `role` distinction it is a `default` artifact
— it prints what the code does — wearing a `counterexample` label.
**A reproduction that cannot fail is a printout.** T05 must fix the
harness before any figure from it is quoted again.
~~It also **explains the maintainer's report** — *"I felt it was too easy
but then we lost, so who knows"* — without needing a solver. He plays at
low seat counts, where 66% is a real game, and had been feeling the 56
seat experience from elsewhere in the same session. Both halves of the
sentence are true of different seat counts.
sentence are true of different seat counts.~~
**T06 must report this to GROUND-WP-0005, which is blocked on exactly
this number.**
**Also withdrawn (C5).** `games/ground/src/lib.rs:2487-2493` records what
actually happened: *"the maintainer played several 3-player games on
2026-08-03 and could not win any of them, because GR-S01 dealt 2/3/4
Problems worth 3/6/10 against thresholds of 5/7/9."* Three seats, on the
pre-ruling deal, **arithmetically unwinnable at 6 against 7** — nothing to
do with the curve fitted to it here.
~~**T06 must report this to GROUND-WP-0005.**~~ **It must not.**
GROUND-WP-0005 is blocked waiting on a difficulty baseline, and this is
not one. Sending it would have invited ground-game to move thresholds on
the strength of one bot's behaviour — the **fifth** wrong premise this
project has sent them, and the second stopped by a review rather than by
a control.
### 1.2 What the search-cost numbers rule out
Branching is small — mean 4.7 to 9.1 — but **`legal_commands` costs
112161 µs per call**, because it constructs candidates and filters them
through full `validate`. That is the price of keeping the rules in one
place (ADR: `legal_commands` builds then validates), and it is the number
that decides this pass.
Branching is small — mean 4.7 to 9.1.
A game is 5 rounds × N seats of decisions. An exhaustive search from
round 1 at 3 seats is roughly `7.4^15` — not a number worth writing down.
**Exhaustive search is out at any seat count**, and this was measured
rather than assumed.
> **CORRECTED (C1). The published figure of 112161 µs/node was wrong by
> 3050×**: the timer bracketed two `setup`s, a whole greedy game and a
> validate+fold replay, divided by player-decision count. The tell was in
> the output — it **fell** as branching **rose**, which no per-enumeration
> cost can do.
>
> Re-measured, clock around `legal_commands` only: **3.0 / 3.5 / 4.1 µs**
> at 2/3/4 seats, now rising with branching. **The reviewer measured
> 15.620.4 µs by a different isolation and we have not settled which is
> right** — T04 must benchmark it properly (`criterion` is already a
> dev-dependency) rather than adopt either.
What is affordable, at ~140 µs/node: a bounded search of ~10⁴10⁵ nodes
costs **1.414 seconds**. That is the budget the ADR has to design inside,
and it is the difference between a feature that answers while the player
is still looking at the page and one that does not.
~~**Exhaustive search is out at any seat count.**~~ **False (C6).** The
reviewer measured ~16 minutes for a full 2p game and **~3 seconds over the
last two rounds at 3p**. With C1's correction the affordable budget is
**~10⁵10⁶ nodes**, and bounded exhaustive search over the endgame is
inside it.
**This changes T03's starting point.** The ADR cannot open with
*"exhaustive is impossible, therefore determinized sampling"* — the
premise is false, and the alternative carries strategy fusion that
exhaustive search does not. Neither the survey nor the review considered
transposition or the co-operative single-agent framing, which cut the
exponent further.
## 2. Prior art: determinized search, and the failure it is famous for