CB-WP-0025 T02: the review withdrew the finding, and the fifth wrong
Some checks are pending
ci / check (push) Waiting to run
Some checks are pending
ci / check (push) Waiting to run
premise never left the repo Separate agent, second tier-L review in this project. Six of seven challenges conceded. The survey's headline finding is WITHDRAWN, not softened. C4 kills it, and the reviewer ranked it fourth. A FirstLegal policy -- take legal[0], no heuristic at all -- scores 0% at five and six seats where GreedyPolicy scores 100%, and 77.5% at two seats where greedy scores 66%. Two unsophisticated agents span the entire range at the same seat count. "The game is too easy at 5-6 seats" is therefore a statement about GreedyPolicy, not about GROUND. The rescue the reviewer offered -- greedy hits the 12-point ceiling in 200/200 deals, so the 6p row is a rules claim -- dies on the same data: FirstLegal reaches that ceiling never. C1: the per-node cost was wrong by 30-50x. The timer started before the seed loop, so "us/node" included two setups, an entire greedy game and a full validate+fold replay, divided by player-decision count. The tell was in my own published output and I did not look at it: the figure FELL (161/139/112) as branching ROSE (4.7/7.4/9.1), which no per-enumeration cost can do. Re-measured with the clock around legal_commands alone: 3.0/3.5/4.1 us, now rising with branching. The reviewer measured 15.6-20.4 by a different isolation; we disagree by ~5x and neither has established which is right, so T04 must benchmark it with criterion rather than adopt either number. C6: "exhaustive search is out at any seat count" is false -- ~3 seconds over the last two rounds at 3p. With C1's correction the budget is ~10^5-10^6 nodes and bounded endgame search fits, so ADR-0013 cannot open with "exhaustive is impossible, therefore determinized sampling" -- especially as sampling carries strategy fusion that exhaustive search does not. C3: the finding failed the admissibility rule this project wrote nine hours earlier. 6/9/12 are sums where GROUND-WP-0004 T02 requires per-priority rows, and the harness has no assertions, no --self-test and no make target, so nothing can turn it red -- a `default` artifact wearing a `counterexample` label, by CB-WP-0022 T05's own distinction. C2: the ratio story explains nothing; 3p and 4p share deal, threshold and ratio and differ by 12.5 points of win rate. C5: "explains the maintainer's report" is contradicted by lib.rs:2487, which records his losses as 3-player games on the pre-ruling deal, arithmetically unwinnable at 6 against 7. T06 exists to report to GROUND-WP-0005, which is BLOCKED waiting on a difficulty baseline. Had this proceeded they would have been invited to move thresholds on the strength of one bot's behaviour. That is the fifth wrong premise this project would have sent them, and the second stopped by an adversarial review rather than by a control. Both tier-L reviews here have now caught a false headline that every gate passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
469d00d679
commit
1f0f652920
5 changed files with 804 additions and 31 deletions
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
id: CB-RES-0008
|
||||
capability: analysis.witness-and-difficulty
|
||||
status: draft — awaiting adversarial review (CB-WP-0025 T02)
|
||||
status: reviewed 2026-08-05 — headline finding WITHDRAWN (C4); numbers corrected (C1)
|
||||
tier: L
|
||||
chaos: d8 = 6 → no override
|
||||
---
|
||||
|
|
@ -39,14 +39,39 @@ bot win rate over 200 seeds (GR-E01 group success):
|
|||
4p random 16/200 won = 8.0% mean total 3.2 of 7.0 median margin -4
|
||||
|
||||
search cost — legal_commands at every real decision point:
|
||||
2p 462 decisions branching mean 4.7 median 5 max 7 161 us/node
|
||||
3p 649 decisions branching mean 7.4 median 8 max 10 139 us/node
|
||||
4p 870 decisions branching mean 9.1 median 10 max 12 112 us/node
|
||||
2p 462 decisions branching mean 4.7 median 5 max 7 3.0 us/call
|
||||
3p 649 decisions branching mean 7.4 median 8 max 10 3.5 us/call
|
||||
4p 870 decisions branching mean 9.1 median 10 max 12 4.1 us/call
|
||||
```
|
||||
|
||||
### 1.1 The finding this produced before any solver exists
|
||||
*(Search-cost row re-measured after C1. The first published figures —
|
||||
161/139/112 "us/node" — timed two `setup`s, a whole greedy game and a
|
||||
validate+fold replay. See §1.2.)*
|
||||
|
||||
**A greedy bot wins 200 of 200 games at five and six seats.** Not 95%,
|
||||
### 1.1 WITHDRAWN — the finding this claimed, and why it is not one
|
||||
|
||||
> **Withdrawn 2026-08-05 by the adversarial review (C4), before it left
|
||||
> the repo.** The section is kept, struck through, because a claim
|
||||
> retracted silently is how three earlier wrong premises survived
|
||||
> (ADR-0012 D5).
|
||||
>
|
||||
> A `FirstLegal` policy — take `legal[0]`, no heuristic — scores
|
||||
> **0% at five and six seats** where `GreedyPolicy` scores 100%, and
|
||||
> **77.5% at two seats** where greedy scores 66%. Two unsophisticated
|
||||
> agents span the entire range at the same seat count. **A measurement
|
||||
> that does that is about the policy, not about the game.**
|
||||
>
|
||||
> | seats | greedy | firstlegal |
|
||||
> |---|---:|---:|
|
||||
> | 2 | 66.0% | **77.5%** |
|
||||
> | 5 | **100.0%** | **0.0%** |
|
||||
> | 6 | **100.0%** | **0.0%** |
|
||||
>
|
||||
> The arithmetic below is right; the **interpretation** is not, and the
|
||||
> table also fails GameDesign §1.2 by reporting **sums** where
|
||||
> GROUND-WP-0004 T02 requires per-priority rows.
|
||||
|
||||
~~**A greedy bot wins 200 of 200 games at five and six seats.**~~ Not 95%,
|
||||
not 99% — every game, with a median margin of **+3** and mean available
|
||||
points of **11.8–12.0 against a threshold of 9**.
|
||||
|
||||
|
|
@ -64,36 +89,69 @@ table *and* a proportionally lower bar *and* more actions per round to
|
|||
clear it with. Three multipliers all pointing the same direction, which is
|
||||
why the curve is not gentle — it is 66% → 100% across four seat counts.
|
||||
|
||||
**This is admissible under GameDesign §1**: the reproduction exists
|
||||
(`examples/difficulty-baseline.rs`), it has the ruled shape (row-level, no
|
||||
sums), and it can fail — change a threshold and the numbers move.
|
||||
~~**This is admissible under GameDesign §1.**~~ **It is not (C3), and the
|
||||
rule it fails was one day old.**
|
||||
|
||||
It also **explains the maintainer's report** — *"I felt it was too easy
|
||||
- **Clause 2, ruled shape.** 6/9/12 are **sums**, and the `hidden dealt`
|
||||
column is a *count*. GROUND-WP-0004 T02 requires Surface and each hidden
|
||||
priority listed separately and explicitly forbids "deal depth N". The
|
||||
table reproduces the prohibited shape while citing the ruling.
|
||||
- **Clause 3, can fail.** The harness has **no assertions, no
|
||||
`--self-test`, and is in no `make` target**. Nothing can turn it red.
|
||||
Under CB-WP-0022 T05's own `role` distinction it is a `default` artifact
|
||||
— it prints what the code does — wearing a `counterexample` label.
|
||||
|
||||
**A reproduction that cannot fail is a printout.** T05 must fix the
|
||||
harness before any figure from it is quoted again.
|
||||
|
||||
~~It also **explains the maintainer's report** — *"I felt it was too easy
|
||||
but then we lost, so who knows"* — without needing a solver. He plays at
|
||||
low seat counts, where 66% is a real game, and had been feeling the 5–6
|
||||
seat experience from elsewhere in the same session. Both halves of the
|
||||
sentence are true of different seat counts.
|
||||
sentence are true of different seat counts.~~
|
||||
|
||||
**T06 must report this to GROUND-WP-0005, which is blocked on exactly
|
||||
this number.**
|
||||
**Also withdrawn (C5).** `games/ground/src/lib.rs:2487-2493` records what
|
||||
actually happened: *"the maintainer played several 3-player games on
|
||||
2026-08-03 and could not win any of them, because GR-S01 dealt 2/3/4
|
||||
Problems worth 3/6/10 against thresholds of 5/7/9."* Three seats, on the
|
||||
pre-ruling deal, **arithmetically unwinnable at 6 against 7** — nothing to
|
||||
do with the curve fitted to it here.
|
||||
|
||||
~~**T06 must report this to GROUND-WP-0005.**~~ **It must not.**
|
||||
GROUND-WP-0005 is blocked waiting on a difficulty baseline, and this is
|
||||
not one. Sending it would have invited ground-game to move thresholds on
|
||||
the strength of one bot's behaviour — the **fifth** wrong premise this
|
||||
project has sent them, and the second stopped by a review rather than by
|
||||
a control.
|
||||
|
||||
### 1.2 What the search-cost numbers rule out
|
||||
|
||||
Branching is small — mean 4.7 to 9.1 — but **`legal_commands` costs
|
||||
112–161 µs per call**, because it constructs candidates and filters them
|
||||
through full `validate`. That is the price of keeping the rules in one
|
||||
place (ADR: `legal_commands` builds then validates), and it is the number
|
||||
that decides this pass.
|
||||
Branching is small — mean 4.7 to 9.1.
|
||||
|
||||
A game is 5 rounds × N seats of decisions. An exhaustive search from
|
||||
round 1 at 3 seats is roughly `7.4^15` — not a number worth writing down.
|
||||
**Exhaustive search is out at any seat count**, and this was measured
|
||||
rather than assumed.
|
||||
> **CORRECTED (C1). The published figure of 112–161 µs/node was wrong by
|
||||
> 30–50×**: the timer bracketed two `setup`s, a whole greedy game and a
|
||||
> validate+fold replay, divided by player-decision count. The tell was in
|
||||
> the output — it **fell** as branching **rose**, which no per-enumeration
|
||||
> cost can do.
|
||||
>
|
||||
> Re-measured, clock around `legal_commands` only: **3.0 / 3.5 / 4.1 µs**
|
||||
> at 2/3/4 seats, now rising with branching. **The reviewer measured
|
||||
> 15.6–20.4 µs by a different isolation and we have not settled which is
|
||||
> right** — T04 must benchmark it properly (`criterion` is already a
|
||||
> dev-dependency) rather than adopt either.
|
||||
|
||||
What is affordable, at ~140 µs/node: a bounded search of ~10⁴–10⁵ nodes
|
||||
costs **1.4–14 seconds**. That is the budget the ADR has to design inside,
|
||||
and it is the difference between a feature that answers while the player
|
||||
is still looking at the page and one that does not.
|
||||
~~**Exhaustive search is out at any seat count.**~~ **False (C6).** The
|
||||
reviewer measured ~16 minutes for a full 2p game and **~3 seconds over the
|
||||
last two rounds at 3p**. With C1's correction the affordable budget is
|
||||
**~10⁵–10⁶ nodes**, and bounded exhaustive search over the endgame is
|
||||
inside it.
|
||||
|
||||
**This changes T03's starting point.** The ADR cannot open with
|
||||
*"exhaustive is impossible, therefore determinized sampling"* — the
|
||||
premise is false, and the alternative carries strategy fusion that
|
||||
exhaustive search does not. Neither the survey nor the review considered
|
||||
transposition or the co-operative single-agent framing, which cut the
|
||||
exponent further.
|
||||
|
||||
## 2. Prior art: determinized search, and the failure it is famous for
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue