tegwick
1f0f652920
Some checks are pending
ci / check (push) Waiting to run
premise never left the repo Separate agent, second tier-L review in this project. Six of seven challenges conceded. The survey's headline finding is WITHDRAWN, not softened. C4 kills it, and the reviewer ranked it fourth. A FirstLegal policy -- take legal[0], no heuristic at all -- scores 0% at five and six seats where GreedyPolicy scores 100%, and 77.5% at two seats where greedy scores 66%. Two unsophisticated agents span the entire range at the same seat count. "The game is too easy at 5-6 seats" is therefore a statement about GreedyPolicy, not about GROUND. The rescue the reviewer offered -- greedy hits the 12-point ceiling in 200/200 deals, so the 6p row is a rules claim -- dies on the same data: FirstLegal reaches that ceiling never. C1: the per-node cost was wrong by 30-50x. The timer started before the seed loop, so "us/node" included two setups, an entire greedy game and a full validate+fold replay, divided by player-decision count. The tell was in my own published output and I did not look at it: the figure FELL (161/139/112) as branching ROSE (4.7/7.4/9.1), which no per-enumeration cost can do. Re-measured with the clock around legal_commands alone: 3.0/3.5/4.1 us, now rising with branching. The reviewer measured 15.6-20.4 by a different isolation; we disagree by ~5x and neither has established which is right, so T04 must benchmark it with criterion rather than adopt either number. C6: "exhaustive search is out at any seat count" is false -- ~3 seconds over the last two rounds at 3p. With C1's correction the budget is ~10^5-10^6 nodes and bounded endgame search fits, so ADR-0013 cannot open with "exhaustive is impossible, therefore determinized sampling" -- especially as sampling carries strategy fusion that exhaustive search does not. C3: the finding failed the admissibility rule this project wrote nine hours earlier. 6/9/12 are sums where GROUND-WP-0004 T02 requires per-priority rows, and the harness has no assertions, no --self-test and no make target, so nothing can turn it red -- a `default` artifact wearing a `counterexample` label, by CB-WP-0022 T05's own distinction. C2: the ratio story explains nothing; 3p and 4p share deal, threshold and ratio and differ by 12.5 points of win rate. C5: "explains the maintainer's report" is contradicted by lib.rs:2487, which records his losses as 3-player games on the pre-ruling deal, arithmetically unwinnable at 6 against 7. T06 exists to report to GROUND-WP-0005, which is BLOCKED waiting on a difficulty baseline. Had this proceeded they would have been invited to move thresholds on the strength of one bot's behaviour. That is the fifth wrong premise this project would have sent them, and the second stopped by an adversarial review rather than by a control. Both tier-L reviews here have now caught a false headline that every gate passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
197 lines
9.2 KiB
Markdown
197 lines
9.2 KiB
Markdown
# 260805 — response to the challenge on CB-RES-0008
|
||
|
||
One round, per InnerLoop §Step 2. Separate agent, as with CB-RES-0007.
|
||
|
||
**Six challenges conceded, one noted. The survey's headline finding is
|
||
withdrawn.** And one challenge lands harder than the reviewer scored it —
|
||
C4, which they ranked fourth, is the one that kills the claim.
|
||
|
||
---
|
||
|
||
## C1 — the timer measured almost everything except the thing it named — **conceded**
|
||
|
||
`difficulty-baseline.rs` started the clock **before the seed loop**, so
|
||
"µs/node" included two `setup`s, an entire greedy game, and a full
|
||
validate+fold replay — then divided by the number of *player decisions*.
|
||
|
||
**The tell was in my own published output and I did not look at it.** The
|
||
figure **fell** as seat count rose — 161 → 139 → 112 — while branching
|
||
**rose** 4.7 → 7.4 → 9.1. A per-enumeration cost cannot do that. The
|
||
denominator was growing faster than the numerator because the numerator
|
||
was mostly fixed per-game work.
|
||
|
||
Re-measured with the clock bracketing `legal_commands` and nothing else:
|
||
|
||
```
|
||
2p 462 decisions branching mean 4.7 median 5 max 7 3.0 us/call
|
||
3p 649 decisions branching mean 7.4 median 8 max 10 3.5 us/call
|
||
4p 870 decisions branching mean 9.1 median 10 max 12 4.1 us/call
|
||
```
|
||
|
||
Now it rises with branching, which is the direction the mechanism
|
||
predicts.
|
||
|
||
**A discrepancy I am not going to paper over.** The reviewer measured
|
||
**15.6–20.4 µs**; I measure **3.0–4.1 µs** bracketing each call with
|
||
`Instant::now()`. We disagree by ~5×, which is small against the 30–50×
|
||
correction we agree on but is not nothing. Two different isolation methods
|
||
gave two different answers and **neither of us has established which is
|
||
right** — so T04 must settle the per-node cost with a proper benchmark
|
||
(this repo has `criterion` already), not with either of these numbers.
|
||
|
||
**This is the wrong-denominator family again** — CB-EV-0019 §1 named it
|
||
with three instances, and this is a fourth, in the *numerator* this time:
|
||
a number computed correctly over the wrong span.
|
||
|
||
## C4 — **conceded, and the reviewer under-ranked it. This is the one that kills the finding.**
|
||
|
||
They ranked it fourth. Measured, it is first.
|
||
|
||
A `FirstLegal` policy — take `legal[0]`, no heuristic at all — against
|
||
`GreedyPolicy`, 200 seeds each:
|
||
|
||
| seats | greedy | firstlegal |
|
||
|---|---:|---:|
|
||
| 2 | 66.0% | **77.5%** |
|
||
| 3 | 82.5% | 25.0% |
|
||
| 4 | 95.0% | 32.0% |
|
||
| 5 | **100.0%** | **0.0%** |
|
||
| 6 | **100.0%** | **0.0%** |
|
||
|
||
**At five and six seats, one unsophisticated policy wins every game and
|
||
another wins none.** The spread between two dumb agents at the same seat
|
||
count is the entire range.
|
||
|
||
The survey said *"a greedy bot wins 200 of 200 at five and six seats"* and
|
||
concluded the game is too easy there. **That conclusion does not follow
|
||
from a measurement whose value ranges over 0–100% depending on which
|
||
trivial agent you pick.** It is a statement about `GreedyPolicy`, not
|
||
about GROUND.
|
||
|
||
**The reviewer offered me a rescue and it does not survive either.** They
|
||
suggested that greedy hitting the 12-point ceiling in 200/200 deals makes
|
||
the 6p row a rules claim — the ceiling is reachable, therefore the game is
|
||
clearable. FirstLegal's 0% at 6p refutes it: reachable-by-greedy is not
|
||
reachable-by-an-agent, and "the ceiling can be hit" was never in doubt
|
||
anyway. What was claimed is that it is *easy* to hit, and nothing here
|
||
supports that.
|
||
|
||
**And at two seats greedy is worse than no heuristic at all** (66% vs
|
||
77.5%), so the 66% figure — which I used as "a real game" — is partly a
|
||
GreedyPolicy defect, not a property of the deal.
|
||
|
||
**The headline finding is withdrawn.** Not softened: withdrawn.
|
||
|
||
## C2 — the ratio does not explain the curve — **conceded**
|
||
|
||
I wrote *"three multipliers all pointing the same direction"* over a table
|
||
of ratios 1.20 / 1.29 / 1.33.
|
||
|
||
**3-player and 4-player share the same deal, the same threshold and
|
||
therefore the same ratio (9 points against 7, 1.29) — and differ by 12.5
|
||
points of win rate** (82.5% vs 95%). My own harness prints `of 7.0` for
|
||
both. A quantity identical across two rows cannot explain a difference
|
||
between them.
|
||
|
||
The mechanism the reviewer identifies — more hands claiming a **fixed
|
||
five-Problem pool** — is consistent with the data and mine was not. I
|
||
constructed a story from a table I had built and did not test it against
|
||
the rows sitting next to each other in my own output.
|
||
|
||
## C3 — the finding is inadmissible under GameDesign §1 — **conceded**
|
||
|
||
The survey claimed its finding met the admissibility rule this project
|
||
wrote nine hours earlier. It fails two of three clauses:
|
||
|
||
- **Clause 2 (ruled shape).** 6/9/12 are **sums**. GROUND-WP-0004 T02
|
||
requires Surface and each hidden priority listed *separately*, and
|
||
explicitly forbids "deal depth N". My table has a `hidden dealt` column
|
||
containing a count. That is the exact shape the ruling exists to
|
||
prohibit, and I reproduced it while citing the ruling.
|
||
- **Clause 3 (can fail).** The harness has **no assertions, no
|
||
`--self-test`, and is not in any `make` target**. Nothing can turn it
|
||
red. Under CB-WP-0022 T05's own distinction it is a `default`-role
|
||
artifact — it prints what the code does — wearing a `counterexample`
|
||
label.
|
||
|
||
**A reproduction that cannot fail is a printout.** The rule was one day
|
||
old and I broke it in the first pass that used it.
|
||
|
||
## C5 — the "explains the maintainer's report" claim is contradicted by this repo — **conceded**
|
||
|
||
I wrote that 66%-at-2p and 100%-at-6p together explain *"I felt it was too
|
||
easy but then we lost."*
|
||
|
||
`games/ground/src/lib.rs:2487-2493`, in a test comment I did not read:
|
||
|
||
> *"the maintainer played several 3-player games on 2026-08-03 and could
|
||
> not win any of them, because GR-S01 dealt 2/3/4 Problems worth 3/6/10
|
||
> against thresholds of 5/7/9."*
|
||
|
||
**Three-player games, on the pre-ruling deal, arithmetically unwinnable at
|
||
6 against 7.** Not 2-player, not a 66% coin-flip, and nothing to do with
|
||
the curve I fitted to it.
|
||
|
||
**This is the same defect as CB-WP-0022's C1**, four days later: a claim
|
||
about the maintainer's own experience, contradicted by a record in this
|
||
repo, written by a pass that had the record available. The remedy there
|
||
was to check `git log`; the remedy here was to read the test comment
|
||
attached to the very test that inverted.
|
||
|
||
## C6 — "exhaustive search is out at any seat count" is false — **conceded, and it changes the ADR**
|
||
|
||
I asserted `7.4^15` and stopped thinking. The reviewer measured ~16 min for
|
||
a full 2p game and ~3 s over the last two rounds at 3p.
|
||
|
||
Combined with C1's correction, **the affordable budget is ~10⁵–10⁶ nodes,
|
||
not ~10⁴–10⁵**, and bounded exhaustive search over the endgame is inside
|
||
it. That is a different starting point for T03: the ADR cannot open with
|
||
*"exhaustive is impossible, therefore determinized sampling"*, because the
|
||
premise is false and the alternative carries strategy fusion that
|
||
exhaustive search does not.
|
||
|
||
Neither I nor the reviewer considered transposition or the co-operative
|
||
single-agent framing, which cut the exponent further.
|
||
|
||
## C7 — the replay loop discards rejections silently — **noted**
|
||
|
||
Correct, and the reviewer measured **zero rejections**, so no published
|
||
number is contaminated. It is still a silent `if let Ok(...)`, and it will
|
||
lie the moment the replay diverges. Fixed alongside the rest.
|
||
|
||
## What survives, and what would have falsified it
|
||
|
||
- **`nodes == widths.len()`** — the denominator is the one the label says.
|
||
- **No games were dropped**: all 200 seeds completed at every seat count,
|
||
so the win-rate denominators are real.
|
||
- **`GreedyPolicy` is a genuine heuristic**, not first-legal in disguise —
|
||
which is what makes C4's comparison meaningful rather than trivial.
|
||
- **The 6/9/12 arithmetic itself** is right against `Problems.csv` and
|
||
GR-S01. It is the *shape* that fails C3 and the *interpretation* that
|
||
fails C2 and C4, not the arithmetic.
|
||
- **The 5×N exponent** was measured at 11.6/16.2/21.8 against my claimed
|
||
10/15/20 — close enough that the exponent was not where C6's error was.
|
||
|
||
---
|
||
|
||
## What the review changed
|
||
|
||
| | before | after |
|
||
|---|---|---|
|
||
| headline finding | *the game is too easy at 5–6 seats* | **withdrawn** — one dumb policy wins 100%, another 0% |
|
||
| per-node cost | 112–161 µs | 3.0–4.1 µs (mine) vs 15.6–20.4 (theirs); **unsettled, T04 must benchmark** |
|
||
| exhaustive search | ruled out | **available** for bounded endgame; ADR's premise changes |
|
||
| the ratio story | three multipliers | 3p and 4p share the ratio and differ by 12.5 points |
|
||
| admissibility | claimed | **fails clauses 2 and 3** of a rule one day old |
|
||
| explains the maintainer | asserted | contradicted by `lib.rs:2487` |
|
||
|
||
**The number was going to ground-game.** T06 exists to report to
|
||
GROUND-WP-0005, which is *blocked waiting on a difficulty baseline*. Had
|
||
this pass proceeded, they would have been told the game is too easy at
|
||
five and six seats and invited to move thresholds — on the strength of one
|
||
bot's behaviour, when a second bot of the same sophistication scores zero.
|
||
|
||
**That would have been the fifth wrong premise this project has sent
|
||
them, and the second stopped by an adversarial review rather than by a
|
||
control.** Two for two: both tier-L reviews in this project have caught a
|
||
false headline that every gate passed.
|