198 lines
9.2 KiB
Markdown
198 lines
9.2 KiB
Markdown
|
|
# 260805 — response to the challenge on CB-RES-0008
|
|||
|
|
|
|||
|
|
One round, per InnerLoop §Step 2. Separate agent, as with CB-RES-0007.
|
|||
|
|
|
|||
|
|
**Six challenges conceded, one noted. The survey's headline finding is
|
|||
|
|
withdrawn.** And one challenge lands harder than the reviewer scored it —
|
|||
|
|
C4, which they ranked fourth, is the one that kills the claim.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## C1 — the timer measured almost everything except the thing it named — **conceded**
|
|||
|
|
|
|||
|
|
`difficulty-baseline.rs` started the clock **before the seed loop**, so
|
|||
|
|
"µs/node" included two `setup`s, an entire greedy game, and a full
|
|||
|
|
validate+fold replay — then divided by the number of *player decisions*.
|
|||
|
|
|
|||
|
|
**The tell was in my own published output and I did not look at it.** The
|
|||
|
|
figure **fell** as seat count rose — 161 → 139 → 112 — while branching
|
|||
|
|
**rose** 4.7 → 7.4 → 9.1. A per-enumeration cost cannot do that. The
|
|||
|
|
denominator was growing faster than the numerator because the numerator
|
|||
|
|
was mostly fixed per-game work.
|
|||
|
|
|
|||
|
|
Re-measured with the clock bracketing `legal_commands` and nothing else:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
2p 462 decisions branching mean 4.7 median 5 max 7 3.0 us/call
|
|||
|
|
3p 649 decisions branching mean 7.4 median 8 max 10 3.5 us/call
|
|||
|
|
4p 870 decisions branching mean 9.1 median 10 max 12 4.1 us/call
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Now it rises with branching, which is the direction the mechanism
|
|||
|
|
predicts.
|
|||
|
|
|
|||
|
|
**A discrepancy I am not going to paper over.** The reviewer measured
|
|||
|
|
**15.6–20.4 µs**; I measure **3.0–4.1 µs** bracketing each call with
|
|||
|
|
`Instant::now()`. We disagree by ~5×, which is small against the 30–50×
|
|||
|
|
correction we agree on but is not nothing. Two different isolation methods
|
|||
|
|
gave two different answers and **neither of us has established which is
|
|||
|
|
right** — so T04 must settle the per-node cost with a proper benchmark
|
|||
|
|
(this repo has `criterion` already), not with either of these numbers.
|
|||
|
|
|
|||
|
|
**This is the wrong-denominator family again** — CB-EV-0019 §1 named it
|
|||
|
|
with three instances, and this is a fourth, in the *numerator* this time:
|
|||
|
|
a number computed correctly over the wrong span.
|
|||
|
|
|
|||
|
|
## C4 — **conceded, and the reviewer under-ranked it. This is the one that kills the finding.**
|
|||
|
|
|
|||
|
|
They ranked it fourth. Measured, it is first.
|
|||
|
|
|
|||
|
|
A `FirstLegal` policy — take `legal[0]`, no heuristic at all — against
|
|||
|
|
`GreedyPolicy`, 200 seeds each:
|
|||
|
|
|
|||
|
|
| seats | greedy | firstlegal |
|
|||
|
|
|---|---:|---:|
|
|||
|
|
| 2 | 66.0% | **77.5%** |
|
|||
|
|
| 3 | 82.5% | 25.0% |
|
|||
|
|
| 4 | 95.0% | 32.0% |
|
|||
|
|
| 5 | **100.0%** | **0.0%** |
|
|||
|
|
| 6 | **100.0%** | **0.0%** |
|
|||
|
|
|
|||
|
|
**At five and six seats, one unsophisticated policy wins every game and
|
|||
|
|
another wins none.** The spread between two dumb agents at the same seat
|
|||
|
|
count is the entire range.
|
|||
|
|
|
|||
|
|
The survey said *"a greedy bot wins 200 of 200 at five and six seats"* and
|
|||
|
|
concluded the game is too easy there. **That conclusion does not follow
|
|||
|
|
from a measurement whose value ranges over 0–100% depending on which
|
|||
|
|
trivial agent you pick.** It is a statement about `GreedyPolicy`, not
|
|||
|
|
about GROUND.
|
|||
|
|
|
|||
|
|
**The reviewer offered me a rescue and it does not survive either.** They
|
|||
|
|
suggested that greedy hitting the 12-point ceiling in 200/200 deals makes
|
|||
|
|
the 6p row a rules claim — the ceiling is reachable, therefore the game is
|
|||
|
|
clearable. FirstLegal's 0% at 6p refutes it: reachable-by-greedy is not
|
|||
|
|
reachable-by-an-agent, and "the ceiling can be hit" was never in doubt
|
|||
|
|
anyway. What was claimed is that it is *easy* to hit, and nothing here
|
|||
|
|
supports that.
|
|||
|
|
|
|||
|
|
**And at two seats greedy is worse than no heuristic at all** (66% vs
|
|||
|
|
77.5%), so the 66% figure — which I used as "a real game" — is partly a
|
|||
|
|
GreedyPolicy defect, not a property of the deal.
|
|||
|
|
|
|||
|
|
**The headline finding is withdrawn.** Not softened: withdrawn.
|
|||
|
|
|
|||
|
|
## C2 — the ratio does not explain the curve — **conceded**
|
|||
|
|
|
|||
|
|
I wrote *"three multipliers all pointing the same direction"* over a table
|
|||
|
|
of ratios 1.20 / 1.29 / 1.33.
|
|||
|
|
|
|||
|
|
**3-player and 4-player share the same deal, the same threshold and
|
|||
|
|
therefore the same ratio (9 points against 7, 1.29) — and differ by 12.5
|
|||
|
|
points of win rate** (82.5% vs 95%). My own harness prints `of 7.0` for
|
|||
|
|
both. A quantity identical across two rows cannot explain a difference
|
|||
|
|
between them.
|
|||
|
|
|
|||
|
|
The mechanism the reviewer identifies — more hands claiming a **fixed
|
|||
|
|
five-Problem pool** — is consistent with the data and mine was not. I
|
|||
|
|
constructed a story from a table I had built and did not test it against
|
|||
|
|
the rows sitting next to each other in my own output.
|
|||
|
|
|
|||
|
|
## C3 — the finding is inadmissible under GameDesign §1 — **conceded**
|
|||
|
|
|
|||
|
|
The survey claimed its finding met the admissibility rule this project
|
|||
|
|
wrote nine hours earlier. It fails two of three clauses:
|
|||
|
|
|
|||
|
|
- **Clause 2 (ruled shape).** 6/9/12 are **sums**. GROUND-WP-0004 T02
|
|||
|
|
requires Surface and each hidden priority listed *separately*, and
|
|||
|
|
explicitly forbids "deal depth N". My table has a `hidden dealt` column
|
|||
|
|
containing a count. That is the exact shape the ruling exists to
|
|||
|
|
prohibit, and I reproduced it while citing the ruling.
|
|||
|
|
- **Clause 3 (can fail).** The harness has **no assertions, no
|
|||
|
|
`--self-test`, and is not in any `make` target**. Nothing can turn it
|
|||
|
|
red. Under CB-WP-0022 T05's own distinction it is a `default`-role
|
|||
|
|
artifact — it prints what the code does — wearing a `counterexample`
|
|||
|
|
label.
|
|||
|
|
|
|||
|
|
**A reproduction that cannot fail is a printout.** The rule was one day
|
|||
|
|
old and I broke it in the first pass that used it.
|
|||
|
|
|
|||
|
|
## C5 — the "explains the maintainer's report" claim is contradicted by this repo — **conceded**
|
|||
|
|
|
|||
|
|
I wrote that 66%-at-2p and 100%-at-6p together explain *"I felt it was too
|
|||
|
|
easy but then we lost."*
|
|||
|
|
|
|||
|
|
`games/ground/src/lib.rs:2487-2493`, in a test comment I did not read:
|
|||
|
|
|
|||
|
|
> *"the maintainer played several 3-player games on 2026-08-03 and could
|
|||
|
|
> not win any of them, because GR-S01 dealt 2/3/4 Problems worth 3/6/10
|
|||
|
|
> against thresholds of 5/7/9."*
|
|||
|
|
|
|||
|
|
**Three-player games, on the pre-ruling deal, arithmetically unwinnable at
|
|||
|
|
6 against 7.** Not 2-player, not a 66% coin-flip, and nothing to do with
|
|||
|
|
the curve I fitted to it.
|
|||
|
|
|
|||
|
|
**This is the same defect as CB-WP-0022's C1**, four days later: a claim
|
|||
|
|
about the maintainer's own experience, contradicted by a record in this
|
|||
|
|
repo, written by a pass that had the record available. The remedy there
|
|||
|
|
was to check `git log`; the remedy here was to read the test comment
|
|||
|
|
attached to the very test that inverted.
|
|||
|
|
|
|||
|
|
## C6 — "exhaustive search is out at any seat count" is false — **conceded, and it changes the ADR**
|
|||
|
|
|
|||
|
|
I asserted `7.4^15` and stopped thinking. The reviewer measured ~16 min for
|
|||
|
|
a full 2p game and ~3 s over the last two rounds at 3p.
|
|||
|
|
|
|||
|
|
Combined with C1's correction, **the affordable budget is ~10⁵–10⁶ nodes,
|
|||
|
|
not ~10⁴–10⁵**, and bounded exhaustive search over the endgame is inside
|
|||
|
|
it. That is a different starting point for T03: the ADR cannot open with
|
|||
|
|
*"exhaustive is impossible, therefore determinized sampling"*, because the
|
|||
|
|
premise is false and the alternative carries strategy fusion that
|
|||
|
|
exhaustive search does not.
|
|||
|
|
|
|||
|
|
Neither I nor the reviewer considered transposition or the co-operative
|
|||
|
|
single-agent framing, which cut the exponent further.
|
|||
|
|
|
|||
|
|
## C7 — the replay loop discards rejections silently — **noted**
|
|||
|
|
|
|||
|
|
Correct, and the reviewer measured **zero rejections**, so no published
|
|||
|
|
number is contaminated. It is still a silent `if let Ok(...)`, and it will
|
|||
|
|
lie the moment the replay diverges. Fixed alongside the rest.
|
|||
|
|
|
|||
|
|
## What survives, and what would have falsified it
|
|||
|
|
|
|||
|
|
- **`nodes == widths.len()`** — the denominator is the one the label says.
|
|||
|
|
- **No games were dropped**: all 200 seeds completed at every seat count,
|
|||
|
|
so the win-rate denominators are real.
|
|||
|
|
- **`GreedyPolicy` is a genuine heuristic**, not first-legal in disguise —
|
|||
|
|
which is what makes C4's comparison meaningful rather than trivial.
|
|||
|
|
- **The 6/9/12 arithmetic itself** is right against `Problems.csv` and
|
|||
|
|
GR-S01. It is the *shape* that fails C3 and the *interpretation* that
|
|||
|
|
fails C2 and C4, not the arithmetic.
|
|||
|
|
- **The 5×N exponent** was measured at 11.6/16.2/21.8 against my claimed
|
|||
|
|
10/15/20 — close enough that the exponent was not where C6's error was.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## What the review changed
|
|||
|
|
|
|||
|
|
| | before | after |
|
|||
|
|
|---|---|---|
|
|||
|
|
| headline finding | *the game is too easy at 5–6 seats* | **withdrawn** — one dumb policy wins 100%, another 0% |
|
|||
|
|
| per-node cost | 112–161 µs | 3.0–4.1 µs (mine) vs 15.6–20.4 (theirs); **unsettled, T04 must benchmark** |
|
|||
|
|
| exhaustive search | ruled out | **available** for bounded endgame; ADR's premise changes |
|
|||
|
|
| the ratio story | three multipliers | 3p and 4p share the ratio and differ by 12.5 points |
|
|||
|
|
| admissibility | claimed | **fails clauses 2 and 3** of a rule one day old |
|
|||
|
|
| explains the maintainer | asserted | contradicted by `lib.rs:2487` |
|
|||
|
|
|
|||
|
|
**The number was going to ground-game.** T06 exists to report to
|
|||
|
|
GROUND-WP-0005, which is *blocked waiting on a difficulty baseline*. Had
|
|||
|
|
this pass proceeded, they would have been told the game is too easy at
|
|||
|
|
five and six seats and invited to move thresholds — on the strength of one
|
|||
|
|
bot's behaviour, when a second bot of the same sophistication scores zero.
|
|||
|
|
|
|||
|
|
**That would have been the fifth wrong premise this project has sent
|
|||
|
|
them, and the second stopped by an adversarial review rather than by a
|
|||
|
|
control.** Two for two: both tier-L reviews in this project have caught a
|
|||
|
|
false headline that every gate passed.
|