coulomb/clay-borg
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

CB-WP-0025 T04: specs/RetrospectiveAnalysis.md, and a benchmark that

Browse source 
caught me repeating C1

specs/RetrospectiveAnalysis.md v1.0 plus games/ground/benches/search.rs,
which exists because ADR-0013 D7 refused to let the spec quote either
disputed figure.

THE BENCHMARK'S OWN FIRST FIXTURE WAS DEFECTIVE, and it is the same defect
class the review caught one layer up. Stopping at a fixed step 20 put 2p
and 4p in states where seat 0 had NO legal commands, so it timed an empty
Vec (~120 ns) and silently skipped validate_fold because there was nothing
to validate. It now advances until the seat has a real branch and ASSERTS
it. A clone benchmark was added too: a search must copy state per branch,
and iter_batched excludes setup from timing, so without it the budget
would again rest on an unmeasured span.

Measured at real decision points: legal_commands 4.06-4.76 us, clone
378-639 ns, validate+fold 0.5-3.8 us. Per-child cost is NOT uniform --
some commands resolve cascades -- so budgets use the upper end (~5
us/child).

That settles D3 with real numbers. Joint branching over the last two
rounds is ~5x10^2 / 1.6x10^5 / 5.7x10^5 at 2/3/4 seats, so K=2 costs
negligible / 0.8 s / 2.9 s and holds at two to four seats. It does NOT
hold at five or six, where the tool must reduce K and say that it did
rather than silently searching less.

§4.1 is a normative prohibition, not a preference: a single policy's win
rate MAY NOT be reported as a difficulty. The spec carries the measured
reason -- greedy 100% against first-legal 0% on identical deals -- because
this project already made that error and nearly exported it to a repo that
is blocked waiting on the number.

§2.3 makes the empty-result wording normative: "no winning line found in
the last K rounds", never "unwinnable". A bounded search cannot establish
unwinnability and that sentence is what a player who just lost reads.

Also corrected: the T01 completion record still asserted all three
withdrawn claims as fact. It now carries claimed / withdrawn / survives
explicitly rather than being rewritten -- a retraction that does not
propagate to every place the claim lives is how the earlier ones survived.

make all: exit 0. loop-lint clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-08-05 18:59:46 +02:00
parent 3a026b1e1f
commit b27aa14df0
4 changed files with 411 additions and 74 deletions
Download patch file Download diff file Expand all files Collapse all files

156
workplans/CB-WP-0025-could-we-have-won.md
View file

@ -127,34 +127,34 @@ per dimension — a number or a reproducible comparison, not an impression.
[CB-RES-0008](../research/CB-RES-0008-could-we-have-won.md), with a
runnable baseline (`games/ground/examples/difficulty-baseline.rs`).
**The baseline produced a finding before any solver exists, and it is the
biggest thing in this pass.** A greedy bot wins **200 of 200** games at
five and six seats — with a median margin of +3 and 11.8–12.0 points
available against a threshold of 9. The curve is 66% / 82.5% / 95% / 100%
/ 100% across 2/3/4/5/6 seats.
> **Everything this record originally claimed was withdrawn by T02 the
> same day.** Kept as a pointer rather than rewritten, because a claim
> retracted silently is how three earlier wrong premises survived
> (ADR-0012 D5).
The row-level table shows why: available points go **6 / 9 / 12** against
thresholds **5 / 7 / 9**, so the ratio *rises* with seat count (1.20 →
1.29 → 1.33) while the table also gains actions per round. **Three
multipliers pointing the same way.**
**Claimed:** a greedy bot wins 200/200 at five and six seats, so the game
is too easy there; the points-to-threshold ratio rises with seat count and
explains the curve; `legal_commands` costs 112–161 µs, so exhaustive
search is out; and all of this explains the maintainer's report.
**It also explains the maintainer's report without a solver.** *"I felt it
was too easy but then we lost"* — 66% at two seats is a real game; 100% at
six is not. Both halves are true of different seat counts.
**Withdrawn:** the win rate is `GreedyPolicy`'s, not the game's (C4 —
`FirstLegal` scores 0% on the same deals); the ratio is identical at 3p
and 4p, which differ by 12.5 points (C2); the cost figure was wrong by
30–50× and exhaustive search is affordable (C1, C6); and the maintainer's
losses were 3-player games on the pre-ruling deal (C5).
**Cost measured, and it rules out the obvious approach.** Branching is
small (mean 4.7–9.1) but `legal_commands` costs **112–161 µs** per call,
because it filters candidates through full `validate`. Exhaustive search
is out at every seat count; ~10⁴–10⁵ nodes is 1.4–14 s, which is the
budget the ADR must design inside.
**What survives:** the harness exists and runs, the 6/9/12 arithmetic is
right against `Problems.csv`, and the branching widths (4.7–9.1) hold. See
[CB-RES-0008](../research/CB-RES-0008-could-we-have-won.md) for the
corrected text and
[the response](../history/260805-could-we-have-won-response.md) for the
full accounting.
**Prior art names the trap.** Determinized search (PIMC) suffers *strategy
fusion* — Frank, Basin & Matsubara 1998 — where the search picks different
actions in states a real player cannot tell apart. A witness built that
way may require knowing what was on top of the deck. **And it would still
replay green**, so the checkability benchmark does not catch it. Honesty
and checkability are different properties, stated so T03 cannot conflate
them.
**Prior art named the trap** — determinized search suffers *strategy
fusion* (Frank, Basin & Matsubara 1998) — **and T03 then established it
does not apply here.** After the game there is one world, so a line found
in it is executable in it. Fusion is an obstacle to a *playing* engine,
which this is not.
## Task: adversarial review
@ -189,47 +189,27 @@ Tier L requires it. Exactly one round: challenge, then response, trail in
[response](../history/260805-could-we-have-won-response.md).
Separate agent, as in CB-WP-0022.
**Six of seven conceded, and the survey's headline finding is withdrawn —
not softened.**
**Six of seven conceded. The survey's headline finding is WITHDRAWN.**
**C4 is the one that kills it, and the reviewer ranked it fourth.**
Measured: a `FirstLegal` policy — take `legal[0]`, no heuristic — scores
**0% at five and six seats** where greedy scores 100%, and **77.5% at two
seats** where greedy scores 66%. **Two unsophisticated agents span the
entire range at the same seat count**, so *"the game is too easy at 5–6
seats"* is a statement about `GreedyPolicy`, not about GROUND. Their
offered rescue — greedy hits the 12-point ceiling every time, so it is a
rules claim — dies on the same data.
**C4 kills it and the reviewer ranked it fourth.** A `FirstLegal` policy —
`legal[0]`, no heuristic — scores **0% at five and six seats** where greedy
scores 100%, and **77.5% at two** where greedy scores 66%. Two
unsophisticated agents span the whole range, so *"too easy at 5–6 seats"*
was about `GreedyPolicy`, not GROUND.
**C1: the per-node cost was wrong by 30–50×.** The timer bracketed two
`setup`s, a whole greedy game and a validate+fold replay, then divided by
player decisions. **The tell was in my own published output**: the figure
*fell* as branching *rose*, which no per-enumeration cost can do.
Re-measured at **3.0–4.1 µs**; the reviewer got 15.6–20.4 by a different
isolation and **that discrepancy is unsettled** — T04 benchmarks it.
**C6: exhaustive search is not out**, which changes T03's premise. With
C1's correction the budget is ~10⁵–10⁶ nodes and bounded endgame search
fits — so the ADR cannot open with *"exhaustive is impossible, therefore
determinized sampling"*, especially since sampling carries strategy fusion
that exhaustive search does not.
**C3: the finding failed the admissibility rule this project wrote nine
hours earlier** — sums where GROUND-WP-0004 T02 requires per-priority
rows, and a harness with no assertions, no `--self-test` and no `make`
target, so nothing can turn it red. A `default` artifact wearing a
`counterexample` label.
**C2: the ratio explains nothing** — 3p and 4p share deal, threshold and
ratio, and differ by 12.5 points of win rate. **C5: the "explains the
maintainer's report" claim is contradicted by `lib.rs:2487`**, which
records his losses as 3-player on the pre-ruling deal, unwinnable at 6
against 7.
**C1**: the node cost was wrong by 30–50× — the timer bracketed whole
games — and **the tell was in my own output**, falling as branching rose.
**C6**: exhaustive search is *not* out, which changes T03's premise.
**C3**: the finding failed the admissibility rule this project wrote nine
hours earlier. **C2**: 3p and 4p share deal, threshold and ratio and differ
by 12.5 points. **C5**: `lib.rs:2487` contradicts the "explains the
maintainer" claim — his losses were 3-player on the pre-ruling deal,
unwinnable at 6 against 7.
**T06 was pointed at GROUND-WP-0005, which is blocked on this number.**
Sending it would have invited threshold changes on one bot's behaviour —
the fifth wrong premise to reach ground-game. **Both tier-L reviews in
this project have now caught a false headline that every gate passed.**
Sending it would have been the fifth wrong premise to reach ground-game.
**Both tier-L reviews here have now caught a false headline that every
gate passed.**
## Task: decide
@ -277,24 +257,21 @@ moved the ground under both.
property of the deal distribution and the threshold, and is what
GROUND-WP-0005 actually needs. **The bot rate never was.**
**D2** searches `GroundState` (the survey's view-only boundary is not
implementable — a view cannot fold events) and moves the guarantee to a
checkable per-move `visible`/`hidden` marking computed from `project()`.
A witness reads *"you could have won, but two of these six moves needed a
card you had no way to know was coming."* **D3** is bounded exhaustive
over the last K rounds — measured affordable at 2–4 seats once C6
corrected the premise — with normative wording: *"no winning line found in
the last K rounds"*, never *"unwinnable"*. **D5** makes the harness an
instrument before any figure is quoted (C3). **D6**: no new crate, no
port — **the L declaration was an over-declaration and is recorded as
one**. **D7**: the node cost is disputed 5× and T04 must benchmark it;
neither figure may be cited, including by the ADR.
**D2** searches `GroundState` — the survey's view-only boundary is not
implementable, since a view cannot fold events — and moves the guarantee to
a checkable per-move `visible`/`hidden` marking from `project()`. **D3**
bounded exhaustive over the last K rounds, with *"no winning line found in
the last K rounds"* normative, never *"unwinnable"*. **D5** makes the
harness an instrument before any figure is quoted (C3). **D6** no new
crate and no port — **the L declaration was an over-declaration, recorded
as one**. **D7** the node cost is disputed 5×; T04 benchmarks it and
neither existing figure may be cited.
## Task: specify
```task
id: CB-WP-0025-T04
status: todo
status: done
priority: high
state_hub_task_id: "7c3ca25d-0001-44ef-a07d-0ab2d51b09a5"
```
@ -320,6 +297,37 @@ this produces ships a runnable reproduction and a row-level table** — never
a summed figure. A difficulty number is arithmetic, and it is exactly the
kind that has already gone wrong twice.
**Done 2026-08-05.**
[specs/RetrospectiveAnalysis.md](../specs/RetrospectiveAnalysis.md) v1.0,
and `games/ground/benches/search.rs` for D7's disputed number.
**The benchmark's own first fixture was defective — the same defect class,
one layer down.** A fixed stop at step 20 put 2p and 4p where seat 0 had
**no legal commands**, so it timed an empty `Vec` (~120 ns) and silently
skipped `validate_fold`. It now advances to a real branch **and asserts
it**.
Measured at real decision points (table in
[RetrospectiveAnalysis §3.1](../specs/RetrospectiveAnalysis.md)):
`legal_commands` 4.06–4.76 µs, `clone` 378–639 ns, `validate+fold`
**0.5–3.8 µs**.
**`clone` is in there because a search must copy state per branch**, and
`iter_batched` excludes setup from timing — leaving the budget resting on
an unmeasured span, which is precisely C1's mistake. **Per-child cost is
not uniform**: `validate+fold` ranges 0.5–3.8 µs by command, so budgets
use the upper end (~5 µs/child).
**That settles D3's affordability with real numbers**: joint branching
over the last two rounds is ~5×10² / 1.6×10⁵ / 5.7×10⁵ at 2/3/4 seats →
negligible / **0.8 s** / **2.9 s**. `K = 2` holds at two to four seats and
**does not at five or six**, where the tool must reduce `K` and *say so*
rather than silently search less.
**§4.1 is a normative prohibition**, not a preference: a single-policy win
rate may not be reported as a difficulty. The spec carries the measured
reason — greedy 100% vs first-legal 0% on identical deals.
## Task: build the witness
```task