132 lines
5.4 KiB
Markdown
132 lines
5.4 KiB
Markdown
---
|
||
id: GROUND-WP-0005-REPORT-260805
|
||
type: report
|
||
title: "clay-borg difficulty measurement, 2026-08-05 — and a retraction of what we nearly sent instead"
|
||
domain: consumer
|
||
repo: ground-game
|
||
status: informational
|
||
owner: bernd
|
||
topic_slug: whynot
|
||
created: "2026-08-05"
|
||
extends: GROUND-WP-0005
|
||
state_hub_workstream_id: "61cabf31-88a8-4ca6-9383-55b8fccddecf"
|
||
---
|
||
|
||
# clay-borg difficulty report — 2026-08-05
|
||
|
||
GROUND-WP-0005 (*difficulty tiers and the Pressure dial*) has both tasks
|
||
in `wait`, blocked on a measured baseline. This is that baseline, with its
|
||
limits stated.
|
||
|
||
Generated by `make difficulty`
|
||
(`games/ground/examples/difficulty.rs`), governed by
|
||
`clay-borg/specs/RetrospectiveAnalysis.md` §4, under
|
||
[CB-WP-0025](../../clay-borg/workplans/CB-WP-0025-could-we-have-won.md) T06.
|
||
|
||
---
|
||
|
||
## 0. First, what we nearly sent you and did not
|
||
|
||
**Earlier today this report was going to say: "a greedy bot wins 200 of
|
||
200 games at five and six seats — the game is too easy there."**
|
||
|
||
It would have invited you to move thresholds. **It was wrong**, and an
|
||
adversarial review caught it before it left our repo.
|
||
|
||
A `FirstLegal` policy — take the first legal command, no heuristic
|
||
whatsoever — scores **0%** at five and six seats on the identical deals
|
||
where `GreedyPolicy` scores **100%**. At two seats it *beats* greedy
|
||
(76.7% vs 60.0%).
|
||
|
||
**Two unsophisticated agents span the entire range.** A single policy's
|
||
win rate is therefore a statement about the policy, not about GROUND, and
|
||
we have written that into our spec as a prohibition rather than a
|
||
preference.
|
||
|
||
This is the fifth claim we would have sent you on a wrong premise. It is
|
||
the second stopped before sending.
|
||
|
||
## 1. The measurement
|
||
|
||
`make difficulty`, 60 seeds per cell:
|
||
|
||
| seats | winnable | greedy | random | first-legal | spread | undecided |
|
||
|---|---:|---:|---:|---:|---:|---:|
|
||
| 2 | **60%** | 60.0% | 5.0% | 76.7% | 71.7 | 0 |
|
||
| 3 | **93%** | 88.3% | 6.7% | 25.0% | 81.7 | 2 |
|
||
| 4 | **100%** | 93.3% | 6.7% | 30.0% | 86.7 | 4 |
|
||
| 5 | **100%** | 100.0% | 3.3% | 0.0% | 100.0 | 0 |
|
||
| 6 | **100%** | 100.0% | 3.3% | 0.0% | 100.0 | 0 |
|
||
|
||
**`winnable`** is the column that is not about a bot. It is the fraction
|
||
of deals in which an **exhaustive search finds a winning line in the final
|
||
round**. The other three columns are what named policies actually achieve,
|
||
and **`spread`** is the range between them.
|
||
|
||
## 2. How to read it, including what it is not
|
||
|
||
**Read `spread` first.** It ranges from 71.7 to 100.0 percentage points.
|
||
Wherever it is large — everywhere — **no single policy's rate tells you
|
||
anything about the game.** That is the finding behind §0.
|
||
|
||
**`winnable` is conditioned on greedy's play up to the final round.** It
|
||
is *"winnable from where greedy got to"*, not a property of the deal
|
||
alone. A genuinely policy-free figure would search from round 1, which we
|
||
measured as unaffordable: our search exceeded 2×10⁶ nodes at **two** seats
|
||
over two rounds. We are telling you this rather than presenting the number
|
||
as cleaner than it is.
|
||
|
||
**It is a lower bound.** A one-round search cannot see a line that needed
|
||
an earlier round, so the true winnable fraction is **at least** these
|
||
figures.
|
||
|
||
**`undecided` deals are excluded, not counted as losses.** Those are deals
|
||
whose search hit the node budget. Folding *"we stopped looking"* into
|
||
*"not winnable"* is the collapse our spec forbids.
|
||
|
||
## 3. What we think this supports, offered as reading not as ruling
|
||
|
||
**The rules are yours. This is what the numbers appear to say.**
|
||
|
||
- **Two seats is the tight configuration.** 60% winnable in the final
|
||
round, and every policy struggles. If any seat count is under-tuned for
|
||
*difficulty*, the evidence does not point here.
|
||
- **Four seats and up: every deal we sampled was still winnable in the
|
||
final round.** 100% at 4, 5 and 6 seats. That is compatible with "too
|
||
generous at high seat counts" — but it is **not** the claim we withdrew
|
||
in §0, because it is a statement about deal winnability rather than
|
||
about a bot's success.
|
||
- **The 5–6 seat rows deserve the most suspicion.** `first-legal` scores
|
||
0% there while `greedy` scores 100%, which is the widest spread in the
|
||
table. Something about those configurations makes play matter *more*,
|
||
not less — the opposite of "too easy" — and we do not yet understand it.
|
||
|
||
**We are not proposing threshold changes.** The measurement's confound
|
||
(§2) is large enough that we would rather you saw it than acted on it.
|
||
|
||
## 4. What would make this a better instrument
|
||
|
||
- **Search from round 1** rather than the last round, which removes the
|
||
greedy confound entirely. Needs transposition or move-ordering; we have
|
||
neither.
|
||
- **A resolution figure** — the smallest threshold change the measurement
|
||
can distinguish, with its N. Our spec requires it and we cannot yet
|
||
supply it.
|
||
- **Your Pressure dial**, if it lands, gives a second axis to measure
|
||
against and would make the table two-dimensional.
|
||
|
||
## 5. Reproduction
|
||
|
||
Per GROUND-WP-0004 T02, arithmetic findings cite their reproduction by
|
||
path:
|
||
|
||
```
|
||
clay-borg$ make difficulty # the table above
|
||
clay-borg$ make self-tests # includes the difficulty controls
|
||
```
|
||
|
||
Controls, so the figure can fail rather than merely print: a witness must
|
||
replay to a win; an unwinnable position must be reported as **searched
|
||
out** rather than as a budget cut; a budget of one node must not claim
|
||
exhaustion; and the policy panel must actually disagree, or reporting
|
||
three policies would be ceremony.
|