From edda653db27e47cba5bd7b51d203c28f2963b10d Mon Sep 17 00:00:00 2001 From: tegwick Date: Wed, 5 Aug 2026 19:17:58 +0200 Subject: [PATCH] clay-borg difficulty report 2026-08-05: a baseline, and a retraction GROUND-WP-0005 has both tasks blocked on a measured difficulty baseline. This is that baseline, with its confound stated rather than hidden. It opens with a retraction. Earlier today clay-borg was going to report that a greedy bot wins 200 of 200 games at five and six seats and the game is too easy there -- which would have invited threshold changes. A FirstLegal policy scores 0% on the identical deals where greedy scores 100%, and at two seats it beats greedy. Two unsophisticated agents span the entire range, so a single policy's win rate is a statement about the policy. clay-borg's adversarial review caught it before it left the repo. Fifth wrong premise avoided, second stopped before sending. What is reported instead is the WINNABLE FRACTION -- the proportion of deals in which an exhaustive search finds a winning line in the final round -- alongside a plural policy panel and the SPREAD between policies. The spread is 71.7 to 100.0 percentage points, which is the direct evidence for why the single-policy figure was meaningless. Stated limits: the winnable figure is conditioned on greedy's play up to the final round (searching from round 1 exceeded 2x10^6 nodes at two seats), it is a lower bound, and budget-cut deals are excluded rather than counted as losses. No threshold changes are proposed. The confound is large enough that clay-borg would rather this repo saw it than acted on it. Co-Authored-By: Claude Opus 5 --- ...UND-WP-0005-clay-borg-difficulty-260805.md | 131 ++++++++++++++++++ 1 file changed, 131 insertions(+) create mode 100644 workplans/GROUND-WP-0005-clay-borg-difficulty-260805.md diff --git a/workplans/GROUND-WP-0005-clay-borg-difficulty-260805.md b/workplans/GROUND-WP-0005-clay-borg-difficulty-260805.md new file mode 100644 index 0000000..5788aac --- /dev/null +++ b/workplans/GROUND-WP-0005-clay-borg-difficulty-260805.md @@ -0,0 +1,131 @@ +--- +id: GROUND-WP-0005-REPORT-260805 +type: report +title: "clay-borg difficulty measurement, 2026-08-05 — and a retraction of what we nearly sent instead" +domain: consumer +repo: ground-game +status: informational +owner: bernd +topic_slug: whynot +created: "2026-08-05" +extends: GROUND-WP-0005 +--- + +# clay-borg difficulty report — 2026-08-05 + +GROUND-WP-0005 (*difficulty tiers and the Pressure dial*) has both tasks +in `wait`, blocked on a measured baseline. This is that baseline, with its +limits stated. + +Generated by `make difficulty` +(`games/ground/examples/difficulty.rs`), governed by +`clay-borg/specs/RetrospectiveAnalysis.md` §4, under +[CB-WP-0025](../../clay-borg/workplans/CB-WP-0025-could-we-have-won.md) T06. + +--- + +## 0. First, what we nearly sent you and did not + +**Earlier today this report was going to say: "a greedy bot wins 200 of +200 games at five and six seats — the game is too easy there."** + +It would have invited you to move thresholds. **It was wrong**, and an +adversarial review caught it before it left our repo. + +A `FirstLegal` policy — take the first legal command, no heuristic +whatsoever — scores **0%** at five and six seats on the identical deals +where `GreedyPolicy` scores **100%**. At two seats it *beats* greedy +(76.7% vs 60.0%). + +**Two unsophisticated agents span the entire range.** A single policy's +win rate is therefore a statement about the policy, not about GROUND, and +we have written that into our spec as a prohibition rather than a +preference. + +This is the fifth claim we would have sent you on a wrong premise. It is +the second stopped before sending. + +## 1. The measurement + +`make difficulty`, 60 seeds per cell: + +| seats | winnable | greedy | random | first-legal | spread | undecided | +|---|---:|---:|---:|---:|---:|---:| +| 2 | **60%** | 60.0% | 5.0% | 76.7% | 71.7 | 0 | +| 3 | **93%** | 88.3% | 6.7% | 25.0% | 81.7 | 2 | +| 4 | **100%** | 93.3% | 6.7% | 30.0% | 86.7 | 4 | +| 5 | **100%** | 100.0% | 3.3% | 0.0% | 100.0 | 0 | +| 6 | **100%** | 100.0% | 3.3% | 0.0% | 100.0 | 0 | + +**`winnable`** is the column that is not about a bot. It is the fraction +of deals in which an **exhaustive search finds a winning line in the final +round**. The other three columns are what named policies actually achieve, +and **`spread`** is the range between them. + +## 2. How to read it, including what it is not + +**Read `spread` first.** It ranges from 71.7 to 100.0 percentage points. +Wherever it is large — everywhere — **no single policy's rate tells you +anything about the game.** That is the finding behind §0. + +**`winnable` is conditioned on greedy's play up to the final round.** It +is *"winnable from where greedy got to"*, not a property of the deal +alone. A genuinely policy-free figure would search from round 1, which we +measured as unaffordable: our search exceeded 2×10⁶ nodes at **two** seats +over two rounds. We are telling you this rather than presenting the number +as cleaner than it is. + +**It is a lower bound.** A one-round search cannot see a line that needed +an earlier round, so the true winnable fraction is **at least** these +figures. + +**`undecided` deals are excluded, not counted as losses.** Those are deals +whose search hit the node budget. Folding *"we stopped looking"* into +*"not winnable"* is the collapse our spec forbids. + +## 3. What we think this supports, offered as reading not as ruling + +**The rules are yours. This is what the numbers appear to say.** + +- **Two seats is the tight configuration.** 60% winnable in the final + round, and every policy struggles. If any seat count is under-tuned for + *difficulty*, the evidence does not point here. +- **Four seats and up: every deal we sampled was still winnable in the + final round.** 100% at 4, 5 and 6 seats. That is compatible with "too + generous at high seat counts" — but it is **not** the claim we withdrew + in §0, because it is a statement about deal winnability rather than + about a bot's success. +- **The 5–6 seat rows deserve the most suspicion.** `first-legal` scores + 0% there while `greedy` scores 100%, which is the widest spread in the + table. Something about those configurations makes play matter *more*, + not less — the opposite of "too easy" — and we do not yet understand it. + +**We are not proposing threshold changes.** The measurement's confound +(§2) is large enough that we would rather you saw it than acted on it. + +## 4. What would make this a better instrument + +- **Search from round 1** rather than the last round, which removes the + greedy confound entirely. Needs transposition or move-ordering; we have + neither. +- **A resolution figure** — the smallest threshold change the measurement + can distinguish, with its N. Our spec requires it and we cannot yet + supply it. +- **Your Pressure dial**, if it lands, gives a second axis to measure + against and would make the table two-dimensional. + +## 5. Reproduction + +Per GROUND-WP-0004 T02, arithmetic findings cite their reproduction by +path: + +``` +clay-borg$ make difficulty # the table above +clay-borg$ make self-tests # includes the difficulty controls +``` + +Controls, so the figure can fail rather than merely print: a witness must +replay to a win; an unwinnable position must be reported as **searched +out** rather than as a budget cut; a budget of one node must not claim +exhaustion; and the policy panel must actually disagree, or reporting +three policies would be ceremony.