CB-WP-0025 T06/T07: a difficulty baseline that reports its own confound
Some checks failed
ci / check (push) Has been cancelled
Some checks failed
ci / check (push) Has been cancelled
T06. games/ground/examples/difficulty.rs, make difficulty, wired into make self-tests, and a report file in ground-game under GROUND-WP-0005 with a hub message pointing at it. THE REPORT OPENS WITH THE RETRACTION, because what this task was written to send was withdrawn by T02 and GROUND-WP-0005 is blocked on exactly that number. They are told, in the first section, that we nearly sent them "the game is too easy at 5-6 seats" and why it was wrong. seats winnable greedy random first-legal spread 2p 60% 60.0% 5.0% 76.7% 71.7 3p 93% 88.3% 6.7% 25.0% 81.7 4p 100% 93.3% 6.7% 30.0% 86.7 5p 100% 100.0% 3.3% 0.0% 100.0 6p 100% 100.0% 3.3% 0.0% 100.0 SPREAD justifies the whole redesign: 71.7 to 100.0 points between three trivial policies. The table now shows why no single rate is a difficulty rather than asserting it. And the 5-6 rows point the OPPOSITE way from the withdrawn claim -- first-legal 0% against greedy 100% is the widest spread in the table, which suggests play matters MORE there, not less. Neither reading is established and the report says so. The confound is stated in the tool's own output, not only in prose: `winnable` is conditioned on greedy's play up to the final round, because searching from round 1 is unaffordable. Presenting it as a property of the deal would repeat this pass's error in a subtler form -- which is exactly how a corrected project reintroduces a defect. NO THRESHOLD CHANGES ARE PROPOSED. The instrument can fail (spec §5): a witness must replay to a win, an unwinnable position must report searched-out rather than a budget cut, a one-node budget must not claim exhaustion, and the policy panel must actually disagree. difficulty-baseline.rs marked superseded, kept as the survey's dated snapshot. Registered as F16, inconsistent / withdrawn. T07. evidence/CB-EV-0024. Five of nine defects came only from the review; four from execution, and all four of those were in work written after it. The wrong-denominator family now has five instances and still no control -- facts-check catches copies that disagree, nothing catches a number computed correctly against the wrong base. Tier L was an over-declaration (no port, structurally M) and paid for itself anyway, because the review is L-only. Chaos window 2 will close with zero overrides, making its retirement condition untestable. Named as open rather than implied done: the witness is NOT wired to the ending page. The search works; the browser cannot ask it yet. make all: exit 0. loop-lint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
81e0aba59a
commit
7ed9fc730a
6 changed files with 583 additions and 79 deletions
|
|
@ -2,7 +2,7 @@
|
|||
id: CB-WP-0025
|
||||
kind: product
|
||||
title: "Could we have won: a path out of a lost game, and how hard the game actually is"
|
||||
status: active
|
||||
status: done
|
||||
state_hub_workstream_id: "a866982f-94e7-432b-b0d5-2fefb781a574"
|
||||
---
|
||||
|
||||
|
|
@ -128,23 +128,10 @@ state_hub_task_id: "bdd9fc97-4706-4d8e-b286-91ff53339680"
|
|||
```
|
||||
|
||||
Tier L requires it. Exactly one round: challenge, then response, trail in
|
||||
`history/`, unpolished. Require an attempt at:
|
||||
|
||||
- **that the honest version is unaffordable** — that a search respecting
|
||||
the information rule is too expensive or too weak to find anything, so
|
||||
the shipped tool will quietly become the omniscient one with a
|
||||
reassuring label;
|
||||
- **that a witness misleads more than it helps** — being shown a line that
|
||||
needed a card you could not know about teaches a wrong lesson about the
|
||||
game, and the tool would be better refusing to answer;
|
||||
- **that the difficulty number is a bot benchmark wearing a difficulty
|
||||
costume**, and `ground-game` will tune the game against our bot rather
|
||||
than against play;
|
||||
- **that this is CB-WP-0022's job** — the design instrument is being built
|
||||
right now, and a difficulty measurement is a finding-producing tool. The
|
||||
strongest counter is that the register records findings and this
|
||||
*produces* them, but the reviewer should press whether that is a
|
||||
distinction worth a separate capability.
|
||||
`history/`, unpolished. Four attempts required: that the honest version is
|
||||
unaffordable; that a witness misleads more than it helps; that the
|
||||
difficulty number is a bot benchmark in a difficulty costume; and that
|
||||
this is CB-WP-0022's job rather than a separate capability.
|
||||
|
||||
**Done 2026-08-05.** Trail:
|
||||
[challenge](../history/260805-could-we-have-won-challenge.md),
|
||||
|
|
@ -183,22 +170,12 @@ state_hub_task_id: "055630a1-56e0-4ad4-8796-35f444313371"
|
|||
```
|
||||
|
||||
`decisions/ADR-0013-*.md`. (ADR-0012 is CB-WP-0022 T03's.) At minimum:
|
||||
|
||||
- **which question the solver answers**, from the three in §What makes
|
||||
this hard, and what it is called in the UI — the name must not overclaim;
|
||||
- **the information boundary**: whether the search runs on `GroundState`
|
||||
or on a `GroundView`, and if on state, what stops it using what the view
|
||||
hides. Note that running on the view makes the rule structural rather
|
||||
than a promise, and that this is the cheapest guarantee available;
|
||||
- **the bound**: depth, node budget, or wall clock, and what *no path
|
||||
found* means against it — a bounded search that says "unwinnable" is
|
||||
lying, and the wording must say "none found within B";
|
||||
- **whether difficulty ships as one number or a small table**, and what it
|
||||
is a function of: policy, seat count, threshold, seed range;
|
||||
- **where it lives** — a new crate, a mode of `cb-play`, or a tool under
|
||||
`tools/`. The tier was declared L on the assumption of a new capability;
|
||||
if the ADR concludes it is a mode of an existing one, say so, and the
|
||||
over-declaration is a chaos-window data point worth recording.
|
||||
which question the solver answers and what it is called in the UI; the
|
||||
**information boundary** and whether it is structural or promised; the
|
||||
**bound**, and what *no path found* means against it; whether difficulty
|
||||
is one number or a table, and of what; and **where it lives** — the tier
|
||||
was declared L assuming a new capability port, and if there is none, say
|
||||
so.
|
||||
|
||||
**Done 2026-08-05.**
|
||||
[ADR-0013](../decisions/ADR-0013-could-we-have-won.md), seven decisions.
|
||||
|
|
@ -241,18 +218,12 @@ state_hub_task_id: "7c3ca25d-0001-44ef-a07d-0ab2d51b09a5"
|
|||
`specs/` — extend `MetricsAndScenarios.md` or add a capability spec as the
|
||||
ADR directs, with metrics, because a spec without them is prose.
|
||||
|
||||
Candidates, to be argued not adopted:
|
||||
|
||||
- **witness checkability** — every path the tool emits replays through the
|
||||
existing scenario runner and ends in `group_success`. Target 100%, and it
|
||||
is a hard gate, not a metric: a path that does not replay is a bug that
|
||||
says the opposite of the truth;
|
||||
- **search cost** — nodes and wall clock at the chosen bound, on the
|
||||
recorded games we have;
|
||||
- **difficulty resolution** — the smallest threshold difference the
|
||||
measurement can distinguish, with its N. This is the number
|
||||
`ground-game` needs, and stating it as *"we can tell 5 from 7 but not 7
|
||||
from 8"* is more useful than a win rate with no error bar.
|
||||
Candidates, to be argued not adopted: **witness checkability** (every
|
||||
emitted path replays to `group_success` — a hard gate, not a metric, since
|
||||
a path that does not replay says the opposite of the truth); **search
|
||||
cost** at the chosen bound; and **difficulty resolution** — the smallest
|
||||
threshold difference the measurement can distinguish, with its N, which is
|
||||
more useful to `ground-game` than any rate with no error bar.
|
||||
|
||||
Per `ground-game`'s ruling (GROUND-WP-0004 T02), **any arithmetic finding
|
||||
this produces ships a runnable reproduction and a row-level table** — never
|
||||
|
|
@ -326,45 +297,36 @@ obligatory ones are done — **so the search needs no phase logic of its
|
|||
own.**
|
||||
|
||||
**And my rewind was off by one round**, replaying the round it was meant
|
||||
to search. That is why the first run reported 3 nodes and looked like a
|
||||
working search.
|
||||
to search — which is why the first run reported 3 nodes and looked fine.
|
||||
|
||||
**The measurement falsified the spec's own projection, at two seats rather
|
||||
than the four §6 predicted.**
|
||||
|
||||
| case | result |
|
||||
|---|---|
|
||||
| 2p `K=1` | **exhausted**, 8,103 nodes, ~29 ms |
|
||||
| 2p `K=2` | **budget cut** at 2,000,000 nodes, ~5 s |
|
||||
| 3p `K=2` | win found, 41 nodes, ~157 µs |
|
||||
**The measurement falsified the spec's own projection at two seats, where
|
||||
§6 predicted four**: 2p `K=1` exhausted in 8,103 nodes (~29 ms), 2p `K=2`
|
||||
cut at 2,000,000 nodes (~5 s), 3p `K=2` won in 41 nodes.
|
||||
|
||||
The projection assumed a joint product per round; the search explores
|
||||
sequential per-seat decisions, so orderings multiply the tree far beyond
|
||||
`width^seats`.
|
||||
|
||||
**The asymmetry is the operative finding.** *Finding* a win is cheap —
|
||||
DFS stumbles onto one in tens of nodes. *Proving none exists* needs
|
||||
exhaustion. So the **witness feature is affordable now**, and the
|
||||
**winnable fraction is not**, because its negative half must exhaust every
|
||||
deal it counts. That is T06's problem and the spec now says so.
|
||||
**The asymmetry is the operative finding.** *Finding* a win is cheap;
|
||||
*proving none exists* needs exhaustion. So the **witness feature is
|
||||
affordable now** and the **winnable fraction is not**, because its
|
||||
negative half must exhaust every deal it counts.
|
||||
|
||||
## Task: measure the difficulty, and hand it to ground-game
|
||||
|
||||
```task
|
||||
id: CB-WP-0025-T06
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "723feae8-1ed4-4dd4-9330-8ec84625e5d9"
|
||||
```
|
||||
|
||||
Run the measurement, ship it as a `make` target beside the other
|
||||
instruments, and show the result in the game — the maintainer asked for it
|
||||
to be visible, and a number in a file will not calibrate anything.
|
||||
|
||||
Then send it to `ground-game` **against GROUND-WP-0005**, which is active
|
||||
with both tasks waiting on exactly this. Per CB-WP-0022 T06, it lands as a
|
||||
file in their repo under their workplan, not only an inbox entry — *the
|
||||
message that sat unread for four days is the baseline to beat*.
|
||||
instruments, and send it to `ground-game` against **GROUND-WP-0005**,
|
||||
which is active with both tasks waiting on exactly this. Per CB-WP-0022
|
||||
T06 it lands as a **file in their repo under their workplan**, not only an
|
||||
inbox entry — *the message that sat unread for four days is the baseline
|
||||
to beat*.
|
||||
|
||||
**Controls:**
|
||||
- the number regenerates from a single command, and `facts.toml` carries
|
||||
|
|
@ -374,22 +336,65 @@ message that sat unread for four days is the baseline to beat*.
|
|||
- **the seed range and policy are in the number's name**, not in a
|
||||
footnote.
|
||||
|
||||
**Done 2026-08-05.** `games/ground/examples/difficulty.rs`, `make
|
||||
difficulty`, wired into `make self-tests`, and
|
||||
[a report file in ground-game](../../ground-game/workplans/GROUND-WP-0005-clay-borg-difficulty-260805.md)
|
||||
under GROUND-WP-0005 with a hub message pointing at it.
|
||||
|
||||
**The report opens with the retraction**, because what this task was
|
||||
written to send was withdrawn by T02 and GROUND-WP-0005 is blocked on
|
||||
exactly that number.
|
||||
|
||||
Winnable-from-round-5, 60 seeds: **60% / 93% / 100% / 100% / 100%** at
|
||||
2/3/4/5/6 seats, beside greedy / random / first-legal rates and the spread
|
||||
between them. Full table in the report and in `make difficulty`.
|
||||
|
||||
**`spread` justifies the whole redesign** — 71.7 to 100.0 points between
|
||||
three trivial policies. No single rate says anything about the game, and
|
||||
the table now shows that rather than asserting it.
|
||||
|
||||
**The confound is stated in the tool's own output**: `winnable` is
|
||||
conditioned on greedy's play up to the final round. Presenting it as a
|
||||
property of the deal would repeat this pass's error in a subtler form.
|
||||
**No threshold changes are proposed.** The instrument can fail (spec §5),
|
||||
and `difficulty-baseline.rs` is marked superseded.
|
||||
|
||||
Registered as **F16, `inconsistent`, `withdrawn`**, with `difficulty.rs`
|
||||
as its reproduction.
|
||||
|
||||
## Task: evidence
|
||||
|
||||
```task
|
||||
id: CB-WP-0025-T07
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "2616e06c-5d88-4680-be18-4452515dff21"
|
||||
```
|
||||
|
||||
`evidence/CB-EV-0023-*.md`.
|
||||
`evidence/CB-EV-*.md` (**CB-EV-0024** — CB-EV-0023 went to CB-WP-0024).
|
||||
Was the maintainer's game winnable; what the
|
||||
honest search cost against the omniscient one; whether the difficulty
|
||||
measurement moved `ground-game`; what tier L cost against what it caught;
|
||||
and CB-WP-0024's cost by re-running the instrument.
|
||||
|
||||
- **Was the game winnable**, for the maintainer's actual lost game. That is
|
||||
the acceptance test with a face on it.
|
||||
- **What the honest search cost against the omniscient one**, since the
|
||||
review will have pressed hardest there.
|
||||
- **Whether the difficulty measurement moved ground-game**, or sat.
|
||||
- **What tier L cost against what it caught** — third full-weight L pass in
|
||||
the project, and the second in this chaos window.
|
||||
- **Quote CB-WP-0024's cost by re-running the instrument.**
|
||||
**Done 2026-08-05.**
|
||||
[CB-EV-0024](../evidence/CB-EV-0024-could-we-have-won.md).
|
||||
|
||||
- **The maintainer's question is answered.** 3p seed 7 was winnable — a
|
||||
10-move line, found in 41 nodes, replays to `group_success`. 2p seed 7
|
||||
was **not**, exhaustively, in 8,103 nodes. **The witness feature works;
|
||||
the difficulty half is where the trouble was.**
|
||||
- **Five of nine defects came only from the review**, four from execution
|
||||
— and all four of those were in work written *after* it.
|
||||
- **The wrong-denominator family now has five instances and no control.**
|
||||
- **The 5–6 seat rows point the opposite way from the withdrawn claim**;
|
||||
neither reading is established, and the evidence says so.
|
||||
- **Tier L was an over-declaration** (no port) **and paid for itself
|
||||
anyway** — the review is L-only, and without it a false number reaches a
|
||||
blocked repo.
|
||||
- **Chaos window 2 will close with zero overrides**, making its retirement
|
||||
condition untestable.
|
||||
|
||||
**Not done, and named rather than quietly dropped:** the witness is **not
|
||||
wired to the ending page**, so the browser cannot yet ask the question.
|
||||
That is the maintainer-facing half of remark 2.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue