CB-WP-0025 T06/T07: a difficulty baseline that reports its own confound
Some checks failed
ci / check (push) Has been cancelled
Some checks failed
ci / check (push) Has been cancelled
T06. games/ground/examples/difficulty.rs, make difficulty, wired into make self-tests, and a report file in ground-game under GROUND-WP-0005 with a hub message pointing at it. THE REPORT OPENS WITH THE RETRACTION, because what this task was written to send was withdrawn by T02 and GROUND-WP-0005 is blocked on exactly that number. They are told, in the first section, that we nearly sent them "the game is too easy at 5-6 seats" and why it was wrong. seats winnable greedy random first-legal spread 2p 60% 60.0% 5.0% 76.7% 71.7 3p 93% 88.3% 6.7% 25.0% 81.7 4p 100% 93.3% 6.7% 30.0% 86.7 5p 100% 100.0% 3.3% 0.0% 100.0 6p 100% 100.0% 3.3% 0.0% 100.0 SPREAD justifies the whole redesign: 71.7 to 100.0 points between three trivial policies. The table now shows why no single rate is a difficulty rather than asserting it. And the 5-6 rows point the OPPOSITE way from the withdrawn claim -- first-legal 0% against greedy 100% is the widest spread in the table, which suggests play matters MORE there, not less. Neither reading is established and the report says so. The confound is stated in the tool's own output, not only in prose: `winnable` is conditioned on greedy's play up to the final round, because searching from round 1 is unaffordable. Presenting it as a property of the deal would repeat this pass's error in a subtler form -- which is exactly how a corrected project reintroduces a defect. NO THRESHOLD CHANGES ARE PROPOSED. The instrument can fail (spec §5): a witness must replay to a win, an unwinnable position must report searched-out rather than a budget cut, a one-node budget must not claim exhaustion, and the policy panel must actually disagree. difficulty-baseline.rs marked superseded, kept as the survey's dated snapshot. Registered as F16, inconsistent / withdrawn. T07. evidence/CB-EV-0024. Five of nine defects came only from the review; four from execution, and all four of those were in work written after it. The wrong-denominator family now has five instances and still no control -- facts-check catches copies that disagree, nothing catches a number computed correctly against the wrong base. Tier L was an over-declaration (no port, structurally M) and paid for itself anyway, because the review is L-only. Chaos window 2 will close with zero overrides, making its retirement condition untestable. Named as open rather than implied done: the witness is NOT wired to the ending page. The search works; the browser cannot ask it yet. make all: exit 0. loop-lint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
81e0aba59a
commit
7ed9fc730a
6 changed files with 583 additions and 79 deletions
161
evidence/CB-EV-0024-could-we-have-won.md
Normal file
161
evidence/CB-EV-0024-could-we-have-won.md
Normal file
|
|
@ -0,0 +1,161 @@
|
|||
# CB-EV-0024 — could we have won
|
||||
|
||||
CB-WP-0025 T07. Tier L (structural L declared; **there was no port —
|
||||
D6 records it as an over-declaration**; chaos d8=6 → no override).
|
||||
Declaration 8 of chaos window 2. Closed 2026-08-05.
|
||||
|
||||
**Delivered:** [CB-RES-0008](../research/CB-RES-0008-could-we-have-won.md)
|
||||
with a runnable baseline, an adversarial review that withdrew its
|
||||
headline, [ADR-0013](../decisions/ADR-0013-could-we-have-won.md),
|
||||
[RetrospectiveAnalysis.md](../specs/RetrospectiveAnalysis.md),
|
||||
`games/ground/src/search.rs`, `benches/search.rs`,
|
||||
`examples/difficulty.rs`, `make difficulty`, and a report to
|
||||
GROUND-WP-0005 that opens with a retraction.
|
||||
|
||||
---
|
||||
|
||||
## 1. Was the maintainer's game winnable?
|
||||
|
||||
The question the pass exists for. **Yes, and the tool answers it.**
|
||||
|
||||
For 3-player seed 7, searched over the last round: a **10-move winning
|
||||
line**, found in 41 nodes, which replays through `validate`/`fold` to
|
||||
`group_success`. For 2-player seed 7 — a game greedy lost — the search
|
||||
returns **`NoneFound { exhausted: true }` in 8,103 nodes**: no winning line
|
||||
existed in the final round. That is a real negative, not a budget cut, and
|
||||
the type keeps the two apart.
|
||||
|
||||
**The feature works. The difficulty half is where the trouble was.**
|
||||
|
||||
## 2. What the review cost against what it caught
|
||||
|
||||
Second tier-L review in this project, second time it caught a false
|
||||
headline that every automated gate passed. `make all` was green
|
||||
throughout.
|
||||
|
||||
| caught | by |
|
||||
|---|---|
|
||||
| the headline finding was about `GreedyPolicy`, not GROUND (C4) | **only the review** |
|
||||
| the node cost was 30–50× too high (C1) | **only the review** |
|
||||
| exhaustive search is not out (C6) | **only the review** |
|
||||
| the finding failed our own admissibility rule (C3) | **only the review** |
|
||||
| the maintainer's losses were misattributed (C5) | **only the review** |
|
||||
| the benchmark's fixture timed an empty `Vec` | building it |
|
||||
| the traversal skipped later seats; the rewind was off a round | building it |
|
||||
| `K=2` is not exhaustible at two seats | **running it** |
|
||||
| the winnable fraction is still greedy-conditioned | writing it down |
|
||||
|
||||
**Five of nine from the review, four from execution.** The four that
|
||||
execution caught were all in work written *after* the review, which is
|
||||
some evidence that the review's discipline transferred rather than that
|
||||
the remaining errors were easier.
|
||||
|
||||
**The reviewer under-ranked their own best challenge.** They put C4
|
||||
fourth. Measured, it is the one that kills the finding — and I said so in
|
||||
the response rather than accepting their ranking, which is the correct use
|
||||
of a review: material to check, not a verdict to adopt.
|
||||
|
||||
## 3. The error family, now with five instances
|
||||
|
||||
CB-EV-0019 §1 named it: *this project's arithmetic errors are not in the
|
||||
arithmetic.* A number computed correctly against the wrong base, span, or
|
||||
subject.
|
||||
|
||||
| # | pass | the wrong thing |
|
||||
|---|---|---|
|
||||
| 1 | CB-WP-0021 | `csv` cost against AM-4a, which never sees the code |
|
||||
| 2 | ground-game report | *"12 in the file"* — a sum with no deal table |
|
||||
| 3 | CB-WP-0018 | SOLVE's inertness attributed to the wrong condition |
|
||||
| 4 | **this pass, C1** | a timer bracketing whole games, divided by decisions |
|
||||
| 5 | **this pass, C4** | a win rate attributed to the game, not the policy |
|
||||
|
||||
**And two projections published in place of measurements**, both in this
|
||||
pass: the survey's `7.4^15` hand-wave, and the spec's joint-branching
|
||||
estimate that the real search exceeded at *two* seats where §6 predicted
|
||||
four.
|
||||
|
||||
**`facts-check` catches copies that disagree. Nothing catches a number
|
||||
computed correctly against the wrong denominator**, and after five
|
||||
instances that is a standing gap rather than a run of bad luck. Naming it
|
||||
is what this file can do; closing it is not this pass's.
|
||||
|
||||
## 4. What the difficulty measurement actually shows
|
||||
|
||||
`make difficulty`, 60 seeds:
|
||||
|
||||
| seats | winnable | greedy | random | first-legal | spread |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| 2 | 60% | 60.0% | 5.0% | 76.7% | 71.7 |
|
||||
| 3 | 93% | 88.3% | 6.7% | 25.0% | 81.7 |
|
||||
| 4 | 100% | 93.3% | 6.7% | 30.0% | 86.7 |
|
||||
| 5 | 100% | 100.0% | 3.3% | 0.0% | 100.0 |
|
||||
| 6 | 100% | 100.0% | 3.3% | 0.0% | 100.0 |
|
||||
|
||||
**The `spread` column is the finding.** 71.7 to 100.0 points between three
|
||||
trivial policies. It is the evidence for the prohibition in
|
||||
RetrospectiveAnalysis §4.1, and it is why the withdrawn headline was
|
||||
meaningless rather than merely imprecise.
|
||||
|
||||
**The 5–6 seat rows point the opposite way from the withdrawn claim.**
|
||||
`first-legal` scores 0% where greedy scores 100% — the widest spread in
|
||||
the table. Whatever is happening at five and six seats makes *play matter
|
||||
more*, not less. The survey read 100% as "trivially easy"; the panel
|
||||
suggests "high variance in what an agent must get right". **Neither is
|
||||
established**, and saying so is the honest state.
|
||||
|
||||
**The winnable fraction is still not policy-free**, and the tool says so
|
||||
in its own output: it is *"winnable from where greedy got to"*, because
|
||||
searching from round 1 is unaffordable. Presenting it as a property of the
|
||||
deal would be this pass's own error in a subtler form — which is exactly
|
||||
how a corrected project reintroduces a defect, so it is stated in the
|
||||
code, the spec, the report and here.
|
||||
|
||||
## 5. Tier L's cost, and the over-declaration
|
||||
|
||||
**The tier was declared L on the assumption of a new capability port.
|
||||
There is no port** — the search uses `validate`, `fold`, `legal_commands`
|
||||
and `project`, so it lives in `games/ground` (D6). By the structural
|
||||
trigger this was an **M**.
|
||||
|
||||
**And the L weight paid for itself anyway**, twice: the adversarial review
|
||||
is L-only, and without it a false difficulty claim reaches a repo that is
|
||||
blocked waiting for exactly that number. **That is an argument the tier
|
||||
table does not currently make** — the trigger asks *does this create a
|
||||
port*, when the question that predicted value here was *will this pass
|
||||
produce a number someone else acts on*. Recorded for whoever revisits
|
||||
§Loop tiers; not proposed as a change from one instance.
|
||||
|
||||
## 6. Chaos window 2
|
||||
|
||||
**Declaration 8 of 12.** Structural L (over-declared), d8 = 6, no
|
||||
override.
|
||||
|
||||
**Eleven declarations, no 8 rolled.** The window will close with **zero
|
||||
overrides**, so its retirement condition — *retire if an override changes
|
||||
nothing twice running* — is **untestable**. Window 1 closed with two
|
||||
overrides at d4 and both changed the outcome.
|
||||
|
||||
**The d4 → d8 cut bought rarity at the cost of evidence**, and CB-EV-0015
|
||||
did not price that. At d8 over twelve declarations the expected count is
|
||||
1.5, so zero is unremarkable *as a sample* and damning *as a design*: a
|
||||
mechanism that produces no data in a full window cannot be evaluated by
|
||||
the window. **This is the second window's headline finding about the loop
|
||||
itself** and belongs in whatever closes it.
|
||||
|
||||
## 7. Cost
|
||||
|
||||
CB-WP-0024's cost, by re-running the instrument: `make cost`. Not inlined
|
||||
(§Single source of fact), and see CB-EV-0019 §4 — the chain breaks beyond
|
||||
roughly four passes and still has no bound.
|
||||
|
||||
## Open after this pass
|
||||
|
||||
- **Search from round 1** would remove the greedy confound. Needs
|
||||
transposition or move-ordering; neither built.
|
||||
- **Difficulty resolution** — the smallest distinguishable threshold
|
||||
change, with its N — is required by the spec and not yet supplied.
|
||||
- **The witness is not wired to the ending page.** T05 built the search;
|
||||
the browser cannot yet ask it. That is the maintainer-facing half of
|
||||
remark 2 and it is not done.
|
||||
- **The wrong-denominator family has no control** (§3), at five instances.
|
||||
- **The chaos window's retirement condition is untestable** (§6).
|
||||
Loading…
Add table
Add a link
Reference in a new issue