tegwick
2806b3acb7
T02 — all chance derives from one root seed. Three chance points, all reading it: the setup deck shuffle, the setup Lead draw, and the reshuffle permutation. The Problems deal is not chance at all. So in extensive-form terms the tree has a single chance node at the root. That test was wrong first, and the mutation caught it. It compared state hashes — and GroundState carries `seed` as a field, so "different seeds differ" was true by construction. Mutating the shuffle away left it green. It now compares the dealt configuration, and the same mutation fails it: a wrong-subject error inside the control written for T02. The reshuffle is a pure function of (seed, round) because K5 requires deterministic replay, where a real table reshuffles independently. That is a modelling restriction, not a defect, and it is now pinned. T03 — commit/reveal checked in both directions: before Reveal each seat sees its own selection and no other; after Reveal the information sets merge, because an encoding that hides forever is not commit/reveal either. T04 — ADR-0020 refuses the EFG port, and the blocker is T02 rather than T01, which inverts what the workplan expected. Perfect recall looked like the risk and is a constraint with a known answer: key on observation histories. Making chance explicit is the expensive one — the reshuffle would become a real chance node and break the K5 purity that every recording, replay bundle and trial-note hash depends on. A port would trade the property this project is built on for one it has never needed. Track B's first move is therefore a question, not a build: take "is exploitability meaningful for a co-operative game with a shared threshold" to OpenSpiel on a toy model, where answering it costs nothing. D4 states what being wrong looks like — OpenSpiel settling on a toy what three rounds of policy sweeps could not — and makes watching for it the next action. Taxonomy §4.1 records the EFG correspondence with the test that checks each row, so a later pass starts from a specification rather than a memory. Chaos window 4 at three declarations. Window 3's verdict is now two windows behind and should be evaluated rather than restated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
10 KiB
Taxonomy — which system is a statement about?
Status: draft, normative for new work. Changes go through a decision
record in decisions/.
This document exists because the words were doing too much work. "Error", "failure", "finding" and "correction" were each being used for the game's design, our formalisation of it, the code, the measuring apparatus, and the sentences we wrote — five different things, one vocabulary.
Every defect statement names a stratum before it names anything else. "A bug in the simulator" and "a gap in the game" are not the same kind of claim, do not have the same owner, and are not fixed by the same person.
Grounded in existing practice rather than invented here: the fault/error/failure chain from Avizienis, Laprie, Randell & Landwehr (Basic Concepts and Taxonomy of Dependable and Secure Computing, IEEE TDSC 1:1, 2004); the model strata and the verification/validation split from Sargent (Verification and Validation of Simulation Models); and MDA (Hunicke, LeBlanc & Zubek, 2004) for the game-facing layers.
1. The strata
| stratum | canon | what it is | who owns it |
|---|---|---|---|
| GAME | Sargent's problem entity; MDA mechanics as designed | the rules as the game's owner defines them — the edition, the printed text | the game's owner, not us |
| MODEL | Sargent's conceptual model | our formalisation: GroundRules.md, the rule ids, the state we chose to represent |
us, answerable to the owner |
| ENGINE | Sargent's computerized model | the kernel that implements the model | us |
| INSTRUMENT | measurement apparatus | harnesses, policies, metrics, gates, the trial protocol | us |
| ACCOUNT | — | evidence files, register entries, messages, this document | us |
| PRESENTATION | MDA aesthetics-facing | the page, and everything Ornamentation.md covers |
us, later clay-animate |
1.1 The two relations that matter
GAME ↔ MODEL is validation. Did we formalise the right game?
MODEL ↔ ENGINE is verification. Did we implement the formalisation correctly?
This is Sargent's split and it is the one we kept losing. A statement that the engine mis-scores a Bond is verification. A statement that the rules do not say what happens when two Bonds break simultaneously is validation — and it is not ours to answer.
Sargent's third leg is data validity, and it is where
editions/ lives: the vendored CSVs, their digests, and the tripwires in
edition.rs are data validation, not verification.
1.2 What each stratum's defects look like
| stratum | a defect here looks like |
|---|---|
| GAME | a rule that does not decide; a dominant strategy; an action that earns its place nowhere |
| MODEL | we represented a rule the edition does not have, or omitted one it does |
| ENGINE | the code does not do what GroundRules.md says |
| INSTRUMENT | the measurement answers a different question than the one asked |
| ACCOUNT | the sentence claims more than the artifact shows |
| PRESENTATION | the player cannot see, or misreads, what the state says |
2. Fault, error, failure
Avizienis' chain, unchanged, applied within a stratum.
| term | definition | example, INSTRUMENT stratum |
|---|---|---|
| fault | the defect in the artifact — the wrong line, the wrong column, the wrong constant | peak maximised over StressSet payloads |
| error | the wrong internal state it produces | the metric holds 1 where the game held 2 |
| failure | the deviation visible at the interface | the evidence file says "peak Stress was 1" |
A fault is dormant until activated. attack-value.rs's missing
assertion was a fault for weeks and produced no error, because no cell was
ever short.
2.1 The rule that explains our history
A failure in one stratum is a fault in the next.
An INSTRUMENT failure (a metric reporting the wrong number) is an ACCOUNT
fault (a sentence built on it). An ACCOUNT failure that reaches the owner
is a fault in their decision-making. That is why
GameDesign.md §1 gates what may leave the repo: it is a
firebreak in this chain, not politeness.
2.2 The wrong-subject error, defined at last
ADR-0018 named a family and could not say precisely what it was. In this vocabulary:
A wrong-subject error is an ACCOUNT failure with no INSTRUMENT fault. The measurement is correct about its subject; the sentence is about a different one.
peak Stress was 1 — the metric correctly reported the maximum assigned
Stress. Reactive is greedy with one preference changed — the policy
correctly implemented what was written. Nothing was miscomputed in
either case, which is exactly why tests did not catch them and why the
fix is never "add a test to the engine".
3. The vocabulary, fixed
| word | means | does not mean |
|---|---|---|
| finding | a statement about the GAME stratum, admissible under GameDesign §1, addressed to the owner | any defect we happen to notice |
| defect | a fault in any stratum we own — MODEL, ENGINE, INSTRUMENT, ACCOUNT, PRESENTATION | a design question |
| fix | a change to the artifact that removes a fault | a change to what we said |
| correction | a change to the ACCOUNT — a claim withdrawn, restated or re-derived | a code change |
| retraction | a correction of something that has already left the repo | an internal edit |
| regression | a fault introduced by a change, in any stratum | any failure |
| inert control | a control that cannot fail, so proves nothing (ADR-0006 D3) | a control that happens to be green |
The register carries findings. Where an entry is really a defect in a stratum we own — F18, F24, F25 are ENGINE/data — the row says so, because "the engine has not read this file" and "the game does not decide this" are different claims to a reader deciding what to act on.
4. What the layers measure, and what they cannot
MDA gives the game-facing split, and it maps onto instruments we already have:
| MDA layer | where it lives here | what measures it |
|---|---|---|
| Mechanics | GAME → MODEL → ENGINE | the rule-coverage gate, the edition tripwires |
| Dynamics — the behaviour that emerges in play | what the ENGINE produces over many games | the policy panels: win rates, DARVO arm rates, action selection |
| Aesthetics — the player's response | not representable in state | trial notes, and nothing else |
Our panels measure dynamics. Our trial logs sample aesthetics. Neither substitutes for the other, and the recurring temptation is to report a dynamics number as though it settled an aesthetics question — "is the game fun at 5 seats" is not answered by a win rate.
MDA's own directional insight is the reason: the designer builds mechanics upward hoping for a feeling; the player meets the feeling first and infers the rules. A simulator sits at the bottom of that arrow and can only ever report upward.
4.1 The MODEL stratum has a formal name now
The extensive-form game is what our MODEL stratum corresponds to, and the correspondence is established rather than assumed (CB-WP-0041):
| EFG component | ours | checked by |
|---|---|---|
| histories | the journal | — |
| actions | bot::legal_commands |
— |
| information sets | the seat's observation history, not project() |
perfect-recall |
| chance | a single root node; the reshuffle is correlated with it | a_game_is_determined_by_its_seed |
| simultaneity | commit/reveal, the textbook encoding | a_pending_selection_is_hidden_until_reveal |
| payoffs | Outcome / score() |
— |
project() is an observation, not an information state, and the
distinction cost 22 measured violations to find. It is the same split
OpenSpiel draws between ObservationString and InformationStateString.
No port was built — ADR-0020 refuses it, because making chance explicit would break the replay purity the evidence discipline rests on.
5. Worked examples from this repo
Because a taxonomy with no instances was invented rather than observed (GameDesign §2).
| what happened | stratum | fault / error / failure |
|---|---|---|
solution_deck() is a Rust literal that never reads Solutions.csv |
ENGINE (data validity) | dormant fault; no error, because the literal agrees |
score() read self while H1's pressure went into work |
ENGINE | fault; error only at round 5; no measured failure |
| GR-P05 — SOLVE offered where it could not act | MODEL, then GAME | our failure, escalated to a GAME finding, ruled by the owner |
| ATTACK earns its place in no mode (F17) | GAME | a finding — not a defect at all |
peak Stress was 1 |
ACCOUNT | failure with no instrument fault (§2.2) |
| the panels ran under no gate | INSTRUMENT | fault — every published figure came from an ungated binary |
| the note channel closed when the game ended | PRESENTATION | failure; the state was right, the player could not reach it |
Read the stratum column. Six rows, six different owners and six different fixes, and every one of them was called "an error" at the time.
6. Falsifier
If a defect arises that cannot be placed in exactly one stratum, the strata are wrong — either too few, or cut in the wrong place. Record the instance rather than widening a definition to swallow it, which is how a taxonomy stops distinguishing anything.
A second falsifier, from §2.2: if a wrong-subject error is ever found that did have an instrument fault, then the family is not what ADR-0018 and this document say it is.
Sources
- Avizienis, Laprie, Randell & Landwehr, Basic Concepts and Taxonomy of Dependable and Secure Computing, IEEE TDSC 1(1), 2004.
- Sargent, Verification and Validation of Simulation Models, Winter Simulation Conference.
- Hunicke, LeBlanc & Zubek, MDA: A Formal Approach to Game Design and Game Research, AAAI Workshop, 2004.