tegwick
2806b3acb7
Some checks failed
ci / check (push) Failing after 3s
T02 — all chance derives from one root seed. Three chance points, all reading it: the setup deck shuffle, the setup Lead draw, and the reshuffle permutation. The Problems deal is not chance at all. So in extensive-form terms the tree has a single chance node at the root. That test was wrong first, and the mutation caught it. It compared state hashes — and GroundState carries `seed` as a field, so "different seeds differ" was true by construction. Mutating the shuffle away left it green. It now compares the dealt configuration, and the same mutation fails it: a wrong-subject error inside the control written for T02. The reshuffle is a pure function of (seed, round) because K5 requires deterministic replay, where a real table reshuffles independently. That is a modelling restriction, not a defect, and it is now pinned. T03 — commit/reveal checked in both directions: before Reveal each seat sees its own selection and no other; after Reveal the information sets merge, because an encoding that hides forever is not commit/reveal either. T04 — ADR-0020 refuses the EFG port, and the blocker is T02 rather than T01, which inverts what the workplan expected. Perfect recall looked like the risk and is a constraint with a known answer: key on observation histories. Making chance explicit is the expensive one — the reshuffle would become a real chance node and break the K5 purity that every recording, replay bundle and trial-note hash depends on. A port would trade the property this project is built on for one it has never needed. Track B's first move is therefore a question, not a build: take "is exploitability meaningful for a co-operative game with a shared threshold" to OpenSpiel on a toy model, where answering it costs nothing. D4 states what being wrong looks like — OpenSpiel settling on a toy what three rounds of policy sweeps could not — and makes watching for it the next action. Taxonomy §4.1 records the EFG correspondence with the test that checks each row, so a later pass starts from a specification rather than a memory. Chaos window 4 at three declarations. Window 3's verdict is now two windows behind and should be evaluated rather than restated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
208 lines
10 KiB
Markdown
208 lines
10 KiB
Markdown
# Taxonomy — which system is a statement about?
|
|
|
|
Status: **draft**, normative for new work. Changes go through a decision
|
|
record in `decisions/`.
|
|
|
|
This document exists because the words were doing too much work. *"Error"*,
|
|
*"failure"*, *"finding"* and *"correction"* were each being used for the
|
|
game's design, our formalisation of it, the code, the measuring apparatus,
|
|
and the sentences we wrote — five different things, one vocabulary.
|
|
|
|
> **Every defect statement names a stratum before it names anything else.**
|
|
> "A bug in the simulator" and "a gap in the game" are not the same kind of
|
|
> claim, do not have the same owner, and are not fixed by the same person.
|
|
|
|
Grounded in existing practice rather than invented here: the
|
|
fault/error/failure chain from **Avizienis, Laprie, Randell & Landwehr**
|
|
(*Basic Concepts and Taxonomy of Dependable and Secure Computing*, IEEE
|
|
TDSC 1:1, 2004); the model strata and the verification/validation split
|
|
from **Sargent** (*Verification and Validation of Simulation Models*); and
|
|
**MDA** (Hunicke, LeBlanc & Zubek, 2004) for the game-facing layers.
|
|
|
|
---
|
|
|
|
## 1. The strata
|
|
|
|
| stratum | canon | what it is | who owns it |
|
|
|---|---|---|---|
|
|
| **GAME** | Sargent's *problem entity*; MDA *mechanics as designed* | the rules as the game's owner defines them — the edition, the printed text | **the game's owner**, not us |
|
|
| **MODEL** | Sargent's *conceptual model* | our formalisation: `GroundRules.md`, the rule ids, the state we chose to represent | us, answerable to the owner |
|
|
| **ENGINE** | Sargent's *computerized model* | the kernel that implements the model | us |
|
|
| **INSTRUMENT** | measurement apparatus | harnesses, policies, metrics, gates, the trial protocol | us |
|
|
| **ACCOUNT** | — | evidence files, register entries, messages, this document | us |
|
|
| **PRESENTATION** | MDA *aesthetics*-facing | the page, and everything [`Ornamentation.md`](Ornamentation.md) covers | us, later `clay-animate` |
|
|
|
|
### 1.1 The two relations that matter
|
|
|
|
> **GAME ↔ MODEL is validation.** *Did we formalise the right game?*
|
|
>
|
|
> **MODEL ↔ ENGINE is verification.** *Did we implement the formalisation
|
|
> correctly?*
|
|
|
|
This is Sargent's split and it is the one we kept losing. A statement that
|
|
the engine mis-scores a Bond is **verification**. A statement that the
|
|
rules do not say what happens when two Bonds break simultaneously is
|
|
**validation** — and it is not ours to answer.
|
|
|
|
**Sargent's third leg is `data validity`**, and it is where
|
|
`editions/` lives: the vendored CSVs, their digests, and the tripwires in
|
|
`edition.rs` are data validation, not verification.
|
|
|
|
### 1.2 What each stratum's defects look like
|
|
|
|
| stratum | a defect here looks like |
|
|
|---|---|
|
|
| GAME | a rule that does not decide; a dominant strategy; an action that earns its place nowhere |
|
|
| MODEL | we represented a rule the edition does not have, or omitted one it does |
|
|
| ENGINE | the code does not do what `GroundRules.md` says |
|
|
| INSTRUMENT | the measurement answers a different question than the one asked |
|
|
| ACCOUNT | the sentence claims more than the artifact shows |
|
|
| PRESENTATION | the player cannot see, or misreads, what the state says |
|
|
|
|
---
|
|
|
|
## 2. Fault, error, failure
|
|
|
|
Avizienis' chain, unchanged, applied **within** a stratum.
|
|
|
|
| term | definition | example, INSTRUMENT stratum |
|
|
|---|---|---|
|
|
| **fault** | the defect in the artifact — the wrong line, the wrong column, the wrong constant | `peak` maximised over `StressSet` payloads |
|
|
| **error** | the wrong internal state it produces | the metric holds 1 where the game held 2 |
|
|
| **failure** | the deviation visible at the interface | the evidence file says *"peak Stress was 1"* |
|
|
|
|
**A fault is dormant until activated.** `attack-value.rs`'s missing
|
|
assertion was a fault for weeks and produced no error, because no cell was
|
|
ever short.
|
|
|
|
### 2.1 The rule that explains our history
|
|
|
|
> **A failure in one stratum is a fault in the next.**
|
|
|
|
An INSTRUMENT failure (a metric reporting the wrong number) is an ACCOUNT
|
|
fault (a sentence built on it). An ACCOUNT failure that reaches the owner
|
|
is a fault in *their* decision-making. That is why
|
|
[`GameDesign.md`](GameDesign.md) §1 gates what may leave the repo: it is a
|
|
firebreak in this chain, not politeness.
|
|
|
|
### 2.2 The wrong-subject error, defined at last
|
|
|
|
[ADR-0018](../decisions/ADR-0018-a-number-that-does-not-move.md) named a
|
|
family and could not say precisely what it was. In this vocabulary:
|
|
|
|
> **A wrong-subject error is an ACCOUNT failure with no INSTRUMENT fault.**
|
|
> The measurement is correct about *its* subject; the sentence is about a
|
|
> different one.
|
|
|
|
`peak Stress was 1` — the metric correctly reported the maximum *assigned*
|
|
Stress. `Reactive is greedy with one preference changed` — the policy
|
|
correctly implemented what was written. **Nothing was miscomputed in
|
|
either case**, which is exactly why tests did not catch them and why the
|
|
fix is never "add a test to the engine".
|
|
|
|
---
|
|
|
|
## 3. The vocabulary, fixed
|
|
|
|
| word | means | does not mean |
|
|
|---|---|---|
|
|
| **finding** | a statement about the **GAME** stratum, admissible under GameDesign §1, addressed to the owner | any defect we happen to notice |
|
|
| **defect** | a fault in any stratum we own — MODEL, ENGINE, INSTRUMENT, ACCOUNT, PRESENTATION | a design question |
|
|
| **fix** | a change to the **artifact** that removes a fault | a change to what we said |
|
|
| **correction** | a change to the **ACCOUNT** — a claim withdrawn, restated or re-derived | a code change |
|
|
| **retraction** | a correction of something that has already **left the repo** | an internal edit |
|
|
| **regression** | a fault introduced by a change, in any stratum | any failure |
|
|
| **inert control** | a control that cannot fail, so proves nothing (ADR-0006 D3) | a control that happens to be green |
|
|
|
|
**The register carries findings.** Where an entry is really a defect in a
|
|
stratum we own — F18, F24, F25 are ENGINE/data — the row says so, because
|
|
"the engine has not read this file" and "the game does not decide this"
|
|
are different claims to a reader deciding what to act on.
|
|
|
|
---
|
|
|
|
## 4. What the layers measure, and what they cannot
|
|
|
|
MDA gives the game-facing split, and it maps onto instruments we already
|
|
have:
|
|
|
|
| MDA layer | where it lives here | what measures it |
|
|
|---|---|---|
|
|
| **Mechanics** | GAME → MODEL → ENGINE | the rule-coverage gate, the edition tripwires |
|
|
| **Dynamics** — the behaviour that emerges in play | what the ENGINE produces over many games | the policy panels: win rates, DARVO arm rates, action selection |
|
|
| **Aesthetics** — the player's response | not representable in state | **trial notes, and nothing else** |
|
|
|
|
> **Our panels measure dynamics. Our trial logs sample aesthetics. Neither
|
|
> substitutes for the other**, and the recurring temptation is to report a
|
|
> dynamics number as though it settled an aesthetics question — *"is the
|
|
> game fun at 5 seats"* is not answered by a win rate.
|
|
|
|
MDA's own directional insight is the reason: **the designer builds
|
|
mechanics upward hoping for a feeling; the player meets the feeling first
|
|
and infers the rules.** A simulator sits at the bottom of that arrow and
|
|
can only ever report upward.
|
|
|
|
---
|
|
|
|
## 4.1 The MODEL stratum has a formal name now
|
|
|
|
The **extensive-form game** is what our MODEL stratum corresponds to, and
|
|
the correspondence is established rather than assumed
|
|
([CB-WP-0041](../workplans/CB-WP-0041-the-extensive-form-foundation.md)):
|
|
|
|
| EFG component | ours | checked by |
|
|
|---|---|---|
|
|
| histories | the journal | — |
|
|
| actions | `bot::legal_commands` | — |
|
|
| **information sets** | the seat's **observation history**, *not* `project()` | `perfect-recall` |
|
|
| chance | **a single root node**; the reshuffle is correlated with it | `a_game_is_determined_by_its_seed` |
|
|
| simultaneity | commit/reveal, the textbook encoding | `a_pending_selection_is_hidden_until_reveal` |
|
|
| payoffs | `Outcome` / `score()` | — |
|
|
|
|
**`project()` is an observation, not an information state**, and the
|
|
distinction cost 22 measured violations to find. It is the same split
|
|
OpenSpiel draws between `ObservationString` and `InformationStateString`.
|
|
|
|
**No port was built** — [ADR-0020](../decisions/ADR-0020-we-do-not-build-the-port.md)
|
|
refuses it, because making chance explicit would break the replay purity
|
|
the evidence discipline rests on.
|
|
|
|
## 5. Worked examples from this repo
|
|
|
|
Because a taxonomy with no instances was invented rather than observed
|
|
(GameDesign §2).
|
|
|
|
| what happened | stratum | fault / error / failure |
|
|
|---|---|---|
|
|
| `solution_deck()` is a Rust literal that never reads `Solutions.csv` | ENGINE (data validity) | dormant **fault**; no error, because the literal agrees |
|
|
| `score()` read `self` while H1's pressure went into `work` | ENGINE | **fault**; error only at round 5; no measured failure |
|
|
| GR-P05 — SOLVE offered where it could not act | MODEL, then GAME | our **failure**, escalated to a GAME finding, ruled by the owner |
|
|
| ATTACK earns its place in no mode (F17) | **GAME** | a finding — not a defect at all |
|
|
| `peak Stress was 1` | ACCOUNT | **failure** with no instrument fault (§2.2) |
|
|
| the panels ran under no gate | INSTRUMENT | **fault** — every published figure came from an ungated binary |
|
|
| the note channel closed when the game ended | PRESENTATION | **failure**; the state was right, the player could not reach it |
|
|
|
|
**Read the stratum column.** Six rows, six different owners and six
|
|
different fixes, and every one of them was called "an error" at the time.
|
|
|
|
---
|
|
|
|
## 6. Falsifier
|
|
|
|
If a defect arises that cannot be placed in exactly one stratum, the
|
|
strata are wrong — either too few, or cut in the wrong place. **Record the
|
|
instance rather than widening a definition to swallow it**, which is how a
|
|
taxonomy stops distinguishing anything.
|
|
|
|
A second falsifier, from §2.2: if a wrong-subject error is ever found that
|
|
*did* have an instrument fault, then the family is not what ADR-0018 and
|
|
this document say it is.
|
|
|
|
## Sources
|
|
|
|
- Avizienis, Laprie, Randell & Landwehr, *Basic Concepts and Taxonomy of
|
|
Dependable and Secure Computing*, IEEE TDSC 1(1), 2004.
|
|
- Sargent, *Verification and Validation of Simulation Models*, Winter
|
|
Simulation Conference.
|
|
- Hunicke, LeBlanc & Zubek, *MDA: A Formal Approach to Game Design and
|
|
Game Research*, AAAI Workshop, 2004.
|