T08: close out - workplan notes and ground-truth record

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 1629012@bnt-lap001
Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
This commit is contained in:
tegwick 2026-08-23 00:08:38 +02:00
parent 416a4b360a
commit 0c52461801
3 changed files with 61 additions and 4 deletions

View file

@ -20,6 +20,6 @@
| task | TD-WP-0002-T05 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
| task | TD-WP-0002-T06 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
| task | TD-WP-0002-T07 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
| task | TD-WP-0002-T08 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
| task | TD-WP-0002-T08 | progress | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
| task | TD-WP-0002-T09 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
| task | TD-WP-0002-T10 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |

View file

@ -1,7 +1,8 @@
# Lab Ground Truth
**Lab version base:** `lab-0.2.0` · **Catalogue:** 20 mutations · **Scenario:**
`scenarios/alice_bob_carol.py`
**Lab version base:** `lab-0.2.0` · **Catalogue:** 23 mutations · **Scenarios:**
`scenarios/alice_bob_carol.py` (API-only) and `scenarios/full_journey.py`
(cross-surface)
Labels are decided by a human from the use case and recorded **before** any run.
They are never inferred from behaviour — that is the whole point, and M12/M19
@ -29,10 +30,23 @@ below show why.
| M18 | Revocation is not audited | DEFECT | preserved | FAIL |
| M19 | Revocation propagates after a delay | DEFECT | preserved | FAIL |
| M20 | Tenant isolation leaks | DEFECT | preserved | FAIL |
| M21 | Controls reworded and test ids dropped | MECHANICAL | **dropped** | PASS *(UI: out of scope)* |
| M22 | Form field names changed | MECHANICAL | **dropped** | PASS *(UI: out of scope)* |
| M23 | Sharing control removed from UI, API left open | DEFECT | preserved | PASS *(UI: out of scope)* |
Baseline: PASS. Detection: **MECHANICAL 0/10 flagged** (correct — semantics
preserved), **DEFECT 6/6 flagged**, **SEMANTIC 2/4 flagged**.
## Coverage is scoped twice over
A scenario protects what it asserts (F-0002) **and** only what it can reach. The
reference scenario is API-only, so the three UI mutations above are outside its
scope entirely — M23 is a genuine defect it cannot see. They are covered by the
cross-surface journey instead, where M23 is the E-003 surface-substitution
attack. Both facts are declared in `OUT_OF_SCOPE` and asserted: an undeclared
invisible mutation fails the suite, and so does a declaration that has gone
stale.
## The two inert mutations
Recorded rather than hidden. `test_inert_semantic_mutations_are_declared` fails
@ -55,6 +69,16 @@ This is why claims require independent provenance (D-06), why `AMBIGUOUS`
escalates to a human rather than resolving itself, and why T08's classifier is
not permitted to guess.
## Classifier response (cross-surface journey)
Measured in `tests/test_classification.py`. **False Adaptation Rate = 0/7.**
| Ground truth | Accepted without a human |
|---|---|
| MECHANICAL (12) | 11 — M22 escalates as `AMBIGUOUS` |
| SEMANTIC (4) | 2 — M13 and M14, both genuinely inert |
| DEFECT (7) | **0** |
## Regenerating
The matrix is asserted in `tests/test_lab_ground_truth.py`. A moved cell fails

View file

@ -358,7 +358,7 @@ The headline result is a narrowing, not a confirmation:
```task
id: TD-WP-0002-T08
status: progress
status: done
priority: high
state_hub_task_id: "e2403d3c-bc35-5dd3-b1c8-2474663e06d0"
```
@ -378,6 +378,39 @@ one publicly normalised authorization defect kills the thesis permanently. Targe
is zero across the set, and a non-zero result is a stop-and-redesign signal, not
a tuning exercise.
**Done 2026-08-22.** `classification.py`, `CompositeDriver`,
`scenarios/full_journey.py` (one asset crossing both surfaces). 141 tests pass.
**False Adaptation Rate = 0/7**, including the three E-003 attacks. The other
side of the trade is asserted too — 11 of 12 mechanical mutations are absorbed
without a human, so the safety result is not bought by escalating everything.
| Ground truth | Accepted without a human |
|---|---|
| MECHANICAL (12) | 11 |
| SEMANTIC (4) | 2 — both genuinely inert here |
| DEFECT (7) | **0** |
- **F-0006 (resolved)** — the T02 decision table listed `SEMANTIC_CHANGE` as an
outcome it could produce. It cannot: M12 and M19 are behaviourally identical,
as the lab has asserted since T05. `PRODUCT_DEFECT` and `SEMANTIC_CHANGE`
collapse into one escalating outcome, `BEHAVIOUR_CHANGED`, and the distinction
becomes a recorded human adjudication. `INTENT_CHANGED` survives but is
detected by the *claim fingerprint* moving — a fact about the recorded use
case, not an inference about behaviour. Erring toward "false alarm" costs one
review; erring toward "false adaptation" ends the project.
- **Two classifier defects found and fixed rather than reported.** Claims
downstream of a failed realization now yield `INCONCLUSIVE` rather than `FAIL`
— accusing the system of a defect on the strength of the test's own inability
to act is the mirror image of a false adaptation. And the browser driver now
records a page signature, so a surface change is detectable even when the
chosen interaction path is identical.
- **What this does not establish.** FAR = 0 follows largely from architecture
(D-02, plus `SAFE_TO_ACCEPT` being a closed two-element set). The experiment
confirms the architecture behaves as designed over 23 hand-written mutations;
it says nothing about mutations nobody thought of. That distinction should
survive into any external claim.
## Crystallize one asset into deterministic regression
```task