diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 9985b2b..67df1d0 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -20,6 +20,6 @@ | task | TD-WP-0002-T05 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T06 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T07 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | -| task | TD-WP-0002-T08 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | +| task | TD-WP-0002-T08 | progress | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T09 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T10 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | diff --git a/lab/GROUND-TRUTH.md b/lab/GROUND-TRUTH.md index e2f4067..1d08882 100644 --- a/lab/GROUND-TRUTH.md +++ b/lab/GROUND-TRUTH.md @@ -1,7 +1,8 @@ # Lab Ground Truth -**Lab version base:** `lab-0.2.0` · **Catalogue:** 20 mutations · **Scenario:** -`scenarios/alice_bob_carol.py` +**Lab version base:** `lab-0.2.0` · **Catalogue:** 23 mutations · **Scenarios:** +`scenarios/alice_bob_carol.py` (API-only) and `scenarios/full_journey.py` +(cross-surface) Labels are decided by a human from the use case and recorded **before** any run. They are never inferred from behaviour — that is the whole point, and M12/M19 @@ -29,10 +30,23 @@ below show why. | M18 | Revocation is not audited | DEFECT | preserved | FAIL | | M19 | Revocation propagates after a delay | DEFECT | preserved | FAIL | | M20 | Tenant isolation leaks | DEFECT | preserved | FAIL | +| M21 | Controls reworded and test ids dropped | MECHANICAL | **dropped** | PASS *(UI: out of scope)* | +| M22 | Form field names changed | MECHANICAL | **dropped** | PASS *(UI: out of scope)* | +| M23 | Sharing control removed from UI, API left open | DEFECT | preserved | PASS *(UI: out of scope)* | Baseline: PASS. Detection: **MECHANICAL 0/10 flagged** (correct — semantics preserved), **DEFECT 6/6 flagged**, **SEMANTIC 2/4 flagged**. +## Coverage is scoped twice over + +A scenario protects what it asserts (F-0002) **and** only what it can reach. The +reference scenario is API-only, so the three UI mutations above are outside its +scope entirely — M23 is a genuine defect it cannot see. They are covered by the +cross-surface journey instead, where M23 is the E-003 surface-substitution +attack. Both facts are declared in `OUT_OF_SCOPE` and asserted: an undeclared +invisible mutation fails the suite, and so does a declaration that has gone +stale. + ## The two inert mutations Recorded rather than hidden. `test_inert_semantic_mutations_are_declared` fails @@ -55,6 +69,16 @@ This is why claims require independent provenance (D-06), why `AMBIGUOUS` escalates to a human rather than resolving itself, and why T08's classifier is not permitted to guess. +## Classifier response (cross-surface journey) + +Measured in `tests/test_classification.py`. **False Adaptation Rate = 0/7.** + +| Ground truth | Accepted without a human | +|---|---| +| MECHANICAL (12) | 11 — M22 escalates as `AMBIGUOUS` | +| SEMANTIC (4) | 2 — M13 and M14, both genuinely inert | +| DEFECT (7) | **0** | + ## Regenerating The matrix is asserted in `tests/test_lab_ground_truth.py`. A moved cell fails diff --git a/workplans/TD-WP-0002-vertical-spike-crystallization.md b/workplans/TD-WP-0002-vertical-spike-crystallization.md index fce0d16..795d038 100644 --- a/workplans/TD-WP-0002-vertical-spike-crystallization.md +++ b/workplans/TD-WP-0002-vertical-spike-crystallization.md @@ -358,7 +358,7 @@ The headline result is a narrowing, not a confirmation: ```task id: TD-WP-0002-T08 -status: progress +status: done priority: high state_hub_task_id: "e2403d3c-bc35-5dd3-b1c8-2474663e06d0" ``` @@ -378,6 +378,39 @@ one publicly normalised authorization defect kills the thesis permanently. Targe is zero across the set, and a non-zero result is a stop-and-redesign signal, not a tuning exercise. +**Done 2026-08-22.** `classification.py`, `CompositeDriver`, +`scenarios/full_journey.py` (one asset crossing both surfaces). 141 tests pass. + +**False Adaptation Rate = 0/7**, including the three E-003 attacks. The other +side of the trade is asserted too — 11 of 12 mechanical mutations are absorbed +without a human, so the safety result is not bought by escalating everything. + +| Ground truth | Accepted without a human | +|---|---| +| MECHANICAL (12) | 11 | +| SEMANTIC (4) | 2 — both genuinely inert here | +| DEFECT (7) | **0** | + +- **F-0006 (resolved)** — the T02 decision table listed `SEMANTIC_CHANGE` as an + outcome it could produce. It cannot: M12 and M19 are behaviourally identical, + as the lab has asserted since T05. `PRODUCT_DEFECT` and `SEMANTIC_CHANGE` + collapse into one escalating outcome, `BEHAVIOUR_CHANGED`, and the distinction + becomes a recorded human adjudication. `INTENT_CHANGED` survives but is + detected by the *claim fingerprint* moving — a fact about the recorded use + case, not an inference about behaviour. Erring toward "false alarm" costs one + review; erring toward "false adaptation" ends the project. +- **Two classifier defects found and fixed rather than reported.** Claims + downstream of a failed realization now yield `INCONCLUSIVE` rather than `FAIL` + — accusing the system of a defect on the strength of the test's own inability + to act is the mirror image of a false adaptation. And the browser driver now + records a page signature, so a surface change is detectable even when the + chosen interaction path is identical. +- **What this does not establish.** FAR = 0 follows largely from architecture + (D-02, plus `SAFE_TO_ACCEPT` being a closed two-element set). The experiment + confirms the architecture behaves as designed over 23 hand-written mutations; + it says nothing about mutations nobody thought of. That distinction should + survive into any external claim. + ## Crystallize one asset into deterministic regression ```task