T08: close out - workplan notes and ground-truth record
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 1629012@bnt-lap001 Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
This commit is contained in:
parent
416a4b360a
commit
0c52461801
3 changed files with 61 additions and 4 deletions
|
|
@ -20,6 +20,6 @@
|
|||
| task | TD-WP-0002-T05 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||
| task | TD-WP-0002-T06 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||
| task | TD-WP-0002-T07 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||
| task | TD-WP-0002-T08 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||
| task | TD-WP-0002-T08 | progress | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||
| task | TD-WP-0002-T09 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||
| task | TD-WP-0002-T10 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||
|
|
|
|||
|
|
@ -1,7 +1,8 @@
|
|||
# Lab Ground Truth
|
||||
|
||||
**Lab version base:** `lab-0.2.0` · **Catalogue:** 20 mutations · **Scenario:**
|
||||
`scenarios/alice_bob_carol.py`
|
||||
**Lab version base:** `lab-0.2.0` · **Catalogue:** 23 mutations · **Scenarios:**
|
||||
`scenarios/alice_bob_carol.py` (API-only) and `scenarios/full_journey.py`
|
||||
(cross-surface)
|
||||
|
||||
Labels are decided by a human from the use case and recorded **before** any run.
|
||||
They are never inferred from behaviour — that is the whole point, and M12/M19
|
||||
|
|
@ -29,10 +30,23 @@ below show why.
|
|||
| M18 | Revocation is not audited | DEFECT | preserved | FAIL |
|
||||
| M19 | Revocation propagates after a delay | DEFECT | preserved | FAIL |
|
||||
| M20 | Tenant isolation leaks | DEFECT | preserved | FAIL |
|
||||
| M21 | Controls reworded and test ids dropped | MECHANICAL | **dropped** | PASS *(UI: out of scope)* |
|
||||
| M22 | Form field names changed | MECHANICAL | **dropped** | PASS *(UI: out of scope)* |
|
||||
| M23 | Sharing control removed from UI, API left open | DEFECT | preserved | PASS *(UI: out of scope)* |
|
||||
|
||||
Baseline: PASS. Detection: **MECHANICAL 0/10 flagged** (correct — semantics
|
||||
preserved), **DEFECT 6/6 flagged**, **SEMANTIC 2/4 flagged**.
|
||||
|
||||
## Coverage is scoped twice over
|
||||
|
||||
A scenario protects what it asserts (F-0002) **and** only what it can reach. The
|
||||
reference scenario is API-only, so the three UI mutations above are outside its
|
||||
scope entirely — M23 is a genuine defect it cannot see. They are covered by the
|
||||
cross-surface journey instead, where M23 is the E-003 surface-substitution
|
||||
attack. Both facts are declared in `OUT_OF_SCOPE` and asserted: an undeclared
|
||||
invisible mutation fails the suite, and so does a declaration that has gone
|
||||
stale.
|
||||
|
||||
## The two inert mutations
|
||||
|
||||
Recorded rather than hidden. `test_inert_semantic_mutations_are_declared` fails
|
||||
|
|
@ -55,6 +69,16 @@ This is why claims require independent provenance (D-06), why `AMBIGUOUS`
|
|||
escalates to a human rather than resolving itself, and why T08's classifier is
|
||||
not permitted to guess.
|
||||
|
||||
## Classifier response (cross-surface journey)
|
||||
|
||||
Measured in `tests/test_classification.py`. **False Adaptation Rate = 0/7.**
|
||||
|
||||
| Ground truth | Accepted without a human |
|
||||
|---|---|
|
||||
| MECHANICAL (12) | 11 — M22 escalates as `AMBIGUOUS` |
|
||||
| SEMANTIC (4) | 2 — M13 and M14, both genuinely inert |
|
||||
| DEFECT (7) | **0** |
|
||||
|
||||
## Regenerating
|
||||
|
||||
The matrix is asserted in `tests/test_lab_ground_truth.py`. A moved cell fails
|
||||
|
|
|
|||
|
|
@ -358,7 +358,7 @@ The headline result is a narrowing, not a confirmation:
|
|||
|
||||
```task
|
||||
id: TD-WP-0002-T08
|
||||
status: progress
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "e2403d3c-bc35-5dd3-b1c8-2474663e06d0"
|
||||
```
|
||||
|
|
@ -378,6 +378,39 @@ one publicly normalised authorization defect kills the thesis permanently. Targe
|
|||
is zero across the set, and a non-zero result is a stop-and-redesign signal, not
|
||||
a tuning exercise.
|
||||
|
||||
**Done 2026-08-22.** `classification.py`, `CompositeDriver`,
|
||||
`scenarios/full_journey.py` (one asset crossing both surfaces). 141 tests pass.
|
||||
|
||||
**False Adaptation Rate = 0/7**, including the three E-003 attacks. The other
|
||||
side of the trade is asserted too — 11 of 12 mechanical mutations are absorbed
|
||||
without a human, so the safety result is not bought by escalating everything.
|
||||
|
||||
| Ground truth | Accepted without a human |
|
||||
|---|---|
|
||||
| MECHANICAL (12) | 11 |
|
||||
| SEMANTIC (4) | 2 — both genuinely inert here |
|
||||
| DEFECT (7) | **0** |
|
||||
|
||||
- **F-0006 (resolved)** — the T02 decision table listed `SEMANTIC_CHANGE` as an
|
||||
outcome it could produce. It cannot: M12 and M19 are behaviourally identical,
|
||||
as the lab has asserted since T05. `PRODUCT_DEFECT` and `SEMANTIC_CHANGE`
|
||||
collapse into one escalating outcome, `BEHAVIOUR_CHANGED`, and the distinction
|
||||
becomes a recorded human adjudication. `INTENT_CHANGED` survives but is
|
||||
detected by the *claim fingerprint* moving — a fact about the recorded use
|
||||
case, not an inference about behaviour. Erring toward "false alarm" costs one
|
||||
review; erring toward "false adaptation" ends the project.
|
||||
- **Two classifier defects found and fixed rather than reported.** Claims
|
||||
downstream of a failed realization now yield `INCONCLUSIVE` rather than `FAIL`
|
||||
— accusing the system of a defect on the strength of the test's own inability
|
||||
to act is the mirror image of a false adaptation. And the browser driver now
|
||||
records a page signature, so a surface change is detectable even when the
|
||||
chosen interaction path is identical.
|
||||
- **What this does not establish.** FAR = 0 follows largely from architecture
|
||||
(D-02, plus `SAFE_TO_ACCEPT` being a closed two-element set). The experiment
|
||||
confirms the architecture behaves as designed over 23 hand-written mutations;
|
||||
it says nothing about mutations nobody thought of. That distinction should
|
||||
survive into any external claim.
|
||||
|
||||
## Crystallize one asset into deterministic regression
|
||||
|
||||
```task
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue