coulomb/test-driver
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

T08: close out - workplan notes and ground-truth record

Browse source 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 1629012@bnt-lap001
Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
This commit is contained in:
tegwick 2026-08-23 00:08:38 +02:00
parent 416a4b360a
commit 0c52461801
3 changed files with 61 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

28
lab/GROUND-TRUTH.md
View file

@ -1,7 +1,8 @@
# Lab Ground Truth
**Lab version base:** `lab-0.2.0` · **Catalogue:** 20 mutations · **Scenario:**
`scenarios/alice_bob_carol.py`
**Lab version base:** `lab-0.2.0` · **Catalogue:** 23 mutations · **Scenarios:**
`scenarios/alice_bob_carol.py` (API-only) and `scenarios/full_journey.py`
(cross-surface)
Labels are decided by a human from the use case and recorded **before** any run.
They are never inferred from behaviour — that is the whole point, and M12/M19
@ -29,10 +30,23 @@ below show why.
| M18 | Revocation is not audited | DEFECT | preserved | FAIL |
| M19 | Revocation propagates after a delay | DEFECT | preserved | FAIL |
| M20 | Tenant isolation leaks | DEFECT | preserved | FAIL |
| M21 | Controls reworded and test ids dropped | MECHANICAL | **dropped** | PASS *(UI: out of scope)* |
| M22 | Form field names changed | MECHANICAL | **dropped** | PASS *(UI: out of scope)* |
| M23 | Sharing control removed from UI, API left open | DEFECT | preserved | PASS *(UI: out of scope)* |
Baseline: PASS. Detection: **MECHANICAL 0/10 flagged** (correct — semantics
preserved), **DEFECT 6/6 flagged**, **SEMANTIC 2/4 flagged**.
## Coverage is scoped twice over
A scenario protects what it asserts (F-0002) **and** only what it can reach. The
reference scenario is API-only, so the three UI mutations above are outside its
scope entirely — M23 is a genuine defect it cannot see. They are covered by the
cross-surface journey instead, where M23 is the E-003 surface-substitution
attack. Both facts are declared in `OUT_OF_SCOPE` and asserted: an undeclared
invisible mutation fails the suite, and so does a declaration that has gone
stale.
## The two inert mutations
Recorded rather than hidden. `test_inert_semantic_mutations_are_declared` fails
@ -55,6 +69,16 @@ This is why claims require independent provenance (D-06), why `AMBIGUOUS`
escalates to a human rather than resolving itself, and why T08's classifier is
not permitted to guess.
## Classifier response (cross-surface journey)
Measured in `tests/test_classification.py`. **False Adaptation Rate = 0/7.**
| Ground truth | Accepted without a human |
|---|---|
| MECHANICAL (12) | 11 — M22 escalates as `AMBIGUOUS` |
| SEMANTIC (4) | 2 — M13 and M14, both genuinely inert |
| DEFECT (7) | **0** |
## Regenerating
The matrix is asserted in `tests/test_lab_ground_truth.py`. A moved cell fails