test-driver/lab/GROUND-TRUTH.md

Lab Ground Truth

Lab version base: lab-0.2.0 · Catalogue: 23 mutations · Scenarios: scenarios/alice_bob_carol.py (API-only) and scenarios/full_journey.py (cross-surface)

Labels are decided by a human from the use case and recorded before any run. They are never inferred from behaviour — that is the whole point, and M12/M19 below show why.

ID	Mutation	Label	Test ids	Reference scenario
M01	Sharing control moves into a modal	MECHANICAL	preserved	PASS
M02	DOM rewritten, test ids not carried forward	MECHANICAL	dropped	PASS
M03	API renames resource_id to id	MECHANICAL	preserved	PASS
M04	UI labels reworded	MECHANICAL	preserved	PASS
M05	Form field order reversed	MECHANICAL	preserved	PASS
M06	API paths shortened	MECHANICAL	preserved	PASS
M07	Buttons become anchors	MECHANICAL	preserved	PASS
M08	Revoke gains a confirmation step	MECHANICAL	preserved	PASS
M09	Responses are slower	MECHANICAL	preserved	PASS
M10	Denials return 401 instead of 403	MECHANICAL	preserved	PASS
M11	A share must be accepted first	SEMANTIC	preserved	FAIL
M12	Revocation is deferred by decision	SEMANTIC	preserved	FAIL
M13	Grants default to WRITE	SEMANTIC	preserved	PASS (inert)
M14	Cross-tenant sharing declared prohibited	SEMANTIC	preserved	PASS (inert)
M15	Revocation updates record but not enforcement	DEFECT	preserved	FAIL
M16	A READ grant confers WRITE	DEFECT	preserved	FAIL
M17	Any authenticated user can read anything	DEFECT	preserved	FAIL
M18	Revocation is not audited	DEFECT	preserved	FAIL
M19	Revocation propagates after a delay	DEFECT	preserved	FAIL
M20	Tenant isolation leaks	DEFECT	preserved	FAIL
M21	Controls reworded and test ids dropped	MECHANICAL	dropped	PASS (UI: out of scope)
M22	Form field names changed	MECHANICAL	dropped	PASS (UI: out of scope)
M23	Sharing control removed from UI, API left open	DEFECT	preserved	PASS (UI: out of scope)

Baseline: PASS. Detection: MECHANICAL 0/10 flagged (correct — semantics preserved), DEFECT 6/6 flagged, SEMANTIC 2/4 flagged.

Coverage is scoped twice over

A scenario protects what it asserts (F-0002) and only what it can reach. The reference scenario is API-only, so the three UI mutations above are outside its scope entirely — M23 is a genuine defect it cannot see. They are covered by the cross-surface journey instead, where M23 is the E-003 surface-substitution attack. Both facts are declared in OUT_OF_SCOPE and asserted: an undeclared invisible mutation fails the suite, and so does a declaration that has gone stale.

The two inert mutations

Recorded rather than hidden. test_inert_semantic_mutations_are_declared fails if an invisible mutation is ever left undeclared.

• M13 only affects grants that omit a permission; the reference scenario passes READ explicitly, so nothing changes.
• M14 is a change of intent with no change of code — cross-tenant sharing was already enforced, and the mutation declares it deliberate. Nothing observable moves. This is the sharpest available demonstration that classification cannot be a diff.

M12 vs M19 — the discrimination problem in one row

Both produce an identical failure: c-bob-revoked, same step, same evidence. One is a deliberate product decision that revocation batches; the other is a propagation race. No observation distinguishes them. Only intent does.

This is why claims require independent provenance (D-06), why AMBIGUOUS escalates to a human rather than resolving itself, and why T08's classifier is not permitted to guess.

Classifier response (cross-surface journey)

Measured in tests/test_classification.py. False Adaptation Rate = 0/7.

Ground truth	Accepted without a human
MECHANICAL (12)	11 — M22 escalates as AMBIGUOUS
SEMANTIC (4)	2 — M13 and M14, both genuinely inert
DEFECT (7)	0

Regenerating

The matrix is asserted in tests/test_lab_ground_truth.py. A moved cell fails the suite: changing the measuring instrument must be a deliberate, reviewed act.