tegwick
cf682281d7
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a0e76f-be98-7ae3-965d-e0b31290a4c4
95 lines
4.8 KiB
Markdown
95 lines
4.8 KiB
Markdown
# Lab Ground Truth
|
||
|
||
**Lab version base:** `lab-0.2.0` · **Catalogue:** 23 mutations · **Scenarios:**
|
||
`scenarios/alice_bob_carol.py` (API-only) and `scenarios/full_journey.py`
|
||
(cross-surface)
|
||
|
||
Labels are decided by a human from the use case and recorded **before** any run.
|
||
They are never inferred from behaviour — that is the whole point, and M12/M19
|
||
below show why.
|
||
|
||
| ID | Mutation | Label | Test ids | Reference scenario |
|
||
|---|---|---|---|---|
|
||
| M01 | Sharing control moves into a modal | MECHANICAL | preserved | PASS |
|
||
| M02 | DOM rewritten, test ids not carried forward | MECHANICAL | **dropped** | PASS |
|
||
| M03 | API renames `resource_id` to `id` | MECHANICAL | preserved | PASS |
|
||
| M04 | UI labels reworded | MECHANICAL | preserved | PASS |
|
||
| M05 | Form field order reversed | MECHANICAL | preserved | PASS |
|
||
| M06 | API paths shortened | MECHANICAL | preserved | PASS |
|
||
| M07 | Buttons become anchors | MECHANICAL | preserved | PASS |
|
||
| M08 | Revoke gains a confirmation step | MECHANICAL | preserved | PASS |
|
||
| M09 | Responses are slower | MECHANICAL | preserved | PASS |
|
||
| M10 | Denials return 401 instead of 403 | MECHANICAL | preserved | PASS |
|
||
| M11 | A share must be accepted first | SEMANTIC | preserved | FAIL |
|
||
| M12 | Revocation is deferred by decision | SEMANTIC | preserved | FAIL |
|
||
| M13 | Grants default to WRITE | SEMANTIC | preserved | PASS *(inert)* |
|
||
| M14 | Cross-tenant sharing declared prohibited | SEMANTIC | preserved | PASS *(inert)* |
|
||
| M15 | Revocation updates record but not enforcement | DEFECT | preserved | FAIL |
|
||
| M16 | A READ grant confers WRITE | DEFECT | preserved | FAIL |
|
||
| M17 | Any authenticated user can read anything | DEFECT | preserved | FAIL |
|
||
| M18 | Revocation is not audited | DEFECT | preserved | FAIL |
|
||
| M19 | Revocation propagates after a delay | DEFECT | preserved | FAIL |
|
||
| M20 | Tenant isolation leaks | DEFECT | preserved | FAIL |
|
||
| M21 | Controls reworded and test ids dropped | MECHANICAL | **dropped** | PASS *(UI: out of scope)* |
|
||
| M22 | Form field names changed | MECHANICAL | **dropped** | PASS *(UI: out of scope)* |
|
||
| M23 | Sharing control removed from UI, API left open | DEFECT | preserved | PASS *(UI: out of scope)* |
|
||
|
||
Baseline: PASS. Detection: **MECHANICAL 0/10 flagged** (correct — semantics
|
||
preserved), **DEFECT 6/6 flagged**, **SEMANTIC 2/4 flagged**.
|
||
|
||
## Coverage is scoped twice over
|
||
|
||
A scenario protects what it asserts (F-0002) **and** only what it can reach. The
|
||
reference scenario is API-only, so the three UI mutations above are outside its
|
||
scope entirely — M23 is a genuine defect it cannot see. They are covered by the
|
||
cross-surface journey instead, where M23 is the E-003 surface-substitution
|
||
attack. Both facts are declared in `OUT_OF_SCOPE` and asserted: an undeclared
|
||
invisible mutation fails the suite, and so does a declaration that has gone
|
||
stale.
|
||
|
||
## The two inert mutations
|
||
|
||
Recorded rather than hidden. `test_inert_semantic_mutations_are_declared` fails
|
||
if an invisible mutation is ever left undeclared.
|
||
|
||
- **M13** only affects grants that omit a permission; the reference scenario
|
||
passes `READ` explicitly, so nothing changes.
|
||
- **M14** is a change of *intent* with no change of code — cross-tenant sharing
|
||
was already enforced, and the mutation declares it deliberate. Nothing
|
||
observable moves. This is the sharpest available demonstration that
|
||
classification cannot be a diff.
|
||
|
||
## M12 vs M19 — the discrimination problem in one row
|
||
|
||
Both produce an identical failure: `c-bob-revoked`, same step, same evidence.
|
||
One is a deliberate product decision that revocation batches; the other is a
|
||
propagation race. **No observation distinguishes them.** Only intent does.
|
||
|
||
This is why claims require independent provenance (D-06), why `AMBIGUOUS`
|
||
escalates to a human rather than resolving itself, and why T08's classifier is
|
||
not permitted to guess.
|
||
|
||
## Classifier response (cross-surface journey)
|
||
|
||
Measured in `tests/test_classification.py`. **False Adaptation Rate = 0/7.**
|
||
|
||
| Ground truth | Accepted without a human |
|
||
|---|---|
|
||
| MECHANICAL (12) | 11 — M22 escalates as `AMBIGUOUS` |
|
||
| SEMANTIC (4) | 2 — M13 and M14, both genuinely inert |
|
||
| DEFECT (7) | **0** |
|
||
|
||
## Regenerating
|
||
|
||
The matrix is asserted in `tests/test_lab_ground_truth.py`. A moved cell fails
|
||
the suite: changing the measuring instrument must be a deliberate, reviewed act.
|
||
|
||
## T03 extension — 2026-09-28
|
||
|
||
M25–M31 drop identifiers while changing field order, dialog placement, anchor
|
||
affordances, fieldset grouping, textarea input, unrelated-form competition and
|
||
sidebar placement. M32–M38 pair the same mechanisms with identifiers preserved.
|
||
All fourteen are mechanical by the pre-run synthetic contract: grant behavior
|
||
and the endpoint are unchanged. These additions are agent-authored instrument
|
||
labels, not a claim of new human review. Both arms and the full classifier are
|
||
measured in `research/evidence/2026-09-28-e001.json`.
|