coulomb/test-driver
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

T05: the lab and its labelled mutation catalogue

Browse source 
lab/app.py (users, tenants, auth, resources, sharing, read/write, revoke,
audit), lab/http_api.py (JSON API + browser UI, stdlib only), 20 labelled
composable version-stamped mutations, ground-truth matrix. 48 tests pass.

Detection against the reference scenario: MECHANICAL 0/10 flagged (correct),
DEFECT 6/6, SEMANTIC 2/4 with both inert cases declared.

- F-0002: M16 and M18 initially escaped detection entirely. A use case
  protects exactly what it asserts. Resolved by adding two claims already
  stated as intent in INTENT.md; the six-mutation catalogue would never have
  surfaced this.
- test-id axis added: stable selectors survive most UI mutations, which would
  make H-001 trivially false. Mutations now vary on preserves_test_ids so the
  hypothesis is analysed split by that axis rather than rigged.
- M12 (semantic deferred revoke) and M19 (defect race) are behaviourally
  identical and asserted as such - the discrimination problem as a test.

lab/minimal.py removed; superseded by lab/app.py.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 1629012@bnt-lap001
Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
This commit is contained in:
tegwick 2026-08-22 23:31:22 +02:00
parent 4d3421ca57
commit 4ddb2f896c
22 changed files with 1091 additions and 207 deletions
Download patch file Download diff file Expand all files Collapse all files

61
lab/GROUND-TRUTH.md Normal file
View file

@ -0,0 +1,61 @@
# Lab Ground Truth
**Lab version base:** `lab-0.2.0` · **Catalogue:** 20 mutations · **Scenario:**
`scenarios/alice_bob_carol.py`
Labels are decided by a human from the use case and recorded **before** any run.
They are never inferred from behaviour — that is the whole point, and M12/M19
below show why.
| ID | Mutation | Label | Test ids | Reference scenario |
|---|---|---|---|---|
| M01 | Sharing control moves into a modal | MECHANICAL | preserved | PASS |
| M02 | DOM rewritten, test ids not carried forward | MECHANICAL | **dropped** | PASS |
| M03 | API renames `resource_id` to `id` | MECHANICAL | preserved | PASS |
| M04 | UI labels reworded | MECHANICAL | preserved | PASS |
| M05 | Form field order reversed | MECHANICAL | preserved | PASS |
| M06 | API paths shortened | MECHANICAL | preserved | PASS |
| M07 | Buttons become anchors | MECHANICAL | preserved | PASS |
| M08 | Revoke gains a confirmation step | MECHANICAL | preserved | PASS |
| M09 | Responses are slower | MECHANICAL | preserved | PASS |
| M10 | Denials return 401 instead of 403 | MECHANICAL | preserved | PASS |
| M11 | A share must be accepted first | SEMANTIC | preserved | FAIL |
| M12 | Revocation is deferred by decision | SEMANTIC | preserved | FAIL |
| M13 | Grants default to WRITE | SEMANTIC | preserved | PASS *(inert)* |
| M14 | Cross-tenant sharing declared prohibited | SEMANTIC | preserved | PASS *(inert)* |
| M15 | Revocation updates record but not enforcement | DEFECT | preserved | FAIL |
| M16 | A READ grant confers WRITE | DEFECT | preserved | FAIL |
| M17 | Any authenticated user can read anything | DEFECT | preserved | FAIL |
| M18 | Revocation is not audited | DEFECT | preserved | FAIL |
| M19 | Revocation propagates after a delay | DEFECT | preserved | FAIL |
| M20 | Tenant isolation leaks | DEFECT | preserved | FAIL |
Baseline: PASS. Detection: **MECHANICAL 0/10 flagged** (correct — semantics
preserved), **DEFECT 6/6 flagged**, **SEMANTIC 2/4 flagged**.
## The two inert mutations
Recorded rather than hidden. `test_inert_semantic_mutations_are_declared` fails
if an invisible mutation is ever left undeclared.
- **M13** only affects grants that omit a permission; the reference scenario
passes `READ` explicitly, so nothing changes.
- **M14** is a change of *intent* with no change of code — cross-tenant sharing
was already enforced, and the mutation declares it deliberate. Nothing
observable moves. This is the sharpest available demonstration that
classification cannot be a diff.
## M12 vs M19 — the discrimination problem in one row
Both produce an identical failure: `c-bob-revoked`, same step, same evidence.
One is a deliberate product decision that revocation batches; the other is a
propagation race. **No observation distinguishes them.** Only intent does.
This is why claims require independent provenance (D-06), why `AMBIGUOUS`
escalates to a human rather than resolving itself, and why T08's classifier is
not permitted to guess.
## Regenerating
The matrix is asserted in `tests/test_lab_ground_truth.py`. A moved cell fails
the suite: changing the measuring instrument must be a deliberate, reviewed act.

BIN
lab/__pycache__/app.cpython-312.pyc Normal file
View file

Binary file not shown.

BIN
lab/__pycache__/http_api.cpython-312.pyc Normal file
View file

Binary file not shown.

BIN
lab/__pycache__/mutations.cpython-312.pyc Normal file
View file

Binary file not shown.

309
lab/app.py Normal file
View file

@ -0,0 +1,309 @@
"""The test-driver lab — the system under test.
Grown from the T04 seed (`lab/minimal.py`, now removed). Users belong to
tenants, own resources, share them with READ or WRITE permission, and revoke
that sharing. Every state change is audited.
Two access paths exist, and the distinction is the whole point:
* the **enforcement path** (`request`) is what an actor uses. It checks
authorization and can therefore be *wrong* — that is where a seeded
authorization defect lives.
* the **observation channel** (`ObservationChannel`) reads stored state directly
and probes enforcement out-of-band. It is the independent channel required by
decision D-07.
An oracle consulting only stored state would verify test-driver's own
reimplementation of the rules rather than the system's enforcement of them. An
oracle consulting only enforcement could not notice that record and enforcement
disagree. test-driver observes both and treats disagreement as meaningful in
itself — that disagreement is the precise signature of an authorization defect
which leaves the audit trail looking correct.
Mutations are applied by `lab.mutations.build_lab`, never by editing this file.
"""
from __future__ import annotations
import time
from dataclasses import dataclass, field
from typing import Any, Callable, Literal
Permission = Literal["READ", "WRITE"]
BASE_VERSION = "lab-0.2.0"
class Denied(Exception):
"""The enforcement path refused the request."""
def __init__(self, reason: str, status: int = 403) -> None:
super().__init__(reason)
self.status = status
@dataclass(slots=True)
class AuditRecord:
sequence: int
event: str
actor_id: str
resource_id: str | None = None
subject_id: str | None = None
permission: str | None = None
@dataclass(slots=True)
class User:
id: str
tenant_id: str
token: str
@dataclass(slots=True)
class LabApp:
"""In-process resource-sharing service.
Mutation hooks are plain attributes holding callables or flags. A mutation
replaces one, so that every mutated build differs from baseline in exactly
one named way and the difference is inspectable at runtime.
"""
version: str = f"{BASE_VERSION}-baseline"
applied_mutations: tuple[str, ...] = ()
users: dict[str, User] = field(default_factory=dict)
_tokens: dict[str, str] = field(default_factory=dict)
resources: dict[str, dict[str, Any]] = field(default_factory=dict)
grants: dict[tuple[str, str], Permission] = field(default_factory=dict)
audit: list[AuditRecord] = field(default_factory=list)
_seq: int = 0
# --- mutation hooks (baseline behaviour) ----------------------------
may_read: Callable[["LabApp", str, str], bool] | None = None
grant_permission_for: Callable[[str | None], Permission] | None = None
audit_revoke: bool = True
revoke_delay_seconds: float = 0.0
denied_status: int = 403
require_share_acceptance: bool = False
enforce_tenant_isolation: bool = True
response_id_field: str = "resource_id"
latency_seconds: float = 0.0
# --- presentation hooks (surface only; never affect domain semantics) ---
ui_share_control: str = "inline"
ui_dom_style: str = "flat"
ui_labels: str = "plain"
ui_field_order: str = "natural"
ui_button_element: str = "button"
ui_test_ids: str = "stable"
ui_confirm_revoke: bool = False
api_path_style: str = "long"
tenant_sharing_announced: bool = False
# -- setup -----------------------------------------------------------
def add_user(self, user_id: str, tenant_id: str = "t-acme") -> str:
token = f"tok-{user_id}"
self.users[user_id] = User(user_id, tenant_id, token)
self._tokens[token] = user_id
return token
def _audit(self, event: str, actor_id: str, **kw: Any) -> None:
self._seq += 1
self.audit.append(AuditRecord(self._seq, event, actor_id, **kw))
def _whoami(self, token: str) -> str:
if token not in self._tokens:
raise Denied("unknown token", status=401)
return self._tokens[token]
def tenant_of(self, user_id: str) -> str | None:
user = self.users.get(user_id)
return user.tenant_id if user else None
# -- enforcement path (what actors use) ------------------------------
def request(self, token: str, op: str, **args: Any) -> Any:
"""The single entry point actors go through. Authorization is enforced here."""
if self.latency_seconds:
time.sleep(self.latency_seconds)
user_id = self._whoami(token)
handler = getattr(self, f"_op_{op}", None)
if handler is None:
raise Denied(f"unknown operation {op!r}", status=404)
return handler(user_id, **args)
def _op_create_resource(self, user_id: str, resource_id: str, content: str) -> dict:
self.resources[resource_id] = {"owner": user_id, "content": content}
self._audit("create", user_id, resource_id=resource_id)
return {self.response_id_field: resource_id}
def _op_read_resource(self, user_id: str, resource_id: str) -> dict:
resource = self.resources.get(resource_id)
if resource is None:
raise Denied("no such resource", status=404)
if not self._may_read(user_id, resource_id):
raise Denied("not authorized to read", status=self.denied_status)
return {self.response_id_field: resource_id, "content": resource["content"]}
def _op_write_resource(self, user_id: str, resource_id: str, content: str) -> dict:
resource = self.resources.get(resource_id)
if resource is None:
raise Denied("no such resource", status=404)
if not self._may_write(user_id, resource_id):
raise Denied("not authorized to write", status=self.denied_status)
resource["content"] = content
self._audit("write", user_id, resource_id=resource_id)
return {"written": True}
def _op_grant(
self,
user_id: str,
resource_id: str,
subject_id: str,
permission: Permission | None = None,
) -> dict:
resource = self.resources.get(resource_id)
if resource is None or resource["owner"] != user_id:
raise Denied("only the owner may grant")
if self.enforce_tenant_isolation:
if self.tenant_of(subject_id) != self.tenant_of(user_id):
raise Denied("cross-tenant sharing is not permitted")
effective = (
self.grant_permission_for(permission)
if self.grant_permission_for
else (permission or "READ")
)
self.grants[(resource_id, subject_id)] = effective
if self.require_share_acceptance:
self.resources[resource_id].setdefault("pending", set()).add(subject_id)
self._audit(
"grant", user_id, resource_id=resource_id,
subject_id=subject_id, permission=effective,
)
return {"granted": effective}
def _op_accept_share(self, user_id: str, resource_id: str) -> dict:
pending = self.resources.get(resource_id, {}).get("pending")
if pending:
pending.discard(user_id)
self._audit("accept", user_id, resource_id=resource_id, subject_id=user_id)
return {"accepted": True}
def _op_revoke(self, user_id: str, resource_id: str, subject_id: str) -> dict:
resource = self.resources.get(resource_id)
if resource is None or resource["owner"] != user_id:
raise Denied("only the owner may revoke")
if self.revoke_delay_seconds:
self.resources[resource_id].setdefault("revoke_after", {})[subject_id] = (
time.monotonic() + self.revoke_delay_seconds
)
else:
self.grants.pop((resource_id, subject_id), None)
if self.audit_revoke:
self._audit("revoke", user_id, resource_id=resource_id, subject_id=subject_id)
return {"revoked": True}
def _may_read(self, user_id: str, resource_id: str) -> bool:
"""Authorization as the system actually enforces it."""
if self.may_read is not None:
return self.may_read(self, user_id, resource_id)
return self.baseline_may_read(user_id, resource_id)
def _may_write(self, user_id: str, resource_id: str) -> bool:
resource = self.resources.get(resource_id)
if resource is None:
return False
if resource["owner"] == user_id:
return True
return self.grants.get((resource_id, user_id)) == "WRITE"
def baseline_may_read(self, user_id: str, resource_id: str) -> bool:
resource = self.resources.get(resource_id)
if resource is None:
return False
if resource["owner"] == user_id:
return True
deadline = resource.get("revoke_after", {}).get(user_id)
if deadline is not None and time.monotonic() < deadline:
return True # revocation is scheduled but not yet effective
if user_id in resource.get("pending", set()):
return False
return (resource_id, user_id) in self.grants
class ObservationChannel:
"""Independent read access to lab state — decision D-07.
Bypasses authorization deliberately. This is the channel test-driver requires
of any system under test, and the main integration burden the framework
imposes on an adopter.
"""
def __init__(self, app: LabApp) -> None:
self._app = app
@property
def version(self) -> str:
return self._app.version
def state_permission(self, user_id: str, resource_id: str) -> str | None:
"""What the stored record says, independent of any enforcement decision."""
resource = self._app.resources.get(resource_id)
if resource is None:
return None
if resource["owner"] == user_id:
return "OWNER"
return self._app.grants.get((resource_id, user_id))
def probe_read(self, user_id: str, resource_id: str) -> bool:
"""Exercise the enforcement path out-of-band and report what it did.
This uses the subject's own credentials, which can look like a violation
of actor isolation but is not: independence means the *actor's report* is
never the evidence. The observer issues its own request and records the
raw outcome. No actor is ever asked whether it succeeded.
"""
user = self._app.users.get(user_id)
if user is None:
return False
try:
self._app.request(user.token, "read_resource", resource_id=resource_id)
except Denied:
return False
return True
def probe_write(self, user_id: str, resource_id: str) -> bool:
"""Out-of-band probe of the write path, non-destructive on refusal."""
user = self._app.users.get(user_id)
if user is None:
return False
original = self._app.resources.get(resource_id, {}).get("content")
try:
self._app.request(
user.token, "write_resource", resource_id=resource_id, content=original
)
except Denied:
return False
return True
def audit_events(self, resource_id: str) -> list[dict[str, Any]]:
return [
{
"sequence": r.sequence,
"event": r.event,
"actor_id": r.actor_id,
"subject_id": r.subject_id,
"permission": r.permission,
}
for r in self._app.audit
if r.resource_id == resource_id
]
def build_baseline() -> tuple[LabApp, dict[str, str]]:
"""Known initial state, so that runs replay from the same starting point."""
app = LabApp()
tokens = {u: app.add_user(u) for u in ("alice", "bob", "carol")}
tokens["mallory"] = app.add_user("mallory", tenant_id="t-other")
return app, tokens

215
lab/http_api.py Normal file
View file

@ -0,0 +1,215 @@
"""HTTP surface for the lab: a JSON API and a minimal browser UI.
Stdlib only. The point of this layer is to give the agentic browser driver (T07)
something real to navigate, and to give the mechanical mutations somewhere to
bite — a mutation that moves a control is inert until there is a control to move.
The domain is not reimplemented here. Every request funnels into
`LabApp.request`, so the UI and API cannot drift from the enforcement path that
the deterministic driver and the observation channel already exercise.
"""
from __future__ import annotations
import json
import re
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from typing import Any
from urllib.parse import parse_qs, urlparse
from .app import Denied, LabApp
def _resource_path(app: LabApp, resource_id: str) -> str:
return (
f"/r/{resource_id}" if app.api_path_style == "short"
else f"/resources/{resource_id}"
)
def render_resource_page(app: LabApp, user_id: str, resource_id: str) -> str:
"""The browser surface. Every mechanical UI mutation shows up here."""
html = _render(app, user_id, resource_id)
if app.ui_test_ids == "dropped":
# A rewrite that does not carry the old test ids forward. See the
# "test-id axis" note in lab/mutations.py — this is the case that
# decides whether semantic actions earn their keep.
html = _TEST_ID_ATTR.sub("", html)
return html
_TEST_ID_ATTR = re.compile(r'\s*data-td="[^"]*"')
def _render(app: LabApp, user_id: str, resource_id: str) -> str:
verbose = app.ui_labels == "verbose"
share_label = "Give access" if verbose else "Share"
revoke_label = "Withdraw access" if verbose else "Revoke"
tag = "a" if app.ui_button_element == "anchor" else "button"
role = ' role="button"' if tag == "a" else ""
href = ' href="#"' if tag == "a" else ""
subject_field = (
'<label for="subject">Person</label>'
'<input id="subject" name="subject_id" data-td="subject">'
)
permission_field = (
'<label for="permission">Permission</label>'
'<select id="permission" name="permission" data-td="permission">'
'<option>READ</option><option>WRITE</option></select>'
)
fields = (
permission_field + subject_field
if app.ui_field_order == "reversed"
else subject_field + permission_field
)
share_form = (
f'<form id="share" data-td="share-form" method="post" '
f'action="{_resource_path(app, resource_id)}/grant">'
f"{fields}"
f'<{tag}{href}{role} id="share-submit" data-td="share-submit">'
f"{share_label}</{tag}></form>"
)
if app.ui_share_control == "modal":
share_form = (
f'<{tag}{href}{role} id="open-share" data-td="open-share">{share_label}…'
f"</{tag}>"
f'<dialog id="share-modal" data-td="share-modal" open>{share_form}</dialog>'
)
confirm = (
'<p data-td="confirm-note">This cannot be undone.</p>'
f'<{tag}{href}{role} data-td="revoke-confirm">Yes, {revoke_label.lower()}</{tag}>'
if app.ui_confirm_revoke else ""
)
revoke_form = (
f'<form id="revoke" data-td="revoke-form" method="post" '
f'action="{_resource_path(app, resource_id)}/revoke">'
f'<input name="subject_id" data-td="revoke-subject">'
f'<{tag}{href}{role} data-td="revoke-submit">{revoke_label}</{tag}>'
f"{confirm}</form>"
)
grants = "".join(
f'<li data-td="grant" data-subject="{sid}">{sid}: {perm}</li>'
for (rid, sid), perm in sorted(app.grants.items())
if rid == resource_id
)
body = (
f'<h1 data-td="resource-title">Resource {resource_id}</h1>'
f'<ul data-td="grant-list">{grants}</ul>{share_form}{revoke_form}'
)
if app.ui_dom_style == "nested":
body = (
'<div class="shell"><section class="panel"><div class="panel-inner">'
f"{body}</div></section></div>"
)
return (
"<!doctype html><html><head><title>Lab</title></head>"
f'<body data-td-version="{app.version}" data-td-user="{user_id}">{body}</body></html>'
)
class LabHandler(BaseHTTPRequestHandler):
app: LabApp
def log_message(self, *args: Any) -> None: # keep the test output quiet
pass
# -- helpers ---------------------------------------------------------
def _token(self) -> str:
header = self.headers.get("Authorization", "")
return header.removeprefix("Bearer ").strip()
def _send(self, status: int, payload: Any, content_type: str = "application/json") -> None:
body = (
json.dumps(payload).encode()
if content_type == "application/json"
else payload.encode()
)
self.send_response(status)
self.send_header("Content-Type", content_type)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def _dispatch(self, op: str, **args: Any) -> None:
try:
self._send(200, self.app.request(self._token(), op, **args))
except Denied as denied:
self._send(denied.status, {"error": str(denied)})
def _resource_id(self, path: str) -> str | None:
parts = [p for p in path.split("/") if p]
if parts and parts[0] in ("resources", "r") and len(parts) >= 2:
return parts[1]
return None
# -- routes ----------------------------------------------------------
def do_GET(self) -> None: # noqa: N802 — stdlib naming
url = urlparse(self.path)
resource_id = self._resource_id(url.path)
if resource_id is None:
self._send(404, {"error": "not found"})
return
if url.path.endswith("/view"):
try:
user_id = self.app._whoami(self._token())
except Denied as denied:
self._send(denied.status, {"error": str(denied)})
return
self._send(200, render_resource_page(self.app, user_id, resource_id), "text/html")
return
self._dispatch("read_resource", resource_id=resource_id)
def do_POST(self) -> None: # noqa: N802
url = urlparse(self.path)
resource_id = self._resource_id(url.path)
length = int(self.headers.get("Content-Length", 0))
raw = self.rfile.read(length).decode() if length else ""
try:
body = json.loads(raw) if raw.startswith("{") else {
k: v[0] for k, v in parse_qs(raw).items()
}
except json.JSONDecodeError:
self._send(400, {"error": "malformed body"})
return
if resource_id is None:
if url.path.rstrip("/") in ("/resources", "/r"):
self._dispatch("create_resource", **body)
return
self._send(404, {"error": "not found"})
return
if url.path.endswith("/grant"):
self._dispatch("grant", resource_id=resource_id, **body)
elif url.path.endswith("/revoke"):
self._dispatch("revoke", resource_id=resource_id, **body)
elif url.path.endswith("/accept"):
self._dispatch("accept_share", resource_id=resource_id)
else:
self._send(404, {"error": "not found"})
def serve(app: LabApp, port: int = 0) -> ThreadingHTTPServer:
"""Start a server on `port` (0 picks a free one). Caller owns shutdown."""
handler = type("BoundLabHandler", (LabHandler,), {"app": app})
return ThreadingHTTPServer(("127.0.0.1", port), handler)
if __name__ == "__main__": # pragma: no cover - manual use
import sys
from .mutations import build_lab
lab, tokens = build_lab(*sys.argv[1:])
lab.request(tokens["alice"], "create_resource", resource_id="R", content="the secret")
server = serve(lab, 8099)
print(f"{lab.version} on http://127.0.0.1:8099 tokens={tokens}")
server.serve_forever()

191
lab/minimal.py
View file

@ -1,191 +0,0 @@
"""Minimal lab: the seed of the system under test.
Deliberately small. Users own resources, share them with read or write
permission, and revoke that sharing. Every state change is audited.
Two access paths exist, and the distinction is the whole point:
* the **enforcement path** (`request`) is what an actor uses. It checks
authorization and can therefore be *wrong* — that is where a seeded
authorization defect lives.
* the **observation channel** (`ObservationChannel`) reads stored state directly,
without authorization. It is the independent channel required by decision
D-07.
An oracle that consulted only stored state would verify the framework's own
reimplementation of the rules rather than the system's enforcement of them. An
oracle that consulted only the enforcement path would have no way to notice that
enforcement and record disagree. test-driver observes both, and treats
disagreement between them as meaningful in its own right — that disagreement is
the precise signature of the M05 authorization defect.
TD-WP-0002-T05 grows this into the full lab with an HTTP API, a browser UI and
the labelled mutation catalogue. It is kept in-process here so that T04 can prove
the kernel without dragging in a web stack.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Any, Literal
Permission = Literal["READ", "WRITE"]
class Denied(Exception):
"""The enforcement path refused the request."""
@dataclass(slots=True)
class AuditRecord:
sequence: int
event: str
actor_id: str
resource_id: str | None = None
subject_id: str | None = None
permission: str | None = None
@dataclass(slots=True)
class MinimalLab:
"""In-process resource-sharing service."""
version: str = "lab-0.1.0-baseline"
users: dict[str, str] = field(default_factory=dict) # user_id -> token
_tokens: dict[str, str] = field(default_factory=dict) # token -> user_id
resources: dict[str, dict[str, Any]] = field(default_factory=dict)
grants: dict[tuple[str, str], Permission] = field(default_factory=dict)
audit: list[AuditRecord] = field(default_factory=list)
_seq: int = 0
# -- setup -----------------------------------------------------------
def add_user(self, user_id: str) -> str:
token = f"tok-{user_id}"
self.users[user_id] = token
self._tokens[token] = user_id
return token
def _audit(self, event: str, actor_id: str, **kw: Any) -> None:
self._seq += 1
self.audit.append(AuditRecord(self._seq, event, actor_id, **kw))
def _whoami(self, token: str) -> str:
if token not in self._tokens:
raise Denied("unknown token")
return self._tokens[token]
# -- enforcement path (what actors use) ------------------------------
def request(self, token: str, op: str, **args: Any) -> Any:
"""The single entry point actors go through. Authorization is enforced here."""
user_id = self._whoami(token)
handler = getattr(self, f"_op_{op}", None)
if handler is None:
raise Denied(f"unknown operation {op!r}")
return handler(user_id, **args)
def _op_create_resource(self, user_id: str, resource_id: str, content: str) -> dict:
self.resources[resource_id] = {"owner": user_id, "content": content}
self._audit("create", user_id, resource_id=resource_id)
return {"resource_id": resource_id}
def _op_read_resource(self, user_id: str, resource_id: str) -> dict:
resource = self.resources.get(resource_id)
if resource is None:
raise Denied("no such resource")
if not self._may_read(user_id, resource_id):
raise Denied("not authorized to read")
return {"resource_id": resource_id, "content": resource["content"]}
def _op_grant(
self, user_id: str, resource_id: str, subject_id: str, permission: Permission
) -> dict:
resource = self.resources.get(resource_id)
if resource is None or resource["owner"] != user_id:
raise Denied("only the owner may grant")
self.grants[(resource_id, subject_id)] = permission
self._audit(
"grant", user_id, resource_id=resource_id,
subject_id=subject_id, permission=permission,
)
return {"granted": permission}
def _op_revoke(self, user_id: str, resource_id: str, subject_id: str) -> dict:
resource = self.resources.get(resource_id)
if resource is None or resource["owner"] != user_id:
raise Denied("only the owner may revoke")
self.grants.pop((resource_id, subject_id), None)
self._audit("revoke", user_id, resource_id=resource_id, subject_id=subject_id)
return {"revoked": True}
def _may_read(self, user_id: str, resource_id: str) -> bool:
"""The authorization rule as the system actually enforces it."""
resource = self.resources.get(resource_id)
if resource is None:
return False
if resource["owner"] == user_id:
return True
return (resource_id, user_id) in self.grants
class ObservationChannel:
"""Independent read access to lab state — decision D-07.
Bypasses authorization deliberately. This is the channel test-driver requires
of any system under test, and the main integration burden the framework
imposes on an adopter.
"""
def __init__(self, lab: MinimalLab) -> None:
self._lab = lab
@property
def version(self) -> str:
return self._lab.version
def state_permission(self, user_id: str, resource_id: str) -> str | None:
"""What the stored record says, independent of any enforcement decision."""
resource = self._lab.resources.get(resource_id)
if resource is None:
return None
if resource["owner"] == user_id:
return "OWNER"
return self._lab.grants.get((resource_id, user_id))
def probe_read(self, user_id: str, resource_id: str) -> bool:
"""Exercise the enforcement path out-of-band and report what it did.
This uses the subject's own credentials, which can look like a violation
of actor isolation but is not: independence means the *actor's report* is
never the evidence. The observer issues its own request and records the
raw outcome. No actor is ever asked whether it succeeded.
"""
token = self._lab.users.get(user_id)
if token is None:
return False
try:
self._lab.request(token, "read_resource", resource_id=resource_id)
except Denied:
return False
return True
def audit_events(self, resource_id: str) -> list[dict[str, Any]]:
return [
{
"sequence": r.sequence,
"event": r.event,
"actor_id": r.actor_id,
"subject_id": r.subject_id,
"permission": r.permission,
}
for r in self._lab.audit
if r.resource_id == resource_id
]
def build_baseline() -> tuple[MinimalLab, dict[str, str]]:
"""Known initial state, so that runs replay from the same starting point."""
lab = MinimalLab()
tokens = {user: lab.add_user(user) for user in ("alice", "bob", "carol")}
return lab, tokens

205
lab/mutations.py Normal file
View file

@ -0,0 +1,205 @@
"""The labelled mutation catalogue — the project's measuring instrument.
Every claim test-driver makes is measured against this catalogue, so its quality
caps the credibility of every downstream result. Six mutations, as the milestones
document originally sketched, cannot support any statement about precision or
recall; there are twenty here.
Each mutation carries a **ground-truth label**, decided by a human from the use
case and recorded before any run:
MECHANICAL the surface changed; protected semantics are identical.
test-driver should recover and report an adaptation.
SEMANTIC intended behaviour genuinely changed. test-driver must escalate
to a human and must never rewrite a claim by itself.
DEFECT the system violates unchanged intent. test-driver must report a
Product Finding and must never adapt to it.
The distinction between SEMANTIC and DEFECT is deliberately *not* inferable from
the code — both change behaviour. It is a statement about intent, which is why
claims must have independent provenance (D-06) and why ambiguity escalates
rather than resolving itself.
Mutations compose: `build_lab("M01", "M15")` applies both and records both in the
version string, so a mechanical change shipping alongside a defect is
reproducible. That combination is decision-table row 3 and the case a naive
self-healing tool gets wrong.
## The test-id axis
The UI carries stable `data-td` attributes, as a well-instrumented application
would. Most mechanical mutations preserve them, and a recorded selector sequence
keyed on those attributes survives such a mutation untouched — which would make
H-001 (semantic actions outlast recorded sequences) trivially *false*.
That is not a flaw to be rigged away. It is the honest shape of the question:
**a semantic action earns its keep exactly when stable identifiers are absent or
not carried forward.** Mutations therefore vary along `preserves_test_ids`, and
H-001 must be analysed split by that axis rather than as a single rate. A
catalogue whose mutations all broke naive selectors would flatter the thesis and
tell us nothing.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Callable, Literal
from .app import BASE_VERSION, LabApp, ObservationChannel, build_baseline
__all__ = [
"CATALOGUE", "BY_ID", "Mutation", "ObservationChannel", "build_lab",
"expected_classification",
]
Label = Literal["MECHANICAL", "SEMANTIC", "DEFECT"]
Layer = Literal["ui", "api", "domain"]
@dataclass(frozen=True, slots=True)
class Mutation:
id: str
title: str
label: Label
layer: Layer
rationale: str
apply: Callable[[LabApp], None]
preserves_test_ids: bool = True
def _m(app: LabApp, **flags) -> None:
for key, value in flags.items():
setattr(app, key, value)
# --- MECHANICAL -----------------------------------------------------------
# The surface moves. What the system means does not.
def _m01(app): _m(app, ui_share_control="modal")
def _m02(app): _m(app, ui_dom_style="nested", ui_test_ids="dropped")
def _m03(app): _m(app, response_id_field="id")
def _m04(app): _m(app, ui_labels="verbose")
def _m05(app): _m(app, ui_field_order="reversed")
def _m06(app): _m(app, api_path_style="short")
def _m07(app): _m(app, ui_button_element="anchor")
def _m08(app): _m(app, ui_confirm_revoke=True)
def _m09(app): _m(app, latency_seconds=0.05)
def _m10(app): _m(app, denied_status=401)
# --- SEMANTIC -------------------------------------------------------------
# Intended behaviour changed. A human must decide; test-driver must not.
def _m11(app): _m(app, require_share_acceptance=True)
def _m12(app): _m(app, revoke_delay_seconds=3600.0)
def _m13(app): _m(app, grant_permission_for=lambda p: p or "WRITE")
def _m14(app): _m(app, enforce_tenant_isolation=True, tenant_sharing_announced=True)
# --- DEFECT ---------------------------------------------------------------
# Unchanged intent, violated.
def _revoke_is_cosmetic(app: LabApp, user_id: str, resource_id: str) -> bool:
"""Record and audit say revoked; enforcement still allows the read."""
resource = app.resources.get(resource_id)
if resource is None:
return False
if resource["owner"] == user_id:
return True
return any(
r.event == "grant" and r.subject_id == user_id and r.resource_id == resource_id
for r in app.audit
)
def _read_permits_everyone(app: LabApp, user_id: str, resource_id: str) -> bool:
return resource_id in app.resources
def _ignores_tenant(app: LabApp, user_id: str, resource_id: str) -> bool:
return app.baseline_may_read(user_id, resource_id) or resource_id in app.resources
def _m15(app): _m(app, may_read=_revoke_is_cosmetic)
def _m16(app): _m(app, grant_permission_for=lambda p: "WRITE")
def _m17(app): _m(app, may_read=_read_permits_everyone)
def _m18(app): _m(app, audit_revoke=False)
def _m19(app): _m(app, revoke_delay_seconds=2.0)
def _m20(app): _m(app, enforce_tenant_isolation=False, may_read=_ignores_tenant)
CATALOGUE: tuple[Mutation, ...] = (
Mutation("M01", "Sharing control moves into a modal", "MECHANICAL", "ui",
"Same action, different place. The canonical mechanical change.", _m01),
Mutation("M02", "DOM rewritten, test ids not carried forward", "MECHANICAL", "ui",
"Selectors break; nothing the user can do changes. Unlike M01 this "
"drops the stable test ids, which is what a real rewrite usually "
"does.", _m02, preserves_test_ids=False),
Mutation("M03", "API renames resource_id to id", "MECHANICAL", "api",
"Compatible representation change of the same field.", _m03),
Mutation("M04", "UI labels reworded", "MECHANICAL", "ui",
"'Share' becomes 'Give access'. Wording, not meaning.", _m04),
Mutation("M05", "Form field order reversed", "MECHANICAL", "ui",
"Permission precedes subject. Same form, same result.", _m05),
Mutation("M06", "API paths shortened", "MECHANICAL", "api",
"/resources/{id} becomes /r/{id}.", _m06),
Mutation("M07", "Buttons become anchors", "MECHANICAL", "ui",
"Element type changes; affordance does not.", _m07),
Mutation("M08", "Revoke gains a confirmation step", "MECHANICAL", "ui",
"An extra legitimate click. The postcondition is unchanged, so this "
"is mechanical — but it is the closest mechanical case to a semantic "
"one, and a good test of where the boundary really sits.", _m08),
Mutation("M09", "Responses are slower", "MECHANICAL", "api",
"Latency only. Tests patience, not semantics.", _m09),
Mutation("M10", "Denials return 401 instead of 403", "MECHANICAL", "api",
"Both mean refused. A driver keying on the exact code breaks.", _m10),
Mutation("M11", "A share must be accepted before it takes effect", "SEMANTIC", "domain",
"Bob genuinely cannot read until he accepts. The old claim 'Bob can "
"read after the grant' is now wrong, and only a human may say so.", _m11),
Mutation("M12", "Revocation is deferred, not immediate", "SEMANTIC", "domain",
"A deliberate product decision that revocation batches. Looks exactly "
"like M19 from the outside; the difference is intent.", _m12),
Mutation("M13", "Grants default to WRITE", "SEMANTIC", "domain",
"Only affects grants that omit a permission. Inert for the reference "
"scenario, which passes READ explicitly — recorded as such rather "
"than quietly dropped.", _m13),
Mutation("M14", "Cross-tenant sharing is announced as prohibited", "SEMANTIC", "domain",
"Codifies existing enforcement as intended behaviour. A change of "
"intent with no change of code — the case that shows classification "
"cannot be a diff.", _m14),
Mutation("M15", "Revocation updates the record but not enforcement", "DEFECT", "domain",
"Audit trail and stored grant both say revoked; the read still "
"succeeds. The defect that looks correct to an operator.", _m15),
Mutation("M16", "A READ grant confers WRITE", "DEFECT", "domain",
"Privilege escalation through the ordinary sharing path.", _m16),
Mutation("M17", "Any authenticated user can read any resource", "DEFECT", "domain",
"Authorization effectively removed. Carol reads R.", _m17),
Mutation("M18", "Revocation is not audited", "DEFECT", "domain",
"Enforcement is correct; the evidence trail is not. Detectable only "
"because audit is observed, not assumed.", _m18),
Mutation("M19", "Revocation propagates after a delay", "DEFECT", "domain",
"Not a decision — a race. Indistinguishable from M12 by behaviour "
"alone, which is exactly the point.", _m19),
Mutation("M20", "Tenant isolation leaks", "DEFECT", "domain",
"A user of another tenant reads the resource.", _m20),
)
BY_ID: dict[str, Mutation] = {m.id: m for m in CATALOGUE}
def expected_classification(mutation_id: str) -> Label:
"""Ground truth. Recorded by a human before any run — never inferred."""
return BY_ID[mutation_id].label
def build_lab(*mutation_ids: str) -> tuple[LabApp, dict[str, str]]:
"""Build a lab with the named mutations applied, versioned by what it carries."""
app, tokens = build_baseline()
for mutation_id in mutation_ids:
BY_ID[mutation_id].apply(app)
app.applied_mutations = tuple(mutation_ids)
suffix = "+".join(mutation_ids) if mutation_ids else "baseline"
app.version = f"{BASE_VERSION}-{suffix}"
return app, tokens