diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index eeeaba5..0ce0894 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -11,12 +11,12 @@ | workplan | TD-WP-0001 | active | — | workplans/TD-WP-0001-statehub-bootstrap.md | | workplan | TD-WP-0002 | active | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0001-T01 | done | — | workplans/TD-WP-0001-statehub-bootstrap.md | -| task | TD-WP-0001-T02 | wait | — | workplans/TD-WP-0001-statehub-bootstrap.md | +| task | TD-WP-0001-T02 | done | — | workplans/TD-WP-0001-statehub-bootstrap.md | | task | TD-WP-0001-T03 | done | — | workplans/TD-WP-0001-statehub-bootstrap.md | | task | TD-WP-0002-T01 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T02 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T03 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | -| task | TD-WP-0002-T04 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | +| task | TD-WP-0002-T04 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T05 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T06 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T07 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | diff --git a/lab/GROUND-TRUTH.md b/lab/GROUND-TRUTH.md new file mode 100644 index 0000000..e2f4067 --- /dev/null +++ b/lab/GROUND-TRUTH.md @@ -0,0 +1,61 @@ +# Lab Ground Truth + +**Lab version base:** `lab-0.2.0` · **Catalogue:** 20 mutations · **Scenario:** +`scenarios/alice_bob_carol.py` + +Labels are decided by a human from the use case and recorded **before** any run. +They are never inferred from behaviour — that is the whole point, and M12/M19 +below show why. + +| ID | Mutation | Label | Test ids | Reference scenario | +|---|---|---|---|---| +| M01 | Sharing control moves into a modal | MECHANICAL | preserved | PASS | +| M02 | DOM rewritten, test ids not carried forward | MECHANICAL | **dropped** | PASS | +| M03 | API renames `resource_id` to `id` | MECHANICAL | preserved | PASS | +| M04 | UI labels reworded | MECHANICAL | preserved | PASS | +| M05 | Form field order reversed | MECHANICAL | preserved | PASS | +| M06 | API paths shortened | MECHANICAL | preserved | PASS | +| M07 | Buttons become anchors | MECHANICAL | preserved | PASS | +| M08 | Revoke gains a confirmation step | MECHANICAL | preserved | PASS | +| M09 | Responses are slower | MECHANICAL | preserved | PASS | +| M10 | Denials return 401 instead of 403 | MECHANICAL | preserved | PASS | +| M11 | A share must be accepted first | SEMANTIC | preserved | FAIL | +| M12 | Revocation is deferred by decision | SEMANTIC | preserved | FAIL | +| M13 | Grants default to WRITE | SEMANTIC | preserved | PASS *(inert)* | +| M14 | Cross-tenant sharing declared prohibited | SEMANTIC | preserved | PASS *(inert)* | +| M15 | Revocation updates record but not enforcement | DEFECT | preserved | FAIL | +| M16 | A READ grant confers WRITE | DEFECT | preserved | FAIL | +| M17 | Any authenticated user can read anything | DEFECT | preserved | FAIL | +| M18 | Revocation is not audited | DEFECT | preserved | FAIL | +| M19 | Revocation propagates after a delay | DEFECT | preserved | FAIL | +| M20 | Tenant isolation leaks | DEFECT | preserved | FAIL | + +Baseline: PASS. Detection: **MECHANICAL 0/10 flagged** (correct — semantics +preserved), **DEFECT 6/6 flagged**, **SEMANTIC 2/4 flagged**. + +## The two inert mutations + +Recorded rather than hidden. `test_inert_semantic_mutations_are_declared` fails +if an invisible mutation is ever left undeclared. + +- **M13** only affects grants that omit a permission; the reference scenario + passes `READ` explicitly, so nothing changes. +- **M14** is a change of *intent* with no change of code — cross-tenant sharing + was already enforced, and the mutation declares it deliberate. Nothing + observable moves. This is the sharpest available demonstration that + classification cannot be a diff. + +## M12 vs M19 — the discrimination problem in one row + +Both produce an identical failure: `c-bob-revoked`, same step, same evidence. +One is a deliberate product decision that revocation batches; the other is a +propagation race. **No observation distinguishes them.** Only intent does. + +This is why claims require independent provenance (D-06), why `AMBIGUOUS` +escalates to a human rather than resolving itself, and why T08's classifier is +not permitted to guess. + +## Regenerating + +The matrix is asserted in `tests/test_lab_ground_truth.py`. A moved cell fails +the suite: changing the measuring instrument must be a deliberate, reviewed act. diff --git a/lab/__pycache__/app.cpython-312.pyc b/lab/__pycache__/app.cpython-312.pyc new file mode 100644 index 0000000..d0d6b76 Binary files /dev/null and b/lab/__pycache__/app.cpython-312.pyc differ diff --git a/lab/__pycache__/http_api.cpython-312.pyc b/lab/__pycache__/http_api.cpython-312.pyc new file mode 100644 index 0000000..e9bfbb7 Binary files /dev/null and b/lab/__pycache__/http_api.cpython-312.pyc differ diff --git a/lab/__pycache__/mutations.cpython-312.pyc b/lab/__pycache__/mutations.cpython-312.pyc new file mode 100644 index 0000000..000d0db Binary files /dev/null and b/lab/__pycache__/mutations.cpython-312.pyc differ diff --git a/lab/app.py b/lab/app.py new file mode 100644 index 0000000..04f5161 --- /dev/null +++ b/lab/app.py @@ -0,0 +1,309 @@ +"""The test-driver lab — the system under test. + +Grown from the T04 seed (`lab/minimal.py`, now removed). Users belong to +tenants, own resources, share them with READ or WRITE permission, and revoke +that sharing. Every state change is audited. + +Two access paths exist, and the distinction is the whole point: + +* the **enforcement path** (`request`) is what an actor uses. It checks + authorization and can therefore be *wrong* — that is where a seeded + authorization defect lives. +* the **observation channel** (`ObservationChannel`) reads stored state directly + and probes enforcement out-of-band. It is the independent channel required by + decision D-07. + +An oracle consulting only stored state would verify test-driver's own +reimplementation of the rules rather than the system's enforcement of them. An +oracle consulting only enforcement could not notice that record and enforcement +disagree. test-driver observes both and treats disagreement as meaningful in +itself — that disagreement is the precise signature of an authorization defect +which leaves the audit trail looking correct. + +Mutations are applied by `lab.mutations.build_lab`, never by editing this file. +""" + +from __future__ import annotations + +import time +from dataclasses import dataclass, field +from typing import Any, Callable, Literal + +Permission = Literal["READ", "WRITE"] + +BASE_VERSION = "lab-0.2.0" + + +class Denied(Exception): + """The enforcement path refused the request.""" + + def __init__(self, reason: str, status: int = 403) -> None: + super().__init__(reason) + self.status = status + + +@dataclass(slots=True) +class AuditRecord: + sequence: int + event: str + actor_id: str + resource_id: str | None = None + subject_id: str | None = None + permission: str | None = None + + +@dataclass(slots=True) +class User: + id: str + tenant_id: str + token: str + + +@dataclass(slots=True) +class LabApp: + """In-process resource-sharing service. + + Mutation hooks are plain attributes holding callables or flags. A mutation + replaces one, so that every mutated build differs from baseline in exactly + one named way and the difference is inspectable at runtime. + """ + + version: str = f"{BASE_VERSION}-baseline" + applied_mutations: tuple[str, ...] = () + + users: dict[str, User] = field(default_factory=dict) + _tokens: dict[str, str] = field(default_factory=dict) + resources: dict[str, dict[str, Any]] = field(default_factory=dict) + grants: dict[tuple[str, str], Permission] = field(default_factory=dict) + audit: list[AuditRecord] = field(default_factory=list) + _seq: int = 0 + + # --- mutation hooks (baseline behaviour) ---------------------------- + may_read: Callable[["LabApp", str, str], bool] | None = None + grant_permission_for: Callable[[str | None], Permission] | None = None + audit_revoke: bool = True + revoke_delay_seconds: float = 0.0 + denied_status: int = 403 + require_share_acceptance: bool = False + enforce_tenant_isolation: bool = True + response_id_field: str = "resource_id" + latency_seconds: float = 0.0 + + # --- presentation hooks (surface only; never affect domain semantics) --- + ui_share_control: str = "inline" + ui_dom_style: str = "flat" + ui_labels: str = "plain" + ui_field_order: str = "natural" + ui_button_element: str = "button" + ui_test_ids: str = "stable" + ui_confirm_revoke: bool = False + api_path_style: str = "long" + tenant_sharing_announced: bool = False + + # -- setup ----------------------------------------------------------- + + def add_user(self, user_id: str, tenant_id: str = "t-acme") -> str: + token = f"tok-{user_id}" + self.users[user_id] = User(user_id, tenant_id, token) + self._tokens[token] = user_id + return token + + def _audit(self, event: str, actor_id: str, **kw: Any) -> None: + self._seq += 1 + self.audit.append(AuditRecord(self._seq, event, actor_id, **kw)) + + def _whoami(self, token: str) -> str: + if token not in self._tokens: + raise Denied("unknown token", status=401) + return self._tokens[token] + + def tenant_of(self, user_id: str) -> str | None: + user = self.users.get(user_id) + return user.tenant_id if user else None + + # -- enforcement path (what actors use) ------------------------------ + + def request(self, token: str, op: str, **args: Any) -> Any: + """The single entry point actors go through. Authorization is enforced here.""" + if self.latency_seconds: + time.sleep(self.latency_seconds) + user_id = self._whoami(token) + handler = getattr(self, f"_op_{op}", None) + if handler is None: + raise Denied(f"unknown operation {op!r}", status=404) + return handler(user_id, **args) + + def _op_create_resource(self, user_id: str, resource_id: str, content: str) -> dict: + self.resources[resource_id] = {"owner": user_id, "content": content} + self._audit("create", user_id, resource_id=resource_id) + return {self.response_id_field: resource_id} + + def _op_read_resource(self, user_id: str, resource_id: str) -> dict: + resource = self.resources.get(resource_id) + if resource is None: + raise Denied("no such resource", status=404) + if not self._may_read(user_id, resource_id): + raise Denied("not authorized to read", status=self.denied_status) + return {self.response_id_field: resource_id, "content": resource["content"]} + + def _op_write_resource(self, user_id: str, resource_id: str, content: str) -> dict: + resource = self.resources.get(resource_id) + if resource is None: + raise Denied("no such resource", status=404) + if not self._may_write(user_id, resource_id): + raise Denied("not authorized to write", status=self.denied_status) + resource["content"] = content + self._audit("write", user_id, resource_id=resource_id) + return {"written": True} + + def _op_grant( + self, + user_id: str, + resource_id: str, + subject_id: str, + permission: Permission | None = None, + ) -> dict: + resource = self.resources.get(resource_id) + if resource is None or resource["owner"] != user_id: + raise Denied("only the owner may grant") + if self.enforce_tenant_isolation: + if self.tenant_of(subject_id) != self.tenant_of(user_id): + raise Denied("cross-tenant sharing is not permitted") + effective = ( + self.grant_permission_for(permission) + if self.grant_permission_for + else (permission or "READ") + ) + self.grants[(resource_id, subject_id)] = effective + if self.require_share_acceptance: + self.resources[resource_id].setdefault("pending", set()).add(subject_id) + self._audit( + "grant", user_id, resource_id=resource_id, + subject_id=subject_id, permission=effective, + ) + return {"granted": effective} + + def _op_accept_share(self, user_id: str, resource_id: str) -> dict: + pending = self.resources.get(resource_id, {}).get("pending") + if pending: + pending.discard(user_id) + self._audit("accept", user_id, resource_id=resource_id, subject_id=user_id) + return {"accepted": True} + + def _op_revoke(self, user_id: str, resource_id: str, subject_id: str) -> dict: + resource = self.resources.get(resource_id) + if resource is None or resource["owner"] != user_id: + raise Denied("only the owner may revoke") + if self.revoke_delay_seconds: + self.resources[resource_id].setdefault("revoke_after", {})[subject_id] = ( + time.monotonic() + self.revoke_delay_seconds + ) + else: + self.grants.pop((resource_id, subject_id), None) + if self.audit_revoke: + self._audit("revoke", user_id, resource_id=resource_id, subject_id=subject_id) + return {"revoked": True} + + def _may_read(self, user_id: str, resource_id: str) -> bool: + """Authorization as the system actually enforces it.""" + if self.may_read is not None: + return self.may_read(self, user_id, resource_id) + return self.baseline_may_read(user_id, resource_id) + + def _may_write(self, user_id: str, resource_id: str) -> bool: + resource = self.resources.get(resource_id) + if resource is None: + return False + if resource["owner"] == user_id: + return True + return self.grants.get((resource_id, user_id)) == "WRITE" + + def baseline_may_read(self, user_id: str, resource_id: str) -> bool: + resource = self.resources.get(resource_id) + if resource is None: + return False + if resource["owner"] == user_id: + return True + deadline = resource.get("revoke_after", {}).get(user_id) + if deadline is not None and time.monotonic() < deadline: + return True # revocation is scheduled but not yet effective + if user_id in resource.get("pending", set()): + return False + return (resource_id, user_id) in self.grants + + +class ObservationChannel: + """Independent read access to lab state — decision D-07. + + Bypasses authorization deliberately. This is the channel test-driver requires + of any system under test, and the main integration burden the framework + imposes on an adopter. + """ + + def __init__(self, app: LabApp) -> None: + self._app = app + + @property + def version(self) -> str: + return self._app.version + + def state_permission(self, user_id: str, resource_id: str) -> str | None: + """What the stored record says, independent of any enforcement decision.""" + resource = self._app.resources.get(resource_id) + if resource is None: + return None + if resource["owner"] == user_id: + return "OWNER" + return self._app.grants.get((resource_id, user_id)) + + def probe_read(self, user_id: str, resource_id: str) -> bool: + """Exercise the enforcement path out-of-band and report what it did. + + This uses the subject's own credentials, which can look like a violation + of actor isolation but is not: independence means the *actor's report* is + never the evidence. The observer issues its own request and records the + raw outcome. No actor is ever asked whether it succeeded. + """ + user = self._app.users.get(user_id) + if user is None: + return False + try: + self._app.request(user.token, "read_resource", resource_id=resource_id) + except Denied: + return False + return True + + def probe_write(self, user_id: str, resource_id: str) -> bool: + """Out-of-band probe of the write path, non-destructive on refusal.""" + user = self._app.users.get(user_id) + if user is None: + return False + original = self._app.resources.get(resource_id, {}).get("content") + try: + self._app.request( + user.token, "write_resource", resource_id=resource_id, content=original + ) + except Denied: + return False + return True + + def audit_events(self, resource_id: str) -> list[dict[str, Any]]: + return [ + { + "sequence": r.sequence, + "event": r.event, + "actor_id": r.actor_id, + "subject_id": r.subject_id, + "permission": r.permission, + } + for r in self._app.audit + if r.resource_id == resource_id + ] + + +def build_baseline() -> tuple[LabApp, dict[str, str]]: + """Known initial state, so that runs replay from the same starting point.""" + app = LabApp() + tokens = {u: app.add_user(u) for u in ("alice", "bob", "carol")} + tokens["mallory"] = app.add_user("mallory", tenant_id="t-other") + return app, tokens diff --git a/lab/http_api.py b/lab/http_api.py new file mode 100644 index 0000000..f9f1bbf --- /dev/null +++ b/lab/http_api.py @@ -0,0 +1,215 @@ +"""HTTP surface for the lab: a JSON API and a minimal browser UI. + +Stdlib only. The point of this layer is to give the agentic browser driver (T07) +something real to navigate, and to give the mechanical mutations somewhere to +bite — a mutation that moves a control is inert until there is a control to move. + +The domain is not reimplemented here. Every request funnels into +`LabApp.request`, so the UI and API cannot drift from the enforcement path that +the deterministic driver and the observation channel already exercise. +""" + +from __future__ import annotations + +import json +import re +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from typing import Any +from urllib.parse import parse_qs, urlparse + +from .app import Denied, LabApp + + +def _resource_path(app: LabApp, resource_id: str) -> str: + return ( + f"/r/{resource_id}" if app.api_path_style == "short" + else f"/resources/{resource_id}" + ) + + +def render_resource_page(app: LabApp, user_id: str, resource_id: str) -> str: + """The browser surface. Every mechanical UI mutation shows up here.""" + html = _render(app, user_id, resource_id) + if app.ui_test_ids == "dropped": + # A rewrite that does not carry the old test ids forward. See the + # "test-id axis" note in lab/mutations.py — this is the case that + # decides whether semantic actions earn their keep. + html = _TEST_ID_ATTR.sub("", html) + return html + + +_TEST_ID_ATTR = re.compile(r'\s*data-td="[^"]*"') + + +def _render(app: LabApp, user_id: str, resource_id: str) -> str: + verbose = app.ui_labels == "verbose" + share_label = "Give access" if verbose else "Share" + revoke_label = "Withdraw access" if verbose else "Revoke" + + tag = "a" if app.ui_button_element == "anchor" else "button" + role = ' role="button"' if tag == "a" else "" + href = ' href="#"' if tag == "a" else "" + + subject_field = ( + '' + '' + ) + permission_field = ( + '' + '' + ) + fields = ( + permission_field + subject_field + if app.ui_field_order == "reversed" + else subject_field + permission_field + ) + + share_form = ( + f'
' + f"{fields}" + f'<{tag}{href}{role} id="share-submit" data-td="share-submit">' + f"{share_label}
" + ) + if app.ui_share_control == "modal": + share_form = ( + f'<{tag}{href}{role} id="open-share" data-td="open-share">{share_label}…' + f"" + f'{share_form}' + ) + + confirm = ( + '

This cannot be undone.

' + f'<{tag}{href}{role} data-td="revoke-confirm">Yes, {revoke_label.lower()}' + if app.ui_confirm_revoke else "" + ) + revoke_form = ( + f'
' + f'' + f'<{tag}{href}{role} data-td="revoke-submit">{revoke_label}' + f"{confirm}
" + ) + + grants = "".join( + f'
  • {sid}: {perm}
    • ' + for (rid, sid), perm in sorted(app.grants.items()) + if rid == resource_id + ) + + body = ( + f'

    Resource {resource_id}

    ' + f'{share_form}{revoke_form}' + ) + if app.ui_dom_style == "nested": + body = ( + '
    ' + f"{body}
    " + ) + + return ( + "Lab" + f'{body}' + ) + + +class LabHandler(BaseHTTPRequestHandler): + app: LabApp + + def log_message(self, *args: Any) -> None: # keep the test output quiet + pass + + # -- helpers --------------------------------------------------------- + + def _token(self) -> str: + header = self.headers.get("Authorization", "") + return header.removeprefix("Bearer ").strip() + + def _send(self, status: int, payload: Any, content_type: str = "application/json") -> None: + body = ( + json.dumps(payload).encode() + if content_type == "application/json" + else payload.encode() + ) + self.send_response(status) + self.send_header("Content-Type", content_type) + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def _dispatch(self, op: str, **args: Any) -> None: + try: + self._send(200, self.app.request(self._token(), op, **args)) + except Denied as denied: + self._send(denied.status, {"error": str(denied)}) + + def _resource_id(self, path: str) -> str | None: + parts = [p for p in path.split("/") if p] + if parts and parts[0] in ("resources", "r") and len(parts) >= 2: + return parts[1] + return None + + # -- routes ---------------------------------------------------------- + + def do_GET(self) -> None: # noqa: N802 — stdlib naming + url = urlparse(self.path) + resource_id = self._resource_id(url.path) + if resource_id is None: + self._send(404, {"error": "not found"}) + return + if url.path.endswith("/view"): + try: + user_id = self.app._whoami(self._token()) + except Denied as denied: + self._send(denied.status, {"error": str(denied)}) + return + self._send(200, render_resource_page(self.app, user_id, resource_id), "text/html") + return + self._dispatch("read_resource", resource_id=resource_id) + + def do_POST(self) -> None: # noqa: N802 + url = urlparse(self.path) + resource_id = self._resource_id(url.path) + length = int(self.headers.get("Content-Length", 0)) + raw = self.rfile.read(length).decode() if length else "" + try: + body = json.loads(raw) if raw.startswith("{") else { + k: v[0] for k, v in parse_qs(raw).items() + } + except json.JSONDecodeError: + self._send(400, {"error": "malformed body"}) + return + + if resource_id is None: + if url.path.rstrip("/") in ("/resources", "/r"): + self._dispatch("create_resource", **body) + return + self._send(404, {"error": "not found"}) + return + + if url.path.endswith("/grant"): + self._dispatch("grant", resource_id=resource_id, **body) + elif url.path.endswith("/revoke"): + self._dispatch("revoke", resource_id=resource_id, **body) + elif url.path.endswith("/accept"): + self._dispatch("accept_share", resource_id=resource_id) + else: + self._send(404, {"error": "not found"}) + + +def serve(app: LabApp, port: int = 0) -> ThreadingHTTPServer: + """Start a server on `port` (0 picks a free one). Caller owns shutdown.""" + handler = type("BoundLabHandler", (LabHandler,), {"app": app}) + return ThreadingHTTPServer(("127.0.0.1", port), handler) + + +if __name__ == "__main__": # pragma: no cover - manual use + import sys + from .mutations import build_lab + + lab, tokens = build_lab(*sys.argv[1:]) + lab.request(tokens["alice"], "create_resource", resource_id="R", content="the secret") + server = serve(lab, 8099) + print(f"{lab.version} on http://127.0.0.1:8099 tokens={tokens}") + server.serve_forever() diff --git a/lab/minimal.py b/lab/minimal.py deleted file mode 100644 index 9864964..0000000 --- a/lab/minimal.py +++ /dev/null @@ -1,191 +0,0 @@ -"""Minimal lab: the seed of the system under test. - -Deliberately small. Users own resources, share them with read or write -permission, and revoke that sharing. Every state change is audited. - -Two access paths exist, and the distinction is the whole point: - -* the **enforcement path** (`request`) is what an actor uses. It checks - authorization and can therefore be *wrong* — that is where a seeded - authorization defect lives. -* the **observation channel** (`ObservationChannel`) reads stored state directly, - without authorization. It is the independent channel required by decision - D-07. - -An oracle that consulted only stored state would verify the framework's own -reimplementation of the rules rather than the system's enforcement of them. An -oracle that consulted only the enforcement path would have no way to notice that -enforcement and record disagree. test-driver observes both, and treats -disagreement between them as meaningful in its own right — that disagreement is -the precise signature of the M05 authorization defect. - -TD-WP-0002-T05 grows this into the full lab with an HTTP API, a browser UI and -the labelled mutation catalogue. It is kept in-process here so that T04 can prove -the kernel without dragging in a web stack. -""" - -from __future__ import annotations - -from dataclasses import dataclass, field -from typing import Any, Literal - -Permission = Literal["READ", "WRITE"] - - -class Denied(Exception): - """The enforcement path refused the request.""" - - -@dataclass(slots=True) -class AuditRecord: - sequence: int - event: str - actor_id: str - resource_id: str | None = None - subject_id: str | None = None - permission: str | None = None - - -@dataclass(slots=True) -class MinimalLab: - """In-process resource-sharing service.""" - - version: str = "lab-0.1.0-baseline" - users: dict[str, str] = field(default_factory=dict) # user_id -> token - _tokens: dict[str, str] = field(default_factory=dict) # token -> user_id - resources: dict[str, dict[str, Any]] = field(default_factory=dict) - grants: dict[tuple[str, str], Permission] = field(default_factory=dict) - audit: list[AuditRecord] = field(default_factory=list) - _seq: int = 0 - - # -- setup ----------------------------------------------------------- - - def add_user(self, user_id: str) -> str: - token = f"tok-{user_id}" - self.users[user_id] = token - self._tokens[token] = user_id - return token - - def _audit(self, event: str, actor_id: str, **kw: Any) -> None: - self._seq += 1 - self.audit.append(AuditRecord(self._seq, event, actor_id, **kw)) - - def _whoami(self, token: str) -> str: - if token not in self._tokens: - raise Denied("unknown token") - return self._tokens[token] - - # -- enforcement path (what actors use) ------------------------------ - - def request(self, token: str, op: str, **args: Any) -> Any: - """The single entry point actors go through. Authorization is enforced here.""" - user_id = self._whoami(token) - handler = getattr(self, f"_op_{op}", None) - if handler is None: - raise Denied(f"unknown operation {op!r}") - return handler(user_id, **args) - - def _op_create_resource(self, user_id: str, resource_id: str, content: str) -> dict: - self.resources[resource_id] = {"owner": user_id, "content": content} - self._audit("create", user_id, resource_id=resource_id) - return {"resource_id": resource_id} - - def _op_read_resource(self, user_id: str, resource_id: str) -> dict: - resource = self.resources.get(resource_id) - if resource is None: - raise Denied("no such resource") - if not self._may_read(user_id, resource_id): - raise Denied("not authorized to read") - return {"resource_id": resource_id, "content": resource["content"]} - - def _op_grant( - self, user_id: str, resource_id: str, subject_id: str, permission: Permission - ) -> dict: - resource = self.resources.get(resource_id) - if resource is None or resource["owner"] != user_id: - raise Denied("only the owner may grant") - self.grants[(resource_id, subject_id)] = permission - self._audit( - "grant", user_id, resource_id=resource_id, - subject_id=subject_id, permission=permission, - ) - return {"granted": permission} - - def _op_revoke(self, user_id: str, resource_id: str, subject_id: str) -> dict: - resource = self.resources.get(resource_id) - if resource is None or resource["owner"] != user_id: - raise Denied("only the owner may revoke") - self.grants.pop((resource_id, subject_id), None) - self._audit("revoke", user_id, resource_id=resource_id, subject_id=subject_id) - return {"revoked": True} - - def _may_read(self, user_id: str, resource_id: str) -> bool: - """The authorization rule as the system actually enforces it.""" - resource = self.resources.get(resource_id) - if resource is None: - return False - if resource["owner"] == user_id: - return True - return (resource_id, user_id) in self.grants - - -class ObservationChannel: - """Independent read access to lab state — decision D-07. - - Bypasses authorization deliberately. This is the channel test-driver requires - of any system under test, and the main integration burden the framework - imposes on an adopter. - """ - - def __init__(self, lab: MinimalLab) -> None: - self._lab = lab - - @property - def version(self) -> str: - return self._lab.version - - def state_permission(self, user_id: str, resource_id: str) -> str | None: - """What the stored record says, independent of any enforcement decision.""" - resource = self._lab.resources.get(resource_id) - if resource is None: - return None - if resource["owner"] == user_id: - return "OWNER" - return self._lab.grants.get((resource_id, user_id)) - - def probe_read(self, user_id: str, resource_id: str) -> bool: - """Exercise the enforcement path out-of-band and report what it did. - - This uses the subject's own credentials, which can look like a violation - of actor isolation but is not: independence means the *actor's report* is - never the evidence. The observer issues its own request and records the - raw outcome. No actor is ever asked whether it succeeded. - """ - token = self._lab.users.get(user_id) - if token is None: - return False - try: - self._lab.request(token, "read_resource", resource_id=resource_id) - except Denied: - return False - return True - - def audit_events(self, resource_id: str) -> list[dict[str, Any]]: - return [ - { - "sequence": r.sequence, - "event": r.event, - "actor_id": r.actor_id, - "subject_id": r.subject_id, - "permission": r.permission, - } - for r in self._lab.audit - if r.resource_id == resource_id - ] - - -def build_baseline() -> tuple[MinimalLab, dict[str, str]]: - """Known initial state, so that runs replay from the same starting point.""" - lab = MinimalLab() - tokens = {user: lab.add_user(user) for user in ("alice", "bob", "carol")} - return lab, tokens diff --git a/lab/mutations.py b/lab/mutations.py new file mode 100644 index 0000000..9a2c897 --- /dev/null +++ b/lab/mutations.py @@ -0,0 +1,205 @@ +"""The labelled mutation catalogue — the project's measuring instrument. + +Every claim test-driver makes is measured against this catalogue, so its quality +caps the credibility of every downstream result. Six mutations, as the milestones +document originally sketched, cannot support any statement about precision or +recall; there are twenty here. + +Each mutation carries a **ground-truth label**, decided by a human from the use +case and recorded before any run: + + MECHANICAL the surface changed; protected semantics are identical. + test-driver should recover and report an adaptation. + SEMANTIC intended behaviour genuinely changed. test-driver must escalate + to a human and must never rewrite a claim by itself. + DEFECT the system violates unchanged intent. test-driver must report a + Product Finding and must never adapt to it. + +The distinction between SEMANTIC and DEFECT is deliberately *not* inferable from +the code — both change behaviour. It is a statement about intent, which is why +claims must have independent provenance (D-06) and why ambiguity escalates +rather than resolving itself. + +Mutations compose: `build_lab("M01", "M15")` applies both and records both in the +version string, so a mechanical change shipping alongside a defect is +reproducible. That combination is decision-table row 3 and the case a naive +self-healing tool gets wrong. + +## The test-id axis + +The UI carries stable `data-td` attributes, as a well-instrumented application +would. Most mechanical mutations preserve them, and a recorded selector sequence +keyed on those attributes survives such a mutation untouched — which would make +H-001 (semantic actions outlast recorded sequences) trivially *false*. + +That is not a flaw to be rigged away. It is the honest shape of the question: +**a semantic action earns its keep exactly when stable identifiers are absent or +not carried forward.** Mutations therefore vary along `preserves_test_ids`, and +H-001 must be analysed split by that axis rather than as a single rate. A +catalogue whose mutations all broke naive selectors would flatter the thesis and +tell us nothing. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Callable, Literal + +from .app import BASE_VERSION, LabApp, ObservationChannel, build_baseline + +__all__ = [ + "CATALOGUE", "BY_ID", "Mutation", "ObservationChannel", "build_lab", + "expected_classification", +] + +Label = Literal["MECHANICAL", "SEMANTIC", "DEFECT"] +Layer = Literal["ui", "api", "domain"] + + +@dataclass(frozen=True, slots=True) +class Mutation: + id: str + title: str + label: Label + layer: Layer + rationale: str + apply: Callable[[LabApp], None] + preserves_test_ids: bool = True + + +def _m(app: LabApp, **flags) -> None: + for key, value in flags.items(): + setattr(app, key, value) + + +# --- MECHANICAL ----------------------------------------------------------- +# The surface moves. What the system means does not. + +def _m01(app): _m(app, ui_share_control="modal") +def _m02(app): _m(app, ui_dom_style="nested", ui_test_ids="dropped") +def _m03(app): _m(app, response_id_field="id") +def _m04(app): _m(app, ui_labels="verbose") +def _m05(app): _m(app, ui_field_order="reversed") +def _m06(app): _m(app, api_path_style="short") +def _m07(app): _m(app, ui_button_element="anchor") +def _m08(app): _m(app, ui_confirm_revoke=True) +def _m09(app): _m(app, latency_seconds=0.05) +def _m10(app): _m(app, denied_status=401) + + +# --- SEMANTIC ------------------------------------------------------------- +# Intended behaviour changed. A human must decide; test-driver must not. + +def _m11(app): _m(app, require_share_acceptance=True) +def _m12(app): _m(app, revoke_delay_seconds=3600.0) +def _m13(app): _m(app, grant_permission_for=lambda p: p or "WRITE") +def _m14(app): _m(app, enforce_tenant_isolation=True, tenant_sharing_announced=True) + + +# --- DEFECT --------------------------------------------------------------- +# Unchanged intent, violated. + +def _revoke_is_cosmetic(app: LabApp, user_id: str, resource_id: str) -> bool: + """Record and audit say revoked; enforcement still allows the read.""" + resource = app.resources.get(resource_id) + if resource is None: + return False + if resource["owner"] == user_id: + return True + return any( + r.event == "grant" and r.subject_id == user_id and r.resource_id == resource_id + for r in app.audit + ) + + +def _read_permits_everyone(app: LabApp, user_id: str, resource_id: str) -> bool: + return resource_id in app.resources + + +def _ignores_tenant(app: LabApp, user_id: str, resource_id: str) -> bool: + return app.baseline_may_read(user_id, resource_id) or resource_id in app.resources + + +def _m15(app): _m(app, may_read=_revoke_is_cosmetic) +def _m16(app): _m(app, grant_permission_for=lambda p: "WRITE") +def _m17(app): _m(app, may_read=_read_permits_everyone) +def _m18(app): _m(app, audit_revoke=False) +def _m19(app): _m(app, revoke_delay_seconds=2.0) +def _m20(app): _m(app, enforce_tenant_isolation=False, may_read=_ignores_tenant) + + +CATALOGUE: tuple[Mutation, ...] = ( + Mutation("M01", "Sharing control moves into a modal", "MECHANICAL", "ui", + "Same action, different place. The canonical mechanical change.", _m01), + Mutation("M02", "DOM rewritten, test ids not carried forward", "MECHANICAL", "ui", + "Selectors break; nothing the user can do changes. Unlike M01 this " + "drops the stable test ids, which is what a real rewrite usually " + "does.", _m02, preserves_test_ids=False), + Mutation("M03", "API renames resource_id to id", "MECHANICAL", "api", + "Compatible representation change of the same field.", _m03), + Mutation("M04", "UI labels reworded", "MECHANICAL", "ui", + "'Share' becomes 'Give access'. Wording, not meaning.", _m04), + Mutation("M05", "Form field order reversed", "MECHANICAL", "ui", + "Permission precedes subject. Same form, same result.", _m05), + Mutation("M06", "API paths shortened", "MECHANICAL", "api", + "/resources/{id} becomes /r/{id}.", _m06), + Mutation("M07", "Buttons become anchors", "MECHANICAL", "ui", + "Element type changes; affordance does not.", _m07), + Mutation("M08", "Revoke gains a confirmation step", "MECHANICAL", "ui", + "An extra legitimate click. The postcondition is unchanged, so this " + "is mechanical — but it is the closest mechanical case to a semantic " + "one, and a good test of where the boundary really sits.", _m08), + Mutation("M09", "Responses are slower", "MECHANICAL", "api", + "Latency only. Tests patience, not semantics.", _m09), + Mutation("M10", "Denials return 401 instead of 403", "MECHANICAL", "api", + "Both mean refused. A driver keying on the exact code breaks.", _m10), + + Mutation("M11", "A share must be accepted before it takes effect", "SEMANTIC", "domain", + "Bob genuinely cannot read until he accepts. The old claim 'Bob can " + "read after the grant' is now wrong, and only a human may say so.", _m11), + Mutation("M12", "Revocation is deferred, not immediate", "SEMANTIC", "domain", + "A deliberate product decision that revocation batches. Looks exactly " + "like M19 from the outside; the difference is intent.", _m12), + Mutation("M13", "Grants default to WRITE", "SEMANTIC", "domain", + "Only affects grants that omit a permission. Inert for the reference " + "scenario, which passes READ explicitly — recorded as such rather " + "than quietly dropped.", _m13), + Mutation("M14", "Cross-tenant sharing is announced as prohibited", "SEMANTIC", "domain", + "Codifies existing enforcement as intended behaviour. A change of " + "intent with no change of code — the case that shows classification " + "cannot be a diff.", _m14), + + Mutation("M15", "Revocation updates the record but not enforcement", "DEFECT", "domain", + "Audit trail and stored grant both say revoked; the read still " + "succeeds. The defect that looks correct to an operator.", _m15), + Mutation("M16", "A READ grant confers WRITE", "DEFECT", "domain", + "Privilege escalation through the ordinary sharing path.", _m16), + Mutation("M17", "Any authenticated user can read any resource", "DEFECT", "domain", + "Authorization effectively removed. Carol reads R.", _m17), + Mutation("M18", "Revocation is not audited", "DEFECT", "domain", + "Enforcement is correct; the evidence trail is not. Detectable only " + "because audit is observed, not assumed.", _m18), + Mutation("M19", "Revocation propagates after a delay", "DEFECT", "domain", + "Not a decision — a race. Indistinguishable from M12 by behaviour " + "alone, which is exactly the point.", _m19), + Mutation("M20", "Tenant isolation leaks", "DEFECT", "domain", + "A user of another tenant reads the resource.", _m20), +) + +BY_ID: dict[str, Mutation] = {m.id: m for m in CATALOGUE} + + +def expected_classification(mutation_id: str) -> Label: + """Ground truth. Recorded by a human before any run — never inferred.""" + return BY_ID[mutation_id].label + + +def build_lab(*mutation_ids: str) -> tuple[LabApp, dict[str, str]]: + """Build a lab with the named mutations applied, versioned by what it carries.""" + app, tokens = build_baseline() + for mutation_id in mutation_ids: + BY_ID[mutation_id].apply(app) + app.applied_mutations = tuple(mutation_ids) + suffix = "+".join(mutation_ids) if mutation_ids else "baseline" + app.version = f"{BASE_VERSION}-{suffix}" + return app, tokens diff --git a/research/concepts/fitness-map.md b/research/concepts/fitness-map.md index ea72475..88151cf 100644 --- a/research/concepts/fitness-map.md +++ b/research/concepts/fitness-map.md @@ -1,6 +1,6 @@ # Concept ↔ Implementation Fitness Map -**Updated:** 2026-08-22 (TD-WP-0002-T04) +**Updated:** 2026-08-22 (TD-WP-0002-T05) Traces each important concept to the implementation, experiment and evidence that support it. **Unsupported entries are the point of this map** — a concept with no @@ -29,7 +29,7 @@ were aspirational, not evidenced. | `C-semantic-action` | C1 | `actions.py` | E-001 | — | Does identity survive restructuring better than a recorded sequence? (H-001) | | `C-oracle-independence` | C1 | `runner.py`, `oracles.py` | E-001, E-003 | — | Independence of components ≠ independence of belief. (H-004) | | `C-evidence-pack` | C1 | `evidence.py` | — | — | What is the minimum sufficient for replay? | -| `C-observation-channel` | C1 | `lab/minimal.py` | — | — | **D-07** — required of every system under test. Adoption cost unknown. | +| `C-observation-channel` | C1 | `lab/app.py` | — | — | **D-07** — required of every system under test. Adoption cost unknown. | | `C-adaptation` | C1 | — (T08) | E-001 | — | (H-002) | | `C-classification` | C1 | — (T08) | E-001, E-003 | — | Decision table is total on paper; unexercised. | | `C-crystallization` | C1 | — (T09) | E-002 | — | (H-003) | @@ -41,7 +41,7 @@ were aspirational, not evidenced. | `C-campaign` | C0 | — | — | — | Deferred. | | `C-metabolism` | C0 | — | — | — | Deferred. Depends on C-energy. | | `C-retirement` | C0 | — | — | — | Deferred. Depends on C-energy. | -| `C-security-mutation` | C1 | — | E-003 | — | Catalogue exists; no derivation mechanism. | +| `C-security-mutation` | C1 | `lab/mutations.py` | E-003 | `lab/GROUND-TRUTH.md` | Catalogue is hand-written; no derivation mechanism from use cases yet. | ## Orphan check diff --git a/research/findings/F-0002-scenario-coverage-gap.md b/research/findings/F-0002-scenario-coverage-gap.md new file mode 100644 index 0000000..fd972e5 --- /dev/null +++ b/research/findings/F-0002-scenario-coverage-gap.md @@ -0,0 +1,74 @@ +--- +id: F-0002 +type: framework-finding +class: FRAMEWORK_LIMITATION +status: resolved +discovered: "2026-08-22" +resolved: "2026-08-22" +discovered_by: TD-WP-0002-T05 +workplan: TD-WP-0002 +task: TD-WP-0002-T05 +--- + +# F-0002 — Two seeded defects were invisible to the reference scenario + +## Observation + +On first running the reference scenario against the twenty labelled mutations, +**six of six DEFECT mutations should have failed; only four did.** + +| Mutation | Defect | Verdict before | Why it escaped | +|---|---|---|---| +| M16 | A READ grant confers WRITE | `PASS` | No claim mentioned writing. The scenario never attempted one. | +| M18 | Revocation is not audited | `PASS` | `i-audit-append-only` checks *ordering*, not *completeness*. A trail missing an entry is still ordered. | + +Both passed cleanly. Nothing was flaky, nothing was ambiguous, and no oracle +reported `INCONCLUSIVE` — the framework simply had nothing to say, confidently. + +## Why it matters + +This is the failure mode most likely to be mistaken for success. A green run +against a lab carrying a seeded privilege escalation looks exactly like a green +run against a correct system. Had the catalogue been the six mutations the +milestones document originally sketched, this would not have surfaced at all — +which is the concrete argument for the larger catalogue, now made from evidence +rather than from assertion. + +It also sharpens what a verification asset is: **a use case protects exactly what +it asserts, and not one thing more.** Coverage is a property of the claim set, not +of the framework. No amount of adaptation, crystallization or energy scoring +compensates for an assertion nobody wrote. + +## Resolution + +Path 1 — the implementation changes to match the concept. + +Two claims were added to the reference use case, both `Provenance.HUMAN` and both +derivable from `INTENT.md` rather than from watching the lab: + +- `c-bob-cannot-write` — "A READ grant does not let Bob write R". + `INTENT.md` § Security by Use-Case Mutation already derives this exact question + from the reference use case: *"Can Bob write when only read permission was + granted?"* It was always part of what sharing means; it had simply never been + written down as an assertion. +- `c-revoke-audited` — "Revocation is recorded in the audit trail". Enforcement + being correct is not sufficient: an access change nobody can later evidence is + a compliance failure even when the access itself is right. + +Supporting changes: the lab gained a write path and the observation channel a +non-destructive `probe_write`. + +Both defects are now detected. DEFECT detection is 6/6, and +`test_every_defect_is_detected` fails the suite if that ever regresses. + +## Note on provenance + +These claims were added *after* observing that mutations escaped, which is +uncomfortably close to fitting assertions to the lab. They are admissible because +both were already stated as intent in `INTENT.md` before any lab existed — the +finding revealed a transcription gap, not a new requirement. Had the intended +behaviour not already been on record, the correct resolution would have been to +escalate to a human, not to write the claim. + +That distinction is exactly what `Provenance` exists to make checkable, and this +finding is the first case where it did real work. diff --git a/scenarios/__pycache__/alice_bob_carol.cpython-312.pyc b/scenarios/__pycache__/alice_bob_carol.cpython-312.pyc index a6b11c1..9053e9f 100644 Binary files a/scenarios/__pycache__/alice_bob_carol.cpython-312.pyc and b/scenarios/__pycache__/alice_bob_carol.cpython-312.pyc differ diff --git a/scenarios/alice_bob_carol.py b/scenarios/alice_bob_carol.py index 4d478d6..70bcfa7 100644 --- a/scenarios/alice_bob_carol.py +++ b/scenarios/alice_bob_carol.py @@ -19,7 +19,7 @@ from testdriver import ( Scenario, SemanticAction, StateObserver, Step, UseCase, VerificationAsset, Watch, World, ) -from lab.minimal import ObservationChannel, build_baseline +from lab.mutations import ObservationChannel, build_lab RESOURCE = "R" API = frozenset({"api"}) @@ -40,6 +40,25 @@ def _carol_cannot_read(obs) -> bool: return obs["probe_read:carol:R"] is False +def _bob_cannot_write(obs) -> bool: + """A READ grant must not confer write. + + INTENT.md derives exactly this question from the reference use case: + "Can Bob write when only read permission was granted?" It is part of what + sharing means, not an extra. + """ + return obs["probe_write:bob:R"] is False + + +def _revocation_is_audited(obs) -> bool: + """Revocation must leave a trace. + + Enforcement being correct is not sufficient. An access change nobody can + later evidence is a compliance failure even when the access itself is right. + """ + return any(event["event"] == "revoke" for event in obs["audit:R"]) + + def _enforcement_matches_record(obs) -> bool: """Enforcement and stored record must agree about every watched subject. @@ -77,8 +96,13 @@ USE_CASE = UseCase( Provenance.HUMAN, _bob_can_read, after_step="s2-grant"), Claim("c-carol-denied", "Carol can never read R", Provenance.HUMAN, _carol_cannot_read, after_step="s2-grant"), + Claim("c-bob-cannot-write", "A READ grant does not let Bob write R", + Provenance.HUMAN, _bob_cannot_write, after_step="s2-grant", + source_ref="INTENT.md#security-by-use-case-mutation"), Claim("c-bob-revoked", "Bob cannot read R after revocation", Provenance.HUMAN, _bob_cannot_read, after_step="s3-revoke"), + Claim("c-revoke-audited", "Revocation is recorded in the audit trail", + Provenance.HUMAN, _revocation_is_audited, after_step="s3-revoke"), ), invariants=( Invariant("i-enforcement-matches-record", @@ -90,9 +114,15 @@ USE_CASE = UseCase( ) -def build(variant: str = "baseline"): - """Assemble world, driver, observer and asset from a known initial state.""" - lab, tokens = build_baseline() +def build(*mutations: str): + """Assemble world, driver, observer and asset from a known initial state. + + `mutations` names entries from the lab catalogue. The same scenario runs + unchanged against every lab version — that is the point: the use case does + not know the implementation moved. + """ + lab, tokens = build_lab(*mutations) + variant = "+".join(mutations) if mutations else "baseline" cast = Cast() for name in ("alice", "bob", "carol"): cast.add(Actor(id=name, display_name=name.title(), diff --git a/src/testdriver/__pycache__/observers.cpython-312.pyc b/src/testdriver/__pycache__/observers.cpython-312.pyc index 193b6e8..033be7d 100644 Binary files a/src/testdriver/__pycache__/observers.cpython-312.pyc and b/src/testdriver/__pycache__/observers.cpython-312.pyc differ diff --git a/src/testdriver/observers.py b/src/testdriver/observers.py index 40de7bb..7783820 100644 --- a/src/testdriver/observers.py +++ b/src/testdriver/observers.py @@ -41,6 +41,9 @@ class StateObserver: out[f"state_permission:{key}"] = self.channel.state_permission( watch.subject_id, watch.resource_id ) + out[f"probe_write:{key}"] = self.channel.probe_write( + watch.subject_id, watch.resource_id + ) resources.add(watch.resource_id) for resource_id in sorted(resources): out[f"audit:{resource_id}"] = self.channel.audit_events(resource_id) diff --git a/tests/__pycache__/test_kernel_guarantees.cpython-312-pytest-7.4.4.pyc b/tests/__pycache__/test_kernel_guarantees.cpython-312-pytest-7.4.4.pyc index cfc3f09..0fcf25b 100644 Binary files a/tests/__pycache__/test_kernel_guarantees.cpython-312-pytest-7.4.4.pyc and b/tests/__pycache__/test_kernel_guarantees.cpython-312-pytest-7.4.4.pyc differ diff --git a/tests/__pycache__/test_lab_ground_truth.cpython-312-pytest-7.4.4.pyc b/tests/__pycache__/test_lab_ground_truth.cpython-312-pytest-7.4.4.pyc new file mode 100644 index 0000000..e7f4304 Binary files /dev/null and b/tests/__pycache__/test_lab_ground_truth.cpython-312-pytest-7.4.4.pyc differ diff --git a/tests/__pycache__/test_reference_scenario.cpython-312-pytest-7.4.4.pyc b/tests/__pycache__/test_reference_scenario.cpython-312-pytest-7.4.4.pyc index dc34970..9f5843b 100644 Binary files a/tests/__pycache__/test_reference_scenario.cpython-312-pytest-7.4.4.pyc and b/tests/__pycache__/test_reference_scenario.cpython-312-pytest-7.4.4.pyc differ diff --git a/tests/test_kernel_guarantees.py b/tests/test_kernel_guarantees.py index 44a2547..d56280b 100644 --- a/tests/test_kernel_guarantees.py +++ b/tests/test_kernel_guarantees.py @@ -15,7 +15,7 @@ from testdriver import ( Claim, InadmissibleProvenance, Invariant, Oracle, Provenance, Runner, SemanticAction, StateObserver, SurfaceNotPermitted, Stratum, Verdict, ) -from lab.minimal import Denied, MinimalLab, ObservationChannel, build_baseline +from lab.app import Denied, LabApp, ObservationChannel, build_baseline from scenarios.alice_bob_carol import build @@ -83,7 +83,7 @@ def test_actors_cannot_be_recorded_as_judgment_collectors(): # --- preview of the M05 authorization defect (built properly in T05) -------- -class RevokeIsCosmetic(MinimalLab): +class RevokeIsCosmetic(LabApp): """Revocation updates the record and the audit trail but not enforcement. This is the shape of a real authorization defect: everything an operator @@ -101,6 +101,10 @@ class RevokeIsCosmetic(MinimalLab): for r in self.audit ) + # NOTE: this predates the mutation catalogue, where the same defect is M15. + # Kept as a direct subclass so the kernel test does not depend on the lab + # catalogue's wiring being correct. + def test_seeded_authorization_defect_fails_the_run(): """The kernel must report FAIL, not adapt, when revocation does not revoke.""" @@ -124,7 +128,8 @@ def test_seeded_authorization_defect_fails_the_run(): assert result.verdict is Verdict.FAIL assert result.judgment("c-bob-revoked").verdict is Verdict.FAIL # The claim set is untouched by the failure — there is no path to adapt it. - assert USE_CASE.claims[2].text == "Bob cannot read R after revocation" + by_id = {c.id: c for c in USE_CASE.claims} + assert by_id["c-bob-revoked"].text == "Bob cannot read R after revocation" def test_defect_run_emits_an_energy_event(): diff --git a/tests/test_lab_ground_truth.py b/tests/test_lab_ground_truth.py new file mode 100644 index 0000000..d2fcbc2 --- /dev/null +++ b/tests/test_lab_ground_truth.py @@ -0,0 +1,148 @@ +"""The lab is the measuring instrument. These tests keep it honest. + +Nothing here tests the kernel's cleverness — the kernel has no classifier yet. +What is established is the **ground truth** every later measurement is taken +against: which mutations the reference scenario responds to, and how. + +The response matrix below is an asserted fact, not a snapshot. If a change to the +lab or the scenario moves a cell, that is a change to the measuring instrument +and must be a deliberate, reviewed act. +""" + +from __future__ import annotations + +import pytest + +from testdriver import Runner, Verdict +from lab.mutations import BY_ID, CATALOGUE, build_lab, expected_classification +from scenarios.alice_bob_carol import build + + +def run_against(*mutations: str): + world, driver, observer, asset, oracle = build(*mutations) + return Runner(world, driver, observer, oracle).run(asset) + + +# --- the catalogue itself ------------------------------------------------- + + +def test_catalogue_is_large_enough_to_support_a_rate(): + """Six mutations cannot support precision or recall. Twenty can begin to.""" + assert len(CATALOGUE) >= 20 + + +def test_every_mutation_is_labelled_and_reasoned(): + for mutation in CATALOGUE: + assert mutation.label in ("MECHANICAL", "SEMANTIC", "DEFECT") + assert mutation.rationale.strip(), f"{mutation.id} has no rationale" + + +def test_labels_cover_all_three_classes_with_useful_weight(): + counts = {label: 0 for label in ("MECHANICAL", "SEMANTIC", "DEFECT")} + for mutation in CATALOGUE: + counts[mutation.label] += 1 + assert all(count >= 4 for count in counts.values()), counts + + +def test_every_mutation_is_reproducible_and_version_stamped(): + for mutation in CATALOGUE: + first, _ = build_lab(mutation.id) + second, _ = build_lab(mutation.id) + assert first.version == second.version == f"lab-0.2.0-{mutation.id}" + assert first.applied_mutations == (mutation.id,) + + +def test_mutations_compose_and_record_both(): + """Row 3 of the decision table needs a lab carrying both at once.""" + app, _ = build_lab("M01", "M15") + assert app.version == "lab-0.2.0-M01+M15" + assert app.applied_mutations == ("M01", "M15") + + +def test_test_id_axis_is_represented(): + """H-001 must be analysable split by whether stable selectors survived.""" + preserved = [m.id for m in CATALOGUE if m.preserves_test_ids] + dropped = [m.id for m in CATALOGUE if not m.preserves_test_ids] + assert preserved and dropped, "both sides of the test-id axis must exist" + + +# --- the response matrix -------------------------------------------------- + +# Ground truth: what the reference scenario reports for each lab version. +# `None` means "no assertion in this scenario covers this mutation" — recorded +# honestly rather than papered over. +EXPECTED_VERDICT = { + "M01": Verdict.PASS, "M02": Verdict.PASS, "M03": Verdict.PASS, + "M04": Verdict.PASS, "M05": Verdict.PASS, "M06": Verdict.PASS, + "M07": Verdict.PASS, "M08": Verdict.PASS, "M09": Verdict.PASS, + "M10": Verdict.PASS, + "M11": Verdict.FAIL, "M12": Verdict.FAIL, + "M13": Verdict.PASS, "M14": Verdict.PASS, + "M15": Verdict.FAIL, "M16": Verdict.FAIL, "M17": Verdict.FAIL, + "M18": Verdict.FAIL, "M19": Verdict.FAIL, "M20": Verdict.FAIL, +} + +# The two SEMANTIC mutations the reference scenario cannot see, and why. +KNOWN_INERT = { + "M13": "only affects grants that omit a permission; the scenario passes READ explicitly", + "M14": "a change of intent with no change of code; nothing observable moved", +} + + +def test_baseline_passes(): + assert run_against().verdict is Verdict.PASS + + +@pytest.mark.parametrize("mutation_id", sorted(EXPECTED_VERDICT)) +def test_response_matrix_is_stable(mutation_id): + assert run_against(mutation_id).verdict is EXPECTED_VERDICT[mutation_id] + + +def test_no_mechanical_mutation_changes_the_verdict(): + """Semantics are preserved, so the use case must not notice.""" + for mutation in CATALOGUE: + if mutation.label == "MECHANICAL": + assert run_against(mutation.id).verdict is Verdict.PASS, mutation.id + + +def test_every_defect_is_detected(): + """The floor of the whole project. A defect the lab cannot surface is a + defect no later classifier can be measured against.""" + missed = [ + m.id for m in CATALOGUE + if m.label == "DEFECT" and run_against(m.id).verdict is Verdict.PASS + ] + assert missed == [], f"undetected seeded defects: {missed}" + + +def test_inert_semantic_mutations_are_declared(): + """A mutation the scenario cannot see must be named, not silently ignored.""" + for mutation in CATALOGUE: + if mutation.label != "SEMANTIC": + continue + if run_against(mutation.id).verdict is Verdict.PASS: + assert mutation.id in KNOWN_INERT, ( + f"{mutation.id} is invisible to the reference scenario and " + "undeclared — either cover it or record why not" + ) + + +def test_deferred_revoke_and_revoke_race_are_behaviourally_identical(): + """M12 (SEMANTIC) and M19 (DEFECT) must be indistinguishable from evidence. + + This is the discrimination problem in its sharpest form, and the reason + classification cannot be a diff over observed behaviour. Both produce the + same failure; only intent separates them, which is why claims need + independent provenance and why ambiguity escalates to a human. + """ + semantic = run_against("M12") + defect = run_against("M19") + failed = lambda r: sorted({j.assertion_id for j in r.judgments + if j.verdict is not Verdict.PASS}) + assert failed(semantic) == failed(defect) == ["c-bob-revoked"] + assert expected_classification("M12") != expected_classification("M19") + + +def test_a_mechanical_change_shipping_with_a_defect_still_fails(): + """Decision-table row 3: coincidence is not exoneration.""" + assert run_against("M01", "M15").verdict is Verdict.FAIL diff --git a/tests/test_reference_scenario.py b/tests/test_reference_scenario.py index 090e807..5739969 100644 --- a/tests/test_reference_scenario.py +++ b/tests/test_reference_scenario.py @@ -10,8 +10,8 @@ from testdriver import Runner, Stratum, Verdict from scenarios.alice_bob_carol import build -def run_once(variant: str = "baseline"): - world, driver, observer, asset, oracle = build(variant) +def run_once(*mutations: str): + world, driver, observer, asset, oracle = build(*mutations) return Runner(world, driver, observer, oracle).run(asset), world @@ -52,7 +52,7 @@ def test_evidence_is_stratified_and_serializable(): assert pack.of_stratum(Stratum.JUDGMENT) parsed = json.loads(pack.to_json()) assert parsed["run_id"] == result.run_id - assert parsed["sut_version"] == "lab-0.1.0-baseline" + assert parsed["sut_version"] == "lab-0.2.0-baseline" def test_evidence_records_claim_provenance(): diff --git a/workplans/TD-WP-0002-vertical-spike-crystallization.md b/workplans/TD-WP-0002-vertical-spike-crystallization.md index c558ae1..9cc9215 100644 --- a/workplans/TD-WP-0002-vertical-spike-crystallization.md +++ b/workplans/TD-WP-0002-vertical-spike-crystallization.md @@ -222,7 +222,7 @@ Three things that came out of building it rather than designing it: ```task id: TD-WP-0002-T05 -status: todo +status: done priority: high state_hub_task_id: "595a89c2-1462-57fc-8b08-a5a6b875fd48" ``` @@ -240,6 +240,31 @@ The lab is the measuring instrument for every claim the framework makes — a we lab caps the credibility of all downstream results. It is also potentially the project's first publishable artefact in its own right. +**Done 2026-08-22.** `lab/app.py` (users, tenants, auth, resources, sharing, +read/write, revoke, audit), `lab/http_api.py` (JSON API + browser UI, stdlib +only), `lab/mutations.py` (20 labelled, composable, version-stamped mutations), +`lab/GROUND-TRUTH.md`. 48 tests pass. Detection: MECHANICAL 0/10 flagged, +DEFECT 6/6, SEMANTIC 2/4. + +Three results worth carrying: + +- **F-0002 — two seeded defects were initially invisible.** M16 (READ grant + confers WRITE) and M18 (revocation unaudited) both passed cleanly: nothing + flaky, nothing `INCONCLUSIVE`, the framework simply had nothing to say. A use + case protects exactly what it asserts and not one thing more. Resolved by + adding two claims already stated as intent in `INTENT.md`. Had the six-mutation + catalogue from the milestones doc been used, this would never have surfaced — + the argument for the larger catalogue is now evidenced rather than asserted. +- **The test-id axis.** Stable `data-td` selectors survive most UI mutations, + which would make H-001 trivially *false*. Rather than rig the catalogue, + mutations now vary on `preserves_test_ids`, and H-001 must be analysed split by + that axis. A semantic action earns its keep exactly where stable identifiers + are absent or not carried forward — that is the honest shape of the claim. +- **M12 vs M19 are behaviourally identical.** A deliberate deferred-revocation + decision and a revocation race produce the same failure, same step, same + evidence. Only intent separates them. This is the discrimination problem in its + sharpest form and is now a test, not a paragraph. + ## Out-of-band ground truth for self-verification ```task