T04: deterministic semantic kernel
Alice/Bob/Carol runs end to end, deterministically, replayable from seed. 16 tests pass, no third-party dependencies. - src/testdriver: intent, provenance, world, actions, drivers, observers, oracles, evidence, energy, scenario, runner - lab/minimal.py: the SUT, exposing the independent observation channel required by D-07 - evidence is stratified S1/S2/S3; Runner refuses to attribute S2/S3 to an actor; claims are frozen and provenance-checked at construction - missing evidence yields INCONCLUSIVE, which outranks PASS in the run verdict - EnergyEvents captured, no scoring (H-005 dormant) The observation channel records both stored state and an out-of-band enforcement probe; their disagreement is an invariant and is what detects an authorization defect that leaves the audit trail intact. A seeded RevokeIsCosmetic lab fails the run via both the claim and that invariant. Also closes TD-WP-0001-T02 (stack and commands now exist). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 1629012@bnt-lap001 Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
This commit is contained in:
parent
8da4c5bf7a
commit
04e9573b5a
42 changed files with 1533 additions and 20 deletions
31
AGENTS.md
31
AGENTS.md
|
|
@ -133,6 +133,37 @@ curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
|
||||||
{CREDENTIAL_ROUTING}
|
{CREDENTIAL_ROUTING}
|
||||||
|
|
||||||
<!-- REPO-AGENTS-EXTENSIONS -->
|
<!-- REPO-AGENTS-EXTENSIONS -->
|
||||||
|
|
||||||
|
## Stack and commands
|
||||||
|
|
||||||
|
Python ≥ 3.11, stdlib only; pytest for the suite. No package manager step is
|
||||||
|
needed — `pyproject.toml` puts `src/` and the repo root on `pythonpath`.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m pytest -q # run everything
|
||||||
|
python3 -m pytest -q -k oracle # narrow
|
||||||
|
```
|
||||||
|
|
||||||
|
Deliberately boring by decision (`docs/TestDriverResearchPrototype.md`): one
|
||||||
|
process, one database, one browser engine, one application under test. Novelty
|
||||||
|
belongs in the verification model, never in the infrastructure. Do not add a
|
||||||
|
dependency without a stated reason in the workplan.
|
||||||
|
|
||||||
|
## Non-negotiables
|
||||||
|
|
||||||
|
These are architectural, not stylistic. Breaking one silently defeats the
|
||||||
|
framework's purpose — see `docs/TestDriverClassificationDesign.md`.
|
||||||
|
|
||||||
|
- **Claims and invariants are run inputs.** Never add a code path that lets
|
||||||
|
adaptation, retry, or a learned trajectory modify them (D-02).
|
||||||
|
- **S2/S3 evidence is never collected by an actor.** `Runner` enforces this;
|
||||||
|
do not weaken the check (D-01).
|
||||||
|
- **Model judgment is confined to S1** — locating controls, proposing paths.
|
||||||
|
Never verdicts, never claim evaluation (Concept Model § 2.3).
|
||||||
|
- **Missing evidence yields `INCONCLUSIVE`**, never a default pass or fail.
|
||||||
|
- **Claims require independent provenance.** `agent-from-implementation` output
|
||||||
|
is an exploratory hypothesis until a human promotes it (D-06).
|
||||||
|
|
||||||
<!-- Append repo-specific agent instructions below this marker.
|
<!-- Append repo-specific agent instructions below this marker.
|
||||||
The state-hub template sync preserves content after this line. -->
|
The state-hub template sync preserves content after this line. -->
|
||||||
|
|
||||||
|
|
|
||||||
40
README.md
40
README.md
|
|
@ -1,3 +1,41 @@
|
||||||
# test-driver
|
# test-driver
|
||||||
|
|
||||||
Agentic framework for integration, end2end, multiuserinteraction, security testing based on usecases.
|
Agentic framework for integration, end-to-end, multi-user interaction and
|
||||||
|
security testing, driven by use cases.
|
||||||
|
|
||||||
|
Tests mature alongside the software they protect: fluid and agentic while
|
||||||
|
behaviour is changing, deterministic once it settles. See `INTENT.md` for the
|
||||||
|
thesis and `SCOPE.md` for boundaries.
|
||||||
|
|
||||||
|
**Status:** research prototype. The deterministic kernel runs; agentic
|
||||||
|
realization, adaptation classification and crystallization are not built yet.
|
||||||
|
Current work: `workplans/TD-WP-0002-vertical-spike-crystallization.md`.
|
||||||
|
|
||||||
|
## Run
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m pytest -q # the whole suite
|
||||||
|
python3 -m pytest -q tests/test_reference_scenario.py
|
||||||
|
```
|
||||||
|
|
||||||
|
No third-party dependencies. Python ≥ 3.11, pytest for the suite.
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
src/testdriver/ the kernel — intent, world, actions, drivers,
|
||||||
|
observers, oracles, evidence, runner
|
||||||
|
lab/ the system under test
|
||||||
|
scenarios/ reference scenarios
|
||||||
|
research/ hypotheses, experiments, findings, fitness map
|
||||||
|
docs/ concept model, improvement loop, milestones, design notes
|
||||||
|
history/ assessments and completed-work write-ups
|
||||||
|
```
|
||||||
|
|
||||||
|
## Reading order
|
||||||
|
|
||||||
|
1. `INTENT.md` — the thesis
|
||||||
|
2. `docs/TestDriverConceptModel.md` — canonical concept set
|
||||||
|
3. `docs/TestDriverClassificationDesign.md` — why adaptation cannot normalize a
|
||||||
|
defect, and where model judgment is and is not permitted
|
||||||
|
4. `docs/TestDriverInitialMilestones.md` — canonical milestones M0–M10
|
||||||
|
|
|
||||||
|
|
@ -8,14 +8,14 @@
|
||||||
|
|
||||||
| Kind | ID | Status | Lane | Source |
|
| Kind | ID | Status | Lane | Source |
|
||||||
| --- | --- | --- | --- | --- |
|
| --- | --- | --- | --- | --- |
|
||||||
| workplan | TD-WP-0001 | ready | — | workplans/TD-WP-0001-statehub-bootstrap.md |
|
| workplan | TD-WP-0001 | active | — | workplans/TD-WP-0001-statehub-bootstrap.md |
|
||||||
| workplan | TD-WP-0002 | active | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
| workplan | TD-WP-0002 | active | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||||
| task | TD-WP-0001-T01 | done | — | workplans/TD-WP-0001-statehub-bootstrap.md |
|
| task | TD-WP-0001-T01 | done | — | workplans/TD-WP-0001-statehub-bootstrap.md |
|
||||||
| task | TD-WP-0001-T02 | wait | — | workplans/TD-WP-0001-statehub-bootstrap.md |
|
| task | TD-WP-0001-T02 | wait | — | workplans/TD-WP-0001-statehub-bootstrap.md |
|
||||||
| task | TD-WP-0001-T03 | done | — | workplans/TD-WP-0001-statehub-bootstrap.md |
|
| task | TD-WP-0001-T03 | done | — | workplans/TD-WP-0001-statehub-bootstrap.md |
|
||||||
| task | TD-WP-0002-T01 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
| task | TD-WP-0002-T01 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||||
| task | TD-WP-0002-T02 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
| task | TD-WP-0002-T02 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||||
| task | TD-WP-0002-T03 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
| task | TD-WP-0002-T03 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||||
| task | TD-WP-0002-T04 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
| task | TD-WP-0002-T04 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||||
| task | TD-WP-0002-T05 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
| task | TD-WP-0002-T05 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||||
| task | TD-WP-0002-T06 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
| task | TD-WP-0002-T06 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||||
|
|
|
||||||
1
lab/__init__.py
Normal file
1
lab/__init__.py
Normal file
|
|
@ -0,0 +1 @@
|
||||||
|
"""The test-driver lab — the system under test."""
|
||||||
BIN
lab/__pycache__/__init__.cpython-312.pyc
Normal file
BIN
lab/__pycache__/__init__.cpython-312.pyc
Normal file
Binary file not shown.
BIN
lab/__pycache__/minimal.cpython-312.pyc
Normal file
BIN
lab/__pycache__/minimal.cpython-312.pyc
Normal file
Binary file not shown.
191
lab/minimal.py
Normal file
191
lab/minimal.py
Normal file
|
|
@ -0,0 +1,191 @@
|
||||||
|
"""Minimal lab: the seed of the system under test.
|
||||||
|
|
||||||
|
Deliberately small. Users own resources, share them with read or write
|
||||||
|
permission, and revoke that sharing. Every state change is audited.
|
||||||
|
|
||||||
|
Two access paths exist, and the distinction is the whole point:
|
||||||
|
|
||||||
|
* the **enforcement path** (`request`) is what an actor uses. It checks
|
||||||
|
authorization and can therefore be *wrong* — that is where a seeded
|
||||||
|
authorization defect lives.
|
||||||
|
* the **observation channel** (`ObservationChannel`) reads stored state directly,
|
||||||
|
without authorization. It is the independent channel required by decision
|
||||||
|
D-07.
|
||||||
|
|
||||||
|
An oracle that consulted only stored state would verify the framework's own
|
||||||
|
reimplementation of the rules rather than the system's enforcement of them. An
|
||||||
|
oracle that consulted only the enforcement path would have no way to notice that
|
||||||
|
enforcement and record disagree. test-driver observes both, and treats
|
||||||
|
disagreement between them as meaningful in its own right — that disagreement is
|
||||||
|
the precise signature of the M05 authorization defect.
|
||||||
|
|
||||||
|
TD-WP-0002-T05 grows this into the full lab with an HTTP API, a browser UI and
|
||||||
|
the labelled mutation catalogue. It is kept in-process here so that T04 can prove
|
||||||
|
the kernel without dragging in a web stack.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from typing import Any, Literal
|
||||||
|
|
||||||
|
Permission = Literal["READ", "WRITE"]
|
||||||
|
|
||||||
|
|
||||||
|
class Denied(Exception):
|
||||||
|
"""The enforcement path refused the request."""
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(slots=True)
|
||||||
|
class AuditRecord:
|
||||||
|
sequence: int
|
||||||
|
event: str
|
||||||
|
actor_id: str
|
||||||
|
resource_id: str | None = None
|
||||||
|
subject_id: str | None = None
|
||||||
|
permission: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(slots=True)
|
||||||
|
class MinimalLab:
|
||||||
|
"""In-process resource-sharing service."""
|
||||||
|
|
||||||
|
version: str = "lab-0.1.0-baseline"
|
||||||
|
users: dict[str, str] = field(default_factory=dict) # user_id -> token
|
||||||
|
_tokens: dict[str, str] = field(default_factory=dict) # token -> user_id
|
||||||
|
resources: dict[str, dict[str, Any]] = field(default_factory=dict)
|
||||||
|
grants: dict[tuple[str, str], Permission] = field(default_factory=dict)
|
||||||
|
audit: list[AuditRecord] = field(default_factory=list)
|
||||||
|
_seq: int = 0
|
||||||
|
|
||||||
|
# -- setup -----------------------------------------------------------
|
||||||
|
|
||||||
|
def add_user(self, user_id: str) -> str:
|
||||||
|
token = f"tok-{user_id}"
|
||||||
|
self.users[user_id] = token
|
||||||
|
self._tokens[token] = user_id
|
||||||
|
return token
|
||||||
|
|
||||||
|
def _audit(self, event: str, actor_id: str, **kw: Any) -> None:
|
||||||
|
self._seq += 1
|
||||||
|
self.audit.append(AuditRecord(self._seq, event, actor_id, **kw))
|
||||||
|
|
||||||
|
def _whoami(self, token: str) -> str:
|
||||||
|
if token not in self._tokens:
|
||||||
|
raise Denied("unknown token")
|
||||||
|
return self._tokens[token]
|
||||||
|
|
||||||
|
# -- enforcement path (what actors use) ------------------------------
|
||||||
|
|
||||||
|
def request(self, token: str, op: str, **args: Any) -> Any:
|
||||||
|
"""The single entry point actors go through. Authorization is enforced here."""
|
||||||
|
user_id = self._whoami(token)
|
||||||
|
handler = getattr(self, f"_op_{op}", None)
|
||||||
|
if handler is None:
|
||||||
|
raise Denied(f"unknown operation {op!r}")
|
||||||
|
return handler(user_id, **args)
|
||||||
|
|
||||||
|
def _op_create_resource(self, user_id: str, resource_id: str, content: str) -> dict:
|
||||||
|
self.resources[resource_id] = {"owner": user_id, "content": content}
|
||||||
|
self._audit("create", user_id, resource_id=resource_id)
|
||||||
|
return {"resource_id": resource_id}
|
||||||
|
|
||||||
|
def _op_read_resource(self, user_id: str, resource_id: str) -> dict:
|
||||||
|
resource = self.resources.get(resource_id)
|
||||||
|
if resource is None:
|
||||||
|
raise Denied("no such resource")
|
||||||
|
if not self._may_read(user_id, resource_id):
|
||||||
|
raise Denied("not authorized to read")
|
||||||
|
return {"resource_id": resource_id, "content": resource["content"]}
|
||||||
|
|
||||||
|
def _op_grant(
|
||||||
|
self, user_id: str, resource_id: str, subject_id: str, permission: Permission
|
||||||
|
) -> dict:
|
||||||
|
resource = self.resources.get(resource_id)
|
||||||
|
if resource is None or resource["owner"] != user_id:
|
||||||
|
raise Denied("only the owner may grant")
|
||||||
|
self.grants[(resource_id, subject_id)] = permission
|
||||||
|
self._audit(
|
||||||
|
"grant", user_id, resource_id=resource_id,
|
||||||
|
subject_id=subject_id, permission=permission,
|
||||||
|
)
|
||||||
|
return {"granted": permission}
|
||||||
|
|
||||||
|
def _op_revoke(self, user_id: str, resource_id: str, subject_id: str) -> dict:
|
||||||
|
resource = self.resources.get(resource_id)
|
||||||
|
if resource is None or resource["owner"] != user_id:
|
||||||
|
raise Denied("only the owner may revoke")
|
||||||
|
self.grants.pop((resource_id, subject_id), None)
|
||||||
|
self._audit("revoke", user_id, resource_id=resource_id, subject_id=subject_id)
|
||||||
|
return {"revoked": True}
|
||||||
|
|
||||||
|
def _may_read(self, user_id: str, resource_id: str) -> bool:
|
||||||
|
"""The authorization rule as the system actually enforces it."""
|
||||||
|
resource = self.resources.get(resource_id)
|
||||||
|
if resource is None:
|
||||||
|
return False
|
||||||
|
if resource["owner"] == user_id:
|
||||||
|
return True
|
||||||
|
return (resource_id, user_id) in self.grants
|
||||||
|
|
||||||
|
|
||||||
|
class ObservationChannel:
|
||||||
|
"""Independent read access to lab state — decision D-07.
|
||||||
|
|
||||||
|
Bypasses authorization deliberately. This is the channel test-driver requires
|
||||||
|
of any system under test, and the main integration burden the framework
|
||||||
|
imposes on an adopter.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, lab: MinimalLab) -> None:
|
||||||
|
self._lab = lab
|
||||||
|
|
||||||
|
@property
|
||||||
|
def version(self) -> str:
|
||||||
|
return self._lab.version
|
||||||
|
|
||||||
|
def state_permission(self, user_id: str, resource_id: str) -> str | None:
|
||||||
|
"""What the stored record says, independent of any enforcement decision."""
|
||||||
|
resource = self._lab.resources.get(resource_id)
|
||||||
|
if resource is None:
|
||||||
|
return None
|
||||||
|
if resource["owner"] == user_id:
|
||||||
|
return "OWNER"
|
||||||
|
return self._lab.grants.get((resource_id, user_id))
|
||||||
|
|
||||||
|
def probe_read(self, user_id: str, resource_id: str) -> bool:
|
||||||
|
"""Exercise the enforcement path out-of-band and report what it did.
|
||||||
|
|
||||||
|
This uses the subject's own credentials, which can look like a violation
|
||||||
|
of actor isolation but is not: independence means the *actor's report* is
|
||||||
|
never the evidence. The observer issues its own request and records the
|
||||||
|
raw outcome. No actor is ever asked whether it succeeded.
|
||||||
|
"""
|
||||||
|
token = self._lab.users.get(user_id)
|
||||||
|
if token is None:
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
self._lab.request(token, "read_resource", resource_id=resource_id)
|
||||||
|
except Denied:
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
def audit_events(self, resource_id: str) -> list[dict[str, Any]]:
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
"sequence": r.sequence,
|
||||||
|
"event": r.event,
|
||||||
|
"actor_id": r.actor_id,
|
||||||
|
"subject_id": r.subject_id,
|
||||||
|
"permission": r.permission,
|
||||||
|
}
|
||||||
|
for r in self._lab.audit
|
||||||
|
if r.resource_id == resource_id
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def build_baseline() -> tuple[MinimalLab, dict[str, str]]:
|
||||||
|
"""Known initial state, so that runs replay from the same starting point."""
|
||||||
|
lab = MinimalLab()
|
||||||
|
tokens = {user: lab.add_user(user) for user in ("alice", "bob", "carol")}
|
||||||
|
return lab, tokens
|
||||||
20
pyproject.toml
Normal file
20
pyproject.toml
Normal file
|
|
@ -0,0 +1,20 @@
|
||||||
|
[project]
|
||||||
|
name = "test-driver"
|
||||||
|
version = "0.1.0"
|
||||||
|
description = "Use-case-driven verification framework whose tests mature alongside the software they protect"
|
||||||
|
requires-python = ">=3.11"
|
||||||
|
dependencies = []
|
||||||
|
|
||||||
|
[project.optional-dependencies]
|
||||||
|
dev = ["pytest>=7.4"]
|
||||||
|
|
||||||
|
[build-system]
|
||||||
|
requires = ["hatchling"]
|
||||||
|
build-backend = "hatchling.build"
|
||||||
|
|
||||||
|
[tool.hatch.build.targets.wheel]
|
||||||
|
packages = ["src/testdriver"]
|
||||||
|
|
||||||
|
[tool.pytest.ini_options]
|
||||||
|
testpaths = ["tests"]
|
||||||
|
pythonpath = ["src", "."]
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
# Concept ↔ Implementation Fitness Map
|
# Concept ↔ Implementation Fitness Map
|
||||||
|
|
||||||
**Updated:** 2026-08-22 (TD-WP-0002-T03)
|
**Updated:** 2026-08-22 (TD-WP-0002-T04)
|
||||||
|
|
||||||
Traces each important concept to the implementation, experiment and evidence that
|
Traces each important concept to the implementation, experiment and evidence that
|
||||||
support it. **Unsupported entries are the point of this map** — a concept with no
|
support it. **Unsupported entries are the point of this map** — a concept with no
|
||||||
|
|
@ -13,25 +13,29 @@ Support levels follow `TestDriverImprovementLoop.md` §13:
|
||||||
|
|
||||||
## Current state
|
## Current state
|
||||||
|
|
||||||
At the time of writing there is **no implementation**, so no concept can honestly
|
The deterministic kernel exists (T04) and its guarantees are covered by unit
|
||||||
exceed `C1`. The initial classifications in §13 of the Improvement Loop
|
tests. **Levels have not moved.** A passing unit test is not an experiment: it
|
||||||
|
shows the code does what its author intended, not that the concept holds under
|
||||||
|
the mutations it claims to survive. Levels rise when E-001/E-002/E-003 produce
|
||||||
|
evidence, not before. The implementation column below moves; the level column
|
||||||
|
does not. The initial classifications in §13 of the Improvement Loop
|
||||||
(`Actor Isolation C2`, `Independent Oracles C2`) are corrected downward here: they
|
(`Actor Isolation C2`, `Independent Oracles C2`) are corrected downward here: they
|
||||||
were aspirational, not evidenced.
|
were aspirational, not evidenced.
|
||||||
|
|
||||||
| Concept | Level | Implementation | Experiment | Evidence | Open question |
|
| Concept | Level | Implementation | Experiment | Evidence | Open question |
|
||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|
|
||||||
| `C-use-case` | C1 | — (T04) | — | — | Is a use case expressible without leaking mechanics? |
|
| `C-use-case` | C1 | `intent.py` | — | — | Is a use case expressible without leaking mechanics? |
|
||||||
| `C-actor-isolation` | C1 | — (T04) | E-001 | — | Isolation is asserted by construction; unverified. |
|
| `C-actor-isolation` | C1 | `world.py` | E-001 | — | Isolation is asserted by construction; unverified. |
|
||||||
| `C-semantic-action` | C1 | — (T04) | E-001 | — | Does identity survive restructuring better than a recorded sequence? (H-001) |
|
| `C-semantic-action` | C1 | `actions.py` | E-001 | — | Does identity survive restructuring better than a recorded sequence? (H-001) |
|
||||||
| `C-oracle-independence` | C1 | — (T04) | E-001, E-003 | — | Independence of components ≠ independence of belief. (H-004) |
|
| `C-oracle-independence` | C1 | `runner.py`, `oracles.py` | E-001, E-003 | — | Independence of components ≠ independence of belief. (H-004) |
|
||||||
| `C-evidence-pack` | C1 | — (T04) | — | — | What is the minimum sufficient for replay? |
|
| `C-evidence-pack` | C1 | `evidence.py` | — | — | What is the minimum sufficient for replay? |
|
||||||
| `C-observation-channel` | C1 | — (T05) | — | — | **D-07** — required of every system under test. Adoption cost unknown. |
|
| `C-observation-channel` | C1 | `lab/minimal.py` | — | — | **D-07** — required of every system under test. Adoption cost unknown. |
|
||||||
| `C-adaptation` | C1 | — (T08) | E-001 | — | (H-002) |
|
| `C-adaptation` | C1 | — (T08) | E-001 | — | (H-002) |
|
||||||
| `C-classification` | C1 | — (T08) | E-001, E-003 | — | Decision table is total on paper; unexercised. |
|
| `C-classification` | C1 | — (T08) | E-001, E-003 | — | Decision table is total on paper; unexercised. |
|
||||||
| `C-crystallization` | C1 | — (T09) | E-002 | — | (H-003) |
|
| `C-crystallization` | C1 | — (T09) | E-002 | — | (H-003) |
|
||||||
| `C-intent-provenance` | C1 | — (T04) | E-003 | — | Constrains provenance, not quality. Accepted residual. |
|
| `C-intent-provenance` | C1 | `provenance.py` | E-003 | — | Constrains provenance, not quality. Accepted residual. |
|
||||||
| `C-lineage` | C0 | — | — | — | Parent pointer only in the spike. |
|
| `C-lineage` | C0 | — | — | — | Parent pointer only in the spike. |
|
||||||
| `C-energy` | C0 | events only (T04) | — | — | Dormant by decision. (H-005) |
|
| `C-energy` | C0 | `energy.py`, capture only | — | — | Dormant by decision. (H-005) |
|
||||||
| `C-temperature` | C0 | — | — | — | Deferred. No implementation planned in TD-WP-0002. |
|
| `C-temperature` | C0 | — | — | — | Deferred. No implementation planned in TD-WP-0002. |
|
||||||
| `C-confidence` | C0 | — | — | — | Deferred. |
|
| `C-confidence` | C0 | — | — | — | Deferred. |
|
||||||
| `C-campaign` | C0 | — | — | — | Deferred. |
|
| `C-campaign` | C0 | — | — | — | Deferred. |
|
||||||
|
|
@ -49,4 +53,7 @@ most at risk of being built because they are easy and satisfying, and never
|
||||||
validated. They are revisited at T10, where the question is not "when do we build
|
validated. They are revisited at T10, where the question is not "when do we build
|
||||||
these" but "does the evidence justify keeping them in the model at all".
|
these" but "does the evidence justify keeping them in the model at all".
|
||||||
|
|
||||||
**Implementation orphans** — none yet; there is no implementation.
|
**Implementation orphans** — none. Every module in `src/testdriver/` traces to a
|
||||||
|
concept above. `energy.py` is the one to watch: it exists solely to capture
|
||||||
|
events for a dormant hypothesis, and if T10 finds no use for the history it
|
||||||
|
should be removed rather than kept out of sentiment.
|
||||||
|
|
|
||||||
BIN
scenarios/__pycache__/alice_bob_carol.cpython-312.pyc
Normal file
BIN
scenarios/__pycache__/alice_bob_carol.cpython-312.pyc
Normal file
Binary file not shown.
135
scenarios/alice_bob_carol.py
Normal file
135
scenarios/alice_bob_carol.py
Normal file
|
|
@ -0,0 +1,135 @@
|
||||||
|
"""The reference scenario.
|
||||||
|
|
||||||
|
Alice owns resource R. Alice grants Bob READ access. Bob can read R.
|
||||||
|
Carol cannot read R. Alice revokes Bob's access. Bob can no longer read R.
|
||||||
|
|
||||||
|
Written as Python rather than YAML on purpose: claims are predicates over
|
||||||
|
observations, and a YAML dialect capable of expressing them would be a
|
||||||
|
programming language with worse tooling. A declarative surface is a later
|
||||||
|
question, once we know which predicates actually recur.
|
||||||
|
|
||||||
|
Every claim here is `Provenance.HUMAN` — authored from the use case in INTENT.md,
|
||||||
|
not derived from watching the lab behave. See D-06.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from testdriver import (
|
||||||
|
Actor, Cast, Claim, DirectDriver, Invariant, Oracle, Provenance,
|
||||||
|
Scenario, SemanticAction, StateObserver, Step, UseCase,
|
||||||
|
VerificationAsset, Watch, World,
|
||||||
|
)
|
||||||
|
from lab.minimal import ObservationChannel, build_baseline
|
||||||
|
|
||||||
|
RESOURCE = "R"
|
||||||
|
API = frozenset({"api"})
|
||||||
|
|
||||||
|
# --- claims ---------------------------------------------------------------
|
||||||
|
# Each reads the independent observation snapshot. None consults an actor.
|
||||||
|
|
||||||
|
|
||||||
|
def _bob_can_read(obs) -> bool:
|
||||||
|
return obs["probe_read:bob:R"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def _bob_cannot_read(obs) -> bool:
|
||||||
|
return obs["probe_read:bob:R"] is False
|
||||||
|
|
||||||
|
|
||||||
|
def _carol_cannot_read(obs) -> bool:
|
||||||
|
return obs["probe_read:carol:R"] is False
|
||||||
|
|
||||||
|
|
||||||
|
def _enforcement_matches_record(obs) -> bool:
|
||||||
|
"""Enforcement and stored record must agree about every watched subject.
|
||||||
|
|
||||||
|
This invariant is what catches an authorization defect that leaves the audit
|
||||||
|
trail looking correct: the grant is recorded as revoked, yet the enforcement
|
||||||
|
path still allows the read. Neither observation alone would notice.
|
||||||
|
"""
|
||||||
|
for key, permitted in obs.items():
|
||||||
|
if not key.startswith("probe_read:"):
|
||||||
|
continue
|
||||||
|
recorded = obs.get("state_permission:" + key.removeprefix("probe_read:"))
|
||||||
|
if permitted != (recorded is not None):
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def _audit_is_append_only(obs) -> bool:
|
||||||
|
events = obs["audit:R"]
|
||||||
|
sequences = [e["sequence"] for e in events]
|
||||||
|
return sequences == sorted(sequences)
|
||||||
|
|
||||||
|
|
||||||
|
USE_CASE = UseCase(
|
||||||
|
id="uc-share-resource",
|
||||||
|
title="Share a resource and revoke the share",
|
||||||
|
narrative=(
|
||||||
|
"Alice owns resource R. Alice grants Bob READ access. Bob can read R. "
|
||||||
|
"Carol cannot read R. Alice revokes Bob's access. "
|
||||||
|
"Bob can no longer read R."
|
||||||
|
),
|
||||||
|
provenance=Provenance.HUMAN,
|
||||||
|
source_ref="INTENT.md#first-reference-scenario",
|
||||||
|
claims=(
|
||||||
|
Claim("c-bob-reads", "Bob can read R after the grant",
|
||||||
|
Provenance.HUMAN, _bob_can_read, after_step="s2-grant"),
|
||||||
|
Claim("c-carol-denied", "Carol can never read R",
|
||||||
|
Provenance.HUMAN, _carol_cannot_read, after_step="s2-grant"),
|
||||||
|
Claim("c-bob-revoked", "Bob cannot read R after revocation",
|
||||||
|
Provenance.HUMAN, _bob_cannot_read, after_step="s3-revoke"),
|
||||||
|
),
|
||||||
|
invariants=(
|
||||||
|
Invariant("i-enforcement-matches-record",
|
||||||
|
"Enforcement and the stored record agree for every subject",
|
||||||
|
Provenance.HUMAN, _enforcement_matches_record),
|
||||||
|
Invariant("i-audit-append-only", "The audit trail is append-only",
|
||||||
|
Provenance.HUMAN, _audit_is_append_only),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def build(variant: str = "baseline"):
|
||||||
|
"""Assemble world, driver, observer and asset from a known initial state."""
|
||||||
|
lab, tokens = build_baseline()
|
||||||
|
cast = Cast()
|
||||||
|
for name in ("alice", "bob", "carol"):
|
||||||
|
cast.add(Actor(id=name, display_name=name.title(),
|
||||||
|
credentials={"token": tokens[name]}))
|
||||||
|
|
||||||
|
world = World(id="w-baseline", sut=lab, sut_version=lab.version,
|
||||||
|
seed={"users": ["alice", "bob", "carol"], "resource": RESOURCE},
|
||||||
|
cast=cast)
|
||||||
|
|
||||||
|
scenario = Scenario(
|
||||||
|
id="sc-share-resource",
|
||||||
|
use_case=USE_CASE,
|
||||||
|
variant=variant,
|
||||||
|
watches=(Watch("bob", RESOURCE), Watch("carol", RESOURCE)),
|
||||||
|
steps=(
|
||||||
|
Step("s1-create", "alice", SemanticAction(
|
||||||
|
"create_resource",
|
||||||
|
{"resource_id": RESOURCE, "content": "the secret"},
|
||||||
|
permitted_surfaces=API,
|
||||||
|
postcondition=lambda obs: "audit:R" in obs,
|
||||||
|
)),
|
||||||
|
Step("s2-grant", "alice", SemanticAction(
|
||||||
|
"grant_access",
|
||||||
|
{"resource_id": RESOURCE, "subject_id": "bob", "permission": "READ"},
|
||||||
|
permitted_surfaces=API,
|
||||||
|
postcondition=lambda obs: obs["state_permission:bob:R"] == "READ",
|
||||||
|
)),
|
||||||
|
Step("s3-revoke", "alice", SemanticAction(
|
||||||
|
"revoke_access",
|
||||||
|
{"resource_id": RESOURCE, "subject_id": "bob"},
|
||||||
|
permitted_surfaces=API,
|
||||||
|
postcondition=lambda obs: obs["state_permission:bob:R"] is None,
|
||||||
|
)),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
driver = DirectDriver(lab, tokens)
|
||||||
|
observer = StateObserver(ObservationChannel(lab), scenario.watches)
|
||||||
|
asset = VerificationAsset(id="va-share-resource", scenario=scenario, maturity="T5")
|
||||||
|
return world, driver, observer, asset, Oracle()
|
||||||
23
src/testdriver/__init__.py
Normal file
23
src/testdriver/__init__.py
Normal file
|
|
@ -0,0 +1,23 @@
|
||||||
|
"""test-driver — verification assets that mature alongside the software they protect."""
|
||||||
|
|
||||||
|
from .actions import SemanticAction, Surface, SurfaceNotPermitted
|
||||||
|
from .drivers import DirectDriver, Realization
|
||||||
|
from .energy import EnergyEvent, EnergyEventType
|
||||||
|
from .evidence import EvidencePack, Observation, Stratum
|
||||||
|
from .intent import Claim, Invariant, UseCase
|
||||||
|
from .observers import StateObserver, Watch
|
||||||
|
from .oracles import Judgment, Oracle, Verdict, overall
|
||||||
|
from .provenance import InadmissibleProvenance, Provenance
|
||||||
|
from .runner import CollectorIndependenceError, Runner, RunResult
|
||||||
|
from .scenario import Scenario, Step, VerificationAsset
|
||||||
|
from .world import Actor, ActorIsolationError, Cast, World
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"Actor", "ActorIsolationError", "Cast", "Claim", "CollectorIndependenceError",
|
||||||
|
"DirectDriver", "EnergyEvent", "EnergyEventType", "EvidencePack",
|
||||||
|
"InadmissibleProvenance", "Invariant", "Judgment", "Observation", "Oracle",
|
||||||
|
"Provenance", "Realization", "RunResult", "Runner", "Scenario",
|
||||||
|
"SemanticAction", "StateObserver", "Step", "Stratum", "Surface",
|
||||||
|
"SurfaceNotPermitted", "UseCase", "VerificationAsset", "Verdict", "Watch",
|
||||||
|
"overall",
|
||||||
|
]
|
||||||
BIN
src/testdriver/__pycache__/__init__.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/__init__.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/actions.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/actions.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/drivers.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/drivers.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/energy.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/energy.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/evidence.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/evidence.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/intent.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/intent.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/observers.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/observers.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/oracles.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/oracles.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/provenance.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/provenance.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/runner.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/runner.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/scenario.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/scenario.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/world.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/world.cpython-312.pyc
Normal file
Binary file not shown.
57
src/testdriver/actions.py
Normal file
57
src/testdriver/actions.py
Normal file
|
|
@ -0,0 +1,57 @@
|
||||||
|
"""Semantic actions and the surfaces they may legitimately use.
|
||||||
|
|
||||||
|
A semantic action names *what* is being accomplished, never *how*. The `how` is
|
||||||
|
a driver's business and is expected to change; the `what` is the stable identity
|
||||||
|
that survives restructuring (H-001).
|
||||||
|
|
||||||
|
Every action declares the surfaces it is permitted to use — decision D-05. An
|
||||||
|
actor that achieves `grant_access` through a surface the scenario did not permit
|
||||||
|
has not recovered from a change; it has performed an unrequested
|
||||||
|
surface-substitution, which is one of the catalogued security mutations. Without
|
||||||
|
this check, an agent can route around a broken authorization control and score
|
||||||
|
as a successful mechanical adaptation.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from typing import Any, Callable, Mapping
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class Surface:
|
||||||
|
"""An interaction surface: an API, a browser UI, a CLI, a queue."""
|
||||||
|
|
||||||
|
id: str
|
||||||
|
kind: str
|
||||||
|
description: str = ""
|
||||||
|
|
||||||
|
|
||||||
|
class SurfaceNotPermitted(Exception):
|
||||||
|
"""An action was realized through a surface the scenario did not permit."""
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class SemanticAction:
|
||||||
|
"""Intent to change or inspect the world, expressed without mechanics.
|
||||||
|
|
||||||
|
`postcondition` is evaluated by an independent Observer (S2), never by the
|
||||||
|
actor that performed the action. An actor never reports its own success.
|
||||||
|
"""
|
||||||
|
|
||||||
|
name: str
|
||||||
|
args: Mapping[str, Any] = field(default_factory=dict)
|
||||||
|
permitted_surfaces: frozenset[str] = frozenset()
|
||||||
|
postcondition: Callable[[Mapping[str, object]], bool] | None = None
|
||||||
|
|
||||||
|
def check_surface(self, surface_id: str) -> None:
|
||||||
|
if self.permitted_surfaces and surface_id not in self.permitted_surfaces:
|
||||||
|
raise SurfaceNotPermitted(
|
||||||
|
f"action {self.name!r} was realized through surface "
|
||||||
|
f"{surface_id!r}, which is not in "
|
||||||
|
f"{sorted(self.permitted_surfaces)} (D-05)"
|
||||||
|
)
|
||||||
|
|
||||||
|
def describe(self) -> str:
|
||||||
|
rendered = ", ".join(f"{k}={v!r}" for k, v in sorted(self.args.items()))
|
||||||
|
return f"{self.name}({rendered})"
|
||||||
76
src/testdriver/drivers.py
Normal file
76
src/testdriver/drivers.py
Normal file
|
|
@ -0,0 +1,76 @@
|
||||||
|
"""Drivers realize semantic actions against a surface.
|
||||||
|
|
||||||
|
A driver knows mechanics. It emits S1 evidence describing *how* it acted and
|
||||||
|
reports which surface it used, but it never reports whether the action was
|
||||||
|
correct — that is S2/S3 and belongs to the observer and the oracle.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import Any, Protocol
|
||||||
|
|
||||||
|
from .actions import SemanticAction, Surface
|
||||||
|
from .world import Actor
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class Realization:
|
||||||
|
"""What a driver did, mechanically. Pure S1."""
|
||||||
|
|
||||||
|
surface_id: str
|
||||||
|
mechanics: dict[str, Any]
|
||||||
|
raised: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class Driver(Protocol):
|
||||||
|
surface: Surface
|
||||||
|
|
||||||
|
def realize(self, actor: Actor, action: SemanticAction) -> Realization: ...
|
||||||
|
|
||||||
|
|
||||||
|
class UnsupportedAction(Exception):
|
||||||
|
"""The driver has no mechanical implementation for this semantic action."""
|
||||||
|
|
||||||
|
|
||||||
|
class DirectDriver:
|
||||||
|
"""Deterministic driver against the lab's enforcement path.
|
||||||
|
|
||||||
|
This is the T5 Deterministic end of the maturity continuum: a fixed mapping
|
||||||
|
from semantic action to mechanics, with no discovery and no model.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, lab: Any, tokens: dict[str, str]) -> None:
|
||||||
|
self._lab = lab
|
||||||
|
self._tokens = tokens
|
||||||
|
self.surface = Surface(
|
||||||
|
id="api", kind="http-like", description="lab enforcement path"
|
||||||
|
)
|
||||||
|
|
||||||
|
_MAPPING = {
|
||||||
|
"create_resource": ("create_resource", ("resource_id", "content")),
|
||||||
|
"grant_access": ("grant", ("resource_id", "subject_id", "permission")),
|
||||||
|
"revoke_access": ("revoke", ("resource_id", "subject_id")),
|
||||||
|
"read_resource": ("read_resource", ("resource_id",)),
|
||||||
|
}
|
||||||
|
|
||||||
|
def realize(self, actor: Actor, action: SemanticAction) -> Realization:
|
||||||
|
if action.name not in self._MAPPING:
|
||||||
|
raise UnsupportedAction(action.name)
|
||||||
|
action.check_surface(self.surface.id)
|
||||||
|
|
||||||
|
op, arg_names = self._MAPPING[action.name]
|
||||||
|
args = {name: action.args[name] for name in arg_names if name in action.args}
|
||||||
|
token = self._tokens[actor.id]
|
||||||
|
|
||||||
|
mechanics: dict[str, Any] = {
|
||||||
|
"operation": op,
|
||||||
|
"arguments": args,
|
||||||
|
"actor": actor.id,
|
||||||
|
}
|
||||||
|
try:
|
||||||
|
result = self._lab.request(token, op, **args)
|
||||||
|
except Exception as exc: # the SUT refusing is data, not a framework error
|
||||||
|
return Realization(self.surface.id, mechanics, raised=f"{type(exc).__name__}: {exc}")
|
||||||
|
mechanics["result"] = result
|
||||||
|
return Realization(self.surface.id, mechanics)
|
||||||
45
src/testdriver/energy.py
Normal file
45
src/testdriver/energy.py
Normal file
|
|
@ -0,0 +1,45 @@
|
||||||
|
"""Energy events — capture only.
|
||||||
|
|
||||||
|
H-005 is dormant by decision: verification energy is not testable at the current
|
||||||
|
scale, and a scoring function producing a number nobody can check is worse than
|
||||||
|
no number. Events are recorded from the first run because history cannot be
|
||||||
|
reconstructed later; scores always can.
|
||||||
|
|
||||||
|
There is deliberately no score() function in this module.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import asdict, dataclass, field
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from enum import Enum
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
|
||||||
|
class EnergyEventType(str, Enum):
|
||||||
|
DEFECT_DETECTED = "DEFECT_DETECTED"
|
||||||
|
REGRESSION_CAUGHT = "REGRESSION_CAUGHT"
|
||||||
|
MECHANICAL_ADAPTATION = "MECHANICAL_ADAPTATION"
|
||||||
|
SEMANTIC_ADAPTATION = "SEMANTIC_ADAPTATION"
|
||||||
|
TEST_DEFECT = "TEST_DEFECT"
|
||||||
|
FALSE_POSITIVE = "FALSE_POSITIVE"
|
||||||
|
DUPLICATE = "DUPLICATE"
|
||||||
|
CRYSTALLIZED = "CRYSTALLIZED"
|
||||||
|
USECASE_DEPRECATED = "USECASE_DEPRECATED"
|
||||||
|
EXECUTED = "EXECUTED"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class EnergyEvent:
|
||||||
|
"""Immutable. Energy is derived from event history, never stored as state."""
|
||||||
|
|
||||||
|
asset_id: str
|
||||||
|
run_id: str
|
||||||
|
event_type: EnergyEventType
|
||||||
|
detail: dict[str, Any] = field(default_factory=dict)
|
||||||
|
at: str = field(
|
||||||
|
default_factory=lambda: datetime.now(timezone.utc).isoformat()
|
||||||
|
)
|
||||||
|
|
||||||
|
def as_dict(self) -> dict[str, Any]:
|
||||||
|
return {**asdict(self), "event_type": self.event_type.value}
|
||||||
86
src/testdriver/evidence.py
Normal file
86
src/testdriver/evidence.py
Normal file
|
|
@ -0,0 +1,86 @@
|
||||||
|
"""Stratified evidence — decision D-01.
|
||||||
|
|
||||||
|
S1 Surface how an action was performed (selectors, routes, payloads)
|
||||||
|
S2 Realization whether it happened, and through which surface
|
||||||
|
S3 Judgment whether that was correct
|
||||||
|
|
||||||
|
Each stratum has a different authority. Models may write S1. Nothing but an
|
||||||
|
independent Observer writes S2 or S3.
|
||||||
|
|
||||||
|
See docs/TestDriverClassificationDesign.md, Part A.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from dataclasses import dataclass, field, asdict
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from enum import Enum
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
|
||||||
|
class Stratum(str, Enum):
|
||||||
|
SURFACE = "S1"
|
||||||
|
REALIZATION = "S2"
|
||||||
|
JUDGMENT = "S3"
|
||||||
|
|
||||||
|
|
||||||
|
def _now() -> str:
|
||||||
|
return datetime.now(timezone.utc).isoformat()
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class Observation:
|
||||||
|
"""One recorded fact, attributed to a stratum and a collector.
|
||||||
|
|
||||||
|
`collector` is never an actor for S2/S3 observations. The runner enforces
|
||||||
|
this; see runner._assert_collector_independence.
|
||||||
|
"""
|
||||||
|
|
||||||
|
id: str
|
||||||
|
stratum: Stratum
|
||||||
|
collector: str
|
||||||
|
step_id: str | None
|
||||||
|
kind: str
|
||||||
|
data: dict[str, Any]
|
||||||
|
at: str = field(default_factory=_now)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(slots=True)
|
||||||
|
class EvidencePack:
|
||||||
|
"""Everything retained from one run.
|
||||||
|
|
||||||
|
The pack must be sufficient to replay the run and to diagnose a finding
|
||||||
|
without the original process. An assertion that cannot be supported from the
|
||||||
|
pack is an EVIDENCE_FAILURE, not a defect in the system under test.
|
||||||
|
"""
|
||||||
|
|
||||||
|
run_id: str
|
||||||
|
scenario_id: str
|
||||||
|
use_case_id: str
|
||||||
|
sut_version: str
|
||||||
|
started_at: str = field(default_factory=_now)
|
||||||
|
finished_at: str | None = None
|
||||||
|
observations: list[Observation] = field(default_factory=list)
|
||||||
|
verdicts: list[dict[str, Any]] = field(default_factory=list)
|
||||||
|
energy_events: list[dict[str, Any]] = field(default_factory=list)
|
||||||
|
provenance_index: dict[str, str] = field(default_factory=dict)
|
||||||
|
|
||||||
|
def record(self, observation: Observation) -> None:
|
||||||
|
self.observations.append(observation)
|
||||||
|
|
||||||
|
def of_stratum(self, stratum: Stratum) -> list[Observation]:
|
||||||
|
return [o for o in self.observations if o.stratum is stratum]
|
||||||
|
|
||||||
|
def latest(self, kind: str) -> Observation | None:
|
||||||
|
for observation in reversed(self.observations):
|
||||||
|
if observation.kind == kind:
|
||||||
|
return observation
|
||||||
|
return None
|
||||||
|
|
||||||
|
def to_json(self) -> str:
|
||||||
|
payload = asdict(self)
|
||||||
|
payload["observations"] = [
|
||||||
|
{**asdict(o), "stratum": o.stratum.value} for o in self.observations
|
||||||
|
]
|
||||||
|
return json.dumps(payload, indent=2, sort_keys=True, default=str)
|
||||||
63
src/testdriver/intent.py
Normal file
63
src/testdriver/intent.py
Normal file
|
|
@ -0,0 +1,63 @@
|
||||||
|
"""The intent layer: what is supposed to be true.
|
||||||
|
|
||||||
|
Claims and invariants are *inputs* to a run and are frozen — decision D-02.
|
||||||
|
There is deliberately no code path by which adaptation, retry, or a learned
|
||||||
|
trajectory can modify them. That absence is what makes False Adaptation Rate = 0
|
||||||
|
an architectural property rather than a tuning target.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from typing import Callable, Mapping
|
||||||
|
|
||||||
|
from .provenance import Provenance, require_admissible
|
||||||
|
|
||||||
|
# A predicate over the independent observations gathered during a run.
|
||||||
|
# It receives the observation mapping and returns True when satisfied.
|
||||||
|
Predicate = Callable[[Mapping[str, object]], bool]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class Claim:
|
||||||
|
"""A statement that must hold at a specific point in a scenario."""
|
||||||
|
|
||||||
|
id: str
|
||||||
|
text: str
|
||||||
|
provenance: Provenance
|
||||||
|
predicate: Predicate
|
||||||
|
after_step: str
|
||||||
|
source_ref: str | None = None
|
||||||
|
|
||||||
|
def __post_init__(self) -> None:
|
||||||
|
require_admissible(self.provenance, f"Claim {self.id!r}")
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class Invariant:
|
||||||
|
"""A statement that must hold after *every* step, not merely at one point."""
|
||||||
|
|
||||||
|
id: str
|
||||||
|
text: str
|
||||||
|
provenance: Provenance
|
||||||
|
predicate: Predicate
|
||||||
|
source_ref: str | None = None
|
||||||
|
|
||||||
|
def __post_init__(self) -> None:
|
||||||
|
require_admissible(self.provenance, f"Invariant {self.id!r}")
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class UseCase:
|
||||||
|
"""Purposeful behaviour, described independently of mechanics."""
|
||||||
|
|
||||||
|
id: str
|
||||||
|
title: str
|
||||||
|
narrative: str
|
||||||
|
provenance: Provenance
|
||||||
|
claims: tuple[Claim, ...] = field(default_factory=tuple)
|
||||||
|
invariants: tuple[Invariant, ...] = field(default_factory=tuple)
|
||||||
|
source_ref: str | None = None
|
||||||
|
|
||||||
|
def __post_init__(self) -> None:
|
||||||
|
require_admissible(self.provenance, f"UseCase {self.id!r}")
|
||||||
47
src/testdriver/observers.py
Normal file
47
src/testdriver/observers.py
Normal file
|
|
@ -0,0 +1,47 @@
|
||||||
|
"""Observers gather evidence independently of the actors.
|
||||||
|
|
||||||
|
An observer never asks an actor what happened. It reads the system directly
|
||||||
|
through the independent observation channel required by decision D-07, and
|
||||||
|
records both what the stored record says and what the enforcement path actually
|
||||||
|
does. Disagreement between those two is meaningful in its own right: it is the
|
||||||
|
signature of an authorization defect, where the audit trail says one thing and
|
||||||
|
enforcement does another.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from typing import Any, Sequence
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class Watch:
|
||||||
|
"""A (subject, resource) pair the scenario wants observed after every step."""
|
||||||
|
|
||||||
|
subject_id: str
|
||||||
|
resource_id: str
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(slots=True)
|
||||||
|
class StateObserver:
|
||||||
|
"""Collects the S3 snapshot: what is true of the domain right now."""
|
||||||
|
|
||||||
|
channel: Any
|
||||||
|
watches: Sequence[Watch] = field(default_factory=tuple)
|
||||||
|
name: str = "state-observer"
|
||||||
|
|
||||||
|
def snapshot(self) -> dict[str, Any]:
|
||||||
|
out: dict[str, Any] = {}
|
||||||
|
resources = set()
|
||||||
|
for watch in self.watches:
|
||||||
|
key = f"{watch.subject_id}:{watch.resource_id}"
|
||||||
|
out[f"probe_read:{key}"] = self.channel.probe_read(
|
||||||
|
watch.subject_id, watch.resource_id
|
||||||
|
)
|
||||||
|
out[f"state_permission:{key}"] = self.channel.state_permission(
|
||||||
|
watch.subject_id, watch.resource_id
|
||||||
|
)
|
||||||
|
resources.add(watch.resource_id)
|
||||||
|
for resource_id in sorted(resources):
|
||||||
|
out[f"audit:{resource_id}"] = self.channel.audit_events(resource_id)
|
||||||
|
return out
|
||||||
106
src/testdriver/oracles.py
Normal file
106
src/testdriver/oracles.py
Normal file
|
|
@ -0,0 +1,106 @@
|
||||||
|
"""Oracles evaluate claims and invariants and produce verdicts.
|
||||||
|
|
||||||
|
An oracle reads only the independent observation snapshot. It has no access to
|
||||||
|
the actor, to the driver, or to what either of them believes happened.
|
||||||
|
|
||||||
|
`INCONCLUSIVE` is a first-class outcome, not a failure mode of the framework. An
|
||||||
|
oracle that cannot support a judgment from the retained evidence must say so
|
||||||
|
rather than defaulting either way — silently defaulting to PASS hides defects,
|
||||||
|
and silently defaulting to FAIL trains people to ignore results.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from enum import Enum
|
||||||
|
from typing import Any, Mapping
|
||||||
|
|
||||||
|
from .intent import Claim, Invariant
|
||||||
|
|
||||||
|
|
||||||
|
class Verdict(str, Enum):
|
||||||
|
PASS = "PASS"
|
||||||
|
FAIL = "FAIL"
|
||||||
|
SUSPICIOUS = "SUSPICIOUS"
|
||||||
|
INCONCLUSIVE = "INCONCLUSIVE"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class Judgment:
|
||||||
|
assertion_id: str
|
||||||
|
text: str
|
||||||
|
verdict: Verdict
|
||||||
|
step_id: str | None
|
||||||
|
detail: dict[str, Any] = field(default_factory=dict)
|
||||||
|
|
||||||
|
def as_dict(self) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"assertion_id": self.assertion_id,
|
||||||
|
"text": self.text,
|
||||||
|
"verdict": self.verdict.value,
|
||||||
|
"step_id": self.step_id,
|
||||||
|
"detail": self.detail,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class Oracle:
|
||||||
|
"""Deterministic evaluation of one assertion against an observation snapshot."""
|
||||||
|
|
||||||
|
def judge(
|
||||||
|
self,
|
||||||
|
assertion: Claim | Invariant,
|
||||||
|
snapshot: Mapping[str, Any],
|
||||||
|
step_id: str | None,
|
||||||
|
) -> Judgment:
|
||||||
|
if not snapshot:
|
||||||
|
return Judgment(
|
||||||
|
assertion.id,
|
||||||
|
assertion.text,
|
||||||
|
Verdict.INCONCLUSIVE,
|
||||||
|
step_id,
|
||||||
|
{"reason": "no observations were collected"},
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
satisfied = assertion.predicate(snapshot)
|
||||||
|
except KeyError as missing:
|
||||||
|
# The evidence needed to judge this assertion was not collected.
|
||||||
|
# That is an evidence failure, never a pass and never a fail.
|
||||||
|
return Judgment(
|
||||||
|
assertion.id,
|
||||||
|
assertion.text,
|
||||||
|
Verdict.INCONCLUSIVE,
|
||||||
|
step_id,
|
||||||
|
{"reason": f"required observation {missing} missing from snapshot"},
|
||||||
|
)
|
||||||
|
except Exception as exc:
|
||||||
|
return Judgment(
|
||||||
|
assertion.id,
|
||||||
|
assertion.text,
|
||||||
|
Verdict.INCONCLUSIVE,
|
||||||
|
step_id,
|
||||||
|
{"reason": f"predicate raised {type(exc).__name__}: {exc}"},
|
||||||
|
)
|
||||||
|
return Judgment(
|
||||||
|
assertion.id,
|
||||||
|
assertion.text,
|
||||||
|
Verdict.PASS if satisfied else Verdict.FAIL,
|
||||||
|
step_id,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def overall(judgments: list[Judgment]) -> Verdict:
|
||||||
|
"""Aggregate run verdict.
|
||||||
|
|
||||||
|
FAIL dominates; INCONCLUSIVE outranks PASS. A run containing an unjudgeable
|
||||||
|
assertion has not passed, whatever else it did.
|
||||||
|
"""
|
||||||
|
verdicts = {j.verdict for j in judgments}
|
||||||
|
if not judgments:
|
||||||
|
return Verdict.INCONCLUSIVE
|
||||||
|
if Verdict.FAIL in verdicts:
|
||||||
|
return Verdict.FAIL
|
||||||
|
if Verdict.INCONCLUSIVE in verdicts:
|
||||||
|
return Verdict.INCONCLUSIVE
|
||||||
|
if Verdict.SUSPICIOUS in verdicts:
|
||||||
|
return Verdict.SUSPICIOUS
|
||||||
|
return Verdict.PASS
|
||||||
46
src/testdriver/provenance.py
Normal file
46
src/testdriver/provenance.py
Normal file
|
|
@ -0,0 +1,46 @@
|
||||||
|
"""Claim provenance — decision D-06.
|
||||||
|
|
||||||
|
A claim may only be authored by a source causally independent of the
|
||||||
|
implementation it constrains. Without this rule the framework's guarantee
|
||||||
|
reduces to "the implementation agrees with itself", which is exactly the failure
|
||||||
|
the project exists to prevent.
|
||||||
|
|
||||||
|
See docs/TestDriverClassificationDesign.md, Part B.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from enum import Enum
|
||||||
|
|
||||||
|
|
||||||
|
class Provenance(str, Enum):
|
||||||
|
"""Where an intent artifact came from."""
|
||||||
|
|
||||||
|
HUMAN = "human"
|
||||||
|
SPEC = "spec"
|
||||||
|
AGENT_FROM_SPEC = "agent-from-spec"
|
||||||
|
AGENT_FROM_IMPLEMENTATION = "agent-from-implementation"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def admissible_as_claim(self) -> bool:
|
||||||
|
"""Whether this provenance may back an assertion that can produce FAIL.
|
||||||
|
|
||||||
|
`agent-from-implementation` is not forbidden as an activity — it is
|
||||||
|
genuinely useful for T0 exploration. It is forbidden as a *claim*.
|
||||||
|
Such output enters as an exploratory hypothesis and requires an explicit
|
||||||
|
human acceptance event before it can constrain the system.
|
||||||
|
"""
|
||||||
|
return self is not Provenance.AGENT_FROM_IMPLEMENTATION
|
||||||
|
|
||||||
|
|
||||||
|
class InadmissibleProvenance(Exception):
|
||||||
|
"""Raised when implementation-derived intent is used as a claim."""
|
||||||
|
|
||||||
|
|
||||||
|
def require_admissible(provenance: Provenance, what: str) -> None:
|
||||||
|
if not provenance.admissible_as_claim:
|
||||||
|
raise InadmissibleProvenance(
|
||||||
|
f"{what} has provenance {provenance.value!r}, which is derived from the "
|
||||||
|
"implementation it would constrain. Promote it through an explicit human "
|
||||||
|
"acceptance event before using it as a claim (D-06)."
|
||||||
|
)
|
||||||
190
src/testdriver/runner.py
Normal file
190
src/testdriver/runner.py
Normal file
|
|
@ -0,0 +1,190 @@
|
||||||
|
"""The orchestrator: executes a scenario and assembles its evidence.
|
||||||
|
|
||||||
|
The runner is the only component that sees everything. Actors see their own
|
||||||
|
credentials and memory; drivers see mechanics; observers see the system; oracles
|
||||||
|
see the observation snapshot. Keeping those views separate is what makes the
|
||||||
|
independence claim structural rather than procedural.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import uuid
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from .actions import SurfaceNotPermitted
|
||||||
|
from .drivers import Driver
|
||||||
|
from .energy import EnergyEvent, EnergyEventType
|
||||||
|
from .evidence import EvidencePack, Observation, Stratum
|
||||||
|
from .observers import StateObserver
|
||||||
|
from .oracles import Judgment, Oracle, Verdict, overall
|
||||||
|
from .scenario import Scenario, VerificationAsset
|
||||||
|
from .world import World
|
||||||
|
|
||||||
|
|
||||||
|
class CollectorIndependenceError(Exception):
|
||||||
|
"""An actor was about to be recorded as the collector of S2/S3 evidence."""
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(slots=True)
|
||||||
|
class RunResult:
|
||||||
|
run_id: str
|
||||||
|
verdict: Verdict
|
||||||
|
judgments: list[Judgment]
|
||||||
|
evidence: EvidencePack
|
||||||
|
|
||||||
|
def judgment(self, assertion_id: str) -> Judgment:
|
||||||
|
for j in self.judgments:
|
||||||
|
if j.assertion_id == assertion_id:
|
||||||
|
return j
|
||||||
|
raise KeyError(assertion_id)
|
||||||
|
|
||||||
|
|
||||||
|
class Runner:
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
world: World,
|
||||||
|
driver: Driver,
|
||||||
|
observer: StateObserver,
|
||||||
|
oracle: Oracle | None = None,
|
||||||
|
) -> None:
|
||||||
|
self._world = world
|
||||||
|
self._driver = driver
|
||||||
|
self._observer = observer
|
||||||
|
self._oracle = oracle or Oracle()
|
||||||
|
|
||||||
|
# -- independence guards ---------------------------------------------
|
||||||
|
|
||||||
|
def _assert_collector_independence(self, stratum: Stratum, collector: str) -> None:
|
||||||
|
"""S2 and S3 evidence may never be attributed to an actor.
|
||||||
|
|
||||||
|
This is the check that makes oracle independence falsifiable rather than
|
||||||
|
merely asserted: wiring an actor in as an observer fails loudly here.
|
||||||
|
"""
|
||||||
|
if stratum is Stratum.SURFACE:
|
||||||
|
return
|
||||||
|
if collector in self._world.cast.actors:
|
||||||
|
raise CollectorIndependenceError(
|
||||||
|
f"{stratum.value} evidence cannot be collected by actor "
|
||||||
|
f"{collector!r}; actors do not judge their own outcomes"
|
||||||
|
)
|
||||||
|
|
||||||
|
def _record(
|
||||||
|
self,
|
||||||
|
pack: EvidencePack,
|
||||||
|
stratum: Stratum,
|
||||||
|
collector: str,
|
||||||
|
kind: str,
|
||||||
|
data: dict[str, Any],
|
||||||
|
step_id: str | None,
|
||||||
|
) -> None:
|
||||||
|
self._assert_collector_independence(stratum, collector)
|
||||||
|
pack.record(
|
||||||
|
Observation(
|
||||||
|
id=f"obs-{len(pack.observations) + 1:04d}",
|
||||||
|
stratum=stratum,
|
||||||
|
collector=collector,
|
||||||
|
step_id=step_id,
|
||||||
|
kind=kind,
|
||||||
|
data=data,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
# -- execution --------------------------------------------------------
|
||||||
|
|
||||||
|
def run(self, asset: VerificationAsset) -> RunResult:
|
||||||
|
scenario: Scenario = asset.scenario
|
||||||
|
run_id = f"run-{uuid.uuid4().hex[:12]}"
|
||||||
|
pack = EvidencePack(
|
||||||
|
run_id=run_id,
|
||||||
|
scenario_id=scenario.id,
|
||||||
|
use_case_id=scenario.use_case.id,
|
||||||
|
sut_version=self._world.sut_version,
|
||||||
|
)
|
||||||
|
pack.provenance_index = {
|
||||||
|
scenario.use_case.id: scenario.use_case.provenance.value,
|
||||||
|
**{c.id: c.provenance.value for c in scenario.use_case.claims},
|
||||||
|
**{i.id: i.provenance.value for i in scenario.use_case.invariants},
|
||||||
|
}
|
||||||
|
pack.energy_events.append(
|
||||||
|
EnergyEvent(asset.id, run_id, EnergyEventType.EXECUTED).as_dict()
|
||||||
|
)
|
||||||
|
|
||||||
|
judgments: list[Judgment] = []
|
||||||
|
claims_by_step: dict[str, list] = {}
|
||||||
|
for claim in scenario.use_case.claims:
|
||||||
|
claims_by_step.setdefault(claim.after_step, []).append(claim)
|
||||||
|
|
||||||
|
for step in scenario.steps:
|
||||||
|
actor = self._world.cast[step.actor_id]
|
||||||
|
|
||||||
|
# --- S1: how it was done -------------------------------------
|
||||||
|
try:
|
||||||
|
realization = self._driver.realize(actor, step.action)
|
||||||
|
except SurfaceNotPermitted as exc:
|
||||||
|
# D-05: routing around a control is a finding, not a recovery.
|
||||||
|
self._record(
|
||||||
|
pack, Stratum.REALIZATION, self._observer.name,
|
||||||
|
"surface_violation",
|
||||||
|
{"step": step.id, "action": step.action.describe(), "error": str(exc)},
|
||||||
|
step.id,
|
||||||
|
)
|
||||||
|
break
|
||||||
|
|
||||||
|
self._record(
|
||||||
|
pack, Stratum.SURFACE, actor.id, "realization",
|
||||||
|
{
|
||||||
|
"action": step.action.describe(),
|
||||||
|
"surface": realization.surface_id,
|
||||||
|
"mechanics": realization.mechanics,
|
||||||
|
"raised": realization.raised,
|
||||||
|
},
|
||||||
|
step.id,
|
||||||
|
)
|
||||||
|
|
||||||
|
# --- S3: what is now true ------------------------------------
|
||||||
|
snapshot = self._observer.snapshot()
|
||||||
|
self._record(
|
||||||
|
pack, Stratum.JUDGMENT, self._observer.name,
|
||||||
|
"state_snapshot", dict(snapshot), step.id,
|
||||||
|
)
|
||||||
|
|
||||||
|
# --- S2: did the action actually take effect ------------------
|
||||||
|
refused = realization.raised is not None
|
||||||
|
postcondition_met: bool | None = None
|
||||||
|
if step.action.postcondition is not None:
|
||||||
|
try:
|
||||||
|
postcondition_met = step.action.postcondition(snapshot)
|
||||||
|
except KeyError:
|
||||||
|
postcondition_met = None
|
||||||
|
self._record(
|
||||||
|
pack, Stratum.REALIZATION, self._observer.name, "realization_check",
|
||||||
|
{
|
||||||
|
"step": step.id,
|
||||||
|
"action": step.action.name,
|
||||||
|
"surface_used": realization.surface_id,
|
||||||
|
"refused_by_sut": refused,
|
||||||
|
"refusal_expected": step.expect_refusal,
|
||||||
|
"postcondition_met": postcondition_met,
|
||||||
|
},
|
||||||
|
step.id,
|
||||||
|
)
|
||||||
|
|
||||||
|
# --- invariants after every step -----------------------------
|
||||||
|
for invariant in scenario.use_case.invariants:
|
||||||
|
judgments.append(self._oracle.judge(invariant, snapshot, step.id))
|
||||||
|
|
||||||
|
# --- claims attached to this step ----------------------------
|
||||||
|
for claim in claims_by_step.get(step.id, ()):
|
||||||
|
judgments.append(self._oracle.judge(claim, snapshot, step.id))
|
||||||
|
|
||||||
|
pack.verdicts = [j.as_dict() for j in judgments]
|
||||||
|
pack.finished_at = datetime.now(timezone.utc).isoformat()
|
||||||
|
|
||||||
|
result_verdict = overall(judgments)
|
||||||
|
if result_verdict is Verdict.FAIL:
|
||||||
|
pack.energy_events.append(
|
||||||
|
EnergyEvent(asset.id, run_id, EnergyEventType.DEFECT_DETECTED).as_dict()
|
||||||
|
)
|
||||||
|
return RunResult(run_id, result_verdict, judgments, pack)
|
||||||
47
src/testdriver/scenario.py
Normal file
47
src/testdriver/scenario.py
Normal file
|
|
@ -0,0 +1,47 @@
|
||||||
|
"""A scenario binds a use case to concrete actors, a world and a schedule."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from typing import Sequence
|
||||||
|
|
||||||
|
from .actions import SemanticAction
|
||||||
|
from .intent import UseCase
|
||||||
|
from .observers import Watch
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class Step:
|
||||||
|
"""One scheduled semantic action, attributed to one actor."""
|
||||||
|
|
||||||
|
id: str
|
||||||
|
actor_id: str
|
||||||
|
action: SemanticAction
|
||||||
|
expect_refusal: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class Scenario:
|
||||||
|
"""UseCase + Actors + World + Schedule + Surfaces + Variant."""
|
||||||
|
|
||||||
|
id: str
|
||||||
|
use_case: UseCase
|
||||||
|
steps: tuple[Step, ...]
|
||||||
|
watches: tuple[Watch, ...] = field(default_factory=tuple)
|
||||||
|
variant: str = "baseline"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(slots=True)
|
||||||
|
class VerificationAsset:
|
||||||
|
"""A test as a durable thing with identity, maturity and lineage.
|
||||||
|
|
||||||
|
Maturity is the T0..T5 continuum. The kernel produces T5 assets: fully
|
||||||
|
deterministic, no model involvement. Agentic assets (T1) arrive in T07, and
|
||||||
|
crystallization walks an asset from T1 toward T5 in T09.
|
||||||
|
"""
|
||||||
|
|
||||||
|
id: str
|
||||||
|
scenario: Scenario
|
||||||
|
maturity: str = "T5"
|
||||||
|
parent_id: str | None = None
|
||||||
|
adaptation_history: list[dict] = field(default_factory=list)
|
||||||
79
src/testdriver/world.py
Normal file
79
src/testdriver/world.py
Normal file
|
|
@ -0,0 +1,79 @@
|
||||||
|
"""The situation layer: who is acting, and in what state of the world.
|
||||||
|
|
||||||
|
Actor isolation is structural. An Actor holds its own credentials and private
|
||||||
|
memory and has no reference to the Cast or to any sibling. The orchestrator may
|
||||||
|
know the whole world; actors must not learn anything merely because they happen
|
||||||
|
to be executed by the same process.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from typing import Any, Iterator
|
||||||
|
|
||||||
|
|
||||||
|
class ActorIsolationError(Exception):
|
||||||
|
"""Raised when one actor is offered another actor's private state."""
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(slots=True)
|
||||||
|
class Actor:
|
||||||
|
"""An independent execution entity.
|
||||||
|
|
||||||
|
Deliberately holds no back-reference to the Cast or the World. An actor that
|
||||||
|
can enumerate its siblings can leak knowledge it was never given, and no
|
||||||
|
later check can reliably detect that it did.
|
||||||
|
"""
|
||||||
|
|
||||||
|
id: str
|
||||||
|
display_name: str
|
||||||
|
credentials: dict[str, str] = field(default_factory=dict)
|
||||||
|
session: dict[str, Any] = field(default_factory=dict)
|
||||||
|
_memory: dict[str, Any] = field(default_factory=dict, repr=False)
|
||||||
|
|
||||||
|
def remember(self, key: str, value: Any) -> None:
|
||||||
|
self._memory[key] = value
|
||||||
|
|
||||||
|
def recall(self, key: str, default: Any = None) -> Any:
|
||||||
|
return self._memory.get(key, default)
|
||||||
|
|
||||||
|
def known_keys(self) -> tuple[str, ...]:
|
||||||
|
return tuple(sorted(self._memory))
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(slots=True)
|
||||||
|
class Cast:
|
||||||
|
"""The set of actors in a scenario. Known to the orchestrator, not to actors."""
|
||||||
|
|
||||||
|
actors: dict[str, Actor] = field(default_factory=dict)
|
||||||
|
|
||||||
|
def add(self, actor: Actor) -> Actor:
|
||||||
|
if actor.id in self.actors:
|
||||||
|
raise ValueError(f"duplicate actor id {actor.id!r}")
|
||||||
|
self.actors[actor.id] = actor
|
||||||
|
return actor
|
||||||
|
|
||||||
|
def __getitem__(self, actor_id: str) -> Actor:
|
||||||
|
return self.actors[actor_id]
|
||||||
|
|
||||||
|
def __iter__(self) -> Iterator[Actor]:
|
||||||
|
return iter(self.actors.values())
|
||||||
|
|
||||||
|
def __len__(self) -> int:
|
||||||
|
return len(self.actors)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(slots=True)
|
||||||
|
class World:
|
||||||
|
"""Initial state plus the handle to the system under test.
|
||||||
|
|
||||||
|
`seed` is everything needed to rebuild the initial state, so that a run can
|
||||||
|
be replayed from a known starting point rather than from wherever the
|
||||||
|
previous run happened to leave things.
|
||||||
|
"""
|
||||||
|
|
||||||
|
id: str
|
||||||
|
sut: Any
|
||||||
|
sut_version: str
|
||||||
|
seed: dict[str, Any] = field(default_factory=dict)
|
||||||
|
cast: Cast = field(default_factory=Cast)
|
||||||
Binary file not shown.
BIN
tests/__pycache__/test_kernel_guarantees.cpython-312.pyc
Normal file
BIN
tests/__pycache__/test_kernel_guarantees.cpython-312.pyc
Normal file
Binary file not shown.
Binary file not shown.
134
tests/test_kernel_guarantees.py
Normal file
134
tests/test_kernel_guarantees.py
Normal file
|
|
@ -0,0 +1,134 @@
|
||||||
|
"""The kernel must enforce the properties the T02 design claims for it.
|
||||||
|
|
||||||
|
These are unit-level checks on the kernel's construction. The behavioural
|
||||||
|
self-verification suite (actor isolation, oracle independence, evidence
|
||||||
|
reproducibility, intent independence) is TD-WP-0002-T06 and lives separately.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import dataclasses
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from testdriver import (
|
||||||
|
Claim, InadmissibleProvenance, Invariant, Oracle, Provenance, Runner,
|
||||||
|
SemanticAction, StateObserver, SurfaceNotPermitted, Stratum, Verdict,
|
||||||
|
)
|
||||||
|
from lab.minimal import Denied, MinimalLab, ObservationChannel, build_baseline
|
||||||
|
from scenarios.alice_bob_carol import build
|
||||||
|
|
||||||
|
|
||||||
|
def test_implementation_derived_claims_are_rejected():
|
||||||
|
"""D-06: a claim derived from watching the system cannot constrain it."""
|
||||||
|
with pytest.raises(InadmissibleProvenance) as exc:
|
||||||
|
Claim(
|
||||||
|
"c-observed", "whatever the system currently does",
|
||||||
|
Provenance.AGENT_FROM_IMPLEMENTATION,
|
||||||
|
lambda obs: True, after_step="s1",
|
||||||
|
)
|
||||||
|
assert "derived from the implementation" in str(exc.value)
|
||||||
|
|
||||||
|
|
||||||
|
def test_spec_and_human_provenance_are_admissible():
|
||||||
|
for provenance in (Provenance.HUMAN, Provenance.SPEC, Provenance.AGENT_FROM_SPEC):
|
||||||
|
Invariant("i-ok", "fine", provenance, lambda obs: True)
|
||||||
|
|
||||||
|
|
||||||
|
def test_claims_are_frozen():
|
||||||
|
"""D-02: claims are run inputs. No adaptation path may rewrite them."""
|
||||||
|
claim = Claim("c", "text", Provenance.HUMAN, lambda obs: True, after_step="s1")
|
||||||
|
with pytest.raises(dataclasses.FrozenInstanceError):
|
||||||
|
claim.text = "something the implementation would prefer"
|
||||||
|
|
||||||
|
|
||||||
|
def test_surface_substitution_is_refused():
|
||||||
|
"""D-05: achieving an action through an unpermitted surface is not recovery."""
|
||||||
|
action = SemanticAction(
|
||||||
|
"grant_access", {"resource_id": "R"}, permitted_surfaces=frozenset({"browser"})
|
||||||
|
)
|
||||||
|
with pytest.raises(SurfaceNotPermitted):
|
||||||
|
action.check_surface("api")
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_evidence_yields_inconclusive_not_pass():
|
||||||
|
"""An unjudgeable assertion must never default either way."""
|
||||||
|
claim = Claim(
|
||||||
|
"c-needs-missing", "depends on evidence nobody collected",
|
||||||
|
Provenance.HUMAN, lambda obs: obs["probe_read:dave:R"], after_step="s1",
|
||||||
|
)
|
||||||
|
judgment = Oracle().judge(claim, {"probe_read:bob:R": True}, "s1")
|
||||||
|
assert judgment.verdict is Verdict.INCONCLUSIVE
|
||||||
|
assert "missing from snapshot" in judgment.detail["reason"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_inconclusive_outranks_pass_in_the_run_verdict():
|
||||||
|
from testdriver.oracles import Judgment, overall
|
||||||
|
|
||||||
|
assert overall([
|
||||||
|
Judgment("a", "", Verdict.PASS, None),
|
||||||
|
Judgment("b", "", Verdict.INCONCLUSIVE, None),
|
||||||
|
]) is Verdict.INCONCLUSIVE
|
||||||
|
|
||||||
|
|
||||||
|
def test_actors_cannot_be_recorded_as_judgment_collectors():
|
||||||
|
"""Wiring an actor in as an observer must fail loudly, not silently pass."""
|
||||||
|
world, driver, observer, asset, oracle = build()
|
||||||
|
observer.name = "alice" # an actor pretending to be the observer
|
||||||
|
with pytest.raises(Exception) as exc:
|
||||||
|
Runner(world, driver, observer, oracle).run(asset)
|
||||||
|
assert "do not judge their own outcomes" in str(exc.value)
|
||||||
|
|
||||||
|
|
||||||
|
# --- preview of the M05 authorization defect (built properly in T05) --------
|
||||||
|
|
||||||
|
|
||||||
|
class RevokeIsCosmetic(MinimalLab):
|
||||||
|
"""Revocation updates the record and the audit trail but not enforcement.
|
||||||
|
|
||||||
|
This is the shape of a real authorization defect: everything an operator
|
||||||
|
would look at says access was removed, and access was not removed.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def _may_read(self, user_id: str, resource_id: str) -> bool:
|
||||||
|
resource = self.resources.get(resource_id)
|
||||||
|
if resource is None:
|
||||||
|
return False
|
||||||
|
if resource["owner"] == user_id:
|
||||||
|
return True
|
||||||
|
return any(
|
||||||
|
r.event == "grant" and r.subject_id == user_id and r.resource_id == resource_id
|
||||||
|
for r in self.audit
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_seeded_authorization_defect_fails_the_run():
|
||||||
|
"""The kernel must report FAIL, not adapt, when revocation does not revoke."""
|
||||||
|
from testdriver import Actor, Cast, DirectDriver, World
|
||||||
|
from scenarios.alice_bob_carol import USE_CASE
|
||||||
|
from scenarios import alice_bob_carol as ref
|
||||||
|
|
||||||
|
lab = RevokeIsCosmetic(version="lab-0.1.0-M05-auth-defect")
|
||||||
|
tokens = {u: lab.add_user(u) for u in ("alice", "bob", "carol")}
|
||||||
|
cast = Cast()
|
||||||
|
for name in tokens:
|
||||||
|
cast.add(Actor(name, name.title(), credentials={"token": tokens[name]}))
|
||||||
|
|
||||||
|
_, _, _, baseline_asset, _ = ref.build()
|
||||||
|
world = World("w-defect", lab, lab.version, cast=cast)
|
||||||
|
driver = DirectDriver(lab, tokens)
|
||||||
|
observer = StateObserver(ObservationChannel(lab), baseline_asset.scenario.watches)
|
||||||
|
|
||||||
|
result = Runner(world, driver, observer, Oracle()).run(baseline_asset)
|
||||||
|
|
||||||
|
assert result.verdict is Verdict.FAIL
|
||||||
|
assert result.judgment("c-bob-revoked").verdict is Verdict.FAIL
|
||||||
|
# The claim set is untouched by the failure — there is no path to adapt it.
|
||||||
|
assert USE_CASE.claims[2].text == "Bob cannot read R after revocation"
|
||||||
|
|
||||||
|
|
||||||
|
def test_defect_run_emits_an_energy_event():
|
||||||
|
"""Energy events are captured; no score is computed (H-005 is dormant)."""
|
||||||
|
import testdriver.energy as energy
|
||||||
|
|
||||||
|
assert not hasattr(energy, "score")
|
||||||
70
tests/test_reference_scenario.py
Normal file
70
tests/test_reference_scenario.py
Normal file
|
|
@ -0,0 +1,70 @@
|
||||||
|
"""The reference scenario must run deterministically and be replayable."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from testdriver import Runner, Stratum, Verdict
|
||||||
|
from scenarios.alice_bob_carol import build
|
||||||
|
|
||||||
|
|
||||||
|
def run_once(variant: str = "baseline"):
|
||||||
|
world, driver, observer, asset, oracle = build(variant)
|
||||||
|
return Runner(world, driver, observer, oracle).run(asset), world
|
||||||
|
|
||||||
|
|
||||||
|
def test_reference_scenario_passes():
|
||||||
|
result, _ = run_once()
|
||||||
|
assert result.verdict is Verdict.PASS, [
|
||||||
|
j.as_dict() for j in result.judgments if j.verdict is not Verdict.PASS
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_every_claim_is_judged():
|
||||||
|
result, _ = run_once()
|
||||||
|
judged = {j.assertion_id for j in result.judgments}
|
||||||
|
assert {"c-bob-reads", "c-carol-denied", "c-bob-revoked"} <= judged
|
||||||
|
|
||||||
|
|
||||||
|
def test_invariants_are_evaluated_after_every_step():
|
||||||
|
result, _ = run_once()
|
||||||
|
per_step = [j for j in result.judgments if j.assertion_id == "i-audit-append-only"]
|
||||||
|
assert len(per_step) == 3
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_is_replayable_from_known_initial_state():
|
||||||
|
"""Two runs from the same seed produce identical judgments."""
|
||||||
|
first, _ = run_once()
|
||||||
|
second, _ = run_once()
|
||||||
|
assert [(j.assertion_id, j.verdict) for j in first.judgments] == [
|
||||||
|
(j.assertion_id, j.verdict) for j in second.judgments
|
||||||
|
]
|
||||||
|
assert first.run_id != second.run_id
|
||||||
|
|
||||||
|
|
||||||
|
def test_evidence_is_stratified_and_serializable():
|
||||||
|
result, _ = run_once()
|
||||||
|
pack = result.evidence
|
||||||
|
assert pack.of_stratum(Stratum.SURFACE)
|
||||||
|
assert pack.of_stratum(Stratum.REALIZATION)
|
||||||
|
assert pack.of_stratum(Stratum.JUDGMENT)
|
||||||
|
parsed = json.loads(pack.to_json())
|
||||||
|
assert parsed["run_id"] == result.run_id
|
||||||
|
assert parsed["sut_version"] == "lab-0.1.0-baseline"
|
||||||
|
|
||||||
|
|
||||||
|
def test_evidence_records_claim_provenance():
|
||||||
|
"""A verdict must be auditable for the independence of the claim behind it."""
|
||||||
|
result, _ = run_once()
|
||||||
|
assert result.evidence.provenance_index["c-bob-revoked"] == "human"
|
||||||
|
|
||||||
|
|
||||||
|
def test_actors_hold_isolated_credentials_and_memory():
|
||||||
|
_, world = run_once()
|
||||||
|
alice, bob = world.cast["alice"], world.cast["bob"]
|
||||||
|
assert alice.credentials["token"] != bob.credentials["token"]
|
||||||
|
alice.remember("secret", "only alice knows this")
|
||||||
|
assert bob.recall("secret") is None
|
||||||
|
assert bob.known_keys() == ()
|
||||||
|
|
@ -32,7 +32,7 @@ Replace generated placeholders with repo-specific facts where needed.
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: TD-WP-0001-T02
|
id: TD-WP-0001-T02
|
||||||
status: wait
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "39da4237-aaa3-5b4f-93c4-9cbf794f0850"
|
state_hub_task_id: "39da4237-aaa3-5b4f-93c4-9cbf794f0850"
|
||||||
```
|
```
|
||||||
|
|
@ -58,8 +58,9 @@ checkout:
|
||||||
statehub fix-consistency
|
statehub fix-consistency
|
||||||
```
|
```
|
||||||
|
|
||||||
Blocked until the stack exists: no code, no dependency manifest and no test
|
**Done 2026-08-22.** Unblocked by TD-WP-0002-T04. `pyproject.toml` added;
|
||||||
runner are present yet. Unblocks with TD-WP-0002-T04 (deterministic semantic
|
`python3 -m pytest -q` is the whole workflow — stdlib only, no install step.
|
||||||
kernel), which introduces the first Python package and pytest configuration.
|
Commands and the architectural non-negotiables are recorded in `AGENTS.md`
|
||||||
|
under the repo-extensions marker, and in `README.md`.
|
||||||
|
|
||||||
Seeded workplan: `workplans/TD-WP-0002-vertical-spike-crystallization.md`.
|
Seeded workplan: `workplans/TD-WP-0002-vertical-spike-crystallization.md`.
|
||||||
|
|
|
||||||
|
|
@ -176,7 +176,7 @@ Three things worth carrying forward:
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: TD-WP-0002-T04
|
id: TD-WP-0002-T04
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "ffbcc9e8-c1bd-5c50-b62d-495ad9e135a4"
|
state_hub_task_id: "ffbcc9e8-c1bd-5c50-b62d-495ad9e135a4"
|
||||||
```
|
```
|
||||||
|
|
@ -198,6 +198,26 @@ structured Evidence Pack; the scenario replays from known initial state.
|
||||||
|
|
||||||
Emit raw `EnergyEvent` records from this point onward. Implement no scoring.
|
Emit raw `EnergyEvent` records from this point onward. Implement no scoring.
|
||||||
|
|
||||||
|
**Done 2026-08-22.** `src/testdriver/` (11 modules), `lab/minimal.py`,
|
||||||
|
`scenarios/alice_bob_carol.py`, 16 passing tests. The reference scenario runs
|
||||||
|
end to end and replays identically from the same seed; evidence comes out
|
||||||
|
stratified 3/3/3 across S1/S2/S3.
|
||||||
|
|
||||||
|
Three things that came out of building it rather than designing it:
|
||||||
|
|
||||||
|
- **The observation channel needs two probes, not one.** Reading stored state
|
||||||
|
alone verifies test-driver's reimplementation of the rules rather than the
|
||||||
|
system's enforcement of them; probing enforcement alone cannot notice that
|
||||||
|
record and enforcement disagree. The lab exposes both, and their disagreement
|
||||||
|
is now an invariant (`i-enforcement-matches-record`). That invariant is what
|
||||||
|
catches an authorization defect which leaves the audit trail looking correct.
|
||||||
|
- **A seeded `RevokeIsCosmetic` lab already fails the run** — both the claim and
|
||||||
|
the independent invariant fire, and the claim set is provably untouched. Early
|
||||||
|
evidence for H-004, though not yet the experiment.
|
||||||
|
- **Scenarios are Python, not YAML.** Claims are predicates over observations; a
|
||||||
|
YAML dialect able to express them would be a programming language with worse
|
||||||
|
tooling. Revisit once we know which predicates actually recur.
|
||||||
|
|
||||||
## Test-driver lab with labelled ground truth
|
## Test-driver lab with labelled ground truth
|
||||||
|
|
||||||
```task
|
```task
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue