diff --git a/INTENT.md b/INTENT.md index 904b763..e357550 100755 --- a/INTENT.md +++ b/INTENT.md @@ -116,6 +116,14 @@ COLD -> deterministic The framework should make it natural for tests to crystallize as software cools. +> **Under review since T10 — see `research/findings/F-0008-temperature-may-be-redundant.md`.** +> Crystallization was built and demonstrated without Temperature ever being +> consulted. What actually triggered freezing was *measured stability of +> realization* — the same path found across several runs — which is observed, +> not declared. Temperature is a label a human would have to maintain and keep +> honest. If measured stability is sufficient, Temperature is a concept the +> framework does not need. + --- ## Energy @@ -248,10 +256,13 @@ Initial verdicts are: ```text PASS FAIL -SUSPICIOUS INCONCLUSIVE ``` +`SUSPICIOUS` was listed here originally and **removed at T10**: no oracle could +produce it, and a verdict nothing can emit is a promise the framework does not +keep. It returns, with an identifier, if a mechanism ever needs it. + --- ## Lineage diff --git a/README.md b/README.md index 8a3d0d8..d5ce8d3 100644 --- a/README.md +++ b/README.md @@ -25,6 +25,7 @@ No third-party dependencies. Python ≥ 3.11, pytest for the suite. ``` src/testdriver/ the kernel — intent, world, actions, drivers, observers, oracles, evidence, runner +usecases/ durable test intent, including not-yet-runnable use cases lab/ the system under test scenarios/ reference scenarios research/ hypotheses, experiments, findings, fitness map diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index a61b979..c2d070d 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -21,5 +21,5 @@ | task | TD-WP-0002-T06 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T07 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T08 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | -| task | TD-WP-0002-T09 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | +| task | TD-WP-0002-T09 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T10 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | diff --git a/history/2026-08-23-td-wp-0002-gate-review.md b/history/2026-08-23-td-wp-0002-gate-review.md new file mode 100644 index 0000000..e599184 --- /dev/null +++ b/history/2026-08-23-td-wp-0002-gate-review.md @@ -0,0 +1,212 @@ +# TD-WP-0002 — Gate Review and First Compression Pass + +**Date:** 2026-08-23 +**Workplan:** `TD-WP-0002` — Vertical spike: falsify the crystallization thesis +**Tests:** 172 passing · **Lab catalogue:** 24 labelled mutations +**Assessment that set the direction:** `history/2026-08-22-concept-assessment-swot.md` + +--- + +## 1. The gate + +The workplan fixed four criteria in advance, before any code existed. + +### Criterion 1 — recover from M01 and M02 with zero claim or invariant changes + +**Met.** + +| Mutation | Classification | Claim set unchanged | Accepted without a human | +|---|---|---|---| +| M01 sharing control moved into a modal | `MECHANICAL_ADAPTATION` | yes | yes | +| M02 DOM rewritten, test ids dropped | `MECHANICAL_ADAPTATION` | yes | yes | + +The claim set is unchanged *by construction*, not by good behaviour: claims are +run inputs and the adaptation machinery has no write path to them (D-02). + +### Criterion 2 — the authorization defect is reported as a Product Finding + +**Met**, with a naming correction. The workplan inherited "M05" from the original +milestones document; in the catalogue as built, the authorization defect is +**M15** (revocation updates the record but not enforcement). All three +authorization defects behave correctly: + +| Mutation | Classification | Regressions reported | +|---|---|---| +| M15 revoke is cosmetic | `BEHAVIOUR_CHANGED` | `c-bob-revoked`, `i-enforcement-matches-record` | +| M17 anyone can read anything | `BEHAVIOUR_CHANGED` | 5 assertions across 3 steps | +| M20 tenant isolation leaks | `BEHAVIOUR_CHANGED` | 5 assertions across 3 steps | + +None is safe to accept. The classifier collapses `PRODUCT_DEFECT` and +`SEMANTIC_CHANGE` into one escalating outcome — see F-0006 — so "Product Finding" +in the original wording is now "escalated to a human with the regressions named". + +### Criterion 3 — one asset crystallizes into a deterministic test + +**Met.** `crystallized/test_grant_access.py` is generated, runs with **no model +involvement**, preserves its ancestor's oracle set as a superset, agrees with the +ancestor across five lab versions, still catches a seeded defect, and carries its +lineage in its own docstring. + +### Criterion 4 — False Adaptation Rate = 0 + +**Met. 0 / 7.** No `DEFECT`-labelled mutation was classified as safe to accept, +including the three E-003 attacks (surface substitution, concurrent +mechanical+defect, evidence starvation) and provenance laundering. + +The other side of the trade is asserted too: **12 of 13** mechanical mutations are +absorbed without a human. A classifier that escalated everything would score a +perfect FAR and be worthless. + +### Gate verdict + +**Passed, on all four criteria.** With one qualification that belongs in the same +sentence: FAR = 0 follows largely from *architecture* — claims are run inputs, +and `SAFE_TO_ACCEPT` is a closed two-element set — not from classifier accuracy. +The experiment shows the architecture behaves as designed over 24 hand-written +mutations. It says nothing about mutations nobody thought of. + +--- + +## 2. What the evidence changed + +Three of the project's own claims were narrowed or corrected by building them. +This is the part worth more than the green gate. + +**H-001 was narrowed (F-0005).** Semantic actions do not generally outlast +recorded interaction sequences. Where an application keeps stable identifiers, +the two arms tie exactly — 9/9 against 9/9 — and the conventional approach is +cheaper and deterministic. Semantic actions earn their keep only where +identifiers are absent or not carried forward (2/3 against 0/3). The concept +model presents this as a general superiority; it is conditional. + +**The classifier cannot infer intent (F-0006).** The T02 design listed +`SEMANTIC_CHANGE` as an outcome the decision table could produce. It cannot: a +deliberate product decision and a defect are behaviourally identical, as M12 and +M19 have asserted since T05. The two collapse into `BEHAVIOUR_CHANGED` and the +distinction becomes a recorded human adjudication. Erring toward false alarm +costs one review; erring toward false adaptation ends the project. + +**Crystallization's economic case is unmeasured (F-0007).** The descendant is 54% +cheaper than its ancestor, and that number must not be quoted in support of the +thesis. The runtime is token-free by design, so the entire saving is one page +fetch, one parse and a two-candidate scoring pass. The saving the concept +actually claims — model tokens, latency, retry variance — is absent from the +measurement. + +--- + +## 3. The compression pass + +### Which concepts have no supporting evidence? + +`Temperature`, `Confidence`, `Campaign`, `Metabolism`, `Retirement` — declared, +unimplemented, never consulted by any decision. + +`Temperature` is the decidable one and is now **F-0008**. Crystallization was +built and demonstrated without it: what triggered freezing was *measured +stability of realization*, which is observed rather than declared. A declared +temperature is a claim nobody verifies, and the project's own first heuristic +argues against it. Not deleted yet — two unevidenced cases for keeping it are +recorded — but gated: if the next workplan completes without a decision +consulting it, it goes. + +### Which implementation abstractions have no conceptual justification? + +Removed at T10: + +| Removed | Why | +|---|---| +| `Verdict.SUSPICIOUS` | No oracle could produce it. A verdict nothing can emit is a promise not kept. | +| `Step.expect_refusal` | Never set by any scenario. Speculative generality. | +| `ActorIsolationError` | Declared, never raised, never caught. | +| `World.seed` | Written at T04, never read. Replay comes from rebuilding the lab. | +| `EvidencePack.latest()` | Never called. | +| `Trajectory.method` | Computed on every capture, never read. | + +### Which metadata has not informed a decision? + +**`EnergyEvent` history.** Events have been recorded since T04 and have informed +nothing — as designed, since H-005 is dormant. Capture is retained because +history cannot be reconstructed later while scores always can, but this is now on +the same gate as Temperature: if the next workplan ends with no decision having +used it, the module goes. + +### Which agentic behaviour can now be deterministic? + +The `grant_access` realization, and it has been — that is T09. Nothing else in +the spike is agentic. + +### What have the experiments falsified? + +Nothing outright. Two claims were **narrowed** (H-001, and the classifier's +output space), and one criterion was met in a sense **too weak to support the +thesis** (H-003's cost clause). No hypothesis was rejected, and none was promoted +past `EXPERIMENTING`: 24 hand-written mutations is a small, self-chosen sample. + +### Two outcomes that had never run + +`INTENT_CHANGED` and `REALIZATION_FAILED` were declared at T08 and produced by no +test. Code that has never run is code nobody has checked. Rather than delete +meaningful outcomes or trust them untested, both now have purpose-built cases, +and `test_no_classification_is_unreachable` fails if a seventh is ever added +without one. + +Building the `REALIZATION_FAILED` case clarified a boundary worth keeping: a run +that asserts *nothing at all* is `AMBIGUOUS`, not `REALIZATION_FAILED` — a use +case with no claims cannot conclude anything, however well its steps ran. + +--- + +## 4. The fitness loop, closed + +The milestone asks that a framework finding become a permanent guarantee rather +than a note. **F-0003 is resolved.** + +The finding: actor isolation left no trace in ordinary evidence. The self-test +caught a shared memory store only because the harness planted per-actor secrets +first. Isolation was a property of *scenarios written to expose it*, not of runs. + +The fix: every `Actor` is seeded at construction with an automatic private +marker, and the runner examines all actors on **every** scenario, recording the +verdict as S3 evidence. Two regressions now stand behind it — +`test_every_run_records_a_verdict_on_isolation`, and a self-check that catches a +leak with the harness planting nothing at all. + +``` +Framework Finding (F-0003) + → classified FRAMEWORK_LIMITATION + → improvement hypothesis: automatic canaries make isolation observable + → implemented at T10 + → permanent self-verification + deterministic regression + → fitness map updated +``` + +That is the first complete Concept–Implementation Fitness Loop, which the +milestones document calls the point at which the self-improvement system +genuinely exists. + +--- + +## 5. Where this leaves the project + +**The thesis survived contact with its own evidence, in narrowed form.** The +distinctive claim — that verification can adapt to a changing surface without +adapting to a defect — is demonstrated, and the demonstration is architectural +rather than statistical, which is the stronger kind. + +**The single highest-value next step is a bounded live-model experiment.** Two +independent findings converge on it: M22 defeats the heuristic runtime while +remaining solvable by reading a visible label (F-0005), and crystallization's +economic case cannot be measured without token costs (F-0007). One experiment +settles whether a model adds capability *and* whether crystallization has a +product rationale. The `RealizationMetrics` fields were populated from the first +run so that comparison is a subtraction rather than a re-run. + +That should come before any further framework feature. + +**Open findings carried forward:** F-0003 resolved · F-0004 (structural +durability only; visual relayout untested) · F-0005 (H-001 narrowed) · F-0007 +(cost unmeasurable; generated tests not fully standalone) · F-0008 (Temperature +gated). The F-0001 residual — the taxonomy has no class for document-to-document +drift — was considered and deliberately not resolved: adding a finding class to a +taxonomy already larger than its evidence would be the opposite of compression. diff --git a/research/concepts/fitness-map.md b/research/concepts/fitness-map.md index c8c4c75..32a0046 100644 --- a/research/concepts/fitness-map.md +++ b/research/concepts/fitness-map.md @@ -1,6 +1,6 @@ # Concept ↔ Implementation Fitness Map -**Updated:** 2026-08-23 (TD-WP-0002-T09) +**Updated:** 2026-08-23 (TD-WP-0002-T10) Traces each important concept to the implementation, experiment and evidence that support it. **Unsupported entries are the point of this map** — a concept with no @@ -29,7 +29,7 @@ were aspirational, not evidenced. | Concept | Level | Implementation | Experiment | Evidence | Open question | |---|---|---|---|---|---| | `C-use-case` | C1 | `intent.py` | — | — | Is a use case expressible without leaking mechanics? | -| `C-actor-isolation` | C1 | `world.py` | E-001 | `td://self/actor-isolation` | **F-0003** — only observable when the scenario plants canaries. | +| `C-actor-isolation` | **C2** | `world.py`, `runner.py` | E-001 | `td://self/actor-isolation`, examined every run | **F-0003 resolved** — automatic canaries; observable on every scenario. | | `C-semantic-action` | **C2** | `actions.py`, `agentic.py` | E-001 (partial) | T07 arm comparison | **F-0005** — supported only where stable identifiers are absent. Narrower than the concept model claims. | | `C-oracle-independence` | **C2** | `runner.py`, `oracles.py` | E-001, E-003 | — | Independence of components ≠ independence of belief. (H-004) | | `C-evidence-pack` | C1 | `evidence.py` | — | `td://self/evidence-reproducibility` | Verdicts are reproducible from S3 alone, on passing and failing runs. | @@ -39,9 +39,9 @@ were aspirational, not evidenced. | `C-crystallization` | **C2** | `crystallization.py` | E-002 | T09 agreement matrix | Fidelity supported; **F-0007** — the economic case is unmeasurable with a token-free runtime. | | `C-intent-provenance` | C1 | `provenance.py` | E-003 | `td://self/intent-independence` | Constrains provenance, not quality. Accepted residual. | | `C-lineage` | C1 | `scenario.py`, generated headers | E-002 | generated module docstring | Parent pointer plus provenance in the artefact; no graph. | -| `C-energy` | C0 | `energy.py`, capture only | — | — | Dormant by decision. (H-005) | -| `C-temperature` | C0 | — | — | — | Deferred. No implementation planned in TD-WP-0002. | -| `C-confidence` | C0 | — | — | — | Deferred. | +| `C-energy` | C0 | `energy.py`, capture only | — | none | Dormant. **Gated**: if the next workplan ends with no decision having used the history, remove. | +| `C-temperature` | **C0 — under review** | — | — | none | **F-0008** — crystallization was built without it; measured stability did the work. Gated for removal. | +| `C-confidence` | C0 | — | — | none | Declared, unimplemented, never consulted. Same position as `C-temperature` without a built alternative. | | `C-campaign` | C0 | — | — | — | Deferred. | | `C-metabolism` | C0 | — | — | — | Deferred. Depends on C-energy. | | `C-retirement` | C0 | — | — | — | Deferred. Depends on C-energy. | @@ -58,6 +58,13 @@ validated. They are revisited at T10, where the question is not "when do we buil these" but "does the evidence justify keeping them in the model at all". **Implementation orphans** — none. Every module in `src/testdriver/` traces to a -concept above. `energy.py` is the one to watch: it exists solely to capture -events for a dormant hypothesis, and if T10 finds no use for the history it -should be removed rather than kept out of sentiment. +concept above. + +**Removed at T10** (compression pass — see the gate review § 3): +`Verdict.SUSPICIOUS`, `Step.expect_refusal`, `ActorIsolationError`, +`World.seed`, `EvidencePack.latest()`, `Trajectory.method`. Each was declared and +never used; `SUSPICIOUS` was additionally a verdict no oracle could emit. + +**Gated for removal**: `C-temperature` (F-0008) and `energy.py`. Both survive the +current review on the strength of being cheap, not of being used. If the next +workplan closes without a decision consulting either, they go. diff --git a/research/findings/F-0003-isolation-needs-canaries.md b/research/findings/F-0003-isolation-needs-canaries.md index 008f659..d0c93b8 100644 --- a/research/findings/F-0003-isolation-needs-canaries.md +++ b/research/findings/F-0003-isolation-needs-canaries.md @@ -2,12 +2,13 @@ id: F-0003 type: framework-finding class: FRAMEWORK_LIMITATION -status: open +status: resolved discovered: "2026-08-22" discovered_by: TD-WP-0002-T06 workplan: TD-WP-0002 task: TD-WP-0002-T06 -carried_to: TD-WP-0002-T10 +resolved: "2026-08-23" +resolved_in: TD-WP-0002-T10 --- # F-0003 — Actor isolation is only observable if the scenario plants canaries @@ -55,9 +56,31 @@ system. Worth stating plainly rather than counting four green self-tests as four equivalent proofs. They are not equivalent. -## Candidate resolutions +## Resolution (T10) -Not resolved now; deciding cheaply here would be guessing. +Candidate 3 — **both**. + +Every `Actor` is seeded at construction with an automatic private marker +(`Actor.canary`), and `Runner._isolation_violations` examines every actor on +**every** scenario, recording the verdict as S3 evidence. An isolation violation +now leaves a trace whether or not anyone thought to look for one. + +Two permanent regressions stand behind it: + +- `tests/test_reference_scenario.py::test_every_run_records_a_verdict_on_isolation` +- `tests/selfverification/test_checks_can_fail.py::test_a_leak_is_caught_without_the_test_planting_anything` + — an actor holding another's marker is caught by the ordinary run, with the + harness planting nothing. + +The second limitation stands as recorded: oracle independence is enforced by +construction, so it is verified once rather than per run. That asymmetry is real +and is now stated in the fitness map rather than hidden behind two equally green +self-tests. + +This finding closed the first Concept–Implementation Fitness Loop — see +`history/2026-08-23-td-wp-0002-gate-review.md` § 4. + +## Candidate resolutions considered 1. **Automatic canaries** — the runtime seeds each actor with a unique private marker at construction and the isolation check runs on every Evidence Pack, diff --git a/research/findings/F-0008-temperature-may-be-redundant.md b/research/findings/F-0008-temperature-may-be-redundant.md new file mode 100644 index 0000000..a867ce3 --- /dev/null +++ b/research/findings/F-0008-temperature-may-be-redundant.md @@ -0,0 +1,70 @@ +--- +id: F-0008 +type: framework-finding +class: UNNECESSARY_COMPLEXITY +status: open +discovered: "2026-08-23" +discovered_by: TD-WP-0002-T10 +workplan: TD-WP-0002 +task: TD-WP-0002-T10 +carried_to: next workplan +--- + +# F-0008 — Temperature may be redundant; measured stability did the work + +## Observation + +`Temperature` (`HOT` / `WARM` / `COOL` / `COLD`) is one of the more prominent +ideas in `INTENT.md` and the Concept Model. It is supposed to govern which +verification mode an asset uses, and it is the mechanism by which "tests +crystallize as software cools". + +The spike built crystallization end to end. **Temperature was never consulted, +never implemented, and never missed.** + +What actually triggered freezing was `assess_stability`: the same realization +path observed across several consecutive runs. Nothing declared the surface cold; +the surface was observed to have stopped moving. + +## The argument for removal + +The two mechanisms answer the same question, and only one of them can be wrong: + +| | Temperature | Measured stability | +|---|---|---| +| source | declared by a human | observed from runs | +| upkeep | must be maintained and kept honest | none | +| failure mode | says `COLD` while the surface churns | none — it is the churn | +| already built | no | yes | + +A declared temperature is a claim about the system that nobody verifies. The +project's own first heuristic — *do not let a declaration substitute for +evidence* — argues against keeping it. + +## The argument for keeping it + +Two cases measured stability cannot cover: + +1. **Forward-looking scope.** A team knows a rewrite lands next month. + Temperature can say `HOT` before any run has observed instability, where + measured stability would happily freeze an asset the day before it breaks. +2. **Cheap prioritisation.** Deciding *where to spend effort* across many + capabilities may want a coarse label without running anything. + +Neither case arose in the spike, and neither is evidenced. + +## Recommendation + +Do not delete yet; do not build either. Carry `Temperature` as **explicitly +unvalidated** in the fitness map, and set a gate: if the next workplan completes +without any decision consulting it, remove it from the concept model. + +Recorded now because the compression review's purpose is to catch concepts that +survive on the strength of being attractive rather than useful, and Temperature +is currently the clearest instance in the corpus. + +## Related + +`Confidence`, `Campaign`, `Metabolism` and `Retirement` are in the same position — +declared, unimplemented, never consulted — but Temperature is the one with a +built alternative already doing its job, which makes it the decidable case. diff --git a/scenarios/__pycache__/alice_bob_carol.cpython-312.pyc b/scenarios/__pycache__/alice_bob_carol.cpython-312.pyc index 9053e9f..619b306 100644 Binary files a/scenarios/__pycache__/alice_bob_carol.cpython-312.pyc and b/scenarios/__pycache__/alice_bob_carol.cpython-312.pyc differ diff --git a/scenarios/alice_bob_carol.py b/scenarios/alice_bob_carol.py index 70bcfa7..e9c10a8 100644 --- a/scenarios/alice_bob_carol.py +++ b/scenarios/alice_bob_carol.py @@ -128,9 +128,7 @@ def build(*mutations: str): cast.add(Actor(id=name, display_name=name.title(), credentials={"token": tokens[name]})) - world = World(id="w-baseline", sut=lab, sut_version=lab.version, - seed={"users": ["alice", "bob", "carol"], "resource": RESOURCE}, - cast=cast) + world = World(id="w-baseline", sut=lab, sut_version=lab.version, cast=cast) scenario = Scenario( id="sc-share-resource", diff --git a/src/testdriver/__init__.py b/src/testdriver/__init__.py index d56c3ca..31418a5 100644 --- a/src/testdriver/__init__.py +++ b/src/testdriver/__init__.py @@ -10,10 +10,10 @@ from .oracles import Judgment, Oracle, Verdict, overall from .provenance import InadmissibleProvenance, Provenance from .runner import CollectorIndependenceError, Runner, RunResult from .scenario import Scenario, Step, VerificationAsset -from .world import Actor, ActorIsolationError, Cast, World +from .world import Actor, Cast, World __all__ = [ - "Actor", "ActorIsolationError", "Cast", "Claim", "CollectorIndependenceError", + "Actor", "Cast", "Claim", "CollectorIndependenceError", "DirectDriver", "EnergyEvent", "EnergyEventType", "EvidencePack", "InadmissibleProvenance", "Invariant", "Judgment", "Observation", "Oracle", "Provenance", "Realization", "RunResult", "Runner", "Scenario", diff --git a/src/testdriver/__pycache__/__init__.cpython-312.pyc b/src/testdriver/__pycache__/__init__.cpython-312.pyc index 216faa6..833fb3f 100644 Binary files a/src/testdriver/__pycache__/__init__.cpython-312.pyc and b/src/testdriver/__pycache__/__init__.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/crystallization.cpython-312.pyc b/src/testdriver/__pycache__/crystallization.cpython-312.pyc index 6e999c0..9eee1fb 100644 Binary files a/src/testdriver/__pycache__/crystallization.cpython-312.pyc and b/src/testdriver/__pycache__/crystallization.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/evidence.cpython-312.pyc b/src/testdriver/__pycache__/evidence.cpython-312.pyc index 887c8f1..c198624 100644 Binary files a/src/testdriver/__pycache__/evidence.cpython-312.pyc and b/src/testdriver/__pycache__/evidence.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/oracles.cpython-312.pyc b/src/testdriver/__pycache__/oracles.cpython-312.pyc index 7980776..d8a11d5 100644 Binary files a/src/testdriver/__pycache__/oracles.cpython-312.pyc and b/src/testdriver/__pycache__/oracles.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/runner.cpython-312.pyc b/src/testdriver/__pycache__/runner.cpython-312.pyc index 48c6c66..d9431ba 100644 Binary files a/src/testdriver/__pycache__/runner.cpython-312.pyc and b/src/testdriver/__pycache__/runner.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/scenario.cpython-312.pyc b/src/testdriver/__pycache__/scenario.cpython-312.pyc index 8bf7e17..3ef1d60 100644 Binary files a/src/testdriver/__pycache__/scenario.cpython-312.pyc and b/src/testdriver/__pycache__/scenario.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/world.cpython-312.pyc b/src/testdriver/__pycache__/world.cpython-312.pyc index 6762964..7b050fb 100644 Binary files a/src/testdriver/__pycache__/world.cpython-312.pyc and b/src/testdriver/__pycache__/world.cpython-312.pyc differ diff --git a/src/testdriver/crystallization.py b/src/testdriver/crystallization.py index 7897e2c..6e90483 100644 --- a/src/testdriver/crystallization.py +++ b/src/testdriver/crystallization.py @@ -35,7 +35,6 @@ class Trajectory: step_id: str action_name: str surface_id: str - method: str target: str fields: tuple[str, ...] @@ -43,7 +42,7 @@ class Trajectory: return json.dumps( { "step": self.step_id, "action": self.action_name, - "surface": self.surface_id, "method": self.method, + "surface": self.surface_id, "target": self.target, "fields": sorted(self.fields), }, sort_keys=True, @@ -63,7 +62,6 @@ def capture(pack: Mapping[str, Any]) -> tuple[Trajectory, ...]: step_id=obs["step_id"], action_name=str(mechanics.get("action", "")).split("(")[0], surface_id=obs["data"].get("surface", ""), - method=("POST" if obs["data"].get("surface") == "browser" else "CALL"), target=target, fields=tuple(sorted(fields)), )) diff --git a/src/testdriver/evidence.py b/src/testdriver/evidence.py index 01a65a8..bdbc78d 100644 --- a/src/testdriver/evidence.py +++ b/src/testdriver/evidence.py @@ -72,12 +72,6 @@ class EvidencePack: def of_stratum(self, stratum: Stratum) -> list[Observation]: return [o for o in self.observations if o.stratum is stratum] - def latest(self, kind: str) -> Observation | None: - for observation in reversed(self.observations): - if observation.kind == kind: - return observation - return None - def to_json(self) -> str: payload = asdict(self) payload["observations"] = [ diff --git a/src/testdriver/oracles.py b/src/testdriver/oracles.py index 705b1ae..17f1c20 100644 --- a/src/testdriver/oracles.py +++ b/src/testdriver/oracles.py @@ -3,6 +3,10 @@ An oracle reads only the independent observation snapshot. It has no access to the actor, to the driver, or to what either of them believes happened. +`SUSPICIOUS` was removed at T10: no oracle could produce it, and a verdict +nothing can emit is a promise the framework does not keep. It returns, with an +identifier, if a mechanism ever needs it. + `INCONCLUSIVE` is a first-class outcome, not a failure mode of the framework. An oracle that cannot support a judgment from the retained evidence must say so rather than defaulting either way — silently defaulting to PASS hides defects, @@ -21,7 +25,6 @@ from .intent import Claim, Invariant class Verdict(str, Enum): PASS = "PASS" FAIL = "FAIL" - SUSPICIOUS = "SUSPICIOUS" INCONCLUSIVE = "INCONCLUSIVE" @@ -101,6 +104,4 @@ def overall(judgments: list[Judgment]) -> Verdict: return Verdict.FAIL if Verdict.INCONCLUSIVE in verdicts: return Verdict.INCONCLUSIVE - if Verdict.SUSPICIOUS in verdicts: - return Verdict.SUSPICIOUS return Verdict.PASS diff --git a/src/testdriver/runner.py b/src/testdriver/runner.py index 833eab0..9aeb9e1 100644 --- a/src/testdriver/runner.py +++ b/src/testdriver/runner.py @@ -56,6 +56,24 @@ class Runner: # -- independence guards --------------------------------------------- + def _isolation_violations(self) -> list[str]: + """Does any actor hold another's canary? + + Run on every scenario, not only on ones written to test isolation. + """ + canaries = {actor.canary: actor.id for actor in self._world.cast} + violations: list[str] = [] + for actor in self._world.cast: + for key in actor.known_keys(): + value = actor.recall(key) + owner = canaries.get(value) if isinstance(value, str) else None + if owner is not None and owner != actor.id: + violations.append( + f"actor {actor.id!r} holds the private marker of {owner!r} " + f"under key {key!r}" + ) + return violations + def _assert_collector_independence(self, stratum: Stratum, collector: str) -> None: """S2 and S3 evidence may never be attributed to an actor. @@ -111,6 +129,12 @@ class Runner: EnergyEvent(asset.id, run_id, EnergyEventType.EXECUTED).as_dict() ) + isolation = self._isolation_violations() + self._record( + pack, Stratum.JUDGMENT, self._observer.name, "actor_isolation", + {"violations": isolation, "actors": sorted(self._world.cast.actors)}, None, + ) + judgments: list[Judgment] = [] scenario_sound = True claims_by_step: dict[str, list] = {} @@ -166,7 +190,6 @@ class Runner: "action": step.action.name, "surface_used": realization.surface_id, "refused_by_sut": refused, - "refusal_expected": step.expect_refusal, "postcondition_met": postcondition_met, }, step.id, @@ -176,7 +199,7 @@ class Runner: # An action that was accepted but did not take effect did happen — # and that is a statement about the system, judged below, not a # reason to stop judging. - if refused and not step.expect_refusal: + if refused: scenario_sound = False # --- invariants after every step ----------------------------- diff --git a/src/testdriver/scenario.py b/src/testdriver/scenario.py index a41de6e..c4ac795 100644 --- a/src/testdriver/scenario.py +++ b/src/testdriver/scenario.py @@ -17,7 +17,6 @@ class Step: id: str actor_id: str action: SemanticAction - expect_refusal: bool = False @dataclass(frozen=True, slots=True) diff --git a/src/testdriver/world.py b/src/testdriver/world.py index 04f2341..55289c8 100644 --- a/src/testdriver/world.py +++ b/src/testdriver/world.py @@ -8,12 +8,20 @@ to be executed by the same process. from __future__ import annotations +import uuid from dataclasses import dataclass, field from typing import Any, Iterator -class ActorIsolationError(Exception): - """Raised when one actor is offered another actor's private state.""" +def _canary() -> str: + """A value private to one actor, unguessable and unique per construction. + + Exists so that isolation is *observable*. Without it a run in which every + actor shares one memory store produces evidence indistinguishable from a + correct one — the guarantee holds only in scenarios written to expose it, + which is no guarantee at all (F-0003). + """ + return f"canary-{uuid.uuid4().hex}" @dataclass(slots=True) @@ -28,8 +36,13 @@ class Actor: id: str display_name: str credentials: dict[str, str] = field(default_factory=dict) - session: dict[str, Any] = field(default_factory=dict) _memory: dict[str, Any] = field(default_factory=dict, repr=False) + canary: str = field(default_factory=_canary) + + def __post_init__(self) -> None: + # Seeded automatically, on every actor, in every scenario. An isolation + # violation now leaves a trace whether or not anyone thought to look. + self._memory.setdefault("__canary__", self.canary) def remember(self, key: str, value: Any) -> None: self._memory[key] = value @@ -65,15 +78,14 @@ class Cast: @dataclass(slots=True) class World: - """Initial state plus the handle to the system under test. + """The handle to the system under test, plus the cast acting on it. - `seed` is everything needed to rebuild the initial state, so that a run can - be replayed from a known starting point rather than from wherever the - previous run happened to leave things. + Replay comes from rebuilding the lab through `build_lab`, not from a seed + dict carried here — the dict was written at T04, never read, and removed at + T10. """ id: str sut: Any sut_version: str - seed: dict[str, Any] = field(default_factory=dict) cast: Cast = field(default_factory=Cast) diff --git a/tests/__pycache__/test_audit_core_e2_use_case.cpython-312-pytest-7.4.4.pyc b/tests/__pycache__/test_audit_core_e2_use_case.cpython-312-pytest-7.4.4.pyc new file mode 100644 index 0000000..78d416e Binary files /dev/null and b/tests/__pycache__/test_audit_core_e2_use_case.cpython-312-pytest-7.4.4.pyc differ diff --git a/tests/__pycache__/test_classification.cpython-312-pytest-7.4.4.pyc b/tests/__pycache__/test_classification.cpython-312-pytest-7.4.4.pyc index ba46d32..e875d95 100644 Binary files a/tests/__pycache__/test_classification.cpython-312-pytest-7.4.4.pyc and b/tests/__pycache__/test_classification.cpython-312-pytest-7.4.4.pyc differ diff --git a/tests/__pycache__/test_reference_scenario.cpython-312-pytest-7.4.4.pyc b/tests/__pycache__/test_reference_scenario.cpython-312-pytest-7.4.4.pyc index 9f5843b..d39b6dd 100644 Binary files a/tests/__pycache__/test_reference_scenario.cpython-312-pytest-7.4.4.pyc and b/tests/__pycache__/test_reference_scenario.cpython-312-pytest-7.4.4.pyc differ diff --git a/tests/selfverification/__pycache__/checks.cpython-312.pyc b/tests/selfverification/__pycache__/checks.cpython-312.pyc index 3c615c3..b46525b 100644 Binary files a/tests/selfverification/__pycache__/checks.cpython-312.pyc and b/tests/selfverification/__pycache__/checks.cpython-312.pyc differ diff --git a/tests/selfverification/__pycache__/test_checks_can_fail.cpython-312-pytest-7.4.4.pyc b/tests/selfverification/__pycache__/test_checks_can_fail.cpython-312-pytest-7.4.4.pyc index 07825a6..003d529 100644 Binary files a/tests/selfverification/__pycache__/test_checks_can_fail.cpython-312-pytest-7.4.4.pyc and b/tests/selfverification/__pycache__/test_checks_can_fail.cpython-312-pytest-7.4.4.pyc differ diff --git a/tests/selfverification/__pycache__/test_self_verification.cpython-312-pytest-7.4.4.pyc b/tests/selfverification/__pycache__/test_self_verification.cpython-312-pytest-7.4.4.pyc index fee743c..f9e0cff 100644 Binary files a/tests/selfverification/__pycache__/test_self_verification.cpython-312-pytest-7.4.4.pyc and b/tests/selfverification/__pycache__/test_self_verification.cpython-312-pytest-7.4.4.pyc differ diff --git a/tests/selfverification/checks.py b/tests/selfverification/checks.py index 47d321e..2329882 100644 --- a/tests/selfverification/checks.py +++ b/tests/selfverification/checks.py @@ -94,6 +94,23 @@ def check_actor_isolation( return violations +def check_isolation_was_examined(pack: Mapping[str, Any]) -> list[str]: + """Every run must carry a verdict on actor isolation — F-0003, resolved. + + Before this existed, isolation was only observable in scenarios written to + expose it: a run in which every actor shared one memory store produced + evidence indistinguishable from a correct one. Actors now carry an automatic + private marker and the runner examines them on every scenario, so the absence + of this observation is itself a failure. + """ + for obs in _observations(pack): + if obs["kind"] != "actor_isolation": + continue + violations = obs["data"].get("violations") or [] + return [f"actor isolation violated: {v}" for v in violations] + return ["this run did not examine actor isolation at all"] + + # --- td://self/oracle-independence --------------------------------------- diff --git a/tests/selfverification/test_checks_can_fail.py b/tests/selfverification/test_checks_can_fail.py index c727c0d..b131778 100644 --- a/tests/selfverification/test_checks_can_fail.py +++ b/tests/selfverification/test_checks_can_fail.py @@ -30,6 +30,7 @@ from tests.selfverification.checks import ( check_actor_isolation, check_evidence_supports_every_verdict, check_intent_independence, + check_isolation_was_examined, check_no_actor_collected_judgment, check_runs_agree, check_verdicts_follow_from_judgment_evidence, @@ -186,3 +187,35 @@ def test_unrecorded_provenance_is_caught(): tampered = copy.deepcopy(pack) tampered["provenance_index"] = {} assert check_intent_independence(tampered) + + +# --- F-0003 resolution: isolation observed on every run ------------------- + + +def test_an_unexamined_run_is_caught(): + """A run that never looked at isolation must not read as isolated.""" + pack, _, _ = run_and_serialize() + tampered = copy.deepcopy(pack) + tampered["observations"] = [ + o for o in tampered["observations"] if o["kind"] != "actor_isolation" + ] + assert check_isolation_was_examined(tampered) + + +def test_a_leak_is_caught_without_the_test_planting_anything(): + """The regression F-0003 leaves behind. + + No secrets seeded by the harness, no scenario written to expose isolation. + An actor holding another's automatic marker is caught by the ordinary run. + """ + from testdriver import Oracle, Runner + from scenarios.alice_bob_carol import build + + world, driver, observer, asset, oracle = build() + world.cast["bob"].remember("overheard", world.cast["alice"].canary) + result = Runner(world, driver, observer, oracle).run(asset) + pack = json.loads(result.evidence.to_json()) + + violations = check_isolation_was_examined(pack) + assert violations + assert "holds the private marker of 'alice'" in violations[0] diff --git a/tests/selfverification/test_self_verification.py b/tests/selfverification/test_self_verification.py index 506eb79..3b06899 100644 --- a/tests/selfverification/test_self_verification.py +++ b/tests/selfverification/test_self_verification.py @@ -17,6 +17,7 @@ from tests.selfverification.checks import ( check_actor_isolation, check_evidence_supports_every_verdict, check_intent_independence, + check_isolation_was_examined, check_no_actor_collected_judgment, check_runs_agree, check_verdicts_follow_from_judgment_evidence, @@ -56,6 +57,14 @@ def test_actor_isolation(): assert check_actor_isolation(pack, SECRETS, memories) == [] +@pytest.mark.parametrize("mutations", [(), ("M15",), ("M01",)]) +def test_every_run_examines_isolation(mutations): + """F-0003 resolved: isolation is now a property of runs, not of scenarios + written to expose it. No canaries need be planted by the test.""" + pack, _, _ = run_and_serialize(*mutations) + assert check_isolation_was_examined(pack) == [] + + # --- td://self/oracle-independence --------------------------------------- diff --git a/tests/test_audit_core_e2_use_case.py b/tests/test_audit_core_e2_use_case.py new file mode 100644 index 0000000..7079f4c --- /dev/null +++ b/tests/test_audit_core_e2_use_case.py @@ -0,0 +1,136 @@ +"""The audit-core use case is durable intent, even before drivers can run it.""" + +from __future__ import annotations + +from copy import deepcopy + +from testdriver import Oracle, Provenance, Verdict +from usecases.audit_core_e2_tenant_boundary import ( + PHASE_CONTRACTS, + PRECEDENT_EVIDENCE_REF, + ROLE_CONTRACTS, + TEST_USE_CASE, +) + + +def passing_observations() -> dict[str, object]: + absent = { + "status": 404, + "schema": ("$", "$.error:str"), + "digest": "absent-surface", + "fixture_match_count": 0, + } + return { + "event_by_id": { + "owner": {"status": 200, "fixture_match_count": 2}, + "attacker": dict(absent), + "absent": dict(absent), + }, + "correlation_slice": { + "owner": {"status": 200, "fixture_match_count": 2}, + "attacker": {"status": 200, "fixture_match_count": 0}, + }, + "append_as_b": { + "attacker": {"status": 400}, + "state_after": dict(absent), + "absent": dict(absent), + }, + "run": { + "attempted_operations": 10, + "maximum_operations": 30, + "max_concurrency": 1, + "rate_per_minute": 10, + "target_revision_matches": True, + "attribution_complete": True, + }, + "cleanup": { + "completed_before_expiry": True, + "runner_absent": True, + "temporary_identities": [], + "kv_paths": [], + "projection_resources": [], + "mounted_secrets": [], + "target_ready": True, + "secret_values_observed": False, + }, + "report": { + "bound_to_projection_receipt": True, + "bound_to_cleanup_receipt": True, + "bounded_assurance_statement": True, + "delivered_to_risk_nexus": True, + }, + } + + +def judgments(observations): + oracle = Oracle() + assertions = (*TEST_USE_CASE.claims, *TEST_USE_CASE.invariants) + return { + assertion.id: oracle.judge( + assertion, + observations, + getattr(assertion, "after_step", None), + ) + for assertion in assertions + } + + +def test_precedent_calibration_satisfies_every_assertion(): + result = judgments(passing_observations()) + assert {judgment.verdict for judgment in result.values()} == {Verdict.PASS} + assert PRECEDENT_EVIDENCE_REF.endswith( + "AUDIT-WP-0008-T05-whitehat-e2-03-pass-2026-08-22.md" + ) + + +def test_cross_tenant_event_disclosure_fails_the_claim(): + observations = passing_observations() + observations["event_by_id"]["attacker"] = { + "status": 200, + "schema": ("$", "$.event_id:str"), + "digest": "tenant-b-event", + "fixture_match_count": 2, + } + result = judgments(observations) + assert result["c-event-id-tenant-scoped"].verdict is Verdict.FAIL + + +def test_missing_cleanup_evidence_is_inconclusive(): + observations = passing_observations() + del observations["cleanup"] + result = judgments(observations) + assert result["c-receipt-bound-cleanup"].verdict is Verdict.INCONCLUSIVE + assert result["i-no-secret-evidence"].verdict is Verdict.INCONCLUSIVE + + +def test_cleanup_residue_fails_closed(): + observations = deepcopy(passing_observations()) + observations["cleanup"]["temporary_identities"] = ["unexpected-residue"] + result = judgments(observations) + assert result["c-receipt-bound-cleanup"].verdict is Verdict.FAIL + + +def test_intent_has_independent_provenance_and_separate_roles(): + assert TEST_USE_CASE.provenance is Provenance.SPEC + assert all( + assertion.provenance is Provenance.SPEC + for assertion in (*TEST_USE_CASE.claims, *TEST_USE_CASE.invariants) + ) + roles = {role.id for role in ROLE_CONTRACTS} + assert { + "authorizer", + "target-owner", + "credential-custodian", + "security-coordinator", + "cluster-executor", + "tenant-a-attacker", + "tenant-b-control", + "independent-observer", + } == roles + + +def test_schedule_requires_cleanup_before_finalization(): + phases = {phase.id: phase for phase in PHASE_CONTRACTS} + assert phases["finalize-and-deliver"].requires == ("cleanup-custody",) + assert phases["cleanup-custody"].requires == ("delete-runner",) + assert phases["run-probes"].requires == ("ready-runner",) diff --git a/tests/test_classification.py b/tests/test_classification.py index 37d5d2f..ccf6c39 100644 --- a/tests/test_classification.py +++ b/tests/test_classification.py @@ -194,3 +194,98 @@ def test_safe_to_accept_is_a_closed_set(): assert SAFE_TO_ACCEPT == { Classification.UNCHANGED, Classification.MECHANICAL_ADAPTATION, } + + +# --- classifications that no lab mutation happens to produce --------------- +# +# Two outcomes were declared at T08 and exercised by nothing. Left that way they +# are decoration: code that has never run is code nobody has checked. Rather than +# delete meaningful outcomes or trust them untested, both are given a case. + + +def test_intent_change_is_detected_when_the_claim_set_moves(baseline): + """`INTENT_CHANGED` is a fact about the recorded use case, not an inference. + + It fires because a human edited what is being asserted — which is why it is + detectable at all, where `SEMANTIC_CHANGE` was not (F-0006). + """ + import copy + + altered = copy.deepcopy(baseline) + altered["provenance_index"]["c-newly-added-claim"] = "human" + outcome = classify(baseline, altered) + assert outcome.classification is Classification.INTENT_CHANGED + assert not outcome.safe_to_accept + + +def test_realization_failure_is_distinguishable_from_ambiguity(): + """`REALIZATION_FAILED` says "we could not act"; `AMBIGUOUS` says "we do not know". + + Every lab mutation that breaks realization also strands a claim, so the + catalogue only ever produces `AMBIGUOUS`. This builds the case the catalogue + cannot: a step that fails while every assertion in the run still holds and + none of them depended on it. + + Note that a run asserting *nothing at all* is `AMBIGUOUS`, not + `REALIZATION_FAILED` — a use case with no claims cannot conclude anything, + however well its steps ran. + """ + from testdriver import ( + Actor, Cast, Invariant, Oracle, Runner, Scenario, SemanticAction, + StateObserver, Step, UseCase, VerificationAsset, World, + ) + from testdriver.agentic import DiscoveryRuntime + from testdriver.browser import BrowserDriver + from testdriver.observers import Watch + from testdriver.provenance import Provenance + from lab.mutations import ObservationChannel + + use_case = UseCase( + "uc-audit-only", "Sharing leaves an ordered audit trail", + "Alice shares R with Bob; the audit trail stays ordered.", + Provenance.HUMAN, + invariants=(Invariant( + "i-audit-ordered", "The audit trail is append-only", Provenance.HUMAN, + lambda obs: [e["sequence"] for e in obs["audit:R"]] + == sorted(e["sequence"] for e in obs["audit:R"]), + ),), + ) + + def run(*mutations): + with journey_lab_server(*mutations) as (app, tokens, base_url): + app.request(tokens["alice"], "create_resource", + resource_id="R", content="x") + cast = Cast() + cast.add(Actor("alice", "Alice", credentials={"token": tokens["alice"]})) + scenario = Scenario( + "sc-audit-only", use_case, + watches=(Watch("bob", "R"),), + steps=(Step("s1", "alice", SemanticAction( + "grant_access", {"subject_id": "bob", "permission": "READ"}, + permitted_surfaces=frozenset({"browser"}), + )),), + ) + driver = BrowserDriver(base_url, tokens, DiscoveryRuntime(), "R") + observer = StateObserver(ObservationChannel(app), scenario.watches) + world = World("w-audit", app, app.version, cast=cast) + return json.loads( + Runner(world, driver, observer, Oracle()) + .run(VerificationAsset("va-audit-only", scenario)) + .evidence.to_json() + ) + + outcome = classify(run(), run("M23")) # the control is gone from the UI + assert outcome.classification is Classification.REALIZATION_FAILED + assert not outcome.safe_to_accept + + +def test_no_classification_is_unreachable(): + """Every declared outcome must be produced somewhere in this suite. + + An outcome nothing can emit is the same kind of dead promise `SUSPICIOUS` + was before T10 removed it. + """ + exercised = set(EXPECTED.values()) | { + Classification.INTENT_CHANGED, Classification.REALIZATION_FAILED, + } + assert exercised == set(Classification) diff --git a/tests/test_reference_scenario.py b/tests/test_reference_scenario.py index 5739969..70554d5 100644 --- a/tests/test_reference_scenario.py +++ b/tests/test_reference_scenario.py @@ -67,4 +67,18 @@ def test_actors_hold_isolated_credentials_and_memory(): assert alice.credentials["token"] != bob.credentials["token"] alice.remember("secret", "only alice knows this") assert bob.recall("secret") is None - assert bob.known_keys() == () + # Every actor carries its own automatic private marker (F-0003) and nothing + # else it was not given. + assert bob.known_keys() == ("__canary__",) + assert alice.canary != bob.canary + + +def test_every_run_records_a_verdict_on_isolation(): + """F-0003: isolation is examined on every scenario, not only on ones + written to expose it.""" + result, _ = run_once() + examined = [ + obs for obs in result.evidence.observations if obs.kind == "actor_isolation" + ] + assert len(examined) == 1 + assert examined[0].data["violations"] == [] diff --git a/usecases/__init__.py b/usecases/__init__.py new file mode 100644 index 0000000..1e646be --- /dev/null +++ b/usecases/__init__.py @@ -0,0 +1,5 @@ +"""Durable test intent, kept separate from concrete scenario realizations.""" + +from .audit_core_e2_tenant_boundary import TEST_USE_CASE + +__all__ = ["TEST_USE_CASE"] diff --git a/usecases/__pycache__/__init__.cpython-312.pyc b/usecases/__pycache__/__init__.cpython-312.pyc new file mode 100644 index 0000000..df4655c Binary files /dev/null and b/usecases/__pycache__/__init__.cpython-312.pyc differ diff --git a/usecases/__pycache__/audit_core_e2_tenant_boundary.cpython-312.pyc b/usecases/__pycache__/audit_core_e2_tenant_boundary.cpython-312.pyc new file mode 100644 index 0000000..8c790cb Binary files /dev/null and b/usecases/__pycache__/audit_core_e2_tenant_boundary.cpython-312.pyc differ diff --git a/usecases/audit_core_e2_tenant_boundary.py b/usecases/audit_core_e2_tenant_boundary.py new file mode 100644 index 0000000..6cae13d --- /dev/null +++ b/usecases/audit_core_e2_tenant_boundary.py @@ -0,0 +1,331 @@ +"""Audit-core E2 tenant-boundary TestUseCase. + +This captures the intent and role hand-offs exercised successfully by +WH-ENG-20260822-AUDIT-E2-03. It is deliberately not a runnable ``Scenario`` +yet. The current kernel has no causal/time-window schedule, external custody +driver, Kubernetes driver, or independent cleanup observer capable of realizing +this use case without an attended operator. + +The claims come from the approved engagement and AUDIT-WP-0008-T05, not from +reverse-engineering the successful responses. The -03 report is precedent +evidence and a calibration example; it is not the source of truth for verdicts. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Mapping + +from testdriver import Claim, Invariant, Provenance, UseCase + + +SPEC_REF = ( + "audit-core/workplans/AUDIT-WP-0008-tenancy-posture-alignment.md" + "#AUDIT-WP-0008-T05" +) +ENGAGEMENT_REF = ( + "whitehat-security/engagements/2026-08-22-audit-core-e2-03.json" +) +PRECEDENT_EVIDENCE_REF = ( + "audit-core/docs/evidence/" + "AUDIT-WP-0008-T05-whitehat-e2-03-pass-2026-08-22.md" +) + + +@dataclass(frozen=True, slots=True) +class RoleContract: + """One independently driven responsibility in the attended protocol.""" + + id: str + responsibility: str + may_observe_secret_values: bool = False + + +@dataclass(frozen=True, slots=True) +class PhaseContract: + """Causal intent for the future orchestrator; never a shell transcript.""" + + id: str + driver_role: str + requires: tuple[str, ...] + completion: str + timing: str + + +ROLE_CONTRACTS = ( + RoleContract( + "authorizer", + "Approve the exact production package, target, scope, window and cleanup " + "obligation; hold no test credential.", + ), + RoleContract( + "target-owner", + "Acknowledge the exact target revision, routes, fixtures and abort limits.", + ), + RoleContract( + "credential-custodian", + "Preflight, project and revoke two short-lived tenant-scoped identities; " + "emit value-safe projection and cleanup receipts.", + may_observe_secret_values=True, + ), + RoleContract( + "security-coordinator", + "Validate the engagement, admit only the receipt-bound plane lease, invoke " + "the bounded probe, finalize the report and route it to risk-nexus.", + ), + RoleContract( + "cluster-executor", + "Create, wait for and delete only the approved runner pod; mount but never " + "read the projected Secret through the control plane.", + ), + RoleContract( + "tenant-a-attacker", + "Use an ordinary tenant-A identity to attempt the declared cross-tenant " + "reads and write.", + may_observe_secret_values=True, + ), + RoleContract( + "tenant-b-control", + "Use an independently scoped tenant-B identity only for known-good owner " + "controls and state-after checks.", + may_observe_secret_values=True, + ), + RoleContract( + "independent-observer", + "Collect sanitized outcomes, target readiness and post-cleanup absence " + "without accepting an actor's report as its own evidence.", + ), +) + + +PHASE_CONTRACTS = ( + PhaseContract( + "authorize", + "authorizer", + (), + "Exact engagement and custody contracts are approved.", + "Before projection and before the engagement window opens.", + ), + PhaseContract( + "acknowledge-target", + "target-owner", + ("authorize",), + "Target owner accepts the exact revision, routes and bounded fixtures.", + "Before any temporary credential exists.", + ), + PhaseContract( + "preflight", + "credential-custodian", + ("acknowledge-target",), + "Target is ready and all engagement-specific resources are absent.", + "Immediately before the projection cutoff.", + ), + PhaseContract( + "project-custody", + "credential-custodian", + ("preflight",), + "Exactly two scoped handles exist and a value-safe receipt is emitted.", + "Inside the narrow projection window; expiry equals engagement end.", + ), + PhaseContract( + "admit-plane", + "security-coordinator", + ("project-custody",), + "Receipt, broker contract, target revision and cleanup authority agree.", + "After projection and before any target packet.", + ), + PhaseContract( + "ready-runner", + "cluster-executor", + ("admit-plane",), + "The exact attributed runner is Ready with its Secret mounted.", + "Inside the authorized engagement window.", + ), + PhaseContract( + "run-probes", + "security-coordinator", + ("ready-runner",), + "Calibrated owner, attacker and absent controls produce a sanitized report.", + "Before credential and engagement expiry.", + ), + PhaseContract( + "delete-runner", + "cluster-executor", + ("run-probes",), + "The exact runner pod is absent.", + "Immediately after evidence collection.", + ), + PhaseContract( + "cleanup-custody", + "credential-custodian", + ("delete-runner",), + "Both identities, exact KV paths, projection resources and Secret are absent.", + "Before credential and engagement expiry.", + ), + PhaseContract( + "finalize-and-deliver", + "security-coordinator", + ("cleanup-custody",), + "The report is bound to projection and cleanup receipts and reaches risk-nexus.", + "Only after cleanup is independently observable.", + ), +) + + +def _same_absence_surface(left: Mapping[str, object], right: Mapping[str, object]) -> bool: + """Compare only the sanitized non-disclosure surface.""" + + return ( + left["status"] == right["status"] + and left["schema"] == right["schema"] + and left["digest"] == right["digest"] + and left["fixture_match_count"] == 0 + and right["fixture_match_count"] == 0 + ) + + +def _event_id_is_tenant_scoped(obs: Mapping[str, object]) -> bool: + probe = obs["event_by_id"] + owner = probe["owner"] + attacker = probe["attacker"] + absent = probe["absent"] + return ( + owner["status"] == 200 + and owner["fixture_match_count"] > 0 + and _same_absence_surface(attacker, absent) + ) + + +def _correlation_slice_is_tenant_scoped(obs: Mapping[str, object]) -> bool: + probe = obs["correlation_slice"] + return ( + probe["owner"]["status"] == 200 + and probe["owner"]["fixture_match_count"] > 0 + and probe["attacker"]["status"] == 200 + and probe["attacker"]["fixture_match_count"] == 0 + ) + + +def _cross_tenant_append_is_refused(obs: Mapping[str, object]) -> bool: + probe = obs["append_as_b"] + return ( + probe["attacker"]["status"] in {400, 403, 404} + and _same_absence_surface(probe["state_after"], probe["absent"]) + ) + + +def _cleanup_is_complete(obs: Mapping[str, object]) -> bool: + cleanup = obs["cleanup"] + return ( + cleanup["completed_before_expiry"] is True + and cleanup["runner_absent"] is True + and cleanup["temporary_identities"] == [] + and cleanup["kv_paths"] == [] + and cleanup["projection_resources"] == [] + and cleanup["mounted_secrets"] == [] + and cleanup["target_ready"] is True + ) + + +def _execution_remains_bounded(obs: Mapping[str, object]) -> bool: + run = obs["run"] + return ( + run["attempted_operations"] <= run["maximum_operations"] + and run["max_concurrency"] == 1 + and run["rate_per_minute"] <= 10 + and run["target_revision_matches"] is True + and run["attribution_complete"] is True + ) + + +def _secret_values_do_not_enter_evidence(obs: Mapping[str, object]) -> bool: + return obs["cleanup"]["secret_values_observed"] is False + + +def _report_preserves_assurance_limit(obs: Mapping[str, object]) -> bool: + report = obs["report"] + return ( + report["bound_to_projection_receipt"] is True + and report["bound_to_cleanup_receipt"] is True + and report["bounded_assurance_statement"] is True + and report["delivered_to_risk_nexus"] is True + ) + + +TEST_USE_CASE = UseCase( + id="uc-audit-core-e2-tenant-boundary", + title="Demonstrate audit-core's E2 tenant boundary with separate drivers", + narrative=( + "Within one explicitly authorized production window, independent target, " + "custody, security, cluster, attacker, control and observer roles establish " + "that an ordinary tenant-A identity cannot read or create tenant-B audit " + "fixtures. The run must remain bounded, preserve non-disclosure semantics, " + "clean every temporary resource before expiry, and deliver a cleanup-bound " + "sanitized report without retaining credential values." + ), + provenance=Provenance.SPEC, + source_ref=SPEC_REF, + claims=( + Claim( + "c-event-id-tenant-scoped", + "Tenant A cannot distinguish tenant B's event id from an absent event", + Provenance.SPEC, + _event_id_is_tenant_scoped, + after_step="run-probes", + source_ref=ENGAGEMENT_REF, + ), + Claim( + "c-correlation-slice-tenant-scoped", + "Tenant A's correlation slice contains no tenant-B fixture", + Provenance.SPEC, + _correlation_slice_is_tenant_scoped, + after_step="run-probes", + source_ref=ENGAGEMENT_REF, + ), + Claim( + "c-cross-tenant-append-refused", + "Tenant A cannot append an audit event attributed to tenant B", + Provenance.SPEC, + _cross_tenant_append_is_refused, + after_step="run-probes", + source_ref=ENGAGEMENT_REF, + ), + Claim( + "c-receipt-bound-cleanup", + "Every engagement-specific runtime and custody resource is absent " + "before expiry while the target remains ready", + Provenance.SPEC, + _cleanup_is_complete, + after_step="cleanup-custody", + source_ref=SPEC_REF, + ), + ), + invariants=( + Invariant( + "i-bounded-execution", + "The run stays within its operation, rate, concurrency, revision and " + "attribution boundaries", + Provenance.SPEC, + _execution_remains_bounded, + source_ref=ENGAGEMENT_REF, + ), + Invariant( + "i-no-secret-evidence", + "Credential values never enter retained observations or reports", + Provenance.SPEC, + _secret_values_do_not_enter_evidence, + source_ref=SPEC_REF, + ), + Invariant( + "i-bounded-assurance-report", + "Delivery is receipt-bound and says only that attempted attacks failed", + Provenance.SPEC, + _report_preserves_assurance_limit, + source_ref=SPEC_REF, + ), + ), +) + +# Alias retained for the existing scenario-module convention. +USE_CASE = TEST_USE_CASE + diff --git a/workplans/TD-WP-0002-vertical-spike-crystallization.md b/workplans/TD-WP-0002-vertical-spike-crystallization.md index dece354..36317cd 100644 --- a/workplans/TD-WP-0002-vertical-spike-crystallization.md +++ b/workplans/TD-WP-0002-vertical-spike-crystallization.md @@ -4,11 +4,11 @@ type: workplan title: "Vertical spike: falsify the crystallization thesis" domain: infotech repo: test-driver -status: active +status: finished owner: codex topic_slug: custodian created: "2026-08-22" -updated: "2026-08-22" +updated: "2026-08-23" state_hub_workstream_id: "68b252b8-03e7-55c7-8dec-ac13f6ca0aa1" --- @@ -461,7 +461,7 @@ to execute than the agentic ancestor. ```task id: TD-WP-0002-T10 -status: todo +status: done priority: medium state_hub_task_id: "884c2e91-b56d-5b5e-a2d3-52d2a2768cae" ``` @@ -478,3 +478,42 @@ retaining them, and update the Concept ↔ Implementation Fitness Map. Convert at least one framework finding discovered during T04–T09 into a permanent self-verification or deterministic regression — that closes the first genuine Concept–Implementation Fitness Loop. + +**Done 2026-08-23.** Full write-up: +`history/2026-08-23-td-wp-0002-gate-review.md`. 178 tests pass. + +**All four gate criteria met.** M01/M02 absorbed with the claim set unchanged; +the authorization defects escalated with regressions named; one asset +crystallized and running with no model; **False Adaptation Rate 0/7**, with 12 of +13 mechanical mutations absorbed so the safety result is not bought by escalating +everything. + +**Fitness loop closed via F-0003.** Actor isolation left no trace in ordinary +evidence — it was a property of scenarios written to expose it. Every actor is +now seeded with an automatic private marker and the runner examines all actors on +every scenario, with two permanent regressions behind it, one of which catches a +leak with the harness planting nothing. + +**Compression — six abstractions removed**, each declared and never used: +`Verdict.SUSPICIOUS` (a verdict no oracle could emit), `Step.expect_refusal`, +`ActorIsolationError`, `World.seed`, `EvidencePack.latest()`, +`Trajectory.method`. + +**F-0008 — Temperature may be redundant.** Crystallization was built end to end +without it ever being consulted; what triggered freezing was *measured stability +of realization*, which is observed rather than declared. A declared temperature +is a claim nobody verifies. Not deleted — two unevidenced cases for keeping it +are recorded — but gated, along with `energy.py`: if the next workplan closes +without a decision consulting either, they go. + +**Two classifications had never run.** `INTENT_CHANGED` and `REALIZATION_FAILED` +were declared at T08 and produced by no test. Both now have purpose-built cases, +and `test_no_classification_is_unreachable` fails if a seventh is added without +one. + +**What this leaves.** The thesis survived contact with its own evidence in +narrowed form, and the demonstration is architectural rather than statistical. +The highest-value next step is a **bounded live-model experiment** — F-0005 and +F-0007 converge on it independently, and one experiment settles both whether a +model adds capability and whether crystallization has a product rationale. That +should come before any further framework feature.