diff --git a/.gitignore b/.gitignore index 144879c..e4e0199 100644 --- a/.gitignore +++ b/.gitignore @@ -3,8 +3,3 @@ .claude/* !.claude/rules/ !.claude/rules/*.md - -# Python -__pycache__/ -*.py[cod] -.pytest_cache/ diff --git a/INTENT.md b/INTENT.md index e357550..904b763 100755 --- a/INTENT.md +++ b/INTENT.md @@ -116,14 +116,6 @@ COLD -> deterministic The framework should make it natural for tests to crystallize as software cools. -> **Under review since T10 — see `research/findings/F-0008-temperature-may-be-redundant.md`.** -> Crystallization was built and demonstrated without Temperature ever being -> consulted. What actually triggered freezing was *measured stability of -> realization* — the same path found across several runs — which is observed, -> not declared. Temperature is a label a human would have to maintain and keep -> honest. If measured stability is sufficient, Temperature is a concept the -> framework does not need. - --- ## Energy @@ -256,13 +248,10 @@ Initial verdicts are: ```text PASS FAIL +SUSPICIOUS INCONCLUSIVE ``` -`SUSPICIOUS` was listed here originally and **removed at T10**: no oracle could -produce it, and a verdict nothing can emit is a promise the framework does not -keep. It returns, with an identifier, if a mechanism ever needs it. - --- ## Lineage diff --git a/README.md b/README.md index d5ce8d3..8a3d0d8 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,6 @@ No third-party dependencies. Python ≥ 3.11, pytest for the suite. ``` src/testdriver/ the kernel — intent, world, actions, drivers, observers, oracles, evidence, runner -usecases/ durable test intent, including not-yet-runnable use cases lab/ the system under test scenarios/ reference scenarios research/ hypotheses, experiments, findings, fitness map diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index c2d070d..a61b979 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -21,5 +21,5 @@ | task | TD-WP-0002-T06 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T07 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T08 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | -| task | TD-WP-0002-T09 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | +| task | TD-WP-0002-T09 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T10 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | diff --git a/crystallized/__pycache__/__init__.cpython-312.pyc b/crystallized/__pycache__/__init__.cpython-312.pyc new file mode 100644 index 0000000..d98adb6 Binary files /dev/null and b/crystallized/__pycache__/__init__.cpython-312.pyc differ diff --git a/crystallized/__pycache__/conftest.cpython-312-pytest-7.4.4.pyc b/crystallized/__pycache__/conftest.cpython-312-pytest-7.4.4.pyc new file mode 100644 index 0000000..f8c9bb1 Binary files /dev/null and b/crystallized/__pycache__/conftest.cpython-312-pytest-7.4.4.pyc differ diff --git a/crystallized/__pycache__/test_grant_access.cpython-312-pytest-7.4.4.pyc b/crystallized/__pycache__/test_grant_access.cpython-312-pytest-7.4.4.pyc new file mode 100644 index 0000000..5efe09d Binary files /dev/null and b/crystallized/__pycache__/test_grant_access.cpython-312-pytest-7.4.4.pyc differ diff --git a/history/2026-08-23-td-wp-0002-gate-review.md b/history/2026-08-23-td-wp-0002-gate-review.md deleted file mode 100644 index e599184..0000000 --- a/history/2026-08-23-td-wp-0002-gate-review.md +++ /dev/null @@ -1,212 +0,0 @@ -# TD-WP-0002 — Gate Review and First Compression Pass - -**Date:** 2026-08-23 -**Workplan:** `TD-WP-0002` — Vertical spike: falsify the crystallization thesis -**Tests:** 172 passing · **Lab catalogue:** 24 labelled mutations -**Assessment that set the direction:** `history/2026-08-22-concept-assessment-swot.md` - ---- - -## 1. The gate - -The workplan fixed four criteria in advance, before any code existed. - -### Criterion 1 — recover from M01 and M02 with zero claim or invariant changes - -**Met.** - -| Mutation | Classification | Claim set unchanged | Accepted without a human | -|---|---|---|---| -| M01 sharing control moved into a modal | `MECHANICAL_ADAPTATION` | yes | yes | -| M02 DOM rewritten, test ids dropped | `MECHANICAL_ADAPTATION` | yes | yes | - -The claim set is unchanged *by construction*, not by good behaviour: claims are -run inputs and the adaptation machinery has no write path to them (D-02). - -### Criterion 2 — the authorization defect is reported as a Product Finding - -**Met**, with a naming correction. The workplan inherited "M05" from the original -milestones document; in the catalogue as built, the authorization defect is -**M15** (revocation updates the record but not enforcement). All three -authorization defects behave correctly: - -| Mutation | Classification | Regressions reported | -|---|---|---| -| M15 revoke is cosmetic | `BEHAVIOUR_CHANGED` | `c-bob-revoked`, `i-enforcement-matches-record` | -| M17 anyone can read anything | `BEHAVIOUR_CHANGED` | 5 assertions across 3 steps | -| M20 tenant isolation leaks | `BEHAVIOUR_CHANGED` | 5 assertions across 3 steps | - -None is safe to accept. The classifier collapses `PRODUCT_DEFECT` and -`SEMANTIC_CHANGE` into one escalating outcome — see F-0006 — so "Product Finding" -in the original wording is now "escalated to a human with the regressions named". - -### Criterion 3 — one asset crystallizes into a deterministic test - -**Met.** `crystallized/test_grant_access.py` is generated, runs with **no model -involvement**, preserves its ancestor's oracle set as a superset, agrees with the -ancestor across five lab versions, still catches a seeded defect, and carries its -lineage in its own docstring. - -### Criterion 4 — False Adaptation Rate = 0 - -**Met. 0 / 7.** No `DEFECT`-labelled mutation was classified as safe to accept, -including the three E-003 attacks (surface substitution, concurrent -mechanical+defect, evidence starvation) and provenance laundering. - -The other side of the trade is asserted too: **12 of 13** mechanical mutations are -absorbed without a human. A classifier that escalated everything would score a -perfect FAR and be worthless. - -### Gate verdict - -**Passed, on all four criteria.** With one qualification that belongs in the same -sentence: FAR = 0 follows largely from *architecture* — claims are run inputs, -and `SAFE_TO_ACCEPT` is a closed two-element set — not from classifier accuracy. -The experiment shows the architecture behaves as designed over 24 hand-written -mutations. It says nothing about mutations nobody thought of. - ---- - -## 2. What the evidence changed - -Three of the project's own claims were narrowed or corrected by building them. -This is the part worth more than the green gate. - -**H-001 was narrowed (F-0005).** Semantic actions do not generally outlast -recorded interaction sequences. Where an application keeps stable identifiers, -the two arms tie exactly — 9/9 against 9/9 — and the conventional approach is -cheaper and deterministic. Semantic actions earn their keep only where -identifiers are absent or not carried forward (2/3 against 0/3). The concept -model presents this as a general superiority; it is conditional. - -**The classifier cannot infer intent (F-0006).** The T02 design listed -`SEMANTIC_CHANGE` as an outcome the decision table could produce. It cannot: a -deliberate product decision and a defect are behaviourally identical, as M12 and -M19 have asserted since T05. The two collapse into `BEHAVIOUR_CHANGED` and the -distinction becomes a recorded human adjudication. Erring toward false alarm -costs one review; erring toward false adaptation ends the project. - -**Crystallization's economic case is unmeasured (F-0007).** The descendant is 54% -cheaper than its ancestor, and that number must not be quoted in support of the -thesis. The runtime is token-free by design, so the entire saving is one page -fetch, one parse and a two-candidate scoring pass. The saving the concept -actually claims — model tokens, latency, retry variance — is absent from the -measurement. - ---- - -## 3. The compression pass - -### Which concepts have no supporting evidence? - -`Temperature`, `Confidence`, `Campaign`, `Metabolism`, `Retirement` — declared, -unimplemented, never consulted by any decision. - -`Temperature` is the decidable one and is now **F-0008**. Crystallization was -built and demonstrated without it: what triggered freezing was *measured -stability of realization*, which is observed rather than declared. A declared -temperature is a claim nobody verifies, and the project's own first heuristic -argues against it. Not deleted yet — two unevidenced cases for keeping it are -recorded — but gated: if the next workplan completes without a decision -consulting it, it goes. - -### Which implementation abstractions have no conceptual justification? - -Removed at T10: - -| Removed | Why | -|---|---| -| `Verdict.SUSPICIOUS` | No oracle could produce it. A verdict nothing can emit is a promise not kept. | -| `Step.expect_refusal` | Never set by any scenario. Speculative generality. | -| `ActorIsolationError` | Declared, never raised, never caught. | -| `World.seed` | Written at T04, never read. Replay comes from rebuilding the lab. | -| `EvidencePack.latest()` | Never called. | -| `Trajectory.method` | Computed on every capture, never read. | - -### Which metadata has not informed a decision? - -**`EnergyEvent` history.** Events have been recorded since T04 and have informed -nothing — as designed, since H-005 is dormant. Capture is retained because -history cannot be reconstructed later while scores always can, but this is now on -the same gate as Temperature: if the next workplan ends with no decision having -used it, the module goes. - -### Which agentic behaviour can now be deterministic? - -The `grant_access` realization, and it has been — that is T09. Nothing else in -the spike is agentic. - -### What have the experiments falsified? - -Nothing outright. Two claims were **narrowed** (H-001, and the classifier's -output space), and one criterion was met in a sense **too weak to support the -thesis** (H-003's cost clause). No hypothesis was rejected, and none was promoted -past `EXPERIMENTING`: 24 hand-written mutations is a small, self-chosen sample. - -### Two outcomes that had never run - -`INTENT_CHANGED` and `REALIZATION_FAILED` were declared at T08 and produced by no -test. Code that has never run is code nobody has checked. Rather than delete -meaningful outcomes or trust them untested, both now have purpose-built cases, -and `test_no_classification_is_unreachable` fails if a seventh is ever added -without one. - -Building the `REALIZATION_FAILED` case clarified a boundary worth keeping: a run -that asserts *nothing at all* is `AMBIGUOUS`, not `REALIZATION_FAILED` — a use -case with no claims cannot conclude anything, however well its steps ran. - ---- - -## 4. The fitness loop, closed - -The milestone asks that a framework finding become a permanent guarantee rather -than a note. **F-0003 is resolved.** - -The finding: actor isolation left no trace in ordinary evidence. The self-test -caught a shared memory store only because the harness planted per-actor secrets -first. Isolation was a property of *scenarios written to expose it*, not of runs. - -The fix: every `Actor` is seeded at construction with an automatic private -marker, and the runner examines all actors on **every** scenario, recording the -verdict as S3 evidence. Two regressions now stand behind it — -`test_every_run_records_a_verdict_on_isolation`, and a self-check that catches a -leak with the harness planting nothing at all. - -``` -Framework Finding (F-0003) - → classified FRAMEWORK_LIMITATION - → improvement hypothesis: automatic canaries make isolation observable - → implemented at T10 - → permanent self-verification + deterministic regression - → fitness map updated -``` - -That is the first complete Concept–Implementation Fitness Loop, which the -milestones document calls the point at which the self-improvement system -genuinely exists. - ---- - -## 5. Where this leaves the project - -**The thesis survived contact with its own evidence, in narrowed form.** The -distinctive claim — that verification can adapt to a changing surface without -adapting to a defect — is demonstrated, and the demonstration is architectural -rather than statistical, which is the stronger kind. - -**The single highest-value next step is a bounded live-model experiment.** Two -independent findings converge on it: M22 defeats the heuristic runtime while -remaining solvable by reading a visible label (F-0005), and crystallization's -economic case cannot be measured without token costs (F-0007). One experiment -settles whether a model adds capability *and* whether crystallization has a -product rationale. The `RealizationMetrics` fields were populated from the first -run so that comparison is a subtraction rather than a re-run. - -That should come before any further framework feature. - -**Open findings carried forward:** F-0003 resolved · F-0004 (structural -durability only; visual relayout untested) · F-0005 (H-001 narrowed) · F-0007 -(cost unmeasurable; generated tests not fully standalone) · F-0008 (Temperature -gated). The F-0001 residual — the taxonomy has no class for document-to-document -drift — was considered and deliberately not resolved: adding a finding class to a -taxonomy already larger than its evidence would be the opposite of compression. diff --git a/lab/__pycache__/__init__.cpython-312.pyc b/lab/__pycache__/__init__.cpython-312.pyc new file mode 100644 index 0000000..930c9db Binary files /dev/null and b/lab/__pycache__/__init__.cpython-312.pyc differ diff --git a/lab/__pycache__/app.cpython-312.pyc b/lab/__pycache__/app.cpython-312.pyc new file mode 100644 index 0000000..93802fc Binary files /dev/null and b/lab/__pycache__/app.cpython-312.pyc differ diff --git a/lab/__pycache__/http_api.cpython-312.pyc b/lab/__pycache__/http_api.cpython-312.pyc new file mode 100644 index 0000000..c0e0551 Binary files /dev/null and b/lab/__pycache__/http_api.cpython-312.pyc differ diff --git a/lab/__pycache__/minimal.cpython-312.pyc b/lab/__pycache__/minimal.cpython-312.pyc new file mode 100644 index 0000000..f1dabc8 Binary files /dev/null and b/lab/__pycache__/minimal.cpython-312.pyc differ diff --git a/lab/__pycache__/mutations.cpython-312.pyc b/lab/__pycache__/mutations.cpython-312.pyc new file mode 100644 index 0000000..0230fbb Binary files /dev/null and b/lab/__pycache__/mutations.cpython-312.pyc differ diff --git a/research/concepts/fitness-map.md b/research/concepts/fitness-map.md index 32a0046..c8c4c75 100644 --- a/research/concepts/fitness-map.md +++ b/research/concepts/fitness-map.md @@ -1,6 +1,6 @@ # Concept ↔ Implementation Fitness Map -**Updated:** 2026-08-23 (TD-WP-0002-T10) +**Updated:** 2026-08-23 (TD-WP-0002-T09) Traces each important concept to the implementation, experiment and evidence that support it. **Unsupported entries are the point of this map** — a concept with no @@ -29,7 +29,7 @@ were aspirational, not evidenced. | Concept | Level | Implementation | Experiment | Evidence | Open question | |---|---|---|---|---|---| | `C-use-case` | C1 | `intent.py` | — | — | Is a use case expressible without leaking mechanics? | -| `C-actor-isolation` | **C2** | `world.py`, `runner.py` | E-001 | `td://self/actor-isolation`, examined every run | **F-0003 resolved** — automatic canaries; observable on every scenario. | +| `C-actor-isolation` | C1 | `world.py` | E-001 | `td://self/actor-isolation` | **F-0003** — only observable when the scenario plants canaries. | | `C-semantic-action` | **C2** | `actions.py`, `agentic.py` | E-001 (partial) | T07 arm comparison | **F-0005** — supported only where stable identifiers are absent. Narrower than the concept model claims. | | `C-oracle-independence` | **C2** | `runner.py`, `oracles.py` | E-001, E-003 | — | Independence of components ≠ independence of belief. (H-004) | | `C-evidence-pack` | C1 | `evidence.py` | — | `td://self/evidence-reproducibility` | Verdicts are reproducible from S3 alone, on passing and failing runs. | @@ -39,9 +39,9 @@ were aspirational, not evidenced. | `C-crystallization` | **C2** | `crystallization.py` | E-002 | T09 agreement matrix | Fidelity supported; **F-0007** — the economic case is unmeasurable with a token-free runtime. | | `C-intent-provenance` | C1 | `provenance.py` | E-003 | `td://self/intent-independence` | Constrains provenance, not quality. Accepted residual. | | `C-lineage` | C1 | `scenario.py`, generated headers | E-002 | generated module docstring | Parent pointer plus provenance in the artefact; no graph. | -| `C-energy` | C0 | `energy.py`, capture only | — | none | Dormant. **Gated**: if the next workplan ends with no decision having used the history, remove. | -| `C-temperature` | **C0 — under review** | — | — | none | **F-0008** — crystallization was built without it; measured stability did the work. Gated for removal. | -| `C-confidence` | C0 | — | — | none | Declared, unimplemented, never consulted. Same position as `C-temperature` without a built alternative. | +| `C-energy` | C0 | `energy.py`, capture only | — | — | Dormant by decision. (H-005) | +| `C-temperature` | C0 | — | — | — | Deferred. No implementation planned in TD-WP-0002. | +| `C-confidence` | C0 | — | — | — | Deferred. | | `C-campaign` | C0 | — | — | — | Deferred. | | `C-metabolism` | C0 | — | — | — | Deferred. Depends on C-energy. | | `C-retirement` | C0 | — | — | — | Deferred. Depends on C-energy. | @@ -58,13 +58,6 @@ validated. They are revisited at T10, where the question is not "when do we buil these" but "does the evidence justify keeping them in the model at all". **Implementation orphans** — none. Every module in `src/testdriver/` traces to a -concept above. - -**Removed at T10** (compression pass — see the gate review § 3): -`Verdict.SUSPICIOUS`, `Step.expect_refusal`, `ActorIsolationError`, -`World.seed`, `EvidencePack.latest()`, `Trajectory.method`. Each was declared and -never used; `SUSPICIOUS` was additionally a verdict no oracle could emit. - -**Gated for removal**: `C-temperature` (F-0008) and `energy.py`. Both survive the -current review on the strength of being cheap, not of being used. If the next -workplan closes without a decision consulting either, they go. +concept above. `energy.py` is the one to watch: it exists solely to capture +events for a dormant hypothesis, and if T10 finds no use for the history it +should be removed rather than kept out of sentiment. diff --git a/research/findings/F-0003-isolation-needs-canaries.md b/research/findings/F-0003-isolation-needs-canaries.md index d0c93b8..008f659 100644 --- a/research/findings/F-0003-isolation-needs-canaries.md +++ b/research/findings/F-0003-isolation-needs-canaries.md @@ -2,13 +2,12 @@ id: F-0003 type: framework-finding class: FRAMEWORK_LIMITATION -status: resolved +status: open discovered: "2026-08-22" discovered_by: TD-WP-0002-T06 workplan: TD-WP-0002 task: TD-WP-0002-T06 -resolved: "2026-08-23" -resolved_in: TD-WP-0002-T10 +carried_to: TD-WP-0002-T10 --- # F-0003 — Actor isolation is only observable if the scenario plants canaries @@ -56,31 +55,9 @@ system. Worth stating plainly rather than counting four green self-tests as four equivalent proofs. They are not equivalent. -## Resolution (T10) +## Candidate resolutions -Candidate 3 — **both**. - -Every `Actor` is seeded at construction with an automatic private marker -(`Actor.canary`), and `Runner._isolation_violations` examines every actor on -**every** scenario, recording the verdict as S3 evidence. An isolation violation -now leaves a trace whether or not anyone thought to look for one. - -Two permanent regressions stand behind it: - -- `tests/test_reference_scenario.py::test_every_run_records_a_verdict_on_isolation` -- `tests/selfverification/test_checks_can_fail.py::test_a_leak_is_caught_without_the_test_planting_anything` - — an actor holding another's marker is caught by the ordinary run, with the - harness planting nothing. - -The second limitation stands as recorded: oracle independence is enforced by -construction, so it is verified once rather than per run. That asymmetry is real -and is now stated in the fitness map rather than hidden behind two equally green -self-tests. - -This finding closed the first Concept–Implementation Fitness Loop — see -`history/2026-08-23-td-wp-0002-gate-review.md` § 4. - -## Candidate resolutions considered +Not resolved now; deciding cheaply here would be guessing. 1. **Automatic canaries** — the runtime seeds each actor with a unique private marker at construction and the isolation check runs on every Evidence Pack, diff --git a/research/findings/F-0008-temperature-may-be-redundant.md b/research/findings/F-0008-temperature-may-be-redundant.md deleted file mode 100644 index a867ce3..0000000 --- a/research/findings/F-0008-temperature-may-be-redundant.md +++ /dev/null @@ -1,70 +0,0 @@ ---- -id: F-0008 -type: framework-finding -class: UNNECESSARY_COMPLEXITY -status: open -discovered: "2026-08-23" -discovered_by: TD-WP-0002-T10 -workplan: TD-WP-0002 -task: TD-WP-0002-T10 -carried_to: next workplan ---- - -# F-0008 — Temperature may be redundant; measured stability did the work - -## Observation - -`Temperature` (`HOT` / `WARM` / `COOL` / `COLD`) is one of the more prominent -ideas in `INTENT.md` and the Concept Model. It is supposed to govern which -verification mode an asset uses, and it is the mechanism by which "tests -crystallize as software cools". - -The spike built crystallization end to end. **Temperature was never consulted, -never implemented, and never missed.** - -What actually triggered freezing was `assess_stability`: the same realization -path observed across several consecutive runs. Nothing declared the surface cold; -the surface was observed to have stopped moving. - -## The argument for removal - -The two mechanisms answer the same question, and only one of them can be wrong: - -| | Temperature | Measured stability | -|---|---|---| -| source | declared by a human | observed from runs | -| upkeep | must be maintained and kept honest | none | -| failure mode | says `COLD` while the surface churns | none — it is the churn | -| already built | no | yes | - -A declared temperature is a claim about the system that nobody verifies. The -project's own first heuristic — *do not let a declaration substitute for -evidence* — argues against keeping it. - -## The argument for keeping it - -Two cases measured stability cannot cover: - -1. **Forward-looking scope.** A team knows a rewrite lands next month. - Temperature can say `HOT` before any run has observed instability, where - measured stability would happily freeze an asset the day before it breaks. -2. **Cheap prioritisation.** Deciding *where to spend effort* across many - capabilities may want a coarse label without running anything. - -Neither case arose in the spike, and neither is evidenced. - -## Recommendation - -Do not delete yet; do not build either. Carry `Temperature` as **explicitly -unvalidated** in the fitness map, and set a gate: if the next workplan completes -without any decision consulting it, remove it from the concept model. - -Recorded now because the compression review's purpose is to catch concepts that -survive on the strength of being attractive rather than useful, and Temperature -is currently the clearest instance in the corpus. - -## Related - -`Confidence`, `Campaign`, `Metabolism` and `Retirement` are in the same position — -declared, unimplemented, never consulted — but Temperature is the one with a -built alternative already doing its job, which makes it the decidable case. diff --git a/scenarios/__pycache__/alice_bob_carol.cpython-312.pyc b/scenarios/__pycache__/alice_bob_carol.cpython-312.pyc new file mode 100644 index 0000000..9053e9f Binary files /dev/null and b/scenarios/__pycache__/alice_bob_carol.cpython-312.pyc differ diff --git a/scenarios/__pycache__/browser_grant.cpython-312.pyc b/scenarios/__pycache__/browser_grant.cpython-312.pyc new file mode 100644 index 0000000..d199de0 Binary files /dev/null and b/scenarios/__pycache__/browser_grant.cpython-312.pyc differ diff --git a/scenarios/__pycache__/full_journey.cpython-312.pyc b/scenarios/__pycache__/full_journey.cpython-312.pyc new file mode 100644 index 0000000..65e17f8 Binary files /dev/null and b/scenarios/__pycache__/full_journey.cpython-312.pyc differ diff --git a/scenarios/alice_bob_carol.py b/scenarios/alice_bob_carol.py index e9c10a8..70bcfa7 100644 --- a/scenarios/alice_bob_carol.py +++ b/scenarios/alice_bob_carol.py @@ -128,7 +128,9 @@ def build(*mutations: str): cast.add(Actor(id=name, display_name=name.title(), credentials={"token": tokens[name]})) - world = World(id="w-baseline", sut=lab, sut_version=lab.version, cast=cast) + world = World(id="w-baseline", sut=lab, sut_version=lab.version, + seed={"users": ["alice", "bob", "carol"], "resource": RESOURCE}, + cast=cast) scenario = Scenario( id="sc-share-resource", diff --git a/src/testdriver/__init__.py b/src/testdriver/__init__.py index 31418a5..d56c3ca 100644 --- a/src/testdriver/__init__.py +++ b/src/testdriver/__init__.py @@ -10,10 +10,10 @@ from .oracles import Judgment, Oracle, Verdict, overall from .provenance import InadmissibleProvenance, Provenance from .runner import CollectorIndependenceError, Runner, RunResult from .scenario import Scenario, Step, VerificationAsset -from .world import Actor, Cast, World +from .world import Actor, ActorIsolationError, Cast, World __all__ = [ - "Actor", "Cast", "Claim", "CollectorIndependenceError", + "Actor", "ActorIsolationError", "Cast", "Claim", "CollectorIndependenceError", "DirectDriver", "EnergyEvent", "EnergyEventType", "EvidencePack", "InadmissibleProvenance", "Invariant", "Judgment", "Observation", "Oracle", "Provenance", "Realization", "RunResult", "Runner", "Scenario", diff --git a/src/testdriver/__pycache__/__init__.cpython-312.pyc b/src/testdriver/__pycache__/__init__.cpython-312.pyc new file mode 100644 index 0000000..216faa6 Binary files /dev/null and b/src/testdriver/__pycache__/__init__.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/actions.cpython-312.pyc b/src/testdriver/__pycache__/actions.cpython-312.pyc new file mode 100644 index 0000000..7b9fe8e Binary files /dev/null and b/src/testdriver/__pycache__/actions.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/agentic.cpython-312.pyc b/src/testdriver/__pycache__/agentic.cpython-312.pyc new file mode 100644 index 0000000..836c867 Binary files /dev/null and b/src/testdriver/__pycache__/agentic.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/browser.cpython-312.pyc b/src/testdriver/__pycache__/browser.cpython-312.pyc new file mode 100644 index 0000000..956759a Binary files /dev/null and b/src/testdriver/__pycache__/browser.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/classification.cpython-312.pyc b/src/testdriver/__pycache__/classification.cpython-312.pyc new file mode 100644 index 0000000..ea83da0 Binary files /dev/null and b/src/testdriver/__pycache__/classification.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/crystallization.cpython-312.pyc b/src/testdriver/__pycache__/crystallization.cpython-312.pyc new file mode 100644 index 0000000..6e999c0 Binary files /dev/null and b/src/testdriver/__pycache__/crystallization.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/drivers.cpython-312.pyc b/src/testdriver/__pycache__/drivers.cpython-312.pyc new file mode 100644 index 0000000..c735df0 Binary files /dev/null and b/src/testdriver/__pycache__/drivers.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/energy.cpython-312.pyc b/src/testdriver/__pycache__/energy.cpython-312.pyc new file mode 100644 index 0000000..f2f2a7f Binary files /dev/null and b/src/testdriver/__pycache__/energy.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/evidence.cpython-312.pyc b/src/testdriver/__pycache__/evidence.cpython-312.pyc new file mode 100644 index 0000000..887c8f1 Binary files /dev/null and b/src/testdriver/__pycache__/evidence.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/html.cpython-312.pyc b/src/testdriver/__pycache__/html.cpython-312.pyc new file mode 100644 index 0000000..2151e4c Binary files /dev/null and b/src/testdriver/__pycache__/html.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/intent.cpython-312.pyc b/src/testdriver/__pycache__/intent.cpython-312.pyc new file mode 100644 index 0000000..7868e9a Binary files /dev/null and b/src/testdriver/__pycache__/intent.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/observers.cpython-312.pyc b/src/testdriver/__pycache__/observers.cpython-312.pyc new file mode 100644 index 0000000..033be7d Binary files /dev/null and b/src/testdriver/__pycache__/observers.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/oracles.cpython-312.pyc b/src/testdriver/__pycache__/oracles.cpython-312.pyc new file mode 100644 index 0000000..7980776 Binary files /dev/null and b/src/testdriver/__pycache__/oracles.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/provenance.cpython-312.pyc b/src/testdriver/__pycache__/provenance.cpython-312.pyc new file mode 100644 index 0000000..71c7136 Binary files /dev/null and b/src/testdriver/__pycache__/provenance.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/runner.cpython-312.pyc b/src/testdriver/__pycache__/runner.cpython-312.pyc new file mode 100644 index 0000000..48c6c66 Binary files /dev/null and b/src/testdriver/__pycache__/runner.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/scenario.cpython-312.pyc b/src/testdriver/__pycache__/scenario.cpython-312.pyc new file mode 100644 index 0000000..8bf7e17 Binary files /dev/null and b/src/testdriver/__pycache__/scenario.cpython-312.pyc differ diff --git a/src/testdriver/__pycache__/world.cpython-312.pyc b/src/testdriver/__pycache__/world.cpython-312.pyc new file mode 100644 index 0000000..6762964 Binary files /dev/null and b/src/testdriver/__pycache__/world.cpython-312.pyc differ diff --git a/src/testdriver/crystallization.py b/src/testdriver/crystallization.py index 6e90483..7897e2c 100644 --- a/src/testdriver/crystallization.py +++ b/src/testdriver/crystallization.py @@ -35,6 +35,7 @@ class Trajectory: step_id: str action_name: str surface_id: str + method: str target: str fields: tuple[str, ...] @@ -42,7 +43,7 @@ class Trajectory: return json.dumps( { "step": self.step_id, "action": self.action_name, - "surface": self.surface_id, + "surface": self.surface_id, "method": self.method, "target": self.target, "fields": sorted(self.fields), }, sort_keys=True, @@ -62,6 +63,7 @@ def capture(pack: Mapping[str, Any]) -> tuple[Trajectory, ...]: step_id=obs["step_id"], action_name=str(mechanics.get("action", "")).split("(")[0], surface_id=obs["data"].get("surface", ""), + method=("POST" if obs["data"].get("surface") == "browser" else "CALL"), target=target, fields=tuple(sorted(fields)), )) diff --git a/src/testdriver/evidence.py b/src/testdriver/evidence.py index bdbc78d..01a65a8 100644 --- a/src/testdriver/evidence.py +++ b/src/testdriver/evidence.py @@ -72,6 +72,12 @@ class EvidencePack: def of_stratum(self, stratum: Stratum) -> list[Observation]: return [o for o in self.observations if o.stratum is stratum] + def latest(self, kind: str) -> Observation | None: + for observation in reversed(self.observations): + if observation.kind == kind: + return observation + return None + def to_json(self) -> str: payload = asdict(self) payload["observations"] = [ diff --git a/src/testdriver/oracles.py b/src/testdriver/oracles.py index 17f1c20..705b1ae 100644 --- a/src/testdriver/oracles.py +++ b/src/testdriver/oracles.py @@ -3,10 +3,6 @@ An oracle reads only the independent observation snapshot. It has no access to the actor, to the driver, or to what either of them believes happened. -`SUSPICIOUS` was removed at T10: no oracle could produce it, and a verdict -nothing can emit is a promise the framework does not keep. It returns, with an -identifier, if a mechanism ever needs it. - `INCONCLUSIVE` is a first-class outcome, not a failure mode of the framework. An oracle that cannot support a judgment from the retained evidence must say so rather than defaulting either way — silently defaulting to PASS hides defects, @@ -25,6 +21,7 @@ from .intent import Claim, Invariant class Verdict(str, Enum): PASS = "PASS" FAIL = "FAIL" + SUSPICIOUS = "SUSPICIOUS" INCONCLUSIVE = "INCONCLUSIVE" @@ -104,4 +101,6 @@ def overall(judgments: list[Judgment]) -> Verdict: return Verdict.FAIL if Verdict.INCONCLUSIVE in verdicts: return Verdict.INCONCLUSIVE + if Verdict.SUSPICIOUS in verdicts: + return Verdict.SUSPICIOUS return Verdict.PASS diff --git a/src/testdriver/runner.py b/src/testdriver/runner.py index 9aeb9e1..833eab0 100644 --- a/src/testdriver/runner.py +++ b/src/testdriver/runner.py @@ -56,24 +56,6 @@ class Runner: # -- independence guards --------------------------------------------- - def _isolation_violations(self) -> list[str]: - """Does any actor hold another's canary? - - Run on every scenario, not only on ones written to test isolation. - """ - canaries = {actor.canary: actor.id for actor in self._world.cast} - violations: list[str] = [] - for actor in self._world.cast: - for key in actor.known_keys(): - value = actor.recall(key) - owner = canaries.get(value) if isinstance(value, str) else None - if owner is not None and owner != actor.id: - violations.append( - f"actor {actor.id!r} holds the private marker of {owner!r} " - f"under key {key!r}" - ) - return violations - def _assert_collector_independence(self, stratum: Stratum, collector: str) -> None: """S2 and S3 evidence may never be attributed to an actor. @@ -129,12 +111,6 @@ class Runner: EnergyEvent(asset.id, run_id, EnergyEventType.EXECUTED).as_dict() ) - isolation = self._isolation_violations() - self._record( - pack, Stratum.JUDGMENT, self._observer.name, "actor_isolation", - {"violations": isolation, "actors": sorted(self._world.cast.actors)}, None, - ) - judgments: list[Judgment] = [] scenario_sound = True claims_by_step: dict[str, list] = {} @@ -190,6 +166,7 @@ class Runner: "action": step.action.name, "surface_used": realization.surface_id, "refused_by_sut": refused, + "refusal_expected": step.expect_refusal, "postcondition_met": postcondition_met, }, step.id, @@ -199,7 +176,7 @@ class Runner: # An action that was accepted but did not take effect did happen — # and that is a statement about the system, judged below, not a # reason to stop judging. - if refused: + if refused and not step.expect_refusal: scenario_sound = False # --- invariants after every step ----------------------------- diff --git a/src/testdriver/scenario.py b/src/testdriver/scenario.py index c4ac795..a41de6e 100644 --- a/src/testdriver/scenario.py +++ b/src/testdriver/scenario.py @@ -17,6 +17,7 @@ class Step: id: str actor_id: str action: SemanticAction + expect_refusal: bool = False @dataclass(frozen=True, slots=True) diff --git a/src/testdriver/world.py b/src/testdriver/world.py index 55289c8..04f2341 100644 --- a/src/testdriver/world.py +++ b/src/testdriver/world.py @@ -8,20 +8,12 @@ to be executed by the same process. from __future__ import annotations -import uuid from dataclasses import dataclass, field from typing import Any, Iterator -def _canary() -> str: - """A value private to one actor, unguessable and unique per construction. - - Exists so that isolation is *observable*. Without it a run in which every - actor shares one memory store produces evidence indistinguishable from a - correct one — the guarantee holds only in scenarios written to expose it, - which is no guarantee at all (F-0003). - """ - return f"canary-{uuid.uuid4().hex}" +class ActorIsolationError(Exception): + """Raised when one actor is offered another actor's private state.""" @dataclass(slots=True) @@ -36,13 +28,8 @@ class Actor: id: str display_name: str credentials: dict[str, str] = field(default_factory=dict) + session: dict[str, Any] = field(default_factory=dict) _memory: dict[str, Any] = field(default_factory=dict, repr=False) - canary: str = field(default_factory=_canary) - - def __post_init__(self) -> None: - # Seeded automatically, on every actor, in every scenario. An isolation - # violation now leaves a trace whether or not anyone thought to look. - self._memory.setdefault("__canary__", self.canary) def remember(self, key: str, value: Any) -> None: self._memory[key] = value @@ -78,14 +65,15 @@ class Cast: @dataclass(slots=True) class World: - """The handle to the system under test, plus the cast acting on it. + """Initial state plus the handle to the system under test. - Replay comes from rebuilding the lab through `build_lab`, not from a seed - dict carried here — the dict was written at T04, never read, and removed at - T10. + `seed` is everything needed to rebuild the initial state, so that a run can + be replayed from a known starting point rather than from wherever the + previous run happened to leave things. """ id: str sut: Any sut_version: str + seed: dict[str, Any] = field(default_factory=dict) cast: Cast = field(default_factory=Cast) diff --git a/tests/__pycache__/test_agentic_realization.cpython-312-pytest-7.4.4.pyc b/tests/__pycache__/test_agentic_realization.cpython-312-pytest-7.4.4.pyc new file mode 100644 index 0000000..d2ac8c2 Binary files /dev/null and b/tests/__pycache__/test_agentic_realization.cpython-312-pytest-7.4.4.pyc differ diff --git a/tests/__pycache__/test_classification.cpython-312-pytest-7.4.4.pyc b/tests/__pycache__/test_classification.cpython-312-pytest-7.4.4.pyc new file mode 100644 index 0000000..ba46d32 Binary files /dev/null and b/tests/__pycache__/test_classification.cpython-312-pytest-7.4.4.pyc differ diff --git a/tests/__pycache__/test_crystallization.cpython-312-pytest-7.4.4.pyc b/tests/__pycache__/test_crystallization.cpython-312-pytest-7.4.4.pyc new file mode 100644 index 0000000..66fed66 Binary files /dev/null and b/tests/__pycache__/test_crystallization.cpython-312-pytest-7.4.4.pyc differ diff --git a/tests/__pycache__/test_kernel_guarantees.cpython-312-pytest-7.4.4.pyc b/tests/__pycache__/test_kernel_guarantees.cpython-312-pytest-7.4.4.pyc new file mode 100644 index 0000000..0fcf25b Binary files /dev/null and b/tests/__pycache__/test_kernel_guarantees.cpython-312-pytest-7.4.4.pyc differ diff --git a/tests/__pycache__/test_kernel_guarantees.cpython-312.pyc b/tests/__pycache__/test_kernel_guarantees.cpython-312.pyc new file mode 100644 index 0000000..6f6df56 Binary files /dev/null and b/tests/__pycache__/test_kernel_guarantees.cpython-312.pyc differ diff --git a/tests/__pycache__/test_lab_ground_truth.cpython-312-pytest-7.4.4.pyc b/tests/__pycache__/test_lab_ground_truth.cpython-312-pytest-7.4.4.pyc new file mode 100644 index 0000000..5deb051 Binary files /dev/null and b/tests/__pycache__/test_lab_ground_truth.cpython-312-pytest-7.4.4.pyc differ diff --git a/tests/__pycache__/test_reference_scenario.cpython-312-pytest-7.4.4.pyc b/tests/__pycache__/test_reference_scenario.cpython-312-pytest-7.4.4.pyc new file mode 100644 index 0000000..9f5843b Binary files /dev/null and b/tests/__pycache__/test_reference_scenario.cpython-312-pytest-7.4.4.pyc differ diff --git a/tests/selfverification/__pycache__/__init__.cpython-312.pyc b/tests/selfverification/__pycache__/__init__.cpython-312.pyc new file mode 100644 index 0000000..0b3bffb Binary files /dev/null and b/tests/selfverification/__pycache__/__init__.cpython-312.pyc differ diff --git a/tests/selfverification/__pycache__/checks.cpython-312.pyc b/tests/selfverification/__pycache__/checks.cpython-312.pyc new file mode 100644 index 0000000..3c615c3 Binary files /dev/null and b/tests/selfverification/__pycache__/checks.cpython-312.pyc differ diff --git a/tests/selfverification/__pycache__/test_checks_can_fail.cpython-312-pytest-7.4.4.pyc b/tests/selfverification/__pycache__/test_checks_can_fail.cpython-312-pytest-7.4.4.pyc new file mode 100644 index 0000000..07825a6 Binary files /dev/null and b/tests/selfverification/__pycache__/test_checks_can_fail.cpython-312-pytest-7.4.4.pyc differ diff --git a/tests/selfverification/__pycache__/test_self_verification.cpython-312-pytest-7.4.4.pyc b/tests/selfverification/__pycache__/test_self_verification.cpython-312-pytest-7.4.4.pyc new file mode 100644 index 0000000..fee743c Binary files /dev/null and b/tests/selfverification/__pycache__/test_self_verification.cpython-312-pytest-7.4.4.pyc differ diff --git a/tests/selfverification/checks.py b/tests/selfverification/checks.py index 2329882..47d321e 100644 --- a/tests/selfverification/checks.py +++ b/tests/selfverification/checks.py @@ -94,23 +94,6 @@ def check_actor_isolation( return violations -def check_isolation_was_examined(pack: Mapping[str, Any]) -> list[str]: - """Every run must carry a verdict on actor isolation — F-0003, resolved. - - Before this existed, isolation was only observable in scenarios written to - expose it: a run in which every actor shared one memory store produced - evidence indistinguishable from a correct one. Actors now carry an automatic - private marker and the runner examines them on every scenario, so the absence - of this observation is itself a failure. - """ - for obs in _observations(pack): - if obs["kind"] != "actor_isolation": - continue - violations = obs["data"].get("violations") or [] - return [f"actor isolation violated: {v}" for v in violations] - return ["this run did not examine actor isolation at all"] - - # --- td://self/oracle-independence --------------------------------------- diff --git a/tests/selfverification/test_checks_can_fail.py b/tests/selfverification/test_checks_can_fail.py index b131778..c727c0d 100644 --- a/tests/selfverification/test_checks_can_fail.py +++ b/tests/selfverification/test_checks_can_fail.py @@ -30,7 +30,6 @@ from tests.selfverification.checks import ( check_actor_isolation, check_evidence_supports_every_verdict, check_intent_independence, - check_isolation_was_examined, check_no_actor_collected_judgment, check_runs_agree, check_verdicts_follow_from_judgment_evidence, @@ -187,35 +186,3 @@ def test_unrecorded_provenance_is_caught(): tampered = copy.deepcopy(pack) tampered["provenance_index"] = {} assert check_intent_independence(tampered) - - -# --- F-0003 resolution: isolation observed on every run ------------------- - - -def test_an_unexamined_run_is_caught(): - """A run that never looked at isolation must not read as isolated.""" - pack, _, _ = run_and_serialize() - tampered = copy.deepcopy(pack) - tampered["observations"] = [ - o for o in tampered["observations"] if o["kind"] != "actor_isolation" - ] - assert check_isolation_was_examined(tampered) - - -def test_a_leak_is_caught_without_the_test_planting_anything(): - """The regression F-0003 leaves behind. - - No secrets seeded by the harness, no scenario written to expose isolation. - An actor holding another's automatic marker is caught by the ordinary run. - """ - from testdriver import Oracle, Runner - from scenarios.alice_bob_carol import build - - world, driver, observer, asset, oracle = build() - world.cast["bob"].remember("overheard", world.cast["alice"].canary) - result = Runner(world, driver, observer, oracle).run(asset) - pack = json.loads(result.evidence.to_json()) - - violations = check_isolation_was_examined(pack) - assert violations - assert "holds the private marker of 'alice'" in violations[0] diff --git a/tests/selfverification/test_self_verification.py b/tests/selfverification/test_self_verification.py index 3b06899..506eb79 100644 --- a/tests/selfverification/test_self_verification.py +++ b/tests/selfverification/test_self_verification.py @@ -17,7 +17,6 @@ from tests.selfverification.checks import ( check_actor_isolation, check_evidence_supports_every_verdict, check_intent_independence, - check_isolation_was_examined, check_no_actor_collected_judgment, check_runs_agree, check_verdicts_follow_from_judgment_evidence, @@ -57,14 +56,6 @@ def test_actor_isolation(): assert check_actor_isolation(pack, SECRETS, memories) == [] -@pytest.mark.parametrize("mutations", [(), ("M15",), ("M01",)]) -def test_every_run_examines_isolation(mutations): - """F-0003 resolved: isolation is now a property of runs, not of scenarios - written to expose it. No canaries need be planted by the test.""" - pack, _, _ = run_and_serialize(*mutations) - assert check_isolation_was_examined(pack) == [] - - # --- td://self/oracle-independence --------------------------------------- diff --git a/tests/test_audit_core_e2_use_case.py b/tests/test_audit_core_e2_use_case.py deleted file mode 100644 index 95bc33e..0000000 --- a/tests/test_audit_core_e2_use_case.py +++ /dev/null @@ -1,147 +0,0 @@ -"""The audit-core use case is durable intent, even before drivers can run it.""" - -from __future__ import annotations - -from copy import deepcopy - -from testdriver import Oracle, Provenance, Verdict -from usecases.audit_core_e2_tenant_boundary import ( - PHASE_CONTRACTS, - PRECEDENT_EVIDENCE_REF, - ROLE_CONTRACTS, - TEST_USE_CASE, -) - - -def passing_observations() -> dict[str, object]: - absent = { - "status": 404, - "schema": ("$", "$.error:str"), - "digest": "absent-surface", - "fixture_match_count": 0, - } - return { - "event_by_id": { - "owner": {"status": 200, "fixture_match_count": 2}, - "attacker": dict(absent), - "absent": dict(absent), - }, - "correlation_slice": { - "owner": {"status": 200, "fixture_match_count": 2}, - "attacker": {"status": 200, "fixture_match_count": 0}, - }, - "append_as_b": { - "attacker": {"status": 400}, - "state_after": dict(absent), - "absent": dict(absent), - }, - "run": { - "attempted_operations": 10, - "maximum_operations": 30, - "max_concurrency": 1, - "rate_per_minute": 10, - "target_revision_matches": True, - "attribution_complete": True, - }, - "cleanup": { - "completed_before_expiry": True, - "runner_absent": True, - "temporary_identities": [], - "kv_paths": [], - "projection_resources": [], - "mounted_secrets": [], - "target_ready": True, - "secret_values_observed": False, - }, - "report": { - "bound_to_projection_receipt": True, - "bound_to_cleanup_receipt": True, - "bounded_assurance_statement": True, - "delivered_to_risk_nexus": True, - }, - } - - -def judgments(observations): - oracle = Oracle() - assertions = (*TEST_USE_CASE.claims, *TEST_USE_CASE.invariants) - return { - assertion.id: oracle.judge( - assertion, - observations, - getattr(assertion, "after_step", None), - ) - for assertion in assertions - } - - -def test_precedent_calibration_satisfies_every_assertion(): - result = judgments(passing_observations()) - assert {judgment.verdict for judgment in result.values()} == {Verdict.PASS} - assert PRECEDENT_EVIDENCE_REF.endswith( - "AUDIT-WP-0008-T05-whitehat-e2-03-pass-2026-08-22.md" - ) - - -def test_cross_tenant_event_disclosure_fails_the_claim(): - observations = passing_observations() - observations["event_by_id"]["attacker"] = { - "status": 200, - "schema": ("$", "$.event_id:str"), - "digest": "tenant-b-event", - "fixture_match_count": 2, - } - result = judgments(observations) - assert result["c-event-id-tenant-scoped"].verdict is Verdict.FAIL - - -def test_missing_cleanup_evidence_is_inconclusive(): - observations = passing_observations() - del observations["cleanup"] - result = judgments(observations) - assert result["c-receipt-bound-cleanup"].verdict is Verdict.INCONCLUSIVE - assert result["i-no-secret-evidence"].verdict is Verdict.INCONCLUSIVE - - -def test_cleanup_residue_fails_closed(): - observations = deepcopy(passing_observations()) - observations["cleanup"]["temporary_identities"] = ["unexpected-residue"] - result = judgments(observations) - assert result["c-receipt-bound-cleanup"].verdict is Verdict.FAIL - - -def test_intent_has_independent_provenance_and_separate_roles(): - assert TEST_USE_CASE.provenance is Provenance.SPEC - assert all( - assertion.provenance is Provenance.SPEC - for assertion in (*TEST_USE_CASE.claims, *TEST_USE_CASE.invariants) - ) - roles = {role.id for role in ROLE_CONTRACTS} - assert { - "authorizer", - "target-owner", - "credential-custodian", - "security-coordinator", - "cluster-executor", - "tenant-a-attacker", - "tenant-b-control", - "independent-observer", - } == roles - - -def test_schedule_requires_cleanup_before_finalization(): - phases = {phase.id: phase for phase in PHASE_CONTRACTS} - assert phases["finalize-and-deliver"].requires == ("verify-cleanup",) - assert phases["verify-cleanup"].requires == ("cleanup-custody",) - assert phases["cleanup-custody"].requires == ("delete-runner",) - assert phases["run-probes"].requires == ("ready-runner",) - - -def test_every_role_participates_in_a_phase(): - declared = {role.id for role in ROLE_CONTRACTS} - scheduled = { - role - for phase in PHASE_CONTRACTS - for role in phase.driver_roles - } - assert scheduled == declared diff --git a/tests/test_classification.py b/tests/test_classification.py index ccf6c39..37d5d2f 100644 --- a/tests/test_classification.py +++ b/tests/test_classification.py @@ -194,98 +194,3 @@ def test_safe_to_accept_is_a_closed_set(): assert SAFE_TO_ACCEPT == { Classification.UNCHANGED, Classification.MECHANICAL_ADAPTATION, } - - -# --- classifications that no lab mutation happens to produce --------------- -# -# Two outcomes were declared at T08 and exercised by nothing. Left that way they -# are decoration: code that has never run is code nobody has checked. Rather than -# delete meaningful outcomes or trust them untested, both are given a case. - - -def test_intent_change_is_detected_when_the_claim_set_moves(baseline): - """`INTENT_CHANGED` is a fact about the recorded use case, not an inference. - - It fires because a human edited what is being asserted — which is why it is - detectable at all, where `SEMANTIC_CHANGE` was not (F-0006). - """ - import copy - - altered = copy.deepcopy(baseline) - altered["provenance_index"]["c-newly-added-claim"] = "human" - outcome = classify(baseline, altered) - assert outcome.classification is Classification.INTENT_CHANGED - assert not outcome.safe_to_accept - - -def test_realization_failure_is_distinguishable_from_ambiguity(): - """`REALIZATION_FAILED` says "we could not act"; `AMBIGUOUS` says "we do not know". - - Every lab mutation that breaks realization also strands a claim, so the - catalogue only ever produces `AMBIGUOUS`. This builds the case the catalogue - cannot: a step that fails while every assertion in the run still holds and - none of them depended on it. - - Note that a run asserting *nothing at all* is `AMBIGUOUS`, not - `REALIZATION_FAILED` — a use case with no claims cannot conclude anything, - however well its steps ran. - """ - from testdriver import ( - Actor, Cast, Invariant, Oracle, Runner, Scenario, SemanticAction, - StateObserver, Step, UseCase, VerificationAsset, World, - ) - from testdriver.agentic import DiscoveryRuntime - from testdriver.browser import BrowserDriver - from testdriver.observers import Watch - from testdriver.provenance import Provenance - from lab.mutations import ObservationChannel - - use_case = UseCase( - "uc-audit-only", "Sharing leaves an ordered audit trail", - "Alice shares R with Bob; the audit trail stays ordered.", - Provenance.HUMAN, - invariants=(Invariant( - "i-audit-ordered", "The audit trail is append-only", Provenance.HUMAN, - lambda obs: [e["sequence"] for e in obs["audit:R"]] - == sorted(e["sequence"] for e in obs["audit:R"]), - ),), - ) - - def run(*mutations): - with journey_lab_server(*mutations) as (app, tokens, base_url): - app.request(tokens["alice"], "create_resource", - resource_id="R", content="x") - cast = Cast() - cast.add(Actor("alice", "Alice", credentials={"token": tokens["alice"]})) - scenario = Scenario( - "sc-audit-only", use_case, - watches=(Watch("bob", "R"),), - steps=(Step("s1", "alice", SemanticAction( - "grant_access", {"subject_id": "bob", "permission": "READ"}, - permitted_surfaces=frozenset({"browser"}), - )),), - ) - driver = BrowserDriver(base_url, tokens, DiscoveryRuntime(), "R") - observer = StateObserver(ObservationChannel(app), scenario.watches) - world = World("w-audit", app, app.version, cast=cast) - return json.loads( - Runner(world, driver, observer, Oracle()) - .run(VerificationAsset("va-audit-only", scenario)) - .evidence.to_json() - ) - - outcome = classify(run(), run("M23")) # the control is gone from the UI - assert outcome.classification is Classification.REALIZATION_FAILED - assert not outcome.safe_to_accept - - -def test_no_classification_is_unreachable(): - """Every declared outcome must be produced somewhere in this suite. - - An outcome nothing can emit is the same kind of dead promise `SUSPICIOUS` - was before T10 removed it. - """ - exercised = set(EXPECTED.values()) | { - Classification.INTENT_CHANGED, Classification.REALIZATION_FAILED, - } - assert exercised == set(Classification) diff --git a/tests/test_reference_scenario.py b/tests/test_reference_scenario.py index 70554d5..5739969 100644 --- a/tests/test_reference_scenario.py +++ b/tests/test_reference_scenario.py @@ -67,18 +67,4 @@ def test_actors_hold_isolated_credentials_and_memory(): assert alice.credentials["token"] != bob.credentials["token"] alice.remember("secret", "only alice knows this") assert bob.recall("secret") is None - # Every actor carries its own automatic private marker (F-0003) and nothing - # else it was not given. - assert bob.known_keys() == ("__canary__",) - assert alice.canary != bob.canary - - -def test_every_run_records_a_verdict_on_isolation(): - """F-0003: isolation is examined on every scenario, not only on ones - written to expose it.""" - result, _ = run_once() - examined = [ - obs for obs in result.evidence.observations if obs.kind == "actor_isolation" - ] - assert len(examined) == 1 - assert examined[0].data["violations"] == [] + assert bob.known_keys() == () diff --git a/usecases/__init__.py b/usecases/__init__.py deleted file mode 100644 index 1e646be..0000000 --- a/usecases/__init__.py +++ /dev/null @@ -1,5 +0,0 @@ -"""Durable test intent, kept separate from concrete scenario realizations.""" - -from .audit_core_e2_tenant_boundary import TEST_USE_CASE - -__all__ = ["TEST_USE_CASE"] diff --git a/usecases/audit_core_e2_tenant_boundary.py b/usecases/audit_core_e2_tenant_boundary.py deleted file mode 100644 index d588367..0000000 --- a/usecases/audit_core_e2_tenant_boundary.py +++ /dev/null @@ -1,342 +0,0 @@ -"""Audit-core E2 tenant-boundary TestUseCase. - -This captures the intent and role hand-offs exercised successfully by -WH-ENG-20260822-AUDIT-E2-03. It is deliberately not a runnable ``Scenario`` -yet. The current kernel has no causal/time-window schedule, external custody -driver, Kubernetes driver, or independent cleanup observer capable of realizing -this use case without an attended operator. - -The claims come from the approved engagement and AUDIT-WP-0008-T05, not from -reverse-engineering the successful responses. The -03 report is precedent -evidence and a calibration example; it is not the source of truth for verdicts. -""" - -from __future__ import annotations - -from dataclasses import dataclass -from typing import Mapping - -from testdriver import Claim, Invariant, Provenance, UseCase - - -SPEC_REF = ( - "audit-core/workplans/AUDIT-WP-0008-tenancy-posture-alignment.md" - "#AUDIT-WP-0008-T05" -) -ENGAGEMENT_REF = ( - "whitehat-security/engagements/2026-08-22-audit-core-e2-03.json" -) -PRECEDENT_EVIDENCE_REF = ( - "audit-core/docs/evidence/" - "AUDIT-WP-0008-T05-whitehat-e2-03-pass-2026-08-22.md" -) - - -@dataclass(frozen=True, slots=True) -class RoleContract: - """One independently driven responsibility in the attended protocol.""" - - id: str - responsibility: str - may_observe_secret_values: bool = False - - -@dataclass(frozen=True, slots=True) -class PhaseContract: - """Causal intent for the future orchestrator; never a shell transcript.""" - - id: str - driver_roles: tuple[str, ...] - requires: tuple[str, ...] - completion: str - timing: str - - -ROLE_CONTRACTS = ( - RoleContract( - "authorizer", - "Approve the exact production package, target, scope, window and cleanup " - "obligation; hold no test credential.", - ), - RoleContract( - "target-owner", - "Acknowledge the exact target revision, routes, fixtures and abort limits.", - ), - RoleContract( - "credential-custodian", - "Preflight, project and revoke two short-lived tenant-scoped identities; " - "emit value-safe projection and cleanup receipts.", - may_observe_secret_values=True, - ), - RoleContract( - "security-coordinator", - "Validate the engagement, admit only the receipt-bound plane lease, invoke " - "the bounded probe, finalize the report and route it to risk-nexus.", - ), - RoleContract( - "cluster-executor", - "Create, wait for and delete only the approved runner pod; mount but never " - "read the projected Secret through the control plane.", - ), - RoleContract( - "tenant-a-attacker", - "Use an ordinary tenant-A identity to attempt the declared cross-tenant " - "reads and write.", - may_observe_secret_values=True, - ), - RoleContract( - "tenant-b-control", - "Use an independently scoped tenant-B identity only for known-good owner " - "controls and state-after checks.", - may_observe_secret_values=True, - ), - RoleContract( - "independent-observer", - "Collect sanitized outcomes, target readiness and post-cleanup absence " - "without accepting an actor's report as its own evidence.", - ), -) - - -PHASE_CONTRACTS = ( - PhaseContract( - "authorize", - ("authorizer",), - (), - "Exact engagement and custody contracts are approved.", - "Before projection and before the engagement window opens.", - ), - PhaseContract( - "acknowledge-target", - ("target-owner",), - ("authorize",), - "Target owner accepts the exact revision, routes and bounded fixtures.", - "Before any temporary credential exists.", - ), - PhaseContract( - "preflight", - ("credential-custodian",), - ("acknowledge-target",), - "Target is ready and all engagement-specific resources are absent.", - "Immediately before the projection cutoff.", - ), - PhaseContract( - "project-custody", - ("credential-custodian",), - ("preflight",), - "Exactly two scoped handles exist and a value-safe receipt is emitted.", - "Inside the narrow projection window; expiry equals engagement end.", - ), - PhaseContract( - "admit-plane", - ("security-coordinator",), - ("project-custody",), - "Receipt, broker contract, target revision and cleanup authority agree.", - "After projection and before any target packet.", - ), - PhaseContract( - "ready-runner", - ("cluster-executor",), - ("admit-plane",), - "The exact attributed runner is Ready with its Secret mounted.", - "Inside the authorized engagement window.", - ), - PhaseContract( - "run-probes", - ( - "security-coordinator", - "tenant-a-attacker", - "tenant-b-control", - ), - ("ready-runner",), - "Calibrated owner, attacker and absent controls produce a sanitized report.", - "Before credential and engagement expiry.", - ), - PhaseContract( - "delete-runner", - ("cluster-executor",), - ("run-probes",), - "The exact runner pod is absent.", - "Immediately after evidence collection.", - ), - PhaseContract( - "cleanup-custody", - ("credential-custodian",), - ("delete-runner",), - "Receipt-bound cleanup revokes both identities and removes exact resources.", - "Before credential and engagement expiry.", - ), - PhaseContract( - "verify-cleanup", - ("independent-observer",), - ("cleanup-custody",), - "Both identities, exact KV paths, projection resources, Secret and runner " - "are independently absent while the target remains Ready.", - "After cleanup and before credential and engagement expiry.", - ), - PhaseContract( - "finalize-and-deliver", - ("security-coordinator",), - ("verify-cleanup",), - "The report is bound to projection and cleanup receipts and reaches risk-nexus.", - "Only after cleanup is independently observable.", - ), -) - - -def _same_absence_surface(left: Mapping[str, object], right: Mapping[str, object]) -> bool: - """Compare only the sanitized non-disclosure surface.""" - - return ( - left["status"] == right["status"] - and left["schema"] == right["schema"] - and left["digest"] == right["digest"] - and left["fixture_match_count"] == 0 - and right["fixture_match_count"] == 0 - ) - - -def _event_id_is_tenant_scoped(obs: Mapping[str, object]) -> bool: - probe = obs["event_by_id"] - owner = probe["owner"] - attacker = probe["attacker"] - absent = probe["absent"] - return ( - owner["status"] == 200 - and owner["fixture_match_count"] > 0 - and _same_absence_surface(attacker, absent) - ) - - -def _correlation_slice_is_tenant_scoped(obs: Mapping[str, object]) -> bool: - probe = obs["correlation_slice"] - return ( - probe["owner"]["status"] == 200 - and probe["owner"]["fixture_match_count"] > 0 - and probe["attacker"]["status"] == 200 - and probe["attacker"]["fixture_match_count"] == 0 - ) - - -def _cross_tenant_append_is_refused(obs: Mapping[str, object]) -> bool: - probe = obs["append_as_b"] - return ( - probe["attacker"]["status"] in {400, 403, 404} - and _same_absence_surface(probe["state_after"], probe["absent"]) - ) - - -def _cleanup_is_complete(obs: Mapping[str, object]) -> bool: - cleanup = obs["cleanup"] - return ( - cleanup["completed_before_expiry"] is True - and cleanup["runner_absent"] is True - and cleanup["temporary_identities"] == [] - and cleanup["kv_paths"] == [] - and cleanup["projection_resources"] == [] - and cleanup["mounted_secrets"] == [] - and cleanup["target_ready"] is True - ) - - -def _execution_remains_bounded(obs: Mapping[str, object]) -> bool: - run = obs["run"] - return ( - run["attempted_operations"] <= run["maximum_operations"] - and run["max_concurrency"] == 1 - and run["rate_per_minute"] <= 10 - and run["target_revision_matches"] is True - and run["attribution_complete"] is True - ) - - -def _secret_values_do_not_enter_evidence(obs: Mapping[str, object]) -> bool: - return obs["cleanup"]["secret_values_observed"] is False - - -def _report_preserves_assurance_limit(obs: Mapping[str, object]) -> bool: - report = obs["report"] - return ( - report["bound_to_projection_receipt"] is True - and report["bound_to_cleanup_receipt"] is True - and report["bounded_assurance_statement"] is True - and report["delivered_to_risk_nexus"] is True - ) - - -TEST_USE_CASE = UseCase( - id="uc-audit-core-e2-tenant-boundary", - title="Demonstrate audit-core's E2 tenant boundary with separate drivers", - narrative=( - "Within one explicitly authorized production window, independent target, " - "custody, security, cluster, attacker, control and observer roles establish " - "that an ordinary tenant-A identity cannot read or create tenant-B audit " - "fixtures. The run must remain bounded, preserve non-disclosure semantics, " - "clean every temporary resource before expiry, and deliver a cleanup-bound " - "sanitized report without retaining credential values." - ), - provenance=Provenance.SPEC, - source_ref=SPEC_REF, - claims=( - Claim( - "c-event-id-tenant-scoped", - "Tenant A cannot distinguish tenant B's event id from an absent event", - Provenance.SPEC, - _event_id_is_tenant_scoped, - after_step="run-probes", - source_ref=ENGAGEMENT_REF, - ), - Claim( - "c-correlation-slice-tenant-scoped", - "Tenant A's correlation slice contains no tenant-B fixture", - Provenance.SPEC, - _correlation_slice_is_tenant_scoped, - after_step="run-probes", - source_ref=ENGAGEMENT_REF, - ), - Claim( - "c-cross-tenant-append-refused", - "Tenant A cannot append an audit event attributed to tenant B", - Provenance.SPEC, - _cross_tenant_append_is_refused, - after_step="run-probes", - source_ref=ENGAGEMENT_REF, - ), - Claim( - "c-receipt-bound-cleanup", - "Every engagement-specific runtime and custody resource is absent " - "before expiry while the target remains ready", - Provenance.SPEC, - _cleanup_is_complete, - after_step="verify-cleanup", - source_ref=SPEC_REF, - ), - ), - invariants=( - Invariant( - "i-bounded-execution", - "The run stays within its operation, rate, concurrency, revision and " - "attribution boundaries", - Provenance.SPEC, - _execution_remains_bounded, - source_ref=ENGAGEMENT_REF, - ), - Invariant( - "i-no-secret-evidence", - "Credential values never enter retained observations or reports", - Provenance.SPEC, - _secret_values_do_not_enter_evidence, - source_ref=SPEC_REF, - ), - Invariant( - "i-bounded-assurance-report", - "Delivery is receipt-bound and says only that attempted attacks failed", - Provenance.SPEC, - _report_preserves_assurance_limit, - source_ref=SPEC_REF, - ), - ), -) - -# Alias retained for the existing scenario-module convention. -USE_CASE = TEST_USE_CASE diff --git a/workplans/TD-WP-0002-vertical-spike-crystallization.md b/workplans/TD-WP-0002-vertical-spike-crystallization.md index 36317cd..dece354 100644 --- a/workplans/TD-WP-0002-vertical-spike-crystallization.md +++ b/workplans/TD-WP-0002-vertical-spike-crystallization.md @@ -4,11 +4,11 @@ type: workplan title: "Vertical spike: falsify the crystallization thesis" domain: infotech repo: test-driver -status: finished +status: active owner: codex topic_slug: custodian created: "2026-08-22" -updated: "2026-08-23" +updated: "2026-08-22" state_hub_workstream_id: "68b252b8-03e7-55c7-8dec-ac13f6ca0aa1" --- @@ -461,7 +461,7 @@ to execute than the agentic ancestor. ```task id: TD-WP-0002-T10 -status: done +status: todo priority: medium state_hub_task_id: "884c2e91-b56d-5b5e-a2d3-52d2a2768cae" ``` @@ -478,42 +478,3 @@ retaining them, and update the Concept ↔ Implementation Fitness Map. Convert at least one framework finding discovered during T04–T09 into a permanent self-verification or deterministic regression — that closes the first genuine Concept–Implementation Fitness Loop. - -**Done 2026-08-23.** Full write-up: -`history/2026-08-23-td-wp-0002-gate-review.md`. 178 tests pass. - -**All four gate criteria met.** M01/M02 absorbed with the claim set unchanged; -the authorization defects escalated with regressions named; one asset -crystallized and running with no model; **False Adaptation Rate 0/7**, with 12 of -13 mechanical mutations absorbed so the safety result is not bought by escalating -everything. - -**Fitness loop closed via F-0003.** Actor isolation left no trace in ordinary -evidence — it was a property of scenarios written to expose it. Every actor is -now seeded with an automatic private marker and the runner examines all actors on -every scenario, with two permanent regressions behind it, one of which catches a -leak with the harness planting nothing. - -**Compression — six abstractions removed**, each declared and never used: -`Verdict.SUSPICIOUS` (a verdict no oracle could emit), `Step.expect_refusal`, -`ActorIsolationError`, `World.seed`, `EvidencePack.latest()`, -`Trajectory.method`. - -**F-0008 — Temperature may be redundant.** Crystallization was built end to end -without it ever being consulted; what triggered freezing was *measured stability -of realization*, which is observed rather than declared. A declared temperature -is a claim nobody verifies. Not deleted — two unevidenced cases for keeping it -are recorded — but gated, along with `energy.py`: if the next workplan closes -without a decision consulting either, they go. - -**Two classifications had never run.** `INTENT_CHANGED` and `REALIZATION_FAILED` -were declared at T08 and produced by no test. Both now have purpose-built cases, -and `test_no_classification_is_unreachable` fails if a seventh is added without -one. - -**What this leaves.** The thesis survived contact with its own evidence in -narrowed form, and the demonstration is architectural rather than statistical. -The highest-value next step is a **bounded live-model experiment** — F-0005 and -F-0007 converge on it independently, and one experiment settles both whether a -model adds capability and whether crystallization has a product rationale. That -should come before any further framework feature.