T10: gate review and first compression pass
All four gate criteria met. False Adaptation Rate 0/7 with 12 of 13 mechanical mutations absorbed. 178 tests pass. TD-WP-0002 finished. Fitness loop closed via F-0003: actor isolation was a property of scenarios written to expose it, not of runs. Actors now carry an automatic private marker and the runner examines all of them on every scenario, with two permanent regressions behind it. Compression - six abstractions removed, each declared and never used: Verdict.SUSPICIOUS (a verdict no oracle could emit), Step.expect_refusal, ActorIsolationError, World.seed, EvidencePack.latest, Trajectory.method. F-0008: Temperature may be redundant. Crystallization was built without it ever being consulted; measured stability of realization did the work, and is observed rather than declared. Gated for removal alongside energy.py. INTENT_CHANGED and REALIZATION_FAILED had never run. Both now have purpose-built cases and a test that fails if a seventh outcome is added without one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 1629012@bnt-lap001 Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
This commit is contained in:
parent
4f4219d8f7
commit
1b9860a8ee
40 changed files with 1074 additions and 46 deletions
13
INTENT.md
13
INTENT.md
|
|
@ -116,6 +116,14 @@ COLD -> deterministic
|
|||
|
||||
The framework should make it natural for tests to crystallize as software cools.
|
||||
|
||||
> **Under review since T10 — see `research/findings/F-0008-temperature-may-be-redundant.md`.**
|
||||
> Crystallization was built and demonstrated without Temperature ever being
|
||||
> consulted. What actually triggered freezing was *measured stability of
|
||||
> realization* — the same path found across several runs — which is observed,
|
||||
> not declared. Temperature is a label a human would have to maintain and keep
|
||||
> honest. If measured stability is sufficient, Temperature is a concept the
|
||||
> framework does not need.
|
||||
|
||||
---
|
||||
|
||||
## Energy
|
||||
|
|
@ -248,10 +256,13 @@ Initial verdicts are:
|
|||
```text
|
||||
PASS
|
||||
FAIL
|
||||
SUSPICIOUS
|
||||
INCONCLUSIVE
|
||||
```
|
||||
|
||||
`SUSPICIOUS` was listed here originally and **removed at T10**: no oracle could
|
||||
produce it, and a verdict nothing can emit is a promise the framework does not
|
||||
keep. It returns, with an identifier, if a mechanism ever needs it.
|
||||
|
||||
---
|
||||
|
||||
## Lineage
|
||||
|
|
|
|||
|
|
@ -25,6 +25,7 @@ No third-party dependencies. Python ≥ 3.11, pytest for the suite.
|
|||
```
|
||||
src/testdriver/ the kernel — intent, world, actions, drivers,
|
||||
observers, oracles, evidence, runner
|
||||
usecases/ durable test intent, including not-yet-runnable use cases
|
||||
lab/ the system under test
|
||||
scenarios/ reference scenarios
|
||||
research/ hypotheses, experiments, findings, fitness map
|
||||
|
|
|
|||
|
|
@ -21,5 +21,5 @@
|
|||
| task | TD-WP-0002-T06 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||
| task | TD-WP-0002-T07 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||
| task | TD-WP-0002-T08 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||
| task | TD-WP-0002-T09 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||
| task | TD-WP-0002-T09 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||
| task | TD-WP-0002-T10 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||
|
|
|
|||
212
history/2026-08-23-td-wp-0002-gate-review.md
Normal file
212
history/2026-08-23-td-wp-0002-gate-review.md
Normal file
|
|
@ -0,0 +1,212 @@
|
|||
# TD-WP-0002 — Gate Review and First Compression Pass
|
||||
|
||||
**Date:** 2026-08-23
|
||||
**Workplan:** `TD-WP-0002` — Vertical spike: falsify the crystallization thesis
|
||||
**Tests:** 172 passing · **Lab catalogue:** 24 labelled mutations
|
||||
**Assessment that set the direction:** `history/2026-08-22-concept-assessment-swot.md`
|
||||
|
||||
---
|
||||
|
||||
## 1. The gate
|
||||
|
||||
The workplan fixed four criteria in advance, before any code existed.
|
||||
|
||||
### Criterion 1 — recover from M01 and M02 with zero claim or invariant changes
|
||||
|
||||
**Met.**
|
||||
|
||||
| Mutation | Classification | Claim set unchanged | Accepted without a human |
|
||||
|---|---|---|---|
|
||||
| M01 sharing control moved into a modal | `MECHANICAL_ADAPTATION` | yes | yes |
|
||||
| M02 DOM rewritten, test ids dropped | `MECHANICAL_ADAPTATION` | yes | yes |
|
||||
|
||||
The claim set is unchanged *by construction*, not by good behaviour: claims are
|
||||
run inputs and the adaptation machinery has no write path to them (D-02).
|
||||
|
||||
### Criterion 2 — the authorization defect is reported as a Product Finding
|
||||
|
||||
**Met**, with a naming correction. The workplan inherited "M05" from the original
|
||||
milestones document; in the catalogue as built, the authorization defect is
|
||||
**M15** (revocation updates the record but not enforcement). All three
|
||||
authorization defects behave correctly:
|
||||
|
||||
| Mutation | Classification | Regressions reported |
|
||||
|---|---|---|
|
||||
| M15 revoke is cosmetic | `BEHAVIOUR_CHANGED` | `c-bob-revoked`, `i-enforcement-matches-record` |
|
||||
| M17 anyone can read anything | `BEHAVIOUR_CHANGED` | 5 assertions across 3 steps |
|
||||
| M20 tenant isolation leaks | `BEHAVIOUR_CHANGED` | 5 assertions across 3 steps |
|
||||
|
||||
None is safe to accept. The classifier collapses `PRODUCT_DEFECT` and
|
||||
`SEMANTIC_CHANGE` into one escalating outcome — see F-0006 — so "Product Finding"
|
||||
in the original wording is now "escalated to a human with the regressions named".
|
||||
|
||||
### Criterion 3 — one asset crystallizes into a deterministic test
|
||||
|
||||
**Met.** `crystallized/test_grant_access.py` is generated, runs with **no model
|
||||
involvement**, preserves its ancestor's oracle set as a superset, agrees with the
|
||||
ancestor across five lab versions, still catches a seeded defect, and carries its
|
||||
lineage in its own docstring.
|
||||
|
||||
### Criterion 4 — False Adaptation Rate = 0
|
||||
|
||||
**Met. 0 / 7.** No `DEFECT`-labelled mutation was classified as safe to accept,
|
||||
including the three E-003 attacks (surface substitution, concurrent
|
||||
mechanical+defect, evidence starvation) and provenance laundering.
|
||||
|
||||
The other side of the trade is asserted too: **12 of 13** mechanical mutations are
|
||||
absorbed without a human. A classifier that escalated everything would score a
|
||||
perfect FAR and be worthless.
|
||||
|
||||
### Gate verdict
|
||||
|
||||
**Passed, on all four criteria.** With one qualification that belongs in the same
|
||||
sentence: FAR = 0 follows largely from *architecture* — claims are run inputs,
|
||||
and `SAFE_TO_ACCEPT` is a closed two-element set — not from classifier accuracy.
|
||||
The experiment shows the architecture behaves as designed over 24 hand-written
|
||||
mutations. It says nothing about mutations nobody thought of.
|
||||
|
||||
---
|
||||
|
||||
## 2. What the evidence changed
|
||||
|
||||
Three of the project's own claims were narrowed or corrected by building them.
|
||||
This is the part worth more than the green gate.
|
||||
|
||||
**H-001 was narrowed (F-0005).** Semantic actions do not generally outlast
|
||||
recorded interaction sequences. Where an application keeps stable identifiers,
|
||||
the two arms tie exactly — 9/9 against 9/9 — and the conventional approach is
|
||||
cheaper and deterministic. Semantic actions earn their keep only where
|
||||
identifiers are absent or not carried forward (2/3 against 0/3). The concept
|
||||
model presents this as a general superiority; it is conditional.
|
||||
|
||||
**The classifier cannot infer intent (F-0006).** The T02 design listed
|
||||
`SEMANTIC_CHANGE` as an outcome the decision table could produce. It cannot: a
|
||||
deliberate product decision and a defect are behaviourally identical, as M12 and
|
||||
M19 have asserted since T05. The two collapse into `BEHAVIOUR_CHANGED` and the
|
||||
distinction becomes a recorded human adjudication. Erring toward false alarm
|
||||
costs one review; erring toward false adaptation ends the project.
|
||||
|
||||
**Crystallization's economic case is unmeasured (F-0007).** The descendant is 54%
|
||||
cheaper than its ancestor, and that number must not be quoted in support of the
|
||||
thesis. The runtime is token-free by design, so the entire saving is one page
|
||||
fetch, one parse and a two-candidate scoring pass. The saving the concept
|
||||
actually claims — model tokens, latency, retry variance — is absent from the
|
||||
measurement.
|
||||
|
||||
---
|
||||
|
||||
## 3. The compression pass
|
||||
|
||||
### Which concepts have no supporting evidence?
|
||||
|
||||
`Temperature`, `Confidence`, `Campaign`, `Metabolism`, `Retirement` — declared,
|
||||
unimplemented, never consulted by any decision.
|
||||
|
||||
`Temperature` is the decidable one and is now **F-0008**. Crystallization was
|
||||
built and demonstrated without it: what triggered freezing was *measured
|
||||
stability of realization*, which is observed rather than declared. A declared
|
||||
temperature is a claim nobody verifies, and the project's own first heuristic
|
||||
argues against it. Not deleted yet — two unevidenced cases for keeping it are
|
||||
recorded — but gated: if the next workplan completes without a decision
|
||||
consulting it, it goes.
|
||||
|
||||
### Which implementation abstractions have no conceptual justification?
|
||||
|
||||
Removed at T10:
|
||||
|
||||
| Removed | Why |
|
||||
|---|---|
|
||||
| `Verdict.SUSPICIOUS` | No oracle could produce it. A verdict nothing can emit is a promise not kept. |
|
||||
| `Step.expect_refusal` | Never set by any scenario. Speculative generality. |
|
||||
| `ActorIsolationError` | Declared, never raised, never caught. |
|
||||
| `World.seed` | Written at T04, never read. Replay comes from rebuilding the lab. |
|
||||
| `EvidencePack.latest()` | Never called. |
|
||||
| `Trajectory.method` | Computed on every capture, never read. |
|
||||
|
||||
### Which metadata has not informed a decision?
|
||||
|
||||
**`EnergyEvent` history.** Events have been recorded since T04 and have informed
|
||||
nothing — as designed, since H-005 is dormant. Capture is retained because
|
||||
history cannot be reconstructed later while scores always can, but this is now on
|
||||
the same gate as Temperature: if the next workplan ends with no decision having
|
||||
used it, the module goes.
|
||||
|
||||
### Which agentic behaviour can now be deterministic?
|
||||
|
||||
The `grant_access` realization, and it has been — that is T09. Nothing else in
|
||||
the spike is agentic.
|
||||
|
||||
### What have the experiments falsified?
|
||||
|
||||
Nothing outright. Two claims were **narrowed** (H-001, and the classifier's
|
||||
output space), and one criterion was met in a sense **too weak to support the
|
||||
thesis** (H-003's cost clause). No hypothesis was rejected, and none was promoted
|
||||
past `EXPERIMENTING`: 24 hand-written mutations is a small, self-chosen sample.
|
||||
|
||||
### Two outcomes that had never run
|
||||
|
||||
`INTENT_CHANGED` and `REALIZATION_FAILED` were declared at T08 and produced by no
|
||||
test. Code that has never run is code nobody has checked. Rather than delete
|
||||
meaningful outcomes or trust them untested, both now have purpose-built cases,
|
||||
and `test_no_classification_is_unreachable` fails if a seventh is ever added
|
||||
without one.
|
||||
|
||||
Building the `REALIZATION_FAILED` case clarified a boundary worth keeping: a run
|
||||
that asserts *nothing at all* is `AMBIGUOUS`, not `REALIZATION_FAILED` — a use
|
||||
case with no claims cannot conclude anything, however well its steps ran.
|
||||
|
||||
---
|
||||
|
||||
## 4. The fitness loop, closed
|
||||
|
||||
The milestone asks that a framework finding become a permanent guarantee rather
|
||||
than a note. **F-0003 is resolved.**
|
||||
|
||||
The finding: actor isolation left no trace in ordinary evidence. The self-test
|
||||
caught a shared memory store only because the harness planted per-actor secrets
|
||||
first. Isolation was a property of *scenarios written to expose it*, not of runs.
|
||||
|
||||
The fix: every `Actor` is seeded at construction with an automatic private
|
||||
marker, and the runner examines all actors on **every** scenario, recording the
|
||||
verdict as S3 evidence. Two regressions now stand behind it —
|
||||
`test_every_run_records_a_verdict_on_isolation`, and a self-check that catches a
|
||||
leak with the harness planting nothing at all.
|
||||
|
||||
```
|
||||
Framework Finding (F-0003)
|
||||
→ classified FRAMEWORK_LIMITATION
|
||||
→ improvement hypothesis: automatic canaries make isolation observable
|
||||
→ implemented at T10
|
||||
→ permanent self-verification + deterministic regression
|
||||
→ fitness map updated
|
||||
```
|
||||
|
||||
That is the first complete Concept–Implementation Fitness Loop, which the
|
||||
milestones document calls the point at which the self-improvement system
|
||||
genuinely exists.
|
||||
|
||||
---
|
||||
|
||||
## 5. Where this leaves the project
|
||||
|
||||
**The thesis survived contact with its own evidence, in narrowed form.** The
|
||||
distinctive claim — that verification can adapt to a changing surface without
|
||||
adapting to a defect — is demonstrated, and the demonstration is architectural
|
||||
rather than statistical, which is the stronger kind.
|
||||
|
||||
**The single highest-value next step is a bounded live-model experiment.** Two
|
||||
independent findings converge on it: M22 defeats the heuristic runtime while
|
||||
remaining solvable by reading a visible label (F-0005), and crystallization's
|
||||
economic case cannot be measured without token costs (F-0007). One experiment
|
||||
settles whether a model adds capability *and* whether crystallization has a
|
||||
product rationale. The `RealizationMetrics` fields were populated from the first
|
||||
run so that comparison is a subtraction rather than a re-run.
|
||||
|
||||
That should come before any further framework feature.
|
||||
|
||||
**Open findings carried forward:** F-0003 resolved · F-0004 (structural
|
||||
durability only; visual relayout untested) · F-0005 (H-001 narrowed) · F-0007
|
||||
(cost unmeasurable; generated tests not fully standalone) · F-0008 (Temperature
|
||||
gated). The F-0001 residual — the taxonomy has no class for document-to-document
|
||||
drift — was considered and deliberately not resolved: adding a finding class to a
|
||||
taxonomy already larger than its evidence would be the opposite of compression.
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
# Concept ↔ Implementation Fitness Map
|
||||
|
||||
**Updated:** 2026-08-23 (TD-WP-0002-T09)
|
||||
**Updated:** 2026-08-23 (TD-WP-0002-T10)
|
||||
|
||||
Traces each important concept to the implementation, experiment and evidence that
|
||||
support it. **Unsupported entries are the point of this map** — a concept with no
|
||||
|
|
@ -29,7 +29,7 @@ were aspirational, not evidenced.
|
|||
| Concept | Level | Implementation | Experiment | Evidence | Open question |
|
||||
|---|---|---|---|---|---|
|
||||
| `C-use-case` | C1 | `intent.py` | — | — | Is a use case expressible without leaking mechanics? |
|
||||
| `C-actor-isolation` | C1 | `world.py` | E-001 | `td://self/actor-isolation` | **F-0003** — only observable when the scenario plants canaries. |
|
||||
| `C-actor-isolation` | **C2** | `world.py`, `runner.py` | E-001 | `td://self/actor-isolation`, examined every run | **F-0003 resolved** — automatic canaries; observable on every scenario. |
|
||||
| `C-semantic-action` | **C2** | `actions.py`, `agentic.py` | E-001 (partial) | T07 arm comparison | **F-0005** — supported only where stable identifiers are absent. Narrower than the concept model claims. |
|
||||
| `C-oracle-independence` | **C2** | `runner.py`, `oracles.py` | E-001, E-003 | — | Independence of components ≠ independence of belief. (H-004) |
|
||||
| `C-evidence-pack` | C1 | `evidence.py` | — | `td://self/evidence-reproducibility` | Verdicts are reproducible from S3 alone, on passing and failing runs. |
|
||||
|
|
@ -39,9 +39,9 @@ were aspirational, not evidenced.
|
|||
| `C-crystallization` | **C2** | `crystallization.py` | E-002 | T09 agreement matrix | Fidelity supported; **F-0007** — the economic case is unmeasurable with a token-free runtime. |
|
||||
| `C-intent-provenance` | C1 | `provenance.py` | E-003 | `td://self/intent-independence` | Constrains provenance, not quality. Accepted residual. |
|
||||
| `C-lineage` | C1 | `scenario.py`, generated headers | E-002 | generated module docstring | Parent pointer plus provenance in the artefact; no graph. |
|
||||
| `C-energy` | C0 | `energy.py`, capture only | — | — | Dormant by decision. (H-005) |
|
||||
| `C-temperature` | C0 | — | — | — | Deferred. No implementation planned in TD-WP-0002. |
|
||||
| `C-confidence` | C0 | — | — | — | Deferred. |
|
||||
| `C-energy` | C0 | `energy.py`, capture only | — | none | Dormant. **Gated**: if the next workplan ends with no decision having used the history, remove. |
|
||||
| `C-temperature` | **C0 — under review** | — | — | none | **F-0008** — crystallization was built without it; measured stability did the work. Gated for removal. |
|
||||
| `C-confidence` | C0 | — | — | none | Declared, unimplemented, never consulted. Same position as `C-temperature` without a built alternative. |
|
||||
| `C-campaign` | C0 | — | — | — | Deferred. |
|
||||
| `C-metabolism` | C0 | — | — | — | Deferred. Depends on C-energy. |
|
||||
| `C-retirement` | C0 | — | — | — | Deferred. Depends on C-energy. |
|
||||
|
|
@ -58,6 +58,13 @@ validated. They are revisited at T10, where the question is not "when do we buil
|
|||
these" but "does the evidence justify keeping them in the model at all".
|
||||
|
||||
**Implementation orphans** — none. Every module in `src/testdriver/` traces to a
|
||||
concept above. `energy.py` is the one to watch: it exists solely to capture
|
||||
events for a dormant hypothesis, and if T10 finds no use for the history it
|
||||
should be removed rather than kept out of sentiment.
|
||||
concept above.
|
||||
|
||||
**Removed at T10** (compression pass — see the gate review § 3):
|
||||
`Verdict.SUSPICIOUS`, `Step.expect_refusal`, `ActorIsolationError`,
|
||||
`World.seed`, `EvidencePack.latest()`, `Trajectory.method`. Each was declared and
|
||||
never used; `SUSPICIOUS` was additionally a verdict no oracle could emit.
|
||||
|
||||
**Gated for removal**: `C-temperature` (F-0008) and `energy.py`. Both survive the
|
||||
current review on the strength of being cheap, not of being used. If the next
|
||||
workplan closes without a decision consulting either, they go.
|
||||
|
|
|
|||
|
|
@ -2,12 +2,13 @@
|
|||
id: F-0003
|
||||
type: framework-finding
|
||||
class: FRAMEWORK_LIMITATION
|
||||
status: open
|
||||
status: resolved
|
||||
discovered: "2026-08-22"
|
||||
discovered_by: TD-WP-0002-T06
|
||||
workplan: TD-WP-0002
|
||||
task: TD-WP-0002-T06
|
||||
carried_to: TD-WP-0002-T10
|
||||
resolved: "2026-08-23"
|
||||
resolved_in: TD-WP-0002-T10
|
||||
---
|
||||
|
||||
# F-0003 — Actor isolation is only observable if the scenario plants canaries
|
||||
|
|
@ -55,9 +56,31 @@ system.
|
|||
Worth stating plainly rather than counting four green self-tests as four
|
||||
equivalent proofs. They are not equivalent.
|
||||
|
||||
## Candidate resolutions
|
||||
## Resolution (T10)
|
||||
|
||||
Not resolved now; deciding cheaply here would be guessing.
|
||||
Candidate 3 — **both**.
|
||||
|
||||
Every `Actor` is seeded at construction with an automatic private marker
|
||||
(`Actor.canary`), and `Runner._isolation_violations` examines every actor on
|
||||
**every** scenario, recording the verdict as S3 evidence. An isolation violation
|
||||
now leaves a trace whether or not anyone thought to look for one.
|
||||
|
||||
Two permanent regressions stand behind it:
|
||||
|
||||
- `tests/test_reference_scenario.py::test_every_run_records_a_verdict_on_isolation`
|
||||
- `tests/selfverification/test_checks_can_fail.py::test_a_leak_is_caught_without_the_test_planting_anything`
|
||||
— an actor holding another's marker is caught by the ordinary run, with the
|
||||
harness planting nothing.
|
||||
|
||||
The second limitation stands as recorded: oracle independence is enforced by
|
||||
construction, so it is verified once rather than per run. That asymmetry is real
|
||||
and is now stated in the fitness map rather than hidden behind two equally green
|
||||
self-tests.
|
||||
|
||||
This finding closed the first Concept–Implementation Fitness Loop — see
|
||||
`history/2026-08-23-td-wp-0002-gate-review.md` § 4.
|
||||
|
||||
## Candidate resolutions considered
|
||||
|
||||
1. **Automatic canaries** — the runtime seeds each actor with a unique private
|
||||
marker at construction and the isolation check runs on every Evidence Pack,
|
||||
|
|
|
|||
70
research/findings/F-0008-temperature-may-be-redundant.md
Normal file
70
research/findings/F-0008-temperature-may-be-redundant.md
Normal file
|
|
@ -0,0 +1,70 @@
|
|||
---
|
||||
id: F-0008
|
||||
type: framework-finding
|
||||
class: UNNECESSARY_COMPLEXITY
|
||||
status: open
|
||||
discovered: "2026-08-23"
|
||||
discovered_by: TD-WP-0002-T10
|
||||
workplan: TD-WP-0002
|
||||
task: TD-WP-0002-T10
|
||||
carried_to: next workplan
|
||||
---
|
||||
|
||||
# F-0008 — Temperature may be redundant; measured stability did the work
|
||||
|
||||
## Observation
|
||||
|
||||
`Temperature` (`HOT` / `WARM` / `COOL` / `COLD`) is one of the more prominent
|
||||
ideas in `INTENT.md` and the Concept Model. It is supposed to govern which
|
||||
verification mode an asset uses, and it is the mechanism by which "tests
|
||||
crystallize as software cools".
|
||||
|
||||
The spike built crystallization end to end. **Temperature was never consulted,
|
||||
never implemented, and never missed.**
|
||||
|
||||
What actually triggered freezing was `assess_stability`: the same realization
|
||||
path observed across several consecutive runs. Nothing declared the surface cold;
|
||||
the surface was observed to have stopped moving.
|
||||
|
||||
## The argument for removal
|
||||
|
||||
The two mechanisms answer the same question, and only one of them can be wrong:
|
||||
|
||||
| | Temperature | Measured stability |
|
||||
|---|---|---|
|
||||
| source | declared by a human | observed from runs |
|
||||
| upkeep | must be maintained and kept honest | none |
|
||||
| failure mode | says `COLD` while the surface churns | none — it is the churn |
|
||||
| already built | no | yes |
|
||||
|
||||
A declared temperature is a claim about the system that nobody verifies. The
|
||||
project's own first heuristic — *do not let a declaration substitute for
|
||||
evidence* — argues against keeping it.
|
||||
|
||||
## The argument for keeping it
|
||||
|
||||
Two cases measured stability cannot cover:
|
||||
|
||||
1. **Forward-looking scope.** A team knows a rewrite lands next month.
|
||||
Temperature can say `HOT` before any run has observed instability, where
|
||||
measured stability would happily freeze an asset the day before it breaks.
|
||||
2. **Cheap prioritisation.** Deciding *where to spend effort* across many
|
||||
capabilities may want a coarse label without running anything.
|
||||
|
||||
Neither case arose in the spike, and neither is evidenced.
|
||||
|
||||
## Recommendation
|
||||
|
||||
Do not delete yet; do not build either. Carry `Temperature` as **explicitly
|
||||
unvalidated** in the fitness map, and set a gate: if the next workplan completes
|
||||
without any decision consulting it, remove it from the concept model.
|
||||
|
||||
Recorded now because the compression review's purpose is to catch concepts that
|
||||
survive on the strength of being attractive rather than useful, and Temperature
|
||||
is currently the clearest instance in the corpus.
|
||||
|
||||
## Related
|
||||
|
||||
`Confidence`, `Campaign`, `Metabolism` and `Retirement` are in the same position —
|
||||
declared, unimplemented, never consulted — but Temperature is the one with a
|
||||
built alternative already doing its job, which makes it the decidable case.
|
||||
Binary file not shown.
|
|
@ -128,9 +128,7 @@ def build(*mutations: str):
|
|||
cast.add(Actor(id=name, display_name=name.title(),
|
||||
credentials={"token": tokens[name]}))
|
||||
|
||||
world = World(id="w-baseline", sut=lab, sut_version=lab.version,
|
||||
seed={"users": ["alice", "bob", "carol"], "resource": RESOURCE},
|
||||
cast=cast)
|
||||
world = World(id="w-baseline", sut=lab, sut_version=lab.version, cast=cast)
|
||||
|
||||
scenario = Scenario(
|
||||
id="sc-share-resource",
|
||||
|
|
|
|||
|
|
@ -10,10 +10,10 @@ from .oracles import Judgment, Oracle, Verdict, overall
|
|||
from .provenance import InadmissibleProvenance, Provenance
|
||||
from .runner import CollectorIndependenceError, Runner, RunResult
|
||||
from .scenario import Scenario, Step, VerificationAsset
|
||||
from .world import Actor, ActorIsolationError, Cast, World
|
||||
from .world import Actor, Cast, World
|
||||
|
||||
__all__ = [
|
||||
"Actor", "ActorIsolationError", "Cast", "Claim", "CollectorIndependenceError",
|
||||
"Actor", "Cast", "Claim", "CollectorIndependenceError",
|
||||
"DirectDriver", "EnergyEvent", "EnergyEventType", "EvidencePack",
|
||||
"InadmissibleProvenance", "Invariant", "Judgment", "Observation", "Oracle",
|
||||
"Provenance", "Realization", "RunResult", "Runner", "Scenario",
|
||||
|
|
|
|||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
|
@ -35,7 +35,6 @@ class Trajectory:
|
|||
step_id: str
|
||||
action_name: str
|
||||
surface_id: str
|
||||
method: str
|
||||
target: str
|
||||
fields: tuple[str, ...]
|
||||
|
||||
|
|
@ -43,7 +42,7 @@ class Trajectory:
|
|||
return json.dumps(
|
||||
{
|
||||
"step": self.step_id, "action": self.action_name,
|
||||
"surface": self.surface_id, "method": self.method,
|
||||
"surface": self.surface_id,
|
||||
"target": self.target, "fields": sorted(self.fields),
|
||||
},
|
||||
sort_keys=True,
|
||||
|
|
@ -63,7 +62,6 @@ def capture(pack: Mapping[str, Any]) -> tuple[Trajectory, ...]:
|
|||
step_id=obs["step_id"],
|
||||
action_name=str(mechanics.get("action", "")).split("(")[0],
|
||||
surface_id=obs["data"].get("surface", ""),
|
||||
method=("POST" if obs["data"].get("surface") == "browser" else "CALL"),
|
||||
target=target,
|
||||
fields=tuple(sorted(fields)),
|
||||
))
|
||||
|
|
|
|||
|
|
@ -72,12 +72,6 @@ class EvidencePack:
|
|||
def of_stratum(self, stratum: Stratum) -> list[Observation]:
|
||||
return [o for o in self.observations if o.stratum is stratum]
|
||||
|
||||
def latest(self, kind: str) -> Observation | None:
|
||||
for observation in reversed(self.observations):
|
||||
if observation.kind == kind:
|
||||
return observation
|
||||
return None
|
||||
|
||||
def to_json(self) -> str:
|
||||
payload = asdict(self)
|
||||
payload["observations"] = [
|
||||
|
|
|
|||
|
|
@ -3,6 +3,10 @@
|
|||
An oracle reads only the independent observation snapshot. It has no access to
|
||||
the actor, to the driver, or to what either of them believes happened.
|
||||
|
||||
`SUSPICIOUS` was removed at T10: no oracle could produce it, and a verdict
|
||||
nothing can emit is a promise the framework does not keep. It returns, with an
|
||||
identifier, if a mechanism ever needs it.
|
||||
|
||||
`INCONCLUSIVE` is a first-class outcome, not a failure mode of the framework. An
|
||||
oracle that cannot support a judgment from the retained evidence must say so
|
||||
rather than defaulting either way — silently defaulting to PASS hides defects,
|
||||
|
|
@ -21,7 +25,6 @@ from .intent import Claim, Invariant
|
|||
class Verdict(str, Enum):
|
||||
PASS = "PASS"
|
||||
FAIL = "FAIL"
|
||||
SUSPICIOUS = "SUSPICIOUS"
|
||||
INCONCLUSIVE = "INCONCLUSIVE"
|
||||
|
||||
|
||||
|
|
@ -101,6 +104,4 @@ def overall(judgments: list[Judgment]) -> Verdict:
|
|||
return Verdict.FAIL
|
||||
if Verdict.INCONCLUSIVE in verdicts:
|
||||
return Verdict.INCONCLUSIVE
|
||||
if Verdict.SUSPICIOUS in verdicts:
|
||||
return Verdict.SUSPICIOUS
|
||||
return Verdict.PASS
|
||||
|
|
|
|||
|
|
@ -56,6 +56,24 @@ class Runner:
|
|||
|
||||
# -- independence guards ---------------------------------------------
|
||||
|
||||
def _isolation_violations(self) -> list[str]:
|
||||
"""Does any actor hold another's canary?
|
||||
|
||||
Run on every scenario, not only on ones written to test isolation.
|
||||
"""
|
||||
canaries = {actor.canary: actor.id for actor in self._world.cast}
|
||||
violations: list[str] = []
|
||||
for actor in self._world.cast:
|
||||
for key in actor.known_keys():
|
||||
value = actor.recall(key)
|
||||
owner = canaries.get(value) if isinstance(value, str) else None
|
||||
if owner is not None and owner != actor.id:
|
||||
violations.append(
|
||||
f"actor {actor.id!r} holds the private marker of {owner!r} "
|
||||
f"under key {key!r}"
|
||||
)
|
||||
return violations
|
||||
|
||||
def _assert_collector_independence(self, stratum: Stratum, collector: str) -> None:
|
||||
"""S2 and S3 evidence may never be attributed to an actor.
|
||||
|
||||
|
|
@ -111,6 +129,12 @@ class Runner:
|
|||
EnergyEvent(asset.id, run_id, EnergyEventType.EXECUTED).as_dict()
|
||||
)
|
||||
|
||||
isolation = self._isolation_violations()
|
||||
self._record(
|
||||
pack, Stratum.JUDGMENT, self._observer.name, "actor_isolation",
|
||||
{"violations": isolation, "actors": sorted(self._world.cast.actors)}, None,
|
||||
)
|
||||
|
||||
judgments: list[Judgment] = []
|
||||
scenario_sound = True
|
||||
claims_by_step: dict[str, list] = {}
|
||||
|
|
@ -166,7 +190,6 @@ class Runner:
|
|||
"action": step.action.name,
|
||||
"surface_used": realization.surface_id,
|
||||
"refused_by_sut": refused,
|
||||
"refusal_expected": step.expect_refusal,
|
||||
"postcondition_met": postcondition_met,
|
||||
},
|
||||
step.id,
|
||||
|
|
@ -176,7 +199,7 @@ class Runner:
|
|||
# An action that was accepted but did not take effect did happen —
|
||||
# and that is a statement about the system, judged below, not a
|
||||
# reason to stop judging.
|
||||
if refused and not step.expect_refusal:
|
||||
if refused:
|
||||
scenario_sound = False
|
||||
|
||||
# --- invariants after every step -----------------------------
|
||||
|
|
|
|||
|
|
@ -17,7 +17,6 @@ class Step:
|
|||
id: str
|
||||
actor_id: str
|
||||
action: SemanticAction
|
||||
expect_refusal: bool = False
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
|
|
|
|||
|
|
@ -8,12 +8,20 @@ to be executed by the same process.
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Iterator
|
||||
|
||||
|
||||
class ActorIsolationError(Exception):
|
||||
"""Raised when one actor is offered another actor's private state."""
|
||||
def _canary() -> str:
|
||||
"""A value private to one actor, unguessable and unique per construction.
|
||||
|
||||
Exists so that isolation is *observable*. Without it a run in which every
|
||||
actor shares one memory store produces evidence indistinguishable from a
|
||||
correct one — the guarantee holds only in scenarios written to expose it,
|
||||
which is no guarantee at all (F-0003).
|
||||
"""
|
||||
return f"canary-{uuid.uuid4().hex}"
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
|
|
@ -28,8 +36,13 @@ class Actor:
|
|||
id: str
|
||||
display_name: str
|
||||
credentials: dict[str, str] = field(default_factory=dict)
|
||||
session: dict[str, Any] = field(default_factory=dict)
|
||||
_memory: dict[str, Any] = field(default_factory=dict, repr=False)
|
||||
canary: str = field(default_factory=_canary)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
# Seeded automatically, on every actor, in every scenario. An isolation
|
||||
# violation now leaves a trace whether or not anyone thought to look.
|
||||
self._memory.setdefault("__canary__", self.canary)
|
||||
|
||||
def remember(self, key: str, value: Any) -> None:
|
||||
self._memory[key] = value
|
||||
|
|
@ -65,15 +78,14 @@ class Cast:
|
|||
|
||||
@dataclass(slots=True)
|
||||
class World:
|
||||
"""Initial state plus the handle to the system under test.
|
||||
"""The handle to the system under test, plus the cast acting on it.
|
||||
|
||||
`seed` is everything needed to rebuild the initial state, so that a run can
|
||||
be replayed from a known starting point rather than from wherever the
|
||||
previous run happened to leave things.
|
||||
Replay comes from rebuilding the lab through `build_lab`, not from a seed
|
||||
dict carried here — the dict was written at T04, never read, and removed at
|
||||
T10.
|
||||
"""
|
||||
|
||||
id: str
|
||||
sut: Any
|
||||
sut_version: str
|
||||
seed: dict[str, Any] = field(default_factory=dict)
|
||||
cast: Cast = field(default_factory=Cast)
|
||||
|
|
|
|||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
|
@ -94,6 +94,23 @@ def check_actor_isolation(
|
|||
return violations
|
||||
|
||||
|
||||
def check_isolation_was_examined(pack: Mapping[str, Any]) -> list[str]:
|
||||
"""Every run must carry a verdict on actor isolation — F-0003, resolved.
|
||||
|
||||
Before this existed, isolation was only observable in scenarios written to
|
||||
expose it: a run in which every actor shared one memory store produced
|
||||
evidence indistinguishable from a correct one. Actors now carry an automatic
|
||||
private marker and the runner examines them on every scenario, so the absence
|
||||
of this observation is itself a failure.
|
||||
"""
|
||||
for obs in _observations(pack):
|
||||
if obs["kind"] != "actor_isolation":
|
||||
continue
|
||||
violations = obs["data"].get("violations") or []
|
||||
return [f"actor isolation violated: {v}" for v in violations]
|
||||
return ["this run did not examine actor isolation at all"]
|
||||
|
||||
|
||||
# --- td://self/oracle-independence ---------------------------------------
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -30,6 +30,7 @@ from tests.selfverification.checks import (
|
|||
check_actor_isolation,
|
||||
check_evidence_supports_every_verdict,
|
||||
check_intent_independence,
|
||||
check_isolation_was_examined,
|
||||
check_no_actor_collected_judgment,
|
||||
check_runs_agree,
|
||||
check_verdicts_follow_from_judgment_evidence,
|
||||
|
|
@ -186,3 +187,35 @@ def test_unrecorded_provenance_is_caught():
|
|||
tampered = copy.deepcopy(pack)
|
||||
tampered["provenance_index"] = {}
|
||||
assert check_intent_independence(tampered)
|
||||
|
||||
|
||||
# --- F-0003 resolution: isolation observed on every run -------------------
|
||||
|
||||
|
||||
def test_an_unexamined_run_is_caught():
|
||||
"""A run that never looked at isolation must not read as isolated."""
|
||||
pack, _, _ = run_and_serialize()
|
||||
tampered = copy.deepcopy(pack)
|
||||
tampered["observations"] = [
|
||||
o for o in tampered["observations"] if o["kind"] != "actor_isolation"
|
||||
]
|
||||
assert check_isolation_was_examined(tampered)
|
||||
|
||||
|
||||
def test_a_leak_is_caught_without_the_test_planting_anything():
|
||||
"""The regression F-0003 leaves behind.
|
||||
|
||||
No secrets seeded by the harness, no scenario written to expose isolation.
|
||||
An actor holding another's automatic marker is caught by the ordinary run.
|
||||
"""
|
||||
from testdriver import Oracle, Runner
|
||||
from scenarios.alice_bob_carol import build
|
||||
|
||||
world, driver, observer, asset, oracle = build()
|
||||
world.cast["bob"].remember("overheard", world.cast["alice"].canary)
|
||||
result = Runner(world, driver, observer, oracle).run(asset)
|
||||
pack = json.loads(result.evidence.to_json())
|
||||
|
||||
violations = check_isolation_was_examined(pack)
|
||||
assert violations
|
||||
assert "holds the private marker of 'alice'" in violations[0]
|
||||
|
|
|
|||
|
|
@ -17,6 +17,7 @@ from tests.selfverification.checks import (
|
|||
check_actor_isolation,
|
||||
check_evidence_supports_every_verdict,
|
||||
check_intent_independence,
|
||||
check_isolation_was_examined,
|
||||
check_no_actor_collected_judgment,
|
||||
check_runs_agree,
|
||||
check_verdicts_follow_from_judgment_evidence,
|
||||
|
|
@ -56,6 +57,14 @@ def test_actor_isolation():
|
|||
assert check_actor_isolation(pack, SECRETS, memories) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mutations", [(), ("M15",), ("M01",)])
|
||||
def test_every_run_examines_isolation(mutations):
|
||||
"""F-0003 resolved: isolation is now a property of runs, not of scenarios
|
||||
written to expose it. No canaries need be planted by the test."""
|
||||
pack, _, _ = run_and_serialize(*mutations)
|
||||
assert check_isolation_was_examined(pack) == []
|
||||
|
||||
|
||||
# --- td://self/oracle-independence ---------------------------------------
|
||||
|
||||
|
||||
|
|
|
|||
136
tests/test_audit_core_e2_use_case.py
Normal file
136
tests/test_audit_core_e2_use_case.py
Normal file
|
|
@ -0,0 +1,136 @@
|
|||
"""The audit-core use case is durable intent, even before drivers can run it."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from copy import deepcopy
|
||||
|
||||
from testdriver import Oracle, Provenance, Verdict
|
||||
from usecases.audit_core_e2_tenant_boundary import (
|
||||
PHASE_CONTRACTS,
|
||||
PRECEDENT_EVIDENCE_REF,
|
||||
ROLE_CONTRACTS,
|
||||
TEST_USE_CASE,
|
||||
)
|
||||
|
||||
|
||||
def passing_observations() -> dict[str, object]:
|
||||
absent = {
|
||||
"status": 404,
|
||||
"schema": ("$", "$.error:str"),
|
||||
"digest": "absent-surface",
|
||||
"fixture_match_count": 0,
|
||||
}
|
||||
return {
|
||||
"event_by_id": {
|
||||
"owner": {"status": 200, "fixture_match_count": 2},
|
||||
"attacker": dict(absent),
|
||||
"absent": dict(absent),
|
||||
},
|
||||
"correlation_slice": {
|
||||
"owner": {"status": 200, "fixture_match_count": 2},
|
||||
"attacker": {"status": 200, "fixture_match_count": 0},
|
||||
},
|
||||
"append_as_b": {
|
||||
"attacker": {"status": 400},
|
||||
"state_after": dict(absent),
|
||||
"absent": dict(absent),
|
||||
},
|
||||
"run": {
|
||||
"attempted_operations": 10,
|
||||
"maximum_operations": 30,
|
||||
"max_concurrency": 1,
|
||||
"rate_per_minute": 10,
|
||||
"target_revision_matches": True,
|
||||
"attribution_complete": True,
|
||||
},
|
||||
"cleanup": {
|
||||
"completed_before_expiry": True,
|
||||
"runner_absent": True,
|
||||
"temporary_identities": [],
|
||||
"kv_paths": [],
|
||||
"projection_resources": [],
|
||||
"mounted_secrets": [],
|
||||
"target_ready": True,
|
||||
"secret_values_observed": False,
|
||||
},
|
||||
"report": {
|
||||
"bound_to_projection_receipt": True,
|
||||
"bound_to_cleanup_receipt": True,
|
||||
"bounded_assurance_statement": True,
|
||||
"delivered_to_risk_nexus": True,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def judgments(observations):
|
||||
oracle = Oracle()
|
||||
assertions = (*TEST_USE_CASE.claims, *TEST_USE_CASE.invariants)
|
||||
return {
|
||||
assertion.id: oracle.judge(
|
||||
assertion,
|
||||
observations,
|
||||
getattr(assertion, "after_step", None),
|
||||
)
|
||||
for assertion in assertions
|
||||
}
|
||||
|
||||
|
||||
def test_precedent_calibration_satisfies_every_assertion():
|
||||
result = judgments(passing_observations())
|
||||
assert {judgment.verdict for judgment in result.values()} == {Verdict.PASS}
|
||||
assert PRECEDENT_EVIDENCE_REF.endswith(
|
||||
"AUDIT-WP-0008-T05-whitehat-e2-03-pass-2026-08-22.md"
|
||||
)
|
||||
|
||||
|
||||
def test_cross_tenant_event_disclosure_fails_the_claim():
|
||||
observations = passing_observations()
|
||||
observations["event_by_id"]["attacker"] = {
|
||||
"status": 200,
|
||||
"schema": ("$", "$.event_id:str"),
|
||||
"digest": "tenant-b-event",
|
||||
"fixture_match_count": 2,
|
||||
}
|
||||
result = judgments(observations)
|
||||
assert result["c-event-id-tenant-scoped"].verdict is Verdict.FAIL
|
||||
|
||||
|
||||
def test_missing_cleanup_evidence_is_inconclusive():
|
||||
observations = passing_observations()
|
||||
del observations["cleanup"]
|
||||
result = judgments(observations)
|
||||
assert result["c-receipt-bound-cleanup"].verdict is Verdict.INCONCLUSIVE
|
||||
assert result["i-no-secret-evidence"].verdict is Verdict.INCONCLUSIVE
|
||||
|
||||
|
||||
def test_cleanup_residue_fails_closed():
|
||||
observations = deepcopy(passing_observations())
|
||||
observations["cleanup"]["temporary_identities"] = ["unexpected-residue"]
|
||||
result = judgments(observations)
|
||||
assert result["c-receipt-bound-cleanup"].verdict is Verdict.FAIL
|
||||
|
||||
|
||||
def test_intent_has_independent_provenance_and_separate_roles():
|
||||
assert TEST_USE_CASE.provenance is Provenance.SPEC
|
||||
assert all(
|
||||
assertion.provenance is Provenance.SPEC
|
||||
for assertion in (*TEST_USE_CASE.claims, *TEST_USE_CASE.invariants)
|
||||
)
|
||||
roles = {role.id for role in ROLE_CONTRACTS}
|
||||
assert {
|
||||
"authorizer",
|
||||
"target-owner",
|
||||
"credential-custodian",
|
||||
"security-coordinator",
|
||||
"cluster-executor",
|
||||
"tenant-a-attacker",
|
||||
"tenant-b-control",
|
||||
"independent-observer",
|
||||
} == roles
|
||||
|
||||
|
||||
def test_schedule_requires_cleanup_before_finalization():
|
||||
phases = {phase.id: phase for phase in PHASE_CONTRACTS}
|
||||
assert phases["finalize-and-deliver"].requires == ("cleanup-custody",)
|
||||
assert phases["cleanup-custody"].requires == ("delete-runner",)
|
||||
assert phases["run-probes"].requires == ("ready-runner",)
|
||||
|
|
@ -194,3 +194,98 @@ def test_safe_to_accept_is_a_closed_set():
|
|||
assert SAFE_TO_ACCEPT == {
|
||||
Classification.UNCHANGED, Classification.MECHANICAL_ADAPTATION,
|
||||
}
|
||||
|
||||
|
||||
# --- classifications that no lab mutation happens to produce ---------------
|
||||
#
|
||||
# Two outcomes were declared at T08 and exercised by nothing. Left that way they
|
||||
# are decoration: code that has never run is code nobody has checked. Rather than
|
||||
# delete meaningful outcomes or trust them untested, both are given a case.
|
||||
|
||||
|
||||
def test_intent_change_is_detected_when_the_claim_set_moves(baseline):
|
||||
"""`INTENT_CHANGED` is a fact about the recorded use case, not an inference.
|
||||
|
||||
It fires because a human edited what is being asserted — which is why it is
|
||||
detectable at all, where `SEMANTIC_CHANGE` was not (F-0006).
|
||||
"""
|
||||
import copy
|
||||
|
||||
altered = copy.deepcopy(baseline)
|
||||
altered["provenance_index"]["c-newly-added-claim"] = "human"
|
||||
outcome = classify(baseline, altered)
|
||||
assert outcome.classification is Classification.INTENT_CHANGED
|
||||
assert not outcome.safe_to_accept
|
||||
|
||||
|
||||
def test_realization_failure_is_distinguishable_from_ambiguity():
|
||||
"""`REALIZATION_FAILED` says "we could not act"; `AMBIGUOUS` says "we do not know".
|
||||
|
||||
Every lab mutation that breaks realization also strands a claim, so the
|
||||
catalogue only ever produces `AMBIGUOUS`. This builds the case the catalogue
|
||||
cannot: a step that fails while every assertion in the run still holds and
|
||||
none of them depended on it.
|
||||
|
||||
Note that a run asserting *nothing at all* is `AMBIGUOUS`, not
|
||||
`REALIZATION_FAILED` — a use case with no claims cannot conclude anything,
|
||||
however well its steps ran.
|
||||
"""
|
||||
from testdriver import (
|
||||
Actor, Cast, Invariant, Oracle, Runner, Scenario, SemanticAction,
|
||||
StateObserver, Step, UseCase, VerificationAsset, World,
|
||||
)
|
||||
from testdriver.agentic import DiscoveryRuntime
|
||||
from testdriver.browser import BrowserDriver
|
||||
from testdriver.observers import Watch
|
||||
from testdriver.provenance import Provenance
|
||||
from lab.mutations import ObservationChannel
|
||||
|
||||
use_case = UseCase(
|
||||
"uc-audit-only", "Sharing leaves an ordered audit trail",
|
||||
"Alice shares R with Bob; the audit trail stays ordered.",
|
||||
Provenance.HUMAN,
|
||||
invariants=(Invariant(
|
||||
"i-audit-ordered", "The audit trail is append-only", Provenance.HUMAN,
|
||||
lambda obs: [e["sequence"] for e in obs["audit:R"]]
|
||||
== sorted(e["sequence"] for e in obs["audit:R"]),
|
||||
),),
|
||||
)
|
||||
|
||||
def run(*mutations):
|
||||
with journey_lab_server(*mutations) as (app, tokens, base_url):
|
||||
app.request(tokens["alice"], "create_resource",
|
||||
resource_id="R", content="x")
|
||||
cast = Cast()
|
||||
cast.add(Actor("alice", "Alice", credentials={"token": tokens["alice"]}))
|
||||
scenario = Scenario(
|
||||
"sc-audit-only", use_case,
|
||||
watches=(Watch("bob", "R"),),
|
||||
steps=(Step("s1", "alice", SemanticAction(
|
||||
"grant_access", {"subject_id": "bob", "permission": "READ"},
|
||||
permitted_surfaces=frozenset({"browser"}),
|
||||
)),),
|
||||
)
|
||||
driver = BrowserDriver(base_url, tokens, DiscoveryRuntime(), "R")
|
||||
observer = StateObserver(ObservationChannel(app), scenario.watches)
|
||||
world = World("w-audit", app, app.version, cast=cast)
|
||||
return json.loads(
|
||||
Runner(world, driver, observer, Oracle())
|
||||
.run(VerificationAsset("va-audit-only", scenario))
|
||||
.evidence.to_json()
|
||||
)
|
||||
|
||||
outcome = classify(run(), run("M23")) # the control is gone from the UI
|
||||
assert outcome.classification is Classification.REALIZATION_FAILED
|
||||
assert not outcome.safe_to_accept
|
||||
|
||||
|
||||
def test_no_classification_is_unreachable():
|
||||
"""Every declared outcome must be produced somewhere in this suite.
|
||||
|
||||
An outcome nothing can emit is the same kind of dead promise `SUSPICIOUS`
|
||||
was before T10 removed it.
|
||||
"""
|
||||
exercised = set(EXPECTED.values()) | {
|
||||
Classification.INTENT_CHANGED, Classification.REALIZATION_FAILED,
|
||||
}
|
||||
assert exercised == set(Classification)
|
||||
|
|
|
|||
|
|
@ -67,4 +67,18 @@ def test_actors_hold_isolated_credentials_and_memory():
|
|||
assert alice.credentials["token"] != bob.credentials["token"]
|
||||
alice.remember("secret", "only alice knows this")
|
||||
assert bob.recall("secret") is None
|
||||
assert bob.known_keys() == ()
|
||||
# Every actor carries its own automatic private marker (F-0003) and nothing
|
||||
# else it was not given.
|
||||
assert bob.known_keys() == ("__canary__",)
|
||||
assert alice.canary != bob.canary
|
||||
|
||||
|
||||
def test_every_run_records_a_verdict_on_isolation():
|
||||
"""F-0003: isolation is examined on every scenario, not only on ones
|
||||
written to expose it."""
|
||||
result, _ = run_once()
|
||||
examined = [
|
||||
obs for obs in result.evidence.observations if obs.kind == "actor_isolation"
|
||||
]
|
||||
assert len(examined) == 1
|
||||
assert examined[0].data["violations"] == []
|
||||
|
|
|
|||
5
usecases/__init__.py
Normal file
5
usecases/__init__.py
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
"""Durable test intent, kept separate from concrete scenario realizations."""
|
||||
|
||||
from .audit_core_e2_tenant_boundary import TEST_USE_CASE
|
||||
|
||||
__all__ = ["TEST_USE_CASE"]
|
||||
BIN
usecases/__pycache__/__init__.cpython-312.pyc
Normal file
BIN
usecases/__pycache__/__init__.cpython-312.pyc
Normal file
Binary file not shown.
Binary file not shown.
331
usecases/audit_core_e2_tenant_boundary.py
Normal file
331
usecases/audit_core_e2_tenant_boundary.py
Normal file
|
|
@ -0,0 +1,331 @@
|
|||
"""Audit-core E2 tenant-boundary TestUseCase.
|
||||
|
||||
This captures the intent and role hand-offs exercised successfully by
|
||||
WH-ENG-20260822-AUDIT-E2-03. It is deliberately not a runnable ``Scenario``
|
||||
yet. The current kernel has no causal/time-window schedule, external custody
|
||||
driver, Kubernetes driver, or independent cleanup observer capable of realizing
|
||||
this use case without an attended operator.
|
||||
|
||||
The claims come from the approved engagement and AUDIT-WP-0008-T05, not from
|
||||
reverse-engineering the successful responses. The -03 report is precedent
|
||||
evidence and a calibration example; it is not the source of truth for verdicts.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Mapping
|
||||
|
||||
from testdriver import Claim, Invariant, Provenance, UseCase
|
||||
|
||||
|
||||
SPEC_REF = (
|
||||
"audit-core/workplans/AUDIT-WP-0008-tenancy-posture-alignment.md"
|
||||
"#AUDIT-WP-0008-T05"
|
||||
)
|
||||
ENGAGEMENT_REF = (
|
||||
"whitehat-security/engagements/2026-08-22-audit-core-e2-03.json"
|
||||
)
|
||||
PRECEDENT_EVIDENCE_REF = (
|
||||
"audit-core/docs/evidence/"
|
||||
"AUDIT-WP-0008-T05-whitehat-e2-03-pass-2026-08-22.md"
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class RoleContract:
|
||||
"""One independently driven responsibility in the attended protocol."""
|
||||
|
||||
id: str
|
||||
responsibility: str
|
||||
may_observe_secret_values: bool = False
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class PhaseContract:
|
||||
"""Causal intent for the future orchestrator; never a shell transcript."""
|
||||
|
||||
id: str
|
||||
driver_role: str
|
||||
requires: tuple[str, ...]
|
||||
completion: str
|
||||
timing: str
|
||||
|
||||
|
||||
ROLE_CONTRACTS = (
|
||||
RoleContract(
|
||||
"authorizer",
|
||||
"Approve the exact production package, target, scope, window and cleanup "
|
||||
"obligation; hold no test credential.",
|
||||
),
|
||||
RoleContract(
|
||||
"target-owner",
|
||||
"Acknowledge the exact target revision, routes, fixtures and abort limits.",
|
||||
),
|
||||
RoleContract(
|
||||
"credential-custodian",
|
||||
"Preflight, project and revoke two short-lived tenant-scoped identities; "
|
||||
"emit value-safe projection and cleanup receipts.",
|
||||
may_observe_secret_values=True,
|
||||
),
|
||||
RoleContract(
|
||||
"security-coordinator",
|
||||
"Validate the engagement, admit only the receipt-bound plane lease, invoke "
|
||||
"the bounded probe, finalize the report and route it to risk-nexus.",
|
||||
),
|
||||
RoleContract(
|
||||
"cluster-executor",
|
||||
"Create, wait for and delete only the approved runner pod; mount but never "
|
||||
"read the projected Secret through the control plane.",
|
||||
),
|
||||
RoleContract(
|
||||
"tenant-a-attacker",
|
||||
"Use an ordinary tenant-A identity to attempt the declared cross-tenant "
|
||||
"reads and write.",
|
||||
may_observe_secret_values=True,
|
||||
),
|
||||
RoleContract(
|
||||
"tenant-b-control",
|
||||
"Use an independently scoped tenant-B identity only for known-good owner "
|
||||
"controls and state-after checks.",
|
||||
may_observe_secret_values=True,
|
||||
),
|
||||
RoleContract(
|
||||
"independent-observer",
|
||||
"Collect sanitized outcomes, target readiness and post-cleanup absence "
|
||||
"without accepting an actor's report as its own evidence.",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
PHASE_CONTRACTS = (
|
||||
PhaseContract(
|
||||
"authorize",
|
||||
"authorizer",
|
||||
(),
|
||||
"Exact engagement and custody contracts are approved.",
|
||||
"Before projection and before the engagement window opens.",
|
||||
),
|
||||
PhaseContract(
|
||||
"acknowledge-target",
|
||||
"target-owner",
|
||||
("authorize",),
|
||||
"Target owner accepts the exact revision, routes and bounded fixtures.",
|
||||
"Before any temporary credential exists.",
|
||||
),
|
||||
PhaseContract(
|
||||
"preflight",
|
||||
"credential-custodian",
|
||||
("acknowledge-target",),
|
||||
"Target is ready and all engagement-specific resources are absent.",
|
||||
"Immediately before the projection cutoff.",
|
||||
),
|
||||
PhaseContract(
|
||||
"project-custody",
|
||||
"credential-custodian",
|
||||
("preflight",),
|
||||
"Exactly two scoped handles exist and a value-safe receipt is emitted.",
|
||||
"Inside the narrow projection window; expiry equals engagement end.",
|
||||
),
|
||||
PhaseContract(
|
||||
"admit-plane",
|
||||
"security-coordinator",
|
||||
("project-custody",),
|
||||
"Receipt, broker contract, target revision and cleanup authority agree.",
|
||||
"After projection and before any target packet.",
|
||||
),
|
||||
PhaseContract(
|
||||
"ready-runner",
|
||||
"cluster-executor",
|
||||
("admit-plane",),
|
||||
"The exact attributed runner is Ready with its Secret mounted.",
|
||||
"Inside the authorized engagement window.",
|
||||
),
|
||||
PhaseContract(
|
||||
"run-probes",
|
||||
"security-coordinator",
|
||||
("ready-runner",),
|
||||
"Calibrated owner, attacker and absent controls produce a sanitized report.",
|
||||
"Before credential and engagement expiry.",
|
||||
),
|
||||
PhaseContract(
|
||||
"delete-runner",
|
||||
"cluster-executor",
|
||||
("run-probes",),
|
||||
"The exact runner pod is absent.",
|
||||
"Immediately after evidence collection.",
|
||||
),
|
||||
PhaseContract(
|
||||
"cleanup-custody",
|
||||
"credential-custodian",
|
||||
("delete-runner",),
|
||||
"Both identities, exact KV paths, projection resources and Secret are absent.",
|
||||
"Before credential and engagement expiry.",
|
||||
),
|
||||
PhaseContract(
|
||||
"finalize-and-deliver",
|
||||
"security-coordinator",
|
||||
("cleanup-custody",),
|
||||
"The report is bound to projection and cleanup receipts and reaches risk-nexus.",
|
||||
"Only after cleanup is independently observable.",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _same_absence_surface(left: Mapping[str, object], right: Mapping[str, object]) -> bool:
|
||||
"""Compare only the sanitized non-disclosure surface."""
|
||||
|
||||
return (
|
||||
left["status"] == right["status"]
|
||||
and left["schema"] == right["schema"]
|
||||
and left["digest"] == right["digest"]
|
||||
and left["fixture_match_count"] == 0
|
||||
and right["fixture_match_count"] == 0
|
||||
)
|
||||
|
||||
|
||||
def _event_id_is_tenant_scoped(obs: Mapping[str, object]) -> bool:
|
||||
probe = obs["event_by_id"]
|
||||
owner = probe["owner"]
|
||||
attacker = probe["attacker"]
|
||||
absent = probe["absent"]
|
||||
return (
|
||||
owner["status"] == 200
|
||||
and owner["fixture_match_count"] > 0
|
||||
and _same_absence_surface(attacker, absent)
|
||||
)
|
||||
|
||||
|
||||
def _correlation_slice_is_tenant_scoped(obs: Mapping[str, object]) -> bool:
|
||||
probe = obs["correlation_slice"]
|
||||
return (
|
||||
probe["owner"]["status"] == 200
|
||||
and probe["owner"]["fixture_match_count"] > 0
|
||||
and probe["attacker"]["status"] == 200
|
||||
and probe["attacker"]["fixture_match_count"] == 0
|
||||
)
|
||||
|
||||
|
||||
def _cross_tenant_append_is_refused(obs: Mapping[str, object]) -> bool:
|
||||
probe = obs["append_as_b"]
|
||||
return (
|
||||
probe["attacker"]["status"] in {400, 403, 404}
|
||||
and _same_absence_surface(probe["state_after"], probe["absent"])
|
||||
)
|
||||
|
||||
|
||||
def _cleanup_is_complete(obs: Mapping[str, object]) -> bool:
|
||||
cleanup = obs["cleanup"]
|
||||
return (
|
||||
cleanup["completed_before_expiry"] is True
|
||||
and cleanup["runner_absent"] is True
|
||||
and cleanup["temporary_identities"] == []
|
||||
and cleanup["kv_paths"] == []
|
||||
and cleanup["projection_resources"] == []
|
||||
and cleanup["mounted_secrets"] == []
|
||||
and cleanup["target_ready"] is True
|
||||
)
|
||||
|
||||
|
||||
def _execution_remains_bounded(obs: Mapping[str, object]) -> bool:
|
||||
run = obs["run"]
|
||||
return (
|
||||
run["attempted_operations"] <= run["maximum_operations"]
|
||||
and run["max_concurrency"] == 1
|
||||
and run["rate_per_minute"] <= 10
|
||||
and run["target_revision_matches"] is True
|
||||
and run["attribution_complete"] is True
|
||||
)
|
||||
|
||||
|
||||
def _secret_values_do_not_enter_evidence(obs: Mapping[str, object]) -> bool:
|
||||
return obs["cleanup"]["secret_values_observed"] is False
|
||||
|
||||
|
||||
def _report_preserves_assurance_limit(obs: Mapping[str, object]) -> bool:
|
||||
report = obs["report"]
|
||||
return (
|
||||
report["bound_to_projection_receipt"] is True
|
||||
and report["bound_to_cleanup_receipt"] is True
|
||||
and report["bounded_assurance_statement"] is True
|
||||
and report["delivered_to_risk_nexus"] is True
|
||||
)
|
||||
|
||||
|
||||
TEST_USE_CASE = UseCase(
|
||||
id="uc-audit-core-e2-tenant-boundary",
|
||||
title="Demonstrate audit-core's E2 tenant boundary with separate drivers",
|
||||
narrative=(
|
||||
"Within one explicitly authorized production window, independent target, "
|
||||
"custody, security, cluster, attacker, control and observer roles establish "
|
||||
"that an ordinary tenant-A identity cannot read or create tenant-B audit "
|
||||
"fixtures. The run must remain bounded, preserve non-disclosure semantics, "
|
||||
"clean every temporary resource before expiry, and deliver a cleanup-bound "
|
||||
"sanitized report without retaining credential values."
|
||||
),
|
||||
provenance=Provenance.SPEC,
|
||||
source_ref=SPEC_REF,
|
||||
claims=(
|
||||
Claim(
|
||||
"c-event-id-tenant-scoped",
|
||||
"Tenant A cannot distinguish tenant B's event id from an absent event",
|
||||
Provenance.SPEC,
|
||||
_event_id_is_tenant_scoped,
|
||||
after_step="run-probes",
|
||||
source_ref=ENGAGEMENT_REF,
|
||||
),
|
||||
Claim(
|
||||
"c-correlation-slice-tenant-scoped",
|
||||
"Tenant A's correlation slice contains no tenant-B fixture",
|
||||
Provenance.SPEC,
|
||||
_correlation_slice_is_tenant_scoped,
|
||||
after_step="run-probes",
|
||||
source_ref=ENGAGEMENT_REF,
|
||||
),
|
||||
Claim(
|
||||
"c-cross-tenant-append-refused",
|
||||
"Tenant A cannot append an audit event attributed to tenant B",
|
||||
Provenance.SPEC,
|
||||
_cross_tenant_append_is_refused,
|
||||
after_step="run-probes",
|
||||
source_ref=ENGAGEMENT_REF,
|
||||
),
|
||||
Claim(
|
||||
"c-receipt-bound-cleanup",
|
||||
"Every engagement-specific runtime and custody resource is absent "
|
||||
"before expiry while the target remains ready",
|
||||
Provenance.SPEC,
|
||||
_cleanup_is_complete,
|
||||
after_step="cleanup-custody",
|
||||
source_ref=SPEC_REF,
|
||||
),
|
||||
),
|
||||
invariants=(
|
||||
Invariant(
|
||||
"i-bounded-execution",
|
||||
"The run stays within its operation, rate, concurrency, revision and "
|
||||
"attribution boundaries",
|
||||
Provenance.SPEC,
|
||||
_execution_remains_bounded,
|
||||
source_ref=ENGAGEMENT_REF,
|
||||
),
|
||||
Invariant(
|
||||
"i-no-secret-evidence",
|
||||
"Credential values never enter retained observations or reports",
|
||||
Provenance.SPEC,
|
||||
_secret_values_do_not_enter_evidence,
|
||||
source_ref=SPEC_REF,
|
||||
),
|
||||
Invariant(
|
||||
"i-bounded-assurance-report",
|
||||
"Delivery is receipt-bound and says only that attempted attacks failed",
|
||||
Provenance.SPEC,
|
||||
_report_preserves_assurance_limit,
|
||||
source_ref=SPEC_REF,
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
# Alias retained for the existing scenario-module convention.
|
||||
USE_CASE = TEST_USE_CASE
|
||||
|
||||
|
|
@ -4,11 +4,11 @@ type: workplan
|
|||
title: "Vertical spike: falsify the crystallization thesis"
|
||||
domain: infotech
|
||||
repo: test-driver
|
||||
status: active
|
||||
status: finished
|
||||
owner: codex
|
||||
topic_slug: custodian
|
||||
created: "2026-08-22"
|
||||
updated: "2026-08-22"
|
||||
updated: "2026-08-23"
|
||||
state_hub_workstream_id: "68b252b8-03e7-55c7-8dec-ac13f6ca0aa1"
|
||||
---
|
||||
|
||||
|
|
@ -461,7 +461,7 @@ to execute than the agentic ancestor.
|
|||
|
||||
```task
|
||||
id: TD-WP-0002-T10
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "884c2e91-b56d-5b5e-a2d3-52d2a2768cae"
|
||||
```
|
||||
|
|
@ -478,3 +478,42 @@ retaining them, and update the Concept ↔ Implementation Fitness Map.
|
|||
Convert at least one framework finding discovered during T04–T09 into a permanent
|
||||
self-verification or deterministic regression — that closes the first genuine
|
||||
Concept–Implementation Fitness Loop.
|
||||
|
||||
**Done 2026-08-23.** Full write-up:
|
||||
`history/2026-08-23-td-wp-0002-gate-review.md`. 178 tests pass.
|
||||
|
||||
**All four gate criteria met.** M01/M02 absorbed with the claim set unchanged;
|
||||
the authorization defects escalated with regressions named; one asset
|
||||
crystallized and running with no model; **False Adaptation Rate 0/7**, with 12 of
|
||||
13 mechanical mutations absorbed so the safety result is not bought by escalating
|
||||
everything.
|
||||
|
||||
**Fitness loop closed via F-0003.** Actor isolation left no trace in ordinary
|
||||
evidence — it was a property of scenarios written to expose it. Every actor is
|
||||
now seeded with an automatic private marker and the runner examines all actors on
|
||||
every scenario, with two permanent regressions behind it, one of which catches a
|
||||
leak with the harness planting nothing.
|
||||
|
||||
**Compression — six abstractions removed**, each declared and never used:
|
||||
`Verdict.SUSPICIOUS` (a verdict no oracle could emit), `Step.expect_refusal`,
|
||||
`ActorIsolationError`, `World.seed`, `EvidencePack.latest()`,
|
||||
`Trajectory.method`.
|
||||
|
||||
**F-0008 — Temperature may be redundant.** Crystallization was built end to end
|
||||
without it ever being consulted; what triggered freezing was *measured stability
|
||||
of realization*, which is observed rather than declared. A declared temperature
|
||||
is a claim nobody verifies. Not deleted — two unevidenced cases for keeping it
|
||||
are recorded — but gated, along with `energy.py`: if the next workplan closes
|
||||
without a decision consulting either, they go.
|
||||
|
||||
**Two classifications had never run.** `INTENT_CHANGED` and `REALIZATION_FAILED`
|
||||
were declared at T08 and produced by no test. Both now have purpose-built cases,
|
||||
and `test_no_classification_is_unreachable` fails if a seventh is added without
|
||||
one.
|
||||
|
||||
**What this leaves.** The thesis survived contact with its own evidence in
|
||||
narrowed form, and the demonstration is architectural rather than statistical.
|
||||
The highest-value next step is a **bounded live-model experiment** — F-0005 and
|
||||
F-0007 converge on it independently, and one experiment settles both whether a
|
||||
model adds capability and whether crystallization has a product rationale. That
|
||||
should come before any further framework feature.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue