Compare commits
No commits in common. "28804f95c93d960592bee5f8e505c4940990d0a0" and "4f4219d8f72f5b6a03648bc212dadad6de4ee544" have entirely different histories.
28804f95c9
...
4f4219d8f7
64 changed files with 46 additions and 1101 deletions
5
.gitignore
vendored
5
.gitignore
vendored
|
|
@ -3,8 +3,3 @@
|
||||||
.claude/*
|
.claude/*
|
||||||
!.claude/rules/
|
!.claude/rules/
|
||||||
!.claude/rules/*.md
|
!.claude/rules/*.md
|
||||||
|
|
||||||
# Python
|
|
||||||
__pycache__/
|
|
||||||
*.py[cod]
|
|
||||||
.pytest_cache/
|
|
||||||
|
|
|
||||||
13
INTENT.md
13
INTENT.md
|
|
@ -116,14 +116,6 @@ COLD -> deterministic
|
||||||
|
|
||||||
The framework should make it natural for tests to crystallize as software cools.
|
The framework should make it natural for tests to crystallize as software cools.
|
||||||
|
|
||||||
> **Under review since T10 — see `research/findings/F-0008-temperature-may-be-redundant.md`.**
|
|
||||||
> Crystallization was built and demonstrated without Temperature ever being
|
|
||||||
> consulted. What actually triggered freezing was *measured stability of
|
|
||||||
> realization* — the same path found across several runs — which is observed,
|
|
||||||
> not declared. Temperature is a label a human would have to maintain and keep
|
|
||||||
> honest. If measured stability is sufficient, Temperature is a concept the
|
|
||||||
> framework does not need.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Energy
|
## Energy
|
||||||
|
|
@ -256,13 +248,10 @@ Initial verdicts are:
|
||||||
```text
|
```text
|
||||||
PASS
|
PASS
|
||||||
FAIL
|
FAIL
|
||||||
|
SUSPICIOUS
|
||||||
INCONCLUSIVE
|
INCONCLUSIVE
|
||||||
```
|
```
|
||||||
|
|
||||||
`SUSPICIOUS` was listed here originally and **removed at T10**: no oracle could
|
|
||||||
produce it, and a verdict nothing can emit is a promise the framework does not
|
|
||||||
keep. It returns, with an identifier, if a mechanism ever needs it.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Lineage
|
## Lineage
|
||||||
|
|
|
||||||
|
|
@ -25,7 +25,6 @@ No third-party dependencies. Python ≥ 3.11, pytest for the suite.
|
||||||
```
|
```
|
||||||
src/testdriver/ the kernel — intent, world, actions, drivers,
|
src/testdriver/ the kernel — intent, world, actions, drivers,
|
||||||
observers, oracles, evidence, runner
|
observers, oracles, evidence, runner
|
||||||
usecases/ durable test intent, including not-yet-runnable use cases
|
|
||||||
lab/ the system under test
|
lab/ the system under test
|
||||||
scenarios/ reference scenarios
|
scenarios/ reference scenarios
|
||||||
research/ hypotheses, experiments, findings, fitness map
|
research/ hypotheses, experiments, findings, fitness map
|
||||||
|
|
|
||||||
|
|
@ -21,5 +21,5 @@
|
||||||
| task | TD-WP-0002-T06 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
| task | TD-WP-0002-T06 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||||
| task | TD-WP-0002-T07 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
| task | TD-WP-0002-T07 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||||
| task | TD-WP-0002-T08 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
| task | TD-WP-0002-T08 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||||
| task | TD-WP-0002-T09 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
| task | TD-WP-0002-T09 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||||
| task | TD-WP-0002-T10 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
| task | TD-WP-0002-T10 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md |
|
||||||
|
|
|
||||||
BIN
crystallized/__pycache__/__init__.cpython-312.pyc
Normal file
BIN
crystallized/__pycache__/__init__.cpython-312.pyc
Normal file
Binary file not shown.
BIN
crystallized/__pycache__/conftest.cpython-312-pytest-7.4.4.pyc
Normal file
BIN
crystallized/__pycache__/conftest.cpython-312-pytest-7.4.4.pyc
Normal file
Binary file not shown.
Binary file not shown.
|
|
@ -1,212 +0,0 @@
|
||||||
# TD-WP-0002 — Gate Review and First Compression Pass
|
|
||||||
|
|
||||||
**Date:** 2026-08-23
|
|
||||||
**Workplan:** `TD-WP-0002` — Vertical spike: falsify the crystallization thesis
|
|
||||||
**Tests:** 172 passing · **Lab catalogue:** 24 labelled mutations
|
|
||||||
**Assessment that set the direction:** `history/2026-08-22-concept-assessment-swot.md`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 1. The gate
|
|
||||||
|
|
||||||
The workplan fixed four criteria in advance, before any code existed.
|
|
||||||
|
|
||||||
### Criterion 1 — recover from M01 and M02 with zero claim or invariant changes
|
|
||||||
|
|
||||||
**Met.**
|
|
||||||
|
|
||||||
| Mutation | Classification | Claim set unchanged | Accepted without a human |
|
|
||||||
|---|---|---|---|
|
|
||||||
| M01 sharing control moved into a modal | `MECHANICAL_ADAPTATION` | yes | yes |
|
|
||||||
| M02 DOM rewritten, test ids dropped | `MECHANICAL_ADAPTATION` | yes | yes |
|
|
||||||
|
|
||||||
The claim set is unchanged *by construction*, not by good behaviour: claims are
|
|
||||||
run inputs and the adaptation machinery has no write path to them (D-02).
|
|
||||||
|
|
||||||
### Criterion 2 — the authorization defect is reported as a Product Finding
|
|
||||||
|
|
||||||
**Met**, with a naming correction. The workplan inherited "M05" from the original
|
|
||||||
milestones document; in the catalogue as built, the authorization defect is
|
|
||||||
**M15** (revocation updates the record but not enforcement). All three
|
|
||||||
authorization defects behave correctly:
|
|
||||||
|
|
||||||
| Mutation | Classification | Regressions reported |
|
|
||||||
|---|---|---|
|
|
||||||
| M15 revoke is cosmetic | `BEHAVIOUR_CHANGED` | `c-bob-revoked`, `i-enforcement-matches-record` |
|
|
||||||
| M17 anyone can read anything | `BEHAVIOUR_CHANGED` | 5 assertions across 3 steps |
|
|
||||||
| M20 tenant isolation leaks | `BEHAVIOUR_CHANGED` | 5 assertions across 3 steps |
|
|
||||||
|
|
||||||
None is safe to accept. The classifier collapses `PRODUCT_DEFECT` and
|
|
||||||
`SEMANTIC_CHANGE` into one escalating outcome — see F-0006 — so "Product Finding"
|
|
||||||
in the original wording is now "escalated to a human with the regressions named".
|
|
||||||
|
|
||||||
### Criterion 3 — one asset crystallizes into a deterministic test
|
|
||||||
|
|
||||||
**Met.** `crystallized/test_grant_access.py` is generated, runs with **no model
|
|
||||||
involvement**, preserves its ancestor's oracle set as a superset, agrees with the
|
|
||||||
ancestor across five lab versions, still catches a seeded defect, and carries its
|
|
||||||
lineage in its own docstring.
|
|
||||||
|
|
||||||
### Criterion 4 — False Adaptation Rate = 0
|
|
||||||
|
|
||||||
**Met. 0 / 7.** No `DEFECT`-labelled mutation was classified as safe to accept,
|
|
||||||
including the three E-003 attacks (surface substitution, concurrent
|
|
||||||
mechanical+defect, evidence starvation) and provenance laundering.
|
|
||||||
|
|
||||||
The other side of the trade is asserted too: **12 of 13** mechanical mutations are
|
|
||||||
absorbed without a human. A classifier that escalated everything would score a
|
|
||||||
perfect FAR and be worthless.
|
|
||||||
|
|
||||||
### Gate verdict
|
|
||||||
|
|
||||||
**Passed, on all four criteria.** With one qualification that belongs in the same
|
|
||||||
sentence: FAR = 0 follows largely from *architecture* — claims are run inputs,
|
|
||||||
and `SAFE_TO_ACCEPT` is a closed two-element set — not from classifier accuracy.
|
|
||||||
The experiment shows the architecture behaves as designed over 24 hand-written
|
|
||||||
mutations. It says nothing about mutations nobody thought of.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 2. What the evidence changed
|
|
||||||
|
|
||||||
Three of the project's own claims were narrowed or corrected by building them.
|
|
||||||
This is the part worth more than the green gate.
|
|
||||||
|
|
||||||
**H-001 was narrowed (F-0005).** Semantic actions do not generally outlast
|
|
||||||
recorded interaction sequences. Where an application keeps stable identifiers,
|
|
||||||
the two arms tie exactly — 9/9 against 9/9 — and the conventional approach is
|
|
||||||
cheaper and deterministic. Semantic actions earn their keep only where
|
|
||||||
identifiers are absent or not carried forward (2/3 against 0/3). The concept
|
|
||||||
model presents this as a general superiority; it is conditional.
|
|
||||||
|
|
||||||
**The classifier cannot infer intent (F-0006).** The T02 design listed
|
|
||||||
`SEMANTIC_CHANGE` as an outcome the decision table could produce. It cannot: a
|
|
||||||
deliberate product decision and a defect are behaviourally identical, as M12 and
|
|
||||||
M19 have asserted since T05. The two collapse into `BEHAVIOUR_CHANGED` and the
|
|
||||||
distinction becomes a recorded human adjudication. Erring toward false alarm
|
|
||||||
costs one review; erring toward false adaptation ends the project.
|
|
||||||
|
|
||||||
**Crystallization's economic case is unmeasured (F-0007).** The descendant is 54%
|
|
||||||
cheaper than its ancestor, and that number must not be quoted in support of the
|
|
||||||
thesis. The runtime is token-free by design, so the entire saving is one page
|
|
||||||
fetch, one parse and a two-candidate scoring pass. The saving the concept
|
|
||||||
actually claims — model tokens, latency, retry variance — is absent from the
|
|
||||||
measurement.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 3. The compression pass
|
|
||||||
|
|
||||||
### Which concepts have no supporting evidence?
|
|
||||||
|
|
||||||
`Temperature`, `Confidence`, `Campaign`, `Metabolism`, `Retirement` — declared,
|
|
||||||
unimplemented, never consulted by any decision.
|
|
||||||
|
|
||||||
`Temperature` is the decidable one and is now **F-0008**. Crystallization was
|
|
||||||
built and demonstrated without it: what triggered freezing was *measured
|
|
||||||
stability of realization*, which is observed rather than declared. A declared
|
|
||||||
temperature is a claim nobody verifies, and the project's own first heuristic
|
|
||||||
argues against it. Not deleted yet — two unevidenced cases for keeping it are
|
|
||||||
recorded — but gated: if the next workplan completes without a decision
|
|
||||||
consulting it, it goes.
|
|
||||||
|
|
||||||
### Which implementation abstractions have no conceptual justification?
|
|
||||||
|
|
||||||
Removed at T10:
|
|
||||||
|
|
||||||
| Removed | Why |
|
|
||||||
|---|---|
|
|
||||||
| `Verdict.SUSPICIOUS` | No oracle could produce it. A verdict nothing can emit is a promise not kept. |
|
|
||||||
| `Step.expect_refusal` | Never set by any scenario. Speculative generality. |
|
|
||||||
| `ActorIsolationError` | Declared, never raised, never caught. |
|
|
||||||
| `World.seed` | Written at T04, never read. Replay comes from rebuilding the lab. |
|
|
||||||
| `EvidencePack.latest()` | Never called. |
|
|
||||||
| `Trajectory.method` | Computed on every capture, never read. |
|
|
||||||
|
|
||||||
### Which metadata has not informed a decision?
|
|
||||||
|
|
||||||
**`EnergyEvent` history.** Events have been recorded since T04 and have informed
|
|
||||||
nothing — as designed, since H-005 is dormant. Capture is retained because
|
|
||||||
history cannot be reconstructed later while scores always can, but this is now on
|
|
||||||
the same gate as Temperature: if the next workplan ends with no decision having
|
|
||||||
used it, the module goes.
|
|
||||||
|
|
||||||
### Which agentic behaviour can now be deterministic?
|
|
||||||
|
|
||||||
The `grant_access` realization, and it has been — that is T09. Nothing else in
|
|
||||||
the spike is agentic.
|
|
||||||
|
|
||||||
### What have the experiments falsified?
|
|
||||||
|
|
||||||
Nothing outright. Two claims were **narrowed** (H-001, and the classifier's
|
|
||||||
output space), and one criterion was met in a sense **too weak to support the
|
|
||||||
thesis** (H-003's cost clause). No hypothesis was rejected, and none was promoted
|
|
||||||
past `EXPERIMENTING`: 24 hand-written mutations is a small, self-chosen sample.
|
|
||||||
|
|
||||||
### Two outcomes that had never run
|
|
||||||
|
|
||||||
`INTENT_CHANGED` and `REALIZATION_FAILED` were declared at T08 and produced by no
|
|
||||||
test. Code that has never run is code nobody has checked. Rather than delete
|
|
||||||
meaningful outcomes or trust them untested, both now have purpose-built cases,
|
|
||||||
and `test_no_classification_is_unreachable` fails if a seventh is ever added
|
|
||||||
without one.
|
|
||||||
|
|
||||||
Building the `REALIZATION_FAILED` case clarified a boundary worth keeping: a run
|
|
||||||
that asserts *nothing at all* is `AMBIGUOUS`, not `REALIZATION_FAILED` — a use
|
|
||||||
case with no claims cannot conclude anything, however well its steps ran.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 4. The fitness loop, closed
|
|
||||||
|
|
||||||
The milestone asks that a framework finding become a permanent guarantee rather
|
|
||||||
than a note. **F-0003 is resolved.**
|
|
||||||
|
|
||||||
The finding: actor isolation left no trace in ordinary evidence. The self-test
|
|
||||||
caught a shared memory store only because the harness planted per-actor secrets
|
|
||||||
first. Isolation was a property of *scenarios written to expose it*, not of runs.
|
|
||||||
|
|
||||||
The fix: every `Actor` is seeded at construction with an automatic private
|
|
||||||
marker, and the runner examines all actors on **every** scenario, recording the
|
|
||||||
verdict as S3 evidence. Two regressions now stand behind it —
|
|
||||||
`test_every_run_records_a_verdict_on_isolation`, and a self-check that catches a
|
|
||||||
leak with the harness planting nothing at all.
|
|
||||||
|
|
||||||
```
|
|
||||||
Framework Finding (F-0003)
|
|
||||||
→ classified FRAMEWORK_LIMITATION
|
|
||||||
→ improvement hypothesis: automatic canaries make isolation observable
|
|
||||||
→ implemented at T10
|
|
||||||
→ permanent self-verification + deterministic regression
|
|
||||||
→ fitness map updated
|
|
||||||
```
|
|
||||||
|
|
||||||
That is the first complete Concept–Implementation Fitness Loop, which the
|
|
||||||
milestones document calls the point at which the self-improvement system
|
|
||||||
genuinely exists.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 5. Where this leaves the project
|
|
||||||
|
|
||||||
**The thesis survived contact with its own evidence, in narrowed form.** The
|
|
||||||
distinctive claim — that verification can adapt to a changing surface without
|
|
||||||
adapting to a defect — is demonstrated, and the demonstration is architectural
|
|
||||||
rather than statistical, which is the stronger kind.
|
|
||||||
|
|
||||||
**The single highest-value next step is a bounded live-model experiment.** Two
|
|
||||||
independent findings converge on it: M22 defeats the heuristic runtime while
|
|
||||||
remaining solvable by reading a visible label (F-0005), and crystallization's
|
|
||||||
economic case cannot be measured without token costs (F-0007). One experiment
|
|
||||||
settles whether a model adds capability *and* whether crystallization has a
|
|
||||||
product rationale. The `RealizationMetrics` fields were populated from the first
|
|
||||||
run so that comparison is a subtraction rather than a re-run.
|
|
||||||
|
|
||||||
That should come before any further framework feature.
|
|
||||||
|
|
||||||
**Open findings carried forward:** F-0003 resolved · F-0004 (structural
|
|
||||||
durability only; visual relayout untested) · F-0005 (H-001 narrowed) · F-0007
|
|
||||||
(cost unmeasurable; generated tests not fully standalone) · F-0008 (Temperature
|
|
||||||
gated). The F-0001 residual — the taxonomy has no class for document-to-document
|
|
||||||
drift — was considered and deliberately not resolved: adding a finding class to a
|
|
||||||
taxonomy already larger than its evidence would be the opposite of compression.
|
|
||||||
BIN
lab/__pycache__/__init__.cpython-312.pyc
Normal file
BIN
lab/__pycache__/__init__.cpython-312.pyc
Normal file
Binary file not shown.
BIN
lab/__pycache__/app.cpython-312.pyc
Normal file
BIN
lab/__pycache__/app.cpython-312.pyc
Normal file
Binary file not shown.
BIN
lab/__pycache__/http_api.cpython-312.pyc
Normal file
BIN
lab/__pycache__/http_api.cpython-312.pyc
Normal file
Binary file not shown.
BIN
lab/__pycache__/minimal.cpython-312.pyc
Normal file
BIN
lab/__pycache__/minimal.cpython-312.pyc
Normal file
Binary file not shown.
BIN
lab/__pycache__/mutations.cpython-312.pyc
Normal file
BIN
lab/__pycache__/mutations.cpython-312.pyc
Normal file
Binary file not shown.
|
|
@ -1,6 +1,6 @@
|
||||||
# Concept ↔ Implementation Fitness Map
|
# Concept ↔ Implementation Fitness Map
|
||||||
|
|
||||||
**Updated:** 2026-08-23 (TD-WP-0002-T10)
|
**Updated:** 2026-08-23 (TD-WP-0002-T09)
|
||||||
|
|
||||||
Traces each important concept to the implementation, experiment and evidence that
|
Traces each important concept to the implementation, experiment and evidence that
|
||||||
support it. **Unsupported entries are the point of this map** — a concept with no
|
support it. **Unsupported entries are the point of this map** — a concept with no
|
||||||
|
|
@ -29,7 +29,7 @@ were aspirational, not evidenced.
|
||||||
| Concept | Level | Implementation | Experiment | Evidence | Open question |
|
| Concept | Level | Implementation | Experiment | Evidence | Open question |
|
||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|
|
||||||
| `C-use-case` | C1 | `intent.py` | — | — | Is a use case expressible without leaking mechanics? |
|
| `C-use-case` | C1 | `intent.py` | — | — | Is a use case expressible without leaking mechanics? |
|
||||||
| `C-actor-isolation` | **C2** | `world.py`, `runner.py` | E-001 | `td://self/actor-isolation`, examined every run | **F-0003 resolved** — automatic canaries; observable on every scenario. |
|
| `C-actor-isolation` | C1 | `world.py` | E-001 | `td://self/actor-isolation` | **F-0003** — only observable when the scenario plants canaries. |
|
||||||
| `C-semantic-action` | **C2** | `actions.py`, `agentic.py` | E-001 (partial) | T07 arm comparison | **F-0005** — supported only where stable identifiers are absent. Narrower than the concept model claims. |
|
| `C-semantic-action` | **C2** | `actions.py`, `agentic.py` | E-001 (partial) | T07 arm comparison | **F-0005** — supported only where stable identifiers are absent. Narrower than the concept model claims. |
|
||||||
| `C-oracle-independence` | **C2** | `runner.py`, `oracles.py` | E-001, E-003 | — | Independence of components ≠ independence of belief. (H-004) |
|
| `C-oracle-independence` | **C2** | `runner.py`, `oracles.py` | E-001, E-003 | — | Independence of components ≠ independence of belief. (H-004) |
|
||||||
| `C-evidence-pack` | C1 | `evidence.py` | — | `td://self/evidence-reproducibility` | Verdicts are reproducible from S3 alone, on passing and failing runs. |
|
| `C-evidence-pack` | C1 | `evidence.py` | — | `td://self/evidence-reproducibility` | Verdicts are reproducible from S3 alone, on passing and failing runs. |
|
||||||
|
|
@ -39,9 +39,9 @@ were aspirational, not evidenced.
|
||||||
| `C-crystallization` | **C2** | `crystallization.py` | E-002 | T09 agreement matrix | Fidelity supported; **F-0007** — the economic case is unmeasurable with a token-free runtime. |
|
| `C-crystallization` | **C2** | `crystallization.py` | E-002 | T09 agreement matrix | Fidelity supported; **F-0007** — the economic case is unmeasurable with a token-free runtime. |
|
||||||
| `C-intent-provenance` | C1 | `provenance.py` | E-003 | `td://self/intent-independence` | Constrains provenance, not quality. Accepted residual. |
|
| `C-intent-provenance` | C1 | `provenance.py` | E-003 | `td://self/intent-independence` | Constrains provenance, not quality. Accepted residual. |
|
||||||
| `C-lineage` | C1 | `scenario.py`, generated headers | E-002 | generated module docstring | Parent pointer plus provenance in the artefact; no graph. |
|
| `C-lineage` | C1 | `scenario.py`, generated headers | E-002 | generated module docstring | Parent pointer plus provenance in the artefact; no graph. |
|
||||||
| `C-energy` | C0 | `energy.py`, capture only | — | none | Dormant. **Gated**: if the next workplan ends with no decision having used the history, remove. |
|
| `C-energy` | C0 | `energy.py`, capture only | — | — | Dormant by decision. (H-005) |
|
||||||
| `C-temperature` | **C0 — under review** | — | — | none | **F-0008** — crystallization was built without it; measured stability did the work. Gated for removal. |
|
| `C-temperature` | C0 | — | — | — | Deferred. No implementation planned in TD-WP-0002. |
|
||||||
| `C-confidence` | C0 | — | — | none | Declared, unimplemented, never consulted. Same position as `C-temperature` without a built alternative. |
|
| `C-confidence` | C0 | — | — | — | Deferred. |
|
||||||
| `C-campaign` | C0 | — | — | — | Deferred. |
|
| `C-campaign` | C0 | — | — | — | Deferred. |
|
||||||
| `C-metabolism` | C0 | — | — | — | Deferred. Depends on C-energy. |
|
| `C-metabolism` | C0 | — | — | — | Deferred. Depends on C-energy. |
|
||||||
| `C-retirement` | C0 | — | — | — | Deferred. Depends on C-energy. |
|
| `C-retirement` | C0 | — | — | — | Deferred. Depends on C-energy. |
|
||||||
|
|
@ -58,13 +58,6 @@ validated. They are revisited at T10, where the question is not "when do we buil
|
||||||
these" but "does the evidence justify keeping them in the model at all".
|
these" but "does the evidence justify keeping them in the model at all".
|
||||||
|
|
||||||
**Implementation orphans** — none. Every module in `src/testdriver/` traces to a
|
**Implementation orphans** — none. Every module in `src/testdriver/` traces to a
|
||||||
concept above.
|
concept above. `energy.py` is the one to watch: it exists solely to capture
|
||||||
|
events for a dormant hypothesis, and if T10 finds no use for the history it
|
||||||
**Removed at T10** (compression pass — see the gate review § 3):
|
should be removed rather than kept out of sentiment.
|
||||||
`Verdict.SUSPICIOUS`, `Step.expect_refusal`, `ActorIsolationError`,
|
|
||||||
`World.seed`, `EvidencePack.latest()`, `Trajectory.method`. Each was declared and
|
|
||||||
never used; `SUSPICIOUS` was additionally a verdict no oracle could emit.
|
|
||||||
|
|
||||||
**Gated for removal**: `C-temperature` (F-0008) and `energy.py`. Both survive the
|
|
||||||
current review on the strength of being cheap, not of being used. If the next
|
|
||||||
workplan closes without a decision consulting either, they go.
|
|
||||||
|
|
|
||||||
|
|
@ -2,13 +2,12 @@
|
||||||
id: F-0003
|
id: F-0003
|
||||||
type: framework-finding
|
type: framework-finding
|
||||||
class: FRAMEWORK_LIMITATION
|
class: FRAMEWORK_LIMITATION
|
||||||
status: resolved
|
status: open
|
||||||
discovered: "2026-08-22"
|
discovered: "2026-08-22"
|
||||||
discovered_by: TD-WP-0002-T06
|
discovered_by: TD-WP-0002-T06
|
||||||
workplan: TD-WP-0002
|
workplan: TD-WP-0002
|
||||||
task: TD-WP-0002-T06
|
task: TD-WP-0002-T06
|
||||||
resolved: "2026-08-23"
|
carried_to: TD-WP-0002-T10
|
||||||
resolved_in: TD-WP-0002-T10
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# F-0003 — Actor isolation is only observable if the scenario plants canaries
|
# F-0003 — Actor isolation is only observable if the scenario plants canaries
|
||||||
|
|
@ -56,31 +55,9 @@ system.
|
||||||
Worth stating plainly rather than counting four green self-tests as four
|
Worth stating plainly rather than counting four green self-tests as four
|
||||||
equivalent proofs. They are not equivalent.
|
equivalent proofs. They are not equivalent.
|
||||||
|
|
||||||
## Resolution (T10)
|
## Candidate resolutions
|
||||||
|
|
||||||
Candidate 3 — **both**.
|
Not resolved now; deciding cheaply here would be guessing.
|
||||||
|
|
||||||
Every `Actor` is seeded at construction with an automatic private marker
|
|
||||||
(`Actor.canary`), and `Runner._isolation_violations` examines every actor on
|
|
||||||
**every** scenario, recording the verdict as S3 evidence. An isolation violation
|
|
||||||
now leaves a trace whether or not anyone thought to look for one.
|
|
||||||
|
|
||||||
Two permanent regressions stand behind it:
|
|
||||||
|
|
||||||
- `tests/test_reference_scenario.py::test_every_run_records_a_verdict_on_isolation`
|
|
||||||
- `tests/selfverification/test_checks_can_fail.py::test_a_leak_is_caught_without_the_test_planting_anything`
|
|
||||||
— an actor holding another's marker is caught by the ordinary run, with the
|
|
||||||
harness planting nothing.
|
|
||||||
|
|
||||||
The second limitation stands as recorded: oracle independence is enforced by
|
|
||||||
construction, so it is verified once rather than per run. That asymmetry is real
|
|
||||||
and is now stated in the fitness map rather than hidden behind two equally green
|
|
||||||
self-tests.
|
|
||||||
|
|
||||||
This finding closed the first Concept–Implementation Fitness Loop — see
|
|
||||||
`history/2026-08-23-td-wp-0002-gate-review.md` § 4.
|
|
||||||
|
|
||||||
## Candidate resolutions considered
|
|
||||||
|
|
||||||
1. **Automatic canaries** — the runtime seeds each actor with a unique private
|
1. **Automatic canaries** — the runtime seeds each actor with a unique private
|
||||||
marker at construction and the isolation check runs on every Evidence Pack,
|
marker at construction and the isolation check runs on every Evidence Pack,
|
||||||
|
|
|
||||||
|
|
@ -1,70 +0,0 @@
|
||||||
---
|
|
||||||
id: F-0008
|
|
||||||
type: framework-finding
|
|
||||||
class: UNNECESSARY_COMPLEXITY
|
|
||||||
status: open
|
|
||||||
discovered: "2026-08-23"
|
|
||||||
discovered_by: TD-WP-0002-T10
|
|
||||||
workplan: TD-WP-0002
|
|
||||||
task: TD-WP-0002-T10
|
|
||||||
carried_to: next workplan
|
|
||||||
---
|
|
||||||
|
|
||||||
# F-0008 — Temperature may be redundant; measured stability did the work
|
|
||||||
|
|
||||||
## Observation
|
|
||||||
|
|
||||||
`Temperature` (`HOT` / `WARM` / `COOL` / `COLD`) is one of the more prominent
|
|
||||||
ideas in `INTENT.md` and the Concept Model. It is supposed to govern which
|
|
||||||
verification mode an asset uses, and it is the mechanism by which "tests
|
|
||||||
crystallize as software cools".
|
|
||||||
|
|
||||||
The spike built crystallization end to end. **Temperature was never consulted,
|
|
||||||
never implemented, and never missed.**
|
|
||||||
|
|
||||||
What actually triggered freezing was `assess_stability`: the same realization
|
|
||||||
path observed across several consecutive runs. Nothing declared the surface cold;
|
|
||||||
the surface was observed to have stopped moving.
|
|
||||||
|
|
||||||
## The argument for removal
|
|
||||||
|
|
||||||
The two mechanisms answer the same question, and only one of them can be wrong:
|
|
||||||
|
|
||||||
| | Temperature | Measured stability |
|
|
||||||
|---|---|---|
|
|
||||||
| source | declared by a human | observed from runs |
|
|
||||||
| upkeep | must be maintained and kept honest | none |
|
|
||||||
| failure mode | says `COLD` while the surface churns | none — it is the churn |
|
|
||||||
| already built | no | yes |
|
|
||||||
|
|
||||||
A declared temperature is a claim about the system that nobody verifies. The
|
|
||||||
project's own first heuristic — *do not let a declaration substitute for
|
|
||||||
evidence* — argues against keeping it.
|
|
||||||
|
|
||||||
## The argument for keeping it
|
|
||||||
|
|
||||||
Two cases measured stability cannot cover:
|
|
||||||
|
|
||||||
1. **Forward-looking scope.** A team knows a rewrite lands next month.
|
|
||||||
Temperature can say `HOT` before any run has observed instability, where
|
|
||||||
measured stability would happily freeze an asset the day before it breaks.
|
|
||||||
2. **Cheap prioritisation.** Deciding *where to spend effort* across many
|
|
||||||
capabilities may want a coarse label without running anything.
|
|
||||||
|
|
||||||
Neither case arose in the spike, and neither is evidenced.
|
|
||||||
|
|
||||||
## Recommendation
|
|
||||||
|
|
||||||
Do not delete yet; do not build either. Carry `Temperature` as **explicitly
|
|
||||||
unvalidated** in the fitness map, and set a gate: if the next workplan completes
|
|
||||||
without any decision consulting it, remove it from the concept model.
|
|
||||||
|
|
||||||
Recorded now because the compression review's purpose is to catch concepts that
|
|
||||||
survive on the strength of being attractive rather than useful, and Temperature
|
|
||||||
is currently the clearest instance in the corpus.
|
|
||||||
|
|
||||||
## Related
|
|
||||||
|
|
||||||
`Confidence`, `Campaign`, `Metabolism` and `Retirement` are in the same position —
|
|
||||||
declared, unimplemented, never consulted — but Temperature is the one with a
|
|
||||||
built alternative already doing its job, which makes it the decidable case.
|
|
||||||
BIN
scenarios/__pycache__/alice_bob_carol.cpython-312.pyc
Normal file
BIN
scenarios/__pycache__/alice_bob_carol.cpython-312.pyc
Normal file
Binary file not shown.
BIN
scenarios/__pycache__/browser_grant.cpython-312.pyc
Normal file
BIN
scenarios/__pycache__/browser_grant.cpython-312.pyc
Normal file
Binary file not shown.
BIN
scenarios/__pycache__/full_journey.cpython-312.pyc
Normal file
BIN
scenarios/__pycache__/full_journey.cpython-312.pyc
Normal file
Binary file not shown.
|
|
@ -128,7 +128,9 @@ def build(*mutations: str):
|
||||||
cast.add(Actor(id=name, display_name=name.title(),
|
cast.add(Actor(id=name, display_name=name.title(),
|
||||||
credentials={"token": tokens[name]}))
|
credentials={"token": tokens[name]}))
|
||||||
|
|
||||||
world = World(id="w-baseline", sut=lab, sut_version=lab.version, cast=cast)
|
world = World(id="w-baseline", sut=lab, sut_version=lab.version,
|
||||||
|
seed={"users": ["alice", "bob", "carol"], "resource": RESOURCE},
|
||||||
|
cast=cast)
|
||||||
|
|
||||||
scenario = Scenario(
|
scenario = Scenario(
|
||||||
id="sc-share-resource",
|
id="sc-share-resource",
|
||||||
|
|
|
||||||
|
|
@ -10,10 +10,10 @@ from .oracles import Judgment, Oracle, Verdict, overall
|
||||||
from .provenance import InadmissibleProvenance, Provenance
|
from .provenance import InadmissibleProvenance, Provenance
|
||||||
from .runner import CollectorIndependenceError, Runner, RunResult
|
from .runner import CollectorIndependenceError, Runner, RunResult
|
||||||
from .scenario import Scenario, Step, VerificationAsset
|
from .scenario import Scenario, Step, VerificationAsset
|
||||||
from .world import Actor, Cast, World
|
from .world import Actor, ActorIsolationError, Cast, World
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
"Actor", "Cast", "Claim", "CollectorIndependenceError",
|
"Actor", "ActorIsolationError", "Cast", "Claim", "CollectorIndependenceError",
|
||||||
"DirectDriver", "EnergyEvent", "EnergyEventType", "EvidencePack",
|
"DirectDriver", "EnergyEvent", "EnergyEventType", "EvidencePack",
|
||||||
"InadmissibleProvenance", "Invariant", "Judgment", "Observation", "Oracle",
|
"InadmissibleProvenance", "Invariant", "Judgment", "Observation", "Oracle",
|
||||||
"Provenance", "Realization", "RunResult", "Runner", "Scenario",
|
"Provenance", "Realization", "RunResult", "Runner", "Scenario",
|
||||||
|
|
|
||||||
BIN
src/testdriver/__pycache__/__init__.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/__init__.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/actions.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/actions.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/agentic.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/agentic.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/browser.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/browser.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/classification.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/classification.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/crystallization.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/crystallization.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/drivers.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/drivers.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/energy.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/energy.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/evidence.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/evidence.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/html.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/html.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/intent.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/intent.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/observers.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/observers.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/oracles.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/oracles.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/provenance.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/provenance.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/runner.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/runner.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/scenario.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/scenario.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/testdriver/__pycache__/world.cpython-312.pyc
Normal file
BIN
src/testdriver/__pycache__/world.cpython-312.pyc
Normal file
Binary file not shown.
|
|
@ -35,6 +35,7 @@ class Trajectory:
|
||||||
step_id: str
|
step_id: str
|
||||||
action_name: str
|
action_name: str
|
||||||
surface_id: str
|
surface_id: str
|
||||||
|
method: str
|
||||||
target: str
|
target: str
|
||||||
fields: tuple[str, ...]
|
fields: tuple[str, ...]
|
||||||
|
|
||||||
|
|
@ -42,7 +43,7 @@ class Trajectory:
|
||||||
return json.dumps(
|
return json.dumps(
|
||||||
{
|
{
|
||||||
"step": self.step_id, "action": self.action_name,
|
"step": self.step_id, "action": self.action_name,
|
||||||
"surface": self.surface_id,
|
"surface": self.surface_id, "method": self.method,
|
||||||
"target": self.target, "fields": sorted(self.fields),
|
"target": self.target, "fields": sorted(self.fields),
|
||||||
},
|
},
|
||||||
sort_keys=True,
|
sort_keys=True,
|
||||||
|
|
@ -62,6 +63,7 @@ def capture(pack: Mapping[str, Any]) -> tuple[Trajectory, ...]:
|
||||||
step_id=obs["step_id"],
|
step_id=obs["step_id"],
|
||||||
action_name=str(mechanics.get("action", "")).split("(")[0],
|
action_name=str(mechanics.get("action", "")).split("(")[0],
|
||||||
surface_id=obs["data"].get("surface", ""),
|
surface_id=obs["data"].get("surface", ""),
|
||||||
|
method=("POST" if obs["data"].get("surface") == "browser" else "CALL"),
|
||||||
target=target,
|
target=target,
|
||||||
fields=tuple(sorted(fields)),
|
fields=tuple(sorted(fields)),
|
||||||
))
|
))
|
||||||
|
|
|
||||||
|
|
@ -72,6 +72,12 @@ class EvidencePack:
|
||||||
def of_stratum(self, stratum: Stratum) -> list[Observation]:
|
def of_stratum(self, stratum: Stratum) -> list[Observation]:
|
||||||
return [o for o in self.observations if o.stratum is stratum]
|
return [o for o in self.observations if o.stratum is stratum]
|
||||||
|
|
||||||
|
def latest(self, kind: str) -> Observation | None:
|
||||||
|
for observation in reversed(self.observations):
|
||||||
|
if observation.kind == kind:
|
||||||
|
return observation
|
||||||
|
return None
|
||||||
|
|
||||||
def to_json(self) -> str:
|
def to_json(self) -> str:
|
||||||
payload = asdict(self)
|
payload = asdict(self)
|
||||||
payload["observations"] = [
|
payload["observations"] = [
|
||||||
|
|
|
||||||
|
|
@ -3,10 +3,6 @@
|
||||||
An oracle reads only the independent observation snapshot. It has no access to
|
An oracle reads only the independent observation snapshot. It has no access to
|
||||||
the actor, to the driver, or to what either of them believes happened.
|
the actor, to the driver, or to what either of them believes happened.
|
||||||
|
|
||||||
`SUSPICIOUS` was removed at T10: no oracle could produce it, and a verdict
|
|
||||||
nothing can emit is a promise the framework does not keep. It returns, with an
|
|
||||||
identifier, if a mechanism ever needs it.
|
|
||||||
|
|
||||||
`INCONCLUSIVE` is a first-class outcome, not a failure mode of the framework. An
|
`INCONCLUSIVE` is a first-class outcome, not a failure mode of the framework. An
|
||||||
oracle that cannot support a judgment from the retained evidence must say so
|
oracle that cannot support a judgment from the retained evidence must say so
|
||||||
rather than defaulting either way — silently defaulting to PASS hides defects,
|
rather than defaulting either way — silently defaulting to PASS hides defects,
|
||||||
|
|
@ -25,6 +21,7 @@ from .intent import Claim, Invariant
|
||||||
class Verdict(str, Enum):
|
class Verdict(str, Enum):
|
||||||
PASS = "PASS"
|
PASS = "PASS"
|
||||||
FAIL = "FAIL"
|
FAIL = "FAIL"
|
||||||
|
SUSPICIOUS = "SUSPICIOUS"
|
||||||
INCONCLUSIVE = "INCONCLUSIVE"
|
INCONCLUSIVE = "INCONCLUSIVE"
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -104,4 +101,6 @@ def overall(judgments: list[Judgment]) -> Verdict:
|
||||||
return Verdict.FAIL
|
return Verdict.FAIL
|
||||||
if Verdict.INCONCLUSIVE in verdicts:
|
if Verdict.INCONCLUSIVE in verdicts:
|
||||||
return Verdict.INCONCLUSIVE
|
return Verdict.INCONCLUSIVE
|
||||||
|
if Verdict.SUSPICIOUS in verdicts:
|
||||||
|
return Verdict.SUSPICIOUS
|
||||||
return Verdict.PASS
|
return Verdict.PASS
|
||||||
|
|
|
||||||
|
|
@ -56,24 +56,6 @@ class Runner:
|
||||||
|
|
||||||
# -- independence guards ---------------------------------------------
|
# -- independence guards ---------------------------------------------
|
||||||
|
|
||||||
def _isolation_violations(self) -> list[str]:
|
|
||||||
"""Does any actor hold another's canary?
|
|
||||||
|
|
||||||
Run on every scenario, not only on ones written to test isolation.
|
|
||||||
"""
|
|
||||||
canaries = {actor.canary: actor.id for actor in self._world.cast}
|
|
||||||
violations: list[str] = []
|
|
||||||
for actor in self._world.cast:
|
|
||||||
for key in actor.known_keys():
|
|
||||||
value = actor.recall(key)
|
|
||||||
owner = canaries.get(value) if isinstance(value, str) else None
|
|
||||||
if owner is not None and owner != actor.id:
|
|
||||||
violations.append(
|
|
||||||
f"actor {actor.id!r} holds the private marker of {owner!r} "
|
|
||||||
f"under key {key!r}"
|
|
||||||
)
|
|
||||||
return violations
|
|
||||||
|
|
||||||
def _assert_collector_independence(self, stratum: Stratum, collector: str) -> None:
|
def _assert_collector_independence(self, stratum: Stratum, collector: str) -> None:
|
||||||
"""S2 and S3 evidence may never be attributed to an actor.
|
"""S2 and S3 evidence may never be attributed to an actor.
|
||||||
|
|
||||||
|
|
@ -129,12 +111,6 @@ class Runner:
|
||||||
EnergyEvent(asset.id, run_id, EnergyEventType.EXECUTED).as_dict()
|
EnergyEvent(asset.id, run_id, EnergyEventType.EXECUTED).as_dict()
|
||||||
)
|
)
|
||||||
|
|
||||||
isolation = self._isolation_violations()
|
|
||||||
self._record(
|
|
||||||
pack, Stratum.JUDGMENT, self._observer.name, "actor_isolation",
|
|
||||||
{"violations": isolation, "actors": sorted(self._world.cast.actors)}, None,
|
|
||||||
)
|
|
||||||
|
|
||||||
judgments: list[Judgment] = []
|
judgments: list[Judgment] = []
|
||||||
scenario_sound = True
|
scenario_sound = True
|
||||||
claims_by_step: dict[str, list] = {}
|
claims_by_step: dict[str, list] = {}
|
||||||
|
|
@ -190,6 +166,7 @@ class Runner:
|
||||||
"action": step.action.name,
|
"action": step.action.name,
|
||||||
"surface_used": realization.surface_id,
|
"surface_used": realization.surface_id,
|
||||||
"refused_by_sut": refused,
|
"refused_by_sut": refused,
|
||||||
|
"refusal_expected": step.expect_refusal,
|
||||||
"postcondition_met": postcondition_met,
|
"postcondition_met": postcondition_met,
|
||||||
},
|
},
|
||||||
step.id,
|
step.id,
|
||||||
|
|
@ -199,7 +176,7 @@ class Runner:
|
||||||
# An action that was accepted but did not take effect did happen —
|
# An action that was accepted but did not take effect did happen —
|
||||||
# and that is a statement about the system, judged below, not a
|
# and that is a statement about the system, judged below, not a
|
||||||
# reason to stop judging.
|
# reason to stop judging.
|
||||||
if refused:
|
if refused and not step.expect_refusal:
|
||||||
scenario_sound = False
|
scenario_sound = False
|
||||||
|
|
||||||
# --- invariants after every step -----------------------------
|
# --- invariants after every step -----------------------------
|
||||||
|
|
|
||||||
|
|
@ -17,6 +17,7 @@ class Step:
|
||||||
id: str
|
id: str
|
||||||
actor_id: str
|
actor_id: str
|
||||||
action: SemanticAction
|
action: SemanticAction
|
||||||
|
expect_refusal: bool = False
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True, slots=True)
|
@dataclass(frozen=True, slots=True)
|
||||||
|
|
|
||||||
|
|
@ -8,20 +8,12 @@ to be executed by the same process.
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import uuid
|
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
from typing import Any, Iterator
|
from typing import Any, Iterator
|
||||||
|
|
||||||
|
|
||||||
def _canary() -> str:
|
class ActorIsolationError(Exception):
|
||||||
"""A value private to one actor, unguessable and unique per construction.
|
"""Raised when one actor is offered another actor's private state."""
|
||||||
|
|
||||||
Exists so that isolation is *observable*. Without it a run in which every
|
|
||||||
actor shares one memory store produces evidence indistinguishable from a
|
|
||||||
correct one — the guarantee holds only in scenarios written to expose it,
|
|
||||||
which is no guarantee at all (F-0003).
|
|
||||||
"""
|
|
||||||
return f"canary-{uuid.uuid4().hex}"
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(slots=True)
|
@dataclass(slots=True)
|
||||||
|
|
@ -36,13 +28,8 @@ class Actor:
|
||||||
id: str
|
id: str
|
||||||
display_name: str
|
display_name: str
|
||||||
credentials: dict[str, str] = field(default_factory=dict)
|
credentials: dict[str, str] = field(default_factory=dict)
|
||||||
|
session: dict[str, Any] = field(default_factory=dict)
|
||||||
_memory: dict[str, Any] = field(default_factory=dict, repr=False)
|
_memory: dict[str, Any] = field(default_factory=dict, repr=False)
|
||||||
canary: str = field(default_factory=_canary)
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
# Seeded automatically, on every actor, in every scenario. An isolation
|
|
||||||
# violation now leaves a trace whether or not anyone thought to look.
|
|
||||||
self._memory.setdefault("__canary__", self.canary)
|
|
||||||
|
|
||||||
def remember(self, key: str, value: Any) -> None:
|
def remember(self, key: str, value: Any) -> None:
|
||||||
self._memory[key] = value
|
self._memory[key] = value
|
||||||
|
|
@ -78,14 +65,15 @@ class Cast:
|
||||||
|
|
||||||
@dataclass(slots=True)
|
@dataclass(slots=True)
|
||||||
class World:
|
class World:
|
||||||
"""The handle to the system under test, plus the cast acting on it.
|
"""Initial state plus the handle to the system under test.
|
||||||
|
|
||||||
Replay comes from rebuilding the lab through `build_lab`, not from a seed
|
`seed` is everything needed to rebuild the initial state, so that a run can
|
||||||
dict carried here — the dict was written at T04, never read, and removed at
|
be replayed from a known starting point rather than from wherever the
|
||||||
T10.
|
previous run happened to leave things.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
id: str
|
id: str
|
||||||
sut: Any
|
sut: Any
|
||||||
sut_version: str
|
sut_version: str
|
||||||
|
seed: dict[str, Any] = field(default_factory=dict)
|
||||||
cast: Cast = field(default_factory=Cast)
|
cast: Cast = field(default_factory=Cast)
|
||||||
|
|
|
||||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
BIN
tests/__pycache__/test_kernel_guarantees.cpython-312.pyc
Normal file
BIN
tests/__pycache__/test_kernel_guarantees.cpython-312.pyc
Normal file
Binary file not shown.
Binary file not shown.
Binary file not shown.
BIN
tests/selfverification/__pycache__/__init__.cpython-312.pyc
Normal file
BIN
tests/selfverification/__pycache__/__init__.cpython-312.pyc
Normal file
Binary file not shown.
BIN
tests/selfverification/__pycache__/checks.cpython-312.pyc
Normal file
BIN
tests/selfverification/__pycache__/checks.cpython-312.pyc
Normal file
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
|
@ -94,23 +94,6 @@ def check_actor_isolation(
|
||||||
return violations
|
return violations
|
||||||
|
|
||||||
|
|
||||||
def check_isolation_was_examined(pack: Mapping[str, Any]) -> list[str]:
|
|
||||||
"""Every run must carry a verdict on actor isolation — F-0003, resolved.
|
|
||||||
|
|
||||||
Before this existed, isolation was only observable in scenarios written to
|
|
||||||
expose it: a run in which every actor shared one memory store produced
|
|
||||||
evidence indistinguishable from a correct one. Actors now carry an automatic
|
|
||||||
private marker and the runner examines them on every scenario, so the absence
|
|
||||||
of this observation is itself a failure.
|
|
||||||
"""
|
|
||||||
for obs in _observations(pack):
|
|
||||||
if obs["kind"] != "actor_isolation":
|
|
||||||
continue
|
|
||||||
violations = obs["data"].get("violations") or []
|
|
||||||
return [f"actor isolation violated: {v}" for v in violations]
|
|
||||||
return ["this run did not examine actor isolation at all"]
|
|
||||||
|
|
||||||
|
|
||||||
# --- td://self/oracle-independence ---------------------------------------
|
# --- td://self/oracle-independence ---------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -30,7 +30,6 @@ from tests.selfverification.checks import (
|
||||||
check_actor_isolation,
|
check_actor_isolation,
|
||||||
check_evidence_supports_every_verdict,
|
check_evidence_supports_every_verdict,
|
||||||
check_intent_independence,
|
check_intent_independence,
|
||||||
check_isolation_was_examined,
|
|
||||||
check_no_actor_collected_judgment,
|
check_no_actor_collected_judgment,
|
||||||
check_runs_agree,
|
check_runs_agree,
|
||||||
check_verdicts_follow_from_judgment_evidence,
|
check_verdicts_follow_from_judgment_evidence,
|
||||||
|
|
@ -187,35 +186,3 @@ def test_unrecorded_provenance_is_caught():
|
||||||
tampered = copy.deepcopy(pack)
|
tampered = copy.deepcopy(pack)
|
||||||
tampered["provenance_index"] = {}
|
tampered["provenance_index"] = {}
|
||||||
assert check_intent_independence(tampered)
|
assert check_intent_independence(tampered)
|
||||||
|
|
||||||
|
|
||||||
# --- F-0003 resolution: isolation observed on every run -------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_an_unexamined_run_is_caught():
|
|
||||||
"""A run that never looked at isolation must not read as isolated."""
|
|
||||||
pack, _, _ = run_and_serialize()
|
|
||||||
tampered = copy.deepcopy(pack)
|
|
||||||
tampered["observations"] = [
|
|
||||||
o for o in tampered["observations"] if o["kind"] != "actor_isolation"
|
|
||||||
]
|
|
||||||
assert check_isolation_was_examined(tampered)
|
|
||||||
|
|
||||||
|
|
||||||
def test_a_leak_is_caught_without_the_test_planting_anything():
|
|
||||||
"""The regression F-0003 leaves behind.
|
|
||||||
|
|
||||||
No secrets seeded by the harness, no scenario written to expose isolation.
|
|
||||||
An actor holding another's automatic marker is caught by the ordinary run.
|
|
||||||
"""
|
|
||||||
from testdriver import Oracle, Runner
|
|
||||||
from scenarios.alice_bob_carol import build
|
|
||||||
|
|
||||||
world, driver, observer, asset, oracle = build()
|
|
||||||
world.cast["bob"].remember("overheard", world.cast["alice"].canary)
|
|
||||||
result = Runner(world, driver, observer, oracle).run(asset)
|
|
||||||
pack = json.loads(result.evidence.to_json())
|
|
||||||
|
|
||||||
violations = check_isolation_was_examined(pack)
|
|
||||||
assert violations
|
|
||||||
assert "holds the private marker of 'alice'" in violations[0]
|
|
||||||
|
|
|
||||||
|
|
@ -17,7 +17,6 @@ from tests.selfverification.checks import (
|
||||||
check_actor_isolation,
|
check_actor_isolation,
|
||||||
check_evidence_supports_every_verdict,
|
check_evidence_supports_every_verdict,
|
||||||
check_intent_independence,
|
check_intent_independence,
|
||||||
check_isolation_was_examined,
|
|
||||||
check_no_actor_collected_judgment,
|
check_no_actor_collected_judgment,
|
||||||
check_runs_agree,
|
check_runs_agree,
|
||||||
check_verdicts_follow_from_judgment_evidence,
|
check_verdicts_follow_from_judgment_evidence,
|
||||||
|
|
@ -57,14 +56,6 @@ def test_actor_isolation():
|
||||||
assert check_actor_isolation(pack, SECRETS, memories) == []
|
assert check_actor_isolation(pack, SECRETS, memories) == []
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("mutations", [(), ("M15",), ("M01",)])
|
|
||||||
def test_every_run_examines_isolation(mutations):
|
|
||||||
"""F-0003 resolved: isolation is now a property of runs, not of scenarios
|
|
||||||
written to expose it. No canaries need be planted by the test."""
|
|
||||||
pack, _, _ = run_and_serialize(*mutations)
|
|
||||||
assert check_isolation_was_examined(pack) == []
|
|
||||||
|
|
||||||
|
|
||||||
# --- td://self/oracle-independence ---------------------------------------
|
# --- td://self/oracle-independence ---------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,147 +0,0 @@
|
||||||
"""The audit-core use case is durable intent, even before drivers can run it."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from copy import deepcopy
|
|
||||||
|
|
||||||
from testdriver import Oracle, Provenance, Verdict
|
|
||||||
from usecases.audit_core_e2_tenant_boundary import (
|
|
||||||
PHASE_CONTRACTS,
|
|
||||||
PRECEDENT_EVIDENCE_REF,
|
|
||||||
ROLE_CONTRACTS,
|
|
||||||
TEST_USE_CASE,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def passing_observations() -> dict[str, object]:
|
|
||||||
absent = {
|
|
||||||
"status": 404,
|
|
||||||
"schema": ("$", "$.error:str"),
|
|
||||||
"digest": "absent-surface",
|
|
||||||
"fixture_match_count": 0,
|
|
||||||
}
|
|
||||||
return {
|
|
||||||
"event_by_id": {
|
|
||||||
"owner": {"status": 200, "fixture_match_count": 2},
|
|
||||||
"attacker": dict(absent),
|
|
||||||
"absent": dict(absent),
|
|
||||||
},
|
|
||||||
"correlation_slice": {
|
|
||||||
"owner": {"status": 200, "fixture_match_count": 2},
|
|
||||||
"attacker": {"status": 200, "fixture_match_count": 0},
|
|
||||||
},
|
|
||||||
"append_as_b": {
|
|
||||||
"attacker": {"status": 400},
|
|
||||||
"state_after": dict(absent),
|
|
||||||
"absent": dict(absent),
|
|
||||||
},
|
|
||||||
"run": {
|
|
||||||
"attempted_operations": 10,
|
|
||||||
"maximum_operations": 30,
|
|
||||||
"max_concurrency": 1,
|
|
||||||
"rate_per_minute": 10,
|
|
||||||
"target_revision_matches": True,
|
|
||||||
"attribution_complete": True,
|
|
||||||
},
|
|
||||||
"cleanup": {
|
|
||||||
"completed_before_expiry": True,
|
|
||||||
"runner_absent": True,
|
|
||||||
"temporary_identities": [],
|
|
||||||
"kv_paths": [],
|
|
||||||
"projection_resources": [],
|
|
||||||
"mounted_secrets": [],
|
|
||||||
"target_ready": True,
|
|
||||||
"secret_values_observed": False,
|
|
||||||
},
|
|
||||||
"report": {
|
|
||||||
"bound_to_projection_receipt": True,
|
|
||||||
"bound_to_cleanup_receipt": True,
|
|
||||||
"bounded_assurance_statement": True,
|
|
||||||
"delivered_to_risk_nexus": True,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def judgments(observations):
|
|
||||||
oracle = Oracle()
|
|
||||||
assertions = (*TEST_USE_CASE.claims, *TEST_USE_CASE.invariants)
|
|
||||||
return {
|
|
||||||
assertion.id: oracle.judge(
|
|
||||||
assertion,
|
|
||||||
observations,
|
|
||||||
getattr(assertion, "after_step", None),
|
|
||||||
)
|
|
||||||
for assertion in assertions
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def test_precedent_calibration_satisfies_every_assertion():
|
|
||||||
result = judgments(passing_observations())
|
|
||||||
assert {judgment.verdict for judgment in result.values()} == {Verdict.PASS}
|
|
||||||
assert PRECEDENT_EVIDENCE_REF.endswith(
|
|
||||||
"AUDIT-WP-0008-T05-whitehat-e2-03-pass-2026-08-22.md"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def test_cross_tenant_event_disclosure_fails_the_claim():
|
|
||||||
observations = passing_observations()
|
|
||||||
observations["event_by_id"]["attacker"] = {
|
|
||||||
"status": 200,
|
|
||||||
"schema": ("$", "$.event_id:str"),
|
|
||||||
"digest": "tenant-b-event",
|
|
||||||
"fixture_match_count": 2,
|
|
||||||
}
|
|
||||||
result = judgments(observations)
|
|
||||||
assert result["c-event-id-tenant-scoped"].verdict is Verdict.FAIL
|
|
||||||
|
|
||||||
|
|
||||||
def test_missing_cleanup_evidence_is_inconclusive():
|
|
||||||
observations = passing_observations()
|
|
||||||
del observations["cleanup"]
|
|
||||||
result = judgments(observations)
|
|
||||||
assert result["c-receipt-bound-cleanup"].verdict is Verdict.INCONCLUSIVE
|
|
||||||
assert result["i-no-secret-evidence"].verdict is Verdict.INCONCLUSIVE
|
|
||||||
|
|
||||||
|
|
||||||
def test_cleanup_residue_fails_closed():
|
|
||||||
observations = deepcopy(passing_observations())
|
|
||||||
observations["cleanup"]["temporary_identities"] = ["unexpected-residue"]
|
|
||||||
result = judgments(observations)
|
|
||||||
assert result["c-receipt-bound-cleanup"].verdict is Verdict.FAIL
|
|
||||||
|
|
||||||
|
|
||||||
def test_intent_has_independent_provenance_and_separate_roles():
|
|
||||||
assert TEST_USE_CASE.provenance is Provenance.SPEC
|
|
||||||
assert all(
|
|
||||||
assertion.provenance is Provenance.SPEC
|
|
||||||
for assertion in (*TEST_USE_CASE.claims, *TEST_USE_CASE.invariants)
|
|
||||||
)
|
|
||||||
roles = {role.id for role in ROLE_CONTRACTS}
|
|
||||||
assert {
|
|
||||||
"authorizer",
|
|
||||||
"target-owner",
|
|
||||||
"credential-custodian",
|
|
||||||
"security-coordinator",
|
|
||||||
"cluster-executor",
|
|
||||||
"tenant-a-attacker",
|
|
||||||
"tenant-b-control",
|
|
||||||
"independent-observer",
|
|
||||||
} == roles
|
|
||||||
|
|
||||||
|
|
||||||
def test_schedule_requires_cleanup_before_finalization():
|
|
||||||
phases = {phase.id: phase for phase in PHASE_CONTRACTS}
|
|
||||||
assert phases["finalize-and-deliver"].requires == ("verify-cleanup",)
|
|
||||||
assert phases["verify-cleanup"].requires == ("cleanup-custody",)
|
|
||||||
assert phases["cleanup-custody"].requires == ("delete-runner",)
|
|
||||||
assert phases["run-probes"].requires == ("ready-runner",)
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_role_participates_in_a_phase():
|
|
||||||
declared = {role.id for role in ROLE_CONTRACTS}
|
|
||||||
scheduled = {
|
|
||||||
role
|
|
||||||
for phase in PHASE_CONTRACTS
|
|
||||||
for role in phase.driver_roles
|
|
||||||
}
|
|
||||||
assert scheduled == declared
|
|
||||||
|
|
@ -194,98 +194,3 @@ def test_safe_to_accept_is_a_closed_set():
|
||||||
assert SAFE_TO_ACCEPT == {
|
assert SAFE_TO_ACCEPT == {
|
||||||
Classification.UNCHANGED, Classification.MECHANICAL_ADAPTATION,
|
Classification.UNCHANGED, Classification.MECHANICAL_ADAPTATION,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
# --- classifications that no lab mutation happens to produce ---------------
|
|
||||||
#
|
|
||||||
# Two outcomes were declared at T08 and exercised by nothing. Left that way they
|
|
||||||
# are decoration: code that has never run is code nobody has checked. Rather than
|
|
||||||
# delete meaningful outcomes or trust them untested, both are given a case.
|
|
||||||
|
|
||||||
|
|
||||||
def test_intent_change_is_detected_when_the_claim_set_moves(baseline):
|
|
||||||
"""`INTENT_CHANGED` is a fact about the recorded use case, not an inference.
|
|
||||||
|
|
||||||
It fires because a human edited what is being asserted — which is why it is
|
|
||||||
detectable at all, where `SEMANTIC_CHANGE` was not (F-0006).
|
|
||||||
"""
|
|
||||||
import copy
|
|
||||||
|
|
||||||
altered = copy.deepcopy(baseline)
|
|
||||||
altered["provenance_index"]["c-newly-added-claim"] = "human"
|
|
||||||
outcome = classify(baseline, altered)
|
|
||||||
assert outcome.classification is Classification.INTENT_CHANGED
|
|
||||||
assert not outcome.safe_to_accept
|
|
||||||
|
|
||||||
|
|
||||||
def test_realization_failure_is_distinguishable_from_ambiguity():
|
|
||||||
"""`REALIZATION_FAILED` says "we could not act"; `AMBIGUOUS` says "we do not know".
|
|
||||||
|
|
||||||
Every lab mutation that breaks realization also strands a claim, so the
|
|
||||||
catalogue only ever produces `AMBIGUOUS`. This builds the case the catalogue
|
|
||||||
cannot: a step that fails while every assertion in the run still holds and
|
|
||||||
none of them depended on it.
|
|
||||||
|
|
||||||
Note that a run asserting *nothing at all* is `AMBIGUOUS`, not
|
|
||||||
`REALIZATION_FAILED` — a use case with no claims cannot conclude anything,
|
|
||||||
however well its steps ran.
|
|
||||||
"""
|
|
||||||
from testdriver import (
|
|
||||||
Actor, Cast, Invariant, Oracle, Runner, Scenario, SemanticAction,
|
|
||||||
StateObserver, Step, UseCase, VerificationAsset, World,
|
|
||||||
)
|
|
||||||
from testdriver.agentic import DiscoveryRuntime
|
|
||||||
from testdriver.browser import BrowserDriver
|
|
||||||
from testdriver.observers import Watch
|
|
||||||
from testdriver.provenance import Provenance
|
|
||||||
from lab.mutations import ObservationChannel
|
|
||||||
|
|
||||||
use_case = UseCase(
|
|
||||||
"uc-audit-only", "Sharing leaves an ordered audit trail",
|
|
||||||
"Alice shares R with Bob; the audit trail stays ordered.",
|
|
||||||
Provenance.HUMAN,
|
|
||||||
invariants=(Invariant(
|
|
||||||
"i-audit-ordered", "The audit trail is append-only", Provenance.HUMAN,
|
|
||||||
lambda obs: [e["sequence"] for e in obs["audit:R"]]
|
|
||||||
== sorted(e["sequence"] for e in obs["audit:R"]),
|
|
||||||
),),
|
|
||||||
)
|
|
||||||
|
|
||||||
def run(*mutations):
|
|
||||||
with journey_lab_server(*mutations) as (app, tokens, base_url):
|
|
||||||
app.request(tokens["alice"], "create_resource",
|
|
||||||
resource_id="R", content="x")
|
|
||||||
cast = Cast()
|
|
||||||
cast.add(Actor("alice", "Alice", credentials={"token": tokens["alice"]}))
|
|
||||||
scenario = Scenario(
|
|
||||||
"sc-audit-only", use_case,
|
|
||||||
watches=(Watch("bob", "R"),),
|
|
||||||
steps=(Step("s1", "alice", SemanticAction(
|
|
||||||
"grant_access", {"subject_id": "bob", "permission": "READ"},
|
|
||||||
permitted_surfaces=frozenset({"browser"}),
|
|
||||||
)),),
|
|
||||||
)
|
|
||||||
driver = BrowserDriver(base_url, tokens, DiscoveryRuntime(), "R")
|
|
||||||
observer = StateObserver(ObservationChannel(app), scenario.watches)
|
|
||||||
world = World("w-audit", app, app.version, cast=cast)
|
|
||||||
return json.loads(
|
|
||||||
Runner(world, driver, observer, Oracle())
|
|
||||||
.run(VerificationAsset("va-audit-only", scenario))
|
|
||||||
.evidence.to_json()
|
|
||||||
)
|
|
||||||
|
|
||||||
outcome = classify(run(), run("M23")) # the control is gone from the UI
|
|
||||||
assert outcome.classification is Classification.REALIZATION_FAILED
|
|
||||||
assert not outcome.safe_to_accept
|
|
||||||
|
|
||||||
|
|
||||||
def test_no_classification_is_unreachable():
|
|
||||||
"""Every declared outcome must be produced somewhere in this suite.
|
|
||||||
|
|
||||||
An outcome nothing can emit is the same kind of dead promise `SUSPICIOUS`
|
|
||||||
was before T10 removed it.
|
|
||||||
"""
|
|
||||||
exercised = set(EXPECTED.values()) | {
|
|
||||||
Classification.INTENT_CHANGED, Classification.REALIZATION_FAILED,
|
|
||||||
}
|
|
||||||
assert exercised == set(Classification)
|
|
||||||
|
|
|
||||||
|
|
@ -67,18 +67,4 @@ def test_actors_hold_isolated_credentials_and_memory():
|
||||||
assert alice.credentials["token"] != bob.credentials["token"]
|
assert alice.credentials["token"] != bob.credentials["token"]
|
||||||
alice.remember("secret", "only alice knows this")
|
alice.remember("secret", "only alice knows this")
|
||||||
assert bob.recall("secret") is None
|
assert bob.recall("secret") is None
|
||||||
# Every actor carries its own automatic private marker (F-0003) and nothing
|
assert bob.known_keys() == ()
|
||||||
# else it was not given.
|
|
||||||
assert bob.known_keys() == ("__canary__",)
|
|
||||||
assert alice.canary != bob.canary
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_run_records_a_verdict_on_isolation():
|
|
||||||
"""F-0003: isolation is examined on every scenario, not only on ones
|
|
||||||
written to expose it."""
|
|
||||||
result, _ = run_once()
|
|
||||||
examined = [
|
|
||||||
obs for obs in result.evidence.observations if obs.kind == "actor_isolation"
|
|
||||||
]
|
|
||||||
assert len(examined) == 1
|
|
||||||
assert examined[0].data["violations"] == []
|
|
||||||
|
|
|
||||||
|
|
@ -1,5 +0,0 @@
|
||||||
"""Durable test intent, kept separate from concrete scenario realizations."""
|
|
||||||
|
|
||||||
from .audit_core_e2_tenant_boundary import TEST_USE_CASE
|
|
||||||
|
|
||||||
__all__ = ["TEST_USE_CASE"]
|
|
||||||
|
|
@ -1,342 +0,0 @@
|
||||||
"""Audit-core E2 tenant-boundary TestUseCase.
|
|
||||||
|
|
||||||
This captures the intent and role hand-offs exercised successfully by
|
|
||||||
WH-ENG-20260822-AUDIT-E2-03. It is deliberately not a runnable ``Scenario``
|
|
||||||
yet. The current kernel has no causal/time-window schedule, external custody
|
|
||||||
driver, Kubernetes driver, or independent cleanup observer capable of realizing
|
|
||||||
this use case without an attended operator.
|
|
||||||
|
|
||||||
The claims come from the approved engagement and AUDIT-WP-0008-T05, not from
|
|
||||||
reverse-engineering the successful responses. The -03 report is precedent
|
|
||||||
evidence and a calibration example; it is not the source of truth for verdicts.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from typing import Mapping
|
|
||||||
|
|
||||||
from testdriver import Claim, Invariant, Provenance, UseCase
|
|
||||||
|
|
||||||
|
|
||||||
SPEC_REF = (
|
|
||||||
"audit-core/workplans/AUDIT-WP-0008-tenancy-posture-alignment.md"
|
|
||||||
"#AUDIT-WP-0008-T05"
|
|
||||||
)
|
|
||||||
ENGAGEMENT_REF = (
|
|
||||||
"whitehat-security/engagements/2026-08-22-audit-core-e2-03.json"
|
|
||||||
)
|
|
||||||
PRECEDENT_EVIDENCE_REF = (
|
|
||||||
"audit-core/docs/evidence/"
|
|
||||||
"AUDIT-WP-0008-T05-whitehat-e2-03-pass-2026-08-22.md"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True, slots=True)
|
|
||||||
class RoleContract:
|
|
||||||
"""One independently driven responsibility in the attended protocol."""
|
|
||||||
|
|
||||||
id: str
|
|
||||||
responsibility: str
|
|
||||||
may_observe_secret_values: bool = False
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True, slots=True)
|
|
||||||
class PhaseContract:
|
|
||||||
"""Causal intent for the future orchestrator; never a shell transcript."""
|
|
||||||
|
|
||||||
id: str
|
|
||||||
driver_roles: tuple[str, ...]
|
|
||||||
requires: tuple[str, ...]
|
|
||||||
completion: str
|
|
||||||
timing: str
|
|
||||||
|
|
||||||
|
|
||||||
ROLE_CONTRACTS = (
|
|
||||||
RoleContract(
|
|
||||||
"authorizer",
|
|
||||||
"Approve the exact production package, target, scope, window and cleanup "
|
|
||||||
"obligation; hold no test credential.",
|
|
||||||
),
|
|
||||||
RoleContract(
|
|
||||||
"target-owner",
|
|
||||||
"Acknowledge the exact target revision, routes, fixtures and abort limits.",
|
|
||||||
),
|
|
||||||
RoleContract(
|
|
||||||
"credential-custodian",
|
|
||||||
"Preflight, project and revoke two short-lived tenant-scoped identities; "
|
|
||||||
"emit value-safe projection and cleanup receipts.",
|
|
||||||
may_observe_secret_values=True,
|
|
||||||
),
|
|
||||||
RoleContract(
|
|
||||||
"security-coordinator",
|
|
||||||
"Validate the engagement, admit only the receipt-bound plane lease, invoke "
|
|
||||||
"the bounded probe, finalize the report and route it to risk-nexus.",
|
|
||||||
),
|
|
||||||
RoleContract(
|
|
||||||
"cluster-executor",
|
|
||||||
"Create, wait for and delete only the approved runner pod; mount but never "
|
|
||||||
"read the projected Secret through the control plane.",
|
|
||||||
),
|
|
||||||
RoleContract(
|
|
||||||
"tenant-a-attacker",
|
|
||||||
"Use an ordinary tenant-A identity to attempt the declared cross-tenant "
|
|
||||||
"reads and write.",
|
|
||||||
may_observe_secret_values=True,
|
|
||||||
),
|
|
||||||
RoleContract(
|
|
||||||
"tenant-b-control",
|
|
||||||
"Use an independently scoped tenant-B identity only for known-good owner "
|
|
||||||
"controls and state-after checks.",
|
|
||||||
may_observe_secret_values=True,
|
|
||||||
),
|
|
||||||
RoleContract(
|
|
||||||
"independent-observer",
|
|
||||||
"Collect sanitized outcomes, target readiness and post-cleanup absence "
|
|
||||||
"without accepting an actor's report as its own evidence.",
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
PHASE_CONTRACTS = (
|
|
||||||
PhaseContract(
|
|
||||||
"authorize",
|
|
||||||
("authorizer",),
|
|
||||||
(),
|
|
||||||
"Exact engagement and custody contracts are approved.",
|
|
||||||
"Before projection and before the engagement window opens.",
|
|
||||||
),
|
|
||||||
PhaseContract(
|
|
||||||
"acknowledge-target",
|
|
||||||
("target-owner",),
|
|
||||||
("authorize",),
|
|
||||||
"Target owner accepts the exact revision, routes and bounded fixtures.",
|
|
||||||
"Before any temporary credential exists.",
|
|
||||||
),
|
|
||||||
PhaseContract(
|
|
||||||
"preflight",
|
|
||||||
("credential-custodian",),
|
|
||||||
("acknowledge-target",),
|
|
||||||
"Target is ready and all engagement-specific resources are absent.",
|
|
||||||
"Immediately before the projection cutoff.",
|
|
||||||
),
|
|
||||||
PhaseContract(
|
|
||||||
"project-custody",
|
|
||||||
("credential-custodian",),
|
|
||||||
("preflight",),
|
|
||||||
"Exactly two scoped handles exist and a value-safe receipt is emitted.",
|
|
||||||
"Inside the narrow projection window; expiry equals engagement end.",
|
|
||||||
),
|
|
||||||
PhaseContract(
|
|
||||||
"admit-plane",
|
|
||||||
("security-coordinator",),
|
|
||||||
("project-custody",),
|
|
||||||
"Receipt, broker contract, target revision and cleanup authority agree.",
|
|
||||||
"After projection and before any target packet.",
|
|
||||||
),
|
|
||||||
PhaseContract(
|
|
||||||
"ready-runner",
|
|
||||||
("cluster-executor",),
|
|
||||||
("admit-plane",),
|
|
||||||
"The exact attributed runner is Ready with its Secret mounted.",
|
|
||||||
"Inside the authorized engagement window.",
|
|
||||||
),
|
|
||||||
PhaseContract(
|
|
||||||
"run-probes",
|
|
||||||
(
|
|
||||||
"security-coordinator",
|
|
||||||
"tenant-a-attacker",
|
|
||||||
"tenant-b-control",
|
|
||||||
),
|
|
||||||
("ready-runner",),
|
|
||||||
"Calibrated owner, attacker and absent controls produce a sanitized report.",
|
|
||||||
"Before credential and engagement expiry.",
|
|
||||||
),
|
|
||||||
PhaseContract(
|
|
||||||
"delete-runner",
|
|
||||||
("cluster-executor",),
|
|
||||||
("run-probes",),
|
|
||||||
"The exact runner pod is absent.",
|
|
||||||
"Immediately after evidence collection.",
|
|
||||||
),
|
|
||||||
PhaseContract(
|
|
||||||
"cleanup-custody",
|
|
||||||
("credential-custodian",),
|
|
||||||
("delete-runner",),
|
|
||||||
"Receipt-bound cleanup revokes both identities and removes exact resources.",
|
|
||||||
"Before credential and engagement expiry.",
|
|
||||||
),
|
|
||||||
PhaseContract(
|
|
||||||
"verify-cleanup",
|
|
||||||
("independent-observer",),
|
|
||||||
("cleanup-custody",),
|
|
||||||
"Both identities, exact KV paths, projection resources, Secret and runner "
|
|
||||||
"are independently absent while the target remains Ready.",
|
|
||||||
"After cleanup and before credential and engagement expiry.",
|
|
||||||
),
|
|
||||||
PhaseContract(
|
|
||||||
"finalize-and-deliver",
|
|
||||||
("security-coordinator",),
|
|
||||||
("verify-cleanup",),
|
|
||||||
"The report is bound to projection and cleanup receipts and reaches risk-nexus.",
|
|
||||||
"Only after cleanup is independently observable.",
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _same_absence_surface(left: Mapping[str, object], right: Mapping[str, object]) -> bool:
|
|
||||||
"""Compare only the sanitized non-disclosure surface."""
|
|
||||||
|
|
||||||
return (
|
|
||||||
left["status"] == right["status"]
|
|
||||||
and left["schema"] == right["schema"]
|
|
||||||
and left["digest"] == right["digest"]
|
|
||||||
and left["fixture_match_count"] == 0
|
|
||||||
and right["fixture_match_count"] == 0
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _event_id_is_tenant_scoped(obs: Mapping[str, object]) -> bool:
|
|
||||||
probe = obs["event_by_id"]
|
|
||||||
owner = probe["owner"]
|
|
||||||
attacker = probe["attacker"]
|
|
||||||
absent = probe["absent"]
|
|
||||||
return (
|
|
||||||
owner["status"] == 200
|
|
||||||
and owner["fixture_match_count"] > 0
|
|
||||||
and _same_absence_surface(attacker, absent)
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _correlation_slice_is_tenant_scoped(obs: Mapping[str, object]) -> bool:
|
|
||||||
probe = obs["correlation_slice"]
|
|
||||||
return (
|
|
||||||
probe["owner"]["status"] == 200
|
|
||||||
and probe["owner"]["fixture_match_count"] > 0
|
|
||||||
and probe["attacker"]["status"] == 200
|
|
||||||
and probe["attacker"]["fixture_match_count"] == 0
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _cross_tenant_append_is_refused(obs: Mapping[str, object]) -> bool:
|
|
||||||
probe = obs["append_as_b"]
|
|
||||||
return (
|
|
||||||
probe["attacker"]["status"] in {400, 403, 404}
|
|
||||||
and _same_absence_surface(probe["state_after"], probe["absent"])
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _cleanup_is_complete(obs: Mapping[str, object]) -> bool:
|
|
||||||
cleanup = obs["cleanup"]
|
|
||||||
return (
|
|
||||||
cleanup["completed_before_expiry"] is True
|
|
||||||
and cleanup["runner_absent"] is True
|
|
||||||
and cleanup["temporary_identities"] == []
|
|
||||||
and cleanup["kv_paths"] == []
|
|
||||||
and cleanup["projection_resources"] == []
|
|
||||||
and cleanup["mounted_secrets"] == []
|
|
||||||
and cleanup["target_ready"] is True
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _execution_remains_bounded(obs: Mapping[str, object]) -> bool:
|
|
||||||
run = obs["run"]
|
|
||||||
return (
|
|
||||||
run["attempted_operations"] <= run["maximum_operations"]
|
|
||||||
and run["max_concurrency"] == 1
|
|
||||||
and run["rate_per_minute"] <= 10
|
|
||||||
and run["target_revision_matches"] is True
|
|
||||||
and run["attribution_complete"] is True
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _secret_values_do_not_enter_evidence(obs: Mapping[str, object]) -> bool:
|
|
||||||
return obs["cleanup"]["secret_values_observed"] is False
|
|
||||||
|
|
||||||
|
|
||||||
def _report_preserves_assurance_limit(obs: Mapping[str, object]) -> bool:
|
|
||||||
report = obs["report"]
|
|
||||||
return (
|
|
||||||
report["bound_to_projection_receipt"] is True
|
|
||||||
and report["bound_to_cleanup_receipt"] is True
|
|
||||||
and report["bounded_assurance_statement"] is True
|
|
||||||
and report["delivered_to_risk_nexus"] is True
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
TEST_USE_CASE = UseCase(
|
|
||||||
id="uc-audit-core-e2-tenant-boundary",
|
|
||||||
title="Demonstrate audit-core's E2 tenant boundary with separate drivers",
|
|
||||||
narrative=(
|
|
||||||
"Within one explicitly authorized production window, independent target, "
|
|
||||||
"custody, security, cluster, attacker, control and observer roles establish "
|
|
||||||
"that an ordinary tenant-A identity cannot read or create tenant-B audit "
|
|
||||||
"fixtures. The run must remain bounded, preserve non-disclosure semantics, "
|
|
||||||
"clean every temporary resource before expiry, and deliver a cleanup-bound "
|
|
||||||
"sanitized report without retaining credential values."
|
|
||||||
),
|
|
||||||
provenance=Provenance.SPEC,
|
|
||||||
source_ref=SPEC_REF,
|
|
||||||
claims=(
|
|
||||||
Claim(
|
|
||||||
"c-event-id-tenant-scoped",
|
|
||||||
"Tenant A cannot distinguish tenant B's event id from an absent event",
|
|
||||||
Provenance.SPEC,
|
|
||||||
_event_id_is_tenant_scoped,
|
|
||||||
after_step="run-probes",
|
|
||||||
source_ref=ENGAGEMENT_REF,
|
|
||||||
),
|
|
||||||
Claim(
|
|
||||||
"c-correlation-slice-tenant-scoped",
|
|
||||||
"Tenant A's correlation slice contains no tenant-B fixture",
|
|
||||||
Provenance.SPEC,
|
|
||||||
_correlation_slice_is_tenant_scoped,
|
|
||||||
after_step="run-probes",
|
|
||||||
source_ref=ENGAGEMENT_REF,
|
|
||||||
),
|
|
||||||
Claim(
|
|
||||||
"c-cross-tenant-append-refused",
|
|
||||||
"Tenant A cannot append an audit event attributed to tenant B",
|
|
||||||
Provenance.SPEC,
|
|
||||||
_cross_tenant_append_is_refused,
|
|
||||||
after_step="run-probes",
|
|
||||||
source_ref=ENGAGEMENT_REF,
|
|
||||||
),
|
|
||||||
Claim(
|
|
||||||
"c-receipt-bound-cleanup",
|
|
||||||
"Every engagement-specific runtime and custody resource is absent "
|
|
||||||
"before expiry while the target remains ready",
|
|
||||||
Provenance.SPEC,
|
|
||||||
_cleanup_is_complete,
|
|
||||||
after_step="verify-cleanup",
|
|
||||||
source_ref=SPEC_REF,
|
|
||||||
),
|
|
||||||
),
|
|
||||||
invariants=(
|
|
||||||
Invariant(
|
|
||||||
"i-bounded-execution",
|
|
||||||
"The run stays within its operation, rate, concurrency, revision and "
|
|
||||||
"attribution boundaries",
|
|
||||||
Provenance.SPEC,
|
|
||||||
_execution_remains_bounded,
|
|
||||||
source_ref=ENGAGEMENT_REF,
|
|
||||||
),
|
|
||||||
Invariant(
|
|
||||||
"i-no-secret-evidence",
|
|
||||||
"Credential values never enter retained observations or reports",
|
|
||||||
Provenance.SPEC,
|
|
||||||
_secret_values_do_not_enter_evidence,
|
|
||||||
source_ref=SPEC_REF,
|
|
||||||
),
|
|
||||||
Invariant(
|
|
||||||
"i-bounded-assurance-report",
|
|
||||||
"Delivery is receipt-bound and says only that attempted attacks failed",
|
|
||||||
Provenance.SPEC,
|
|
||||||
_report_preserves_assurance_limit,
|
|
||||||
source_ref=SPEC_REF,
|
|
||||||
),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
# Alias retained for the existing scenario-module convention.
|
|
||||||
USE_CASE = TEST_USE_CASE
|
|
||||||
|
|
@ -4,11 +4,11 @@ type: workplan
|
||||||
title: "Vertical spike: falsify the crystallization thesis"
|
title: "Vertical spike: falsify the crystallization thesis"
|
||||||
domain: infotech
|
domain: infotech
|
||||||
repo: test-driver
|
repo: test-driver
|
||||||
status: finished
|
status: active
|
||||||
owner: codex
|
owner: codex
|
||||||
topic_slug: custodian
|
topic_slug: custodian
|
||||||
created: "2026-08-22"
|
created: "2026-08-22"
|
||||||
updated: "2026-08-23"
|
updated: "2026-08-22"
|
||||||
state_hub_workstream_id: "68b252b8-03e7-55c7-8dec-ac13f6ca0aa1"
|
state_hub_workstream_id: "68b252b8-03e7-55c7-8dec-ac13f6ca0aa1"
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -461,7 +461,7 @@ to execute than the agentic ancestor.
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: TD-WP-0002-T10
|
id: TD-WP-0002-T10
|
||||||
status: done
|
status: todo
|
||||||
priority: medium
|
priority: medium
|
||||||
state_hub_task_id: "884c2e91-b56d-5b5e-a2d3-52d2a2768cae"
|
state_hub_task_id: "884c2e91-b56d-5b5e-a2d3-52d2a2768cae"
|
||||||
```
|
```
|
||||||
|
|
@ -478,42 +478,3 @@ retaining them, and update the Concept ↔ Implementation Fitness Map.
|
||||||
Convert at least one framework finding discovered during T04–T09 into a permanent
|
Convert at least one framework finding discovered during T04–T09 into a permanent
|
||||||
self-verification or deterministic regression — that closes the first genuine
|
self-verification or deterministic regression — that closes the first genuine
|
||||||
Concept–Implementation Fitness Loop.
|
Concept–Implementation Fitness Loop.
|
||||||
|
|
||||||
**Done 2026-08-23.** Full write-up:
|
|
||||||
`history/2026-08-23-td-wp-0002-gate-review.md`. 178 tests pass.
|
|
||||||
|
|
||||||
**All four gate criteria met.** M01/M02 absorbed with the claim set unchanged;
|
|
||||||
the authorization defects escalated with regressions named; one asset
|
|
||||||
crystallized and running with no model; **False Adaptation Rate 0/7**, with 12 of
|
|
||||||
13 mechanical mutations absorbed so the safety result is not bought by escalating
|
|
||||||
everything.
|
|
||||||
|
|
||||||
**Fitness loop closed via F-0003.** Actor isolation left no trace in ordinary
|
|
||||||
evidence — it was a property of scenarios written to expose it. Every actor is
|
|
||||||
now seeded with an automatic private marker and the runner examines all actors on
|
|
||||||
every scenario, with two permanent regressions behind it, one of which catches a
|
|
||||||
leak with the harness planting nothing.
|
|
||||||
|
|
||||||
**Compression — six abstractions removed**, each declared and never used:
|
|
||||||
`Verdict.SUSPICIOUS` (a verdict no oracle could emit), `Step.expect_refusal`,
|
|
||||||
`ActorIsolationError`, `World.seed`, `EvidencePack.latest()`,
|
|
||||||
`Trajectory.method`.
|
|
||||||
|
|
||||||
**F-0008 — Temperature may be redundant.** Crystallization was built end to end
|
|
||||||
without it ever being consulted; what triggered freezing was *measured stability
|
|
||||||
of realization*, which is observed rather than declared. A declared temperature
|
|
||||||
is a claim nobody verifies. Not deleted — two unevidenced cases for keeping it
|
|
||||||
are recorded — but gated, along with `energy.py`: if the next workplan closes
|
|
||||||
without a decision consulting either, they go.
|
|
||||||
|
|
||||||
**Two classifications had never run.** `INTENT_CHANGED` and `REALIZATION_FAILED`
|
|
||||||
were declared at T08 and produced by no test. Both now have purpose-built cases,
|
|
||||||
and `test_no_classification_is_unreachable` fails if a seventh is added without
|
|
||||||
one.
|
|
||||||
|
|
||||||
**What this leaves.** The thesis survived contact with its own evidence in
|
|
||||||
narrowed form, and the demonstration is architectural rather than statistical.
|
|
||||||
The highest-value next step is a **bounded live-model experiment** — F-0005 and
|
|
||||||
F-0007 converge on it independently, and one experiment settles both whether a
|
|
||||||
model adds capability and whether crystallization has a product rationale. That
|
|
||||||
should come before any further framework feature.
|
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue