Require boolean judgments and intact actor isolation guards
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a0e76f-be98-7ae3-965d-e0b31290a4c4
This commit is contained in:
parent
9cdade2116
commit
0f8559f442
6 changed files with 156 additions and 2 deletions
|
|
@ -271,3 +271,24 @@ through direct and composite dispatch, driver reuse and invalid frozen lookup.
|
||||||
Decision: `e8fabf6e-3e5e-424e-9a60-152bf3dec641`.
|
Decision: `e8fabf6e-3e5e-424e-9a60-152bf3dec641`.
|
||||||
|
|
||||||
Validation: the complete suite passed **343 tests** (153.67 seconds).
|
Validation: the complete suite passed **343 tests** (153.67 seconds).
|
||||||
|
|
||||||
|
## Predicate and isolation guard integrity (2026-09-28)
|
||||||
|
|
||||||
|
Oracle predicates must return actual booleans. None, numbers, strings,
|
||||||
|
containers and objects now produce INCONCLUSIVE; their truth-value methods are
|
||||||
|
never invoked. Existing True/False results retain PASS/FAIL semantics. Predicate
|
||||||
|
results are not copied into the diagnostic, avoiding disclosure of their values.
|
||||||
|
|
||||||
|
At each existing isolation boundary, the runner now checks for shared memory
|
||||||
|
stores, missing or changed stored canaries, and invalid or duplicate actor
|
||||||
|
canaries. Removing markers can no longer make shared actor memory appear clean.
|
||||||
|
These findings use the existing abort/acceptance/crystallization gates and record
|
||||||
|
actor identities without marker or private memory values. This remains a
|
||||||
|
boundary diagnostic, not a sandbox against arbitrary or transient memory access.
|
||||||
|
|
||||||
|
Twenty regressions cover claims and invariants, non-coercible objects, real
|
||||||
|
boolean controls, invalid-result admission and crystallization, plus broken
|
||||||
|
isolation guards before and during runs. Sixteen failed before implementation;
|
||||||
|
the focused suite passed 69 tests afterward. Decision: `2ecac1c5-5254-4ce9-b092-b47c3e1283a9`.
|
||||||
|
|
||||||
|
Full-suite validation: **363 tests passed** in 153.10 seconds.
|
||||||
|
|
|
||||||
|
|
@ -11,3 +11,4 @@ file is a pointer table, not a second source of truth.
|
||||||
| TD-WP-0003 T08 safety follow-up | Run-input manifests and incomplete-run handling | `afb38ade-893b-48a2-b1b3-c55e49cc40ee` | `docs/TestDriverGeneralisationReview.md` |
|
| TD-WP-0003 T08 safety follow-up | Run-input manifests and incomplete-run handling | `afb38ade-893b-48a2-b1b3-c55e49cc40ee` | `docs/TestDriverGeneralisationReview.md` |
|
||||||
| TD-WP-0003 T08 admission follow-up | Qualified crystallization, passing baselines and intent revisions | `88e51e10-cd32-44d2-a2d6-055675b5ce04` | `docs/TestDriverGeneralisationReview.md` |
|
| TD-WP-0003 T08 admission follow-up | Qualified crystallization, passing baselines and intent revisions | `88e51e10-cd32-44d2-a2d6-055675b5ce04` | `docs/TestDriverGeneralisationReview.md` |
|
||||||
| TD-WP-0003 T08 isolation/replay follow-up | Isolation acceptance gate and step-bound frozen replay | `e8fabf6e-3e5e-424e-9a60-152bf3dec641` | `docs/TestDriverGeneralisationReview.md` |
|
| TD-WP-0003 T08 isolation/replay follow-up | Isolation acceptance gate and step-bound frozen replay | `e8fabf6e-3e5e-424e-9a60-152bf3dec641` | `docs/TestDriverGeneralisationReview.md` |
|
||||||
|
| TD-WP-0003 T08 guard-integrity follow-up | Strict boolean judgments and intact isolation guards | `2ecac1c5-5254-4ce9-b092-b47c3e1283a9` | `docs/TestDriverGeneralisationReview.md` |
|
||||||
|
|
|
||||||
|
|
@ -83,6 +83,11 @@ class Oracle:
|
||||||
step_id,
|
step_id,
|
||||||
{"reason": f"predicate raised {type(exc).__name__}: {exc}"},
|
{"reason": f"predicate raised {type(exc).__name__}: {exc}"},
|
||||||
)
|
)
|
||||||
|
if type(satisfied) is not bool:
|
||||||
|
return Judgment(
|
||||||
|
assertion.id, assertion.text, Verdict.INCONCLUSIVE, step_id,
|
||||||
|
{"reason": "predicate did not return a boolean"},
|
||||||
|
)
|
||||||
return Judgment(
|
return Judgment(
|
||||||
assertion.id,
|
assertion.id,
|
||||||
assertion.text,
|
assertion.text,
|
||||||
|
|
|
||||||
|
|
@ -57,12 +57,32 @@ class Runner:
|
||||||
# -- independence guards ---------------------------------------------
|
# -- independence guards ---------------------------------------------
|
||||||
|
|
||||||
def _isolation_violations(self) -> list[str]:
|
def _isolation_violations(self) -> list[str]:
|
||||||
"""Does any actor hold another's canary?
|
"""Check separate stores, intact unique markers and cross-actor leaks.
|
||||||
|
|
||||||
Run on every scenario, not only on ones written to test isolation.
|
Run on every scenario, not only on ones written to test isolation.
|
||||||
"""
|
"""
|
||||||
canaries = {actor.canary: actor.id for actor in self._world.cast}
|
canaries: dict[str, str] = {}
|
||||||
|
stores: dict[int, str] = {}
|
||||||
violations: list[str] = []
|
violations: list[str] = []
|
||||||
|
for actor in self._world.cast:
|
||||||
|
store = id(actor._memory)
|
||||||
|
if store in stores:
|
||||||
|
violations.append(
|
||||||
|
f"actor {actor.id!r} shares a memory store with {stores[store]!r}"
|
||||||
|
)
|
||||||
|
stores[store] = actor.id
|
||||||
|
if not isinstance(actor.canary, str) or not actor.canary:
|
||||||
|
violations.append(f"actor {actor.id!r} has an invalid private marker")
|
||||||
|
continue
|
||||||
|
if actor.canary in canaries:
|
||||||
|
violations.append(
|
||||||
|
f"actor {actor.id!r} has a duplicate private marker"
|
||||||
|
)
|
||||||
|
canaries[actor.canary] = actor.id
|
||||||
|
if actor.recall("__canary__") != actor.canary:
|
||||||
|
violations.append(
|
||||||
|
f"actor {actor.id!r} has a missing or changed private marker"
|
||||||
|
)
|
||||||
for actor in self._world.cast:
|
for actor in self._world.cast:
|
||||||
for key in actor.known_keys():
|
for key in actor.known_keys():
|
||||||
value = actor.recall(key)
|
value = actor.recall(key)
|
||||||
|
|
|
||||||
94
tests/test_guard_integrity.py
Normal file
94
tests/test_guard_integrity.py
Normal file
|
|
@ -0,0 +1,94 @@
|
||||||
|
"""Reject invalid predicate results and broken actor isolation instrumentation."""
|
||||||
|
from dataclasses import replace
|
||||||
|
import json
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from scenarios.alice_bob_carol import build
|
||||||
|
from testdriver import Runner, Verdict
|
||||||
|
from testdriver.classification import classify
|
||||||
|
from testdriver.crystallization import assess_stability
|
||||||
|
from testdriver.oracles import Oracle
|
||||||
|
|
||||||
|
|
||||||
|
class CannotCoerce:
|
||||||
|
def __bool__(self):
|
||||||
|
raise AssertionError('oracle must not coerce predicate results')
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('value', [None, 'unknown', {}, [], 0, 1, CannotCoerce()])
|
||||||
|
def test_non_boolean_predicate_results_are_inconclusive(value):
|
||||||
|
_, _, _, asset, _ = build()
|
||||||
|
for assertion in (*asset.scenario.use_case.claims, *asset.scenario.use_case.invariants):
|
||||||
|
assertion = replace(assertion, predicate=lambda snapshot: value)
|
||||||
|
judgment = Oracle().judge(assertion, {'observed': True}, 'step')
|
||||||
|
assert judgment.verdict is Verdict.INCONCLUSIVE
|
||||||
|
assert 'boolean' in judgment.detail['reason']
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('value,verdict', [(True, Verdict.PASS), (False, Verdict.FAIL)])
|
||||||
|
def test_boolean_predicate_results_keep_their_meaning(value, verdict):
|
||||||
|
_, _, _, asset, _ = build()
|
||||||
|
assertion = replace(asset.scenario.use_case.claims[0], predicate=lambda snapshot: value)
|
||||||
|
assert Oracle().judge(assertion, {'observed': True}, 'step').verdict is verdict
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('value', [None, 'unknown', 1])
|
||||||
|
def test_invalid_predicates_cannot_authorize_acceptance_or_crystallization(value):
|
||||||
|
packs = []
|
||||||
|
for _ in range(3):
|
||||||
|
world, driver, observer, asset, oracle = build()
|
||||||
|
case = asset.scenario.use_case
|
||||||
|
asset.scenario = replace(asset.scenario, use_case=replace(case, claims=tuple(
|
||||||
|
replace(c, predicate=lambda snapshot: value) for c in case.claims)))
|
||||||
|
result = Runner(world, driver, observer, oracle).run(asset)
|
||||||
|
assert result.verdict is Verdict.INCONCLUSIVE
|
||||||
|
packs.append(json.loads(result.evidence.to_json()))
|
||||||
|
assert not classify(packs[0], packs[1]).safe_to_accept
|
||||||
|
assert not assess_stability(packs).stable
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('damage', ['shared', 'missing', 'changed', 'duplicate'])
|
||||||
|
@pytest.mark.parametrize('during_run', [False, True])
|
||||||
|
def test_corrupt_isolation_guards_stop_execution(damage, during_run):
|
||||||
|
packs = []
|
||||||
|
for _ in range(3):
|
||||||
|
world, driver, observer, asset, oracle = build()
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def corrupt():
|
||||||
|
alice, bob = world.cast['alice'], world.cast['bob']
|
||||||
|
if damage == 'shared':
|
||||||
|
alice._memory = bob._memory = {}
|
||||||
|
alice.remember('private', 'private-value')
|
||||||
|
assert bob.recall('private') == 'private-value'
|
||||||
|
elif damage == 'missing':
|
||||||
|
bob._memory.pop('__canary__')
|
||||||
|
elif damage == 'changed':
|
||||||
|
bob.remember('__canary__', 'replacement')
|
||||||
|
else:
|
||||||
|
bob.canary = alice.canary
|
||||||
|
bob.remember('__canary__', alice.canary)
|
||||||
|
|
||||||
|
class Driver:
|
||||||
|
def realize(self, actor, action):
|
||||||
|
calls.append(action.name)
|
||||||
|
result = driver.realize(actor, action)
|
||||||
|
corrupt()
|
||||||
|
return result
|
||||||
|
|
||||||
|
if not during_run:
|
||||||
|
corrupt()
|
||||||
|
result = Runner(world, Driver(), observer, oracle).run(asset)
|
||||||
|
assert result.verdict is Verdict.INCONCLUSIVE
|
||||||
|
assert len(calls) == int(during_run)
|
||||||
|
pack = json.loads(result.evidence.to_json())
|
||||||
|
guards = [o['data']['violations'] for o in pack['observations']
|
||||||
|
if o['kind'] == 'actor_isolation']
|
||||||
|
assert guards[-1]
|
||||||
|
assert 'private-value' not in result.evidence.to_json()
|
||||||
|
if damage == 'shared':
|
||||||
|
assert any('memory store' in v for v in guards[-1])
|
||||||
|
packs.append(pack)
|
||||||
|
assert not classify(packs[0], packs[1]).safe_to_accept
|
||||||
|
assert not assess_stability(packs).stable
|
||||||
|
|
@ -353,3 +353,16 @@ evidence despite passing product verdicts, repeated action names with distinct
|
||||||
targets, reusable frozen drivers and invalid dispatch. `git diff --check` is
|
targets, reusable frozen drivers and invalid dispatch. `git diff --check` is
|
||||||
clean. Decision: `e8fabf6e-3e5e-424e-9a60-152bf3dec641`. No new task or workplan;
|
clean. Decision: `e8fabf6e-3e5e-424e-9a60-152bf3dec641`. No new task or workplan;
|
||||||
T01/T06/T07 remain waiting and this workplan remains blocked.
|
T01/T06/T07 remain waiting and this workplan remains blocked.
|
||||||
|
|
||||||
|
|
||||||
|
**2026-09-28 guard-integrity follow-up — done.** Oracle now requires actual
|
||||||
|
boolean predicate results; other values produce INCONCLUSIVE without truth-value
|
||||||
|
coercion. Actor isolation checks detect shared memory stores, missing/changed
|
||||||
|
stored markers and invalid/duplicate actor canaries, feeding the existing abort
|
||||||
|
and acceptance/crystallization gates without disclosing private values.
|
||||||
|
|
||||||
|
Validation: 20 new regressions (16 failed before implementation), 69 focused
|
||||||
|
checks passed, and the full suite passed **363 tests** in 153.10 seconds.
|
||||||
|
`git diff --check` is clean. Decision:
|
||||||
|
`2ecac1c5-5254-4ce9-b092-b47c3e1283a9`. No new task or workplan was opened;
|
||||||
|
T01/T06/T07 remain waiting and this workplan remains blocked.
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue