test-driver/research/findings/F-0003-isolation-needs-canaries.md

id	type	class	status	discovered	discovered_by	workplan	task	resolved	resolved_in
F-0003	framework-finding	FRAMEWORK_LIMITATION	resolved	2026-08-22	TD-WP-0002-T06	TD-WP-0002	TD-WP-0002-T06	2026-08-23	TD-WP-0002-T10

F-0003 — Actor isolation is only observable if the scenario plants canaries

Observation

Writing td://self/actor-isolation out-of-band exposed something the concept model does not acknowledge: an isolation violation leaves no trace in ordinary evidence.

The self-test only detects a shared memory store because the test harness first gives each actor a distinguishable private secret and then looks for it in the wrong places. Remove that instrumentation and a run in which Alice, Bob and Carol share one memory dict produces an Evidence Pack indistinguishable from a correct one — same observations, same verdicts, same everything.

So the guarantee holds in the reference scenario because it was instrumented to be checkable, not because runs generally reveal it.

Why it matters

INTENT.md and the Concept Model treat actor isolation as a property of the framework. As implemented it is a property of a scenario that was written to expose it. Every ordinary scenario — the ones users will actually write — carries no canaries, and for those the framework's central multi-user claim is asserted rather than verified.

This matters more once actors are agentic (T07). A shared session, a cached HTTP client, or a module-level default is exactly how isolation breaks in practice, and none of those announce themselves.

A second, related limitation

td://self/oracle-independence has a mirror-image problem. The framework prevents an actor from collecting S2/S3 evidence — Runner raises CollectorIndependenceError — so no real run can produce the violating artefact. The check is therefore exercised against a hand-built pack.

That is the right architecture and a weak self-test at once: a guarantee enforced by construction cannot be verified by observing real runs. The suite now asserts both halves — that the door is locked, and that we would notice someone coming through the window — but the second half tests the check, not the system.

Worth stating plainly rather than counting four green self-tests as four equivalent proofs. They are not equivalent.

Resolution (T10)

Candidate 3 — both.

Every Actor is seeded at construction with an automatic private marker (Actor.canary), and Runner._isolation_violations examines every actor on every scenario, recording the verdict as S3 evidence. An isolation violation now leaves a trace whether or not anyone thought to look for one.

Two permanent regressions stand behind it:

• tests/test_reference_scenario.py::test_every_run_records_a_verdict_on_isolation
• tests/selfverification/test_checks_can_fail.py::test_a_leak_is_caught_without_the_test_planting_anything — an actor holding another's marker is caught by the ordinary run, with the harness planting nothing.

The second limitation stands as recorded: oracle independence is enforced by construction, so it is verified once rather than per run. That asymmetry is real and is now stated in the fitness map rather than hidden behind two equally green self-tests.

This finding closed the first Concept–Implementation Fitness Loop — see history/2026-08-23-td-wp-0002-gate-review.md § 4.

Candidate resolutions considered

1. Automatic canaries — the runtime seeds each actor with a unique private marker at construction and the isolation check runs on every Evidence Pack, making isolation continuously observable rather than specially tested. Cheap, and the obvious first move.
2. Accept and document — isolation is enforced by construction, and the self-test covers the construction rather than each run. Honest, but leaves the multi-user claim resting on code review.
3. Both — canaries for observability, plus an explicit statement that some guarantees are structural and are verified once rather than per run.

Carried to TD-WP-0002-T10, where it should be answered together with the wider question of which guarantees are structural and which are observed. Reassess after T07, when agentic actors make the failure mode concrete rather than theoretical.