coulomb/test-driver
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

T06: out-of-band self-verification

Browse source 
Checks written as plain functions over a serialized Evidence Pack, outside the
framework - no Oracle, no Runner, no Verdict aggregation. 12 tests that they
hold, 12 that they can fail. All four td://self identifiers covered.

The substantive check is verdict reproducibility from S3 evidence alone,
asserted on failing runs as well as passing ones.

F-0003 (open): actor isolation leaves no trace in ordinary evidence - the
self-test catches a shared memory store only because the harness plants
per-actor canaries. Isolation is currently a property of a scenario written
to expose it, not of runs in general. The mirror-image case is noted too: a
guarantee enforced by construction cannot be verified by observing real runs,
so four green self-tests are not four equivalent proofs. Carried to T10.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 1629012@bnt-lap001
Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39
This commit is contained in:
tegwick 2026-08-22 23:38:23 +02:00
parent de25673c5d
commit 5734b280c6
12 changed files with 619 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

23
workplans/TD-WP-0002-vertical-spike-crystallization.md
View file

@ -269,7 +269,7 @@ Three results worth carrying:
```task
id: TD-WP-0002-T06
status: todo
status: done
priority: medium
state_hub_task_id: "4fb31485-8650-556c-bee4-b4e067a89d04"
```
@ -282,6 +282,27 @@ Exit: deliberately breaking actor isolation fails the out-of-band check;
allowing an actor to determine its own verdict fails oracle-independence;
each failure produces a Framework Finding.
**Done 2026-08-22.** `tests/selfverification/` — checks as plain functions over a
serialized Evidence Pack, plus 12 tests that they hold and 12 that they *can
fail*. All four `td://self/...` identifiers are covered. 72 tests pass overall.
- The strongest check turned out to be **verdict reproducibility from S3 alone**:
re-evaluating each assertion against the stored snapshots, with no actor,
driver or live system in reach, must reproduce exactly what the run reported.
It is asserted on failing runs too — a check that only works when everything is
green verifies luck, not independence.
- **F-0003 (open)** — actor isolation leaves no trace in ordinary evidence. The
self-test catches a shared memory store only because the harness plants
per-actor canaries first; without them, a run where all three actors share one
memory dict produces an Evidence Pack identical to a correct one. Isolation is
currently a property of a scenario written to expose it, not of runs in
general. Carried to T10, reassess after T07.
- The same finding notes the mirror-image problem: oracle independence is
enforced by construction, so no real run can produce the violating artefact and
the check must be exercised against a hand-built pack. **A guarantee enforced
by construction cannot be verified by observing real runs.** Four green
self-tests are not four equivalent proofs, and the suite now says so.
## Agentic realisation of one semantic action, fully instrumented
```task