From 5734b280c68cf7eba69d548a44b17be060849e95 Mon Sep 17 00:00:00 2001 From: tegwick Date: Sat, 22 Aug 2026 23:38:23 +0200 Subject: [PATCH] T06: out-of-band self-verification Checks written as plain functions over a serialized Evidence Pack, outside the framework - no Oracle, no Runner, no Verdict aggregation. 12 tests that they hold, 12 that they can fail. All four td://self identifiers covered. The substantive check is verdict reproducibility from S3 evidence alone, asserted on failing runs as well as passing ones. F-0003 (open): actor isolation leaves no trace in ordinary evidence - the self-test catches a shared memory store only because the harness plants per-actor canaries. Isolation is currently a property of a scenario written to expose it, not of runs in general. The mirror-image case is noted too: a guarantee enforced by construction cannot be verified by observing real runs, so four green self-tests are not four equivalent proofs. Carried to T10. Co-Authored-By: Claude Opus 5 Assistant: claude-code Assistant-Model: opus Assistant-Process: 1629012@bnt-lap001 Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39 --- WORK-RECORDS.md | 2 +- research/concepts/fitness-map.md | 8 +- .../F-0003-isolation-needs-canaries.md | 75 ++++++ tests/selfverification/__init__.py | 0 .../__pycache__/__init__.cpython-312.pyc | Bin 0 -> 156 bytes .../__pycache__/checks.cpython-312.pyc | Bin 0 -> 9566 bytes ...ecks_can_fail.cpython-312-pytest-7.4.4.pyc | Bin 0 -> 17844 bytes ..._verification.cpython-312-pytest-7.4.4.pyc | Bin 0 -> 14275 bytes tests/selfverification/checks.py | 219 ++++++++++++++++++ .../selfverification/test_checks_can_fail.py | 188 +++++++++++++++ .../test_self_verification.py | 110 +++++++++ ...-WP-0002-vertical-spike-crystallization.md | 23 +- 12 files changed, 619 insertions(+), 6 deletions(-) create mode 100644 research/findings/F-0003-isolation-needs-canaries.md create mode 100644 tests/selfverification/__init__.py create mode 100644 tests/selfverification/__pycache__/__init__.cpython-312.pyc create mode 100644 tests/selfverification/__pycache__/checks.cpython-312.pyc create mode 100644 tests/selfverification/__pycache__/test_checks_can_fail.cpython-312-pytest-7.4.4.pyc create mode 100644 tests/selfverification/__pycache__/test_self_verification.cpython-312-pytest-7.4.4.pyc create mode 100644 tests/selfverification/checks.py create mode 100644 tests/selfverification/test_checks_can_fail.py create mode 100644 tests/selfverification/test_self_verification.py diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 0ce0894..0b2c90f 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -17,7 +17,7 @@ | task | TD-WP-0002-T02 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T03 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T04 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | -| task | TD-WP-0002-T05 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | +| task | TD-WP-0002-T05 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T06 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T07 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T08 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | diff --git a/research/concepts/fitness-map.md b/research/concepts/fitness-map.md index 88151cf..d41bdb9 100644 --- a/research/concepts/fitness-map.md +++ b/research/concepts/fitness-map.md @@ -1,6 +1,6 @@ # Concept ↔ Implementation Fitness Map -**Updated:** 2026-08-22 (TD-WP-0002-T05) +**Updated:** 2026-08-22 (TD-WP-0002-T06) Traces each important concept to the implementation, experiment and evidence that support it. **Unsupported entries are the point of this map** — a concept with no @@ -25,15 +25,15 @@ were aspirational, not evidenced. | Concept | Level | Implementation | Experiment | Evidence | Open question | |---|---|---|---|---|---| | `C-use-case` | C1 | `intent.py` | — | — | Is a use case expressible without leaking mechanics? | -| `C-actor-isolation` | C1 | `world.py` | E-001 | — | Isolation is asserted by construction; unverified. | +| `C-actor-isolation` | C1 | `world.py` | E-001 | `td://self/actor-isolation` | **F-0003** — only observable when the scenario plants canaries. | | `C-semantic-action` | C1 | `actions.py` | E-001 | — | Does identity survive restructuring better than a recorded sequence? (H-001) | | `C-oracle-independence` | C1 | `runner.py`, `oracles.py` | E-001, E-003 | — | Independence of components ≠ independence of belief. (H-004) | -| `C-evidence-pack` | C1 | `evidence.py` | — | — | What is the minimum sufficient for replay? | +| `C-evidence-pack` | C1 | `evidence.py` | — | `td://self/evidence-reproducibility` | Verdicts are reproducible from S3 alone, on passing and failing runs. | | `C-observation-channel` | C1 | `lab/app.py` | — | — | **D-07** — required of every system under test. Adoption cost unknown. | | `C-adaptation` | C1 | — (T08) | E-001 | — | (H-002) | | `C-classification` | C1 | — (T08) | E-001, E-003 | — | Decision table is total on paper; unexercised. | | `C-crystallization` | C1 | — (T09) | E-002 | — | (H-003) | -| `C-intent-provenance` | C1 | `provenance.py` | E-003 | — | Constrains provenance, not quality. Accepted residual. | +| `C-intent-provenance` | C1 | `provenance.py` | E-003 | `td://self/intent-independence` | Constrains provenance, not quality. Accepted residual. | | `C-lineage` | C0 | — | — | — | Parent pointer only in the spike. | | `C-energy` | C0 | `energy.py`, capture only | — | — | Dormant by decision. (H-005) | | `C-temperature` | C0 | — | — | — | Deferred. No implementation planned in TD-WP-0002. | diff --git a/research/findings/F-0003-isolation-needs-canaries.md b/research/findings/F-0003-isolation-needs-canaries.md new file mode 100644 index 0000000..008f659 --- /dev/null +++ b/research/findings/F-0003-isolation-needs-canaries.md @@ -0,0 +1,75 @@ +--- +id: F-0003 +type: framework-finding +class: FRAMEWORK_LIMITATION +status: open +discovered: "2026-08-22" +discovered_by: TD-WP-0002-T06 +workplan: TD-WP-0002 +task: TD-WP-0002-T06 +carried_to: TD-WP-0002-T10 +--- + +# F-0003 — Actor isolation is only observable if the scenario plants canaries + +## Observation + +Writing `td://self/actor-isolation` out-of-band exposed something the concept +model does not acknowledge: **an isolation violation leaves no trace in ordinary +evidence.** + +The self-test only detects a shared memory store because the test harness first +gives each actor a distinguishable private secret and then looks for it in the +wrong places. Remove that instrumentation and a run in which Alice, Bob and Carol +share one memory dict produces an Evidence Pack indistinguishable from a correct +one — same observations, same verdicts, same everything. + +So the guarantee holds in the reference scenario *because it was instrumented to +be checkable*, not because runs generally reveal it. + +## Why it matters + +`INTENT.md` and the Concept Model treat actor isolation as a property of the +framework. As implemented it is a property of a *scenario that was written to +expose it*. Every ordinary scenario — the ones users will actually write — +carries no canaries, and for those the framework's central multi-user claim is +asserted rather than verified. + +This matters more once actors are agentic (T07). A shared session, a cached HTTP +client, or a module-level default is exactly how isolation breaks in practice, +and none of those announce themselves. + +## A second, related limitation + +`td://self/oracle-independence` has a mirror-image problem. The framework +*prevents* an actor from collecting S2/S3 evidence — `Runner` raises +`CollectorIndependenceError` — so no real run can produce the violating artefact. +The check is therefore exercised against a hand-built pack. + +That is the right architecture and a weak self-test at once: **a guarantee +enforced by construction cannot be verified by observing real runs.** The suite +now asserts both halves — that the door is locked, and that we would notice +someone coming through the window — but the second half tests the check, not the +system. + +Worth stating plainly rather than counting four green self-tests as four +equivalent proofs. They are not equivalent. + +## Candidate resolutions + +Not resolved now; deciding cheaply here would be guessing. + +1. **Automatic canaries** — the runtime seeds each actor with a unique private + marker at construction and the isolation check runs on every Evidence Pack, + making isolation continuously observable rather than specially tested. + Cheap, and the obvious first move. +2. **Accept and document** — isolation is enforced by construction, and the + self-test covers the construction rather than each run. Honest, but leaves the + multi-user claim resting on code review. +3. **Both** — canaries for observability, plus an explicit statement that some + guarantees are structural and are verified once rather than per run. + +Carried to `TD-WP-0002-T10`, where it should be answered together with the wider +question of which guarantees are structural and which are observed. Reassess +after T07, when agentic actors make the failure mode concrete rather than +theoretical. diff --git a/tests/selfverification/__init__.py b/tests/selfverification/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/selfverification/__pycache__/__init__.cpython-312.pyc b/tests/selfverification/__pycache__/__init__.cpython-312.pyc new file mode 100644 index 0000000000000000000000000000000000000000..0b3bffb7e06c44edbba6b2ebce24549aeeae6687 GIT binary patch literal 156 zcmX@j%ge<81QB9gSs?l`h(HIQS%4zb87dhx8U0o=6fpsLpFwJVS?g!y=cekH=NA&ryk0@&FAkgB{FKt1 YRJ$TppjnJSTnu7-WM*V!EMf+-0F8+zPyhe` literal 0 HcmV?d00001 diff --git a/tests/selfverification/__pycache__/checks.cpython-312.pyc b/tests/selfverification/__pycache__/checks.cpython-312.pyc new file mode 100644 index 0000000000000000000000000000000000000000..3c615c3a8e84ea49b9a4ffa70e4658a2dd07b4aa GIT binary patch literal 9566 zcmbVSU2qgvcJ7|(`5#H6{~r)E!eEaq2@DwXXMzmE2ph~Iz(KY=w5EHchM8%ldq(oi zjvOVssS+_2gLkutqFgIiUgW~5Vk^mpyt%S@a8+Jr5EZ7nToo#thvZENduz>8zH_^K zG?InwN@I2R^u71od(S=heBU|O|6WrQD`@l54-Ucl0wQe>`8fty(!f|OljAuk{mTT~=$M0G^8*=Vw zOZG1?fw!`mUMym{E*OudJ*1{=6Y-Q)w}wWz#Y%my+UG zR??+(Mp2BuVDQ|yVklyKT!~*XL`hdfSxKs+iY{f8WKQhvPHUO&ZgEOiGZ`f9nF>vPv!} zx~#@Cmqcl7OjpL(BK3(_mYk?7k~J^`ipAn&l!O$g;fhjLRx_F&v}Vid8S49DN=s+P z4Kazu(|g#ouzYN`O43j;odGF6ZY?&1XRfFb_Z%r8YH zHId1QDMdnE1I0!{RpdVLENhNhLD-cf!d}Kno#Qk(YwpnT6-Ajam}+eU%Nj5WMPy4M zp#-YN2-0#-Wz0s9mgrJOKCoj4t@)0=zP?MVEu7?|8S14_W0INiS!E2f&~r2oYSz1I zPtZESlvG7G4g`ZDt`+&vsHXR-hCTN+rIwxcu9#?CSyMx?t)N#|CUi~C#??_Zsb+F( zs;Ss;>C9?D!G3!!*cORPX{?r*kklk?18gMNtG#|bqJb@iIpypYP_`nDLxoZ>uIajB zOlWDDtTrxTJHuQrk#1v_+G7wMBe{JOxl6Q0=tWUv1KSGAU%lajqq=ql`@G85WP3eX z?YNd=y8)i#f64Vlc{9X@&FscF9d(=T{&ddt55fXzsZHN0X<`BciwgPnXDU74xl*)_ z#`9e(#m`cAwTN|eeb|lBKCP3+Xb}>+mg+Sol(;FdyJtL`lG1V8nKPH}#T`xfC;RYn zlgm`f3fyH9wdNxDxZ|=u-m&WraZ&eaGpvom6HHqA`HYLYOo8^B=^j%upYf(J0pDRz z3-Rgx9phR`*@1;O;^RAL{cI=B-VL@Ls?B)$Wnba}vBR=j@xwvbjk>jTM0x$FJs4cs$ zyufjK0KeVz8&83rzi?e^7EpVdAHtnyZ&O^1?rSgCv}L|& zv2pt!JVMy}C|p+#)y_^{Jy`7Mz8~sd@^FFh?BLbfIrrT0TcMJF>#`fwmN>!V{dSqd zt3hnxZ`z_lyn5Y3_{i)7Yyhen%lkeFZc?=l~Et-j#C9B?GBJY`ow zxb2eN1t0FhCmv_9dG1=bz+D>=xD3sgT?Ni5#f*N2*#ruK%Qa5v%2TdLnGP0$1%8s( z`D^VzFz@vxd7U2rpIYf#rIp~1){3tXko{!Dw>ddb2tqeOrlHU$K4(PG&`xORuLQ2( z#(ocHr(pu8HIab|C`nEn*FZ-!9WYAHHl4HJcmm`Mc2B%TA{S4Df^Y~BP3Ux#=onZ_ zW=a#Mq@2+MCKH#k@a&3eRZePYxOt$2B3-c|ts87DrKB`H2f{|kZ4IIyY|H7+Ko^Rb z$^!2RYT}>^L~X2!uc4x$#K8*=vO$nM$J;s*K`&R_nFz+EtE1TrjntWQf0cMi2VOxe z6DQijd&H@66)eP}F2pMsK}cdEAdPWp0@C2XHTG3kfNnA&Us_ZXBIaVtvL-4fHNcfD zf=Wv1K2`)o^2y*y*<>c~wI-JLu~jg{d`FcoCZ%MSA*ChC48*l$QlUlA$!q4T9k*z@ zR5@P@g|RB=aWp690{~lHSHWWQk+p)OKGUOu#2Th>+0fFaN6w}u4ATvcVtRpypu5bq z>9y!98nXOb#8N-=Wo+nXJsTXlAopf{3RY;EVS21jUV93Dome#JuIXVmWqK`vIvKIv ziJL+Dm@Le`EH~QBfW(p?t1uy*5ab{=>V;9ya7!Dx&doC?AGLQsY~NXG-?`YnYnk(U z8m`vNdS)lTYHYoE;KqTu#OLt`jk{+3<<8!Rox4k&yB9lOh93Ol zaFNcrbBlFd%UsaYFzYTiwcdRF#_Mx$EH=F`8z_gGiW_@Mp`LQM$r(!g*JHQF<`cKI ze>5I69$Y&lPxX)ji*;M(6L<$m=q^_!uv%ptTZ2PPV0HwQZ#mh)?0WlSo>1jb2D4;_$Yj8l^=`4OH?7$QT-l_t z%UO9W1;VHAA%)@3&~XDN@N>2G*V{jQZ{|2&9eUvN0QbFIO@s(S?&{&WcT1t@Lj3NF zrM|;oY%lelSn_e?B)aFvYSM49!4SX zYIxA?trz+QVUM_(IT;th;1?nwJC9p+cw-un}GQ5j~+c z`U@9Ax4BO$;{=NTIZ*tUKyg=PR|r$wRxU$p#IEN!-GlT`a5{96yLNDjd*6MLo8qH2 z+0U^|11vJLzzX1iUlh_?NQjNI5*Z@)rQk!X-eeMCxefg1#)0Yhk78gWI|_1)xOI|X z%1M#L?PLu_;Jpfg7(xrgeF{bPcI2(485Bxm0Czw=6Gq4{iw@7WLi;I@3Cd2ey~VN- zR?>+10enwV*H$tHBVaO-YCu24E|_m{W(H#NKBOS!GUa4@^lX|YjwqJW1%n9+Hf0T9 z3Fgs5W3mJq2ze|{q-8S({0Y%L7+8hJw6WL zjU%~%T$U=0At?oZAgL+DtmHIIzunj{!xeqO$LOtCXVYgFK6Pq|Px&MHO)S)gMzC&< z3W@R{?ndS$Z3<`lhlWhyc>my=W_a-Q(YH3>7X4d}Y2N5huS3|8i3 z=-XLkXTXNZPnw_z)^N-~rJ|1Q%(3a~6z!URr-kWP-ba*-1vF?MkoKI*a=ks4>va?w zOMo9~Q!z5}it~**Gt*y(j6cJlvD@MkEsRfm73rGavlw|{X0SYX>dVWe!HfS>30^g{o_*uur)-gCvK^IsDO0VdS9FS*<`-Q{h454Y_rZQFOR?ZLL!ZteZj zd)<%HNZZ^i;3ew|x_kMH)-M8}4k!8j<^BElg2Wq+@fLCD#Bi6~n8ea{i#U9{{2~{5 zhyUZ(8@Z;9%N&nBH$*@7qfJfsQ-dt@Z<~*Lgn#ul9}Nn>3x82hD+oCC0C%tsDFm+&>b6q z`De)pJ2jpuqsFNL87mnOB3!lkll>K-a2&C-Rlv58wM86cCB%_xzA$)f;Plaf7`d49 zX9vhxQ|=B92!T$Y-VDW>AvouxnrEpdYdi17XMI22*cLy4w|tA^%UI^6;?D9t&UZWJ zLLzlso3g<}nP9waLeUeNo=c^K|2g*(d)9q?&~OB;3-J9fh#zI}JF`07wG zat>~)rWTGU+*s`Bxlb|Bvs=Iu9ggBwAB77-t2*>bVN2wreOLEgKlLEgRrWVL^tYG% z?Zu8$5B#Ugp}O^oUVq>}Q4TdD*!gz(c}&Eh&lKHOFjT}p8=e{=>fIB(;B z&P}^qAX_V84*OLJg&4tf%dRo(8IWHmqsX3inLuS@9Fi`c{DE`GlfXg@i(E0jLnkH% zlH3kYA=*+Ya8TFiTje*&Lm_hGpHSpm9i-8SF%Cwyk8uuVcJ?+)^nORsZYO*E2FF8K zfg6FzvVLT3kvRhx@K5xs@N47QZE7QQZEA`{N9^%(+^E9;Hf`^&ZtUMC-eq>I;JOi!;PQRF!7=**2C z^PS&2pnsw!TW~|Bx~};#>wRGG%kI(u0dPz33FnFgv0DN`7FXx{C@r?ma>Ju!RGe>G zj!jk#Hg{P+3DNm()?kW`O)?TY;;`AVtR@opinB-CO&OM&-KHm@>d4en5(S4V`Z*dV z%$R(JzH;PwO`2RVOe_T``DZw$?~{NkPaKCv7*QmpGQ`uoei z@a#y*w_$E_K2_Sh=e`dakweA0SBw5v%b}W?K`W0y+H|f8v_HZZ7Blhyxat~qO_^7@ zO6C|F%`x7QpV*+zHWT8*b+gt5!7*+E^O+VvB+Y!yg)9T*jXj^Zo!YS9myo@t4bLtH z-Uq(7HR4jpSRDZl0k*NdqkhSYsUe?gyIz)2wMjQ632D`k@Isg(uWWr4M^p-h{R{~0 zz-q8MfC992(@7TMu5`xeu7oAx*mq(rsyd1jaV62+C-w&kE}XF-Ow-a7@vwusl{f}@ zN^&$T=BNVlIC&%%{83c^n{J;ZqVDP;BRxWA*o5#b+_K}A2#yz#2gN~^5~FzVeYW+O zkI1jEeldFinn%W$jv}EJsWM)NWH=E3)v@KVfDUKuYvN4nK$0>)blj}S`3U0(RqgWo zoOCX<@dGPCYz7;NB07YnQFm29)7*dT)Zoz2;A?LV#Lk?3>%ze4evqB0cLmguO=VN2 z0EJE246Y2rpwkjs8fRm+TN~4|5T-2a|1SXikD%(m;m;s1j-+NS3qs1kzY|wa6q}#N z0a>_bX5dj{tBuNGhRUs*<~}I3_RWUN%^f!{-nckdSZwZ@4H5!|I;-$@gy@^$pxYGl>$TN-9Ho}(J8B}4p( zzKyY5~)Nw#WQvtOBy z)BZeKSzpQa*OoV$zF17w;<1<+jKvaJ$_&I}I{7nQ#Le_(a&$tilh@LT1L$N=W@u>O z&Ev77CkBqbF=Pg?bMs1?GL*wqgV9eqYGhe*mSAM@EhC2v2AHKWC}CclA;wexp%nII zQaP+A(3adbvqQvpJpb6$&I{k{<@l|?=R*Iqn78FJa~c!w!O0Cqc=b2t$Pxnx3heZ_p+MweBtFK qH=d6?wNE^FE_HO-V(1y&yX3}`72^5i4Bv|To)BOAbr;^)Wd09MF)5t@ literal 0 HcmV?d00001 diff --git a/tests/selfverification/__pycache__/test_checks_can_fail.cpython-312-pytest-7.4.4.pyc b/tests/selfverification/__pycache__/test_checks_can_fail.cpython-312-pytest-7.4.4.pyc new file mode 100644 index 0000000000000000000000000000000000000000..07825a633dfd7627b697379b2e5b8ba9caf8810b GIT binary patch literal 17844 zcmd5^ZEzdMdADrknGwDRCoK~66wC?mz(wSz`$&VVL5~^3V>NJ_A{nbQrGf?fO#Y(<=nw7j)qqK071c=n@6t zH9-{;f-0)MteEp9d?MxjS${&}=Rhu)!o6H55w1RINHp*#kwk=_qlqZaQnoSIlxWH| zCz^9DiI!YzqBYl+XybJP+4fvVqJ!sy*(JHoL}#umftRCR2=$gGmgbfvmhtOwc6siu z#9g_&6L<6LhU|*m%EZdts>G_?J&AjAazf6nPOQ$YNvz?|BiXgNb%}Mk^@;Vkj=m%$ zHmHqA?^T-D(HYxVi-CR<#r9Hnj`s_M$(ww9w@j zgkIUuvV+~GW|*=$teA32$>()b9#qmihvJdQ;Zqtb%Bf*3by7~} z<>z?Lh>sMXqtb(#Dl4+8Wzz#1Q%o&eln0o8Qp?K(XF5Eax=I*U@~ux68om!=2PDaB0ddD$qWO$~$zuE?2SY=$(uGACM%@O0#mYfOU9;6^1qFL zo%d*{pfDwGYP>dvq@ZLmPN&j(79*z8sG>`Sd>*YuWZ5VhCdl20f;ux~W0;dG=d_&8 zim|wS0tDHggT@n|&rA)=X?#GY=lJWKA&7IrYTto(us&JUDS)x95oHcDy)ZP)8ysXx zPQ$!73F-#4lu|Hgc<~`vY8YRJx~<8BIvYr1DAVQ^+0gm(>ZCDb?X<>HX@lm<5I#Mq zR?yTi=0x{EA)Phl5rxk#bX-v}F=)!51x!wSk8E0_R%zr7lMk478aGpEGh4)r(T6n9 z!q6U{0z{XgjbOkrWfa5E7@tDCF?v?(i>&HsmuZ9OO+50S#6M!973GV8PdlS87O(=n z^c5`WpkkO-@Cl}*vYHkA6h4;5tiaPc%c`h#*Fim-#W?8fvAn8{Xh<+u4l|~+n9pi| zVgO{E;*B0er}J9YYM_3r$*eMf0*!~#47CrjG^V}P)K9&%>u6Lz{AB;(u z)XBN$Qb3hTKBu%Ol>C_{_o~x->ym+xH7RwFhQs=%&UE@bdaHuJTGIu$5-E#&)cJ@(g zF=F21Jj=*V&OJN*!T8@R2>d%m%Z;UQW{q3M$(5o+Qf9q#?Vi<8Df))+O<$?ec`wE- zhH*oi_E8F#8cGo}=3G1Pbw(t!*?HogOChJ<m&bFrfX(8o%b|1cVOQ|KZ%WcQW;qHA--aR{SK`X5(A8%s`+Gs7c zqWlAHIVV?Yb$a@$@8vED?<}_>cR&)#ZRM77s|gD1qttA!s^#09d0cKUwHu!^4>(UU zJx`Iu9}^PZ#5J@;JdaNgsbUla*)mD&#p4z=Ya=5eP+X^B&=+*#^0{zz?7 zn-|^Zu3O0{ z3MMMvSclBxf(YSW5}t*CEsH9|)+yl|;%g#UeMAz9;`jWcB0xi6SkJ0P-v!}<&+^5$ zS>h>!q?;^%@aWcIJ*RC2Ks8dsTS>9$Ryma7d1EW-JNQ~PeS+Uka?D~R0ezFC`aq%i zpkY70Ilv|B8QV?GB<-$Op^!aPKwGE-WNgOwXY~|Du_s=uM(oBQ)bby~|BMSW(zeUe zw)bz4aDK75Rs3L!XN-2mjL7F`u#mxQs1DmkI`4=Xu}InSLwUL2XEJKjXzrk7S6}R2 z7R4P#c!p8S__RR60Hyjs1|P$UCV{4aAWfD`q77AL1Tq+*7b}EeFz_`P(Z~p(FjA?l zu#wX8P&0HZG)n4(X0R^0AHdwr8H^;o6=oVn09qa+@WjAhK^E|vC1ngi&j9IGh81>T zo)t3nB!A*dt5)P>ULVaTPijTO3Igjw0b;GxLgb;5A|GmnS#1E0ET@=Bdzf;@kkyFC zNv!JX3(sKoSrpuiKjXi!I>&`~TZN7luWo+f@$rKbT~|6f#~-;GZTa$^@k3Wy zW7h?LVA+IpwY76nojUZbqpu%*@UAQ2#T| zZvOV>ONTBuJ}`dxYSZ$mwNt02n>K)kmKB#*ZoTB6Ub*X%_E-7u=V$gDtL!;8-Ew^V z$S+r{{nqi!%;-{yZ_RKWxtu*a@d-rrxFYo%5wDgZ3skHR}vTegT^R1^Z4Ntc{IDYh+w0~yNzsu78 zYtl9_mD~Bff4xCi`r!N1E&awaeE7e+cg^G7!5=j&e|%%}kJbc`|H(#5Q+QT8h;@Kt zi#JXnt*eINKB-||B`ia{Y1ZK?9A$qA${E3aCHk2~eN7t(VB&M$+zT9XMi{XdL z)iIO^@DjoOxGM^I5sE-VwFne0w>%UdP((*LPzp?n#xXUdhE3v|{Kyddn{_xa51@3p ztCnSmNzFQ3Met%(rU9VU6uC?t6>V+}0JN~$uxN@pJ?+<_sN}Cv;w@Q+61OBMXiy`I zroh|7Iuy8LU!%lZqYfo=Yk(qB9TxkQygjW$5$1sNb-X3(P~w&Z1yRpP*DG67;MK-A z0~`(qU6tEYm&E6+qZD)~a?XImP31tbDb_r8wAW)X<(4`CUYM&0mC2fsQbq{wRBy6=EZKi9bAIU zEL^fZW^Ca=1pvj)?BqzrmgHem$&EntsvJ77&)_w&Kn=ZYXQW;egxN{S?!H(fBM}C; zsgD$`u&QYzl(Q?-_hbhtIYdb>C5I^?O@=*6$q^*65POR9^e)y%iHgMXqyJtpy2~p$ z_ZS3Us$$cYlr|nz0Gmi)CuzuGiG^JZ2nOE(C?lma)dhYmhZc_E&-fQ46SI)N3OIioocSM+)MJaeZqew+C1^OO7!lj(dp=htLrwO zKm6^!iG!2DGmlS&D~+rFv9XQhWczzzA<#1;9r(Zf{Zu+|13eT{lsE?sAGoV{+f@+GS)Q4m}+T&Bj|ordw0Bn>~D6|x(A-sK2n(kl5%Qq8zS2zJ)t zaya--eDu#X9Tk0Rh3$|nqfj_MgO)|1=sWHIi~#+@KlbPeEVr91+9QmXm}y`krZM4R z)P@z>HI+lzQwu3*Mx=tlkPK}Qa5&BvSRbY)hut(yte}l?0R@#%`RY5efSp2_>Co;>%+l_kr^kGypJYD3G}m1kC7Ub=m{VaIQP zvGxJuO%x^vr-si5r^8!k!aFPBotF+@4)1*rplaWHRD_rNz#vBc{Z+lo{6Ab4=v^JE zKZ(8r9j@*q!W6p=NCCTPJxBo?EkJ|^KG0<5N68QCE$Ndus-VHI2Gx)YFf!cvXqRx( zn?OTtGG>~c=k9r~xC3Z#Yt#{7b8A365saxpo&_lC^t4}xB6p#{wl*&%Zb?u;u;ors z;B%Jwf)vzBf`WzxsC7kjFlGfh>k);M`Mj+m3TZ(UQCEw@By;_ zz%K~9i0v`tPCQcaY$qoCn%(+f_W)JkmIhdixWT4soq8DlRXD)1WW|AbMb2xm%E0wS zOlUC8;2|@VB5@u#A9a|4Al;$-p48NRoEvd#6E88UMTk4t&QW-6VN@at4Om3TB9$WZ znAev#3U`k_YMYAKa~J?C1XG)07IK!aeNOu_9BcH25A7x?oSE4-ipRtrWWZHwU` zXF|pWiOFJ+lr(w7UlAv4bz=8CA{WA$}LLhb9sN7ij&Z18A_g^iC@kGL-4%P`2?+hw6HhHD^owVDfC!?i zuiUc^@;+K=T;Y}V56(ys_aE=?ved)!Aih2${?mmRvFJM37SQc{B=a3@jxeVx5z94e z%+{6jby-*yEz~VvPjjo5r!Q3YyBHiu!CzzPVEk}D2RKWwIo0avY95x}=_30~Jv=XY z*jLT%mLauz)&ay)rIA>8)j1I`q4(N{=RGT&d+ynF8ak~&ZwQxz5M_QE-EycDEQL-9 zh; z7Tx2B(^Kb+9yh~r5X4`&94a=)TE-qCRteUn=Z7ei#g6i@d9ATm26rY`x4~mh5Su^_ zX*)uqeUi*lH9RTMaD%Q4BzSj!)c; zIw__#Sg9|FmM@*ZAda<@Jr^vJVmzwG-U~jP6%rS`=_Pw?tNEV3SnDj-Xi3>LdP+jv z3Z@V|lQXPHI)4flTZEMuj3(bA)Fl{L$?(dprWAz(xdSfDH7L$ZiH}+lyW3>4=f+7! zFwv_`ZR1?lduIn{xn3pfJ`VmkYPAy^yY8`huoH&>_6lQfO3y?7@Q;;D^H2plPQc~F z?5iN*dHflFfdpu^ZOwGsx)+Xe&ilz(oHw?0{K!P2A}yP0t4McGJywx6Ty5+8`Y>7X zrrS2WwdAegnRss{-a8$8xYG9U_|dCv2i_k3ab&u!pX_m+GYkEGE_Gg$z^`xNHxJv5 z0Oxw~uUa!(JmVnXTVT8n+l5gLh>{!GIEzpv$Eb!l0>A1fcKmWX%xAv~mMq=GlBF6; zwoP+3%3dr>UhZK*by%`VON1W|;{eK@mo?9=K`a_5d$ANPBqVCc6&QD3N}zwZLV{5C zouZ&7d={jjR+6La`KhgkvKK4RY68XSG1$9t?B^isr55*h1?jTGBx2(D#&iYfnA&SU> z9?ZF>Sas5Z|+R1^XW(*t;h(=`v@cmtJ*85!S3 zGA>-52S)uXxJF;4WAktQJ~?=$x#j%xm%1(#-`Rg`dh4+-iPz~aVAf|V(u((jLSWyF zwD(T^D$?HfuZPgS5doDKn+|nKe=Q+N;i`7j*g^3BIXh4#1d&OeATkeQaP>OqazO-< z(IQ=}_si>DtFxS34cl1~AIV12 zz(6@zZ-KL~W0qJ^GRLRKxC`vIBai&U`e+|phj!U|B-OAZuF|j=Zj626Lswzdb)*j_ z(FQjsA;f(oiQNG1x~vmN^vCoXGIdnrh~k%}`1@=VZm`XiY@wtZN!5$yv7&K)!=EpL z>4ax79rijkOiC{MJxW~ELY^6$@yN2R%pJI`PP21p@KyX7#Fah@GkW{@^w!62=0yZx z?z?n)y6quS<>E89_d}?J%Z3j2fDIw3&xU>rSfcC2P=i&3AuSqV)HEY66Ds?Qa6$K3U^fP2t`l4L4NPbk zBCyx!atg_X!0gpV@}2RuKFi5x8%mr|I@?Y{+f|ZKb$Fl~XC>7ax}1dl2^#z+{tQxX z=3i7-R>_dy&%#D>?am2rfB?wvx$vFL41R*|gA2>+suB#V@y(_XAL5HVrV#joU56bO z1)_ZX@b>!W_b@vPlSlc?>&7z48psdllS6{KHHbOSI_B1Z8chq8J}y7%`Mf<8b$Z$- z@H6Xhqo)A8=cU9g2?}Uo-YE)dCW-|qsFef-9?VlO^6I9AX*=vZCL>Qz>n-Z}7ABj- z2->10U*|3>;Lq85f)Uq>c2}R`zSe2i7O#zEn%69M>TEmEmEY% z+Z(2sRt+C!7f}B@_%qHR!D4M+JKeq>v6qJ@wofQmJJy|ha=K&l3&+MEndqH({+hID z=HvfemNw0^czb3o-dPKjLOwhTbU`PP51MqJ652 zOV*oQwUU^`j@#z;WJVaV1%9=GyKRD{Kq&~fO{f%t+s37&4hV+BCIq)llxld~Hg27I z)Yr6PPuum%x!}C9>b42fewI?W>b7ZEw3KXcSfz7zpgQ0>cC$fRw)F8uIkueduIQ#6 zg57EC2S#Xl9x>~*hp`oLH3+*=uz`|qO*}o;l)YGaYvBAsJra+$C%Q^MH; z7xQHMV_!kS^;!o?J79VX=N2~;o~(9^0?{x!9x?wGn)x0z^$HR$=I1j2?s{VU$U9B< zR3!NvlI>U9Hl4qJx^34jcsp(_3~eq4sj2IC5+|nt0Oqm8**KX1!D9Kw?*=E|3UdE= zaI&-jKd5m%HE>68au7H<7z-7?h{@WEeG=F?Kcer-!hODwj5BiwfUE zg*cm>auOjj6bDoF(?)3R6!Jo~B%EF+DrK?k;5(>3J63NUU)6=YZ3Vy{YPa8>)%<6u`Il6?ganZ# zP-549xPNo)dS^A?zJ}{nkSkK0FWlG;>DWESB@60X#7u*it-Gny{8F)B~FjYvKS@6RC?HLLZozriJGPzL$I zu+K#lf8%QwrT>-%aqX`J;U(b<(SH)6zYv;!B|Pvm;eme^?!PA7cTL!FP1yaOFC_Zj z6DVmF#3#k;jjP1%*B=(cV$byz0de2;)<*Hc>#M@zL*n(Wu-GSFUlGLJ_K>*s`VtAb TX1}=kdV5gpxq)OWpH2S@B{y&y literal 0 HcmV?d00001 diff --git a/tests/selfverification/__pycache__/test_self_verification.cpython-312-pytest-7.4.4.pyc b/tests/selfverification/__pycache__/test_self_verification.cpython-312-pytest-7.4.4.pyc new file mode 100644 index 0000000000000000000000000000000000000000..fee743c0a0ab273411dc5d95a745be441e0f53c4 GIT binary patch literal 14275 zcmeHOU2NP|b|yKTKaNJSWJ{6bBumG!*OSO&$+m1c>p$6+Ydf)>MowBxld{y1l4isq zl}nD5Q8|IUYhb;_qLI8!Pw=a$1EgGONeJr{~fuaw!JZ)vL1`KptH2YE{cH3?I zQ1qP3UrA$+oNAM8u>|Mx^77ondr98!eD|C?f0NA$3~c3|-zk%JhWQDe_*>k~unz>5 zVXiU?Q(zQU;WW0w6*w02F)dc$=`mhO6rrq=ETp_E=|Y-b5efnvc`Z|EE40yTaV=YE zFSJ)W3LW%3p{=WQ7CI|kg|5o_!uraF!iGwBp}W#k=&3wacnV~aN=iv9!nIhTS9wZl zyTlhZD%nd+;c2BE(oISSq|c~A`DMT9;jw%$%=HCFZ7gvG6se z3$%&KdPp}b8z6mN>4tQR(gW!WwOFpV`fnh7eo__3iCj@<3^Mg~Qye#{x*}U8Lzgvi zqAC+vw^Y^a6ECQwG+t5_kTxo!QMJsHqKa0udR`DfovDgMl@(Fe#gPjjqZd{2HMuxN zWoP6XUXZIw$ufvIV^lRo{HM9^iLxo49zA!qPn;=PlcH{jTIqr+7-MC%Xqm2ru837o zKCQ_mU7W62s%hoLvxaz%$VE-rQ5pW@@mu}((T zb;F{C=2K3bBhb6N*J#7%nlvH?bH3<@IjI+@dE5S3>Evcj{>a+?8JR)badr3Du z-9~J|t_Z2>rX)kJRHx4^@zk5rI8-ubBz#3dHE4qS7G^tStyENZ z7`j&5v44DbaW*?18Kfsd`VY=zVPT z&<1IeVb}-T;B&mnSTNY!&1Qa=c?$+%ovm~ASe>uOE$@Sc0qYG;Ulx!4Cw!cF`vr!1 z3uNnwdeUagiKvvDw-Xy7Q%<%H4i0Jo-HgG^d;Q>NxYh@P4KkD=|gx?QcN;1zRZh zgnIlOlw$XnZT?+AOQy(C9UDML#?FBJre(r#YG?fR{2uq+9v<#l&p_@V&(z!M0@O>} zZPv4Xz2+@0Kj&W$kG|IN*eK{^soohk7prIL!VHt+X8#ceULPWZ4hb=4L>Uo+4jOB+ zq)&+IyG3={qS#>q0>S4v3mD{o(ttQ3Y{yltG^P^SQeUQ$bX1sPrD|FLF=WccEW|3A zSn4exAlhOsKpIb-p+jWwoBgbAJIX~*eOO{$O9lj`=mXjiCMXf;d34V@F z42HMEmrX-=;+i2Vrjzn}>m)5hqF1<*;s{f^F{4XUYRz;KL@mmi=I~kxG$trInYk?K zM2&HH1e)R`6lje~oRl#JPZD!+*)&xUHz%K1AaTK?66Z# z3L0yW5nZo3vQMQbfq0 zI6qx;I^i)$vaY~Y@2!)act7}=Xx`_Te{W;*yFWg4YsaAl{>8bG%X{aCK6vB$iM!8i z`Qe!l&-~!reTI$iUr6;{PR(z-lghp?d`FlcT1Y*!nA*CK+Iqe8anIkUhCYpf&_8F_ z&yRet_xjLBZ#J@f7PCVO*`Yt{Ze$P7ow%Fnc>m?^ygWaC)wp6T_Uu^b+0p3P*~s+I z9lvw(%-;%sEi877{`tT>dzHV!fAr3u*f;E(@BKs1p^rAtzw^Q7A8!9}`}KhzymaNg z+dYRC)*ZUlF* zSbD}=ILzr5(bU8yg*MIC_)S(}-ev3fF=gjSR9LSkDp)MYDOZVnXC#I=7!4L8V@5De5lL1V2W3u;QJAw|esFop= zpRNt%1TqLUXf;PT;&MAlCQhe>uqF|8niP%7v`kc|4K7Pa&8(6Rz^54?VLH8%If-17 zib@ljF(zx04z$)uOQt-oq6VjZnBE$=?FbJ*Z4aVLX!a9g=6 zdNLS2+2fwL3id@$_M14IC@7LVX6Ja z_jqyfOWFU97azbKnI%YqW11q#7^G1a=a1*NpuA4GoCP^|vl$CN8oJr+O|QHjn_%iZ z@GPDpP27&pv*w(Ip9bB^DL)q;fwjc}YmlAs5v+-Y+^yUMLHxE+tO@lu8FMS~5U?h! z(S+TttN|lC2I!aZTiv2X-4XpwS`?xHiApwUNhs-|-D+D9?O~G^gzwHrg_h4Gc-wvr zL^~z1X3r!*yq3=-kc1W_B2T*6^NabdZ6OyheK@m2A6L&k|;ce1d@4x$qCg9l1 zH<*EVa|r+F;cX1?HeTleZ-ujD7gWo!vquq52YsZ4%u1x~qoBDf2eWYxjU(6#SW0k2 zkry$+){q^jIYIVeeh3o8+FfM7mm4GpAP1Ae5RV4{)`9HR;FzIw%hgDgp#y>(!sIX} zM=%-2`4v1y5Y>Z9_ojQvDZES}-9^%Y0TVmnQ%V;=H&6ie z*r`9|Cg(wJFZ|4pA(>xhbH^`lpC7)*A71=L-52=72VWV5uzwy zOUQeh;9@B0;q6+rfb-FyWeWmH#Mc(B4bkoKy$aDSaV_ql8?wm|-B!ycTccarMK|D) z`w_7A_~-_i$DmsT)b;_Wp_y|rWi#Y%#6vtIXP^K+tK|tX z>@}>2H;n?=Zt^-7p9qRw;2K(jV*O1}EJso7G5)>4QAZ?Dj!W`kL@_4Uhn=d~L<@E@Y1TSDLz)l{xYTV}SIOVDdI{{pEpM@R{ zx|N&Y17K%@a#g6cgsTSQrW=7d z(jwsY-g^b)dTdKVNe?^Mss#bcwQNBkNwM?stqm}-MX%z1|6wNXVrP=h#FHK~Oj&_H zHlpci|0@cQcFNb}AAuRBffe%bOVs1FbWW&#ALLIOnqrCu#5;g7p+6K{V&b@5(lC#K z0B~fMrc^gTB2vF6cqYLciB5j%`!vNF@U9BTs3LCFk||;UK^Gwdkq|e!Q$wZEphO?| zGEss$=jc_fTAa#L%1YjZ<`I0Z6r3HwrF1a4Hp#DHX#tbpfaGkh-Nj%LOVBzwi6R7X zzym6j^b0ch_jJp2Rjn4G)6});ngt5eD(ZdLX$(67?ICNziu;75^zD;EGH!U0bf7y49J#c$$LruiowK{@x_| z3>%%>ezf67lZ*M`h5T?McXXli=-lah{JF)iZg24CR?<(L1XKEXq!s;SL*NZ5Qk??; zjbT`Y0w{0u=6^;AOLOYUQ}~2BOWmQqRb9Y+{CB0^osi3%LLg zmAL0dgekj{wBsIlC06Z5O!#;7kKVMFLjbGCU|O4Ui67YmUlqX@>W);Lt`P`mF?8zb zQ!{#fl>`4qGBV~zP!kd{I?E&{F?R|Q6F+8exVwXW4;}1FVzKCh^P;p6V;!r(=(ixS z40%7?LZ(2?In-B!1ZY_2^Nr3I{1JYSKd|`o-*52;R*uzxP<^tDQ2FE{m@2Yi7d}^LtciwfNSm1;My)*@8flQlZCB4+E5F zfhGZE3aAiZ$dOPkpl!A^sn$zYw(&P3`o z!_@D?JdB_#);oXrPw>};18uMs!9!5Wx^=Q!#YB)<;YW}|@_2;>l4h2%pt z-j1viqH}Geg5s2x>6m*IEsIX8`w60XG>$Sz0o}w~2h#AGce6c`Itbd(TFm!P?lP_G>X} zkJ78QKxw}R{?TLgDc|~_V=prGr#68iJq7utbN(d`KwzoNZVYYoLFN^P1t*}*ybbkX zAqp%&z&EGRCMm`{>1R9SI?5)0%7npa`}FTymeO`!f_Y!E9e7eD)asHU#a!t?k5Qy_So)9_%xqj9Okl z)m!Y>7bC{? z`Wub)FZ}|USA38EoBcLFywc$Q(s9D9bhQAa;TnmxGG&2@d?I7Yw*b{(TGT==B>7*m z;{=CjGsnpdFQ!GWuNE-MI8;|SMtbRpZ?YbpTm*S^sy@Q`@nvx3TYUdx#@o?lj>=^e z7Nbf_TpYex`vR^?4pt-K3dp6d%GH10tw_ZG8~WN}Bt7|v^H)5^sdHg%T$LDS@m-Y= zXPJHPJS+^az`|u6*O^1?1eOHCx;*n`Q5KPOU<3sJvr;TpO}a*~k49mZ6bWcYVYM_Y zN}RxOO$itGp+;I^hXRY3FW}N}`o|Ac2y3LRNexy}=V`#yHwk{m&j#;aBp!ZQ*%M@X@);)ZG>M8_lAsLsZHz^Yq^@68!lE)}g{*V`%WoUCU1wb(;C*E86q!Uk~U? zR(J6f+CW0zX$EtY%?4 zVY2_iJbRBB{27;Gxt}qRe1^%VJxu%8?qr_6lYQ>9B%k5tk`LP3*xmP^W7*C3+d1|K zdw(6r4zr&M443-$ri+^v+1>@V_k&Hhx1DXU=RV`&>}%|U7{k5BeoFKAdpP#seSv3p V-f!pGqwM_+96Rs;&If4I{|7-!M56!z literal 0 HcmV?d00001 diff --git a/tests/selfverification/checks.py b/tests/selfverification/checks.py new file mode 100644 index 0000000..47d321e --- /dev/null +++ b/tests/selfverification/checks.py @@ -0,0 +1,219 @@ +"""Out-of-band verification of test-driver's own foundational guarantees. + +These checks are deliberately **not** written using test-driver. Using the +framework to establish that the framework's oracles are independent is a system +certifying itself: any flaw serious enough to matter would likely be shared by +both the thing under test and the thing testing it. + +So: plain functions over an Evidence Pack, plain pytest assertions, no `Oracle`, +no `Runner`, no `Verdict` aggregation. They read the same artefact an auditor +would read six months later, and nothing else. + +Each check returns a list of violation strings — empty means satisfied. Returning +data rather than asserting keeps the checks usable both as tests and, later, as +the `td://self/...` verification assets themselves. + +Registry of self-verification identifiers: + + td://self/actor-isolation + td://self/oracle-independence + td://self/evidence-reproducibility + td://self/intent-independence + +A check that cannot fail is worth nothing, so every check here has a +corresponding case in `test_checks_can_fail.py` that feeds it a deliberately +broken artefact and asserts it complains. +""" + +from __future__ import annotations + +import json +from typing import Any, Iterable, Mapping + +SELF_CHECKS = ( + "td://self/actor-isolation", + "td://self/oracle-independence", + "td://self/evidence-reproducibility", + "td://self/intent-independence", +) + +# Provenance values that may back an assertion capable of producing FAIL. +ADMISSIBLE_PROVENANCE = frozenset({"human", "spec", "agent-from-spec"}) + + +def _observations(pack: Mapping[str, Any]) -> list[dict]: + return list(pack.get("observations", ())) + + +def _snapshots_by_step(pack: Mapping[str, Any]) -> dict[str, dict]: + return { + obs["step_id"]: obs["data"] + for obs in _observations(pack) + if obs["kind"] == "state_snapshot" + } + + +# --- td://self/actor-isolation ------------------------------------------- + + +def check_actor_isolation( + pack: Mapping[str, Any], + secrets_by_actor: Mapping[str, str], + memories_by_actor: Mapping[str, Mapping[str, Any]], +) -> list[str]: + """No actor may hold or emit anything private to another actor. + + Checked two ways, because either alone is weak: + + * **memory** — actor A's private store must not contain actor B's secret; + * **evidence** — an observation attributed to actor A must not carry B's + secret, which would mean the leak escaped into the record even if A's + memory looked clean. + """ + violations: list[str] = [] + + for actor, memory in memories_by_actor.items(): + rendered = json.dumps(memory, default=str) + for other, secret in secrets_by_actor.items(): + if other != actor and secret in rendered: + violations.append( + f"actor {actor!r} holds {other!r}'s private value in memory" + ) + + for obs in _observations(pack): + collector = obs.get("collector") + if collector not in secrets_by_actor: + continue + rendered = json.dumps(obs.get("data"), default=str) + for other, secret in secrets_by_actor.items(): + if other != collector and secret in rendered: + violations.append( + f"observation {obs['id']} collected by {collector!r} " + f"carries {other!r}'s private value" + ) + return violations + + +# --- td://self/oracle-independence --------------------------------------- + + +def check_no_actor_collected_judgment( + pack: Mapping[str, Any], actor_ids: Iterable[str] +) -> list[str]: + """S2 and S3 evidence must never be attributed to an actor.""" + actors = set(actor_ids) + return [ + f"{obs['stratum']} observation {obs['id']} was collected by actor " + f"{obs['collector']!r}" + for obs in _observations(pack) + if obs["stratum"] in ("S2", "S3") and obs["collector"] in actors + ] + + +def check_verdicts_follow_from_judgment_evidence( + pack: Mapping[str, Any], assertions: Iterable[Any] +) -> list[str]: + """Every recorded verdict must be reproducible from S3 evidence alone. + + This is the substantive independence check. Re-evaluating each assertion + against the stored snapshots — with no actor, no driver and no live system in + reach — must reproduce exactly what the run reported. If it does not, then + something outside the judgment stratum influenced the verdict, whatever the + architecture diagram claims. + """ + violations: list[str] = [] + snapshots = _snapshots_by_step(pack) + recorded = {(v["assertion_id"], v["step_id"]): v["verdict"] for v in pack["verdicts"]} + + for assertion in assertions: + for (assertion_id, step_id), verdict in recorded.items(): + if assertion_id != assertion.id: + continue + snapshot = snapshots.get(step_id) + if snapshot is None: + violations.append( + f"verdict {assertion_id}@{step_id} has no S3 snapshot to rest on" + ) + continue + try: + expected = "PASS" if assertion.predicate(snapshot) else "FAIL" + except KeyError: + expected = "INCONCLUSIVE" + if expected != verdict: + violations.append( + f"verdict {assertion_id}@{step_id} recorded {verdict}, but the " + f"retained judgment evidence yields {expected}" + ) + return violations + + +# --- td://self/evidence-reproducibility ----------------------------------- + + +def check_evidence_supports_every_verdict(pack: Mapping[str, Any]) -> list[str]: + """A verdict with no evidence behind it is an EVIDENCE_FAILURE, not a result.""" + violations: list[str] = [] + snapshots = _snapshots_by_step(pack) + steps_with_surface = { + obs["step_id"] for obs in _observations(pack) if obs["kind"] == "realization" + } + + for verdict in pack["verdicts"]: + step_id = verdict["step_id"] + if step_id not in snapshots: + violations.append( + f"verdict {verdict['assertion_id']}@{step_id} has no state snapshot" + ) + if step_id not in steps_with_surface: + violations.append( + f"verdict {verdict['assertion_id']}@{step_id} has no record of how " + "the step was performed" + ) + if not pack.get("sut_version"): + violations.append("evidence does not identify the version under test") + if not pack.get("finished_at"): + violations.append("evidence does not record when the run ended") + return violations + + +def check_runs_agree(first: Mapping[str, Any], second: Mapping[str, Any]) -> list[str]: + """Two runs from the same seed must reach the same judgments.""" + def key(pack): + return sorted( + (v["assertion_id"], v["step_id"], v["verdict"]) for v in pack["verdicts"] + ) + + if key(first) != key(second): + return ["two runs from the same initial state disagreed"] + if first["sut_version"] != second["sut_version"]: + return ["runs were taken against different versions"] + return [] + + +# --- td://self/intent-independence ---------------------------------------- + + +def check_intent_independence(pack: Mapping[str, Any]) -> list[str]: + """No verdict may rest on intent derived from the implementation. + + Independence of *components* does not give independence of *belief*. A + verdict is only worth something if the claim behind it came from somewhere + the implementation could not reach. + """ + violations: list[str] = [] + index = pack.get("provenance_index", {}) + + for verdict in pack["verdicts"]: + assertion_id = verdict["assertion_id"] + provenance = index.get(assertion_id) + if provenance is None: + violations.append( + f"assertion {assertion_id!r} produced a verdict with no recorded " + "provenance — its independence cannot be audited" + ) + elif provenance not in ADMISSIBLE_PROVENANCE: + violations.append( + f"assertion {assertion_id!r} has provenance {provenance!r}, which is " + "derived from the implementation it constrains" + ) + return violations diff --git a/tests/selfverification/test_checks_can_fail.py b/tests/selfverification/test_checks_can_fail.py new file mode 100644 index 0000000..c727c0d --- /dev/null +++ b/tests/selfverification/test_checks_can_fail.py @@ -0,0 +1,188 @@ +"""A self-test that cannot fail is worth nothing. + +Every check in `checks.py` is fed a deliberately broken artefact here, and must +complain. Without this file, the self-verification suite would be a row of green +ticks with no evidence that any of them is load-bearing. + +Two kinds of break appear below, and the difference is worth noticing: + +* **structural breaks** — the guarantee is actually violated in a running + system (actors sharing memory). The check must catch it from the evidence. +* **artefact breaks** — the guarantee cannot be violated through the framework + because the architecture forbids it, so the check is exercised against a + hand-built pack instead. Where that applies it is stated explicitly, together + with the separate assertion that the framework does refuse. +""" + +from __future__ import annotations + +import copy +import json + +import pytest + +from testdriver import Actor, Cast, Oracle, Runner, World +from testdriver.runner import CollectorIndependenceError +from lab.mutations import ObservationChannel, build_lab +from testdriver import DirectDriver, StateObserver +from scenarios.alice_bob_carol import USE_CASE, build +from tests.selfverification.checks import ( + check_actor_isolation, + check_evidence_supports_every_verdict, + check_intent_independence, + check_no_actor_collected_judgment, + check_runs_agree, + check_verdicts_follow_from_judgment_evidence, +) +from tests.selfverification.test_self_verification import SECRETS, run_and_serialize + + +# --- td://self/actor-isolation ------------------------------------------- + + +def test_shared_actor_memory_is_caught(): + """A structural break: two actors handed the same private store. + + This is what an isolation bug actually looks like when the same agent + technology executes several actors in one process — nobody writes + `bob.memory = alice.memory`, but a shared default or a cached client + achieves it by accident. + """ + lab, tokens = build_lab() + shared: dict = {} + cast = Cast() + for name in ("alice", "bob", "carol"): + actor = Actor(name, name.title(), credentials={"token": tokens[name]}) + object.__setattr__(actor, "_memory", shared) # the leak + cast.add(actor) + + _, _, _, asset, _ = build() + world = World("w-leaky", lab, lab.version, cast=cast) + driver = DirectDriver(lab, tokens) + observer = StateObserver(ObservationChannel(lab), asset.scenario.watches) + + for actor_id, secret in SECRETS.items(): + world.cast[actor_id].remember(f"private-{actor_id}", secret) + + result = Runner(world, driver, observer, Oracle()).run(asset) + pack = json.loads(result.evidence.to_json()) + memories = {a.id: {k: a.recall(k) for k in a.known_keys()} for a in world.cast} + + violations = check_actor_isolation(pack, SECRETS, memories) + assert violations, "shared actor memory went undetected" + assert any("holds" in v for v in violations) + + +def test_a_secret_leaking_into_the_record_is_caught(): + """An artefact break: the leak escapes into evidence even if memory is clean.""" + pack, memories, _ = run_and_serialize() + tampered = copy.deepcopy(pack) + for obs in tampered["observations"]: + if obs["collector"] == "alice": + obs["data"]["stolen"] = SECRETS["bob"] + break + assert check_actor_isolation(tampered, SECRETS, memories) + + +# --- td://self/oracle-independence --------------------------------------- + + +def test_the_framework_refuses_to_let_an_actor_collect_judgment_evidence(): + """The architecture prevents this, so it cannot be produced by a real run.""" + world, driver, observer, asset, oracle = build() + observer.name = "bob" + with pytest.raises(CollectorIndependenceError): + Runner(world, driver, observer, oracle).run(asset) + + +def test_actor_collected_judgment_evidence_is_caught_in_the_record(): + """...and if it ever did appear in a pack, the out-of-band check sees it. + + Both halves are needed. The first says the door is locked; this one says we + would notice if someone came through the window. + """ + pack, _, actors = run_and_serialize() + tampered = copy.deepcopy(pack) + for obs in tampered["observations"]: + if obs["stratum"] == "S3": + obs["collector"] = "bob" + break + assert check_no_actor_collected_judgment(tampered, actors) + + +def test_a_verdict_that_contradicts_its_evidence_is_caught(): + """The strongest of the checks: verdicts must follow from S3 alone.""" + pack, _, _ = run_and_serialize() + tampered = copy.deepcopy(pack) + tampered["verdicts"][0]["verdict"] = "FAIL" # a verdict nothing supports + assertions = list(USE_CASE.claims) + list(USE_CASE.invariants) + violations = check_verdicts_follow_from_judgment_evidence(tampered, assertions) + assert violations + assert "retained judgment evidence yields" in violations[0] + + +def test_a_pass_silently_replacing_a_fail_is_caught(): + """The specific corruption the project most needs to notice.""" + pack, _, _ = run_and_serialize("M15") + tampered = copy.deepcopy(pack) + for verdict in tampered["verdicts"]: + if verdict["verdict"] == "FAIL": + verdict["verdict"] = "PASS" + assertions = list(USE_CASE.claims) + list(USE_CASE.invariants) + assert check_verdicts_follow_from_judgment_evidence(tampered, assertions) + + +# --- td://self/evidence-reproducibility ----------------------------------- + + +def test_a_verdict_without_a_snapshot_is_caught(): + pack, _, _ = run_and_serialize() + tampered = copy.deepcopy(pack) + tampered["observations"] = [ + o for o in tampered["observations"] if o["kind"] != "state_snapshot" + ] + assert check_evidence_supports_every_verdict(tampered) + + +def test_a_verdict_without_a_record_of_how_the_step_ran_is_caught(): + pack, _, _ = run_and_serialize() + tampered = copy.deepcopy(pack) + tampered["observations"] = [ + o for o in tampered["observations"] if o["kind"] != "realization" + ] + violations = check_evidence_supports_every_verdict(tampered) + assert any("how" in v for v in violations) + + +def test_evidence_that_does_not_name_the_version_is_caught(): + pack, _, _ = run_and_serialize() + tampered = copy.deepcopy(pack) + tampered["sut_version"] = "" + assert check_evidence_supports_every_verdict(tampered) + + +def test_disagreeing_runs_are_caught(): + first, _, _ = run_and_serialize() + second, _, _ = run_and_serialize("M15") + assert check_runs_agree(first, second) + + +# --- td://self/intent-independence ---------------------------------------- + + +def test_implementation_derived_intent_is_caught(): + pack, _, _ = run_and_serialize("M15") + tampered = copy.deepcopy(pack) + tampered["provenance_index"]["c-bob-revoked"] = "agent-from-implementation" + violations = check_intent_independence(tampered) + assert violations + assert "derived from the implementation" in violations[0] + + +def test_unrecorded_provenance_is_caught(): + """Absent provenance is not a benign omission — it makes the verdict + unauditable, which for this framework is the same as unusable.""" + pack, _, _ = run_and_serialize() + tampered = copy.deepcopy(pack) + tampered["provenance_index"] = {} + assert check_intent_independence(tampered) diff --git a/tests/selfverification/test_self_verification.py b/tests/selfverification/test_self_verification.py new file mode 100644 index 0000000..506eb79 --- /dev/null +++ b/tests/selfverification/test_self_verification.py @@ -0,0 +1,110 @@ +"""The framework's foundational guarantees, verified from outside the framework. + +These read an Evidence Pack the way an auditor would — as JSON, with no live +objects — and use plain pytest. No Oracle, no Runner, no Verdict aggregation. +""" + +from __future__ import annotations + +import json + +import pytest + +from testdriver import Runner +from scenarios.alice_bob_carol import USE_CASE, build +from tests.selfverification.checks import ( + SELF_CHECKS, + check_actor_isolation, + check_evidence_supports_every_verdict, + check_intent_independence, + check_no_actor_collected_judgment, + check_runs_agree, + check_verdicts_follow_from_judgment_evidence, +) + +SECRETS = { + "alice": "alice-only-9f3c", + "bob": "bob-only-2a71", + "carol": "carol-only-55e0", +} + + +def run_and_serialize(*mutations: str): + """Run, then throw away everything except the serialized evidence. + + Deliberate: the checks must work from the artefact alone, exactly as they + would months later, with no access to the objects that produced it. + """ + world, driver, observer, asset, oracle = build(*mutations) + for actor_id, secret in SECRETS.items(): + world.cast[actor_id].remember("private", secret) + result = Runner(world, driver, observer, oracle).run(asset) + pack = json.loads(result.evidence.to_json()) + memories = {a.id: {k: a.recall(k) for k in a.known_keys()} for a in world.cast} + return pack, memories, list(world.cast.actors) + + +def test_the_registry_is_complete(): + assert len(SELF_CHECKS) == 4 + + +# --- td://self/actor-isolation ------------------------------------------- + + +def test_actor_isolation(): + pack, memories, _ = run_and_serialize() + assert check_actor_isolation(pack, SECRETS, memories) == [] + + +# --- td://self/oracle-independence --------------------------------------- + + +def test_no_actor_collected_judgment_evidence(): + pack, _, actors = run_and_serialize() + assert check_no_actor_collected_judgment(pack, actors) == [] + + +@pytest.mark.parametrize("mutations", [(), ("M15",), ("M11",), ("M01", "M15")]) +def test_verdicts_are_reproducible_from_judgment_evidence_alone(mutations): + """Holds on passing and failing runs alike — a check that only works when + everything is green is not verifying independence, it is verifying luck.""" + pack, _, _ = run_and_serialize(*mutations) + assertions = list(USE_CASE.claims) + list(USE_CASE.invariants) + assert check_verdicts_follow_from_judgment_evidence(pack, assertions) == [] + + +# --- td://self/evidence-reproducibility ----------------------------------- + + +def test_every_verdict_is_supported_by_retained_evidence(): + pack, _, _ = run_and_serialize() + assert check_evidence_supports_every_verdict(pack) == [] + + +def test_two_runs_from_the_same_seed_agree(): + first, _, _ = run_and_serialize() + second, _, _ = run_and_serialize() + assert check_runs_agree(first, second) == [] + assert first["run_id"] != second["run_id"] + + +def test_reproducibility_holds_on_a_failing_run(): + first, _, _ = run_and_serialize("M15") + second, _, _ = run_and_serialize("M15") + assert check_runs_agree(first, second) == [] + assert any(v["verdict"] == "FAIL" for v in first["verdicts"]) + + +# --- td://self/intent-independence ---------------------------------------- + + +def test_intent_independence(): + pack, _, _ = run_and_serialize() + assert check_intent_independence(pack) == [] + + +def test_intent_independence_holds_when_a_claim_fails(): + """The moment that matters: a FAIL is an accusation, and an accusation + resting on implementation-derived intent is worthless.""" + pack, _, _ = run_and_serialize("M15") + assert check_intent_independence(pack) == [] diff --git a/workplans/TD-WP-0002-vertical-spike-crystallization.md b/workplans/TD-WP-0002-vertical-spike-crystallization.md index 9cc9215..7c9c3ca 100644 --- a/workplans/TD-WP-0002-vertical-spike-crystallization.md +++ b/workplans/TD-WP-0002-vertical-spike-crystallization.md @@ -269,7 +269,7 @@ Three results worth carrying: ```task id: TD-WP-0002-T06 -status: todo +status: done priority: medium state_hub_task_id: "4fb31485-8650-556c-bee4-b4e067a89d04" ``` @@ -282,6 +282,27 @@ Exit: deliberately breaking actor isolation fails the out-of-band check; allowing an actor to determine its own verdict fails oracle-independence; each failure produces a Framework Finding. +**Done 2026-08-22.** `tests/selfverification/` — checks as plain functions over a +serialized Evidence Pack, plus 12 tests that they hold and 12 that they *can +fail*. All four `td://self/...` identifiers are covered. 72 tests pass overall. + +- The strongest check turned out to be **verdict reproducibility from S3 alone**: + re-evaluating each assertion against the stored snapshots, with no actor, + driver or live system in reach, must reproduce exactly what the run reported. + It is asserted on failing runs too — a check that only works when everything is + green verifies luck, not independence. +- **F-0003 (open)** — actor isolation leaves no trace in ordinary evidence. The + self-test catches a shared memory store only because the harness plants + per-actor canaries first; without them, a run where all three actors share one + memory dict produces an Evidence Pack identical to a correct one. Isolation is + currently a property of a scenario written to expose it, not of runs in + general. Carried to T10, reassess after T07. +- The same finding notes the mirror-image problem: oracle independence is + enforced by construction, so no real run can produce the violating artefact and + the check must be exercised against a hand-built pack. **A guarantee enforced + by construction cannot be verified by observing real runs.** Four green + self-tests are not four equivalent proofs, and the suite now says so. + ## Agentic realisation of one semantic action, fully instrumented ```task