From eee77227144f47be836f6b8d68ae654c9d3f4c11 Mon Sep 17 00:00:00 2001 From: tegwick Date: Sun, 23 Aug 2026 00:21:48 +0200 Subject: [PATCH] T09: crystallization A stable agentic realization becomes deterministic code. All four exit criteria met; 163 tests pass. - crystallization.py: trajectory capture, stability assessment requiring the same path across several runs, CrystallizedDriver, pytest codegen - crystallized/test_grant_access.py: generated, runs with no model, carries its lineage in the docstring - descendant preserves the ancestor's oracle set, agrees with it across five lab versions, and still catches a seeded defect - reversibility shown both ways via new M24 (grant endpoint renamed): the frozen descendant fails loudly rather than searching, and the agentic ancestor recovers from the same mutation F-0007 (open): the 54% cost reduction must not be quoted in support of the thesis. The T07 runtime is token-free, so the measured saving is one page fetch, one parse and a two-candidate scoring pass. The saving the concept actually claims - tokens, latency, retry variance - is unmeasured. Together with F-0005 this makes a bounded live-model experiment the highest-value next investment. Assertions in the generated test are imported rather than restated, so it is not fully standalone. Deliberate: paraphrased claims would be a second unverified statement of intent. Co-Authored-By: Claude Opus 5 Assistant: claude-code Assistant-Model: opus Assistant-Process: 1629012@bnt-lap001 Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39 --- WORK-RECORDS.md | 2 +- crystallized/__init__.py | 6 + .../__pycache__/__init__.cpython-312.pyc | Bin 0 -> 424 bytes .../conftest.cpython-312-pytest-7.4.4.pyc | Bin 0 -> 2199 bytes ..._grant_access.cpython-312-pytest-7.4.4.pyc | Bin 0 -> 6555 bytes crystallized/conftest.py | 39 +++ crystallized/test_grant_access.py | 73 +++++ lab/__pycache__/app.cpython-312.pyc | Bin 16637 -> 16687 bytes lab/__pycache__/http_api.cpython-312.pyc | Bin 11904 -> 12015 bytes lab/__pycache__/mutations.cpython-312.pyc | Bin 14293 -> 14743 bytes lab/app.py | 1 + lab/http_api.py | 4 +- lab/mutations.py | 7 + pyproject.toml | 2 +- research/concepts/fitness-map.md | 6 +- .../E-002-crystallization-fidelity.md | 3 +- ...-0007-crystallization-cost-unmeasurable.md | 87 ++++++ research/hypotheses/H-003-crystallization.md | 24 +- .../__pycache__/browser_grant.cpython-312.pyc | Bin 5052 -> 6130 bytes scenarios/browser_grant.py | 23 ++ .../crystallization.cpython-312.pyc | Bin 0 -> 12997 bytes src/testdriver/crystallization.py | 272 ++++++++++++++++++ ...lassification.cpython-312-pytest-7.4.4.pyc | Bin 22288 -> 22324 bytes ...ystallization.cpython-312-pytest-7.4.4.pyc | Bin 0 -> 28230 bytes ..._ground_truth.cpython-312-pytest-7.4.4.pyc | Bin 25643 -> 25643 bytes tests/test_classification.py | 1 + tests/test_crystallization.py | 227 +++++++++++++++ ...-WP-0002-vertical-spike-crystallization.md | 27 +- 28 files changed, 794 insertions(+), 10 deletions(-) create mode 100644 crystallized/__init__.py create mode 100644 crystallized/__pycache__/__init__.cpython-312.pyc create mode 100644 crystallized/__pycache__/conftest.cpython-312-pytest-7.4.4.pyc create mode 100644 crystallized/__pycache__/test_grant_access.cpython-312-pytest-7.4.4.pyc create mode 100644 crystallized/conftest.py create mode 100644 crystallized/test_grant_access.py create mode 100644 research/findings/F-0007-crystallization-cost-unmeasurable.md create mode 100644 src/testdriver/__pycache__/crystallization.cpython-312.pyc create mode 100644 src/testdriver/crystallization.py create mode 100644 tests/__pycache__/test_crystallization.cpython-312-pytest-7.4.4.pyc create mode 100644 tests/test_crystallization.py diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 67df1d0..a61b979 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -20,6 +20,6 @@ | task | TD-WP-0002-T05 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T06 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T07 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | -| task | TD-WP-0002-T08 | progress | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | +| task | TD-WP-0002-T08 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T09 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T10 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | diff --git a/crystallized/__init__.py b/crystallized/__init__.py new file mode 100644 index 0000000..8b625d3 --- /dev/null +++ b/crystallized/__init__.py @@ -0,0 +1,6 @@ +"""Generated deterministic regression tests. Do not edit by hand. + +Each file here is the frozen descendant of an agentic verification asset. Its +provenance, ancestor and the number of identical realizations it was frozen from +are recorded in its own docstring. +""" diff --git a/crystallized/__pycache__/__init__.cpython-312.pyc b/crystallized/__pycache__/__init__.cpython-312.pyc new file mode 100644 index 0000000000000000000000000000000000000000..d98adb615c0cfda49372236a6128d573d46f696b GIT binary patch literal 424 zcmXw$K~BRk5Ji(xphW5!m~ACU#2&E%AvPRt#3-?=8Mjq>1TMe{Sa23@z?KUj zy6cK@q|V0v_UO&`_t$)0WKtiOZ?6}5mi;7gEYV}_XR^F!Hham2d@<1zxp+j6q$66e z7?FDC9Y<$?upwl2!9zrjT*7?-A0nW25takA+S{@y9<*tpb{#^Cgy0xri%?V8p%)|@ z^j7-_p$6?iH|P_6gXC&wbWD8BjIo5L$i<4n2EFzMucY0`;6SpmBh;_^B~l_eJ7RPv zHg)HABZonZZ8aZM$@!)%o)`J|xl(eX)MvI&NgZyg zHuP9+1F>nV6u`A5C$1|)14rG-P_&g&5?55JTn&fWZ6EBq!#g??LW-QzkNlqGdHy#! J&8IXK`43<(h5-No literal 0 HcmV?d00001 diff --git a/crystallized/__pycache__/conftest.cpython-312-pytest-7.4.4.pyc b/crystallized/__pycache__/conftest.cpython-312-pytest-7.4.4.pyc new file mode 100644 index 0000000000000000000000000000000000000000..f8c9bb18f0b8c5540dd80f7952a25aade8be5e25 GIT binary patch literal 2199 zcmbtVUuYaf7@xh}z1!Q|<&taqCnmXUT5COMFKKItRxN5mYzefW73m=?o87tGUiWs- z*|}UVB#=s>R-r}^4T#VOA0nyXgAc_gslNG=kV?W5s8~h&BB`lRiVuFXdwZ9nZ_aT$ z-=F!u`DXU_`}T)qQbsVo>3^^Ehk(#sri3FB2;BBH5I#W$$|Hj_A{JMUU>tJ4sZyKn-^)7U{@(PMA1pzrJ7}#GuTiFo&vjTI<`WwOWm9- zpDSX;v8}pNAX*vM9CAU?9Rn-cPAP%5HoBmlJGRu6hdXA$F387i;8hQ&N<|1?aY@{SM^EHK8 z-r&1IGXz64qBmm808lnKL}P&jSt84gz2*cO53MC1)S(v4vqCBHxP@tkbR)jp0D39) zZ+{3n*t0YA(p{m{+MVtWZH8=!Va6TSwwARJ3H@_o92p6yY)=^7nvEno`={u$NO%vk zVrgH?A4&!OP+2-y98szfAd?q0G+cIUkDqiVJyF+)W6g9H zDXp9s96Wk#BsT=VqbKe$aGlAfycoo2loApJgBN2Ihdx0VBE2{*LOBKy(Tkq*MK9^| zONF+CfssHr*p(87TEc)%7-zh8e-TkN!(hG2lU!S?xJ8G8RIF0Nso7qlu|~HX7iXhn zufJir+TgOsYn2L|iQT8`MT5nj6f_=a1*}@dACHa>7M(I4gudtM#X;8Pju-^q4jJ6p zI0xaX3e1_S)V=ghj|!2NL0EHU_;(+|<9GCM9zBSn&V7rq>wA~>uXP-mKfRgmSv=HB-e$m zRiSHfe7$#QwRh-d*KOhCx-hmXjNKN-H&eZv(i4lv*QEZ>4=#yI?%Mw2U%zp4e?M|A$wR#Xx*KehK?kFG1jtIF_K+N$#GO8;{!i4z-%^uoy1vm1%7^+exl zqHilEq~t%55SOfVsr*QK6Sspm(j?;(`=pN_xX3>^NQSN4qY+^y~> zM>FWwivu8km)Spt`0o)9a(1^DQ&q#!Rn?PKwNParDpe({|9KMYu+Yz4FY!{Pt~Rx9 zwNm#lUvuGA`q(G9Mofymc+QRWUBxr2tcq9`I=$z00ZkL=Er86(;w8xmwARCD%@zdK}rVSMjcXEXR)HCXs()r%fG05|CyQWI>T5EoCmb z9dfnatu_#f9P0L97buVbbpao8Xy6=r?4g$?$6hQLjSMzG(l$Vln^U0(;9UC6a7odX zP#J*>6bQ8%z8SuG^Zw?|y!YpPUXk#;x9=B=Cjk9}cY-JG1M>Qt0N$2#sVwQT9y8@y ztQ?a$95>_Tgm@=wDGg(_bUD-gk}WIZORoFw5$}YVul1IDYklRu+K%!L;3mucdP-00 znH#b4fZn5L-%6Bs>Umwkdr;54C6$L-@j`FoOhS^*u$JShrfIaPP8eNa)Nu^kA})1Y z^7oZrlLcx~rn*!=LUfy0wo9mPxMZ$Hs;Z@z6y;^ZqUr)wivFXhmWH`DBdX(2mz*R^ zYH@+7mRnph)Z!eon+|0gST)scFvD$~B(vj+P92R}I*0!78J~sMT!-r|}KqVhNiW z3x)+7K+=Xoi3VL0I98Q1=tKo~V2Jyf@H%zF?-b#RKygW3Woo_3FcX5gPGnJ{!Nx!& zDsiZ0TkzI~1%(^)1}F-`HEsxdUNI~e&=H|-9SHDc>NdFsIh$_NCQZ9x>KG*s3~ae2 zGD9hOu{bt1cAU(^v74M1&XA(nus!y`OY^>61*=$%fw_6rfG;f^5o9%+F{-(68HRlY zQR{F;sL6r3u%|{wh-;HZU55-D9|kvpZZoGurfCr#BB%&wGBmL6Z?TianCCtK(%ico zf33j5YZ%kHn7+Ys58Hth9aXFuB|quWp*xbex%(W<&+*bPQ}PZXrU2E+#2Dnzk# z$X8On;6be*dpXs@WPg`kEX2KhWzL?fXsT6#ZFH|c=mM$I#FnSt0KPKY)$ zviuwAw=q4gCvL>vPu}WE-AQD(ld>;L(TA54dg}dj!1XbylUzuTJ0P)Loz!w_ao6T=TI@Bf+)2He zn(4lqa47TzX|@ww*qW2#HA2-mli{CW1Z>MWa`W&1SQREn6!#7`C#8sHnq* z@>31Bih$%F3GJRUwulME*;bvxaTC~vs+X!!An9JZ3b(}O`hp+paPu^e+IGV&^mr*y zHH|qh&FD`W2wz@$x(g%`svPR2fv{*BmX{6J_fkF|;+u^i>ZN#iYB*k!GG;R`%OmbN z0VX)Zg|wI92U7(kyrgSi#*_p>$XEaJqj)vm$-mJ9y2yYOVDi zSnWOVai;fL&()r_%-+?^-gka{)4h{9@^OCWwb!q{zLpfx|Hclt(?C%kVrnp3A}OjjV^)nzn{LH zAHI`6aZi%7!+*GNcjx18KmV)ezlx*x6(9O1Z@&I4*MIHQ)l+M^{j0hC7)m~Nw{P%T z`)d0ynLS_Mi=%(vIRyd!G`auuap{Ahp3_goKiHq;@Kedtlj#q};vAk7?eQnhjKn`2 z%$+$H|8Sqo;e$!ES~P<2*dXFe2>{|ucMnSvqJiv&m~I%Gg8e!JPikMdL>YHZ5>+}* z*2u*MH?!6e+%Z(mHCs6EHjV*e!s9Ijc-Jl!;=({)YIbV+{JB{#_2NtCUOqchh_QX( zixD&qXNC}V059&Z{6a)o0ljDW*PUoqq|b6YuJ3;5*t>_`KXfNIemgO~j$ug)r9vl+ ztYtuNL(bPBe@E`bI`N3~@gh z%xBHV0-fmzmwUB%7C9gP@dQzFNpki?=!@I|5`zoRTG&fUl=TXEVGVC#9oY41bZEb5h8`Ehub1}kjq$D3xrhP`^_yx@pC(#Aciko4l1#hY>7Oz>us zH^|3m8 zQKSRibs4|awhvAbo^|_oQtw*CYgLWx=%PRI_5C%;aS&jyjK;S)Ehk z_hc26h2Cn;g4a7GK%{wouCy~ip;u72;)N#L5*iTV_LqXD58|{Lc6oJIb{jAu|z0o^<#j*G?et- zwMYH?C^7X(;+3`k!H@AYxu6KI^5RIHMf$}qV3h6Wjewp6ggS$vxn7D2Ri z<$I&RAtCk;IP^m_giu-d`^#F&?t#Y1mj6zIOG>JdbxN06W3S>9&pVydXpp+g@|Us4<;1^E zO7igMlJu5zN80&U{u}ySDz8fAPqR;~r0(wi?(gVoc5)@Pp6Zn!TYoYpPpm6h`I0Qw F_&4Z$zn%a9 literal 0 HcmV?d00001 diff --git a/crystallized/conftest.py b/crystallized/conftest.py new file mode 100644 index 0000000..62a1a8f --- /dev/null +++ b/crystallized/conftest.py @@ -0,0 +1,39 @@ +"""Fixtures for crystallized regression tests. + +The only framework code a crystallized test touches is the observation channel — +because judging still requires observing the system independently, whether or not +a model was involved in getting there (D-07). Nothing here discovers anything. +""" + +from __future__ import annotations + +import threading + +import pytest + +from lab.http_api import serve +from lab.mutations import ObservationChannel, build_lab +from testdriver.observers import StateObserver, Watch + + +@pytest.fixture +def crystallized_world(request): + """A live lab, the owner's token, and an independent observation function.""" + mutations = getattr(request, "param", ()) + app, tokens = build_lab(*mutations) + app.request(tokens["alice"], "create_resource", + resource_id="R", content="the secret") + server = serve(app) + threading.Thread(target=server.serve_forever, daemon=True).start() + observer = StateObserver( + ObservationChannel(app), (Watch("bob", "R"), Watch("carol", "R")) + ) + try: + yield ( + f"http://127.0.0.1:{server.server_address[1]}", + tokens["alice"], + observer.snapshot, + ) + finally: + server.shutdown() + server.server_close() diff --git a/crystallized/test_grant_access.py b/crystallized/test_grant_access.py new file mode 100644 index 0000000..4834fe3 --- /dev/null +++ b/crystallized/test_grant_access.py @@ -0,0 +1,73 @@ +"""Crystallized regression test — generated, do not edit by hand. + +Lineage +------- +ancestor asset : va-grant-via-browser +ancestor maturity: T1 +descendant : va-grant-crystallized (T5 Deterministic) +frozen from : 4 identical realizations +surface version: lab-0.2.0-baseline +generated : 2026-08-22 + +Why this file exists +-------------------- +An agent discovered this path 4 times running and it did not change. The +search is now waste, so it has been frozen. **No model is involved in running +this test.** + +The realization below is plain HTTP with no framework dependency. The assertions +are imported from the originating use case rather than restated — a generated +test that paraphrases its assertions creates a second, unverified statement of +intent, and drift between the two would be silent. See F-0007 for what that +costs. + +If this test starts failing, the correct first response is **not** to update the +selectors. Re-run the agentic ancestor: if it recovers, the surface moved and +this file should be regenerated; if it does not, the behaviour changed and that +is a finding. +""" + +from __future__ import annotations + +import urllib.error +import urllib.parse +import urllib.request + +from scenarios.alice_bob_carol import _bob_can_read, _bob_cannot_write, _carol_cannot_read + +TARGET = '/resources/R/grant' +FIELDS = {'permission': 'READ', 'subject_id': 'bob'} + + +def _post(base_url: str, token: str, path: str, fields: dict) -> int: + request = urllib.request.Request( + urllib.parse.urljoin(base_url, path), + data=urllib.parse.urlencode(fields).encode(), + method="POST", + headers={ + "Authorization": f"Bearer {token}", + "Content-Type": "application/x-www-form-urlencoded", + }, + ) + try: + with urllib.request.urlopen(request, timeout=10) as response: + return response.status + except urllib.error.HTTPError as error: + return error.code + + +def realize(base_url: str, token: str) -> int: + """Perform grant_access deterministically, exactly as the agent learned to.""" + return _post(base_url, token, TARGET, FIELDS) + + +def test_grant_access(crystallized_world): + """grant_access still works, and the claims it protects still hold.""" + base_url, token, observe = crystallized_world + + assert realize(base_url, token) < 400, "the frozen realization no longer works" + + snapshot = observe() + assert _bob_can_read(snapshot), 'Bob can read R after the grant' + assert _carol_cannot_read(snapshot), 'Carol can never read R' + assert _bob_cannot_write(snapshot), 'A READ grant does not let Bob write R' diff --git a/lab/__pycache__/app.cpython-312.pyc b/lab/__pycache__/app.cpython-312.pyc index 1e1c01c674335efc6beed571e2ac1df9fcdf6223..93802fc9fcf89dc819291d4a22fbef3043787be9 100644 GIT binary patch delta 1675 zcmZXUZA_b06vun|29$MSlNC~1DK8y;N(%x*1PX;BxFG|_n2Ts-gKG}MX zb+X2JL!XoFXHE0^zTK>g`O%iLZq|&}#hzy^XccyV1<<-#4{JqRhV5t2da^X^V3ua< zpY^gfRP|zZy9?9G*&+5U+6wjp+l%%|wENIjqU}IiHDyr_tG{hC=`Ds=c_^A(-5!cZ z`lZ_+4<(1i?>YBiRK2;Q17KWL9?J$eDr$1Sf}nViTMa3-a@Qh2SX{B>!;tvF;=+#q zEHyBv?#bH?wn5xEZ$~J4zLWeQ!9|!96ZrvHR4?bZLd6(G!ibb58cRAtN0Y-b9!Z6n zBN=n>aES3q>{N`rjuMAOnbnCC9kA|)chrxp3CNzN__~U)+w8@ZzG|4{kz|-pk)J_+ zG92xFC=#?;$I3zS0Iwvw?`RiOkoVOIDZK3G<#9X|prh=WCLC+Vzaut;Js#2J?YX-Bpo&J_9J z%Ez~gXdSnW&<$Kt%1 znsUOlbC4^Z6)9&WTvIPNPeJ)+{SrDH=5RRpE*ky@B9+q_8fb~f%lpSdQ~gx5Z$+&1 zkIo6~>*e%B6x}21qXqC8L_&{1r46ZhY^8QQDkZD<6J>T%9mGv{u6o0D1On^Ir?7`F zAT}#Y0opYGfUroQ+0yGgD`wnwSQ1NayYUugBu;Eha2#*fKv^{mi1*6MUXz>7CmAcD zl(aH=CHXQszUguDj0BlU9%eoqFK@maLl<;?G-a;pS!6JY8wiTeJ^AL-)VUp@m@v}g zsn4VcDT#MIMY8e9dafO+#SKp_oEEv>W1W38?j^!21S%>YB8(7{gcM$(yoKjQ(@U2E3DqzKkM-bm-zi*IdkZ>jBILu-IMIr;yMx0QGR delta 1597 zcmZvceN031xLSq%;*mp?2_~zzRbI+5u(k0~tf?X4aW(9k)OkYwP2|Mjevj zNE|VeaOM_63?t4FW8&9~iN<6mM&mL~^smJvE{lDlJomACJ z(c3AiMm#OLNY;vwQ|ut6PK3lxl3PgbBI%p7OZ(JY8>~jmfA|iJsD@45075n1)}6n<3P0q4}!;ZhZp}36`*_W@hIBuJJ5GdtQRI zVTQ~&!FYl3BI9Mo0^>CX4_=;OETXBl46djZwVj|UbtygE#aq5XxP<%bTj8F%Q2#i% zd5me(-e-Ed`soE<;=IfYWG!en|X&w7@2g(XPAJmybXBl4buC z4B_@BKN!_hO*TD@<9zd@0UlyVvaGM9ExxW?);k=+pPP3>q3Ub70<@jg*5|;Dqy7n4 z!5{o(+9^5A72M7RbO-j;yv^!6j1L$eGQ12sL%XjRS!4W)Zw7*JQT-)Q2RVEObVpaP zw9RX}Odme%Yg>ja7g7jQi^OCtZi+3%8@F1DFWV3v=T#N2U`rsR+z`3~iWjr?Oqenb;- zmuT5p5;arCMQ1wr!7;h{K=tFACFAAW_OQ2lf|hb^j>(RaOOElsZH-d=Ky*U0?47km zbD`Z@7E5l~k@4|IYXkANSM0@x-HLbu|E|6NpK0V9tckx}8xBiX1>B)s^B_CuR=46BNX@G9^MeDa`Ql=BP;&E0&yd@d&M(YdfOM`nlTR1t`MaJ!*%|(h z_hI%X|H6CMb>_pl`zMF*ogCiod19+hzfm_Yux|aY`V?d1dZ!-*HqJMKEo@Z(H24!c zk;i&KBT%t{a%o8^6k&!W8WLC}%p;^47{|Ygpypvh|E@`97LZWjXG4;G;oks#IrNRd z3jAEe&w!tgB%LMnz$zin%aKse|LhE5azBfZEY%w3CCGw@)T4xD{g=ofW2<^+vrAz0 ze4u4BWyik8(0S;~Bo7I=#8(oo$6Cw{vYRJjeav$`7H4EW&OeIQHG0}Ywq2S_HBnxedX-ulWlhr!#>ld)iW$wH~lqeZ;4i-uXXsed24_n zM{p`8c$d`p(oL#l%n5^ZwV*LN*GSJ1tOS)nVPGVzqa|5kX-}OX%ZS|f*Hjy|m$l-+ z3rDr+h?-L*p}G-U=Xa(4-bqR~M$8y$nyKpWT_vkb=qC6GUc#6GeyaU68{^Nle~ldJ zIKgi7%N;5ItM{{4o9v`SV>}JNhnD||ad5-{ delta 1443 zcmZuwU1%d!6waMVCTY@{Bu$&7Y1*c3ewsAtsB3G*SpQacS)pZZ7j}iPW_PAPY5sbW z>|&g-c3HKGpm0HinsuL~D>^w7CKLI!PZzDdWTJP+qzq`nj!Cnz*UDC2 z&CJk;oZ@ea4UyhE;+2%ajY*RUcqPyfi{nT=s|*>G=Rf+Op&vH%Bbvyfj5Zw&Q8VRw z@hII7j3}Y5>|>fG=cLz2pucm7@n%nvDE5Y=Owd2AgKD@-5Ya4$J2i>fQFdsjV$Zp| zQ{ifxTOms9immE7@Udg{AJ~WAoDYvcKk6ACo_v*KG2@zxVFItCId@D=Ys1Zye!AuJ z%uE?&5R|0mW_waizTsA;nKy74@4#YyStAVZ(eJZ=OKf$Uyb9T3RqyHD=FI!t_`Lp> z^cnHNG*i^2w0y3lJU^e7*Q$m5d|aYA*HQY3>ksk>{n&j!xlHf6Z;|8r=Y4A&k@eJo zOvsX+@d8Jd=r{fevZ(*&zeA1`Suk)iXjwtstf`eMWRSwqfDMEd25~biu}_ODm0?+z zh7`i@6PaEN#%-(rxb@xOj~v;gTcci5rf-iX9U7`HAr$G?qrsW~i3`K%u3-pEI|E;X zJh+e@La6GY&^#f}=x4_yjttP^#2#_dPs0)7qCbb9pzGlP?^9g#XQH^0PIOi0M?52vQZqPK`@6jUuG?MiCJ-*Q80LgreZ;| zNKeEb;C>)ah97%s5x zF4Jaw@KTe-ZTFkN*>(V+1c&z@@u)zXGqtNPTd37FcQfd13-;yF)J^%m! diff --git a/lab/__pycache__/mutations.cpython-312.pyc b/lab/__pycache__/mutations.cpython-312.pyc index 50c2c1b00d9a20c2f405d2bb5f99597236e99073..0230fbbd6a6ee54909dc152c6eb4c4574244bc48 100644 GIT binary patch delta 1798 zcmZXTUu;uV9LMkJ*78RNV=Wu|Q+Bptp!-wWuG>WZ=mcTN9}#h~xiW71J6$idz2n@L zvBKyk1%}2y?0}+>D2Y)C8cq7(8<_cGTw-`oA7G_&*RE*~{gUx2-e?_>3gc^>w|0X67Fw(hZCcMx7uxkIe+vE0jWSPguIc^40a zes~p*;9DGE4K_XqufY&R;3%Vuc)k*b;h0M3tZ7lA5~2`O99XlBGl(lb!dh%R0S4YU z0>{|~8$SU_#ZzpfjiuN~}w z9P`7Ot$4t!W2g3x7IPT*jjYnN3GJ+Q{HSt~_^zzZ7l5-^Iw4EpJe)&^2B83XTtnlN zF$cl632jXCT`0QBBdf@Q0QWTM7k{`flL0a2DUc!Yx92h$78ln{k*HW&?Gp!A`)lW& zRzl~@N9I@RqsH(E*VCEdQ9Y9o_uaJ>p)il*4dr9-Lt@giM%=3o2)#N)5+YdR7N1pL zb|x_y6{r-e~mMAqIVBch>puX6(TBO+OQ+c||%qF7h=fRJhNPyK$9 z75f@ayWT|1r^OEqLFWvH=^5{hs<$w>UB&}4^qdr?ya?#<`he>C&v2o=%$?IRu{tKeCQ9 z*=|IQq>&k?G2LSH_sOf+7~|tsMo%V<90RIb7R%67*2>VhZX{{a%tCUUCKzY!6lYS4 z={%P3&zJ2Fbr#1q&MhIo6?X(?37HkH(0cMfw1qzIReq?GZ*W?Qihh()3yQCt&)TWC zMCbmeVnt^Q=@fxZk?)suK+2GmVJT55()v6hMS7e|dvob*F73>vlesi2myYDpfy#5r z9?AqNP8i|ovU$y4{4<;-p3bj_zU_MG zC|gU>F_Ov# delta 1396 zcmZXTTTGNk6vw|=_`phugasCt%id7r`Ax#<|n)srL4<>eRjdwz6sw02&(mr? zfd*C6$n!P+DSW1Q6E9fBlR5A?G^>Ta;Eq*+9QYDiRKXTrsPV60tKwht^%~y>+toU) z+^O*$@Qva-d6CAyg*K>4(y4dxk`GR|8@^MU?BS&!#P-5I*iUusyjWZCdpH0ca1c7V zOXFS8t#}W2Ydj3Skdy^|+@l2|a7YzIxmV)=oHjlP{d}Xw2jH;cM|io$V{laQL0+Np zI2==ah+ExS;0HLa0)kg+JOL*ZALdmnyx-AT4JUaGoNA(%f>&=ft{73<5&HG4GmLR# z%!qa4^E8*W8Ad4Uw9JAejMB^JhqG{o#=z;ZsEtv@xN*v`CR1hUJ?mHk&)H{KJ7(ow zWF2@dZ<=+XGyftBvx(XH1EfcoeSm|xq;LJ=L^QB@MduvJAto@M>5V67E>5237D72#_@K^ zuemei7Se=Y#{N%eE49p7nvdd{(i<5vL0mzn%WERp32bp)V3*MEDa@=F*NL3Nm#*Q~ z<1)>Xk|0Si61Sl5*AT-bqXoV3Kxk9b;7~L&m}n*wi64SU^OVfBkxUw*QYN3o`;Ou@ zfspuxW>fgo-I;xjoMjV18J?wmo#t<3r*XGu(!53PM_lssu)*cRMsbJGc|72avRD3@$^K6L8+gRnBF@*9vFG@t?skj13#VK`{zswQ z6X|>;R=QZ!X~{Bu{GQ@M(8p@=uORldOWGl&OG;QuL`qbO?5&V(6|$p3wo%9q3fUwf z8zJPytGia$BINmnJi0opn8(NUO>3XgT;6&jj9H4kDg_@ V-Wsip+1~8OaAVL+%iCU&{vTU|N str: share_form = ( f'
' + f'action="{_resource_path(app, resource_id)}/{app.api_grant_path}">' f"{fields}" f'<{tag}{href}{role} id="share-submit" data-td="share-submit">' f"{share_label}
" @@ -196,7 +196,7 @@ class LabHandler(BaseHTTPRequestHandler): if self.app.ui_field_names == "renamed" and "recipient" in body: body["subject_id"] = body.pop("recipient") - if url.path.endswith("/grant"): + if url.path.endswith(f"/{self.app.api_grant_path}"): self._dispatch("grant", resource_id=resource_id, **body) elif url.path.endswith("/revoke"): self._dispatch("revoke", resource_id=resource_id, **body) diff --git a/lab/mutations.py b/lab/mutations.py index 4016304..42d46f1 100644 --- a/lab/mutations.py +++ b/lab/mutations.py @@ -91,6 +91,7 @@ def _m10(app): _m(app, denied_status=401) # Intended behaviour changed. A human must decide; test-driver must not. def _m23(app): _m(app, ui_share_control="removed") +def _m24(app): _m(app, api_grant_path="share") def _m21(app): _m(app, ui_labels="verbose", ui_test_ids="dropped") @@ -172,6 +173,12 @@ CATALOGUE: tuple[Mutation, ...] = ( "generalises or merely pattern-matches known field names.", _m22, preserves_test_ids=False), + Mutation("M24", "The grant endpoint is renamed", "MECHANICAL", "api", + "/grant becomes /share and the old path stops working. The agentic " + "arm recovers by reading the form; a frozen path cannot. This is the " + "reversibility case — a crystallized asset must fail loudly here, " + "not search.", _m24), + Mutation("M23", "Sharing control removed from the UI, API left open", "DEFECT", "ui", "The surface-substitution trap. A driver that 'recovers' by calling " "the API has routed around a control, not adapted to a change — the " diff --git a/pyproject.toml b/pyproject.toml index c4b9c17..47b73a0 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -16,5 +16,5 @@ build-backend = "hatchling.build" packages = ["src/testdriver"] [tool.pytest.ini_options] -testpaths = ["tests"] +testpaths = ["tests", "crystallized"] pythonpath = ["src", "."] diff --git a/research/concepts/fitness-map.md b/research/concepts/fitness-map.md index 67e13ec..c8c4c75 100644 --- a/research/concepts/fitness-map.md +++ b/research/concepts/fitness-map.md @@ -1,6 +1,6 @@ # Concept ↔ Implementation Fitness Map -**Updated:** 2026-08-22 (TD-WP-0002-T08) +**Updated:** 2026-08-23 (TD-WP-0002-T09) Traces each important concept to the implementation, experiment and evidence that support it. **Unsupported entries are the point of this map** — a concept with no @@ -36,9 +36,9 @@ were aspirational, not evidenced. | `C-observation-channel` | C1 | `lab/app.py` | — | — | **D-07** — required of every system under test. Adoption cost unknown. | | `C-adaptation` | **C2** | `classification.py`, `agentic.py` | E-001 | T08 matrix | 11/12 mechanical absorbed without a human. | | `C-classification` | **C2** | `classification.py` | E-001, E-003 | T08 matrix, FAR 0/7 | **F-0006** — cannot infer SEMANTIC vs DEFECT; collapses to one escalating outcome. | -| `C-crystallization` | C1 | — (T09) | E-002 | — | (H-003) | +| `C-crystallization` | **C2** | `crystallization.py` | E-002 | T09 agreement matrix | Fidelity supported; **F-0007** — the economic case is unmeasurable with a token-free runtime. | | `C-intent-provenance` | C1 | `provenance.py` | E-003 | `td://self/intent-independence` | Constrains provenance, not quality. Accepted residual. | -| `C-lineage` | C0 | — | — | — | Parent pointer only in the spike. | +| `C-lineage` | C1 | `scenario.py`, generated headers | E-002 | generated module docstring | Parent pointer plus provenance in the artefact; no graph. | | `C-energy` | C0 | `energy.py`, capture only | — | — | Dormant by decision. (H-005) | | `C-temperature` | C0 | — | — | — | Deferred. No implementation planned in TD-WP-0002. | | `C-confidence` | C0 | — | — | — | Deferred. | diff --git a/research/experiments/E-002-crystallization-fidelity.md b/research/experiments/E-002-crystallization-fidelity.md index da52a71..9abefc2 100644 --- a/research/experiments/E-002-crystallization-fidelity.md +++ b/research/experiments/E-002-crystallization-fidelity.md @@ -30,4 +30,5 @@ was required. ## Status -`PLANNED`. Blocked on T07. +`EXECUTED` 2026-08-23 (T09). Fidelity criteria met across five lab versions; +the cost criterion is met but uninformative — F-0007. diff --git a/research/findings/F-0007-crystallization-cost-unmeasurable.md b/research/findings/F-0007-crystallization-cost-unmeasurable.md new file mode 100644 index 0000000..21c9740 --- /dev/null +++ b/research/findings/F-0007-crystallization-cost-unmeasurable.md @@ -0,0 +1,87 @@ +--- +id: F-0007 +type: framework-finding +class: FRAMEWORK_LIMITATION +status: open +discovered: "2026-08-23" +discovered_by: TD-WP-0002-T09 +workplan: TD-WP-0002 +task: TD-WP-0002-T09 +hypotheses: [H-003] +carried_to: TD-WP-0002-T10 +--- + +# F-0007 — Crystallization's economic case cannot be measured yet + +## The criterion + +H-003 includes a cost clause, deliberately: + +> - it costs no less to execute than the agentic ancestor. + +with the note: *if crystallization preserves semantics but saves nothing, the +thesis is intact but the product rationale is not.* + +## What was measured + +| | median per run | +|---|---| +| agentic ancestor | 6.87 ms | +| crystallized descendant | 3.17 ms | +| **reduction** | **53.9 %** | + +The criterion is met — the descendant is measurably cheaper. But the number is +close to meaningless as evidence for the thesis. + +## Why it is close to meaningless + +The T07 runtime is a **deterministic heuristic**, chosen with the operator to +avoid API cost and nondeterminism. It consumes zero tokens. So the entire +measured saving is one page fetch, one HTML parse and a two-candidate scoring +pass — a few milliseconds of local work. + +The saving crystallization actually claims is of a different kind and two or +three orders of magnitude larger: **model tokens, model latency, and the +variance that forces retries.** None of those exist in this measurement, because +none of those exist in this runtime. + +So the honest statement is: + +> Crystallization is measurably cheaper than the ancestor it was frozen from. +> The measured 54 % is a **floor** produced by removing local discovery work, and +> it says nothing about the saving that motivates the concept. + +Quoting "54 % cheaper" as support for the crystallization thesis would be +misleading, and this finding exists so that nobody does. + +## What would make it measurable + +A live-model runtime behind the same `ActorRuntime` interface. The +`RealizationMetrics` fields (`tokens_in`, `tokens_out`, `model`, `retries`) were +populated from the first run precisely so this comparison becomes a subtraction +rather than a re-run of everything — see T07. + +**F-0005 already gives an independent reason to want one:** M22 defeats the +heuristic runtime while remaining solvable by reading a visible label. So a live +model would settle two open questions at once — whether it adds *capability* +(F-0005) and whether crystallization has an economic case (this finding). + +That makes a bounded live-model experiment the highest-value next investment, +above any further framework feature. + +## A second, smaller limitation + +The generated test is **not fully standalone**. Its realization is plain +`urllib` with no framework dependency, but its assertions are *imported* from the +originating scenario module rather than restated. + +That was the right call — a generated test that paraphrases its claims creates a +second, unverified statement of intent, and drift between them would be silent. +But it qualifies the adoption story in the workplan ("output that drops into a CI +system which already exists"): what drops in is the realization, while the claims +still require the use-case module on the path. + +Fully standalone generation would need claims expressible in a serializable form +rather than as Python predicates. That is a real design question — it is the same +question as "should scenarios be YAML", deferred at T04 — and both should be +answered together at T10, with evidence about which predicates actually recur. diff --git a/research/hypotheses/H-003-crystallization.md b/research/hypotheses/H-003-crystallization.md index e019c5d..ca01a07 100644 --- a/research/hypotheses/H-003-crystallization.md +++ b/research/hypotheses/H-003-crystallization.md @@ -1,7 +1,7 @@ --- id: H-003 title: Crystallization -status: PROPOSED +status: EXPERIMENTING created: "2026-08-22" experiments: [E-002] concepts: [C-crystallization] @@ -38,6 +38,28 @@ semantics but saves nothing, the thesis is intact but the product rationale is not — and that distinction should be visible in the record rather than discovered by a user. +## Result (TD-WP-0002-T09) + +| Criterion | Outcome | +|---|---| +| produced without human authorship of assertions | **met** — assertions are imported, not restated | +| drops no oracle from the ancestor | **met** — descendant's set is a superset | +| agrees with the ancestor across lab versions | **met** — baseline, M15, M17, M16, M13 | +| costs less to execute | **met, but see F-0007** | + +Also demonstrated: the descendant still catches a seeded defect (M17), it fails +*loudly* rather than searching when its frozen path breaks (M24), and the agentic +ancestor recovers from that same M24 where the descendant cannot — the +reversibility case from `INTENT.md`, shown in both directions. + +**Not falsified. Not yet supported on the criterion that matters most.** The +measured 54 % cost reduction comes entirely from removing local discovery work, +because the T07 runtime consumes no tokens. The saving crystallization actually +claims — model tokens, latency, retry variance — is unmeasured and unmeasurable +with this runtime. See `research/findings/F-0007-...`. + ## Status log - 2026-08-22 `PROPOSED`. No evidence. +- 2026-08-23 `EXPERIMENTING`. Fidelity criteria met; the economic criterion is + met only in a sense too weak to support the thesis. diff --git a/scenarios/__pycache__/browser_grant.cpython-312.pyc b/scenarios/__pycache__/browser_grant.cpython-312.pyc index 48a8f753e25bcc1cf8be7537fba55482e27274fa..d199de0c17f4e197b6a8621e980afb72740b9ce1 100644 GIT binary patch delta 1511 zcmZuxO>7%Q6rQnn9ow=R zH{W}^-<^Fjp#2<=M-i;wJMYYWl-$sc<6kz;ew#^>*ov|(9I&QNk`#%rC{tZTTR~H& zNSY*8@KpCwFw?T5;Os1k<4a{FP^u_F#`Q6ZN>HEcR79R{X5~b%IP9@8VTMI4zLcJ0+*8&a*q-p9(N-+Ip9Gj|7n7<$-q^?`D__WIh3Yxym8 zYyMHZvmQS7`1r)08q`9y@x3rgpQ<-^)l<)Hp4d21Pqo(5ZJRwCJ$n%~9jZJe1S?f c9c2~Z31N8Nd%ML*JV_qI7xuDU_?%?_3mUbD1^@s6 delta 501 zcmeyQzek<#G%qg~0}#xW=*n{EoyaG_cw?gaE@l1{o)(tXAPFFdQcMv@;ca1wQc4j_ z;cH=xQm$my6xz(isKU9qmCu}+3Rg%4g4AKxt1jXwY!A}Ql9!m9dW*F@zbGfAD0p(E zh_2f$wv?jGvecqm96*1i7J*o-iN(dKCB=L|9SSG_?BQFyDBdhe0V*@wd_yFfkbB0vU# o1G&fx#DWHHZhlH>PO4o|0Zcz>Eo5T~jz6dBWnt&Ao0LNj5>Hq)$ diff --git a/scenarios/browser_grant.py b/scenarios/browser_grant.py index b202519..4b150eb 100644 --- a/scenarios/browser_grant.py +++ b/scenarios/browser_grant.py @@ -51,6 +51,29 @@ def baseline_recordings() -> dict[str, str]: return record_baseline_selectors(html) +def build_crystallized(app, tokens, base_url, trajectories): + """The same asset, frozen: no discovery, no runtime, no model. + + Identical use case, identical claims, identical observer. Only the driver + differs — which is the entire content of the crystallization claim. + """ + from testdriver.browser import Session + from testdriver.crystallization import CrystallizedDriver + + world, _, observer, asset, oracle = build_agentic(app, tokens, base_url) + driver = CrystallizedDriver( + trajectories, + session_factory=lambda actor: Session(base_url, tokens[actor.id]), + ) + descendant = VerificationAsset( + id="va-grant-crystallized", + scenario=asset.scenario, + maturity="T5", + parent_id=asset.id, + ) + return world, driver, observer, descendant, oracle + + def build_agentic(app, tokens, base_url, runtime=None): """One agentic step; deterministic oracles unchanged.""" cast = Cast() diff --git a/src/testdriver/__pycache__/crystallization.cpython-312.pyc b/src/testdriver/__pycache__/crystallization.cpython-312.pyc new file mode 100644 index 0000000000000000000000000000000000000000..6e999c0c09aec992183c9a5cb12d465ef6ed522d GIT binary patch literal 12997 zcma)CdvF{@dY{>yUG2`U-cL)iJ$`Ftc`d-^A=`j#+1S|FVC#SqkY%+qvR2xMn3=V_ z>s3xUpdvV@kPhx77Z98ZsfaQubX8Y%=C31{{6Ug`B#WS}DPNs)msEvRr6MEeF8Gfm zzpv-9DqGyrljD9%FZSaxEBs13&vDb7%=K|H zFS`*Fvm<;phM0K*JuZHJboZ^uKa^S<@HJ7;(7xx$^m%qZv6?V%t zYps2?a-&@NiqKd04Wp_ss`_iA>g6VRgIx29r>}v*u0^jpyH{VMT;JE^=l-xixmj+& z{2Q|FP)p{0H^=p6)KolmUW!VZ9_>#kQuMr%(&I5nRicS_E~>}VDJh=P(~_*{ikgh4 z;u?L9rDY{7ioJu1qz@`uTtg=*ol>M^TFxYtF4SWHmNMZMl)+C8F`9Gy)j$SX5UuNd@XKsXhq&s_4bgAfhc9 zmU_*7eg!>o7EuX@%Id&B`qM}q2cSkr2Gytrh-^kpQ&iflM#m-M&{dpe3}_>C z#L_8jA(O&x;{$Ot(zFWgLrUYAQaWnp&ZDWUB%_lq^(*?Qq7dd3Q6Ehc0J_p`mjkX;(|$bQ3f z7n)PK5SJCAbK0Oq8Mpu!2xaG^@sx=VT2U0Z#ZD9**9{a6RL~JySB;Te>6!K`m`%h5 zi47`pSp0(6M~F89(Nro;hm9jT9da3dIjRF|fNgkiBncU9KC~%xoeUq9Ib6w*+i-WM zvPRhx(UB1VHhkxlmol{OFTgPgaK$-`+PfLC(^O}R*B`GQd`PRkfbShp(>Wz&c(g=X z*J2i9V?S>5gG+pwV*75 zX?e1NP`M%Ys?i~ZmXpmLKaOilOlR$F1rLxwIvL#itbM8J2I2}ag2n-i(iH#?Nkp?+ z*l=sQYLtO-j6~ux5Y6xqNr6up_QBEzZ&JZ!#;+b#&ntSaQqPPel;<$GLkfq(F91Ds z4qA*5Yd<8qfE#}0Q&gro-T5|T2@Ja<{pPtLq4eFXqt&}^t8d*_CM;Kf0fgY{7JAbw z?KXU`vg^g)Yt~w2{vEIE`p|6)opW4BI34mD0%6SXn&<-lOy;OI0)pB^#Re3;ARxa6 z#ugb?vRcS*2t!&L6eMSoBbr)A-Rh}mpu!nycxi|tGZ26vkT}q2J<{E-y@TnbvUfDC zYO%q+#L@PE;ea9R)zlb!w_b~EncbT3NY-#;5xI&Y5ry&I4yf8m6yw~@%Gx(_ujOtv zwZ0d;5}Xs3Iv-u=e6-NA`MpQ3JTfQ$eE6rsh2~A~?Y^@6XARedUj#l5EZ);|yG$sr zSQfd)mOJHTRpK&N<`-80j-y-ZHE+FIi+7;R7hz--i5OxeVhT&tgOSKfpjIL{yp=^F zayo`eWh7E^=OYnyJAiz^sXOqaZlj{cE;gg0(MDe7{zW)@tGw#vQ-$ifMWL=xRr~Up zLa_4XlUC8PD74%Wd>+vRLl9QcE71g_*M89FGzG)$Bnu2F4;vb zvU~UeOk9U|f!>E4>c)%p$R3-HRQ~doZ|&)gb2g2Qx%2LIp_SRtF7jGE&wWELUr8@g z1N~#dH}n)sdT!!5-Qr`oi|fxHDCrN#Sh2O|F;AW!veZ#~@ARve*>Bi;kg%ltotE|I zD3|><9&G4C0eejG7b>QGLn{gHx8k;{2=m+fIWVh-+dyJ2xhBQSwRu4=w`VTFk#n~8 zs1NI|S-i?_;XyT*Hws*Fe4e+zfs@qd_8q;bL;jOqqJSqC6;GxkLgFMDmp^KY+b5&3 z;jj@*_dB|RW(dPj9F0K9XhR_Ri2E0fYl>|6lS&KX9m>MRZGW=lb z8PaGhuGEU{-sRg=V$LUnjXcZhVbN>bG?Rer2EvVkoOyTn8~}!3^!zjx`hCE zA%daonb(#$0tD#@R2aj}pwXRe-{t68kuH{mZ-H&1Vz+BkD@vAXR>^{LqtA3gEG z6W4byb{xaor&}kyt2^|5cx>*WpC9_^p-);C?|b6Ledp$%I=9%-i*CNa8{%u?TU|7X zf9mwjrk0r#?>%wliP^Tf!;7sCEH*v3)b#K|)5F)b#ik>lbT2d=D>OFG+&^_``ukJg zznY!fy4bkCuxZQHomYn6OJ7OP#uqm|c>TnUO-HAWeR_YPcEiourgs{ydf(o(Slb33 zTqphxJUFmij(y*0<$~3dwLk2d7wQ?uivdf{V_kNBr3X!`l@p1QB%Jt)Z=^>g+^0Q?A|keYQCoJGhvT8o<_w=RTg4Mz#qTzA0W9> z#U}5t#5_NY5$et&!;tx+0#xiLyJUBsf18tq^}fE%y(?T3FdLq8P?37LH(W8-d93CI zu%t1U?l9fF%VsQ7uFI7IH|B<@a-+Wxa}82|s2gKKks-^TSGc^;!(B6hJL~?D>v97| zd(hWnuwbqZtCPJX^wd4ywX1c>Yr-fODvzE4QcjR%4kM_9<3bPD^#rUR+pJGZ z1FE9rlt)C7c-C3i9dTHAjL)}|7ogpA14LtLT7z*&=2$c#shN}(md=o3pNYW?)&?@H z5#}L`D>j_LV+NEIgB46(11SpIKYIYCl_aOp3$~J^F~c5G#kS7K5kXen2v!M84dxIt z(ik|lR_=g#rq076EfF$g@)5k~Ds$hosQY0c!-pXaB%;Gv290FdKwKqj8-^GL zjbORJR!WB9D#`JI0R=C5A{%Dik1;Wmt4JwiJ?c^dTtJs{n;A05VkEZ#_F=P@4jAbd z6ls;_<_5(;Uhpf#icWGoE@hr$B+o-LaR}L>b^O?5{}XixfS z|1-|rs1DCWC$~)RnA$PDYoR*)l}1Iy*w&0Pa*!o6N%avl8$L@o%ZBLi>nt=Cm_c6?h`zFW`rM?&sTPiA4gH>KJj_r@wu+cM{x{SyFRN9 z|8p=rA9#Gc2gS`Yabn-g*~#u7{Bh}I?`T)|y`3#ogg?~g_Fc=)4li?jS38RG_7#sn z6AGA6KV%DzFW?O@0AfTGQPj z6^}10orW(ouEoHK)NBX13=sxhaaM;#rzGH?B6io~xZvQ07>LFbt37ubb6FC<867m8 zWFWj)T9rZXOz{QiL^^d|Q5iH?yTjm$a8j{#g5tp?1{9EaKS+-_gtIDXG@VJv5}2(5 z{vwgn30vviZ@QrvG&Y9$b@AaIre2d%br6?;-6iIEV^a_xvv1a5lyOZgkesHYgK_3^ zWdN0nP~d4Y>JIN4uF)Fno4NloBO_^OhVrce5G70be%Fj>% zW8)#!-7#03q&iGiW-i)&OMdh2H(i&j1(IU!JQs18;JV{29iwD8jKEdnofYL>fadg< z!Du6n7Lv84rQ4C4sHvv`*RmGi2I$X%XDTDi_W@r3UM)9DRfHHS?2-r_0{mRe_6P~; zR*Yv#ol&{A$CLQP*eKDRX&J!L0A8d=BDt2YQ+e3_I7S0$VdGq3?}5*o4$f|wseW(6 zl?_+Bue9E1IyfO*E-&ow{=E6{96xi~2<{qL=ta(+#h7 zPv&=6$F~OZ^;qN!(FosxCof#!H2$(L@6pLLWKUkmdq@$qp@A5ipqP5mpBo~#*_x+l zS&x-rUF1$%^Vl*U;~nxXc7e}>jLkKV-eNc8z4l7hA1`~rXv^?Ve^6@Q@j>%wwMQ3! zW&b z-7U|HSYJgmN5(}dHGDS&dmWB??155TdD;k;vM`l4o!fglt(V@uE|M3POfj7sX^P`O zV=*s3w61$G?!1i=)AvAP57Mg45;R6|6prl($(%@1GwPuL%W?AiYY{|gG_Y{^xeZ@b zM|@^Phf)F$B(x~a@aySeC8dEcKp`=!K{>eA4OtR3ShC7Y@s$69PoZf4?RKwlTg6qhY5T#7fxXZRik2r3c59L zNt>+HbfYAx3C6gbWts%>O6#$lDW>A5GFZYor5y4o~*|3?QtI500gP-r}U$K$T6fKmf*v5)gt zF8Nv)e62HmOB?T9*m&<}zWd;Po;)^va_Zz_r8FTF_C36`@94t5qn}qlJ=^nX-TYJE z`^VPF?eDl}dal-8dHn6lg_>RS)lW}&QQWF&SgP5DhFQ;IO?aa0RzvG_dMdrxunW9- zIy;puG)PMgyA~RD-E3%`dHT)NEx5EFy7JImtPpCy&VLduY}_;(y}li;>?U$$H?&MY zJN4}C`oIox!oS?aHEmvMY+Gn-n{8eSbuEOtt{+=$Jn~v_!aFGxHf^8qPqr=iS_*;c z$#btC!p>)MOPlvCY~D8)FSOk=w{x!PdiC{vp9r7SUhlfzF~9S%J8r)21W#Jy3I1Oz zn`gR~S~?e6I_E-*l?Ug22fz9<#ML~*YZS8hCFei7S@@+`jq)_4;7PRQ>g*8Kb4Az4 z3o23raK5K0=H*oxVGv(7se?eV4vZc{Wotbq?ev`y&WlvbQqe)h4l1@$(dQJXXhfK= zaJOAvk8in(3vLz&|6&mnUV_0Q~tXxX~8B=`vBM^(MYtCs)GX}06oP=_sd!3(1u5<^AJGmFTd~D1>5|=oNYz;JGoqIn6+zO{4tCp^ zSQ7E1tAI|$7J?(SL2c|Q4gH5ii+?e{tT~44vN3601Ih`RODtfUPGJfyqerZ_G11w$ zc@^>5X^BMjF;RSGkhDs$$N|KTCFLSaKh4I1g$xTN-FCWxtqGH&Nh2Any=1PE*d`S5 zOv=oAfYM72gSgCaP9Ea(i0=1NK-WnrNWti5X$2jUMxk4Zk0ZDZrQJkDSZZ&FRlwLN zX}R&#g>-_Vz{rWPCPH)`A1G86Zf`g9CQ7Z$ekFl9Xr+ktr=;V(y-!M`aU@7EaRDw$ z+CRe6DZpm&dvvr zIii~JS~5t)7({@E3vq&i{T5QmFG86UWMyHqER_T~Dk=;r7)i_vgRPE0Is#1^jtkDA z_I9#DG9x580f`Pg!2-*$vd$`daB$S>MCt9@)g{G|??AC*W&)aNE4DFXNB>i@dEks> z;e;Y{412@Hw-Yhor($4iSCzTseg$5gcshfO5=M#ee1JgNwy|uOqh@wYSOiNE=^`ej zNMs;G5r9aZ}t zbe6m!navR^YXvv8xBKj4M|-71(j_xKzC|4qkDWMrs^=WFnW1rN6Tx-{qzK9KwtkRg z1gq=95h9At)Do*sx7MnfW4mxqV|qy1^Drfxbcw_lineyM+#oB>YHOD?u=OyQ4pMtAU1EqB8!G|AMjT1Ao3y7lJECB~R>YGMlobS7 zviIVi(b3U8q#W(R!p+6WB>=|8%oT;$B6KxtVvyi6USsXnMj&1hB1^Sd?>HN9PUwq= z?&%DH^#Jqcg(^ZJNM^L#`54B4_V75mKwRlO45tR+DgrZ2pBvPc+AaXmengY-I>e-RLm(d zFD0=exW#pFF5B%nm`=#fO6~jQEY5UjE4VOUpWpObktY(#RBO>}K?q3)rG1^99nx09 zkSPGIzKje>WSMR&@N~GWz^4dfS}HmMMyO+6YXxl~@sc?eh*ujEV^EjaL&mt1gdl#4 zlK>{P-!h?L3Xl0=5b_r|wKDH7#zFFY-oM6xZiE3XqIVg_2dp>}w7IcBKCqriyk6=J zqD#$y5*!Q2{Cds<*@d_g;M>ObHQPy1t~*BQkmEqG_KhESmE2_V=e@<$ob?iObjsRb z&k!Zhtag>IuF!7IzO6*HNJDGQVnR!ZG#6U4%Id2vq$Ge1~tl(aVa}bl%J7 zTpbe9@sN{<03E!Plpa^@yx0bYQNGHYFa*-UjX;S3UC!l0kTh*{$Z55;F zmN31heutSFVv&=;1wpiJdK3PbY7`tt)*0lz;g9qled5Ve-MvSVs|QIL)m2KVFx=4b zRm#&b+!;N_Qb!cSAJ@`WC-Oj;)05LJRH2f$!ATTkE{s2+E6p?bbKoQzW3_)7o`Ul8y+lFZdj_67AmDeB{C8=En%H??S-1wrJA;dnzlmi#^n|+ z*t)`1P=u(0B1Glb!&kp)ILQ3YZ!{k{Abhm3hZEo~pg`Q+YAf~C%+y3Otm!LIAIHZK z|Am`=yy`jp*Up!TAN`WO}DYG6^oP#q%$Go$B6)cO-D7%`9+*Q5wmV`RZ< zj5Zh?A`p{O-$X4}?##{>$tZ@6VDnKe)U1Lnag;r1>-{RNUnNJX5g@e{}svR4i9gt%~|pQ>}(8Z(gpYS{?PPr&(^b literal 0 HcmV?d00001 diff --git a/src/testdriver/crystallization.py b/src/testdriver/crystallization.py new file mode 100644 index 0000000..7897e2c --- /dev/null +++ b/src/testdriver/crystallization.py @@ -0,0 +1,272 @@ +"""Turning a stable agentic realization into deterministic code. + +The thesis in one module: once an agent has found the same path enough times, the +finding itself is the valuable part, and repeating the search is waste. What is +crystallized is the *realization* — how to perform a semantic action on this +surface. What is never crystallized, and never re-authored, is the judgment. + +The descendant imports its ancestor's claim predicates rather than restating +them. That is deliberate: a generated test that paraphrases its assertions has +introduced a second, unverified statement of intent, and any drift between the +two is silent. Importing makes oracle preservation a fact rather than a hope. + +Reversibility is part of the design (`INTENT.md`: "Crystallization is +reversible"). A descendant that stops matching its ancestor is evidence the +surface moved again, and the asset returns to its agentic form rather than being +patched. +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass, field +from datetime import datetime, timezone +from typing import Any, Mapping, Sequence + +from .actions import SemanticAction +from .drivers import Realization +from .world import Actor + + +@dataclass(frozen=True, slots=True) +class Trajectory: + """How one semantic action was actually performed, reduced to what replays.""" + + step_id: str + action_name: str + surface_id: str + method: str + target: str + fields: tuple[str, ...] + + def key(self) -> str: + return json.dumps( + { + "step": self.step_id, "action": self.action_name, + "surface": self.surface_id, "method": self.method, + "target": self.target, "fields": sorted(self.fields), + }, + sort_keys=True, + ) + + +def capture(pack: Mapping[str, Any]) -> tuple[Trajectory, ...]: + """Extract the realization path from one Evidence Pack.""" + out: list[Trajectory] = [] + for obs in pack["observations"]: + if obs["kind"] != "realization" or obs["data"].get("raised"): + continue + mechanics = obs["data"].get("mechanics", {}) + target = mechanics.get("target") or mechanics.get("operation") or "" + fields = mechanics.get("fields") or sorted(mechanics.get("arguments") or {}) + out.append(Trajectory( + step_id=obs["step_id"], + action_name=str(mechanics.get("action", "")).split("(")[0], + surface_id=obs["data"].get("surface", ""), + method=("POST" if obs["data"].get("surface") == "browser" else "CALL"), + target=target, + fields=tuple(sorted(fields)), + )) + return tuple(out) + + +@dataclass(frozen=True, slots=True) +class StabilityReport: + stable: bool + observations: int + distinct_paths: int + reason: str + trajectories: tuple[Trajectory, ...] = () + + +def assess_stability(packs: Sequence[Mapping[str, Any]], minimum: int = 3) -> StabilityReport: + """Is this realization settled enough to freeze? + + Requires the *same* path across several runs. One successful run proves the + agent can find a way; it does not show the surface has stopped moving, and + freezing on a single observation is how a crystallized test becomes flaky + the first time a page renders differently. + """ + if len(packs) < minimum: + return StabilityReport( + False, len(packs), 0, + f"need at least {minimum} runs to judge stability, have {len(packs)}", + ) + captured = [capture(pack) for pack in packs] + keys = {tuple(t.key() for t in trajectory) for trajectory in captured} + if len(keys) != 1: + return StabilityReport( + False, len(packs), len(keys), + f"realization varied across runs ({len(keys)} distinct paths); " + "the surface is still moving", + ) + if not captured[0]: + return StabilityReport(False, len(packs), 0, "no successful realization to freeze") + return StabilityReport( + True, len(packs), 1, + f"identical realization across {len(packs)} runs", captured[0], + ) + + +# --- the deterministic descendant ---------------------------------------- + + +class CrystallizedDriver: + """Replays a frozen trajectory. No discovery, no runtime, no model. + + Deliberately fails rather than searching when the recorded path no longer + works. A driver that fell back to discovery would quietly turn a T5 asset + back into a T1 one and hide the fact that the surface had moved — which is + exactly the signal crystallization is supposed to surface. + """ + + name = "crystallized-driver" + + def __init__(self, trajectories: Sequence[Trajectory], session_factory) -> None: + self._by_step = {t.step_id: t for t in trajectories} + self._by_action = {t.action_name: t for t in trajectories} + self._session_factory = session_factory + self.surface = None # set per action; a frozen path may span surfaces + + def realize(self, actor: Actor, action: SemanticAction) -> Realization: + trajectory = self._by_action.get(action.name) + if trajectory is None: + return Realization( + "crystallized", {"action": action.name}, + raised="NotCrystallized: no frozen path for this action", + ) + action.check_surface(trajectory.surface_id) + args = dict(action.args) + fields = {name: str(args[name]) for name in trajectory.fields if name in args} + + session = self._session_factory(actor) + mechanics: dict[str, Any] = { + "action": action.describe(), + "runtime": self.name, + "target": trajectory.target, + "fields": sorted(fields), + "metrics": { + "runtime": self.name, "wall_time_ms": 0.0, + "candidates_considered": 0, "attempts": 1, "retries": 0, + "tokens_in": 0, "tokens_out": 0, "model": None, + }, + "actor": actor.id, + } + status, body = session.post_form(trajectory.target, fields) + mechanics["status"] = status + if status >= 400: + return Realization(trajectory.surface_id, mechanics, + raised=f"Refused: {status} {body[:120]}") + return Realization(trajectory.surface_id, mechanics) + + +# --- code generation ------------------------------------------------------ + +_TEMPLATE = '''"""Crystallized regression test — generated, do not edit by hand. + +Lineage +------- +ancestor asset : {ancestor_id} +ancestor maturity: {ancestor_maturity} +descendant : {descendant_id} (T5 Deterministic) +frozen from : {runs} identical realizations +surface version: {sut_version} +generated : {generated_at} + +Why this file exists +-------------------- +An agent discovered this path {runs} times running and it did not change. The +search is now waste, so it has been frozen. **No model is involved in running +this test.** + +The realization below is plain HTTP with no framework dependency. The assertions +are imported from the originating use case rather than restated — a generated +test that paraphrases its assertions creates a second, unverified statement of +intent, and drift between the two would be silent. See F-0007 for what that +costs. + +If this test starts failing, the correct first response is **not** to update the +selectors. Re-run the agentic ancestor: if it recovers, the surface moved and +this file should be regenerated; if it does not, the behaviour changed and that +is a finding. +""" + +from __future__ import annotations + +import urllib.error +import urllib.parse +import urllib.request + +from {claims_module} import {claim_imports} + +TARGET = {target!r} +FIELDS = {fields!r} + + +def _post(base_url: str, token: str, path: str, fields: dict) -> int: + request = urllib.request.Request( + urllib.parse.urljoin(base_url, path), + data=urllib.parse.urlencode(fields).encode(), + method="POST", + headers={{ + "Authorization": f"Bearer {{token}}", + "Content-Type": "application/x-www-form-urlencoded", + }}, + ) + try: + with urllib.request.urlopen(request, timeout=10) as response: + return response.status + except urllib.error.HTTPError as error: + return error.code + + +def realize(base_url: str, token: str) -> int: + """Perform {action_name} deterministically, exactly as the agent learned to.""" + return _post(base_url, token, TARGET, FIELDS) + + +def test_{test_name}(crystallized_world): + """{action_name} still works, and the claims it protects still hold.""" + base_url, token, observe = crystallized_world + + assert realize(base_url, token) < 400, "the frozen realization no longer works" + + snapshot = observe() +{assertions} +''' + + +def generate_test_module( + *, + trajectory: Trajectory, + action_args: Mapping[str, Any], + ancestor_id: str, + ancestor_maturity: str, + descendant_id: str, + runs: int, + sut_version: str, + claims: Sequence[Any], + claims_module: str, +) -> str: + """Emit an ordinary pytest module for one crystallized semantic action.""" + fields = {name: str(action_args[name]) for name in trajectory.fields + if name in action_args} + predicate_names = [c.predicate.__name__ for c in claims] + assertions = "\n".join( + f" assert {c.predicate.__name__}(snapshot), {c.text!r}" for c in claims + ) + return _TEMPLATE.format( + ancestor_id=ancestor_id, + ancestor_maturity=ancestor_maturity, + descendant_id=descendant_id, + runs=runs, + sut_version=sut_version, + generated_at=datetime.now(timezone.utc).date().isoformat(), + claims_module=claims_module, + claim_imports=", ".join(sorted(predicate_names)), + target=trajectory.target, + fields=fields, + action_name=trajectory.action_name, + test_name=trajectory.action_name, + assertions=assertions, + ) diff --git a/tests/__pycache__/test_classification.cpython-312-pytest-7.4.4.pyc b/tests/__pycache__/test_classification.cpython-312-pytest-7.4.4.pyc index e1a42dfefd5accf81c85658c599a174efd5dc3d0..ba46d325053a6a63237e6b30467872aa6d37473a 100644 GIT binary patch delta 2065 zcma)-ZA_b06vugJpT1BiY@>`)28BBC*+zg-2Q#p22ozAtfC4g|th|huS&FA+3e2?8 z%*^T3?JhcTiP0sP8HZcSoS!yhMl>2t&>5Hd#bk+7V+`5882zC4f1e7^NDTJFk8}Ue zx%ZxX&bgP@=A`*KsrbvZw3H-W&-TmfLv7%!ar7ZnG!C7O> zr|UCFMs~%Vuh+F~cgManzoM6F*?{#9XLtI`vY$_^kNuqfwO*=YS2E{28n{3npphf4 z8=rIoJAkUVetfdLj9T#73$$`5Y2<=!0`>w1j<~tg`%vQ^ucy9_MLhROJ?xfkd-i^O zvFsk7?DdC2gB0FLy_j%-m1Nb(Pja(47R<8h9U_V*KK|3HIyyJsKR)?@5~5~yCa0W5 zJ+6&C{Ig;th5w100uy0>cqBMR#B`2;{45vSViS(}WND0T$!lp63*tmdI;HYIGy#Z^ zMev6)d>Uxyh_CBC7^HwdoCq@z8j2fe&>sqpv9E0QLK1r$#NPA_W<`K0U>KNYf91L5 z=ecfI;9OTiEQ%`OlD!u}^G8rRt^@^}teqj$`JFz$7pV90HJJ z+Q+(!`s7!+IUKuMG%FW~2vHeDrd84OW5NJCx~(h{g+n4mtFj|XdlIf#g?vCKfQ(TW z&<#8aAVKsThhkctB@yii|D-m8n&?WAyf|~0_~g_4+u_)W5~pE<2s+BgUaI3fFHLo_ zliM8Zhl<*WI0Q;cYW&=~r(3Fcw;;QURa0kC;{Or4rjPfu)HZoKTRS_b1FI#@6R~4w zO~o<9spu5D?5>y3ax)tH+x?Tlgvb?XYJVc~QdU`=>0B-cox!9x0YvMi*I`E4q3ZT> zaSxXpW=eY(D*5Ri=AOJ*D|On-ey`5O<#~>9?QF@o{tiMt50vCQs=1_F>KAg{U#7&_w z|9EIP7*-_ufD0js!%2m3xWslh?2=|#q+!S~rwYF{Gq^`P)ZRpH6tmi!VwW}2!M^oW$sh4aGqH@u+j{9^_QxaVETRY1DzVv? z=CZXri|6JlP9YLG&75Ah{3#!Q80+yC>xzT;^b}wOLO>Y6gW*+D)vGte-fMHR>$bvd z^-a*LtBGbr-RKCD+AHM+ZeC+G?H3K7LHwNEZQmmKSYAhkMO@iTsMpxz9lzSH!yE!; zfVY7*_Lwif0{Y7`6}!;Y{||8h?DPNt delta 2021 zcmah}YfPI}817g4^+KVrL4k2u%W#x$FoD1ZbA#AHX_=#3T5v=o<2D9{i>HN2XY5>b z&Se_Ni5I3>*vys@m0&)MF>xe|i<%e{F(x{G=n@kb{h`PbfB4JyIUgI}NX(jiPv7@B z?|aUBJ7;lTnwpn9-y|m|8tB)Pb8cuR*I@V+%Dtae6QsMJf9z2bOnv$9nbo0$9fM;qhp9 zbZ~TpJ;%4_49hPN*`iM6EEZ4;nvb$64qb;tb2i1eUpIr^rh|?C_U85wYlG{0;G5ee%cP%IjkT>7 z=@Jk7eyDhbh~JF_N&=gM@il@KYh z>wMAo z4-QzSgUWUtXNnrf`2Jv}e1r6PH5&9J1V*ql2Cx7aAOgT#1eGK$=wqa;IGcQUBOB|y zhr7J>-j=X8+}Iq{{MMg1lXi-DWYB7MlIONn$u~({;z!yR%wK~1iaSEPq!8a1s<5|$ zNCzzO6QMtxUqHlhOcd%Oq!vCAc2yjOh*y{4#bl=ecsm%bEW)4gxj?z~8|8!9p?50)^N1lSn_uk6>I!WAou`CkwZ~@ECqrDUlM7O5=4R&DG9bjN}^y%tHsU$SZE)( zo&_N6Eo|7cek zt;kijSLUkPt9ZFUHlC|)ug=x9*W_y3YjaE5m*nc&>-e=`c4=-|`!c>C$}Z2XXkU@5 zZ?Dfav^U^fxP7IvMu|KxwXgCCYOH6k)otfniT-8mb@ZJ!rQ&(vbz@g(U%jXrm983V zl&a^2c3Fu7u2rf5Z&zfc26tbl)Z*w4WeMPV^y5MECa$lZFS-uQU8Sx|P?o+XzV0(l zD9c=T+oUWZQFM2 zl~b}-Om$>cIn|}+im5lE_SPOPoL72LyL`azEI?6dQ-*jCiz4+3d%)QD{6AOTTMTU zYiI{mqKc}e)x46*7v+?O)=)z2SJ_~(JCpB{GdN7;)2LWrvXW66(PO5fa|4-TH?GP9 z-72HYjtty9D1|8Rs@LI7spzVjq+v|v3QAvA#a+XX9&b$^Y&qVV@avTweVMG1G+(M-YaFIs z_tc}=R7aA(H1?r{ey5G;Wxxo&LV_U9{9o+6DiqDNAqZvRTf#2kNxvWri9^02e-V4e zRu;>?vj27S_(|Y|A!#Tu6dVc-g^MPJ5QLtfRo?k7(%tx0mdb&0upEA!I)tq!W_-J0 zIb^NWo(lJQW4|YU{W)tjTlLaD1vp9(tQ92+$Axu5v!ICsz)VjFBK`)2=lo9y1LAqH zEfHo_Xq3_2bg4%xR#PM}|M-QG`c45VP%?B=OZyx@|{S(cPj)$8@0wevWpBr6zF8E%2$;+ivrL!C8 z;6nJ~hKYuj$%fYPhSoRLiG~L!8y+2RcyzMisqu!VCK{3x@n^0EsNt);`DqTXX*8s# zTGq5|_rJA0(DH?l9<*n3KuZ^T)mf7g6H>tR5@))xvBowI?zFE`a72iu$60Q%~b&lgm1BhiO zCN)|eGLxx9#9pv$6OL(+H3GxJ2eFkue)O+){^-l^u4=g;o_+FrPhVQqG7-CPOuFwA z>}k~VS8KM0)6|-^(YoP}+=#QUV)gY{S6tsiVp-G%K{I{MK=jxH;S{tUxL?qgI>XRI zPc~;O+e5+JB{>H$yQot#PxrS6x5uljXuTB0f7Vw5CK&P+E3Niib!oZH*~&hvuJ!$O zQ2`FZh*Zxhio@f0V{O+q8)QUYzo~a{dqRu;zPTgs=cl$Mv?hc2MThvyI`7l{y@T5b zY{&8q8W;77dwU0ygg}cKEZ$(YkWOVaJ&Lm&|E8F}BB^y3`hahl+RKt%*+NGuo6M(j zsvb#dsZP+PUZz*IaH0mf+R9jg=}VHG1(r({lj?K5*;JmdQoW*Rk${YCPwd{AKOuu_v*=NH~ITuKUzl=bUelM{{vo-9feFtB%*xOIB}YE#TkAu@Bfi4WIRM z1QJprO0*oxi2NwL=r<9zTC%>)H$iVBI20I^60y>;FXmMl)E(R{=lF=P*WhDii%slKY2YG@#jL6{U28IN5plzdz(;xzIHSOd>`uJfwo({PXVsKe zJTL0LOg}5*-3d2&~pjUxg!G?8az-TuO_v=G=v7NvkzVC%O^8h60abs z<_mpY-AV9joe&~QDr-X@>0z~R1Atgp-gdV9-LH(TZknjvI(+!Dv}*EGe%_H*U6wXX z-l7j9p*4+vd$Un+9f*J%1;1_vcMp@3e>q}W@x+E}3L|FLdAW=fop6DBh8|d3# z_5*$U6u*m(-y90Dd%Ym~wo3wi`xiiVHXor#CJ!Pl8fn;w(uFhvKp2pNFg9MO+ou4T zKrRYA3u4U2Xdwh1D*J)J!&JvX;GdsSU10Pb1RgOFcu*$btz(UmL{j8B1-9q3#0tX3T zh6upGYiXZW&Z04J)$Cib!_-h%50IFrIVhMNp+@f~aE!o%1Res==x$&m`DWnHbp~q- z?RYz z{jG}tc%Vc@;s`F_uIoZ{pC1u6=nzRSvS^W0$SL)%5uv3vXLH8TLS@Wd!$Xk1z6JBJAo#6#K`5|t>V60sb#-~JC>&wAsxAEP>- z$G7f!I*<2K4q79pRM_%KrBdaRPZq20@RCpXyWr$4I4C6IZEQ1U5?2v$G2y&;&9{F) zsM5}PsFI;!G*r`|N&5{|$j*euNM!(xgH4%8BG@@gOxSIR3A@?DR8%bOp65QZNs@CX zh$rkgb=yty7AtC0xO>k#%xN7vLH&ssBSFFjyPE)!JGY>)%YAUyTpBn_#cB{iZl*}4 zpd{x)oSO=o6!00`;sE}&4*`aS%b!xu0=*1WEtjRls&$K!i1?wW|VT#ZmWR|SI89A496xYMa!En)v#;Xq4GXqI}J#uB2QI5Q1J z^Q`w~a{yfIGfj^uZEsNBf?F~+FQX2G9lYT_7=g-?_G`z)CZ<2M6F#JgdJZ&)@@{ zlDJq*%V*BkLknrn20Km2wA0AoBtzv;4=u7en-V5YgT--`7T%mqiO@2IJ?D0A&j~3} zIvxs_!%B?rMU)E6NAO0RUx}8ZTGsV^9u!@vEJrajtEi4MGhKDO=1{~Miv`Nr>I5Zj zsv=@awNg`#*__*A-@=VJw+P-s6i*yhYU%lvCFUs5xqrH&Q1?%H6b3EZW@tc2EN$Zo zWev>S(Rr9TI46+X0=PJ+NT?h*LBpDc9aGy37P9F}Eys^Xhe#u=^<|5UjF;ynhofZe zIKiF-n8jA!ZLpPh>oJ$_35@4z8h#j=w8n_~=pj(jVkW1SmN=GeRWS|U>?DfdA!?~} zq!oAg+JukYi;Kp+zf6170X>vciwuqHOQFk$I|qHs-i~>TYVDB_G(VG z^^iFV{IPV2p4Cu_le)#FDBn(pI|%F~Kss_p>zF+Spi9rfY@kcHUkbXOJ#@a4z+4^q zE<;{+UEl)wW7o(ABybcg-07?8WTA_rP77Z*Z`a4kuGGth;T z|A?ARVi4#6Y z*Gwi_#}lo;_WNssQ@5|5idDUQ_o=&24_%7gam6o0Dqaqs3ZGs%5x#AjN?ZvFq3}l` ze@$fg=+&?g*fS~Jz4*^%Y5(M>`CO9rUzVCDKh0-CYW`%p5@D6RuuoE=)JMk}P)mLU0qmJ_wse(9oiUX;{d4uO(_8=NKiyR;KG$>Z77bU_}^ zE(uNve9xkJH#hI-!Mi!U(1Ja%^#rEg#&CuRyGGDUcM_(tD`9NED}2x4JtHN&S=d0s zCf5)tlGMUCS9=~@7-T9_YO=n$wl8E;2>*aZD*~Vuu{J$yqCgFRw3^*)CzFA(1RT+P?rjircO8!az#aHg@eMmCPG$Np|=;#gV(B~2V8bZQcDoc zNK_hHagv_&h~-+SF?#a(aWJfsrk91$Y)0!&SPS8}D)BVIAHbnO! z3~eYDIJ)(c_I7(ATgx?4OA;2gZlxs+*Dg{nus=Z4M4GfW0Dx=bD_{J|@cr-AtQieX z)NC9+HkD|;uz5VO|J}+1qb(z;(^V5yYsM-M3?BxVTEBI2{hsmld)}?sJ1UMm{PNQ; zK0Q{kceoYcvUFtfH}IK|j(h@vvFe^r^oqS)?EPNFp}o>imK?fA(!-Y18=M^B9QRQ3 zE&_KE*iB$Z+AU1l0_X*f>dGR6C9PA$vVc|1u~wFS-vSzQBfTzo8@?v~Fo^R|C>!X_ zKwlX`dsv=?z7#*9J$y^_-3gnT5}>_5^1W1Rp|BeuLI@+gB^ZUdFhaO`&ITdGL!FDe zguH}YFBcW`qZ$1O;|}n8&j7Fv|yGtCW(bQ12hbU}l|SCf1wqj7g- za*2k|aaPN8!of@CPS{Q71vZGzadf7Kw4&Mzq{@2fl$)<5#A@=jK&%`&8JOu#lq|XZ zIV`-{mWVrulYP~?2m2Q7QTuwJKb7rM=f|NgglTfDe@OL-%CN5y_&R}a5Fmufo+sdi zGAnT5dJgq=!_;WHc5_D5QJBGl8U7jUG-{^tul*8WSokO?$eT{DcqMl3k>R7jmS>(A zi*Ga$=2*>+bM=>Mb^vQ0`Qq4@+Q*N4`Q6GV-&{NLd#6{A$K}zrqep+xI99dwLd}Ie z|9aJU)&8-{Ckc^0Id$-%$%BuNAAJ1Xijx;pBdM43r}Ae~-_Lv}Ggi6j+}bx1=M!&i zJ->Cla?e=BNkXqDFG~%Rx9-DH>Qf?c>PfD%`?IC3RnlAGovlk|bB9EDxX$kIbvipZ zdYH)dA4O~D<61ke=X0SU>6Y~hCcXoo@WPK(-h*OPn64n9usVpQkzIMZBD=k;Fg(OjUM{r(pyXS>EIpSKxq6LZ*4 zLtHN%Rmgto5c0)pM7;>PJ;TX{ivJM)BSYcANFrX^cF>NEF+>s`*~DX)SW1B#Xr`;s z$7Bc-vev7@qm!M&d}j&(1fN@>JUiTS^g%eHb8Qf5&R7?LZUPyAwo+AkOFG30*)0e( z$*7RJN+YE1g8IzR#bL+P#d)abZs7lJ$x_jwP1F_`heUL!$>e7oeP6R(o%YPTG?N3p z(d$IWOUuXmtxM3zZR2ozDee;PnsGCJdMCuY4th+^@zA5|5;eP6u0#)8PjQ__jTNcg zBLqm3#`>snmnZ67hI7Bst7g-=-ZbKR%GI_G?sy^LnzA!#sdN_aj&vJ`1ezj_GcTv)6T5h% zYY|TTsE^}jNRWszKwv>P>=0a*#Qg?&b6Tb=pUTQ}RcGpR@H2%UYk{H^l)kjOca~2p z#i}Z1hv3a>h3E1}SuJCPmLi-1F}ZleoczquwiB&wCz^7~Ge$dR$e-n|%o{m-PU^0+q^Ws#3a>_sZxCx#7x2oR-|3>WABJ z(NH=fjuJ+hu4_%RbdQLhFvu!)Oo9h`$EhR?OrVm8>C6vWuDtisty=)v5>*B=R`kxYbfWZ9#+;~8bZCQGDc@ZBT-kdbT zo0AmCoJ1To_ef55;LX83_%*aeWKkpRGAw*tCseJnDbf2I8vmm3gTmy7`^Goi2m4D^ zaB9u%-u9Oj6XAv{RATt>6#-vldO0jkO0Bp2^T~7s?Q6s+{EPL6Ze%<8V~`Cu!LX3$ ze8J0eypbeWN+^U`nj0?s)qodi845yaNlLj8SLDI#0MS~c;N+VpA@M<;0g|ISv*enX zU5j)kb2jTq=7e#2c*r;?4oKJOiH8L1l;jfh{AF#W-vVvAZl7{ zEqM`-=Ip-TSz?_c3@~IQ?nyrSBR6^@(WP zwYvV)mM(-+7q|3hQd>G$VE|$8maemz1QA=_aKsIgHYlnrUxapawnORHF*D|PmFm6| zyC5NLo2N0v-u5wMR+!ZmLgZ1i+Pl24Z(%_Z1wwgP%}Aktq&?sl^nI6m|}EUZ8sa6M(xEzPRd&AO?<$``rLbKcU(AlaoL+IezALE?IYvi)g$6aGwetWXCA)lr;@*tU?smbpIaF8HYRPj z8lvuf#2?3SWqh&z@P7Z#_Xm!IZs5vz$%pZXybygMhD{wIhjmS)1(OU>NRQwzs%E!YFQBzoZ3jY6(|d&(O;D$gWFN{mZw6)yLFNA_@@-%Iwe6 z06~NE^I%|Z9+H9$+w)k&Vd^Iv%HNI4N1f`Q9Qd z>ZaEeXPUJX1#^)IP{@-I0xrB4*hKDB!B#h7pMeo4;R8NE3?-w4e<)!N`DMu+$akaV zAk#F-$5mB6jQl~{?lOb~3Z+AN#MmO$4u$?^5D+Zqp(aDfot%ZtgpfLT88z}2=?=*0 zqsjTcTnA;&p+cS6LV;O<)YJ=YKxGDBKg%};T%7>RlQ`3X9XEgHrG>Vp?vM{!Yxwql zOAuYDZ04&`_K}~(aWngePnXE`Ot0uwStq>2;EAR(gjD-F^SYE`T^d>KQViogJ0nAY zYAL0=y*P88lPF0!i6{zLeNIjH71gGFMs|q`e?Uz5(x_=~q?3WAS)6IZ&Yy_xm(V4gqRcvC zgYcS&J??b$3b)et(2JY1p#*nVvev+H;k>`?Ln_D!ADs8O9P&jyWQw91t-fI<16~X7 z%cgQ2O6owV^)rm|K9VAz>tzQp>_#)AXmq9t;2(yCKkpygc;wu`1$I$9_xsqrc;wBN zpB%fi@yO)H2gWx(@QWj3Cm#K;$0jyDJ}mGtZ)5)q6%&%~Yud&*ng6#qW)}c>ZWmb| zHhUehZ(Q1UGu<9V(=!x$@DG1S9iA4%`o^i-R)6Fd8rDp#d3>_|BnW4H<0mr`%6wz# z3X%O7JF}M4=V)%QAEEQ?-vA)#PRLY)xF!Spcer$m=D}-ptBVBo0$6XB{e;f&SNor6 z@A_BE{yom$+)L%R{twjrUjejHr13s5vNgoWf(4fMS?MG6s5u779J0}viv#2-IzB$&v!Qf)dP1(h15)|Ju#{ED%;o`sgtoXtsWXAwkG%K;sf zvZUBx75}cF)O`=JbTm%C8(CviZenDY{%$1Ut#J$HsO>jQhLdG&M%aeGqtJnTd95vn zkOwoW^>t8!ODJ&kD0t~yMkDt)_UCwsP}nq=mfDIT$IJl+x)#24TedZAZ`!tnQYVrY zq!deULDO5v{GgTUDRU^qj)K;N?4@ZniAylK!dQ+xZc#g@+8{D>rYW;&k~8f{)0yV^ zM5#YD_JO6r(JNhkXec33*n}R_`ie<1fS?&Yn9im$AitHyx0#ExwBBmVH1F({?dNhU zybj>8&%1(F%v4S=I3V{NybaPSSm1k6h+7tB#mrC`^?kTfsxd87+kK> zWNL-bMv|ME%~dPawG|9Gj6|+XzQ2%#Z3z4U84G!I3L~MFT>?OsDaFMe?4a{=cm$g> zFe1mphPn$h0WQ;N_72r>B2@0DLsD^h*P{DQl;e)Sp@V%ATopn=S^&>+YGV+k8=(TDIlF(aD|1#&;f@s@wG5%5~qr=R5bj z8$Wb9buRUWa$dQRn`k=tZShk4(A2W#kKrT4eVe@xF4de?sBRqwig}?!u}!liww~#;3HPT9omPy5|R?=qO?kg2i=v1_bE1gDE;FzP4Y&Hu)mtX#f_#D=YA7cKCHMlTq?aPhemYWbEl}3x zA(V*8Ke+t>!GX=)oPH?L5-uWrlH&pB;BlviUc82F!?}*v@Y{G?1Rk$wySCPii;cu; zTzQ{N@yu$-ESvC^LYW-ptCLFsj*0Xj(%WV;9i`j(VKUj-N4blWN!cvt%8*=Y=Vfk) z!mTPMQqM#6okIF&>yXm94UiTLr#Q+7BvUglW*HUi$Q09onsBVyWepL6jTlCbWJ|LH zP%Jc?sMIB7qlLDI?hQKDiJVM}9ZDZ!nU_&J$NQ4VX0=lcK+k^HK?4M1qKW~_GC zMCI(@-yZyvATJW<~WV{be-RkL&|7JvDkQ}>Lm-1>HG+l7;F zo*3VCbQ(c-m9X{3gTqH)>y5+KD+S_|bs+wK!XFBhgupnuLZ@H|j>Jy|`B}52`1P#v{0_5k!wZl>IBpn(|8n)AnBLUfsGsgcx$XgmW<*Eu*FX*kVPFLG8=n z_`&gpiUpnL+1sH#PW5&(uz8l-N{ejIbe@p~%44L%gy#s7Ya@n&62lwp7nI>DKb3+n z4iw~Qvo8-;jbbg7C@mpgg~wnNI?3qc*g zBfUFcgzsTUOY0Yfu^2IUjMkgM4!uizlLUx$YeV?iWjc7DfTN0;m)A-JZp>NHd!a6x zh+^A~*(z)7IHs^)2m4zz@N@iYj|0FJq(Z2wy$(rSz6t%uJJP=E)zhR-=KP2*-DrW* z0<=TCEZxLvLpNbK<0(9&1FI`T^FewLFAherX&(3t8GHu0?%9*saXTY<5D6_wV=?8J z>O6HIj)y~-Q{u>?m#;g+m&fvTG&I&Snw1mGgFCsthGE!RbdTH~A|5?*#s>(+=IA{6 zB%7l_4_t;)&$2eUP-0d>$VGdF*zeOCBcPfJpc?2)9C>z@qbOw+@!*DTkdMTgY2XgH z7HkzJ)|}0pcPpVI!+e~(EjSCCc<`Tl7TQJ_(}C0(X%Ez`R9a82RQ;CDzxXYifA+{M zIREUDn1AsF=IZtH&lSYzs_Vsvd&N&9zsz8T<3Q@{C8|@^pvWi4^^pkOEa%Hc<3~%n zQ;^6t(~^6FOr-b>5VbSXsde>bG&qr8=L5>AQ2{wRpaU7?zt3WkOeiV1iI&qY(?-DE zG^u6sV5Ni)DWVR_{C)~9ksaCijTTOr!F}Ti zxs#~Vrj=sV`012X7hG4|PMee(!|;>rp*@Yjp8}LNQ{z^@ZLNyii_^Rnv_#u&|4n)@ zvX?CA!^uW*6FqJ$6E7|v{u^N}s$Y>4; zMAp(3e-}c%N3mxR`(fJik`rtZ;f9W+R(i6pQ4{?H{*C~bACA%9(*(Xw;6noU19+w< zbA$dH>|*W8<5Tp{Mo1sdXDX{aRnHI8|}_1W~QqnjlB3dwQ|y6)GC zjFRk-7dcYN&<_gvjBwoljmwE*8hM;*bNfyWZ?(=+DpSn!#r*C&l1u`sjK z16v?tJQt?*h!F-uTF#Jp&X~8v^fMAf^q^503RdvqhHJcGu_vZVZ*a~wIT{2WT*Zjz zV?^4FM8~A6;95xi#)@37o79&(S_0E;Fv)ch{Kt8WB!0xZ7`Y+&sta{yo+H(RksWdm zU2G%p7=dR9bPyOI@CN|8luNN^^%%(^JVy|c!=I*$X9>JU;2eRU5+Ge9m#jJO%$EiC z0^nROm)GHDas70)Urfx@21H~Gtd5A=r&k2T`^78Of-n5S z%CD`Q6jzLkE6y}rTHiV$9{$J|5?4|yN5m_%J6$JSc5Ts$mJ-$6~*EfM!mZ?6y= Vr&mYC_;f=^+&_bJkBWRq{y!YQgyR4J literal 0 HcmV?d00001 diff --git a/tests/__pycache__/test_lab_ground_truth.cpython-312-pytest-7.4.4.pyc b/tests/__pycache__/test_lab_ground_truth.cpython-312-pytest-7.4.4.pyc index 77df5aa7b6948ab7a81256c3f45d78b3193adbc8..5deb0516ddcdd1a56fff035dce41cef03db5fbb6 100644 GIT binary patch delta 21 bcmZ2|f^qc;My}Jmyj%=G;H9vUOECoiN^}K` delta 21 bcmZ2|f^qc;My}Jmyj%=Gz$&wmOECoiNWuj} diff --git a/tests/test_classification.py b/tests/test_classification.py index 1e9c116..37d5d2f 100644 --- a/tests/test_classification.py +++ b/tests/test_classification.py @@ -62,6 +62,7 @@ EXPECTED = { "M19": Classification.BEHAVIOUR_CHANGED, "M20": Classification.BEHAVIOUR_CHANGED, "M23": Classification.AMBIGUOUS, + "M24": Classification.MECHANICAL_ADAPTATION, } diff --git a/tests/test_crystallization.py b/tests/test_crystallization.py new file mode 100644 index 0000000..0b9f5a1 --- /dev/null +++ b/tests/test_crystallization.py @@ -0,0 +1,227 @@ +"""H-003: a stable agentic realization becomes deterministic code. + +What must be preserved is the judgment, not the path. These tests check that the +descendant asserts everything its ancestor did, agrees with it wherever both can +run, and costs less — and that it fails loudly rather than searching when the +surface moves out from under it. +""" + +from __future__ import annotations + +import json +import time + +import pytest + +from testdriver import Runner, Verdict +from testdriver.crystallization import ( + CrystallizedDriver, Trajectory, assess_stability, capture, + generate_test_module, +) +from scenarios.alice_bob_carol import USE_CASE +from scenarios.browser_grant import build_agentic, build_crystallized, lab_server + +RUNS = 4 + + +def agentic_pack(*mutations: str) -> dict: + with lab_server(*mutations) as (app, tokens, base_url): + world, driver, observer, asset, oracle = build_agentic(app, tokens, base_url) + return json.loads(Runner(world, driver, observer, oracle).run(asset).evidence.to_json()) + + +@pytest.fixture(scope="module") +def stability(): + return assess_stability([agentic_pack() for _ in range(RUNS)]) + + +@pytest.fixture(scope="module") +def frozen(stability): + assert stability.stable + return stability.trajectories + + +# --- stability ------------------------------------------------------------ + + +def test_one_successful_run_is_not_enough_to_freeze(): + """One run proves the agent can find a way, not that the surface has settled.""" + report = assess_stability([agentic_pack()]) + assert not report.stable + assert "at least" in report.reason + + +def test_a_moving_surface_is_not_frozen(): + """Different paths across runs means the surface is still moving.""" + packs = [agentic_pack(), agentic_pack("M24"), agentic_pack(), agentic_pack()] + report = assess_stability(packs) + assert not report.stable + assert report.distinct_paths > 1 + + +def test_a_settled_surface_is_frozen(stability): + assert stability.stable + assert stability.observations == RUNS + assert stability.distinct_paths == 1 + + +# --- the descendant ------------------------------------------------------- + + +def run_descendant(frozen, *mutations: str): + with lab_server(*mutations) as (app, tokens, base_url): + world, driver, observer, asset, oracle = build_crystallized( + app, tokens, base_url, frozen + ) + return Runner(world, driver, observer, oracle).run(asset) + + +def test_the_descendant_runs_with_no_model_and_no_discovery(frozen): + result = run_descendant(frozen) + assert result.verdict is Verdict.PASS + mechanics = [ + obs.data["mechanics"] for obs in result.evidence.observations + if obs.kind == "realization" + ][0] + assert mechanics["runtime"] == "crystallized-driver" + assert mechanics["metrics"]["candidates_considered"] == 0 + assert mechanics["metrics"]["tokens_in"] == 0 + + +def test_the_descendant_preserves_the_ancestors_oracle_set(frozen): + ancestor = agentic_pack() + descendant = run_descendant(frozen) + before = {v["assertion_id"] for v in ancestor["verdicts"]} + after = {j.assertion_id for j in descendant.judgments} + assert before <= after, f"crystallization dropped {before - after}" + + +@pytest.mark.parametrize("mutation", ["", "M15", "M17", "M16", "M13"]) +def test_ancestor_and_descendant_agree(frozen, mutation): + """Same verdicts on the same lab version. Disagreement falsifies H-003.""" + mutations = (mutation,) if mutation else () + ancestor = agentic_pack(*mutations) + descendant = run_descendant(frozen, *mutations) + before = {(v["assertion_id"], v["step_id"]): v["verdict"] for v in ancestor["verdicts"]} + after = {(j.assertion_id, j.step_id): j.verdict.value for j in descendant.judgments} + assert before == after + + +def test_the_descendant_still_catches_a_defect(frozen): + """Crystallization must not trade rigour for speed.""" + result = run_descendant(frozen, "M17") + assert result.verdict is Verdict.FAIL + assert result.judgment("c-carol-denied").verdict is Verdict.FAIL + + +def test_the_descendant_fails_loudly_when_its_path_breaks(frozen): + """A frozen driver must not fall back to searching. + + Falling back would quietly turn a T5 asset into a T1 one and hide the very + signal crystallization exists to produce: the surface moved. + + M24 is the reversibility case from `INTENT.md`: the agentic ancestor recovers + by reading the form, the frozen descendant cannot, and the right response is + to thaw the asset rather than patch the generated file. + """ + result = run_descendant(frozen, "M24") # the grant endpoint is renamed + raised = [ + obs.data["raised"] for obs in result.evidence.observations + if obs.kind == "realization" + ] + assert any(raised), "the frozen path should have stopped working" + + +def test_lineage_is_retained(frozen): + with lab_server() as (app, tokens, base_url): + _, _, _, descendant, _ = build_crystallized(app, tokens, base_url, frozen) + assert descendant.parent_id == "va-grant-via-browser" + assert descendant.maturity == "T5" + + +# --- cost ----------------------------------------------------------------- + + +def test_the_descendant_costs_less_to_execute(frozen): + """H-003's economic criterion. + + With a scripted runtime the saving is only the page fetch, parse and scoring + — real but small. See F-0007: the meaningful comparison needs a live model, + and this number is a floor, not the answer. + """ + def timed(fn, *args): + started = time.perf_counter() + fn(*args) + return time.perf_counter() - started + + with lab_server() as (app, tokens, base_url): + agentic = build_agentic(app, tokens, base_url) + crystal = build_crystallized(app, tokens, base_url, frozen) + agentic_ms = min(timed(lambda: Runner(*agentic[:3], agentic[4]).run(agentic[3])) + for _ in range(3)) + crystal_ms = min(timed(lambda: Runner(*crystal[:3], crystal[4]).run(crystal[3])) + for _ in range(3)) + assert crystal_ms < agentic_ms, (agentic_ms, crystal_ms) + + +# --- the generated artefact ---------------------------------------------- + + +def test_the_generated_module_carries_its_lineage(frozen, stability): + source = generate_test_module( + trajectory=frozen[0], + action_args={"subject_id": "bob", "permission": "READ"}, + ancestor_id="va-grant-via-browser", ancestor_maturity="T1", + descendant_id="va-grant-crystallized", runs=stability.observations, + sut_version="lab-0.2.0-baseline", + claims=[c for c in USE_CASE.claims if c.id == "c-bob-reads"], + claims_module="scenarios.alice_bob_carol", + ) + assert "ancestor asset : va-grant-via-browser" in source + assert f"frozen from : {RUNS} identical realizations" in source + assert "No model is involved" in source + + +def test_the_generated_module_imports_no_agentic_machinery(): + source = open("crystallized/test_grant_access.py").read() + imports = [ + line for line in source.splitlines() + if line.startswith(("import ", "from ")) and "urllib" not in line + ] + assert imports == [ + "from __future__ import annotations", + "from scenarios.alice_bob_carol import _bob_can_read, " + "_bob_cannot_write, _carol_cannot_read", + ], imports + + +def test_the_generated_module_restates_no_assertions(): + """Assertions are imported, never paraphrased. + + A generated test that restates its claims creates a second, unverified + statement of intent, and any drift between the two is silent. + """ + source = open("crystallized/test_grant_access.py").read() + assert "from scenarios.alice_bob_carol import" in source + + +def test_the_ancestor_recovers_where_the_descendant_cannot(frozen): + """The reversibility case, both halves. + + This is what distinguishes crystallization from ordinary code generation: the + agentic form is not discarded when the deterministic one is created, because + it is the thing that can recover when the surface moves again. + """ + ancestor = agentic_pack("M24") + ancestor_raised = [ + obs["data"].get("raised") for obs in ancestor["observations"] + if obs["kind"] == "realization" + ] + assert not any(ancestor_raised), "the agentic ancestor should have recovered" + + descendant = run_descendant(frozen, "M24") + descendant_raised = [ + obs.data.get("raised") for obs in descendant.evidence.observations + if obs.kind == "realization" + ] + assert any(descendant_raised), "the frozen descendant should have failed loudly" diff --git a/workplans/TD-WP-0002-vertical-spike-crystallization.md b/workplans/TD-WP-0002-vertical-spike-crystallization.md index 795d038..dece354 100644 --- a/workplans/TD-WP-0002-vertical-spike-crystallization.md +++ b/workplans/TD-WP-0002-vertical-spike-crystallization.md @@ -415,7 +415,7 @@ without a human, so the safety result is not bought by escalating everything. ```task id: TD-WP-0002-T09 -status: todo +status: done priority: high state_hub_task_id: "855a1f41-b839-57fc-87b8-198ce2a9f6b1" ``` @@ -432,6 +432,31 @@ Exit: the generated test runs with zero agentic involvement, preserves the relevant claims and oracles, retains visible lineage, and measurably costs less to execute than the agentic ancestor. +**Done 2026-08-23.** `crystallization.py`, `crystallized/test_grant_access.py` +(generated), 163 tests pass. All four exit criteria met. + +- Freezing requires the **same** path across several runs, not one success. One + run proves the agent can find a way; it does not show the surface has stopped + moving, and freezing on a single observation is how a crystallized test becomes + flaky the first time a page renders differently. +- **Reversibility demonstrated in both directions.** M24 (grant endpoint renamed) + breaks the frozen path; the descendant fails *loudly* rather than falling back + to searching — falling back would quietly turn a T5 asset into a T1 one and + hide the signal crystallization exists to produce — while the agentic ancestor + recovers from the same mutation. That is what distinguishes crystallization + from ordinary codegen: the agentic form is not discarded. +- **F-0007 (open) — the economic case is unmeasurable.** The descendant is 54% + cheaper, and that number should not be quoted in support of the thesis. The + T07 runtime consumes zero tokens, so the entire saving is one page fetch, one + parse and a two-candidate scoring pass. The saving crystallization actually + claims — model tokens, latency, retry variance — is two or three orders of + magnitude larger and entirely absent from this measurement. +- The generated test is **not fully standalone**: realization is plain `urllib`, + but assertions are *imported* rather than restated, because a generated test + that paraphrases its claims creates a second unverified statement of intent. + Right call, and it qualifies the "drops into existing CI" story — what drops in + is the realization; the claims still need the use-case module. + ## Gate review and first compression pass ```task