From 44faf3de8e4ea5843da422304c4855c4784db691 Mon Sep 17 00:00:00 2001 From: tegwick Date: Sat, 22 Aug 2026 23:50:29 +0200 Subject: [PATCH] T07: agentic realization over a stdlib browser surface Two decisions taken with the operator: stdlib HTML driver instead of Playwright (F-0004), and a deterministic discovery runtime instead of a live model. Both sit behind interfaces so the alternatives drop in later. - html.py: stdlib DOM parse and query - agentic.py: DiscoveryRuntime (agentic arm, ignores data-td by construction) and RecordedSelectorRuntime (control arm, uses the strongest identifier the page offers) - browser.py: per-actor sessions over real HTTP, constructed per call so no actor inherits another's connection state - cost/nondeterminism metrics recorded from the first run F-0005 (CONCEPT_DRIFT): the H-001 result is a narrowing. Where test ids are preserved, discovery 9/9 and recorded selectors 9/9 - the semantic action buys nothing. Where they are dropped, discovery 2/3 and recorded 0/3. The concept model presents semantic actions as generally superior; the evidence says conditionally superior. M21 and M22 added mid-task: the deciding side of the axis was N=1. M22 (field names renamed) defeats the heuristic and is the first concrete evidence that a live model would add capability, not just cost. Co-Authored-By: Claude Opus 5 Assistant: claude-code Assistant-Model: opus Assistant-Process: 1629012@bnt-lap001 Assistant-Session: 78d4fb13-8a1e-474b-87a3-9b9261c49a39 --- WORK-RECORDS.md | 2 +- lab/__pycache__/app.cpython-312.pyc | Bin 16581 -> 16637 bytes lab/__pycache__/http_api.cpython-312.pyc | Bin 11471 -> 11777 bytes lab/__pycache__/mutations.cpython-312.pyc | Bin 12948 -> 13835 bytes lab/app.py | 1 + lab/http_api.py | 6 +- lab/mutations.py | 15 ++ research/concepts/fitness-map.md | 10 +- ...004-stdlib-driver-instead-of-playwright.md | 60 +++++ ...mantic-actions-earn-their-keep-narrowly.md | 80 ++++++ .../H-001-semantic-action-stability.md | 24 +- .../hypotheses/H-002-mechanical-adaptation.md | 16 +- .../__pycache__/browser_grant.cpython-312.pyc | Bin 0 -> 5052 bytes scenarios/browser_grant.py | 88 +++++++ .../__pycache__/agentic.cpython-312.pyc | Bin 0 -> 11662 bytes .../__pycache__/browser.cpython-312.pyc | Bin 0 -> 8516 bytes .../__pycache__/html.cpython-312.pyc | Bin 0 -> 7178 bytes src/testdriver/agentic.py | 244 ++++++++++++++++++ src/testdriver/browser.py | 156 +++++++++++ src/testdriver/html.py | 108 ++++++++ ...c_realization.cpython-312-pytest-7.4.4.pyc | Bin 0 -> 21982 bytes tests/test_agentic_realization.py | 175 +++++++++++++ ...-WP-0002-vertical-spike-crystallization.md | 32 ++- 23 files changed, 1008 insertions(+), 9 deletions(-) create mode 100644 research/findings/F-0004-stdlib-driver-instead-of-playwright.md create mode 100644 research/findings/F-0005-semantic-actions-earn-their-keep-narrowly.md create mode 100644 scenarios/__pycache__/browser_grant.cpython-312.pyc create mode 100644 scenarios/browser_grant.py create mode 100644 src/testdriver/__pycache__/agentic.cpython-312.pyc create mode 100644 src/testdriver/__pycache__/browser.cpython-312.pyc create mode 100644 src/testdriver/__pycache__/html.cpython-312.pyc create mode 100644 src/testdriver/agentic.py create mode 100644 src/testdriver/browser.py create mode 100644 src/testdriver/html.py create mode 100644 tests/__pycache__/test_agentic_realization.cpython-312-pytest-7.4.4.pyc create mode 100644 tests/test_agentic_realization.py diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 0b2c90f..65aa4e3 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -18,7 +18,7 @@ | task | TD-WP-0002-T03 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T04 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T05 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | -| task | TD-WP-0002-T06 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | +| task | TD-WP-0002-T06 | done | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T07 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T08 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | | task | TD-WP-0002-T09 | todo | — | workplans/TD-WP-0002-vertical-spike-crystallization.md | diff --git a/lab/__pycache__/app.cpython-312.pyc b/lab/__pycache__/app.cpython-312.pyc index d0d6b766bda7f5bc219c7c723603a2ab2712f6bc..1e1c01c674335efc6beed571e2ac1df9fcdf6223 100644 GIT binary patch delta 1824 zcmaJ>YfRHu6z}N^%4;&DR8m5D6#9n(Rsx8WhamVE;8bw4E`i$nM+ztPQ2a zUS_*dR-oOEveKkshfJD1Ce8DIc9?ZwS|`%?AgziWVVx+eSs&{{S%Y#f%Izrkp>)N} zQonLP;~U*Jlg~RA8uRI}iCjJilb*dr`#oQGZa58zKQ&4s#|}%ns!KWUhl5B|WbUtX?GXE(*}| z7fC-(;2;qrTQhqrqa=?<~-a45{%$o%iqt5wtC zUB)kw6;ZG!g(7ClRz9;2g5_z4BRX8Q*=#(Qa!rcu<+<>^=qmrCJBp4zPID&d{#6|9 z&ZiI}dIXG2hQ5W?M0U6L#O`po`dNn_5VM;$SC$-6m`P;4gkC;}@LYH4wEQx`c>>yL zb6cm;8&74jmj(P|a>Rfp6_KoB5zPrpvWgSlh)g848jcaaYQ)8gW&BhdD{Em;1S>01 zzfxH^6T&cVAqW!~3G$z4Qfgu#)OB?#RE<8_L<}+Gdgk{<@i^+uZlJ2EmWZE)Af;_2 ziJ0>R(d8^iB@S&TMxDiFWPJEP30sN0LnzJ$SQ5WGBS-tu!H*KWN^p$eH3D_O8EL}= zoZt*MnrL=VLFD;BQV1@G432w-SFm%#nvcfH~D74;x!BNN@ zVR43pXuM9d#7RWH z=idLr8TfhzwtttGmt(-+l67w6LFMxH(H!WN(>aUqx;yM-_Nax`2639zg-e(TNA+st z4!hLc0X2G>9b$pA=CGR`W({XE!lkT}HKO#eXIK#B9@fR0P?oVHtQloFJIY#6YB+91 z>CHBhKjimea^#Z6l|PW)@Dh0}6>!5**= z;Es6*Ld)PgN$)3=6Lc|A7=kzD)xv{N6(dUok+O}&lg`L^@})SBrJ~H4j5~QW!nlr; znuQ0*k%@{T4;JbvIs)@@p-6|^Q)FMyA{=&yo7`8;uklzi%Ew91Bt01&>wA>sguO^D zP&22kjj~M%VQ;gXM^)FwvZq-7ZBH11i;?1~d`1r22cuJ%^)$oBXoA5q?n#s=)E{NOy61XPU{G}lM zRc{&LzWAbg7?woozDD>Fh_FjD3OB5F%sF%?|T1s54IRyVmC zr-8#@m1}_;fZMsz_$s(WLvR8viREAkb|2edp+@r$#l7Hj{>!9ZA$&rR1RudhP;X$7 zG{Ub!ZwkSp{H&=8vZx1m?=OkpntkRK9PAZOwtNI5;>(t1SP*M1Zi@5dY{jnxQ}y{||xLi8%lO diff --git a/lab/__pycache__/http_api.cpython-312.pyc b/lab/__pycache__/http_api.cpython-312.pyc index e9bfbb7d47de37250d5abe3e7ec21fede64eadd7..91d5fde0611d2f7459c914531573a8833fbb0086 100644 GIT binary patch delta 1771 zcmZuxYiv_x7(U-=dwN@MYu9#L>AJI?Zf9jHmvM=dktsSNWG-wdM3jwv2Wjbgy02Xg z^^}XEh=z-K`Js*y_+fxZjHZ8#mw+J=qcJcH0qYO>;UA+Rm=I(5LHxc`HeBLK`tur2L@yOw}QI%Fw$ z=ep7wZ&vL4+^kJ@M4Ek}r_$jAD8vX$A@f*+@3iuNtrB$O!!0 z`~*5y^|fq4mR*mWFx9#NoveP*dYljDDEGjcRSS__>U0*DOo?Jjd6lRz9$svJjXKFb z29xl9IJDw_nvq4!U7#SF>TM0vBtzVcZDo*$-@`rTQ`F4gG&AYO7zjp*qMZ>P&1#7GY@jX z2V#f8kKJ%dlu(`X=v-g{*9hA>+D%@KzvzSg9Tj76i0pt!)IY#BQF2a>XLhNCYAM1l zmlP=}^Ceq?D0y|7OepyhpCIG_rO81C5`!3nPS_n?ir#<^qun7tu`p_We>FoYTOcnm zZY$i0N?XRLSu)Da#hSn<+DpsB*L?wh>cugSmo>Mt}$@=Y+Rg5;| z==~m|mTb`%Pu_x!Qiuv>x3o;SNcsFNI3xYt?K-~t$m&~man>%LN!_p~KHYe^Z?=P$yal}r@$Q#a%uiB3u%&(*c2-H(y6)Gd|DQ}7iF;cX9_oQ5IjQ35jM7iOg1hle{y~I<`=k6QLVedgo%cCr+|R=B;{E>u DhWN+u delta 1567 zcmZuwUrbwN6z_N2d)waj_78=QL1_!6z@-p$gDlw?Zt7qgt_vg6Ok8N;UjCG}7rwSc zy4x`p|83EzJ8?6TYK)1-%($2)z8Eu?JxsHQ?Zue%#psJKM)yE8?!ow-TVP_m$^G5) z{m$R-oO9P+xOc{NSCT{%dlL1RQ^n91uD=EH4M^5kor<$$SEQ1tn^G27Yx9L%3tc7? z<#kcF7nwuWqT6*5rIJH&=dD+4yJU~y(#qXyNu`ix(){vqTX~lG0)n zFc#e564LqY)B#;$a+nq1QpA)Sue|eljhi7#>r1x0cZ+9*)`-)NlIah7Nou^BTl)0kVN7bBGpwnuu$n{Zr4!w1lj6P1z!`spB zfREac+HOGAJfF?eQ^<|O57FrG1QV-#T1zT9YDM}067J*UC*x16)0(0lp;V(32UHa( zo6G#7(IvXd7};jDimt&}M}SQ<+p$Nu&iLGQ__*T___p(9wgvI7!KlM@TkwqV!fn50 z#9_L_3y6-+kpc6ae!?)$MjMA4UAtGWGWXiB2|ZIwVa)#$@-qkm!Yo1p!Em_7Bw3)D zZ8naQVfjdJskI@2DrAcyyG~dS$F-~~bG1O^8XT1m44ymH&qb8pL5Lyv z5WEPZTTppQrn1sH|plZ&oEu*Z$ZHEKnFFNd`B`_D*H0|J?YYZ7`!Lg|s?E jzhuU2Q=y}G*gs5&uyKkTBc%Vnw{ydU!p0n2>|Ol_gxP~g diff --git a/lab/__pycache__/mutations.cpython-312.pyc b/lab/__pycache__/mutations.cpython-312.pyc index 000d0db55881285ea55d98f3ca7a0da830b74b60..ef915da8a64b97ebf8b4a728f00f859722f44dfd 100644 GIT binary patch delta 1922 zcmZXV`%e^C6vtc@o|l5W32wrq&4)1Ci{alHpaBSw6#tBp)rj=nfM1(YyW_rdy$r@lihQ__jArY z_p#^BUZoe3H4pOhmn-=E?Ehr&Mo3Yfiq7Iwz97^W-(9Y`rcPYGcf7KEM(H6?fwmGR zAvojhS&uEjwhG(I882^E18JznfDqK6t@S8$HEeQWxI~`(&O@8gTLba6s~dv}}Q=6-Dmu5WFrEb<^`76FR;VpO@ zy5T6Ta82|Myes*8H0a{TphxoKlt_;AFD&2$^h(D`TIm|l2k(Q16ZF%KF0KPePHDA^ z$6-M71g&v#0|q5OMK>+wM<5A9(qYojQU_QtEcpnnUF7lNo;qN(9_$ueHm%#ETvT+A zf#2R$igH#tqYU=&6IsDf7#xfm6;W^+#<1iX;S8j(jZBQ|9u-HNRZ>dmZ03us+iUo* z!CXFDQpDq{3yRr39K>4DggmTM=r%O`{+iPASd{gl?!~X2k7i1lWtXaMmColrTD6;x zlteJW+6_63n%WSBmpLA<95wpl22Dbr!1j`KF{WddlmAO+%pZHSs*|r-yNjPLYVyU= zt?_S*F8Kyg_4C8U_f(zdmmKmr*w=Y)$!*^_D#+ydpQ>t_|5@6lj_}sNc=kEOEY0r* zqQ3Je2l?~B^@15xx3gG8_+AElUKaArVn4-yDf=v21h5PI)VlS)izrX?E9*W|&-3XD zf6i9Mk)(N3`DodY2*bC4!YnkliiWT{(I4SIm6sJw3F<*Jt*}~w85(hn%c9%ZHPll) zP|=fr85JJ%77;OvlLUO9qMYWpDyDtcQJv$}!Cr3y)xy{&b_4YX{Hvf=Foud|P zA|@zV7K^Z(2xl_QWXv0nMJKB}EYo3D(k6_iECv*aW&&}j?GOWO0?Zm7rZ73zowA5+ z444{*f?+$OtY5@B#4>f-Oq>KIV!UAv5Iv!p1B;0g^h%yTJz$wMO!kgD)WoDB7^rT@ zXxjivthj_?+6DvDjvID8NhJ(DK~kDc4!1^$7I!Ek-I47zHi;Vy*T+W9@Z+qmSZvb2 z&tgMJv66_aq{TH%-nB>j2WeV&`Y>%j)h#SD+^TL28`N~_U)gAwbmQXMj~pPjVkb$k#q)ba1Ho7f_ zNf}K}kYUZiEtw5NnxiMsbIP<*=8|1x<~BUcR)5Q!srz14@9?&U4eIY48g8}A+6;(a zSiGW`R6ht+g(l=>cidF`{~*2pALV<)Rce$Q;V(PIt(A9;iQB})MPlOiFmZ91xEoB= zKNIyXOI#Kx6P3tB5z3llH+ffMOWp(Qi(45Qn=TJa8i*A|NmugA5O3oM96babFSlbr6_gs!?zTq{j^%7zl_#T+kK` z@q#BMn0T0r(MX8I6CO;A7q4E7Clmews~e5Hcc}i=S4*d=-)QIwbIqrutRVPxXWZ>L zRY=GW>HPb07X@9o@yPX*c0ZbFba1c?CHm~Z!HTB?PaB^3U>_nrCpd`Bb-;fi4{!8 zmae-QT2#Q#)|(0f(5iTl6`D8zQ@oA2O#A{|RJ@%P{o`#JJ_9;fF}t38m06?vF~3O^WsT{8s#$!1e?g`h(zK(AP&9U4BK=EfZf3TGy;bhoP**SziqxS9gj*p{ zCvS{sXa)!`mdPQ}otd}HTgAuFi;5AOFJ%;!q45eC9>#lf^v5WBMXG&3A4kvPj?lr=r8_q#MNp@5`-zN#!eX`7~U<3YUxLa=BbC zQ|(WU%H>R4j-v+P6CzqxyK)YFxyi_aCg#e9 str: role = ' role="button"' if tag == "a" else "" href = ' href="#"' if tag == "a" else "" + subject_name = "recipient" if app.ui_field_names == "renamed" else "subject_id" subject_field = ( '' - '' + f'' ) permission_field = ( '' @@ -188,6 +189,9 @@ class LabHandler(BaseHTTPRequestHandler): self._send(404, {"error": "not found"}) return + if self.app.ui_field_names == "renamed" and "recipient" in body: + body["subject_id"] = body.pop("recipient") + if url.path.endswith("/grant"): self._dispatch("grant", resource_id=resource_id, **body) elif url.path.endswith("/revoke"): diff --git a/lab/mutations.py b/lab/mutations.py index 9a2c897..0ea3e83 100644 --- a/lab/mutations.py +++ b/lab/mutations.py @@ -90,6 +90,10 @@ def _m10(app): _m(app, denied_status=401) # --- SEMANTIC ------------------------------------------------------------- # Intended behaviour changed. A human must decide; test-driver must not. +def _m21(app): _m(app, ui_labels="verbose", ui_test_ids="dropped") +def _m22(app): _m(app, ui_field_names="renamed", ui_test_ids="dropped") + + def _m11(app): _m(app, require_share_acceptance=True) def _m12(app): _m(app, revoke_delay_seconds=3600.0) def _m13(app): _m(app, grant_permission_for=lambda p: p or "WRITE") @@ -154,6 +158,17 @@ CATALOGUE: tuple[Mutation, ...] = ( Mutation("M10", "Denials return 401 instead of 403", "MECHANICAL", "api", "Both mean refused. A driver keying on the exact code breaks.", _m10), + Mutation("M21", "Controls reworded and test ids dropped", "MECHANICAL", "ui", + "Two signals disturbed at once: the wording changed and the stable " + "identifiers are gone. Extends the deciding side of the test-id " + "axis, which was N=1 after T07's first run.", _m21, + preserves_test_ids=False), + Mutation("M22", "Form field names changed", "MECHANICAL", "ui", + "subject_id becomes recipient, test ids dropped. The API still means " + "the same thing. This is the case that probes whether discovery " + "generalises or merely pattern-matches known field names.", _m22, + preserves_test_ids=False), + Mutation("M11", "A share must be accepted before it takes effect", "SEMANTIC", "domain", "Bob genuinely cannot read until he accepts. The old claim 'Bob can " "read after the grant' is now wrong, and only a human may say so.", _m11), diff --git a/research/concepts/fitness-map.md b/research/concepts/fitness-map.md index d41bdb9..03c9b78 100644 --- a/research/concepts/fitness-map.md +++ b/research/concepts/fitness-map.md @@ -1,6 +1,6 @@ # Concept ↔ Implementation Fitness Map -**Updated:** 2026-08-22 (TD-WP-0002-T06) +**Updated:** 2026-08-22 (TD-WP-0002-T07) Traces each important concept to the implementation, experiment and evidence that support it. **Unsupported entries are the point of this map** — a concept with no @@ -13,8 +13,12 @@ Support levels follow `TestDriverImprovementLoop.md` §13: ## Current state +`C-semantic-action` is the first concept to reach `C2`: it has an experiment +behind it (the T07 two-arm comparison), and that experiment narrowed the claim +rather than confirming it. Everything else still rests on unit tests. + The deterministic kernel exists (T04) and its guarantees are covered by unit -tests. **Levels have not moved.** A passing unit test is not an experiment: it +tests. **Levels do not move for those.** A passing unit test is not an experiment: it shows the code does what its author intended, not that the concept holds under the mutations it claims to survive. Levels rise when E-001/E-002/E-003 produce evidence, not before. The implementation column below moves; the level column @@ -26,7 +30,7 @@ were aspirational, not evidenced. |---|---|---|---|---|---| | `C-use-case` | C1 | `intent.py` | — | — | Is a use case expressible without leaking mechanics? | | `C-actor-isolation` | C1 | `world.py` | E-001 | `td://self/actor-isolation` | **F-0003** — only observable when the scenario plants canaries. | -| `C-semantic-action` | C1 | `actions.py` | E-001 | — | Does identity survive restructuring better than a recorded sequence? (H-001) | +| `C-semantic-action` | **C2** | `actions.py`, `agentic.py` | E-001 (partial) | T07 arm comparison | **F-0005** — supported only where stable identifiers are absent. Narrower than the concept model claims. | | `C-oracle-independence` | C1 | `runner.py`, `oracles.py` | E-001, E-003 | — | Independence of components ≠ independence of belief. (H-004) | | `C-evidence-pack` | C1 | `evidence.py` | — | `td://self/evidence-reproducibility` | Verdicts are reproducible from S3 alone, on passing and failing runs. | | `C-observation-channel` | C1 | `lab/app.py` | — | — | **D-07** — required of every system under test. Adoption cost unknown. | diff --git a/research/findings/F-0004-stdlib-driver-instead-of-playwright.md b/research/findings/F-0004-stdlib-driver-instead-of-playwright.md new file mode 100644 index 0000000..b61722a --- /dev/null +++ b/research/findings/F-0004-stdlib-driver-instead-of-playwright.md @@ -0,0 +1,60 @@ +--- +id: F-0004 +type: framework-finding +class: FRAMEWORK_LIMITATION +status: open +discovered: "2026-08-22" +discovered_by: TD-WP-0002-T07 +workplan: TD-WP-0002 +task: TD-WP-0002-T07 +carried_to: TD-WP-0002-T10 +--- + +# F-0004 — The browser surface is driven without a browser + +## Decision + +`TD-WP-0002-T07` names Playwright as the browser driver. It was not used. +Instead, `src/testdriver/html.py` parses the lab's server-rendered HTML with +`html.parser`, and `src/testdriver/browser.py` drives it over real HTTP. + +Reasons, in order of weight: + +1. **The lab's surface has no JavaScript.** It is server-rendered forms. A + browser engine would not change what H-001 or H-002 can be measured against — + the mutations that matter are DOM restructuring, rewording and field renaming, + all of which a parser faces in full. +2. **Establishing Playwright is a real cost to the operator** — installation plus + a licence review, and the usual snag is the Chromium binaries the installer + pulls rather than the library licence itself. Blocking the vertical spike on + that was not worth it for a surface that gains nothing from it. +3. **The stack is deliberately boring.** Zero dependencies remains true. + +`Driver` is a Protocol, so a Playwright implementation sits alongside this one +without touching the kernel, the oracles or the evidence format. + +## What this costs — stated, not buried + +The driver cannot: + +- execute JavaScript, so **no single-page-application surface can be driven**; +- take screenshots, so that evidence type listed in `INTENT.md` is unavailable; +- read an accessibility tree, or observe visual layout at all. + +The third is the most consequential for the thesis. A real mechanical change +often moves a control *visually* while leaving the DOM largely intact, and this +driver is blind to that entire class. H-001's mutation set is therefore narrower +than the hypothesis's wording implies: it tests **structural** durability, not +**visual** durability. + +That should be said plainly whenever the H-001 result is quoted. + +## Carried to T10 + +Two questions, neither answerable now: + +1. Does a Playwright driver actually change the H-001 result, or merely widen the + surface it can reach? Worth one experiment, not a rewrite. +2. Is the screenshot evidence type in `INTENT.md` load-bearing, or was it listed + because screenshots are conventional in this space? If nothing has needed one + by T10, that is a candidate for compression. diff --git a/research/findings/F-0005-semantic-actions-earn-their-keep-narrowly.md b/research/findings/F-0005-semantic-actions-earn-their-keep-narrowly.md new file mode 100644 index 0000000..742c866 --- /dev/null +++ b/research/findings/F-0005-semantic-actions-earn-their-keep-narrowly.md @@ -0,0 +1,80 @@ +--- +id: F-0005 +type: framework-finding +class: CONCEPT_DRIFT +status: open +discovered: "2026-08-22" +discovered_by: TD-WP-0002-T07 +workplan: TD-WP-0002 +task: TD-WP-0002-T07 +hypotheses: [H-001] +carried_to: TD-WP-0002-T10 +--- + +# F-0005 — Semantic actions earn their keep more narrowly than claimed + +## The claim as written + +H-001, from `TestDriverImprovementLoop.md` § 3: + +> A semantic action survives implementation restructuring better than a recorded +> UI interaction sequence. + +`INTENT.md` treats this as foundational — semantic actions are "the bridge +between agentic exploration and deterministic crystallization". + +## The measurement + +Twelve mechanical mutations, two arms, same semantic action +(`grant_access(Bob, R, READ)`), same surface, same oracles. + +| | Discovery (agentic) | Recorded selectors (control) | +|---|---|---| +| test ids **preserved** (9 mutations) | 9/9 | **9/9** | +| test ids **dropped** (3 mutations) | 2/3 | 0/3 | + +The control arm is not a straw man: it uses stable `data-td` attributes, which is +what a well-instrumented application provides and what good practice recommends. +`test_the_control_arm_is_not_a_straw_man` asserts it keeps winning where those +attributes survive. + +## What this actually says + +**Where an application is well instrumented and keeps its identifiers, the +semantic action buys nothing.** Nine mutations, two arms, identical results. The +conventional approach is not merely adequate there — it is cheaper, faster and +deterministic. + +The semantic action earns its keep in exactly one circumstance: **when stable +identifiers are absent or are not carried forward through a change.** That is a +real and common circumstance — a rewrite rarely preserves test ids, and a large +share of applications never had them — but it is much narrower than "survives +implementation restructuring better", which reads as a general claim. + +## The second boundary: M22 + +Discovery fails on M22, where form field names change (`subject_id` → +`recipient`) with test ids dropped. The heuristic runtime scores candidates +partly on field names, so renaming them removes a signal it depends on. + +This is an honest limit rather than a bug. It marks where a scripted runtime +stops and where a model plausibly starts: the page still carries the label +"Person" next to the field, which a model could read and a keyword heuristic +cannot. **M22 is the first concrete piece of evidence that a live model would add +capability rather than merely cost** — worth more than a general argument that it +might. + +## Consequences + +1. **H-001 must never be quoted as a single rate.** Split by the test-id axis or + it is misleading. `research/hypotheses/H-001-semantic-action-stability.md` now + records the split. +2. **The concept model overstates this.** `INTENT.md` and the Concept Model + present semantic actions as generally superior. The evidence says + conditionally superior. Classified `CONCEPT_DRIFT` — the concept should be + revised to match the evidence (§ 5 path 2), not the other way round. +3. **Three mutations is still thin.** 2/3 and 0/3 are directionally clear and + statistically nothing. Any stronger statement needs more mutations on the + dropped-identifier side. Recorded rather than rounded up. +4. See also **F-0004**: this driver tests structural durability only. Visual + relayout, the other major mechanical change class, is untested entirely. diff --git a/research/hypotheses/H-001-semantic-action-stability.md b/research/hypotheses/H-001-semantic-action-stability.md index dcf2291..3e18733 100644 --- a/research/hypotheses/H-001-semantic-action-stability.md +++ b/research/hypotheses/H-001-semantic-action-stability.md @@ -1,7 +1,7 @@ --- id: H-001 title: Semantic Action Stability -status: PROPOSED +status: EXPERIMENTING created: "2026-08-22" experiments: [E-001] concepts: [C-semantic-action] @@ -36,6 +36,28 @@ The comparison is unfair if arm B is built naively — a brittle straw man makes H-001 trivially true and worthless. Arm B uses the most robust selector strategy reasonably available (roles, labels, test ids where the lab provides them). +## Result so far (TD-WP-0002-T07) + +Twelve mechanical mutations, both arms, same semantic action: + +| | Discovery (agentic) | Recorded selectors (control) | +|---|---|---| +| test ids preserved (9) | 9/9 | **9/9** | +| test ids dropped (3) | 2/3 | 0/3 | + +**Not falsified, but substantially narrowed.** Where stable identifiers survive, +the control arm matches the agentic arm exactly — the semantic action buys +nothing. It earns its keep only where identifiers are absent or not carried +forward. + +Three mutations on the deciding side is directionally clear and statistically +nothing. See `research/findings/F-0005-...` — the concept model overstates this +and should be revised to match the evidence. + +Scope caveat (F-0004): the driver tests **structural** durability only. Visual +relayout is untested. + ## Status log - 2026-08-22 `PROPOSED`. No evidence. +- 2026-08-22 `EXPERIMENTING`. Partial evidence from T07; awaiting E-001 in full. diff --git a/research/hypotheses/H-002-mechanical-adaptation.md b/research/hypotheses/H-002-mechanical-adaptation.md index 92d7191..0100494 100644 --- a/research/hypotheses/H-002-mechanical-adaptation.md +++ b/research/hypotheses/H-002-mechanical-adaptation.md @@ -1,7 +1,7 @@ --- id: H-002 title: Mechanical Adaptation -status: PROPOSED +status: EXPERIMENTING created: "2026-08-22" experiments: [E-001] concepts: [C-adaptation] @@ -33,8 +33,20 @@ at all. (`docs/TestDriverClassificationDesign.md` D-02). Any non-empty diff is both a falsification signal **and** a framework defect, since no write path should exist. +## Result so far (TD-WP-0002-T07) + +Discovery recovered from 11 of 12 mechanical mutations with **no claim or +invariant diff in any run**, as D-02 requires structurally. The single failure +(M22, field names renamed) failed *loudly* — a `RealizationFailed` recorded in +evidence, not a silent pass. That distinction is the one that matters: the +framework reported that it could not act, rather than reporting that nothing was +wrong. + +Full classification of recovery vs defect is T08. + ## Status log -- 2026-08-22 `PROPOSED`. Design decision D-02 makes the second falsification +- 2026-08-22 `PROPOSED`. +- 2026-08-22 `EXPERIMENTING`. Recovery demonstrated; classification pending T08. Design decision D-02 makes the second falsification branch structurally unreachable; the measurement is retained anyway, as an assertion that the architecture is what we believe it is. diff --git a/scenarios/__pycache__/browser_grant.cpython-312.pyc b/scenarios/__pycache__/browser_grant.cpython-312.pyc new file mode 100644 index 0000000000000000000000000000000000000000..48a8f753e25bcc1cf8be7537fba55482e27274fa GIT binary patch literal 5052 zcmbtYT}&L;6~41Gvpf60EEpT}lL745i}9|12*KD*g#nw`DcFGHdgEv_%na<3nH_Rx z)|l-gZ7NH#>#8AAR1>99sMH5XP|1C1-Iq2=Um~>+Y;Xh_5v6VX(7d@ZQ6j4kJ$Ghj z!9VIt@0vUJp8I$1z2|)AyZ&o1=tGd++wrr>lR<>OA{8gmc7eLT28A0)LP;c1l1rgf zSJFk1GOf@_8orFesBDt8WUS&=JxPzsB{|ib^s2t3PxUALY9JXfAtg=_BQ+FnJs`bfwtL{@8)W&3^ z+LUafkPFFAN&YmoNZEQPd6|{MQs4@+Sl&^xS&B%(D@?LQs**xisATI59gF5ZfB~av znHMJHtdS9UEDK5|FBlmu%j=??6>vuD=C!QM>#{0XT?CP|wu6uttp2>7!{dS|$9=vL zQJa$ajLu84k{Oe+V93ghHQXK*=U z`RQ~2o6JqS_v=fKiWBMO|^C_4#41@sGa(S^_~HR~!+lVzvhI=tYLs5%fx zE`xF861~JMmTMF;JdV6jalcvG@H)~-3uK&{{VmXwfrlleCb3KIMbL(%S*E~DhVAm! z+;L}s01wO(+1h58C64I(E_%;3%X(3PD!2;lIQ>i44T?fAqWbX>n{z_@w*S>W<8A%F z=a@#9=?iF@`Yk<8#eDhadw2!xl80-x5oH|9GC!qZBhC|GDr1WS4gA39=$UpN%pbsK zJeRdBmKOy@iJL4ymnfTI5uD7BQ&`ru92Vs~KnZ~_5F21Qen+0lNT$c~oUAbl4Q@lg z6JXYENsv`7o3Bb6##DDAaq!Ty@dNNX*!?vzoWYnMlS48+Sbi-h>jtic8YbszdI7X# zK?0C5-J@2~3|c0g8rJ}2z?oPbRx!+w?OQ29l0Y5ZdI@iamBW7@a$OJy|@l9;{srHkN{o3oowjK2q9! zDmW9sM)~-@(*X^oO>(S+wW6R+m6bJ6P7`ACGR5jN* z-?bcS{gi9{I~RI0d@X#3t6Sqjt6Y7FtN)B^{2mtdhM&;zAH4~b|7lNWubcYli061S z`WQ7s`Qzr6UYh<3O+)!H$N(sy?*AUfZG{WRQJz5;p?%hZ4FgJc0!O24u&6jO4GzUR z?VDc!6LLvbdKXFbd$bd`Eihtc*-iK;uuhxkk{I%HW~a~yYDEW;PE7+Nxd6~cp&7<@ zt{9si=oO}n9K;S{{pDcD3s~KwgD;d}7(7mtA?QsAnJh$4(leK}3{I9D&x_-oplIHo zaQsG3TYbIZ9-`SqsqVmh>-?pK%j?m)`MnD#Za(|#p_ORI zRnJhhCMODFiY%Wc^9*81L&=tB)1W& zlsytRNiOS=1T#h-5kb|_;WDKY0_-Ib23T4pUPyi^U~G5BNkJ(D1T^w1m$9QFtGa@g zqK;P1-o~k=>c#T;%zC66X`2E>d6pxjw!k^(auIsWdg13I`R0BKo=M`}k0g-tkBFcO z(v~MkeUrPMAoWZ>{kU|gc2iR$=Cl*XFVJuppoPDvjO9YTHj4Hn!5e<4r^8qYX=96iR_5fs~*KQshf-eLPX zFuS7<6WcQH)d2`ZAYK!)6pzss1eo-gHfFM-0Ib?%tuREmD1=psyd4BZ&wHo$m%~Yn zF<$AkNGFb0L^oTP<0ndOCqL;dwVf@Z`yfPw|5_^>hAgGZ1V}f7Q<`o7 zkV_fD(gAyqL0^;0LD+{gl}*njc&x$O%JfVNhM1OhLN??n9h3bMa&2i<9*2JfLVSi~ zLnIr7EMId;c>GpO*XTj80DCcqHkh0$KyZPFO{^Z1SO(+-s0?vqFpZ3%$e6$k=wx3z zCUh7RbTFAoEt55?fXgTu*j|O+V8VzoAjS13A&NErw@a9LCo`q zuXh&v)*`WQfKr{Mt}<&};7#rt_x83qV}A7gS`csPfB4Doe)37poxotR|898un!kCys%|a1bN=xAUGH=)^seqYQrdN7 zrM~;q`hnY%%R8T6j+|PHMDGS7b35nH|0cT@Xq&m->8n$I9-`64O%bDZaMsYnT@ns@BP()o&iXc64FKYV)B| z^P$^&{~Z5gd@a(l8fhy<+HMW3cJ!4x`Zida%(US{q5Ao*g%@w0|8)PUJ6!`SU8nvw zuo656WHQ`vr!jGhUTN&Om0s;UQR+Og5Z!sIC;0EgsF`ECJ7SJrk7b>K!P@YiDcH> z#qvwMy${>M+@| dict[str, str]: + """Capture the control arm's selectors once, against the unmutated surface.""" + with lab_server() as (app, tokens, base_url): + session = Session(base_url, tokens["alice"]) + _, html = session.get(f"/resources/{RESOURCE}/view") + return record_baseline_selectors(html) + + +def build_agentic(app, tokens, base_url, runtime=None): + """One agentic step; deterministic oracles unchanged.""" + cast = Cast() + for name in ("alice", "bob", "carol"): + cast.add(Actor(name, name.title(), credentials={"token": tokens[name]})) + + world = World(id="w-browser", sut=app, sut_version=app.version, cast=cast) + + scenario = Scenario( + id="sc-grant-via-browser", + use_case=USE_CASE, + variant=f"browser/{'+'.join(app.applied_mutations) or 'baseline'}", + watches=(), # filled from the reference scenario below + steps=( + Step("s2-grant", "alice", SemanticAction( + "grant_access", + {"subject_id": "bob", "permission": "READ"}, + permitted_surfaces=BROWSER, + postcondition=lambda obs: obs["state_permission:bob:R"] == "READ", + )), + ), + ) + from scenarios.alice_bob_carol import build as build_reference + _, _, reference_observer, _, _ = build_reference() + scenario = Scenario( + id=scenario.id, use_case=scenario.use_case, steps=scenario.steps, + watches=reference_observer.watches, variant=scenario.variant, + ) + + driver = BrowserDriver(base_url, tokens, runtime or DiscoveryRuntime(), RESOURCE) + observer = StateObserver(ObservationChannel(app), scenario.watches) + asset = VerificationAsset( + id="va-grant-via-browser", scenario=scenario, maturity="T1" + ) + return world, driver, observer, asset, Oracle() diff --git a/src/testdriver/__pycache__/agentic.cpython-312.pyc b/src/testdriver/__pycache__/agentic.cpython-312.pyc new file mode 100644 index 0000000000000000000000000000000000000000..836c8670f13714cc7b7697023a594ee785e8e743 GIT binary patch literal 11662 zcmb_iU2q%Mb>0OQ{{TS%Bt?q)SxKZM&=O=tQe#mv9mS+9SvDO@%A{r@!%OUv1c@JJ zcR?Bmm@=`GKpD?SirkQn<1sURC_Q#2nm&1^ovGSPr=7MBWS~^Yj%K1XnQ8n+iE0x2 zso%M~3y2iuC~bEJ7kmHCJ@?*o&i&50`){F;pTm=D|L*wXJ2~#xl(8R|`NFS$hUd6B z?u(qv4RJCrJ2HIMG34Os-I;L?Iq@%Kgsf}Gm30rfS-C6Y$$E#pS>KQkdAIDz__Kkb zKsGoOMBXd=Wd9ExLm@e!G|9o^7)TDuO{^@;%EDCU;1rkK{6XY>j8Bh)dxn#vuXA$j zeTP}{zWK)uHE-Cq1#Md&Xd7X4+EBMiiN5bNNesnUX*)_g6kolxMHc1FuM0!1@(#HZ z|81<*7PQ)G(;eEx%C@0wd%di^=#1|y{EfhICsKM|71cscPiGaaR~*Yvi;}1*SqUX6 zQA+9Qd`=uzQh8LPCY8^nR7F?BbWT@PvpV7TpRh@YX-yoHahVgDN~Btpoc1j@%x{%Z-S!5 zwK$rdQgX=GtH&fwl+#)&kKT&x&*k;8bZ(UKk&<%Ain5Z;=QLfFFcV3ZCUuGCiHV$_ z&f9b~F$2Ql7-;NceThlTAHBf(u%b^Z7#4+DVwEaS?SK67gQD{4q@rRqb9%4e-y^=# zZ*fvQYp(JuBC${HrBO7ZRaUbJ@mwmeVu`_^oR*0%HdH>NiP!*pL;RwsrAKoTO7~8s zwe)aC5q0HNeV;gzRx+}fldw(5c zUzcA28O$mzYdt*)@$?8bLY&6TMJbcXPh&UryjajM7@IEbTmi!kqh3@Y94a=Pc(XTX zT9H(?v7gZCn7LQkJUXcPVG={OT=@~|4{8`5@Bg3%vy*wvl4ocjM1<6{ zkirU$U}?-H0KJ6&BeIBb-Uu*FW_qg)#vO)_BsrClG|h0asi4exB3CrLUr_UUK9$dW zM1~rr-hRsx8J>ZRLW&x987^?ADy0A!6PUe}LUcw|m_fsx&P}CtrR0}nS%hwb<+xzD zRb>imS_)66^)Z?D-|STisW2`LBA!Byk;cG5MlWemM>`^5WlNjobV~m+CcRJ83zHee zd`%=07cLmyb1x0PF!<8*=Rm}-6^6%Q29s&o@PnId8tVp*cr}LU8R4@7FMjd#*@6D# z$*V1a6>%KPX|s>9w(+l5o9Mj_k_X1X za-Jc#>_yIJNjObY? zSGvSD^?5~C(aJF$;?cnxG+R6XtPZ_8`-! zD$^=NgpE0@GF2lJuun{>MHsHk8hR!t66(hk($z^-VJ0$#IpGXs(oVyOQ&Oe?6;$)k zjWj!;M63j~uA#<^K~i+$0y0hq4NIvBQqctKY&aovh6kEW5Mb065GYY5 zFX={b8r&zTDw)+(0yc&dXTQ-}Kd7`MDQjAW-IEQkwG$dB(grG;;bW#pOQv&%-+IN7 zKUnf(T(K+;VlNFBGaxAoNiiOLNFk;C>c8VbmxN9JwJgL?P+ zTGq?EVJX9%YimFuxVD6HC~exXwE4kO+3`c~IWF!T{D|0&d)3YOQ@2pEjgsw@h?MN0 z1c;YYcOi-USWCmD<_kIbQ9AI3fMak(qXiW|`OJYaz_$ZHTUu)D014)PS%sZb4`^zN z<*lp*ND`Ap!y{=)vPh-P59(yXsyhMd+6zc#x!apM=gL>gx3=t9=(@S{(DKeh*Iuk{ z`E0E_ezW_jLinfKbeH=6f*Q4OeVitkTO<{Cz%A~L~%Ho zyzi_?CLdt|)i^kKk5hZ_r|zYs-A)pC(T0$Gi@PJZUEY;ut|{``bG6M|XP>*=T@|+0 zcI;ddI_3{mg`KsQ*4a~)R8?rL#ad>cshq0{Ew$FR+0zxRDzw$wHqAbJ*;y4fVRdE) zD!o-<%RRyG@|(lbh$;N)k0tB;>L7|f#b%TB7MN{qSZ*rwurhk{vNs>znC&hG;z2|B zLPpA!5+?}TJOJ^TVKaq<5Gu^5k}Bv*LhS%6hC8e1V|m$doBqHj_w5&S{DR>&?Umt! zd1J6jF@gYpWR{bJ`q|6w;*fAWM3fOaxzZHz4>xkZHP;g$f`7Ss_b zr6dO9!iLjcH)Gm4l@by`jhx`uIYT&WPKe6aOsFn0B5E2FqGMjcG0()nB|C8d3q>LB zHi9R}zdCE41?J`oq)w^zwiSf=RVtk0irjnrAln$sU=NB6ukGpqq%jAQDXSsCA<;DI zM29p}D?F6FoYEu7)Y{5VZ{ z-6Qi;IJrD*)i z(A)GV4KFPc@VTTu1;Pdz83KUGgyGbsQNsn$p=ySc++BxyhFW+m`@&?%7PerL86qZ) zn))TQo4`*yk7SlxadYik=89K}SHJcstDtRm$Q03Z`&dZH*x zYZflHLJSmCDOyf186iTCpr}kLNPv{>2m?Gv;9VWWP?Q@*sKK&TCMpIL9z{n@9D(DN zL9Bzc`e1@tI6F20$RQI}&PsX;$W@d?tq6Cwm?aPos4J2wJjQx{#_Y*_aSOjhDNShIiQ7jL0Af%KE7ZrrrdQsd#G?zW2kbjNIke6Ez)!4B!%m2sJaj>%O zf_F%30u+st0e*!nWe{Lu^!i_T-u5N-i3rR|l(W{5ycC2mXd85(cPx7vfsRozlP}1a z2ez9wvmQ6X&M=aQF%=LGL5or5x!0p0Ok^e$WsOuQ$W4aA<^}faFjC+)u@xbJT?VgSQI=|ksx1YQ4 zD;Q3n9nG6GA&uk_Dj)Jm#CZun&dsSLSc-iy1 zx9sZYY_9Cl)wXg-_Lf56E>sSV(*e5fk?WlDT-mL5mfdCl1iX|-=;^mdK~Kkc=Q_g3 z!WXkvQg+kOGfm~Da$tPxgDq{oal{bEfluW)Xj8?wA&r0I@5`QzW(GQ;adNQMJ>`1; zvTwxMp!LSD1y0{#H-?wD(|&LK>)rNy*=zTdn?4BJ@%fo>*`xJ7nxvNnItbe7hiFb zOw2SxYU?96q!y4Ml5(U85&(6~M9Pt}6MGXbM|8sI>?u3T5!zLohpUb^HVNEJ6s_G2 ztt&_EdYbJ_tQ=F1{=coyCbmB9I(ZB1DaY*5-savB?Duc-|HlytLz?s-U28k!&GxE~ z6GmP4*b@K0vjgR*+*uAm2DX%&X{MlZWZeMB*2f5cu=RZ_7E%XZ5dg1kg9e}cgcnVa zOcc}?DIvhJ22Vm*I81_HC5P)T`7<2PP~`=LQV1_zbWCyI=jV8yd&4ilJ@~P6nrD#L zbP}9~Bk?%k8;&(YAY7*rJyD!xc=r?EahzDVLs9!md)K2M<{Bi9XYx3@wZ25X<(nYD z0P!Dw%l%=N`&snJLjUE%^M~I&z8pRBF>^UyYl`qH>1(OoK2U8?Vk3*vmNmFTWf*`W z-*mO2eMHMp+Wc^kQF0s;Rf;CXon~NX8{Sz==cC_ZhNWG=Q8olv=lmha4cr4vR|<)H z8__he6l~lFY4J6CK-@r8`Bgf{2-C|vsUN~ijzP9#=3?`PQ_WA`=TCg%J)V)W!?M&@ z+VjVj*J}7<&}F{SEVuCVBLAb4v!+jW9xjY?LP4wuk!*y;l~T~2A7&$#BBGoJ=3`eR z2&hlu`rs z@J|(0jZ*y=kq`rL8dC~lnTR2@Qjv@KQOsRpNM9wCZ#X4f2)J>n1E`CzI9zgFeU>VO z@q9XG26>o=;mjyGGfqT4GNNgDRkwm2^s30r(==QKkWw)tL^0y6%=Ym2-L)LuRgHF6Jhx(< zbH}e7UkEHt+)SKUPMo;D>$+A=oLSocd^I*$@!pEGyqUb&x_7yC?_%e*;cDyAYNWT~ zzJ2ge#a(M|pKH3(RBaY3u3AgST`uZ7!B_fk#l(e{YHT;&TIj81cV)l|2F|^F<>iG_ ziy(8ba`N_$U6p4LOREgr3P-EaBa5eQM33AEAE`CB&PT5JFS}~Zn~46>_MyFWg98s%KY63MuNK>OGuE{n>smNhjl~ytEXVd$`fK6XB0Zl#+hb#E^h>#nu5&z-(<`qs|w z1?8`L5b(R_5n}#|=Z>FiZoAdB_h#3T<*p;uuFrnBXSwUd_4CWSPFJ3}5fpDlch#at zu1#D^u5kPSzx~$soeM|aF3o2a_pLbj$P=qhXRK+3bB3B${G2a*DR42c+}?AozY@6N zJzCqb2f7K@bX&z$Db9D@^Ku>C3+JotJ#c5E{@cwhG)etp8IEiy^5(9~^8Dc|<5v$X z2Y0ME(V*gn>l*cga;)Qrhvui=KDHd)UGe-P5dC8uZ=3(xe12(v-``DLAN||ESn7LW zsrd`5P6x5-2sMF^9o;v@gUjNtFc2jV@H=`N2{@8YpXtAjUA}PI%@6P<_q(q3uhO73twAmJ4CVMj<-7A?yR+R z-V1psgzWM8Kfc?E$!K)U{d4E$Gj3r}J&E_Yd&3wT(<9Of^XTiLHx~7MiQXvD=m&d? zXxkk}6QaH%7u-JEe^2PM!?n~Zg@1;pRnMOWNwYg@Z!VO(Qb-fsta!}JlveluW|SLhrvnoO zr}W33vd88ItKzdGr#m3`#8!A5( zc87`BiP@6J`XovvYhLQKHM{=Bk@#h+!IP!P1uL@2E`7+Ln9!SGu9~K#?nRZl%#V>^ z8b5{NC;SC<9R1=Eb%MHQC?QP7aBc&invjtot*TSPTn&mRv#TK!@Y04dl&VhC+p9?8 zfsG)lT11uMwb?T9Xxe&YiDUj| zY}ayZ*Ui|T<=CFZKs9z4crx0xc>X6Zz4y}f$c^aJ6%XR`+jhVE@>?%gx9tU$mb|AJC==@PH!(;mw#}>b`boAM!<}(dY17?~x&+oW6QxSfX=>70;HSxLM zIUP}dZFkT0{(tHJ$LA{j>kt+y31Q*y9{RSdg*K0ZdX9hi=N~?MU0P~8^)SS<9drHo zZaaq1=zRHyjy(grg`al$1|AcBnus8!E5`4jMy0DOc3)-)^!bGejXV#BMipwm5k7jI zDk%9^{4~0;CVb=pK8kX|J+u9*0`J;=&)sQ*Bx(vKNVrt8bBqb*S3OW#^1_#+kDxBZ zJJ$D=++ag5!7nnhX4r+xtsS{4Z|TN|0+EicU;cBj#=-LziX9q@(wCL zSZnP<@d3wdcq#H2E4~xt9D&O%H~6;SJGSr;ZX~Nt&ar8QvUmC#wO(Fa;gDFhc)Qoe z^XK?^-#w20?|Gg4S^n}9_b}R7z!9F`x$1TCq2D*Z$b%9`>F;xn9^QYKL&7HcKieZn A6951J literal 0 HcmV?d00001 diff --git a/src/testdriver/__pycache__/browser.cpython-312.pyc b/src/testdriver/__pycache__/browser.cpython-312.pyc new file mode 100644 index 0000000000000000000000000000000000000000..30939fe22664f600e912cf8f0681da0e51108ded GIT binary patch literal 8516 zcmbt3ZEPFImAm9Fm){ggeOcDWI<_Ox)|YKLP8`RMW!bTVL``j}vE}5L6?bJ(BDw19 z(u!EhG0+r{dpX#`9Y7mNVFg7I1ql!rC{XuTb4h`wxIa{xLhPP{rq=*>`Nu>qa&Z3K zd$YT=L@ReeZY0jfn>VvFZ{EDG`L{s8%Ru@`^G~OIjSTZQY}ko6E9}}Gs1z8P8DnHt zcBI&}W6Z%~-I;QZIcdp_@w9YVr9ewAt0|;M%aREx zp@=d`UQ~#vVx~_jVoDnC)Wnk`&psoLp6c~_k4uS3QA+445icmZCMMyxYnM8k*|kg5 z6irL28Sy_Z{|ki!R2-KyMI1f-j5wam$jQuwZo9{uAd0MH^rV#1sz5QJs?$lONBLnk zc~MG1cdt0C>XR@~OlqPwDZwCdT+wG0C1Wl_6D8f7R^@C;5n)k@l%#25O1Y?{x&bjE zW#RjhJu;C>!kjpj@-l29H78E0v(Qvil8!4MCyF!;o62}+2=pg`AW}w#C5suDkw7D( zW-|$e?psMsh+QXo`uqF$#egy+KrM@U%F9Y3s{@uKzVuv5%Dg1b5LjUX*FFc6h%X&Q z3ZBlw_S4Esz2YfdlvTjbs8CT7Nm=pEPD;9{tFWPDV%lCs4U#xJslXwbC#7cfgql{w zSxEyj5~3vZgY*a|l0?(J<4Q`+T+nbzJXd0VyPGyNC85GLtuA#O7{c5wzyMbTTBMX- z@8`%|G1l-Q8FWf9Z5Y}fS<(Rm2oDYC(ahZE$e%HX5j>-$CD>yEwhQDngfk|WL6?6$ z1&r`%MM@=ebjlM_GNs6X;#&ta!lM~2J2RsaU6IY%;9Kq?HIYrji9(&H6OLd_OR2h+ zuxD4t*~JjDklBcVz}{kfbYc2AK?;0kV}vSnG}fOH*btxw+SHYC198 zcv0e6k}`N*omMglOBB(CyCH`yuxoqa&K4M*0x_Giu6-4Nu=OLYj&&nk>v7Hcl;=TM zo|&Rza$`^4VGEPIGtYwfa{!)yBM!D+XMW~LvUw)~!sfXon_-j8FW7gvcQ_Mwp7;D; zy9hlvYo3?+cU|@h_A~RYylbj%6Fk>c^g*aC8mAh*xo^UeaL6F`Wybyr%ba1hGkX|~ zodpGOj$z?%j(OR6j+te*uipcxlG-%YTnk~C9Sj+m7xKbX%Y%IY{eP_c9ISd?xFnph z%2{^ST4zicHUdYp`lL#%JC+j$6%eIF%=wS0869p=&&b@25_1|H-etp;2DzfjhA^o} zvO=`akr&`<2`H~X&;awhfLBw=aYG;=tQAc+gws}qpOJ{B7y=+nsmTn%J&}6IC`g@A zGKNb7@sZUG4&<$DxIpBovSRQGAu2IECr3t}J5H+{!eefO(?Hk~B&Wg6NcyC~jjQsU z;fFhhi;wHDc+!U5ZlY*2Y6c|OPH1X)b3`lM+uJt@0 zYFbLMcV^Duyc-w60_6!1BO*2j4!p++EyupyG5k1pm7Kv1`Li&#XlE+$I$s z02L1tjjb{6sCPlAL^#j>JK?56`(pcQxN{}kxh&j{6vLgR@SzIB29C1tyUR@-g;y3| zS$4k{csuY;XtnF`O4s31*OSGjC%dCMGzHdyP#5FK!jNK3^c1r zW`4~Qb^fx>WtX_E8VM+S`n#At%- zGE)vkuja1guJ2r`U(%OPmqPtTcmEo6F$qgcV4bP1@l|Mi7m*91(%RLOKxf>L*}`bv z3A^7!{NnEOpqV!Ao_p?lXGU_{q?s8o3aGN|d%0(JcD83iCFvfxZ=g5e!pJe6w8G~| z2V{n)3O8J4zezmqKJg6KL{dq~nju&dXdH~hYh%*WGoxyH96yo^+f=QB4gjjQA2RbY zbT4+BmtncF6zN}$99W4QC`BGy5NyIfytHFkDDD|4g+`0+QL+z4ZlF9;i*iQ_D*qdI z=+*O@!sXVaZ%6+c3KQH%u5LYnr!+Ma<0#1t4jTq9XVWtp!F|V^>xd}KNOLJwk~Lgc z4efAX4eb#0IACgc3-0qJkmp_QJ@4DzTL(&AkFRzOu5=BSx`v7;&Xz*witcliV8byt zm4+k&V=4ceHJmEvAD=_pGY^(?Mj^Wq1JPg;cu{EGa1lk%lFYr3o}Ec4=aU(|TLd%Y z#hB|p5_uHR?*-*#LO&0ECIWd9=Qs-42Gqj<$gM*KBk;v_s2;-|k7uN`5|10+c-%DC zq3(~zf0&h0R*w*m%W49^p?G|qi5ZVm>XW_r<%5_Vf$VLDQi3ul2_qCU)Bzew{+}>^ z1Kr`Nb3zkf|+`8{hc2mb^1uAAk@SaSv(mw;b2S+a^<8v@XMYY}Zl z5A)!kEXWS92|3v@m~+K^Mqto%w1%jEl{;zLws1_EWg?^f2)3ez z9>Rp_D^aUItD*CQijJr$O$2Ka{0Hif%}5uN-VJslT7LBEjMIgK2w~#YUFE@6i&q6L z*l$d{DwOjaw5r!huL>lz@MI_0ZB`#(Rjtc8CdLmNf~6}BZcXB$r;>c%w>4uRg+hUQ=3-)#tp%;pFz4& zv5$6z;j%^w_CVjdaT+cbZlW#6fC-bCq#lpw8aEtAuMHmrd`p&9f=tUp3*2kLa#I@= z{&I5%6oGPU=K}Y!d&`}s_HwBC`o-eDV|Sg7X72*`PkvLDRf!67RS)C^Xn+>EPyS6! zchCi!Niue76Qg|#%xD{_pc~m?xZm^|d>=Z~aqoTM34VOLy>al*UzqoNr*&?AYXp6pJ@@833uc4I=0tsa6}Nh4WbYfC?8|#L z-noQ>`umeufEX_EVpV`ONjk*qoZxF2ZquR}S4bMCKw1z~Z)6%XBogeLhAgdp zNLm~xr&estZ<;Z5PZU?MbS~W?K`{bfup*p<;FKD|gb9d{W{F~W(n?}d${=A-x@Z`P zqt=d_F6(pPW)=BZ?gjkx3UH_n$DV;k^GO+Dp?mcrS^vtLBLJG~2@N;A{CeRXU1%GU1X zk@wEOeg5{2-|qg+?$Xv1EAD2XfxFzkW5IK+YsJ0QqQ;#K)L7bb>)h>&w)bquCXTgX>dN_a{E~akl^?{p*ZXCLKD-&MfW5BefJ5bZrhiP z*Av`0=cnyEZcg8rF1K~v>3A5-viUdXZ%y1zlsX3QJhXF3{(0;R4_AcS{dzIz%53O@99KGo0szTb1I!TI~gj)qUwIsf2gq49@09`fOs z*No($f+s(~Y!EZb^yn~9d11PtPeRk+(ES8iLHQf)3DX}$O96~hL^0h`*E$Tm1_bRR zvmdW0c)G_BFofB0%uZl-4l{bA>1CeB=1I&jo?tqKPh$-;bQiT#kiE+MnLAzK0=)Or zNJE8(vf^UG9Tfp$rIi z^!v9RhhZm&`3k4e{J*|uw6Db%y@y~3lu-ifqf0O zzdKhtCe1)tLZPotmdh(jBn_`B5CDe96o}l46GTmm5`9=qCE?wLy1`yA%Df>3BGL+| z6a`-PE+`QFP_x=We7@7AY4El2F$*FY@D2vCUlCr};B5*bU*`K6&LgT56AO;;foY$zmEN6Hmg!-yRXv(nWZ|gwN?DYd#i2cv+uJfoEmTk`pkZ-lNL| z?K-Lya<0A$1m49AKP}=?0$vogoB)o`MR@x$0%mE!A@oxLx~}=^r7M@N%caov%O}g*wqJg_T-RI(E(S|=;+Kq@-@Cwp)+p>+ z+=a36r7g>YrAQw{0K*N1mcT0HJ1JLAS|yQxpL(CqaXWrlwL|l|TABD`EIRCBpa{i(CV}h#+L>#;Ig_5*A4% zQ>>Lr3XC^RWRa^Nj^xBnWIDrdK40|GCkGlWt%A(wi=c&IKX3S|kDQ7|z$wJg!`uwe zm*A~61S@=FN-wtw0v3eFPeP@mmx9a(2F)}+P+yxX#{)Ka;GC39sHj9KNo5zMevM?i z=RRac&<>M-20WBNnhY7l>RI*+hnMC4-psJu|HkO8$#nmP8Tbpc z?-Qow6Q=nK7vu0234RKBN%EB1tvm!-coAK@RqQ5JiQfcbC@-(_c{ON`~UyUKUY;n1e8OK@1NdPEeQWgFMd(IhPeFqXv_$P&?gw8 zA!Wp@)F+A5mNRlz=~J?5pUU5rOduQV3-Y#_31!25VNs9-Gcqg~fj<*J7Cnl6QS>QB z@Mg&Gk%VswM);CoL_U_hIUjoup)bbAq8N*r!H?w z*>op7x(Sk_gjfq`1UgTKnvJb;&j@Mtmhzjk+rq)v@?wDX>aIf^;0R99(DE*1Ised z%-m2KoEW#pz!}}ZFJhRZB!*lnj{|6#bP7$AqyQd)KXfMB#BL$|;jx@S;>wys`as@+ zAgw`dAf3}$8g2TxHReQ8;2OR%1kOWl$8e07I2CGrm5Xn{1PL9dA>Iz@X)vly`(789 z@ApU|x^|LTj+L@9wEqZRS2&fDg)MN5;wx*=Yl6GYkqOZ%7{dFLNzt znue{Vowms5WDNW%lADs-3YMJLOrdS1aB_T>c9*W&~5{69FsIw*V%lY)A*uONMQ# zH)S6h`T~Z6F||AvTs0Ol0vHR@m=Wp=b1GqfrV%lsH)B2=^hJy+%&q2g>y7v&r7vp6 zj0U6Tl2QhnDx==0#Yi>!>+p={BZ&=e@Q5cUSB4?DvZD{-baBHR)WZI+=?_#Dbf>l|LS8r^Oa=c+9v5$jJqd|z0|8eV*E1v7z5-ww zbX9&dn;=}%p6=Z{Y-P>8;})}1!+T+6&Wj!#?X_8ozn5NzoouFUG|zToT_INDR2xyo z;RJghs{asvb5U5Vu6sYSxMA~5|8)N&K@2xf99i77b*3;~xLUjyYnqQWJ(SVA7;AcJ z$FS(#mi^f4OM$-(?HAYZErdawPh7r@20WX1dNo(kN5zmX5f@AT$m{d!Arbv0RbJiS zC0rI$(vXmn`e9a_%YoJ7UVB=as!=fKf>=ODSDEX+!k-R5yT+HyDXbIu+HoNv_dFG~ zHRqze?x%A#qv7jAKTb<~VMq+?S{vyP*wX-#`_bS zIQMRKjhIkf+kp;Gox)|}Ko1AF4?Ww3QCCc@-WPibeTCX*c0_x&-#~Ryco46< zd~WjGL)yZF+6^X;Tz5SKqNI_6(ya|CdMX#@;@NYwUJ+?!Asuv1_Ms3QKlAx0}OKOvX26uyz$ zkWGBhG>uyk*_AWA0&Mj5b&T7GQ9)CoosH9tS0Z!0fAifN-(7g|wV!QkpA%>G56utE zPq($t*R)Tli=I0FICO2p2l0h?>zz>RlSi_!{SepB59>Q>12EzMGo3M(E7be$eU;yV zBDLdtY%{7vjT=fPb9&ZHCf!IfnYE0u47Fp)4mbajyAJsPS%S@6Q{y%yus{v`#2f;l)i` zX7bbdDM-= z$~n>IDeF~fsv6E0y%*%l&fSp6LJ(t?zI#FavU!!lfYW7g0-;hVsy;j+G!q>{T3*hI zY-?ru{~Hx@7Cac>XBii6C_QXDXny`*DC}!!B~;J6PN40%*ML4sIg(rT_KDZKl1JV= za{Ne7@2MLBE>RENxLu1Qu6WiBnqJszlX?NnP&_<~Q#uK}fIct>(Qu)$EbnzAomgp~ zK?SMTZJuqqz2V-DL!UI>lPvdv%zC96K z48<--C!<$7C*$`*+I&d67iyUgwaf+YhW0H+tACiCk2WpFYA$zAcHfI_pO0;y9sexW zvLp+!<|oUlpdA)DE>6{Uim}QH^iSB-|5pWCD|q%6jsoYzp|mV(){9>DHmYaDSGQV7 zPzRFWmWmSe29FUD`pEy45I+m;djecuky6LDSPI5kE`)g8%>NpV)lzcGU;v{jqg#2M z;2W7C762871jH<@K<2X-a3ED)SL8-t2}g_K2%<_`oYMZMm4@(r={4j?d4Zy8;Oysw zN{r6KPN7ODuF7zeV7hX~%rPz@3M5KF12;R$r494k+OVOjB&U*yHm(Y zdlnlSXS%1muc_AqcN$vm*KB+k#B>jRSBr&M$;C2<2ExFKX_FuBfz=W_Tp8O8kubKB z01b(fq9U&=C1pPKFQc0*@0lvBv|Us?_+>ZUIcSe+8R$cpBt@GBSMCW^S+g}U;!k0Il7Mxn3Z zW;jf-lrL(;SP>(@W2~qVq!=q7O2phyS1D}qfUgHDJw9weYYmzx=-_JmduXgi6aNdO zs+I0Tisc(?IBls?9^zA=TkaywAy>JsfJ#XrPYFEVhSZ2dt}=*=JlvUNaFNPr{|Om4 za2eZG=t9H|4tUYyT3w|4!Zr)l{^{HJdp}m&z&GgMztuf=bvJp?_~qDS>`L$5&~wzT znyi|N-;dW_IWTi@`rx&Oh4`*JpA}-%TV>Po@n887`Y-5;KDqqGR zip@$_BTW>Q73E5-GQ<&NSF8@mQC)dPQ927~Ey_p}ck)XVxt#9FxDt_iST}Y9$dzr= z;Y{WE2nSj4eHR9C>820aZeU!`Ii|sW7hU{HVkN!_`GHTNw#;U~i|oxN3>evz7qMktYD>nh1|q=2|=ImCD&dGd{^P@72OW9~Gmlr^q{`9w-(r(P7MI zF@^^nn%i_Kd{g-y`zjUU?;|v;78M{2ZsNk^h1s_k;t0{3xsvpNdlp7lu%-}c@wwB2 zK}us*2A@{(YtPaRO~0Rf61XQq>*1*ry>66e6S!XPhECEkjuYUFXXsP6QL(E6T4+zlp^hL!SC z;K4Cu&Pah7VU-&#-=Lc|Q_$n3zESU{lHZ~;GIclLAHbb=}i=y~Yiiygvw+dp@zYDFu5MKC&u;rIEn;%J{ z`tq_Y*GdxSIdZpp6xbX{Vo0wP dict[str, Any]: + return { + "runtime": self.runtime, + "wall_time_ms": round(self.wall_time_ms, 3), + "candidates_considered": self.candidates_considered, + "attempts": self.attempts, + "retries": self.retries, + "tokens_in": self.tokens_in, + "tokens_out": self.tokens_out, + "model": self.model, + } + + +@dataclass(slots=True) +class Plan: + """A concrete interaction the driver can execute.""" + + method: str + action: str + fields: dict[str, str] + rationale: str + element_path: str + metrics: RealizationMetrics + + +class RealizationFailed(Exception): + """The runtime could not find a way to accomplish the goal on this surface.""" + + +class ActorRuntime(Protocol): + name: str + + def plan(self, document: Document, action_name: str, args: dict) -> Plan: ... + + +# --- the agentic arm ------------------------------------------------------ + + +def _visible_text(form: Element) -> str: + return form.full_text.lower() + + +def _field_names(form: Element) -> set[str]: + return { + element.attrs.get("name", "") + for element in form.walk() + if element.tag in ("input", "select", "textarea") + } - {""} + + +class DiscoveryRuntime: + """Finds a control by what it means, not where it is. + + Scores every form on three independent signals so that no single surface + change is fatal: + + 1. the visible wording matches a synonym of the goal; + 2. the form collects the fields the goal needs; + 3. the form's target names the operation. + + A mutation typically disturbs one signal. Requiring agreement across three is + what lets the same semantic action survive a control moving into a modal, a + DOM rewrite, or a rewording — while still failing loudly when the control is + genuinely gone, which is what keeps a removed authorization check from + reading as a recovery. + """ + + name = "discovery-runtime" + + def plan(self, document: Document, action_name: str, args: dict) -> Plan: + started = time.perf_counter() + synonyms = SYNONYMS.get(action_name, (action_name.replace("_", " "),)) + needed = set(REQUIRED_FIELDS.get(action_name, ())) + + scored: list[tuple[int, str, Element]] = [] + forms = document.forms() + for form in forms: + text = _visible_text(form) + names = _field_names(form) + target = form.attrs.get("action", "") + + reasons: list[str] = [] + score = 0 + if any(word in text for word in synonyms): + score += 3 + reasons.append("wording matches the goal") + if needed and needed <= names: + score += 3 + reasons.append("collects the required fields") + verb = action_name.split("_")[0] + if verb in target.lower(): + score += 2 + reasons.append("target names the operation") + if score: + scored.append((score, "; ".join(reasons), form)) + + metrics = RealizationMetrics( + runtime=self.name, + candidates_considered=len(forms), + attempts=1, + ) + if not scored: + metrics.wall_time_ms = (time.perf_counter() - started) * 1000 + raise RealizationFailed( + f"no control on this surface affords {action_name!r} " + f"(considered {len(forms)} candidates)" + ) + + scored.sort(key=lambda row: row[0], reverse=True) + score, rationale, form = scored[0] + + fields = { + name: str(args[name]) + for name in _field_names(form) + if name in args + } + missing = needed - set(fields) + if missing: + metrics.wall_time_ms = (time.perf_counter() - started) * 1000 + raise RealizationFailed( + f"control for {action_name!r} does not accept {sorted(missing)}" + ) + + metrics.wall_time_ms = (time.perf_counter() - started) * 1000 + return Plan( + method=form.attrs.get("method", "post").upper(), + action=form.attrs.get("action", ""), + fields=fields, + rationale=f"score {score}: {rationale}", + element_path=form.path(), + metrics=metrics, + ) + + +# --- the control arm ------------------------------------------------------ + + +class RecordedSelectorRuntime: + """Replays test-id selectors captured against a baseline surface. + + The conventional approach at its strongest: stable `data-td` attributes are + what a well-instrumented application provides and what good practice says to + use. Where a mutation preserves them this runtime is unbeatable; where a + rewrite drops them it has nothing left. That asymmetry is precisely what + H-001 is asking about. + """ + + name = "recorded-selector-runtime" + + def __init__(self, recordings: dict[str, str]) -> None: + self._recordings = recordings # action_name -> data-td value of the form + + def plan(self, document: Document, action_name: str, args: dict) -> Plan: + started = time.perf_counter() + recorded = self._recordings.get(action_name) + metrics = RealizationMetrics(runtime=self.name, attempts=1) + + candidates = [ + form for form in document.forms() + if form.attrs.get("data-td") == recorded + ] + metrics.candidates_considered = len(document.forms()) + metrics.wall_time_ms = (time.perf_counter() - started) * 1000 + + if recorded is None or not candidates: + raise RealizationFailed( + f"recorded selector [data-td={recorded!r}] matched nothing" + ) + form = candidates[0] + fields = { + name: str(args[name]) for name in _field_names(form) if name in args + } + return Plan( + method=form.attrs.get("method", "post").upper(), + action=form.attrs.get("action", ""), + fields=fields, + rationale=f"replayed recorded selector [data-td={recorded}]", + element_path=form.path(), + metrics=metrics, + ) diff --git a/src/testdriver/browser.py b/src/testdriver/browser.py new file mode 100644 index 0000000..1cf8529 --- /dev/null +++ b/src/testdriver/browser.py @@ -0,0 +1,156 @@ +"""A browser-surface driver over the lab's HTML UI. + +Each actor gets its **own** session — its own base URL binding, its own +credentials, its own cookie-equivalent. Nothing is shared between actors at +module or class level, because a shared client is exactly how isolation breaks in +practice and it never announces itself (F-0003). + +The driver executes a `Plan` produced by an `ActorRuntime`. It does not decide +what to click and it does not decide whether the outcome was correct: the first +belongs to the runtime, the second to the observer and the oracle. +""" + +from __future__ import annotations + +import json +import urllib.error +import urllib.parse +import urllib.request +from dataclasses import dataclass +from typing import Any + +from .actions import SemanticAction, Surface +from .agentic import ActorRuntime, RealizationFailed +from .drivers import Realization, UnsupportedAction +from .html import Document +from .world import Actor + + +@dataclass(slots=True) +class Session: + """One actor's private connection to the surface. Never shared.""" + + base_url: str + token: str + + def _open(self, method: str, path: str, body: bytes | None, content_type: str): + request = urllib.request.Request( + urllib.parse.urljoin(self.base_url, path), + data=body, + method=method, + headers={ + "Authorization": f"Bearer {self.token}", + **({"Content-Type": content_type} if body else {}), + }, + ) + try: + with urllib.request.urlopen(request, timeout=10) as response: + return response.status, response.read().decode() + except urllib.error.HTTPError as error: + return error.code, error.read().decode() + + def get(self, path: str) -> tuple[int, str]: + return self._open("GET", path, None, "") + + def post_form(self, path: str, fields: dict[str, str]) -> tuple[int, str]: + encoded = urllib.parse.urlencode(fields).encode() + return self._open("POST", path, encoded, "application/x-www-form-urlencoded") + + def post_json(self, path: str, payload: dict[str, Any]) -> tuple[int, str]: + return self._open("POST", path, json.dumps(payload).encode(), "application/json") + + +class BrowserDriver: + """Realizes semantic actions by finding and using controls on an HTML page.""" + + def __init__( + self, + base_url: str, + tokens: dict[str, str], + runtime: ActorRuntime, + resource_id: str, + ) -> None: + self._base_url = base_url + self._tokens = tokens + self._runtime = runtime + self._resource_id = resource_id + self.surface = Surface( + id="browser", kind="html", description="lab browser UI" + ) + + def _session_for(self, actor: Actor) -> Session: + # Constructed per call: no actor can inherit another's connection state. + return Session(self._base_url, self._tokens[actor.id]) + + def _view_path(self) -> str: + return f"/resources/{self._resource_id}/view" + + def realize(self, actor: Actor, action: SemanticAction) -> Realization: + action.check_surface(self.surface.id) + session = self._session_for(actor) + + status, body = session.get(self._view_path()) + if status != 200: + # A surface the actor cannot even load is not an adaptation problem. + return Realization( + self.surface.id, + {"action": action.name, "stage": "navigate", "status": status}, + raised=f"NavigationFailed: {status}", + ) + + document = Document.parse(body) + args = {"resource_id": self._resource_id, **dict(action.args)} + + try: + plan = self._runtime.plan(document, action.name, args) + except RealizationFailed as failure: + return Realization( + self.surface.id, + { + "action": action.name, + "stage": "discovery", + "runtime": getattr(self._runtime, "name", "?"), + "page_bytes": len(body), + }, + raised=f"RealizationFailed: {failure}", + ) + + mechanics: dict[str, Any] = { + "action": action.describe(), + "stage": "submit", + "runtime": plan.metrics.runtime, + "rationale": plan.rationale, + "element_path": plan.element_path, + "target": plan.action, + "fields": sorted(plan.fields), + "metrics": plan.metrics.as_dict(), + "actor": actor.id, + } + + status, response = session.post_form(plan.action, plan.fields) + mechanics["status"] = status + if status >= 400: + return Realization( + self.surface.id, mechanics, raised=f"Refused: {status} {response[:120]}" + ) + return Realization(self.surface.id, mechanics) + + +def record_baseline_selectors(html: str) -> dict[str, str]: + """Capture the control arm's selectors from a baseline page. + + Deliberately generous: it takes the strongest identifier the page offers. + A weak control arm would make H-001 trivially true and worthless. + """ + document = Document.parse(html) + recordings: dict[str, str] = {} + for form in document.forms(): + test_id = form.attrs.get("data-td", "") + target = form.attrs.get("action", "") + if not test_id: + continue + if target.endswith("/grant"): + recordings["grant_access"] = test_id + elif target.endswith("/revoke"): + recordings["revoke_access"] = test_id + return recordings diff --git a/src/testdriver/html.py b/src/testdriver/html.py new file mode 100644 index 0000000..af39544 --- /dev/null +++ b/src/testdriver/html.py @@ -0,0 +1,108 @@ +"""A minimal DOM for the browser surface — stdlib only. + +Not a browser. It parses server-rendered HTML into a queryable tree so that a +driver can *look for* a control rather than being told where one is. That +distinction is the whole point: a driver handed a selector has not discovered +anything, and cannot demonstrate adaptation. + +Chosen over Playwright deliberately (see F-0004): the lab's surface is +server-rendered forms with no JavaScript, so a browser engine would add a +dependency, a licence question and several hundred megabytes of binaries without +changing what H-001 and H-002 can be measured against. `Driver` is a Protocol, so +a Playwright implementation can sit alongside this one when a surface needs it. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from html.parser import HTMLParser +from typing import Iterator + +VOID_ELEMENTS = frozenset( + {"area", "base", "br", "col", "embed", "hr", "img", "input", + "link", "meta", "param", "source", "track", "wbr"} +) + + +@dataclass(slots=True) +class Element: + tag: str + attrs: dict[str, str] = field(default_factory=dict) + children: list["Element"] = field(default_factory=list) + text: str = "" + parent: "Element | None" = field(default=None, repr=False) + + def walk(self) -> Iterator["Element"]: + yield self + for child in self.children: + yield from child.walk() + + @property + def full_text(self) -> str: + return " ".join( + part for part in ( + [self.text] + [c.full_text for c in self.children] + ) if part + ).strip() + + def ancestors(self) -> Iterator["Element"]: + node = self.parent + while node is not None: + yield node + node = node.parent + + def path(self) -> str: + """A human-readable location, for evidence. Never used to find anything.""" + parts = [self.tag] + [a.tag for a in self.ancestors()] + return "/".join(reversed(parts)) + + +class _Builder(HTMLParser): + def __init__(self) -> None: + super().__init__(convert_charrefs=True) + self.root = Element("#document") + self._stack = [self.root] + + def handle_starttag(self, tag: str, attrs) -> None: + element = Element(tag, {k: (v or "") for k, v in attrs}, parent=self._stack[-1]) + self._stack[-1].children.append(element) + if tag not in VOID_ELEMENTS: + self._stack.append(element) + + def handle_startendtag(self, tag: str, attrs) -> None: + element = Element(tag, {k: (v or "") for k, v in attrs}, parent=self._stack[-1]) + self._stack[-1].children.append(element) + + def handle_endtag(self, tag: str) -> None: + for index in range(len(self._stack) - 1, 0, -1): + if self._stack[index].tag == tag: + del self._stack[index:] + return + + def handle_data(self, data: str) -> None: + stripped = data.strip() + if stripped: + self._stack[-1].text = (self._stack[-1].text + " " + stripped).strip() + + +@dataclass(slots=True) +class Document: + root: Element + source: str + + @classmethod + def parse(cls, html: str) -> "Document": + builder = _Builder() + builder.feed(html) + builder.close() + return cls(builder.root, html) + + def elements(self, *tags: str) -> list[Element]: + wanted = set(tags) + return [ + element for element in self.root.walk() + if element.tag != "#document" and (not wanted or element.tag in wanted) + ] + + def forms(self) -> list[Element]: + return self.elements("form") diff --git a/tests/__pycache__/test_agentic_realization.cpython-312-pytest-7.4.4.pyc b/tests/__pycache__/test_agentic_realization.cpython-312-pytest-7.4.4.pyc new file mode 100644 index 0000000000000000000000000000000000000000..d2ac8c23123c916bfe9acdc3c9da267789ae2fac GIT binary patch literal 21982 zcmdUXX>c6Zb#C`e&yK-H5(F0sLy@96B(ag=B8r3rZW1L5R!Av|5ZmL%bORW07Wj4# zLBKOrkgY093S-exV$o%}9`#a*rAUsI?LEDERY~l675{ko0R*zZ>QX{yE60Blnv_k8 zKT`S5y}bc!0iZ&;lE&clcJJ-mw{M^2JLmNOPp6X-mezGIj(u~TB>f}pI7>uq$k#1d zl3tNCX;9K+EmV|Cp}~+$+u>q(P~q!HDVoQzQfx48pGgcRa7HO6OR2$BsbR38lpai% z8V4Ks`AD&;)I8W+Y8h-PwGOuOW6|QO((1w0d^=WLQ)(M*K) z)&jakYX!YmTLrpRTMhatZ4Kx)tqnA*uRAKeB;`XR`2Xx`NvcaVY3xa7c)Yc?_RKa( zGUW5CYSPz2=i3yNuS1wesrJ99GIa3+VRLXkB z&`T;mr{?MK&Wt*u7RrX1QF-aiBTqj0M8>GHVKuLJC6iB%>L?*v8&{c`DHxf2r950H z>srRFWVT~VE0jkvCJN$&$tuO|U76FmSsmYr7SC4l;__-)OX_EJHkqj~HDArQ9(zB)k3k#^vtLZf(v>_a7A%XdgdRde>+*G78jl#t7;?Y_)loK zv$B;^%jJs6yI>4t6)SqGS}yC%YCLU^Lq8t|D|Xsss#z^rv0v4hR>+&#u+@CDVB{;* zXH+r^CEaQ{wWsuag=xBWS}*E(v%>7Mxb~{kVi*T1<`Z;}rm1T<96wsg zS4$YuY{-fnekOO~sFgU<|78EkPe1(BFuD@R*Z@M`khtYROt^tl2y7Yz>FaR4kidEm$#XkkJTOVT`L4*UuI-y`0yrWMw#Kh-t#2bUt?a zsZ+=Mj~ufUO*PeQ16xH$6Q!y>;jFMaK5j+L${D?ESaEu;xhgAKk%bXtfRh#(9uw>HWm23(w3XI<895$d=j72QD3Xz3uJx_j~&; zH#A*nyV!nd-}P{4S8_Jq^?c;{vlq@?TJw7JgQ(Qnelc?K?4@&Wx6QTepKm!h+j8)o zr{-FYJRiN>+Wt!S*ShChch0u%eCz4&=HAH7wH|s$oo#*S`Pk*AH5VpcseP?B)3o_U z8dtnQm%T>hCcobo`nwN@!ru=?4yR%ZCOAz;Og#DeQEa>-`6sw0*QA2{Rrxb`t~AMO z;W2lTqp-bbA$LMaRZNg+`D+C9;?^4IdBS&%KWlX!nf4_wh3)Xd+YmjB*w(4>@Y9bmX+2?|B1V7wtu zwcfjJe6lxdkP1ed_hpSYLRJ{Zh}?Ixk)#z=LOQoMYtX^HsCI((cV&&_7wwJAAzq{> zYji=}k#d!+J8hz#D2ENZ94^!KME~j2$vWGN?l5A3Zz$Ay50PDLHyzyrvY-e1MGy8v z{_6QrwOq&>oMqo^(Hu3d)7F#TkZ*<#9rEAdA<;!#8d7{KnzoV;jZcz1&8f_4$>r5z z5%P1K;US~o9x1vR;%0dQoow&hZF@6zD9_BZh-Y`MAxeXJ%F3V(&2C30 z{u`#NR0jz}Ey7ts>)0=W)nA^iPyCJ)n^mtgzpC?11 z)6_oCYtd_-8G?b0>j)zf=DDyUGZVVX$gWm-gIZyjxJ}v2jH)JV_Oh0Fq_d}|H#6b1 zfgZw2pc?4nDArxPBDZV?V0Ff9({Us=`1D5qTWM&&uxe@9Z_`9im5oMR2PfoLdoZvL8pTVB|P( zkb{(f;hs~$wZy11dTG1yZ{$FxrRxn+WYs?__xvcmdOm&kZ2Int6Z0MWW;^!HrT0%C zzoOhT|8ajmRPI@VOT=NfEUD~EV`Q-|T7uMjfL<_$M+x-WPPR}j6r?7a^j!Ed7NC4r zW!8Xt0SfkSh0gC(pgw~S$)m)903UF*DN>7o7d5%91{mouJR;oiKpl@4Y08S#)HS~L z1JqQHZ-K{nJqh9pE{6k;QI;vIIJ{Sj*t{sJMPU{=YH7^e;M}#ZrH#%pZ(Va$bEie9 zrE&102zXK4hV(*!^b%8#xF}qvq)uiCa^w7EsA`&(a#!Yr$&b^!tLb_0rp&~sjv3ET zN)_CR6-GwQuBk=?(1hOU01m7V{mI&z9`v(}tq0MopUhW^;;k@X2=7NhMqnrV43W=* z3}lEpUtp?GiK-z#<_;3x3kzEPP?bzTii ziH4WbzmdL51*RXpDq(daE=6|ED?2{+@55^eG-D9-@%!CJ9#n{5xr>VxByrx)J~Y{x zVm`DWWLrB6(Sk4etTM%I+1&t6Kxl#T3EMi(s}=F}#${1%XC%!<1+>@{4r_6rO5my$ zzBU3_SfEO<5LpONOMoo6EG>$1p#`xKlSVBx7{dhR0AT7!!$q_swug(o~E z($`YZ+*uA;kj4lvT(m$WCvB0u10tEw@qiiTjYZJ>s*;B)gPH+?1wy$KLOE1VYE4@6 zD8Zk=_EwB$wD32)^_TUAYpXtqw{{W=5BG^BAwX9TSmDQe_i=E8b>V5S9wNOUQ=5FC zICOF-g2NeMf5Qou&+EE2)tu~G2y}4l0aEi$03FBsPn`4v9c&+U?EsMnsDF+5&b$hr zS|<#S0y?v?(5aVU(C2mXAaw#}$dBqqr?4NmIA;%0<@-P!`Mgly2AW~x6XLE)#)zkL zv*T2m1c{|k*A^+~R)XAXbY;V85Y+*D}LhHX$X@d!964qz)JyIL_Kc0ayUp< z)f2UNEddV+q!c_P-n|5AV<>~sPf-nT^tFaNHmZI-#_LJYsJbOFstvwg2bJ}-f%J-d zy3W^T!1MRU)OiljW0vgktu@f|^wM?Ue+R3hMQgP6m=sC;TB;ztYYi)&Hw{ip&f1#)&MH*p9wiJ6W|_D z%iKpWT;SrT1NGd0u^~ z{2S$&^uFoiAXk)a^B?#3uCncyCf(mtvL9ZHNR9is;r^{v5AXd5vHDjKhK_fjB-bS# zA{Q=(N@4O4gy1946cQ6!?A4Ix{&WPB7I)nDT(FXEwP-;L5C~e*35$8tW*}rD1cl># zWO)DpA6Xtw+FRO>5HeVm>(N@&cz=FB;S(cGx8Zy6#CVX9bVV+NvsW@e zIC^3n%ovRIj`hG*IhO5#TM|8J)>>@!Am;10E55x)t+hJORa?bjk=3=BEfV5Bbke)` z03*_KzdLO~B&@;n3y#?Q{SHvDv^K6ltfe~tm7ct> z*Fm$y1EJdQ*bIp{qrU76aZ6%`v@g?BPhnW+Dh%tj4L*gz!>*T}9YG4i8Yl}5^fm~M*$)lACtSUtj6FLOr`wXOMIWvx!P{e=gnQh16*6P`v?Ghn* zvXwpzQH?NzqdZm)j;shGHY;qZb<{M+4|I3;?%LPYgTLMby;BiH+I6a8HH7F&S+(RL zOFqTuEwJQ%_C>xvoQ3NPoq<9EE1tW)2>TXCxa$ki3sz*f0A&CkFu3N*8a!b8QJLKj zGPMnwiUGsd5Kcl;Y~aPqtI&S5jGAW^0}2Ep28>160m2J-@B;jCH$$5a;o^@h5i7yN z1`#Q-NyH|6M5KgW%8GEMYOw~x@QCp{ zB&firbv!~PJXhTj+AR?|N`#zt7Ci&K43&c}BAV==t!X095_ygY34FifZnuC5I5@B} z)nf}+J#-8@#Trx)DgiHJ4iHWbnwiso_9CwHdHfq&A?l{3e^z?_R@wUZ>Gd-ky5FAo z%cp01AAf(tj=2q;uN&9Ha^ro|$A8q&I^WPS+tBe^~eb5d^$6m z&b+qflJWYH-|d)7_f8+b99Z^>^631X{>>?mUQzCy|Kxvj%DuPxC23B;;FqsohS$;W zSa38CpId>4=4y^}A$(N&>ea29L@#*wr z5gZ<5@cHFG?n*2{-pvK_KdOc9SP$Ho25|SKd*GG?cMp-)X9rQn;V$rkC|&QSueE09 z&b)P?t0P%g&i{=f)k0c|JA`6dgK%D)!8;A}Ke228X&Ta+^tAQE~6B1%# zXQogZ2SYCmA!h-nhSdC-VE~dD$S|FCv4a?2D^}7?hJdqdTm%p|Oi;WSr3~ashP5US zyR?9?3&coKCq*`=ym$)YzBMn<;->o-1V~syF36G}Ge5eDv{Fs3bw+96A;dcS6{o z{dDv&ks$t$X=2A~q}X=a-a_O&k*^aWPS0K-GC<@pBAmlNPFrMQ2BW|SIe*VrCz<%X;hJ0600J1VpVkN+x?kR0m2hdE2bPJ2Rde!fxIOir$|DTLCRs` zVO7>_O!!FxeeomQcBHB@ig*IpYnYXBN(e;uAP`zuTUB$k!kipO=Nc4sOo@U#Gf_2? zG)q+B&@~Lej~P`{$3>Esz;LI!5009pV&6eRdHW8I_Vx{kC`t}UAMD0~gWcj>H!o;4 zj4J@FM9wRs_-FhHgbgFgY^1D6`05#NdRfrBIAG z%vxm}2&w@iMY7`-MU=H0yKXx`idu@^a4lgGQm-Et<{un#{IjqSA$XV0Dkd`8l&6?2l$!lzEqe6&;XupWYA z+N_EsT7m$G$G~YY*~PwvzF0Ba+QzeO9e2_{b5157r3MxTzy}ekh1fVu*eT=)LJfQk z`|N)bp}Vruz!dUloN8z*N!Q-#8oJc=4%s3|^tjESH6cX>8@vToILuG&~!y2eu@(sn!_`#xD8q(B0_PC9WPvyeHf_Xq5VM-x(*MA^}Ej3K5?K$iwX5nERHmQYm(+5UjhS>Zm9uV6kdoG z_h^l;{eT`VNPJteCtgYPAmVGa(doap_QrpiaxNMGguzuiml@;b?#ShBuNPMxi4R9tp^TK_AZ3#j3bkt&cRo7sBVzbP3KllJRIP1f9U=}` zoWh2E3W{7;K}*)NUJlnjSdKf;4fb6Sqi^+i`**_2i4niYtq0T zu1R{r*XHd8a=P;RxJmLCF~rFKK%N%mW*IqBN7(n#17bP`slL*E!ojU!D*$Supp6_01k{0B5>gIp6(lj z`h8?xf{U^Rr4l>6=NvfY^lvQSEj8U>TIWu|fxQ|59?M&6*c^DJJ@V&R1@y?{tKe5l z_QWg6Iq=eDUF;H^^-lHF<(JF$)GbMzb(y>JWI*JO_0$bPTDGTd$rX&LtJOkQ#O$PK z!h*f*la*$|xd^N?3xS8avH}naWucI~U^HPIdGTcu3IUENfRW3|3w#Vle5B+MwLutw zCqQ~pU&tBSvIPzDdAr7ue@7oVL2fCf;zDObtn3cz5Q!R&wZhQaPushQa4EB!w)PNt z00f5BWN*(@laK-&>Fg!<9v$8VHx+RvM$^+1KG4^1x$ZJfu%pz#F%TfYZr4O`&9C67 zUn<@2-`Tf75cMwD+Z~YAa=$=0>})|dZ_OYB;VcpmA)jC$a03xRP41}WvjQI*2d9yMdwTxA!*518y##Rb3ka95Bv&S0UQC(_27)iXFGE4q z;$MbxnY2@eWZwB^{!OS=h)qRs&hr%QC_jhq2OM?3?gZiFDv_QFN+y8uI|QZp7*T*K z{HA>|E<~1OnO?%4^ZG)Fcr^l;{H-;>Zo*5~@$O)mavlR!UJY}$qx zxuGV@wCM%Jr0g)0xYmHMu6RMjr6yK9H{5^~XJqZULGvS4k4{E>Up{M`y6`mrU>LDT zBxrM}hIrymj@|$$Lw+_vCo15-@o`o`=&PN;L)K41EDDu6ZVW@ zCtP!AaY!d@Xc#U)K;=LohbA*BWo(~uP)>m=E>MBV5QJ!wub;%m@5qkviiFGiY!a#?^2{m`8#3yeifI|16Xs>NJbPsBNpoFVSk2AE+q|c zMk$k-<~aGy#6jdc!c1d*QwO*}A?(OO5>f88tKcMw#ZXEeS#NKUx;~ZEV48qlA#a)6)P6AVK3B1 zgvjen4cFh-Z!V)9#~0ju3G*0QN5JlIe=182$i$NC&tZlpeisgOaT(Qepm_wmwbd z6cHLwgE$jrqP;vMyc?fd;iF>jq4u|_nSTJmlldSiHEs0?`}a3)eXse!3txDB)BN_n zx$S+w)iTrk;PeTQE52vwKFr(Nm{E>hQSP4q zkPxuKn7eq=w9V$}0KTSXR$19nhANd}w%3Z~pcdtGIV+jV4OekW&*d1sC-_pE{ZSVF zxUF?Lf!=qCsSFm<#dIDCZYA79-CVfu&w|B@U>dHFZvL~fNoC=0MZeUDLm z8iD%-X!0$H+p-~Ffe1IzG_V7@$Q2>{3~782u*<Q5?L|!Cvk;pfR{2r0-fLIY+TrV|fA;jDO?bL9T{nS^jA#A}c>@l;q7nmZayT zFQ#D&{!8k&M_wCwd*j^JLocd-lkS^N;u@j&S2jGmVP0;ZmD?|_zclzqpPP}}=j5Y5 zMbCP!ho#Ul83|o-&(-9TMW2+tve-_><&F=pbx4g58l;kZuW$8{ZOZpoDWLxWbvDEQ literal 0 HcmV?d00001 diff --git a/tests/test_agentic_realization.py b/tests/test_agentic_realization.py new file mode 100644 index 0000000..552b318 --- /dev/null +++ b/tests/test_agentic_realization.py @@ -0,0 +1,175 @@ +"""Agentic realization of one semantic action, against a real HTTP surface. + +The agentic part is confined to *finding the control*. Setup, revocation and +every oracle remain deterministic, so a failure here is a failure to realize — +never a failure to judge. +""" + +from __future__ import annotations + +import pytest + +from testdriver import Runner, SemanticAction, Stratum, Verdict +from testdriver.agentic import ( + DiscoveryRuntime, RealizationFailed, RecordedSelectorRuntime, +) +from testdriver.actions import SurfaceNotPermitted +from testdriver.html import Document +from lab.mutations import BY_ID, CATALOGUE +from scenarios.browser_grant import baseline_recordings, build_agentic, lab_server + + +def realize(*mutations: str, runtime=None): + with lab_server(*mutations) as (app, tokens, base_url): + world, driver, observer, asset, oracle = build_agentic( + app, tokens, base_url, runtime + ) + result = Runner(world, driver, observer, oracle).run(asset) + surface = result.evidence.of_stratum(Stratum.SURFACE)[0] + return result, surface.data + + +def realized_ok(*mutations: str, runtime=None) -> bool: + _, surface = realize(*mutations, runtime=runtime) + return surface.get("raised") is None + + +# --- the realization itself ---------------------------------------------- + + +def test_an_agent_realizes_the_semantic_action_from_intent(): + result, surface = realize() + assert surface["raised"] is None + assert result.verdict is Verdict.PASS + assert surface["mechanics"]["runtime"] == "discovery-runtime" + + +def test_the_runtime_is_never_handed_a_selector(): + """Discovery must rest on meaning, not on identifiers it was given. + + If the runtime consulted `data-td` it would be a recorded selector wearing a + different hat, and H-001 would be measuring nothing. + """ + import inspect + + from testdriver import agentic + + source = inspect.getsource(agentic.DiscoveryRuntime) + assert "data-td" not in source + + +def test_deterministic_oracles_are_unchanged_by_agentic_realization(): + """The agent may find the button. It may not decide whether that was right.""" + result, _ = realize() + collectors = { + obs.collector for obs in result.evidence.observations + if obs.stratum is not Stratum.SURFACE + } + assert collectors == {"state-observer"} + + +def test_a_defect_still_fails_under_agentic_realization(): + result, surface = realize("M17") + assert surface["raised"] is None, "realization itself should succeed" + assert result.verdict is Verdict.FAIL + assert {j.assertion_id for j in result.judgments if j.verdict is Verdict.FAIL} == { + "c-carol-denied", "i-enforcement-matches-record", + } + + +def test_the_agentic_asset_judges_fewer_claims_than_the_full_reference(): + """A one-step scenario reaches fewer claims, and must not imply otherwise. + + M15 passes here purely because this asset never revokes. That is correct and + it is also exactly the kind of narrowing that silently overstates coverage if + nobody writes it down. + """ + result, _ = realize("M15") + judged = {j.assertion_id for j in result.judgments} + assert "c-bob-revoked" not in judged + assert result.verdict is Verdict.PASS + + +# --- isolation and cost --------------------------------------------------- + + +def test_each_actor_gets_its_own_session(): + """A shared client is how isolation breaks in practice (F-0003).""" + from testdriver.browser import BrowserDriver + from testdriver.world import Actor + + driver = BrowserDriver("http://127.0.0.1:1", {"a": "tok-a", "b": "tok-b"}, + DiscoveryRuntime(), "R") + first = driver._session_for(Actor("a", "A")) + second = driver._session_for(Actor("a", "A")) + other = driver._session_for(Actor("b", "B")) + assert first is not second, "sessions must not be cached across calls" + assert first.token != other.token + + +def test_cost_and_nondeterminism_are_recorded_from_the_first_run(): + """Free to collect now, impossible to backfill later.""" + _, surface = realize() + metrics = surface["mechanics"]["metrics"] + for field in ("runtime", "wall_time_ms", "candidates_considered", + "attempts", "retries", "tokens_in", "tokens_out", "model"): + assert field in metrics + assert metrics["candidates_considered"] > 0 + + +def test_runtime_identity_is_recorded_in_evidence(): + """Which agent, which configuration — auditable after the fact.""" + _, surface = realize() + assert surface["mechanics"]["metrics"]["runtime"] == "discovery-runtime" + assert "rationale" in surface["mechanics"] + + +# --- failing loudly ------------------------------------------------------- + + +def test_discovery_fails_loudly_when_the_control_is_absent(): + """A missing control is not something to route around. + + This is the guard that stops a removed authorization control from reading as + a successful adaptation. + """ + page = Document.parse("

Nothing here

") + with pytest.raises(RealizationFailed): + DiscoveryRuntime().plan(page, "grant_access", {"subject_id": "bob"}) + + +def test_the_browser_action_may_not_be_performed_through_the_api(): + """D-05: routing around the UI is surface substitution, not recovery.""" + action = SemanticAction( + "grant_access", {}, permitted_surfaces=frozenset({"browser"}) + ) + with pytest.raises(SurfaceNotPermitted): + action.check_surface("api") + + +# --- H-001: the two arms -------------------------------------------------- + +MECHANICAL = [m for m in CATALOGUE if m.label == "MECHANICAL"] + + +@pytest.mark.parametrize("mutation", MECHANICAL, ids=lambda m: m.id) +def test_discovery_recovers_except_where_field_names_move(mutation): + expected = mutation.id != "M22" + assert realized_ok(mutation.id) is expected + + +@pytest.mark.parametrize("mutation", MECHANICAL, ids=lambda m: m.id) +def test_recorded_selectors_survive_exactly_while_test_ids_do(mutation): + runtime = RecordedSelectorRuntime(baseline_recordings()) + assert realized_ok(mutation.id, runtime=runtime) is mutation.preserves_test_ids + + +def test_the_control_arm_is_not_a_straw_man(): + """Where test ids survive, the conventional approach loses nothing. + + Asserting this protects H-001 from being flattered by a weak control. + """ + runtime = RecordedSelectorRuntime(baseline_recordings()) + preserved = [m for m in MECHANICAL if m.preserves_test_ids] + assert all(realized_ok(m.id, runtime=runtime) for m in preserved) + assert len(preserved) >= 9 diff --git a/workplans/TD-WP-0002-vertical-spike-crystallization.md b/workplans/TD-WP-0002-vertical-spike-crystallization.md index 7c9c3ca..69569d2 100644 --- a/workplans/TD-WP-0002-vertical-spike-crystallization.md +++ b/workplans/TD-WP-0002-vertical-spike-crystallization.md @@ -307,7 +307,7 @@ fail*. All four `td://self/...` identifiers are covered. 72 tests pass overall. ```task id: TD-WP-0002-T07 -status: todo +status: done priority: high state_hub_task_id: "b58cf7ce-dc18-5218-8eac-179291626467" ``` @@ -324,6 +324,36 @@ run than simply asking an agent to rewrite the broken test, the crystallization argument is an aesthetic preference rather than a value proposition. This data is free to collect from run one and impossible to backfill. +**Done 2026-08-22.** `html.py` (stdlib DOM), `agentic.py` (two runtimes), +`browser.py` (per-actor sessions over real HTTP), `scenarios/browser_grant.py`. +107 tests pass. Two decisions taken with the operator: **stdlib HTML driver +instead of Playwright** (F-0004) and **a deterministic discovery runtime instead +of a live model**, both behind interfaces that let the alternatives drop in later. + +The headline result is a narrowing, not a confirmation: + +| | Discovery (agentic) | Recorded selectors (control) | +|---|---|---| +| test ids preserved (9 mutations) | 9/9 | **9/9** | +| test ids dropped (3 mutations) | 2/3 | 0/3 | + +- **F-0005 — semantic actions earn their keep more narrowly than claimed.** Where + an application keeps stable identifiers, the conventional approach matches the + agentic one exactly and is cheaper, faster and deterministic. The semantic + action wins only where identifiers are absent or not carried forward. Filed as + `CONCEPT_DRIFT`: the concept model overstates this and should be revised to + match the evidence. Three mutations on the deciding side is directionally clear + and statistically nothing — recorded rather than rounded up. +- **M22 marks where a scripted runtime stops.** Renaming form fields defeats the + heuristic, but the page still carries a "Person" label a model could read. This + is the first concrete evidence that a live model would add *capability* rather + than only cost — worth more than the general argument that it might. +- Two mutations (M21, M22) were added mid-task because the deciding side of the + test-id axis was N=1 after the first run. Extending the instrument when the + evidence shows it is too thin is the intended behaviour. +- Recovery happened with **no claim or invariant diff in any run**, and the one + failure failed *loudly* — `RealizationFailed` in evidence, not a silent pass. + ## Adaptation detection and the defect-vs-adaptation classifier ```task