From e25bac3af38c94cd1a56194e36885e63843166bb Mon Sep 17 00:00:00 2001 From: tegwick Date: Fri, 31 Jul 2026 17:27:23 +0200 Subject: [PATCH] =?UTF-8?q?CB-WP-0005=20T02:=20M-D1-MUT=20=E2=80=94=204=20?= =?UTF-8?q?of=2014=20acceptance=20rows=20are=20enforced?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit For each acceptance row in GameKernel §5, invert the property and require the verifying command to go red. adapted:mutation-testing, with the denominator changed from source lines to acceptance rows. M-D1-MUT: 4/14 rows enforced PARTIAL 2 (AM-7, AM-8 — some clauses live, some inert) unmutatable 8 (no property to invert, reason stated per row) SURVIVED 0 Two corrections to our own numbers. There are 14 rows, not the twelve ADR-0005 and CB-WP-0005 both asserted — AM-4 splits into a/b/c. And the prediction of 9-of-12 (75%) becomes >=10 of 14; measured 4 (29%), badly unmet. No target moved in this commit. The second correction matters more. My first run reported two SURVIVED rows and both were my own no-op mutations: `pub struct NullRng;` -> `pub struct NullRng {}` is semantically identical, and renaming max_age_days does nothing because CA-17 reads it with a default of 90. Both would have been published as "this row asserts nothing" — a false accusation against code that is fine. Replaced with real inversions (a per-construction counter in the ChaCha seed; reverting AC-9's output resolution to the first-wins bug it was fixed for), after which both go red. T08 asks whether writing a weak mutation is the new grep. It is, demonstrably, on the first attempt. The finding is larger than the workplan assumed. 8 of 14 rows are unmutatable — AM-2, AM-3, AM-4c, AM-5, AM-6, AM-9, AM-10, AM-11 have no instrument at all. AM-6 is the sharpest: nothing in the workspace compares any number to 100,000 events/s, the headline throughput claim. The problem is not three unimplemented rules, it is that more than half the acceptance table has nothing behind it. Harness controls: a stale find-string reports HARNESS-BROKEN rather than scoring the baseline as the mutant; a red baseline reports inconclusive rather than red; the tree is restored in a finally and the restoration is verified. Not in `make all` — it rebuilds the workspace once per row. Co-Authored-By: Claude Opus 5 --- Makefile | 10 +- facts.toml | 12 + tools/__pycache__/cb-cost.cpython-312.pyc | Bin 37064 -> 37064 bytes tools/__pycache__/dep-weight.cpython-312.pyc | Bin 9295 -> 9295 bytes .../mutation-check.cpython-312.pyc | Bin 0 -> 19077 bytes tools/facts.py | 5 + tools/mutation-check.py | 383 ++++++++++++++++++ workplans/CB-WP-0005-assertion-coverage.md | 46 +++ 8 files changed, 455 insertions(+), 1 deletion(-) create mode 100644 tools/__pycache__/mutation-check.cpython-312.pyc create mode 100644 tools/mutation-check.py diff --git a/Makefile b/Makefile index d786a54..17f3117 100644 --- a/Makefile +++ b/Makefile @@ -24,7 +24,7 @@ TOOLS := $(REPO)/tools # Every cargo recipe runs at the repo root; the shell does not persist cd. IN_REPO := cd $(REPO) && -.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget cost-mix loop-lint self-tests env-test task-done status facts-check facts-gen loc all +.PHONY: check test sim bench bench-test coverage dep-weight cost cost-test cost-pin cost-budget cost-mix loop-lint self-tests env-test task-done status facts-check facts-gen mutation-check loc all ## fmt + clippy (deny warnings) + HashMap deny-lint check: @@ -64,6 +64,7 @@ self-tests: $(PY) $(TOOLS)/task-done.py --self-test $(PY) $(TOOLS)/status.py --self-test $(PY) $(TOOLS)/facts.py --self-test + $(PY) $(TOOLS)/mutation-check.py --self-test # T01 positive control: prove the environment fix, do not assume it. Runs # every tool from a foreign working directory with a PATH that has no @@ -82,6 +83,13 @@ env-test: @$(MAKE) -C $(REPO) coverage >/dev/null \ && echo " [ok ] make -C works from any directory" +# M-D1-MUT (CB-WP-0005 T02): invert each acceptance row's property and +# require the verifying command to go red. Deliberately NOT in `make all`: +# it rebuilds the workspace once per mutated row. Run it on demand and in +# CI, not in the inner loop. +mutation-check: + $(PY) $(TOOLS)/mutation-check.py $(ARGS) + # T04: single source of fact (InnerLoop v1.2) — the DFD gate. # facts.toml is GENERATED; facts-check fails if it disagrees with the # instruments, or if a tagged artifact disagrees with it. diff --git a/facts.toml b/facts.toml index 5358384..4c0e46a 100644 --- a/facts.toml +++ b/facts.toml @@ -33,6 +33,18 @@ text = "350,000" fmt = "{:,}" by = "tools/dep-weight.py TARGETS" +[am_rows] +value = 14 +text = "14" +fmt = "{:,}" +by = "tools/mutation-check.py" + +[am_unmutatable] +value = 8 +text = "8" +fmt = "{:,}" +by = "tools/mutation-check.py" + [gr_covered] value = 58 text = "58" diff --git a/tools/__pycache__/cb-cost.cpython-312.pyc b/tools/__pycache__/cb-cost.cpython-312.pyc index a5d92f2722efbd3ea57ad0ba0ce09974307a2cb7..5a58df85b547af68311b6977c542eaca7a4907b8 100644 GIT binary patch delta 21 bcmX@HkmG*S-k=Pm~85 diff --git a/tools/__pycache__/dep-weight.cpython-312.pyc b/tools/__pycache__/dep-weight.cpython-312.pyc index 912597bf8720249c0e4b853e9d854580774f5732..d9edb508b1c358192af2bddfc3c9e6709abee393 100644 GIT binary patch delta 19 ZcmX@_ao&ULG%qg~0}%M^+sNgr0suWQ1y%q6 delta 19 ZcmX@_ao&ULG%qg~0}vdD*vRFo0suYg1!MpK diff --git a/tools/__pycache__/mutation-check.cpython-312.pyc b/tools/__pycache__/mutation-check.cpython-312.pyc new file mode 100644 index 0000000000000000000000000000000000000000..e483abd2e4ebba2f4847dc1ed66318c5fb6f1f97 GIT binary patch literal 19077 zcmbt+dr%x#x?lH9&oD1$cnJX#QVSu$AZ7pw2@5?Dptq0&NM390MZL) z*x1*%3@dF3?`|S=ZI`iY>sl%)Wn4E^SGSVf>)pFI*-BEWLC9-7t3PxTR}%e`!jiVK zR;7~szSG^qfUx7c=@LENeNLbAo$vj9=WG3k>}&^zW5?eP_W#=^j{7fkV;q8h@#JrD zah;R63!KDDra}G!&wfo8OzhWu!OVVz3j%)4gO(xd1?y1Og{&dl1sl&vLN9;8ZsLw` zlI1NXgQIj_}=C*O%DPK#S@6@@@Mp25$su&25M*BkH zUa=<-8WczR0}e;?A>WH@T;~^CFUiWN7>V|wB@|YpN^D3DM{y})PBv)29E>`GkytpY zid9wNz>us~Rf+$3{1+mFv-$LSQE$B%j9@gi@uCVvZ%_JfHz6*!D@gXEJqT%M@1m0y|shzQ7-}tS)}zz+Vb%d1_;WS zik<+sez9{TqV*e!MFRw(4`|28jbhU&UjwLcu`i(Z`9c!G9}Em$6oZ3-m?}Hem~ts} z3DoR~3`#`IfLK>sJK!U*qT*l#)KwxwMA%qZk=YW&UO6nQSd)sWd8rBFcdv5WkFr zXw4VZVL7PQ903ka$Vylq^bbiF#b1BCvl^`+ZIn>yNHZGYMUh{Pg@8B0F?xi0Mrm=u z$j}hAMk8Wx1YD!YQdJe^7jXonz%U5Zn8sjKR-^P1u@nsh+(m{$;Xo9CV)e_x1Y3n8 zQ8ti99dQt(0WY;)?gOHuz%gcx460%!R^A)&(KKM)YOy1#j1I_3wRq-Or&m;i5kMVa z0}X@*2So)q7Kwkg0rvou0OKfiFfu~82TuloX^>Ej49S{AjsV(78X%B5G~^dggX`EO zVAnAN=BOeu1sG}sBS%!st+x$FhGU>NcnJvFyjeUigJ57%4O_q*Ax%5F1FB5R6?+v~ z4p)oX+CT#-jXX?3NRz`Lb5NGp`Z~_FpFeiKwS^7xEn_sD>^Q9ryKJa{s)Fn)8^(Zw z{elRjlL$?i4*xXDk$8#SGLf?n8bb#l7G}f_bVIU%ttC|j{?Hh!S{#9pLns)p6I%t; zNSJ|ARi&wy=V1u+21sQ=8`0#Ti4qEmm6`g@>ld2{@0heZB8`f{K=}D#bk^3FO4Ov4 z`B}$MI2Z{B2V*MnRT_1HLBNOMfOtmGvL!;4i6fC1WRj@`V=PA3GZ8h!*b^g06>zA- z(Znc7q!;@FkT;dk2(D8@gU||tqfF!DSwsvB592lj)x|JLZz%F`1f&t6L6lJkEi#~L z6!aN@)-XQNMS$_}U?eg?lSG&T@rxZ1O&2J!utP_HW+gCQBSUKsV1Yb?me5myU^EsO z1SG+}K)6?yi2NCN(Z${|OhZYK`)(O*D1$&6)Wk1LE#M_K?g&7JL_yNgJ}j^veF> z(TnIO_K|G}^hO|a=Ttmz1i6P{XCvVqn*OLU5bX>0$-x1v z;}X`(ylUdPUJg$!MxRmc7T@Nzrc5CT2q5j7vT8;sWn}=%DQAZ6No8rqSoLP51VXUl z#7K&ZPT6#F)5$}p9Gp~Il}_|${+d2${hE=8q6YhFFwUs2JEHW~FkACHucT~UU7>I& z+SL_z0kwYPZV!#S1E+EBVP3(6a5ZaTO$mM+3x(_P<1~KOZJiMA+lm&gT+zm#jm?xz z2tRc_!E2mXXu|=DVFL11y1G)1uCAeo1g{C#uCA_EA)$IpR#%r433hcU`FQgdrxf5v z$)gjzOzrdnE>v3WTilmI)5D`D7m7=tn(dY?T4(CFjHze{M%~At;&m>HTkYsi)6sas zalQN)7fn-(yzc}jnS1#-x=6wu%Wa*$8GWo$mSnqQzpe8@MvFsoO0HE~vL!dh_2}c~ ztU7M4l$X&bf7Lz((i*97)fv`GMN;vqEhW-AY5l4#rP79|)xbqBCvE&lTepboO&{@h z%5LkJ$ynoNseIWxTl9CfN)=LNuW8kHyi%34ZB+=WC7p6aEY zy(UJbT}znib$4*yhS)b)QzsO7hN6g)Wlx*%w5ovrw$h-J^BIPU)_uX#iUg+{svj;y zSX{;XB%$sItHGd#3kOq&}!gt?7p~aDQADI^w77~gF zVvW zz|dgahWXlScU6li?v`muXECU@$Gv;^iS@M%jhYk7;yo5mVJW>7x#?5Hufhw<2=`#P zhhwldq|L>PLx>#^k_0cpivhY-@#N`dW(W+6D}^p0!cl3lY;@+Ji6AYF13EC5rM`9# zg;%ZTDY~iy7J*(#C0cGLdQ+G~4vZ}My*yL8Y!y4=73ydhXh--2S4tTej07Z#+en)d zWw6GbwFFW8^<`@yp8*lu^Y6kph1(znBNE61Jao%pSah`}-PNxkSRg()QTKwEt&{kS z){IxfCwi*M* z?vtjMVJ=BFE*cbPF&vTvO0Nw1Uo-@bpUl$yd(H8+b;uSc;u@GE;;CUCMF!t7pwG;mX3p7SAr6?}4ke=OZ!huFT z)j^?fhU^T6RICf!wo?oYMW9wlEiDNU(|3NcO$%H^Vmb2hb$B#bcg5H3-$yA2;zAZq zglIa-K0p-W|3KZFl@R<%5;7(uy5gguShN5Q!K6cv69{mYmI`!7&_bGnBrp<2cwlq^ z*DVt>Bj!a02*Hkz&;Vfx48xjGwg9hd5c!a(()6D=917FYpjsq~Lv;Ou?2kbfFo3=+ zZKVzICL}!&Cig-Ui?#mUmo+?t`QWvY(q{arO~llR2xGB4Z3F*GAm{s;$u}H#G0o}I zlylte^C1HA^?+9>8IW>n<95-YUUPhHrkbr$m0-sgl z{z#JnV;Dw6v`FbiNG67JFGs-P*cael5($(<(y~}AY1KnQ zTDg1oZZESb2#pEL+SB^&UJ=4UnJ)s6kg5f#8n@7_fB@+zbkJBwGdhFeO@8rM6cndu znq;OMOv17e;9J;_5il#31jZR10a=aY5oOZUVWbK*dxyADCZVfG^0R&i{_6dHKbGrR z!6Jv_V^B9~9%HtU;6+v_6wz{sX&E@k_y1*aNK*K{5^6s*w+=1q)KS*RmNr>B>1|qu^obS8N9ql z{1a9oZWF^^uQ&z_6}IiGgK(@wH8_MZL@bOHWB(hgqctpLV~HH{&@g;DUALsYl8eSb$nKLg z8P188^<`d#64R5(y0rVn7xi><8g6pGNmXK+0Ny0}X@rIDTuHkTmN8zQ8x6u(tKx}W zVr7Po0N-f}2RZ`L12{Dxw$zgb9w09PV1V9|5`goz+$XBnF;^F_566%W--dB|5Yf>7 zQfO!x`E(TanV1FT-^~NP$G;u;^1jgQYhw zYUy@zut!C5!W9`N6+B5PJmf9VcZd-dp+-;%f&SPeiILT1oTiCXd)pB}7GgUE5K1K& zqRb=K7R64DD#Iyf1VYT|gQG!SD48IZVN9ot83nrG1#18yW~C)*Uc=Iw&?_!05x>|1 z*SA**kXZp+BEJOTT7k?x*lw%W+(FImu@Q9?Z;#`>0T~M=MnBilf`#k8A??pa1M0xD z94$-$qXi>C;PB`(rw^K6&k4d>Y$pA_9pk7!EQ9+6->mEvD9oY2<;rNr6N znC)AhIyk$F+7B$6OvO$7Q?uEAh)+M=M^AUr!v^iyk>r{~Y$SiOsAev|hDJKW(@~O^{>5|V2=B5kauMEQWjVfJX5xfEXB+v(nA$I%QVw>-LFuP{O2)1@ zjM4jZS2W<(Xd5$QOiFaFI3$6#<`ZVco-j+6Vzf#~IF@idrVdWB(Rp)bN|Ug$;wa}7 zGta%mjhaWe%jTE35#DQ$@2QLoR6}q>gI>11N(u~GZTV&>G&5{Gc6UgnkdlT%jzs;P zlzk75NGUU=^xZ*hjzYtAX);Yw$mM7%i+l(?@tO{&vO15QYCV0fQz5cF`ad|@9v#M| zLRo&VBZZyjZUo2J4pEgddQRk2h)OAUr#4LMecI)-?=>J+W3WY>zs4zytDEP z=Z-n&j(O+K@s)PJQz0-Ab&h?)P)6tnjH!j~a-^`mHOFH)@g?)F+wBJ8I&Ei~z zUpUL>oaGQri}VHttv;@q#{HHyf!f9dCQ7S_TEcA1ESU$eVr8vlk*wb`7K8s?PRdG{-!;82Ff7rY z1Uv8!?JBG|jL~1l*khJ4Yr>*pXF#$itnYH~n`i_(2O(d(;x*i2R|#8N#ty<*Rw655 zVih^1P-hvKlN<&{`)R2wj)cj80JS;Jf5LlNOYddAWjy^ZI{kmc>5BeSuxcB#XF~Y> z@sat~_gq&4IHUj^lKtHrqER+sQ}|K0=5G{wu@)yR2wD`=_D5hMQl?CxaW5-_1C@nxQe#C214q}RU35)pQZbP?A zF;sS7Onn#L=u%865qQoaEnh+vK%(436&dk70$%f1u~J&D=ryHmMm{H2jVTCbs{GQ6 z2=u52xA5%r%nRjYST`XpH&|oVVqtMBzdzLrc$Peni7Ts(Y}<>9|?su zYef63DFNkbVfc-Rfv|Z?sawrn5sgz>`f^hiwkX&&Gr&;Dw@le0N~jn0L4zqXvQQ~A zk_M?PLSR>9AZ2G4MA}rAwgn73hYM=6(J!-Fq+`iyX@};)+F4jkPO|bMn*SsI)Zc){ z*s0BOy#4x{uP=DkPX|AU+=wJSzNF3fl{5QV!(;=Qr1u0g^xx>enR{bs-dR1~^1zcn z)p8|%?e)pm=RM;1(FIT5wfJOwdeh|CQqPI}@#aMbXUlo}wKray3eIPhOrO1%wPC@t zalyUqX7eY1u%$%Q?)$?X({z54NOqy5oZjH!i%paZV^4 zKQi&sm(IKePib;f!(IFAkCPjllb+_Jt$9&!S}GUvO1{V|o69Sk*#Vn*c0R9WBI}_u zZ>sN}bHhS@6ZyW`6P;5#u6Vw%mCo5pr~5v)RXojJ`Ly@GZR-Q4XG*xzc!9GFOM-Z>lm>;6ysKg<1eXtpc4sWs_oP1;)jF)Mq)b8NwxGgWynZ%=Z= zo}_b6QrPpyr}?Wv@k}iOHUB=R=tvg#8%qPu|4>ZV|1PWaNcnR6PmU=6 z4Gh<^g}cTFO}$*u^hzNt6Z`-rC&91o9Fx^i`KxOZ{5z(rMFKZwg6+=O4>xotpSeqJ zT5_bHpWS!^w$%K7+5#_oUoiUcT!Wv|xaWVl$D5LUC1Yei&}d7YMy{foz-ZLw>NRz%aDmN=a_3OJ7biR zgiA3=>%zRW9_uJfIB_mbWCOYlE1|o(SpWxKZN@BuQj;;pMkMgotXLjy6KO8Sn48`= zM;)xL$&@IBK0-k3Xsd(v1K^JKuaIpN04wuA@g^7k!En3A>t zBgk`%<$McAv|eHyd z@)Dg6;`Gccia&g2dPR(z_U?Mdi;wE@@DUqxEgRtZQ@r^3XY7ejjKe3B#FWs6Os~>J zBauFgmEz=RZEru_&RE4LSnXCQmx-J$Ca)IbYxa<1s(aalW1u)BAzvs^!Dl=hI5s%0 zQH;CU)^)mC*(+#kTE5P>K!C-~d+T<@590j{N>ZWc5@p0nq8gYAHS#*yt3{%V#SInOV@}J7*>wbi(BZl3nykg-%FLEa5Y5pl^vYcTlBhPs2 zLbhl8FiuanLW_5Te^9b{rs>`4NzX*qR3W*y@;&Fe8UADYNA?AA)5rD??Y}G7HC;EG zJ6rcx1wT8REZ8;Ciqn^#f`yXJQ&|s6#hKiXi$5%W_eXbvvs-TW&y~KAT)+En?!Vq~ z_tn1}{hQH$wLjT-9RBwS-iPx#|CIG%@rDnIZxqio+-#aJ_Rr?d6>p!gO!Q8=7o2&? z!man5TNhk;*G^5In%?w5`Hk}Xu8j+8%V&KH8>{bz7uHoSl+}ElWzMNXq38D9bk#T^ zkSFb)GEcQk?V8S;Zu+3*M$1gz^zNCanZU=rANJl1-c)7=XMU7)*Wb78{L-2Gpr~}Z zXQq3;$ct=O_6t+OgPb+UiO&i*FC`0ieV)^Rup#?}hnvek-u>b3$DGOTMS^XjWvX(z z`GG4pS>XE{+pV6@dXm0VzdgIKrg-Y`blG&rO#V#S^be+w8}d^a9Uc z*K#)ZSGMeAj{j!#lhIqFcMIoh4krCgz{w%L9B0p#2iyFgG~R0bgPFHCJoMx|ba<|L zC%sdx({=NX(s4}enKE5DGTut$Mp|%s%k--=Yi6=$UQ0T5-4_}bvYP>^#W%sj$G^6F za`)8PpY8>;Q!UfE=$*xSzxrX(j3b%1?Q>W4q6MA5wjzjF@Z?}jz>r3zj-e2M>6lA4V7<@#L6~nJFg@ zhXSGux0v2$$5L`F<5}7XAvaZG<(Ub70J>5sfQyDjiAV!F^gRxD#H{3gkC6qGg$ivf|ICb{@(Sigw%EcfveJh;rJqj70dX*8e*7S= z4@pQ0Sg>a-sY;9`_$T3UGk_X?f11EcB~F2=cqwd=0O---V`G1GoC zZz_MHo|Hk}W6r|#fYUt@y|Vik>U7=DFU>hiXZShirkU1zPVa)tbM5%#@u}#2*ZPJ0 zg2|SLg(Xwxr|YKAPB%~O`yzkqT>jSi{L0CeMJGo3+C>~?b6#^yI;J3QMGH>%_;GF9 z1W=?8GAfMp{}t!AJ=amE{u7V1ZNk%on&-FdsDe(jLK?-q3I3jB&LD6)m#b}4tkweZ zS>U(pjKXtl3~Pqphi?@Vd`o&G7Ev*yJxm0*Lpv=0vOTTw#Y8&Ys!#(&xCB zhn4;D^{e15-Fl30C*sBbPT0GbU4OG_B5%b2uUO#c*;G7ipZVC!8#qn$fvVayG?cd9-Sl2?X4A*M4}J3+YG=29zG3HF zX{(NevT{CtC=&(vOb~%LWT+woFBQaGeY6?DFodEY6b@_KHd>i?e3QmvX~xobU)ZOH ztP)9G##lCh$B>xBMH-1e`gX=r&in~`yTnB7SamlzosBhmp^$jkj5i0NX#YhE&3c7Cw@2DSrZe&k%hn!@%f^z{6d!ke5Gw_Jbe% z#SfmEIZy7j-pSr;;mPoO?H{~&%|`ZJ*li=6>p&FWvivdmoBn+!fz2 zckj)n@+OD^eO~i}R8~*uGF3sKa5YG~N2-d?A8n_wp~aF2C8Sw>+W#Vu{uI>52%O?C zD`dpt4FS?9^gjvcAL{F$8xRj^5h`VUSnfhA)l>)AHyc{EhQN3!-iE?Agt91MSF0oH z9{TlTmS| zv}f68UJ@Jk8DE>miuX;n=+# z@^}RVldy`+24xax+RR-}4bKD~@lAZ20fF{`EsFK@>Nc@*c*UpE-cIF*^olt$fV|Y< zrei0SmvJA{;snYBYMC}l$-xvt4RYQbEw9pt>y#OJ5N$hBpfW%0>qnM2*@|W@)pCrk z(n69U9LK4l7bgt)=X518AF3Bm=rR^y}qODESr5(zj z(HqP%TZx%bdSiwwpuC00?AuE%)=8faYF65wHZ#gc=y`x7ZYQ(}%*sP|&iIi(2)t$6 zQ!D3iUnv|v^d%)w{QuHCd+4vS?r#4y`!}0E>;C(S&x(J$C3)b?ec>!?Z+PIyhV7Zn zyPsV@??81KZ`t**sN{F9>Z!Wvzz0L`4*le8(p5d)iW7UG?t#NS5xcVOo}=}>%~05P zWA{s2=Sx~2Wamt+xpHCRBm(E+jW{_)l-^l3eQo0`j--eDwo(B;4&>a1^}P^2@xB*M7M6;t64*XDWz{#FY8Q`FC1x z7AEs+XHB#9vs;ty-Cu6rik#5q={4_cxOwPnGw-RTj78l;n`5G9(lu#X%)%fbL2mv; zT~a8X=5hMw@hP6m_Nf#Y{VG5Ah+p{C76IoV?$Zak6K4E9`CmA#w(AL>pW4~6njJHu zgfXfnTBX8CO2@Wv%vSGVlFWAm!>X}ftJuc$JmmVN?W6%2q|C7tn;3mjk(J*0!&|Gw zg@z49BE!-ax3Ew3m2!-VS~x0}=)!-CBO`}x#s7Z*ht+FS$YD>}@PQZ#j8!@1Zc=*L zf1GoMZk3b^A1rjCl|4)0|3ko2x>uPbv)riCh9OXdVG0bV1Zv83uqYvmVu?HGLPfUf z1zaG9D!AV|`M{q0geNax5c6E9A&|4t3<{1|#wBM%u zC7qadB!`J5ovCyK=8@Qfl@~IcC>3-|j+Lf<%jk;ADYP{Y&=n~&wlsw{{!#*&RE1m` z28ZTt+xBXowC`7D(Et>1>IhEAM)LgE!Zu!b@&lgZbAHd+f6wLqo^$*+u6mBEerV5` zu#a2*r`5%8`I^J&@rDtew?2uQZT$Yng;o6CCl~l#eD-67ckrIaC3Sqs<38TTuU{=1 passing scenario", + verify=py + ["tools/rule-coverage.py"], + mutate=("scenarios/ground/gr-r06-round-resolve.yaml", + "GR-R06, ", "")), + + Row("AM-2", "<= 40 spec lines per rule in games/ground", + unmutatable="no instrument computes it. `make loc` prints LOC " + "and nothing divides by rule count or compares to 40; " + "CB-EV-0001 records AM-2 as unreported."), + + Row("AM-3", "synthetic workload definition <= 50 LOC", + unmutatable="no instrument. The synthetic workload is hardcoded " + "in benches/synthetic.rs (see K18) and its LOC is " + "never measured or compared."), + + Row("AM-4a", "third-party LOC, shipped runtime <= 250,000", + verify=py + ["tools/dep-weight.py"], + mutate=("tools/dep-weight.py", + '"shipped-runtime": 250_000,', '"shipped-runtime": 1_000,')), + + Row("AM-4b", "third-party LOC, dev toolchain <= 350,000", + verify=py + ["tools/dep-weight.py"], + mutate=("tools/dep-weight.py", + '"dev-toolchain": 350_000,', '"dev-toolchain": 1_000,')), + + Row("AM-4c", "own source per third-party 100k lines", + unmutatable="declared `reported, not targeted` in the spec. " + "There is no threshold, so there is no property to " + "invert. Counts against the metric by ADR-0005 §1, " + "which is the honest treatment: an untargeted number " + "cannot fail."), + + Row("AM-5", "clean release build <= 60 s", + unmutatable="declared `recorded not gated`, and not recorded " + "either — CB-EV-0001 lists AM-5 among the unreported " + "rows. Nothing times the build."), + + Row("AM-6", ">= 100,000 applied events/s", + unmutatable="the Criterion bench reports throughput and asserts " + "nothing about it. The only asserts in synthetic.rs " + "are the stress-gate shape and the events-per-round " + "pin. No code compares any number to 100,000."), + + Row("AM-7", "scaling >= 0.9x, and replay of 100k events <= 5 s, " + "hash-identical", + verify=CARGO + ["test", "-p", "games-ground", "--all-features", + "replay_100k"], + mutate=("games/ground/src/lib.rs", + 'assert!(elapsed.as_secs_f64() < 5.0,', + 'assert!(elapsed.as_secs_f64() < 0.0,'), + clauses=[ + ("timing <= 5 s", True, + "the elapsed assert is live; tightening it to 0.0 s goes red"), + ("hash-identical", False, + "MUTATION-PROVEN SURVIVED: folding from fresh(999) instead " + "of fresh(42) leaves the test green. The hash reaches only a " + "println!. It could not be asserted as written anyway — the " + "log spans games seeded 42, 43, 44..."), + ("scaling >= 0.9x", False, + "no code computes the ratio of throughput @100k to @5k or " + "compares it to 0.9; Criterion reports both and nothing " + "relates them."), + ]), + + Row("AM-8", "determinism: same-seed replays bit-identical", + verify=CARGO + ["run", "-q", "-p", "cb-sim", "--", + "scenarios/ground/gr-r06-round-resolve.yaml"], + mutate=("crates/cb-kernel/src/rng.rs", + "Self(rand_chacha::ChaCha12Rng::seed_from_u64(seed.0))", + "{ static N: std::sync::atomic::AtomicU64 = " + "std::sync::atomic::AtomicU64::new(0); " + "let n = N.fetch_add(1, std::sync::atomic::Ordering::Relaxed); " + "Self(rand_chacha::ChaCha12Rng::seed_from_u64(seed.0 + n)) }"), + clauses=[ + ("N=10 same-seed replays", False, + "the runner executes each scenario TWICE (K8), not ten " + "times; the spec says N=10. The double-run is live and the " + "count is not."), + ("HashMap deny lint clean", True, + "clippy.toml denies HashMap/HashSet and `make check` runs " + "with -D warnings"), + ]), + + Row("AM-9", "peak RSS <= 64 MB", + unmutatable="no instrument. Nothing in the workspace measures " + "resident memory; CB-EV-0001 lists AM-9 as " + "unreported and 'very unlikely to bind' — an " + "unmeasured judgment call."), + + Row("AM-10", "0 foreign types in cb-*-api-visible signatures", + unmutatable="the population is empty — there is no `cb-*-api` " + "crate, so the claim is true over nothing. What is " + "measured instead is a clippy deny of " + "HashMap/HashSet whose stated reason cites K6 " + "(determinism), reported under a D4 leak row. " + "Withdrawn by ADR-0005 §4."), + + Row("AM-11", "null + reference impls passing ONE conformance suite", + unmutatable="the suite does not exist. `grep -rn conformance` " + "over every .rs returns one doc comment describing " + "future work; the RNG pair is exercised by two " + "separate, non-shared tests. The metric is a bool " + "over a suite, and the suite is zero. Downgraded to " + "unmet by ADR-0005 §4; T04 builds the suite."), + + Row("AM-12", "tokens and USD recorded per task", + verify=py + ["tools/cb-cost.py", "--self-test"], + mutate=("tools/cb-cost.py", + 'toks["output"] = max(t["output"] for t in per_row)', + 'toks["output"] = per_row[0]["output"]')), + ] + + +def run(cmd, timeout=900): + """(ok, tail) for one verifying command, run at the repo root.""" + try: + r = subprocess.run(cmd, cwd=ROOT, env=cargo_env(), + capture_output=True, text=True, timeout=timeout) + except subprocess.TimeoutExpired: + return False, "TIMEOUT" + tail = (r.stdout + r.stderr).strip().splitlines() + return r.returncode == 0, (tail[-1][:70] if tail else "") + + +def check_row(row): + """(verdict, detail). Restores the tree even when the command fails.""" + if row.unmutatable: + return "unmutatable", row.unmutatable + + path = os.path.join(ROOT, row.mutate[0]) + original = open(path).read() + old, new = row.mutate[1], row.mutate[2] + + # Positive control 1: the mutation must be applicable at all. A + # find-string that no longer matches would otherwise mutate nothing + # and report the baseline result as the mutant result. + if original.count(old) < 1: + return "HARNESS-BROKEN", ( + f"mutation target not found in {row.mutate[0]}: {old!r}") + + # Positive control 2: the baseline must be green, or "mutant red" + # proves nothing. + base_ok, base_tail = run(row.verify) + if not base_ok: + return "inconclusive", f"baseline already red: {base_tail}" + + try: + mutated = original.replace(old, new, 1) + # Positive control 3: the file content must actually differ. + if mutated == original: + return "HARNESS-BROKEN", "replace produced no change" + open(path, "w").write(mutated) + if open(path).read() == original: + return "HARNESS-BROKEN", "write did not take effect" + + mut_ok, mut_tail = run(row.verify) + finally: + open(path, "w").write(original) + + # Positive control 4: restoration must have worked, or every later + # row runs against a corrupted tree. + if open(path).read() != original: + return "HARNESS-BROKEN", f"failed to restore {row.mutate[0]}" + + if mut_ok: + return "SURVIVED", "mutant is green — this row asserts nothing" + return "red", mut_tail or "verifier failed as required" + + +def report(only=None): + rs = [r for r in rows() if not only or r.id == only] + print("M-D1-MUT — does anything fail when the property is false?") + print(f" {len(rows())} acceptance rows in specs/GameKernel.md §5") + print(" NOTE: ADR-0005 and CB-WP-0005 both say 'twelve'. There are " + "**14** — AM-4\n splits into a/b/c. The 9-of-12 prediction " + "(75%) is evaluated below\n on the same basis: >=10 of 14.\n") + + tally = {} + broken = [] + for r in rs: + verdict, detail = check_row(r) + # A multi-clause row is red only if every clause is enforced. + if verdict == "red" and r.clauses: + unmet = [c for c in r.clauses if not c[1]] + if unmet: + verdict = "PARTIAL" + detail = (f"{len(r.clauses) - len(unmet)}/{len(r.clauses)} " + f"clauses enforced") + tally[verdict] = tally.get(verdict, 0) + 1 + if verdict == "HARNESS-BROKEN": + broken.append(r.id) + mark = {"red": "red ", "SURVIVED": "SURVIVED ", + "unmutatable": "unmutatable", "inconclusive": "inconclusive", + "PARTIAL": "PARTIAL ", + "HARNESS-BROKEN": "BROKEN "}[verdict] + print(f" [{mark}] {r.id:<6} {r.claim[:52]}") + if detail: + for line in _wrap(detail, 66): + print(f" {line}") + for name, enforced, why in r.clauses: + print(f" - {'red ' if enforced else 'NONE'} " + f"{name}: {why[:60]}") + + if broken: + # The harness asserting it did the work it reports. + print(f"\nERROR — harness broken on {', '.join(broken)}; " + f"no verdict is valid", file=sys.stderr) + return 1 + + red = tally.get("red", 0) + total = len(rs) + print(f"\n M-D1-MUT: {red}/{total} rows enforced") + for k in ("PARTIAL", "SURVIVED", "unmutatable", "inconclusive"): + if tally.get(k): + print(f" {k:<13} {tally[k]}") + if only: + return 0 + print(f"\n prediction was >=10 of 14 (the ADR's 9-of-12, 75%). " + f"Measured {red}: " + f"{'MET' if red >= 10 else 'UNMET'}") + if red <= 3: + print(" CB-WP-0005 stop condition: at or below 3 of 12-equivalent, " + "the pass is\n under-scoped and must re-plan before Phase C.") + return 0 + + +def _wrap(text, width): + words, line, out = text.split(), "", [] + for w in words: + if len(line) + len(w) + 1 > width: + out.append(line) + line = w + else: + line = f"{line} {w}".strip() + if line: + out.append(line) + return out + + +def self_test(): + """Each assertion pins a failure this harness must detect.""" + results = [] + + def check(name, ok, detail=""): + results.append((name, ok, detail)) + + rs = rows() + check("the acceptance table is enumerated", len(rs) == 14, f"{len(rs)} rows") + check("every row has a mutation or a stated reason", + all(r.mutate or r.unmutatable for r in rs)) + unmut = [r for r in rs if r.unmutatable] + check("no unmutatable row has a token reason", + bool(unmut) and all(len(r.unmutatable) > 40 for r in unmut), + f"{len(unmut)} unmutatable; a bare 'unmutatable' is how a row " + f"escapes the metric") + check("every mutation target file exists", + all(os.path.isfile(os.path.join(ROOT, r.mutate[0])) + for r in rs if r.mutate)) + # The control that matters: a mutation whose find-string no longer + # matches must be reported BROKEN, not silently skipped. + stale = Row("AM-X", "fixture", verify=[sys.executable, "-c", "pass"], + mutate=("Makefile", "this-string-does-not-exist", "x")) + v, _ = check_row(stale) + check("a stale mutation target is reported broken, not skipped", + v == "HARNESS-BROKEN", v) + # And a mutation that does apply must be detected as applying. + live = Row("AM-Y", "fixture", + verify=[sys.executable, "-c", + "import sys,os; sys.exit(0 if 'ZZMARKER' " + "not in open('Makefile').read() else 1)"], + mutate=("Makefile", "PY := python3", "PY := python3 # ZZMARKER")) + v2, _ = check_row(live) + check("an applied mutation that breaks the verifier reports red", + v2 == "red", v2) + check("the tree is restored after a mutation run", + "ZZMARKER" not in open(os.path.join(ROOT, "Makefile")).read()) + # A verifier that is already red must not be scored. + dead = Row("AM-Z", "fixture", verify=[sys.executable, "-c", "raise SystemExit(3)"], + mutate=("Makefile", "PY := python3", "PY := python3 ")) + v3, _ = check_row(dead) + check("a red baseline is inconclusive, not red", v3 == "inconclusive", v3) + + print("mutation-check self-test (positive control)") + ok = True + for name, passed, det in results: + print(f" [{'ok ' if passed else 'FAIL'}] {name}" + + (f" — {det}" if det else "")) + ok &= passed + return 0 if ok else 1 + + +def main(): + enter_root() + ap = argparse.ArgumentParser() + ap.add_argument("--row") + ap.add_argument("--self-test", action="store_true") + args = ap.parse_args() + if args.self_test: + return self_test() + return report(args.row) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/workplans/CB-WP-0005-assertion-coverage.md b/workplans/CB-WP-0005-assertion-coverage.md index 5df223d..228a49c 100644 --- a/workplans/CB-WP-0005-assertion-coverage.md +++ b/workplans/CB-WP-0005-assertion-coverage.md @@ -138,6 +138,52 @@ than claimed, the finding shrinks to three absent rules, and M-D1-MUT was not worth its CI cost); **3 of 12 means the pass is under-scoped and must stop and re-plan** rather than proceed to Phase C. +**Delivered. Measured: 4 of 14 rows enforced. The prediction is badly +unmet.** + +```text + M-D1-MUT: 4/14 rows enforced + PARTIAL 2 (AM-7, AM-8 — some clauses live, some inert) + unmutatable 8 (no property to invert, reason stated per row) + SURVIVED 0 +``` + +**First correction: there are 14 acceptance rows, not 12.** ADR-0005 and +this workplan both said twelve; AM-4 splits into a/b/c. The 9-of-12 (75%) +prediction is evaluated as ≥10 of 14 on the same basis. Measured 4 (29%). + +**Second correction, and the one that matters: my first run reported two +SURVIVED rows, and both were my own bad mutations.** + +- AM-8: `pub struct NullRng;` → `pub struct NullRng {}` — semantically + identical, a no-op. +- AM-12: renaming `max_age_days` in the price sheet — CA-17 reads it with + `.get("max_age_days", 90)`, so removing it changes nothing. + +Both would have been published as *"this row asserts nothing"* — a false +accusation against code that is in fact fine. Replaced with real +inversions (inject a per-construction counter into the ChaCha seed; +revert the AC-9 output resolution to the first-wins bug it was fixed for), +after which both go red. **T08's question — "is writing a weak mutation +the new grep?" — is answered on the first attempt: yes, demonstrably.** + +**The finding is larger than the workplan assumed.** 8 of 14 rows are +`unmutatable`: AM-2, AM-3, AM-4c, AM-5, AM-6, AM-9, AM-10, AM-11 have no +instrument at all. AM-6 is the sharpest — **nothing in the workspace +compares any number to 100,000 events/s**, the headline throughput claim. +The problem is not three unimplemented rules; it is that **more than half +the acceptance table has nothing behind it.** + +Harness controls that earned their place: a stale find-string reports +`HARNESS-BROKEN` rather than silently scoring the baseline as the mutant; +a red baseline reports `inconclusive` rather than `red`; the tree is +restored in a `finally` and the restoration is verified. + +`make mutation-check` is deliberately **not** in `make all` — it rebuilds +the workspace once per mutated row. `--self-test` is in `make self-tests`. + +**Stop condition: see the note in T07 and the decision recorded there.** + ## Phase B — correct the record ## Task: correct three committed verdicts and restore the fourth