From bb35fcb168a28285aeba24aa55bffcb26c2cf258 Mon Sep 17 00:00:00 2001 From: tegwick Date: Fri, 31 Jul 2026 16:54:07 +0200 Subject: [PATCH] CB-WP-0005 T01: spec->code link over every numbered spec and every crate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AGGREGATE becomes a list of source roots and rule patterns become per-spec, so the link runs over every numbered spec x every crate rather than GroundRules.md x games/ground/src/lib.rs. The prediction held on the first run: AM-1b kernel spec->code link: 15/18 (83%) across 10 source files unlinked: K10 K14 K18 Kernel rules are link-only by design, and the output says so: they are kernel invariants with no aggregate, setup preset or command vocabulary, so scenarios/kernel/*.yaml with covers: [K11] would be a tag in a directory the runner cannot dispatch. Claiming scenario coverage for them is the inflation this gate exists to prevent. Per ADR-0005 §5 the kernel arm reports without feeding the exit code until 2026-08-31, then binds — the date in the tool, not in prose, with days remaining printed every run, because open-ended "gate it later" is how AM-4's targets went unratified for four workplans. The self-test asserts the gate returns 0 before that date and 2 after. The zero-rules positive control is replicated on the new denominator: a kernel regex that stops matching aborts rather than printing 0/0 as though it were 100%. The self-test passed while the tool was completely broken. A print( inside say() became say(), so every real `make coverage` died with RecursionError while --self-test reported all-ok — it only ever called kernel_arm(quiet=True) and never executed the reporting path. The control named the behaviour and did not assert it, which is precisely what this workplan is about. Fixed by exercising the loud path and asserting it prints, then verified by re-breaking say() and confirming both new checks go red. Seventh instance of the harness-does-nothing shape, in the tool written to find that shape. Also caught by its own gate: a self-test label that printed "0 K-ids" beside a passing ">5" assertion, because the detail string rebuilt the pattern with different escaping. A label that contradicts its own check is worse than no label. k_rules, k_linked and k_unlinked are registered facts under facts-check. A limit of that checker is recorded rather than patched: it is line-based, so a tagged value that prose-wraps fails. Co-Authored-By: Claude Opus 5 --- facts.toml | 18 ++ specs/MetricsAndScenarios.md | 29 ++- .../__pycache__/rule-coverage.cpython-312.pyc | Bin 10007 -> 18261 bytes tools/facts.py | 12 + tools/rule-coverage.py | 213 ++++++++++++++++-- workplans/CB-WP-0005-assertion-coverage.md | 36 +++ 6 files changed, 289 insertions(+), 19 deletions(-) diff --git a/facts.toml b/facts.toml index 4081f65..5358384 100644 --- a/facts.toml +++ b/facts.toml @@ -57,6 +57,24 @@ text = "21" fmt = "{:,}" by = "tools/rule-coverage.py" +[k_linked] +value = 15 +text = "15" +fmt = "{:,}" +by = "tools/rule-coverage.py" + +[k_rules] +value = 18 +text = "18" +fmt = "{:,}" +by = "tools/rule-coverage.py" + +[k_unlinked] +value = 'K10 K14 K18' +text = "K10 K14 K18" +fmt = "{}" +by = "tools/rule-coverage.py" + [pinned_main] value = 92.03371920000004 text = "$92.03" diff --git a/specs/MetricsAndScenarios.md b/specs/MetricsAndScenarios.md index 9d523f8..372a8e7 100644 --- a/specs/MetricsAndScenarios.md +++ b/specs/MetricsAndScenarios.md @@ -45,7 +45,34 @@ and add capability-specific rows only when these don't cover the claim. | M-D4-LEAK | D4 | foreign types in canonical interfaces | count | novel — must be 0; enforced by grep/deny rule, the Clay-Borg hard rule | | M-D4-SWAP | D4 | capability has null + reference impls passing the same conformance suite | bool | adapted:hexagonal-architecture port testing | -### 1b. The coverage gate's two numbers (M-D1-COV, M-D1-LNK) +### 1b. The coverage gate's numbers (M-D1-COV, M-D1-LNK) + +> **Widened 2026-07-31 (CB-WP-0005 T01).** Until then the instrument +> matched `GR-` only, against `GroundRules.md` only, and linked against +> one source file, so all **18** K-rules were outside it. +> `58/58 (100%)` read as "all rules". +> +> `make coverage` now reports a second, separate denominator: +> **15 of 18** K-rules are named across the source. +> Unlinked: **K10 K14 K18**. +> +> **Kernel rules are link-only, by design.** They are kernel invariants, +> not game rules: there is no kernel aggregate, setup preset or command +> vocabulary, so a `scenarios/kernel/*.yaml` carrying `covers: [K11]` +> would be a tag in a directory the runner cannot dispatch. Claiming +> scenario coverage for them would be the inflation this gate exists to +> prevent. +> +> Per ADR-0005 §5 the kernel arm **reports without feeding the exit +> code until 2026-08-31**, then binds. A newly widened denominator is not +> a regression. The date lives in `tools/rule-coverage.py`, not here, and +> the tool prints the days remaining on every run. + +**Both numbers count names.** Neither proves anything fails when a rule is +violated. That is **M-D1-MUT** (`make mutation-check`, CB-WP-0005 T02) — +four of the seven defects found by CB-RES-0004 were named in the source +and inert, and therefore invisible to everything on this page. + M-D1-COV counts tags. It proves no rule is unclaimed and no claimed rule is invented; it does **not** prove a scenario exercises what it names, and diff --git a/tools/__pycache__/rule-coverage.cpython-312.pyc b/tools/__pycache__/rule-coverage.cpython-312.pyc index a921196bc950f46737ee989270f64a558f1062eb..ccfb67daf772ef6cd3a2dc18e4fe7877a11e5ade 100644 GIT binary patch literal 18261 zcmbV!eNY@nmSR|p-|peKf2JS$WlcRi z=6H^q=0xr+C-R~x$e-oe-E`K(?&h;*cDJ0hu)FoF6?bzmYshxi#&e?OtX;I8%@(uH zIz-#q9MOK(Y2y6+V3}S~`YvXR4l(BgvzBw#B|6W#{qBBV%zd3d>+#$1)hD_F%M~>n2)qbEI?W;79uSXi;$LzrD8G4uJV_OC6_8WPAWzH*1=urdi5vd zmwu4?@d0{G7tf8DMoivSks~&aYdKLMA5qy$g81nn%2nadEKz~0dhLqq)v%?_>7o~6nEfG4pYNSZeFAISX zt4yxzU~jr8vvx?V7W$}dXiD#haDhF`%@>4VKvw7*P|`2C&mZzhfv{Ys(XOki9q|nX zt67hOfzTz;P{PzS!FTbZ6pf)fuGlTJ|fE zVPRPEFpw(S&7Z>z5sF0^!XbhUT1oNh;#d@=?KH3ov{7Pv(f5&s3WSHYM%hN#^T zj0=#%3v5ss8>w7s3~2^vgd&O@5SeVFv9cmXhR_ZN{=gatgK(^VTQ!mfR@1TiM&SyL za3mBM8V>rYEb5`Y_e18;cj?k_&=-=0D+6eq@3tz8{ z7ec=j9%9T-%{Qt;45Nn5lcyc5GB9lfO@iq3`vO7Yia!KN)ahIfgdrX1_x~8X2|_-F zW)54^PMkUIsMNSO6ajA_K{XiCzDpOXm&(R=FoR@(u|M3eHHTygYgs>JTXx_L-eVzx zOm($)lTbBigNXhGVL6}#F4GW(6e%3c9Gc7Zwe`Z^ z-|1HT!@^E)v%pr_%=J|8$+BOf1x<1I`jm(-h|$toLky5>g@b-b4#^(6O;YU>MH&j9 z&?|*6p$kk(v8MU23L^o35HH!-$opP7&pJI6kYzAe|4H(#gwaaOFyiVqLGqJq5dSxB ztJ}6Q#b>``z=vf~mTPFVYB1T^5MlUbENS4BL5PisMg(&X0zTmPiCA_04rAVHn@X+F zB82>q$@F#tLRbs~=s%H5F%Wx_MEsEamjPyZV=)E>t>@kc5i{>|Rq(5*K9RZhU zWsot4nz3jML#D!^9jvc%9r3b8Z%*y-$afYHYHDPEu)jv}%ZlSm+FCoke3GWTU+R&< zVMWb9S9Pwc5{t(xEj4G)Z#^e&^{RQPT}n9jN@Z=;bKZ03)e>6yjg3i``s%QI)=Ki1 z>p)X=W`6(W|Hh{Mr5*WSu1DfEC$su!ZzHD$W)wBQ}`Euu*@t>&Vy|PSp$w-mIi4EGI2je8Ee}EE&_xFDC8&kSNnWoU~vsLOK`@guM2ol~GQbK`CjW z=cEl=FDqil16!n%R@%_yBrhiUewkRL>p)Ly>IT9?{yOX~a^FB5=IKaHZ&Hbmw-7_Bz{R_tzR?X!t zl$1`ZQ)+C}{gTQjX0E7Yx?`$i`uNoGSjp^$PYbIbQL(Q$KDTP3^(!l?cxXn=PaK>x zf3p1Ap17s>>qj9yD{-aj?Yxkp;=eIgJO`IQlYWoLIx=MHe2Xx%5m> zD3@JnsTj=C^V0Vy=8-N}y3FYJZti`)(`%93D4sM!HA|W#f6~?;2#Hv~y{2R~Ey_KL z|0>2XrEkd8IzfFO#nq_|vTj8Y&X! z9M^T}hB~3nY>71_+?9zOZ`|UQHsR%x?$YYPx=ZUsW_NWvXw<}3Kad|aM@=h@b<|jG zR$7a6I#FI9d$ka>OxLR{T-3x^RDUK!sw&h78oo?k^+{n_7NDEMEMmq5TR`MmNDAW< zbvkJY5Bo!y-`FF)mZX)Yx6Js3sZB~qk{_07#Ase?!jh|Vt%uAZsSYpC;!kcw0@gX* z*V@P07jpA4hpbKG{DM9A=DORK*s1qycbxA!-*e5`5_ygD_Qnb8Lo-Tzp5qxiuCdnG z9ze;MF#jI8)10x9{1@tedita{jX`4~;p^0;vv1V=?UPI7qh`gB%axXT-K@rjbMQN8 znZ7TrQOkEwGJkJM)+oQ?W@L(5q`atQ5Zll4OSI_AM~D}#Q7g(>zq?$&A+9`k{pS!u z9zvLEF<9!y#X=`J7w@!^W?!&dlv$70|oNbKOvI)+~9cRcpNclH{q=Ym=&Qm$p(OAi>7x z_ep&NN&ZAK%Qp=17opI=)F1X{N!3&eQ;d`HDA_{gt*p{ymPTHtHI8P3J|FTjTHQlE zk|ml7z4()FBgx#p?AB&%U+#j*wrlE*A58op;TGt3-i?uok@%`@a}|GQ`ibSomiz8~ z3(mqtPyTfFRCc_q@ovS>Oh31LY)N?9$E}Nb1=FjhR>fC0-);NZ#-CSwT#?A@7|&X; z=iG2iI40X3Y~DKAK7DNJSd72fIal~#OU>jVwzcKmJUQ3!&}u4d`VD8wZhn}H);!GT zobIus-<*G>KCz|E!{%Rl#;v5frRPqR%TSqC*v9mE1htTt9ZEK8VV6EryCG34NwR1t zN-N4}hU$33)MpyNGJU?BR*I-;kaopomtGrYbK$p__*2vMO-zNuVw+OXOi;_rol-q$ zKIN;uc5T1ej-SbFz*_BH-926HNxoZKWOq`&RUQro6x!Gzsxx_v*B*bPda|w@t=>3x7_HM==hT(pX3~hTMm7V zo9qPbk4uXVnk^sK<|FOHZ{so+;z!5=l1V5gH%fPNkRLLQ@;n#izHRRsHJLD|>4kbH z$8+g{S90{42B{g#F3}=dMXN3}V%CR-ZVhY{n!ae#_eHegTZg)vE9dHAFI<6^{W4nc z-Q};EU*@jxUVEe!jd~G=J{ikQ@lN3}K50k;M_>pOjP+_H;8*sY#-hb+)}e^35QvpB zRcj?ulsez%lWt&$48!V?72qL?M01VF)9Lq3WK@a1k))MTgBorMgp;sbVdz~|V5&)N zD4eu+Ly|%lkDN?8w8r$HF-g0KMk;_tk`{VO+M)Pg_5~w;h}>bSgMDl6jr1=SIaiMi zEk305XhwtdJj!0hpL`z4828VX-2Y~&f)0LT>%`V;HDj$0vP)+Q64`5^AX^Xci}w6m zrSGhLb8SqSJvGNCR&Bpq@u~f}f5qm0fd4$lbz|?u-fR2gmR0|1U7qt)rqO?|yk(vF z<8{`SUA85XM~eeZjmH3b=pjD0cE!jtW?^WQ94&Io!;Q}C1-_tT#8|Z zif;YOXV&A%D4nO5{;qn&e6b*C8RgTnO)*pgEV#xpto4!gvrwnK{LbbxCl)Ej!kV7j zS9vKPvzzu#cENXmdD+J4z1L|e1$v3}w{LM`$=fD=BM5%Fz|SJZObzLZeFP*XrEg`JCnwi%rR2P zMr4;lCI!0&h15gFO{^5XqF1k5tom@P!87akrcwP&8d*%f^G0pB*97y23P$Z%-))25 zj3%TWjWNu|SlOf5xHn)YAJ*zLMjc`S{ipX0ZO9(n`Ymn9Ve(S<14qjFmG zfqiXTH0LrsTku>@&lzKsGg$L%dU`uB(%UmfiPf?rnlnhY+OkXMhlZ!2)?CON={D@j zc>B%uu_0;|^Hc4%qOV!vPWmP8idxZv-O((hjc*&fH|Xxr>55HyThYg=QRjpQm><-1 zhWX`=x<=g@Wt*ce$t3Q9nX(rY_eFEx;@-AI-I7(@A9dr|gg3kj(rKDeXVj&-pa43r zSHnng;`{t=PV|0L3%LGCYM(!#HnjWt22ubxnbgDZtq4Bm1%TraJ3kO_23512UGyt} z>T8o+AMH{4MO!(|*2lF-{*0gICd`AWokK4|_bAhdGL5MbP9g->%S#P(vm`A9umP?E zP=~7EiAdVuc7vfL>-PAVW{+DL@UNBoAA+7-jO4dt+=AVG z!!_ZWvCQaPl_U($RfG7K{686a#z_x0|pxhY?Y3_M-(t2_Ne!wP@} z@<}nA-|*4Mdw@9vLX}?T@bLlk1Du|6rm345NTK<K$ZV9%~Xf&(I*9$V| z)%7WFgy!y&^`4~q=2uv+Q#3F6N7Qn{K57D@Q2}9WaF);?TptW&Q7h}}r~Q~_cSczPSJPDu|2@Uxqau%lu7ilDf*FaVgt_u5v`aQy^ zrdK{K(M@av+}FI$7%2uIXIQClX29+qpy35@L?%1)%=l7vyFkk05>(#;>7lv%cpH*q z(4D6t!RIl**nH(jO@wY??SO6_HUBw34V&P)$HI+pe`UVHGr4}BMA}rlEy?#J`EZh# zbRpJ2rKj0@FZx6NtHaWMbddbLuZ%jMcyO z;{2MXuNm!sTUgAih0K4Lf)rS{D`l@*sp-tk-!1fo5c-t1&%jY8`%hiW)pCqnEcB$- z==7F|ohI|}@-CO{ni{z7;i82Uw~)6B&^SQ*kOy$`v;TFd1^ktxh41?F=1O~~7fjE9i z3!CafkO6&?Dy(K{h@x7h(@Wuu%yWRQ$>AXUkZ@Eo9x$@1!*FHD8HSJ44*G%r14USB z#S@mnLsX$bnEV}S#~^`CXhh0A$sE+Ox>k!N(VD^*lbZcBzu=$_U20b0vr_!cXfnA1 zWwlmsGPAP;IWkO94Kh4?nh{hj1j7+ga}9&VYLi}`g#+mB#a`VRihvWW1cYnREE^wTiL~7a;;KQqDi2a(J14 z*Q=TZ;T6>!z9b|qhgyytS54=Ih=x*;j~curr-Qdls=$pQ3Yqxt#bCIX0alv(z|sq! zUeXM|LDCGHka=Pl#w(G*02h$|awrlEO2>%SaY{~5a+s1Mlt3%xq@$E{B1t+J^uc&8 z?WbHaE`c*sP=Pm3dWn9SdM?8E1ne(qfkRSG+AsA4LOpm)W@%hYS{cvMMyzy_zO&E} zX=ojR_tI&~%o-*awkW+oIfR0@#6gn$8!s=;u(89 zcf(g^K4;?-v&mZim7Nt4Zdb?ajwCi7%_vd6@d<3`@?TrBmTKnj3%q)ck={)b7%2=f_XZ?EkQQ zR{X#_yL#?W+_8r`urF2lu0==gxOmMw`RYXNt?pRiTQAQXi>dLl?bL<_xcFS9w+7xB zdUI&D4Q|9!iFLd0D))2S#ttnyi*LR3&iOab&o<2E&9x-f?z$`9ceakTKlX4|*Rv6l zJ&W#Dx9xXw-_4yB=UV1oO$bf*-GG4!AldV|Blkx2MD?{=pei&XyA~X-$;}hBaGvC> zjj!E%&m3Rd%6=@kie?JNL)68B>2*`<5_uaExf|oo^4PApqw0yxY^_cSTJr*kdMCWs zw$9|wJ63<<*mc{L5Vp-7oELUcCmY#VwJ$h4lPwclKXFvdY>L&tS+(FSm^?JuII}yp zGqxq}ZI5GU!MDTw6HR`1-M8#s$j%)*xFBqS+tpe$Z(lQR8SkI$U$o~=n%{h3s(rRN zUa)P>G}kbkvR%t?+knxfk}-7BJ#KpIne{7L|{{;ghSluM%{X*6Vz+{DYSbhLr-Ej+#QnzYR) zS~7Y;u8C#Wht{;cIg$lfJ@Z@J@A7RLx6u0o0G)?rwgr|H`7c-LRi|;J4F9EpDCwnO zk)zNNgEUc^A85fsC2gF`HhDOsW>6&%;UL=*_(NQ@{DAXXFw5;a}s0pi88qPjQV}m=(6B4y?q*lI2MVc^?9$79uMy79s3+CdX!erY92q6Dm$d|uylKQF z9igH^&C19(TCPHlN1YjO(luuCC^tiDvNK8>5iqcy9U1usedOmvok%n5FkCLk&wblK zN;2^%(M8ty5}KKHxzoIbot~xhcL@bgIydT&GVmzTKsT3^2a-9sK_9X7J?hZuq-R4k zN2<=?NZ~WSG_c+8oGY~6;6hY$mQt=g!CK@@8{U&okPwHQgrt zDQcj;B*}x-dVcy2s^&ohA4=zo2LHuUS|&z|GkUTrS}c~mWf50L3$F4qtVD5*xE8)$r8c-W5xa$YoV&_IgPHAIWk{Z`1W#;$0QV#K2BBaao~hsY-{EV~FPMc!02 ze`UI8Lk??+0DLsH%EN+vAIMRjv^t|78=fIShNS8(KtJ-O@{BJxrdxyB%HK9WgHpa; zf0|MTl?weW=-ZaGY|}I78Jtbii`tjWip|gX(yrI_JIxB5gU~L0(2^_mN z*(4SLh1+7dQnRkWE2j`t#;yuZ%7Sb_a$n{q`RhBca4BrAvQy2;2!K=z33gf@pD(=( zB9U$+?Ok0byO<9S<~xuW&AMWKRe&)R0KC%odthMpM<_N`vsft#)d+)z+s;DvD|6FD z$g){diGjVOVr2;oT7C%NTj_r(f}~o=jz~~*jf$Ivr+nR0lTfMJH+i+g5ip?*;%bRz z^-_m2&0RrVtA|FXj7tE5VHPgT;)>Nn0!2KxH=`mJbB#cI0CxtWK-L(=-|H12zR)lA zA*zGop(*BB(W11~{X*J`txN@D(*cR}r-%(+7J6-*F7P%#h3$$S3Xx{?Ul7tvWlR9j z22L^~o<;Q%3cwIYqn!vIfjQsIFfer;3*gq#o~7X>GBNVGFaIwt)g%b2dGC$}*y0TR z+LyF+p6qN#9xyp8&Sd!aNwkKkCea(WTuOIIh8s7x4*d3_;bT2M+vcy>t&grR6D z0r&%~d%0Qd*7@(hj8OhipLR3=;aUW^(3ukQG|T=yU?f;ULfIb-3~O=6`k-hK4KYOn zCsZd527@d8VsnJRx=fZDqZ7g4(P0V3JH}$2vCUFHKJu=U%(x+xa0J1kh#$SotYZy7 z07{<A)D{+=CBnOPT>BB#WN!I{Ctp?jt8VTaNe8SoO5E zyxh$|3U4VQSuvwz7Tc4wrP#ybdy>{mIFt}dng;@k)I*eQDL^=B*AEgT?IeE`C7879 zcnzW`2XJ^HJd`vK_bJIN4aJd$seud=VI8tXh&Uo_5%=N4(n0%_w3})%L&P++B3m1| z9>5;tD#T=1@zT%=3*qY0EtLBlfAWtoNMqa+nECG4k1yB?Z<*h*ziFSh35)L1Yp=!a zWnVZ;Zdu}McO*(1J}YgSFKtSc?nyZJj`piv$J${YJM&=4 z!S-@jFK*xYw^e^r^&2yvz5jE!hXPDie)EMZ@0x$CgT?0Z)*a&(co=T(MY!MT_Z)ZJ z@46F0eWGCdynXwGjiUW_ptUeyt=EpC5wJgtW=f{kCmb6P-CMT)m#&7%Q?dHlj!(*W z{PC-CSHoBbl0|pXNW}PqElyGex>wu|!(>&Swh8=c)Ys~uA(Rs(_u~xFDH_dFB zs*AlepI4K}t%acvL)-<8J>#TsYpKkS|>_~514qbR+*c(Hi>~I(!gxT zzao3*v z_Pq&M%`hT?_%qVd)T?xKe#bKbgz)k3WC&c1i|eOA7I zzI^}P%Zc(s^Q#YiR(5>8?D%Bs!z`|_bh>$}d8Y4;=VIIE3pS6pEjsfi4^ApGt&@W> z+pP6JxaW%Rn&Vsc-fh2Eey`}BFP?w+zVpa~g3<*~$+TMox( z->+>=R8pw-v~$WCv&>wMbh)_zsSISWuzJiqDgj-NFCxba>` zV$bpW&J!f2$S!QSzw>Ck;n**B#q+z+qv^`2%9)*S)B*d*E&oE3V;ZsSD#*N@hG0B2 z)A!E6n**^ciFI}N?c4N$IdHH1=anB<{-QB)@bqFqF?E(Wzn(bnDSN{?cI1K0v*0d` z+e;r5t|ky;V=NEJ*N;Yd&b{-uPY&~3=|;deU}o9cH(DVBdF!6aiHyg`EOmtBTd z#*OocqTye+BbYWzFEnV^pVIdggYis2EmFt|Vj&E>Kf~TNA_fMHSdcVLy%mVp(WzLX z7VK^0Ib|0DQq9nlv)(o<2I4VDKFnoTW<1!4J&jN3H2^iR$~!XX;*9J{JyOba!n)Q= zHo@;$yAeGAzqr%rSyqM>XIK}bHpQ@Eqqb<)5~Rb<&bwr%PO1>nksZY;mlV>GjZ-Cr zbT~${GiYQ*ZA-9+9A%^4qbSF0xeEPv`abH6I>p>e==*4{=u&|4XjjzG{TUvS&Ntf6 z@CX;;3QVGVrSHH!Pe#7+9r9tTBF)cun~4|{Wa12rzG0EV=E;>jEUzftyQTJUsDQmM zW>`i^W+|~0kqCxu2c$t@C0C{M*zYoN^-A*b40OTEzFG5(Z_>EJcgZkmTeyHL*wHdG z`pJ*19m!1voMt27bY0{(D4^;3Op#1kFo+R2rk@BFemb3swr0n)gvzA{QoT5iM#{x; z{m?MeXJtP{HsE+P&b7hp5W{^Gq)O^0RMsm4BgV1f)Ct?vnQBH!*F%j1ycclFx(}LB zKpv_Ux^PxlVi6SZXVaa>U@06gY4|L=J4694Hj6{Miz{2K2|nl4?DI`AtM z0hx-SFw1bcm85hrgi@wMDDa{UVxtF8Ja8A%9#;<+gX!AIhi1-G!~!TN#1|&1n%j{% zB9ad^MX0Hcv}FT>K|iXUX_BfHD;kJen+$VM8UZoQ2&HMclx*fHkCY**ofcEgb=88p zsqTB#K$a-8r=-{Lq^@^-t6x=+qEvpAik8Ac)w@Q~D~KS`yvk~BDuhAv7e}U0TDN~l zcZN7XCdgT-e@-X%Q(aFVUDv{F!9PRr3~_^<#12&>L?`nQv1TD@XIkOqmW4`^_Y3TV z^RhF#UbPti76AE(g@Z>r+m3X02rx2Wm#M4u{Imm_wU=fHA`FJ;I4q16L>5#tFECu{ zWEu&TEp1)eaqz#tv(u~Y$+VP3KddqOw*+9{>c$ru%a{hMK|kZ9ji|R-`aNRu50FI2 zfXKA!$Rq!Ktz>506)rQpm}6Hv?!as_+9)!POwiPk{(zD{q=dloq_g|vnXcCMp01N8 zPj_qHVVabh2RZK>Q(79Qmt_5U-OR7gW}x&gk+RUdk^Y1V5ZaiuG9E}bC}WxuGRvf! zl>8YbL@H@ZIoqW_$1j;N`+|^pF?&h#Q6EE*Jv4)Vi{!Us+(Q?aoBxTk>gG$+=cmqp zR#Z7(RGBEM8f$-IF@zUkxO#9Rc z`2WTZK}-Lod&i_N)-rqMFFXH~kCW+xgsh)4GiRpGCtT%Y9bY)t!V+?B`YdPbe9qRf zwgp%Iq;Dbs%Z*t@hu+A6r|^~>+xeC{@9>f{(gv^MhdX93f3Rn^X71I$E4X{+Kb75e z-aB|tzUN0DkMkQBH*Sjg->ZtPnLUr@XYVB=Xy30bwud-fQvQT9Su2=+{lK0RcU9cq zv@2e*JML`!)ZX-K%JI%tz+G6^x|p{y?kSHw2NJmrq?Padwarc6IFH=7AH~CX`{a)4 z#;L}c!-;}T_wAcu7O~gxhd;|Kg8n|$vEcSBa zCf85xd$rwqzx3D0gIII&Pmq9vJpYwt6K{EZ90wdczv6O!#bvYq9siSC{kdcPm~Fu+ zjAbu4H;rXKv~#Og$GM^fd&!s;lcH$jhrj9VxQwbxg<7fR7y?kIIfzQo=pmF;j+FN+w0uz z9y5DR&$8}_)I>fj$U8B>Gt>u*)pKJ3x7zvJd`7zc~}pK0$7p4ir_MlHG+ zB^ompV=THDXR*Zui!UZwV$o#D*R{nIGl8aA3TTFNXpz-82SsI2pf z9Gekt!(k=RBS&kDS}T_A$Sx1Bnb#{WSdz5Y3KGRoqolJz2iC~8(g^b z<|=onWgeQRV8pITYK4Z;12jgN3;sB+2W~9e4j-n{rGW`+h0}`VKqO$#T;gD9xoUaf z=TZ~2c+I*yz^E&Lzs{W$3JyY2|22yI0(fgnx&k*qU&8^{JZnXc(pk@}3y1-&hBx3? z!wK929hSpTaJUUr!hygD*ueXt&sXObO!e2Z$6FtPX?xVMYFw%x$HvZwTtGldE)XY= zS{1KhRjW;N8Q~3V10JwC#7O-sHj*@(Hm@@9L=D9!qbBN6^>|60Nc>j4kn13eDxc|6 z-$>_8FL9VTs-C73Q9mYK0Y3V=ZmK3{3j*(l?)W3WES*Y>Pn`QOemh|Rnw7{%XDzz%u|KYf*~*lkwk)p=tpO# zLN|>^9#WA(NjCJc;xb;gnG{EWd0KsvIggoax&U2JK)j#Vw& z9j0?ZL+r zcDd;MYm%AceZ;o zDkAK-L)a7VX{=|{cu)JOu^AIlmfwuQIsy{10eTwyDGYw)wcm zdgrx5pL)>qS^^<15Yer^ogX3@^?JUaJg0t{?*%x2pT8az~9$kqUu5Mj&;-&m=-OTi-v1jRR&T(|1$SO|I*k|J6cw&THL zokeM+yaL$(Z!E@OD@KtQt7F|VaRxa>AQJO5ND3g&hL4k!yt)jP1P?g{(o1;k7TNJn z9wq9F!CH^V0jC1<(MZE8c%qNuDx2-txnJ<&>P)dn2Gmk=Co{RO-eZvwf>wy~~A{`(Rok$P?@idZW zfk2P&7#AW$3ll0RBtJPhGgF$LnOv9_m|?!L=GsmGp`T#TLp{63eFIm1)H7UWHPzhs zld)5%pj+J>KRs0F^rK+|c>)#0gC-6T=5ZIj;Xsl3`l{`Tr_?9oljpyVHy4mx3ywGN z0Q4>>3kzcJMF~ZT4nKn6QDmojVWPVu#NMdEKS|tx11?PY{?7L&HuaXz+mGKaJ}{oV z-wNbEdkvMp(7$aOztfC4)A&Pna4r}5IG38sM?dZ%K>b9yjC#uD!Vy1SE;F}ME{9kJ z9cd2jLkFq50n#D`dx5`gQ9|-El2#^#xJY91cwBBdL_kf4(R5pM-)CnQ~G}?)=Kctbg`uU}km-b?Wd}S{YA%lBnn&kHK z2ALsy*;Ax@uS6CIIq}emXh!<>#;1DMpN;gbi*H=1 scenario. +"""AM-1 / M-D1-COV and AM-1b / M-D1-LNK, over **every** numbered spec. -Compares the rule IDs declared in specs/GroundRules.md against the -`covers:` lists in scenarios/ground/*.yaml. Exits non-zero when a -scenario claims a rule the spec does not define, so coverage can never -be inflated by a typo'd or invented rule ID. +Two denominators, deliberately separate because they are measured +differently: -Stated limit (InnerLoop implementation rule 4): this gate counts tags. It -proves no rule is unclaimed and no claimed rule is invented. It does NOT -prove a scenario exercises the rule it names. + ground GR-rules in specs/GroundRules.md, covered by `covers:` lists in + scenarios/ground/*.yaml, and linked to the aggregate source. + kernel K-rules in specs/GameKernel.md. **Link only.** K-rules are + kernel invariants, not game rules: there is no kernel aggregate, + no setup preset and no command vocabulary, so a + `scenarios/kernel/*.yaml` with `covers: [K11]` would be a tag in + a directory the runner cannot dispatch. Claiming scenario + coverage for them would be the inflation this gate exists to + prevent. -Positive control (InnerLoop v1.1 §Step 5): the run asserts it actually -found rules and scenarios. Before this was added, a broken spec regex -yielded rules=[] and missing=[] and the tool exited 0 reporting "0/0" — -the harness-does-nothing class, in the tool that reports our headline -coverage number. +Until CB-WP-0005 T01, `AGGREGATE` was one file and the rule pattern +matched `GR-` only, so the kernel spec was outside the instrument +entirely. K10, K14 and K18 were unimplemented for four workplans while +`make coverage` printed `58/58 (100%)`. + +Stated limit (InnerLoop implementation rule 4), now doubly important: +**this gate counts names.** It proves no rule is unclaimed, no claimed +rule is invented, and no rule is absent from the source. It does NOT +prove anything fails when a rule is violated — that is M-D1-MUT +(`make mutation-check`, CB-WP-0005 T02), and four of the seven defects +found in CB-RES-0004 were invisible to a name-based check. + +Positive control (InnerLoop v1.1 §Step 5): every denominator asserts it +actually found rules. Before this was added, a broken spec regex yielded +rules=[] and missing=[] and the tool exited 0 reporting "0/0" — the +harness-does-nothing class, in the tool that reports our headline +coverage number. A new denominator inherits the control, or it +reintroduces the defect the old one was fixed for. Usage: python3 tools/rule-coverage.py python3 tools/rule-coverage.py --self-test """ +import datetime import glob +import os import re import sys @@ -35,14 +54,45 @@ AGGREGATE = "games/ground/src/lib.rs" PROVISIONAL_WARN_DAYS = 30 ID_RE = r"GR-[A-Z]+\d+" +# ADR-0005 §5: a newly widened denominator is not a regression, so the +# kernel arm reports without feeding the exit code — but only until a +# date that lives in the tool rather than in prose. An open-ended "we +# will gate it later" is how AM-4's targets went unratified for four +# workplans. The remaining days are printed on every run. +KERNEL_GATES_FROM = datetime.date(2026, 8, 31) -def parse_rules(spec_text): - return sorted(set(re.findall(RULE_RE, spec_text))) +# Source roots searched for rule IDs. A list, not one file: K-rules live +# in cb-kernel, cb-events and cb-game-runtime, so a single-file AGGREGATE +# would report every one of them unlinked forever. +SOURCE_ROOTS = ("crates", "games", "tools") -def parse_code_ids(text): - """Rule IDs named anywhere in the aggregate source (T09).""" - return set(re.findall(ID_RE, text)) +def source_files(roots=SOURCE_ROOTS): + """Every .rs file under the given roots, excluding build output.""" + out = [] + for root in roots: + for dirpath, dirnames, files in os.walk(root): + dirnames[:] = [d for d in dirnames if d != "target"] + out += [os.path.join(dirpath, f) for f in sorted(files) + if f.endswith(".rs")] + return sorted(out) + + +def parse_rules(spec_text, pattern=RULE_RE): + return sorted(set(re.findall(pattern, spec_text))) + + +def parse_code_ids(text, pattern=ID_RE): + """Rule IDs named anywhere in the source (T09).""" + return set(re.findall(pattern, text)) + + +def code_ids_over(paths, pattern): + """Union of rule IDs named across many source files.""" + found = set() + for p in paths: + found |= parse_code_ids(open(p).read(), pattern) + return found def provisional_items(paths): @@ -67,6 +117,17 @@ def parse_covers(text): return {c.strip() for c in match.group(1).split(",") if c.strip()} +def _silent_output(): + """Whatever the kernel arm prints with quiet=True — must be nothing.""" + import io + from contextlib import redirect_stdout + + buf = io.StringIO() + with redirect_stdout(buf): + kernel_arm(today=datetime.date(2026, 1, 1), quiet=True) + return buf.getvalue() + + def self_test(): """Each assertion pins a failure this tool must detect.""" results = [] @@ -99,6 +160,58 @@ def self_test(): bool(prov) and all(o and r for _, o, r in prov), f"{len(prov)} provisional item(s)") + # --- CB-WP-0005 T01: the kernel denominator --- + # The control the old arm was fixed for, replicated. A pattern that + # stops matching must abort, not report 0/0 as though it were 100%. + check("kernel: zero rules detected as a failure", + parse_rules("no kernel rules here", r"\*\*(K\d+)\*\*") == []) + check("kernel: matcher works on the real spec format", + parse_rules("- **K10** A replay bundle\n- **K9** A snapshot", + r"\*\*(K\d+)\*\*") == ["K10", "K9"]) + check("kernel: matcher does not match GR-rules", + parse_rules("**GR-R06** lead first", r"\*\*(K\d+)\*\*") == []) + # A single-file AGGREGATE reported every K-rule unlinked forever; the + # union across roots is the fix, so assert it actually unions. + # Compute once and report the same value that was asserted. Building + # the detail string with a second, re-escaped copy of the pattern + # printed "0 K-ids" beside a passing ">5" assertion — a label that + # contradicts its own check is worse than no label. + srcs = source_files() + k_in_src = code_ids_over(srcs, r"\bK\d+\b") + check("kernel: ids union across many files, not just one", + code_ids_over([os.devnull], r"\bK\d+\b") == set() and len(k_in_src) > 5, + f"{len(k_in_src)} K-ids across {len(srcs)} files") + check("kernel: source roots resolve to real files", + len(srcs) >= 5, f"{len(srcs)} .rs files") + + # The gate date must actually change behaviour, in both directions. + # A "binds later" that never binds is the AM-4 failure this replaces. + before = kernel_arm(today=datetime.date(2026, 1, 1), quiet=True) + after = kernel_arm(today=datetime.date(2027, 1, 1), quiet=True) + + # The reporting path must be exercised, not only the quiet one. When + # this control ran `quiet=True` exclusively, a broken `say()` made + # every real `make coverage` die with RecursionError while the + # self-test printed all-ok — a positive control that named the + # behaviour without asserting it, which is exactly the defect + # CB-RES-0004 is about. + import io + from contextlib import redirect_stdout + + buf = io.StringIO() + with redirect_stdout(buf): + loud = kernel_arm(today=datetime.date(2026, 1, 1)) + out = buf.getvalue() + check("kernel: the reporting path actually prints", + loud == before and "AM-1b kernel spec->code link:" in out + and "gate:" in out, + f"{len(out.splitlines())} lines") + check("kernel: quiet suppresses output, loud does not", + out.strip() != "" and _silent_output() == "") + check("kernel: gate reports before the binding date, fails after", + before == 0 and after == 2, + f"before={before} after={after}; unlinked rules exist today") + print("rule-coverage self-test (positive control)") ok = True for name, passed, detail in results: @@ -192,8 +305,72 @@ def main(): print(" ERROR — claimed but not defined in the spec:", " ".join(invented), file=sys.stderr) return 1 + + kernel_rc = kernel_arm() + if kernel_rc: + return kernel_rc return 0 if not missing else 2 +def kernel_arm(today=None, quiet=False): + """AM-1b over specs/GameKernel.md × every crate (CB-WP-0005 T01). + + Link only — see the module docstring for why K-rules cannot use the + scenario `covers:` mechanism. Returns a non-zero code only once + KERNEL_GATES_FROM has passed. + """ + today = today or datetime.date.today() + + def say(*a, **kw): + if not quiet: + print(*a, **kw) + + spec = os.path.join("specs", "GameKernel.md") + rules = parse_rules(open(spec).read(), r"\*\*(K\d+)\*\*") + + # The inherited positive control. A pattern that stops matching must + # abort, not report 0/0 as though it were an answer. + if not rules: + print(f"\nERROR — no K-rules parsed from {spec}; refusing to report " + f"kernel coverage", file=sys.stderr) + return 1 + paths = source_files() + if not paths: + print("\nERROR — no source files found under " + f"{'/, '.join(SOURCE_ROOTS)}/; refusing to report kernel coverage", + file=sys.stderr) + return 1 + + named = code_ids_over(paths, r"\bK\d+\b") + known = set(rules) + linked = sorted(known & named) + unlinked = [r for r in rules if r not in named] + phantom = sorted(named - known) + + days = (KERNEL_GATES_FROM - today).days + binding = days <= 0 + pct = 100 * len(linked) // len(rules) + say(f"\nAM-1b kernel spec->code link: {len(linked)}/{len(rules)} ({pct}%) " + f"K-rules named across {len(paths)} source files") + say(" NOTE: link only — K-rules are kernel invariants with no scenario " + "mechanism; and this counts names, not assertions (see " + "`make mutation-check`)") + if binding: + say(f" gate: BINDING since {KERNEL_GATES_FROM}") + else: + say(f" gate: reporting only for {days} more day(s), binds " + f"{KERNEL_GATES_FROM} (ADR-0005 §5)") + if unlinked: + say(" unlinked (declared in the spec, named nowhere in source):") + say(" ", " ".join(unlinked)) + if phantom: + say(" ERROR — K-id in code that the spec does not define:", + " ".join(phantom), file=sys.stderr) + return 1 + if unlinked and binding: + return 2 + return 0 + + if __name__ == "__main__": sys.exit(main()) diff --git a/workplans/CB-WP-0005-assertion-coverage.md b/workplans/CB-WP-0005-assertion-coverage.md index e574642..d20d952 100644 --- a/workplans/CB-WP-0005-assertion-coverage.md +++ b/workplans/CB-WP-0005-assertion-coverage.md @@ -66,6 +66,42 @@ output: **Refuted if** any numbered rule in any spec is still unnamed in source after the pass and the tool does not say so. +**Delivered, and the prediction held on the first run:** + +```text +AM-1b kernel spec->code link: 15/18 (83%) K-rules named across 10 source files + NOTE: link only — K-rules are kernel invariants with no scenario + mechanism; and this counts names, not assertions + gate: reporting only for 31 more day(s), binds 2026-08-31 (ADR-0005 §5) + unlinked (declared in the spec, named nowhere in source): + K10 K14 K18 +``` + +`AGGREGATE` is now a list of source roots, rule patterns are per-spec, and +the link runs over every numbered spec × every crate. The binding date +lives in the tool and the days remaining are printed every run. The +kernel figures (`k_rules`, `k_linked`, `k_unlinked`) are registered facts, +so they are under `make facts-check` from the day they first exist rather +than after they drift. + +**The self-test passed while the tool was completely broken.** A +`print(` inside `say()` was rewritten to `say(`, so every real +`make coverage` died with `RecursionError` — while `--self-test` reported +all-ok, because it only ever called `kernel_arm(quiet=True)` and never +executed the reporting path. + +That is this task's own thesis in miniature: **the control named the +behaviour and did not assert it.** Fixed by exercising the loud path under +`redirect_stdout` and asserting it prints, and verified by re-breaking +`say()` and confirming the two new checks go red. Seventh instance of the +harness-does-nothing shape, found in the tool written to find that shape. + +**A limit of `facts-check` surfaced here and is recorded, not patched:** +the check is line-based, so a tagged value that prose-wraps onto the next +line fails. It cost three edits to place two tags. Reported for T07 — +either the checker spans a paragraph, or the rule is stated as +"tagged values must not wrap". + ## Task: M-D1-MUT — one mutation per acceptance row ```task