From 6a996bd71e5e36d7483e7a79b3301bd895907644 Mon Sep 17 00:00:00 2001 From: tegwick Date: Sun, 23 Aug 2026 14:26:36 +0200 Subject: [PATCH] security: rotate exposed KeyCape credentials Assistant: codex Assistant-Model: gpt-5.6-sol Assistant-Session: 01a02e3f-7301-7622-9be1-12e5f352881c --- WORK-RECORDS.md | 6 +- ...Y-WP-0011-live-secret-exposure-recovery.md | 71 ++++ ...tated-credentials-2026-08-23-final.tar.age | Bin 0 -> 20680 bytes tools/rotate-live-exposed-secret.sh | 373 ++++++++++++++++++ ...Y-WP-0011-live-secret-exposure-recovery.md | 30 +- 5 files changed, 463 insertions(+), 17 deletions(-) create mode 100644 history/KEY-WP-0011-live-secret-exposure-recovery.md create mode 100644 history/KEY-WP-0011-rotated-credentials-2026-08-23-final.tar.age create mode 100755 tools/rotate-live-exposed-secret.sh diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index df8b6a9..d0ec89e 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -18,7 +18,7 @@ | workplan | KEY-WP-0008 | finished | — | workplans/KEY-WP-0008-registration-handoff-and-client-mfa-policy.md | | workplan | KEY-WP-0009 | finished | — | workplans/KEY-WP-0009-provider-capabilities-and-service-identities.md | | workplan | KEY-WP-0010 | finished | — | workplans/KEY-WP-0010-openbao-operator-loopback-callback.md | -| workplan | KEY-WP-0011 | blocked | — | workplans/KEY-WP-0011-live-secret-exposure-recovery.md | +| workplan | KEY-WP-0011 | finished | — | workplans/KEY-WP-0011-live-secret-exposure-recovery.md | | task | KEY-WP-0001-T01 | done | — | workplans/KEY-WP-0001-keycape-implementation.md | | task | KEY-WP-0001-T02 | done | — | workplans/KEY-WP-0001-keycape-implementation.md | | task | KEY-WP-0001-T03 | done | — | workplans/KEY-WP-0001-keycape-implementation.md | @@ -81,5 +81,5 @@ | task | KEY-WP-0009-T04 | done | — | workplans/KEY-WP-0009-provider-capabilities-and-service-identities.md | | task | KEY-WP-0010-T01 | done | — | workplans/KEY-WP-0010-openbao-operator-loopback-callback.md | | task | KEY-WP-0011-T01 | done | — | workplans/KEY-WP-0011-live-secret-exposure-recovery.md | -| task | KEY-WP-0011-T02 | wait | — | workplans/KEY-WP-0011-live-secret-exposure-recovery.md | -| task | KEY-WP-0011-T03 | wait | — | workplans/KEY-WP-0011-live-secret-exposure-recovery.md | +| task | KEY-WP-0011-T02 | done | — | workplans/KEY-WP-0011-live-secret-exposure-recovery.md | +| task | KEY-WP-0011-T03 | done | — | workplans/KEY-WP-0011-live-secret-exposure-recovery.md | diff --git a/history/KEY-WP-0011-live-secret-exposure-recovery.md b/history/KEY-WP-0011-live-secret-exposure-recovery.md new file mode 100644 index 0000000..99952c5 --- /dev/null +++ b/history/KEY-WP-0011-live-secret-exposure-recovery.md @@ -0,0 +1,71 @@ +# KEY-WP-0011 — Live Secret exposure recovery evidence + +Date: 2026-08-23 + +## Outcome + +The accidental rendering of the `sso/keycape-config` Secret was treated as a +live credential exposure. With explicit operator approval for session +invalidation and brief SSO disruption, recovery replaced: + +- the KeyCape RS256 signing key; +- the shared LLDAP admin/bind credential used by LLDAP, Authelia, and KeyCape; +- the Authelia-KeyCape OIDC client credential and verifier; and +- the privacyIDEA application signing secret, invalidating the exposed admin + JWT. + +KeyCape now carries the non-credential marker +`ROTATED_DISABLED_ADMIN_TOKEN` instead of a privacyIDEA admin JWT. The current +client set has no enrollment handoff URL, and `requireForAll: true` keeps the +normal MFA path fail-closed. A durable, least-privilege privacyIDEA service +credential remains a follow-up architecture gap. + +## Custody + +The authoritative replacement material is encrypted to the NetKingdom age +recipient in +`KEY-WP-0011-rotated-credentials-2026-08-23-final.tar.age` (mode `0600`). +It contains no pre-exposure value. The temporary first-cutover archive was +securely removed after its contents were superseded by the final rotation. +The final archive was not decrypted during verification. + +## Non-secret verification evidence + +- Pre-recovery public JWKS fingerprint: + `4172efd1109e2a07b728c59b28558bf6e0171b22afb95d1692f87fccf284039b` +- First-cutover public JWKS fingerprint: + `8ec2651168468caa595f185bb7021cfdec228eb4992f1f6c1521c6e62762c73e` +- Final public JWKS fingerprint: + `c6faac5dfeef2453daf9cfc14671f62b535dd321befdf6014e3ee5c2cf1f1156` +- Final Secret resource versions: + - `sso/keycape-config`: `51346058` + - `sso/keycape-pi-token`: `51344120` + - `sso/authelia-secrets`: `51346049` + - `sso/lldap-secrets`: `51345775` + - `mfa/privacyidea-config`: `51345601` +- KeyCape, Authelia, LLDAP, and privacyIDEA each reported one Ready, updated, + available replica. +- LLDAP accepted the replacement credential before and after removal of + `LLDAP_FORCE_LDAP_USER_PASS_RESET`; the final deployment has zero copies of + that emergency setting. +- Authelia accepted the replacement KeyCape client credential and rejected a + deliberately wrong credential. Its active ConfigMap verifier matches the + copy held in `authelia-secrets`. +- privacyIDEA rejected the synthetic invalid MFA proof with HTTP 400, + `result.status=false`, and error code 907. +- OIDC discovery returned issuer `https://kc.coulomb.social`. +- The registered OpenBao local callback returned HTTP 302 to + `auth.coulomb.social`. + +## Discovered operational gap + +The NetKingdom manifest currently code-defines the Authelia client verifier in +`authelia-config`, while its legacy rotation script updates only +`authelia-secrets/keycape_client_secret_hash`. The live recovery synchronized +both sources. NetKingdom should migrate the client verifier to an actually +consumed secret-file setting or update its rotation procedure so a later +manifest apply cannot restore a stale verifier. + +The credential-routing catalog also has no concrete native rotation entries +for these legacy KeyCape backend credentials; generic and interactive-login +routes were insufficient for incident execution. diff --git a/history/KEY-WP-0011-rotated-credentials-2026-08-23-final.tar.age b/history/KEY-WP-0011-rotated-credentials-2026-08-23-final.tar.age new file mode 100644 index 0000000000000000000000000000000000000000..8eef81bac925dacfa8408bdfb4e5a2d672ba903e GIT binary patch literal 20680 zcmV(dK>WX9XJsvAZewzJaCB*JZZ2sL_rErVmL@{LwHYFNpVgwaY!;@QEhZ{ zRW~&&IdoKHPD^iZRBLQiMMO72W_JoLEiE8VMPy}TYiLtAc{Nc}FG^}mLr`}@Wok() zQ*LK6X;w@*bxvnTGfY@SS}+Q7E`&?y;WP)2B@{1E+U3e~)SWPO)Ku-xjcnNWGB{n7 zZ7SDx>y?Bniv3U(;|W>nB()vXByFPzr{KNlH5*ZnL_oazOmV+FDqYyZ#QA}O?2eh5 ziER?L*Q{zBlk|+N{gr?Kox*4qsTSFvjaLn>LG1s8(K*K~X6Icvw9la5@W$pBu|fC` zLBb{|{7ki}i~DfZ6kS}nrU@VcN)@;N$H~40dqffP_D9fjB!Fl0(cxSht8tc>LHqjW zWl)O?vRId|Z(#m*?dJC(pPH0WBicG;7{d0le8dhy_)ZCt2aExu68zrFKGsvr77T@o zF%JYBQ%~z?I-|(s$Bs)bM2TJ8Lq%szSt;zR3G7KdK|SF9tTGz%bvPGAM99N7ITnig zTY^y0xpVc)+SCaQ(;WicMBAH7p zAU|pG|AA9)V#RI_xX^!1TvdiiIo(5J4R_;_Mg|=foDS>w488!VPBMtNoXunkPG>%P z^M8MpavjDej|ir`Im+6%(FiC$;DJ+N;0{qOB(n!76%X#&!Cur7QF<{|N`Dx{n6w?i zp(*a_S(--`EbUj}@^g7{f-!DLXnav9gOC6C6RKymVJ%83jy?JaO@)bype+M_=pzdtmI7wn3zit|7L@IKvR|QxsXvW}h_s{k%ffGc9 z(1ST6dwrXCmB+RnY^Fncu*%WHTofflnoPL7kdN{f)|)x2%n`U}0}X~Kr5lzHi(8P3 z%a=lGCj@qgg1LCJfT%)-ZrYRQrBJf(vu3{>hSlu*4uYVk2R3%hnpn-KsYrm|iWt-OxJa@{3$|ahjek-4UXd*=YVo z1J=0uLMEy_(qqA@UGh4|7ak*c$EZ()SUgW#lcN;`oL>`XC})I=t5+XH|`93?B4 z<-ZHMB=p>=W*k&a#()VIUnq!o?E!noth}cnLAW1}Q~SevY^P4B>IxOl(cRmScm~)E0ac2#L9TXShO5IDMOA9Rg?Hf^oQzZEW z5CuUiI%IM-zva5g{)RSB^8evTxz@;}G{t{OOfL)m_>#pcAa54~``S4^EbX#GEVit? zc*%|xS5&mGLA2N9pWs9V+mHUbrlYn+!Q~4kcoG|~b+m$4FS$x1?V9mc9gHk5>zsKl zPN;3b@e(<{f*qo5c6uLOQ9gW&QRFRR>Jb`8FH`bn&Y`SIw^n9(A2SUNORj3U*>ved zF@`{J95keug5+t)IAy!rGC^YP^W!8(#hoil@_3ge0BRYYY=#+h9sz%9hdglN90iu^ zm$h0=NVde@!D-utS(pu!G4td$#ka0${L!3N2KQV{OKoD&9$iy8ByjrnG>rGIG&1W}KPyo%yWLtPSkgnImUL zJj~a68jw89E&5~#76&M%#AHL)?TcYXl3%*3_?@eX>O~aakjwcH zK(2iNxp5bsV~LI80Nr79gnQXj!Zs7aHl(i2#aj|e$M}jY>hG1 zm>+3YG{1wQR;7IL%*C>p?{84AvN_a>=1Hx^FA{G;elUn^NddR>67l?C8=a-Dc@d|g97`%@15ettgjpKzF|imI~vkL}HPTyp|+wqiA-zgPAgV7Icp0G!az z#@{n@@uYdBv6!d_a3x5d2#qbPDJJXNzR@0e_|L#4V+4%slKwDl_c)vCv@=o@WVsDZ z+Z|Sa&}P*PcVSk{fkoth^CWf6T zD7?_IATPEaQJdFwINB%shNPCrki89GTg!4gRhg1gBv(~lpG+dtJ9d&qAb`J-m!{<7 z4`R2AoAD9BPdAtW`;LCsH6*=MER@nrqdt7a%-NMiFt<}bHs92;oE|ZFOeTXy2NH96 z0yzvjk&6kcKZNN5rMvIFxmkfUylN2~_rRukMe=NAg`WeD1R*1Rg7L5WK>bESz+@@I1Z%-GEbOp*3NCfeAEfSz~C=7 z9Ve3@#)AW`Jy|xuZ!lGKp?)^TEDxWYkr@;xC~kg+P(l)StKxNCv@xxaWi&bv?`&u8 z%p3qLX}Bu!3c>M?c8xoZ)2h%KYyJI87Psxf6J9EwkP&E3f^P5kTs)j}^&!bPw(Ds0 z8pSJb$!|OISdd`bPip_Oe{?zB>fzO1@$tJF*mV*MZwG6DA^)Xw8HO$5b0pu9PKZwT zS_bSSLxU3R`QxLnxte)FPZx8gFjpCcO2?uyQkkuoU6|QrK?gfPyXOEP!Uw|CpC$n) zgW7;uh8Epo@4|E&QI_%<%7dqT@mkwVjfMZPPM>dCdu zR}R9t+&DI9?p!xLvk#eb|2!N5?15yV6!2~5%WS2d>b;0_U(!{B3LO7x$5wYb`_hES z&tHo|2GE5$+MS$3P`E8lIwS2eJpG=`z)Z6=d;I8WgxAjKf^t|96!(cL|Io>Mq)!qrkrO#oliP#l~ZGzjr}*cL1b&$)qc%TL4L#vFX%nshHdD1>To`&d1=Cy)OY0iosZ*xX#s&;z()5daWvNc9|c@^Av zqAG~Kx2pfQC?XZq)f4oZBQ%X^5^%{<_E;-`gbU3Z2ohOqrbYd{EY8NBZCC*}dI+*L}XHCo2& zOC0~+g0FPZaIQ8zrD6voW zgRsefhc3TOHfl2dK&1}IKlbAMV-REsORj%Xn}GK3D=0G-sX0dnoEH4 zW|lvEavm=NhQqKW9Hg67mGk4Hpc;`PN7=hClDC}6SzPx#O*d!NVwLYdgeA1P&j^xb zY%|Ew$TH)T5H5rF(J$&TF;5cH?I<@9y(Eef95)Qq*X;}k(Af1#cmd_ofD?pG7H9pX zX+!Z{6ZEHS(v`a=;L`$1x?YU+7{s~jZ|=wfF+kK~v*-Zb&KO3*i8>>1EChv2Y1rW0 zz!P)?C!U#ZyOY>Wmg$Ts*gXMCt3-J?czq2d7WJHoV?st1QRpVLtMFeE5Fy-Y1AQ7+ za;d1BUZaS72Jl<ncj zPK??iA>7#U@yo{|muYDT5@`!&4_d(F9*Cv|?*tFL(zEs=k?hycG9j*LB% z$KZC`n837YY%R0R;73qc_b=5pY#ZnRMBKRk%KqVkeQ@;UJ1sm#sm5GGJ`La8?y5zCS9mEfA+?BCp` zts#8CgHf;R5pi(Q6E&kBAZP_4{^g?8U$Y5DvA~jP`sMX9&1ePUlC}#+ zWLa}4OB8c2Ud3Y9mlAzPo-j!9jwGasWf2hDAKxo`1^?Q zknp9bZVq%XgP2)bqbb|#2Q7S|OVvn!?`PnZ7(*mmqE~)XRP5TEQ6bDn{~{r~KqVam zs1|I_ck8D(%kFwd|Ds<;`NKhr`|=|t_L6wgT_&PF+p7&z<7f5qah-$KWU_<-LA)eH z-Alxra|@x%-nd4auPCy_T|;46y1wYNU=IOdi z?c4S^+PL9RZ=e+k8&eA)up?&0xkaC5>gRtULi6&_<~mwzOjDwi4zLP{czNyF9vGDv zP4Rcam2Xa!Jxq58XbA()A)P2kYICqNckFZ|@F9*q%T~g>KJl3NyvxLosa^n|5dydZ zVv(!+fM$*CZYBD_5kh~{4{y^E(@!o3N=>H7Ao=! zj~c5~Fj9|Op~mX-f3MoW0S=*^3`a^=u565OW2DwT7Wy(FVj`4kd1)Bg_bR*ws~|SR zRDXffH$3svS9?+Hl7@q2(aLclR9d<70SbMdot>?=-S}>=0mx+>+$OBI4i5b{iZz}uV3hpyM)W!OHl1# zW7zd5GmGl=u>WmvIq)7^Y5}l4H#mJ0sb4zrx=__DW(zq|Hc^BBmP^MbdmO9c`FS2t%?3Y@A_zX5Vm{j0 zq!BNDq^cjm&b^Yn{~@4m2OY{7qpTyQ#nYC8awdC^?{u8egqO<5->$Cqhl=Z5_iJkc z_}^_bNrPHbTDU0F!?hN_;Y1MNVDq5Y|Dx#tbo0`!u5UaU!4Cx4)GwmeJfXXeI3^HR=4=8)~ z`R^U`YZoqBz&v$(oC5_WMeAV_a_T($*0^LXeCvII0$eYMM=KU6Wz3joaEAt_vm`~g z0ZN*ve?#QF#$9J%?_6qsKlSYpcd)-q7A#k1=S|MIm{eMvEV+t*3TQx!ZdRff+eDUe z^+pML5$bpfA*>Id;=9I3G@r%lJ*XA?Qh87rwMO-@grL%1Bv)>`iCwlDF`z+0IqDJ2 z?zG&7{4})C2E9=vpUTR#8<2Rr{}>1TWbr|((pkK=7rM=ad~t`G$-P0lP-Ou`j-fwP z_bZ^otR{%e67U+Tk~j*cY&g80i#d0oMFg2cZyL!Jwj|*9+)i&TYE@@c%7p=c@=kp9 zk!`jgc5UF;5k0|5IO}Tft<{yj#iPI1Ws$plt8(6iP^fZ{gHxF1Sqt0K;!YWO95jsy z1y;e}7*iE7$att&sw+`@g=NzQ<{X7ll{fD+)wZz)QeboBXIY{8LViSs=MQF%XvCpxz!HPq_4@F+qkFxsGjJDbj3GDonfu9KK4YT(E%BXO~_<>zJu2w-v$j|yVNnZWl6Fq1~Y zlKjo4I3|+~kz}2NO9Ge=tJ=;QT1PvmEYx=<@U@PmlFT3CmP?XQ`2m zw!&HF{Y4%mP;|40*aJvZZMyiDWP?Go}n2}vOm^zf? z27aU&>z+LqBGfC;pYJ2WI{3k)_mW(Bv_f*t1}W6Fm_<_dlal$r3!3tb_f}^zIQRI2 z8T*{IR)ii(2t3=r10H=@nauTo)e*Uh5?$*;d8B0Qhr+VkH?X#rS6nlwGkKE@4FQ5k z9!%mWl=CfzQ3pI+!(XQs2$Xav))j=!UUa&X`n8K*!*CXI>S4|;z)>P?F4C44e!jfw5? z3Rww8C*bnl2R#)NC}5zEm{Y|MWd#P8Z)KOL)&)YY-6fxavV%vizP=M`%FQ#0cz^ z6?YPR6CgQ@(*ukUzCG8DC@5+FaGlvO40H*3>3r@x`Uq;v1%&p0?7nA1wbp6O^}QQT zQr>I+I2sc9j)Sdk*jY{a=u%LC?e!J|5Cr0 zA>sB%-FmSXhTl)PGU{#Mo*L=T7FTCWJVlM!H3{1b?L^uX+nQE4FtuOPxc^*Qq`r9q zbK7-z>$*Q-cLm@tBwh*TWzfI53WF|+y7ChWi0sy7ZArfqEoOF-{WPIRkJ0jNnAe&- zbPBd>x{6DYTdp$dC}5;2iS%S(K>TSVAJQ+5wtaCJJ^U773%)x`VLcua7EV>aMa>kV zY);7p5-b%-@4lVpG(SR!@<}*QJHjBkB)9Vt-!d0V3@v<1=t~$-I*5JoQrkUG0`0RVLA4cWNw9dW;*RV` zj*sNN{!GW{H4j|Y?^ty7->1P)FKle!CsqXCvMMZ zl><#pnG@1K^9Y=`FE$x`9j}W)LiYFb6muX$GTc}Iq(i$IC!gKM!W`jhY2Xy@AU8Gf zp{jUgQO7<;A1QxseaG->Ive!S7^8DXW1U93%9s!)8D!lK+4p@_<1w@m&T=W!!Wv*Q z-t16mL;wvLW~rRN#5hPdH3lpXur1asXbq=ikqoS5LRA7(&YVLP^n$21#j4LnOz15U zQRaxEF?f|dQA%q62)ldkHCZoY&qZ#^+T6(QOJ(gQW-E>0bVpFT6jRjyqMMv{$yJJu z$Bmki8ZL~Sd(W3AkT1Fey+UYR7Q`0PGLHFp<>rKpq#5T2vlgkQwL6|4Z=F-Jp{X`7 z_NZ^^yNXzMrzBxnO`}6#s8?Y5$T)65g#aao;@l>9R-fChaRW&AB95I~84&yiGUiLN z&z}xbD*QbAb8IxBSr2bx6!JrVGjt4YID#Le>E>YXh7o=C1m{H_9%TRxC$1g=7>g#G%wXzG_Ay`3octZ2B-FhUPly1Fd#AzRc&QlbP_%lJNg7+nr%a#VVIc)Nf z{}3%SSDs}@NCyP!+Vg8nYlX;MXydbZM-~3V(DY&-V>yY_L?2Hx3eMf{vG-rk8+!vp zsuQS-X{2Q_`^|$0?QF;QysdarHYd(w(odAM*)3kC_J`<-0BPO6k!wnG?kK~^mjhvK z44x!}{e6@lJWC=fQ08&IUl+u1eb^2O?qmG853?cPTQ6j#I0DJ%OfX~CvkEgP>}(E& zVSMQCwPj&q9ymx(t~P~{;0fRSH?>2Tv>OTV+9I|s?>E}QxZUcN!>LDSN-pEujiR6O zf?uG16vA9xh<)CV6HyE@%G(1Ds~Ff!$cb7rUltBaQ=Gx3k0;Ly!$G2l*c9Q~X0(AL zQ;#<;R4&nDH9r^2kV;Xyl=6g_JOo>r)h=^rL+$@T8b3`zDV#bMhn^2O+x2@BJ4h+aLt#kI#q~IyH=5s%@~Q*FV;>X-!x||( zi*Eg}wu_t#C#h5|>Dtr_L$GL$)(kq9l{K8Z`7^=|Zr9X+??Bo2_Prl7`%18VSIPMO z`gm*^-&5p8LF0U=>kwTpp3yl3m!+so=H5VXT}<27o2d;2_gAAISH3;i_SZqH;WVp* z8KTTk_+>5%^}rFqS1SdF?TA$_O|hjLozFf zbRRRr$Y*D{cCXk&Eo&ISi?`FeG-sb>^d*b6BAB#e^~oSTsHuUsf7D>tBC7vr1nVT7 zp90$o?O)X~x6C(%)L07S8m3$Sh7&B-*@6i6$3nF!v%`z;2#eW;MMRbj9#Nl|RuLqV zdNFm&7;^(tfLo>JSniJ(`m~3NQNP-o3RL$LdaW9=90ao$e!}KA|>hlN!NDz5)EE z3XlOOiG%K0i*k(I@svc04l0mg7s~+wxwWy|7OW&{PtMyq5Y5G}7Hl(0D%7_CDCuJ; z6I}1_GUUM9)kw^*`U*xW;*JP*%ZZJ>H5j&=?MztvjEg`D^sFIQZ9qX*y9bQt7ax2r zK_Y|eit&3jBEA3Dfss{mQ%6c*X{Bai4zeCmYR~Dz4_*|hP!Y7s(1C35FN>s=er*%u zg#zxP>(N{oxrMU~G5^J%edTyuj`lHED7IZIcld{DN-mEL+4l`7xvvd@q{!O*xg>@ zxfk8Cj^&=di?&3!am}vod$+A}D}{?bmgT;hlG+84kRn0*RhpCf|7Pej;YQ$XGC+G1 z?OMr+SU9Xbt#AX@+aw0Q+XDRK`WI4i<2;Svjf58v=`*Eq&@b&N z`%bJ3?7dbgtqdi4mNh^p0D{oqO*TGa zr^=+fnkHyHGMo@xr2LzTW%~a* zR>s2|1&=@Jl6vHMa*W5(nA-<1wI<1B5hnO;@4zTExV0Im#Y6Y4RYH&db#cb{%sa9p z-39Rt17;Ut;07s8AuH+_Yn#_65Y#Tbz9=#|ZRF!-lP?v>U+#5XE1%gu$*L9D$G$!D zJ^}7@xo{P2H0a80{me(_3?Jre1P~&l9T7`Z3OfMj#Ui$5DNL-3Q2{G?-Ni?!S8Ygl zx$qvr_Hmj>`QOau2cn!j=pfb;U_EBH>5^ljEk(z9Xy}!O9yo)eQwY^#k;@P{@nX&5 zs7qUOK(@lZVoBzgK*Zqj4XBW6*eUZ5F@xdzpje0s9%U=01UZ9hW-~*#hRQN?RjQ@i zIoaw(__mUR={qD9fHTwJ+;^xPuncJw6nh`cUdarmB zJG$m@A;ns~wIbok=?JJ2BOwt3;ksrDd0I{k%?G06c{?Tg7bHqoq@Zh51Gvtgv-7m* zfHPT|79bIv_OHde$?eAv)>>DQRl4iYS#d@s#$LKczV=HfvVb802@vxlB_Z4$289k+ ziHnn*z%eebf)D6Sba0!VpPt68K)D4hKA(kjRvx>+E&=U*H=bcf17ca5BMV=pHb(aG zR%+u*su4dHMBrWj%*64Ad|!$Su~hEt#t3ZAEt&0izz!7Iam+*;4&d z6hptm%d05f;Pyr^@s6&L8$BkJ?;q22Y0LU-v<5Lus^rUm*VD@ZMc!i0`+^nJ-NF%$ zzm^KziDswtx4Or>6G|B_fF+$sjD9?L79&D1n6Jfp&wn8otHP!<#oVn#wmL(xaO0RI zveF^L2hYsg)a4;^c{ZmFK`JCoWd8vAoH4wjdBgno*$vQeE{rOMacj&=K`|L#Hwd5- z@ko86#GSCF)Sa{EIC!cR*)B+5ySxW1>TUA(m;BNv4taT^ZK~= z126mP)q~~eY3#exVY=A$Hd_4MaCZ#opkIX5VR^7Db7a((BJOXgcc!1QHk3L%0Ggw{ z7whUaf(Sju;AX{cYJllj36L&T1``u*O6qYVAAyxlKX&yKc~RMq3MlHj z{L2^~v;cv@WaeGs0^~g7$y23HGo!r3f>^IU7_D9PqX0&0X+H@3EHUezL6N)d>hIW%BU*hZuqHY*_c(1YELB z-!ODknpo!a1m=~s$+z_DR|q~MJjt5UIi$2X5AIipX5Ey+1+pR1^1sQP{+Kog?;%O#c@-PCk(>660C1dDqsua<%RTh>RK*8vVi9 z@0>Qyn7lUEPF*gAZir68(Mx)SXJ13ntdr}_$pwtNKQsN#TuCp~x6g5`YcI+az~8X}UAAkOJaX2@a?}V~>Et8;cyp zv4eof=xq!=Ax$>Lr>K2!Gmm%WxSRTg!O<+x=btCGJFu!gCK;sM+V*!O{jD1exO-a$ zKW)S&oi_qTvn)MjdCVxwag6+zQ~dd9HedD(%i9~OASX)(7T&kY{!GY!&J$8)JpAry zx5XjKrjao?CE(B}90)85m?4JD5Evv8MN!Em)SA*kIi=I}Xf0iNWzL;Ts>#9FM%=8Y ze2CveH7;b6weY<}W@8X z)7}HSJbB#>BQ61x_kV$<5{xK0-YY+R@`gJGVu-Ju72$&0!Xc6g!5{SdJ{!SIm6@Z+ z0yG^w)MT1Xy3!zIB)5ba9s4i9&s7qA)~CzJ5u6R8xH#ybCOehWH#Ih2bSJtTmi&Yb z^CYZBHwp30ZrQE6PjoqZtOM?{%q5O%@R(W&4;wnIl%VC8}D ziPD;_>y2GDh-5ssF0|e6|Dt-C%Kx^lNpB*N+*s3MeqTg&DcQ@V*TC#04krVAZgJf< zE8CimoQT=1Txn6>P7H4a62YgV=y5o2C!z8}`JX2`p73YUKNXr98^?Xo!jMpGWL%^* z;#cHuR51vMcaQxS&S4pHe9dX^QkXoz0<+czZ!F2O29fLJyFcLZaqy`)pO~_q?|?)8 zK9ciwoqiv$)5aEy^r4DSpb*U9jfu}&SzFyT@n*Dg-70GHJIaS+Bl@4a`Y-sSToRoe zUfPK5z;jYlc$C_+2J1M9cit&brI)b`rGm`wy%FVuM&)@09}B(5HTo`0?NCc7>F-K2 zbQa(`shaR+!xIU zURV}p6N)Scaq&{i_*aV}+EeX@GDJEP*=EU$c- zR5omb^KDDZ2erhk+g2tnzeHOe)r1s(@YbYXn-H!qRbGDySEukkx>l4)&jLOcEqh2- z(;U{ifSU@s*Dwh9j58~g!lg&#kD-BEizi|D{K&zBR~xxEZ0;l#HU>z4jSl)1jcekL z$T071IhZz}ZI>1~JF1~96Y zvbSiTMN9sa9#An<^h+q}4dZLk7Gn6ifM}3H;w0K$l7Mhb`)bjf?5Lm8NoSwHz;F@4PizuFT9A90BwJtx6A5c z$nFqD8`Z}%n~P0oZSL7L%3{Us3_(z&h)gTCqm|IWFH@+}+G&}&s=gOJ;gEOC*kW2% zORVj4GU7((0y+4Md#>`+og`esiZW(drPMWy@4Tn42PVPX9r2LBWEUJ_uZ*xVOa`*> ze9umUJ`Ca@&P@|pRs7P6CKb|N5$BO0h|+T*%)QfqM?pQ9r+l#H9o2Ta2ok4+>W31> zuF3Wt<4{f3Qp7Dt)Dfnj1wap6Q{ zY9zcG(-fqd@6W?|Ct{X)ytDm<+KA=kpSq+Susc6g9>( zN*}k+Yal%YXXsW6ZH4?}T7%{4Ej3%+ZbjG}UG7-2i=dEdJzInROcbw2G*$@w?O7^C_2# z@%QQ{P|quJk1k&~`OvSqJnhqRe=SigSgHjhV7NC3bZ;HsiKto?LEk_=Sz+G>7iG0; zb@I_2Ge8?+#Yl8doC~I2l)b8WRK{LfxkCFM$inF}!%C>Y9{IY$VI(^p-r&!q8o1&Y zkx()U^Avc>Twk2XFWal}4aCbgQOIw zB6iNY4*K6m;!;bB`3J%pYD&xtLNtIa29Abne z(ih^I67(Uo+gW}#bk>m07+Il#3s*|;2AeuD=ndJDhBsPMfYm+hA6GF4j+3Kv;KGj4 zLHjP|I#X?G%U7x5>?z_9m(;C=C~CK%cyB4jV!Rg9hW%fLMLrtvPHEmsSd{YR?|d4e z@jRJ4P2U;|oM(5jxjVQx+Q;0UPSH@?&MpSH69*maQd+YuQk9>(wu}S}Yr6RTm1^tx zU+eG)1Ci8}lxCo5%|apjWqZDWp>*l&?+pm8sx;9J%;GQEhrbrw8!;dy3#_ukNJDXV zLyr6rr1cJ#Ha8CP2Js?LvBzEvy*ke;zoz$e?3M?4i;Gv#vD&gnaLK-(fGJ$kXv0K1 z%EwPcEaP11$-Ns^D{{wBg!In^AH~q|sSSV<0o% z3*3Zy8Z>BW5v@ZMtA~3-do^Q3Te4ZwOiyD`vX7LSNov0#=K-V+&1}$p9LNzo;l=^F zT&gGY4fl(r!jjTy)pN?amFxRgt1jJNWDm1VH9xnYN~2LJ{(+YR$~Q*MUd7h9DKNTi zAgrZY?%AfdzC(D(O0=4R7H4M5;nrHU)}Sb3&6#^yy@@wu;V_h5Au*1&>w(~hMGx_R z(P0(QN%jwBkcnae_7nWl#HQok#Tkm-NMYE!H!Oa=NJjfooJHnQTy7XOt46f4d!10k z4-n;b8Q)e-N2XX}tjX-1ui8oqD${?$!KH-5v9ZcjzfME0Xt2V_V0J3t(2V|#o5`PGE}-B6A_4L~)^w%lEj; zw}mecr4WD+}fSUjDUgu;Fl5@SdPzY;zST9(n`?Z?eWqj@Tp%rVeX7j(5z z(j--!Fh8SPhYR<>4>Gnf^+c01fBOAj@o8SDf6;=r5xvi+(Q^lP; z3kH-u=a>(9^sZMn`r~E*s`8y{KOo23e9mk7sPWT6d@yzrb*Z8m^(+^z^wVx_)Aex$ zHX2vlK4i)x~hNMzNY#zm-fX;t~g|T3mD#kF52z#p_cff&N zz_B+gANg$w*H$b26`OgePv2htSYV}$jq1SH=6wy)t+)mGIp$_{>xC@B1I8C!q=^`d zFu!ni{GCRrx@Hcppi0yyMsFA+#hjL@M(;oQtDAj1NaO6HkVoyV8jr`X>DJcZ=7`*m z(yclb*2GCSqzJbZJimXV!yixmX0|((?jE_?jY}uIf5A?)a3;YNIGMm~$O z?S><|)DazrPGajtaRSATswh$W46*9(61{>{h?XXE%seGD7V$nC?=wdto%m%Hmz6-s zeO%BSy2C6i9v(H_c{+FMV|5{LQGm~Y*PE&bqu=qpK{G>jy?*rn}rIViW2 zv(y=_7KYOD z+lmXZE+XCw&IdeH9h|BTh)Qs*ut$?$LZzM-a~3|-(Nwttp0dFjJ70qMN?P>T5I?}X z+r`(LG8Y4av~yiIjJ7X=o!*+)kEI{KQ$)Nk^(y|;MYcTTL8(ulqV+lSUc)pzF4NWJ zRb|}Ngi{A;Rmcv@-jHwyG+6qba=3;*MTgGXfLIcUe)>g*oyF`g{v+wd=Q^+-xrAZJ z^H5zXC;xidU!JE-^BSCY50)4SzW3jpt9$MABTwqnz9S>J#Cw}vuV?$yNWO&Z1!Gab zr&pseUb1Cj**>$zv$q{i=1uy~H&Qm8kOO*B3BnxKG*Q{$c_ z83<4cIUcA}%%>vlDMrKT^Km3=ta7>`V|u z$?li@M&^`mZJ`Y7YrEWj9@s568b+qp1eHr@OLywv9;J8z^gj_%Tx+0~M9nep-G$04 z5MoPsfJF;za_|Lewdr{}P3=E;o4|TtS?i~1q=rDiVN66Rx#f-2y-FzE z6^h@yA(xACq?+osmCg2cS2ct(g{O)K5)*`&f|(yBQ=u3f8MH^4oMKU@cFN&-Vl86L z5#?3VApp)bEt90WkXr0bTVC{G0Nw$M;Yr7DkbX7n;0Pz=xT5WT>8f{ccYk$L)7DI| z=*Lm`=ZZ^4eDe||w1$jUawM-dAu!a&LRid`lH4E-ol_|5Y58hkB}=K!n*At?@%_591BN-H~FE?y7Ig$>Ajhd#Yj^ry9+OxyftPi zx-+I_6wM`noES%*DdBPYZ{iR&PpCsz{DA7@xRpijF)h8uAC zNoS9SG-%f*#jU4IOGWTz52?!5pc@7+lriHc0E!HAW8Hp4Ei==AaHRz2+n0a1B1L)*ZzS~S@Uo}yeQsW>c8C%v z{7}a#1>-DY+@0MTSoQ;cIwe6EBr^#Zep* zF@YMemt+DH@MhGQ{5uG_Ch07#Nro9o10^ypTy~t0g9gPTb1QnA&9AoGXPD1Pyj5`( z{4dm3hgT{WjoCD?U)G9wN3p(w<(_+AI{=S4DqUQlmlXs}yWoRoCUCQxM^C8P zL}fqAu<4-y)tda3G^!f&%8*GP0!gO4TG6TmemN3{!FcJfWvQ@NpA0#?LTypDc}WSW z+7uL7JjbB1>K5-M|01W)FfNXxN1l(MFqSrr@;wY&kji8-Ml zuA5B0x0iZ0W%oy<-WQQkCahmdRv7lRp@HZB8L*>PmmcQN3!yVT47C?*Az^XZeo_2& z4A3E>o4WZdTwWxop~_xntj8d@-vac!NrvrxZJO~_o%^;j-TO*B);^E6E&r``T<$x& zk~|abHfrVE6xRiG&$p+ijyWDQR+;{#7CEsrc~1fq(jJ1Q-GFF!_r*nP5}qbNVr*~P z%@Q-|?*wY5M?QsLE%lUqfIa;ARqlQf^ZYK~En<1WXzx_I2ssU_&a4gPaXYB?zLGyJ zY7u|$?l+c%(348dgY6gnR z4-jlZ6-{dhoKnWO5Y36$k=xoj|C=Om?_E6>Hjw!DIMEk($#^V05lHbNqOWZV0&a2A zsU(28x~xxvgfE2Z03ucfunV`EqE0tj!h;ai%I-$5AF-5ej4$`v#H`S7wf^#^m~6<% zO4DMbf6iw#(5&Fy$4`7*LX z^YfML+`7m~bL$dhc^4Flfz~)KO4qbVqG3-nZN%%Rw9bKrWIVPW$saKhjc4~bLb1m@ zY>VS@YYrA_*;9&UBg=&_5sjxj32o~3>2ZpiL0EnV;AwrD9s+LREp($AVY43^*9H~q zL~mF0?I2mE<}1R5b#|)b6Lx~~*h2RJ6<99f)}aMInG(3GMG6>v z>{W#5EX`ciZ5|q;!C2rynMCBmZ&9$#(PvW%o91hZX}rj!bg*W-Ia+w5awA}8fesLa zjGK5FWDff++%~?oXDKZ$Wop$W;$oSYh-NvA$J=0PHTZ-QJ#pbIY|P}q{~MfVV`DK} zm(lfQ9&m@sv2)rx917mQ6U-BJE0gh{U9F%x&#+6mV5ul}tpc(U22{tiK}G}-s!Y-l zpG-V11ZVsc4vmc~*NcqL00{&M@W5>|aWkU{+yS@Kke*l92gd@e=eyjGnekP87&8k6 z?FqCFE^9N7)eZjTUx>Adt!t^PWj^r!=dKehw@Xn6W6`pAwF%qwOGD4sSyZ#;>+#Q#c@>fhFdJ(%NzM(m;{ct_5iV4^F%kVZrQZyxX zDr(dXNL%+?A1wXIEon%igR6#`uj{n-9Z*LD$pvKaCN*&+r@kd1!QDFfq^5d>p0cp_ z!;Cv%5@1qIyFOO^o`|*#Wio5-wbMD}jV>^weTENI1%Y9nsaqpHW~=LaCLDlZJSzk2 zYaUr=ypNobIMmQW9>_Ij=GafQL(7nS?+0j9Bq7#YdaPLuRCu5X1~Nv+%)(nk1av=c4Zwi17wiOC@%T0<+#Bke+ns=4yuSly z)R5#rmYo8ciURquLh2_*I789t0CDBRVnb#*XQ-GmU*hy9KT9f3uu z#Oy~tQ zKg*zEQ?mJsNETY0sf4RSQ-L7RnN_VUjt%Q44~OtU4}7#+fU4Qp0;W+?-?CDIs6+rg zoEmcYj+c@Xh>MK*eCrh*q2Xy()z6O0BbHTJqLCCd9)t2a^lunne7H7Rz}6*ZKkuf4 zejy+ds&4err&K8WOwJ5b-}*ctd~+PwmPS2Is}n(9*F^8P}{`1pczWjiI6nwbbP8Xd$cwD#Hx^TEDLtx4xNo>YiZ!U$$d@07( z-m_>q_vTDO-loVhUgi;YywL|l_O3}SaJcTGPvuQR{filunAI3)rNVoKGbEYw zyDT_lczF7qKDf|$H!5%9kJ!#I=gBn}(b=^iVC+I@iO13(si%6?f3rd=R+P3)j)rOaso9=WWa((e#_AwMcM`}tuZ!$Wy9-KV7?r#DR z>;tidg%28S$IN@@1wv;@bMI*8ei`MJ0dR35Z^<}2_Jq9n16@e2N_;J%z1y_O@vE>^ z5fJl#RHNYX4SoLYO~&1||6v~wrEgfEQ90+3^`ayJY?pWaU zy<#5W=iV}j5jTwYIP|mL8e{`Lw!sb{F;$Q+;_vj@Kx;wKf?^pXb0w(g)9mbJLq*o{E?mWZL>Fq)P&0*GN;F1abUFZdH#-O}GvAFewKQ|LwRQ;U}_ir^2uwQS<`ndN(S_OT^X|^LdcOgyE zt=nH%$HZ=vF|~sd?l()$o`n_N&wNA6q2f_g-%kD%QNxG$_gY=z7t)k`^+W|r|1Lud zioL3xyDQG{TpA`;KG=6wKbr2Y>2k-I%xw5${XGc;)X*P^Y;|q(xYv zJqd%~3pXl&+n2Oml67cgg(=P${C&&Gj@0}&SW`=#drv{cL}F$OcD4+1*FY`2XZ~Ok z@Jyy!<)+I0H~baUD~=A#e4#cMH+BOV-wRl9iY?^I0`NBS5MLwzqT@Zj5hLJc`d+O^Og{H&HjP?t3A#FO`TD{vHySqx)ce?iXmqv1|14ZK z&8#5e?m%G#eEFX|6v?3DCycO*wHDq1SNie>t2hdM+j=>KDHZo~0zXH)wv#$u)ZaQB z*qJ`o>ej`a_?b-A8D4&tduqH=V{deOLe*-nB0~XWJKxp@TvNff_bCt(k^IGGOO+M(g$M3iS;G>-&7IO^x3fH>a!1`h{SlcYL&v09nE@IOPC%- z^7~vGy0r}eWk?q;zfK4!mT`hP1}d&>hExl8`;lee(2WvH7S-Zo#R00*q{dF7YDS*+ zMi}vK8fzsn;ds>{h7-OwlA9}ZjRBm~8d#Z^TY2|$3>s*>rLaNDTz9mHbbljnpUw#2 zWMIzhZIs?M&3k9+`t0J~VOjE>cO`wRkG;o;&1})s$k(pQG*ZF%B0?5>(aAG5#3$9C{_tlI@j)QJF#qy?}g?=yJ%BAURU94nb1R;l2@GW zq!rhZjC^t|l*g%FI)WVgI;8Kq*LCN7r{HAQ7cr(cUZA*YkTTVcH~y8Kq`(UA?$n{N z^vy{#2`Ex+ISHq7?YgfY!0Futk&s;JyqBmdNHO_6lwy8`1uH-6f>1Vm2m?16j3YYh za2W1|sfIIYO;Wb^n9uB0ocxb0>2dF|M--$p>;)T9@2YXN0Iz&HQLqeBkajIVB}Mho z&fmhWOo&}grh94lY2aH~nNoMh6)_7waV`6FzZ8wVW-#OjmjKi#a8=UhB3!{iyp<=l zkFjDJYC9Y=;|0(C=?MSC*Cwia&k~uW>LuZ1s}&;d2vrioKt_~yio_P~;fmVWGX)*o_#KWZbiWv%;=vTy+IB<5yKPAoC%-`6S(7ymoHT+Kb+ghoYBrUfb+ znakr!eeO&YeFHyP{LUZZcZ=~!u$?GtHqdTUY<4scMJ~K0JBVWE^dgKTKjg~`co;A^ zl@h@>=W?--hot9(1diCT*HJk#K3cQ$Ro?rDF(vZXbP!bNRW|#sTfdCWOPz#}oXAv2 zOCHJQdxC6H2Mu}uqvKD7UX+_~HLaZ(pPKi`^(Ji{_e_Lb%sb;z^28i0Lr2H9+KKT2 z3ti_OeS|cg!Ur?HR~}KD<#6Sd#b{r?SH8TlTQOw~u(h)w4OM^5b6x&3QrW!I>SkZR z(dCxokaYrfe9t;K%ea z+nOZoi*oh*a%XEEgT5Em6J5okC18B{d>sK7Z|tW+_g&mhV8d#KTrA3`PUAbK^}We% z;pdEh|DDZd@i1EtS$S*{#khzUXdCe%6G2+R_cc${2I%#=tvHF};)@pYld?FN}_a>ivzeu6=0*au|;y1)BlHq2o zu3O_kVSCwiTGE#(* z>Xn|1emocYB}$YzIhnm7CO*3`@_7$JQFLsj51MF zj?U>?Zo7fD$@pjae_n7#zABCzb-J(}@K?LXn<(wbHv+y`;UA!)${)R%(B}y&<~3*V z=eC`zoYvwSXPjSMoPY@J`lvzz<|pm$9Fu;Em+{{_F#(zz_BiDAvj>-jZs!pEeyP1= zO9b{61$~0jNI$V9E3b;rE-v6S_*PJFla1UHAE?uhb00Xx|CNR9Dkd*bNV+t6fQuu4 z(n5bXXGX`H;ZPJ#6Y1R9EL+i|s*x^K5D|8fkmgAZz{45QJ;o()ble);9ws3}(c4p} z6dLDo)^(U@N@ctOhd-}TgsiQnw3OuLGk+kLrrUY!wPzk6mMF+eziHrh4tM_ksG2%< z4tz8%dj@4~zlq@7no5wSVu6CNFT9c=_Ad9dBhk!}M9 literal 0 HcmV?d00001 diff --git a/tools/rotate-live-exposed-secret.sh b/tools/rotate-live-exposed-secret.sh new file mode 100755 index 0000000..46a948f --- /dev/null +++ b/tools/rotate-live-exposed-secret.sh @@ -0,0 +1,373 @@ +#!/usr/bin/env bash +# Rotate every credential class formerly embedded in sso/keycape-config without +# rendering a Secret value. This is an emergency, deliberate-invalidation path. + +set -euo pipefail + +MODE="${1:-}" +if [[ "$MODE" != "--preflight" && "$MODE" != "--execute" ]]; then + echo "usage: $0 --preflight|--execute" >&2 + exit 2 +fi +if [[ "$MODE" == "--execute" && "${KEYCAPE_ACKNOWLEDGE_SESSION_INVALIDATION:-}" != "1" ]]; then + echo "refusing execution without KEYCAPE_ACKNOWLEDGE_SESSION_INVALIDATION=1" >&2 + exit 2 +fi + +for tool in age base64 curl jq kubectl openssl python3 sha256sum tar; do + command -v "$tool" >/dev/null || { + echo "missing required tool: $tool" >&2 + exit 1 + } +done +python3 -c 'import bcrypt' 2>/dev/null || { + echo "missing required Python bcrypt module" >&2 + exit 1 +} + +ROOT_DIR="$(cd "$(dirname "$0")/.." && pwd)" +RECIPIENT_FILE="${NETKINGDOM_AGE_RECIPIENT_FILE:-/home/worsch/net-kingdom/keys/age.pub}" +ARCHIVE_FILE="${KEYCAPE_RECOVERY_ARCHIVE:-$ROOT_DIR/history/KEY-WP-0011-rotated-credentials-2026-08-23.tar.age}" + +[[ -s "$RECIPIENT_FILE" ]] || { + echo "age recipient file not found: $RECIPIENT_FILE" >&2 + exit 1 +} + +required_secrets=( + "sso/keycape-config" + "sso/keycape-pi-token" + "sso/authelia-secrets" + "sso/lldap-secrets" + "mfa/privacyidea-config" +) +required_deployments=( + "sso/keycape" + "sso/authelia" + "sso/lldap" + "mfa/privacyidea" +) + +for ref in "${required_secrets[@]}"; do + namespace="${ref%%/*}" + name="${ref#*/}" + kubectl get secret "$name" -n "$namespace" -o name >/dev/null +done +for ref in "${required_deployments[@]}"; do + namespace="${ref%%/*}" + name="${ref#*/}" + kubectl get deployment "$name" -n "$namespace" -o name >/dev/null +done + +jwks_fingerprint() { + jwks_uri="$(curl -fsS https://kc.coulomb.social/.well-known/openid-configuration | jq -er '.jwks_uri')" + curl -fsS "$jwks_uri" | + jq -cS '.keys | map({alg,e,kid,kty,n,use})' | + sha256sum | + awk '{print $1}' +} + +before_jwks="$(jwks_fingerprint)" +before_keycape_rv="$(kubectl get secret keycape-config -n sso -o jsonpath='{.metadata.resourceVersion}')" +before_pi_rv="$(kubectl get secret privacyidea-config -n mfa -o jsonpath='{.metadata.resourceVersion}')" +before_lldap_rv="$(kubectl get secret lldap-secrets -n sso -o jsonpath='{.metadata.resourceVersion}')" +before_authelia_rv="$(kubectl get secret authelia-secrets -n sso -o jsonpath='{.metadata.resourceVersion}')" + +echo "preflight: deployments and Secrets exist" +echo "preflight: current public JWKS fingerprint: $before_jwks" +echo "preflight: current resource versions: keycape=$before_keycape_rv privacyidea=$before_pi_rv lldap=$before_lldap_rv authelia=$before_authelia_rv" + +if [[ "$MODE" == "--preflight" ]]; then + exit 0 +fi +if [[ -e "$ARCHIVE_FILE" ]]; then + echo "refusing to overwrite recovery archive: $ARCHIVE_FILE" >&2 + exit 1 +fi + +umask 077 +work_dir="$(mktemp -d /tmp/keycape-recovery.XXXXXX)" +port_forward_pids=() +lldap_force_reset_active=false + +cleanup() { + status=$? + for pid in "${port_forward_pids[@]:-}"; do + kill "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true + done + if [[ "$lldap_force_reset_active" == "true" ]]; then + kubectl set env deployment/lldap -n sso LLDAP_FORCE_LDAP_USER_PASS_RESET- >/dev/null 2>&1 || true + fi + case "$work_dir" in + /tmp/keycape-recovery.*) + find "$work_dir" -type f -exec shred -u -- {} + 2>/dev/null || true + rm -rf -- "$work_dir" + ;; + esac + exit "$status" +} +trap cleanup EXIT INT TERM + +openssl rand -hex 32 >"$work_dir/lldap-bind-password" +openssl rand -hex 32 >"$work_dir/authelia-keycape-client-secret" +openssl rand -hex 32 >"$work_dir/privacyidea-secret-key" +openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out "$work_dir/key.pem" 2>/dev/null + +python3 -c ' +import bcrypt +import sys +secret = sys.stdin.buffer.read().rstrip(b"\n") +sys.stdout.buffer.write(bcrypt.hashpw(secret, bcrypt.gensalt(rounds=12))) +' <"$work_dir/authelia-keycape-client-secret" >"$work_dir/authelia-keycape-client-secret.bcrypt" + +lldap_password="$(tr -d '\n' <"$work_dir/lldap-bind-password")" +authelia_secret="$(tr -d '\n' <"$work_dir/authelia-keycape-client-secret")" + +cat >"$work_dir/config.yaml" <"$work_dir/README.txt" +tar -C "$work_dir" -cf "$work_dir/recovery.tar" README.txt config.yaml key.pem lldap-bind-password authelia-keycape-client-secret privacyidea-secret-key +age -R "$RECIPIENT_FILE" -o "$ARCHIVE_FILE" "$work_dir/recovery.tar" +chmod 600 "$ARCHIVE_FILE" +echo "custody: encrypted recovery overlay created at $ARCHIVE_FILE" + +patch_field() { + namespace="$1" + secret_name="$2" + field_name="$3" + value_file="$4" + patch_file="$work_dir/patch-${namespace}-${secret_name}-${field_name}.json" + encoded="$(base64 -w0 <"$value_file")" + printf '{"data":{"%s":"%s"}}' "$field_name" "$encoded" >"$patch_file" + unset encoded + kubectl patch secret "$secret_name" -n "$namespace" --type=merge --patch-file "$patch_file" >/dev/null + shred -u -- "$patch_file" +} + +echo "rotation: invalidating the exposed privacyIDEA admin JWT" +patch_field mfa privacyidea-config PI_SECRET_KEY "$work_dir/privacyidea-secret-key" +printf '%s' 'ROTATED_DISABLED_ADMIN_TOKEN' >"$work_dir/pi-token-disabled" +patch_field sso keycape-pi-token token "$work_dir/pi-token-disabled" +kubectl rollout restart deployment/privacyidea -n mfa >/dev/null +kubectl rollout status deployment/privacyidea -n mfa --timeout=180s + +echo "rotation: resetting the LLDAP admin/bind credential" +patch_field sso lldap-secrets LLDAP_LDAP_USER_PASS "$work_dir/lldap-bind-password" +kubectl set env deployment/lldap -n sso LLDAP_FORCE_LDAP_USER_PASS_RESET=always >/dev/null +lldap_force_reset_active=true +kubectl rollout status deployment/lldap -n sso --timeout=180s + +kubectl port-forward -n sso service/lldap 11717:17170 >"$work_dir/lldap-port-forward.log" 2>&1 & +port_forward_pids+=("$!") +for _ in {1..30}; do + curl -fsS -o /dev/null http://127.0.0.1:11717/health >/dev/null 2>&1 && break + sleep 1 +done +printf '{"username":"admin","password":"%s"}' "$(tr -d '\n' <"$work_dir/lldap-bind-password")" >"$work_dir/lldap-login.json" +lldap_status="$(curl -sS -o /dev/null -w '%{http_code}' -H 'Content-Type: application/json' --data-binary @"$work_dir/lldap-login.json" http://127.0.0.1:11717/auth/simple/login)" +[[ "$lldap_status" == "200" ]] || { + echo "LLDAP new-credential verification failed with HTTP $lldap_status" >&2 + exit 1 +} +echo "verification: LLDAP accepted the replacement bind credential" +kill "${port_forward_pids[-1]}" 2>/dev/null || true +wait "${port_forward_pids[-1]}" 2>/dev/null || true +unset 'port_forward_pids[-1]' + +kubectl set env deployment/lldap -n sso LLDAP_FORCE_LDAP_USER_PASS_RESET- >/dev/null +lldap_force_reset_active=false +kubectl rollout status deployment/lldap -n sso --timeout=180s + +echo "rotation: updating Authelia and KeyCape consumers" +authelia_patch="$work_dir/patch-authelia.json" +ldap_b64="$(base64 -w0 <"$work_dir/lldap-bind-password")" +client_hash_b64="$(base64 -w0 <"$work_dir/authelia-keycape-client-secret.bcrypt")" +printf '{"data":{"ldap_password":"%s","keycape_client_secret_hash":"%s"}}' "$ldap_b64" "$client_hash_b64" >"$authelia_patch" +unset ldap_b64 client_hash_b64 +kubectl patch secret authelia-secrets -n sso --type=merge --patch-file "$authelia_patch" >/dev/null +shred -u -- "$authelia_patch" + +# The current Authelia deployment injects most secret fields from files, but +# its OIDC client verifier is still code-defined in authelia-config. Keep the +# Secret copy for the intended file-backed migration and update the active +# ConfigMap atomically for the live 4.38 deployment. +kubectl get configmap authelia-config -n sso -o jsonpath='{.data.configuration\.yml}' >"$work_dir/authelia-configuration.yml" +AUTHELIA_CLIENT_HASH="$(tr -d '\n' <"$work_dir/authelia-keycape-client-secret.bcrypt")" python3 -c ' +import os +import re +import sys +source = sys.stdin.read() +updated, count = re.subn( + r"(?m)^(\s+secret:\s*).*$", + lambda match: match.group(1) + chr(34) + os.environ["AUTHELIA_CLIENT_HASH"] + chr(34), + source, +) +if count != 1: + raise SystemExit(f"expected one Authelia OIDC client secret field, found {count}") +sys.stdout.write(updated) +' <"$work_dir/authelia-configuration.yml" >"$work_dir/authelia-configuration.updated.yml" +kubectl create configmap authelia-config -n sso --from-file=configuration.yml="$work_dir/authelia-configuration.updated.yml" --dry-run=client -o yaml | + kubectl apply -f - >/dev/null + +kubectl create secret generic keycape-config -n sso --from-file=config.yaml="$work_dir/config.yaml" --from-file=key.pem="$work_dir/key.pem" --dry-run=client -o yaml | + kubectl apply -f - >/dev/null + +kubectl rollout restart deployment/authelia deployment/keycape -n sso >/dev/null +kubectl rollout status deployment/authelia -n sso --timeout=180s +kubectl rollout status deployment/keycape -n sso --timeout=180s + +echo "verification: checking replacement Authelia client credential" +kubectl port-forward -n sso service/authelia 19091:9091 >"$work_dir/authelia-port-forward.log" 2>&1 & +port_forward_pids+=("$!") +for _ in {1..30}; do + curl -fsS -o /dev/null http://127.0.0.1:19091/api/health >/dev/null 2>&1 && break + sleep 1 +done +basic="$(printf 'keycape:%s' "$(tr -d '\n' <"$work_dir/authelia-keycape-client-secret")" | base64 -w0)" +printf 'Authorization: Basic %s\nContent-Type: application/x-www-form-urlencoded\n' "$basic" >"$work_dir/authelia-new.headers" +unset basic +printf '%s' 'grant_type=authorization_code&code=definitely-invalid&redirect_uri=https%3A%2F%2Fkc.coulomb.social%2Fauthorize%2Fcallback&client_id=keycape' >"$work_dir/authelia-token.body" +new_client_status="$(curl -sS -o "$work_dir/authelia-new.response" -w '%{http_code}' -H @"$work_dir/authelia-new.headers" --data-binary @"$work_dir/authelia-token.body" http://127.0.0.1:19091/api/oidc/token)" +new_client_error="$(jq -r '.error // empty' "$work_dir/authelia-new.response" 2>/dev/null || true)" +printf 'Authorization: Basic a2V5Y2FwZTp3cm9uZy1yZWNvdmVyeS1wcm9iZQ==\nContent-Type: application/x-www-form-urlencoded\n' >"$work_dir/authelia-wrong.headers" +wrong_client_status="$(curl -sS -o "$work_dir/authelia-wrong.response" -w '%{http_code}' -H @"$work_dir/authelia-wrong.headers" --data-binary @"$work_dir/authelia-token.body" http://127.0.0.1:19091/api/oidc/token)" +wrong_client_error="$(jq -r '.error // empty' "$work_dir/authelia-wrong.response" 2>/dev/null || true)" +if [[ "$new_client_error" == "invalid_client" || "$new_client_status" == "401" ]]; then + echo "replacement Authelia client credential was rejected" >&2 + exit 1 +fi +if [[ "$wrong_client_error" != "invalid_client" && "$wrong_client_status" != "401" ]]; then + echo "Authelia wrong-client-secret negative check did not reject authentication" >&2 + exit 1 +fi +echo "verification: Authelia accepted the replacement client and rejected a wrong secret" +kill "${port_forward_pids[-1]}" 2>/dev/null || true +wait "${port_forward_pids[-1]}" 2>/dev/null || true +unset 'port_forward_pids[-1]' + +echo "verification: checking fail-closed privacyIDEA behavior" +kubectl port-forward -n mfa service/privacyidea 18081:8080 >"$work_dir/privacyidea-port-forward.log" 2>&1 & +port_forward_pids+=("$!") +for _ in {1..30}; do + curl -sS -o /dev/null http://127.0.0.1:18081/ >/dev/null 2>&1 && break + sleep 1 +done +printf 'Authorization: Bearer ROTATED_DISABLED_ADMIN_TOKEN\nContent-Type: application/x-www-form-urlencoded\n' >"$work_dir/privacyidea.headers" +printf '%s' 'user=__keycape_recovery_probe__&pass=000000&realm=coulomb' >"$work_dir/privacyidea.body" +pi_status="$(curl -sS -o "$work_dir/privacyidea.response" -w '%{http_code}' -H @"$work_dir/privacyidea.headers" --data-binary @"$work_dir/privacyidea.body" http://127.0.0.1:18081/validate/check)" +pi_result="$(jq -r 'if .result | has("value") then (.result.value | tostring) else empty end' "$work_dir/privacyidea.response" 2>/dev/null || true)" +pi_api_status="$(jq -r 'if .result | has("status") then (.result.status | tostring) else empty end' "$work_dir/privacyidea.response" 2>/dev/null || true)" +if [[ "$pi_result" == "true" ]] || + ! { [[ "$pi_status" == "200" && "$pi_result" == "false" ]] || + [[ ( "$pi_status" == "400" || "$pi_status" == "401" ) && "$pi_api_status" == "false" ]]; }; then + echo "privacyIDEA fail-closed probe failed (HTTP $pi_status, result=$pi_result)" >&2 + exit 1 +fi +echo "verification: privacyIDEA rejected the invalid MFA proof" +kill "${port_forward_pids[-1]}" 2>/dev/null || true +wait "${port_forward_pids[-1]}" 2>/dev/null || true +unset 'port_forward_pids[-1]' + +issuer="$(curl -fsS https://kc.coulomb.social/.well-known/openid-configuration | jq -r '.issuer')" +[[ "$issuer" == "https://kc.coulomb.social" ]] || { + echo "unexpected live issuer: $issuer" >&2 + exit 1 +} +authorize_status="$(curl -sS -o /dev/null -w '%{http_code}' 'https://kc.coulomb.social/authorize?client_id=openbao-admin&redirect_uri=http%3A%2F%2F127.0.0.1%3A18200%2Fui%2Fvault%2Fauth%2Fnetkingdom%2Foidc%2Fcallback&response_type=code&scope=openid%20profile%20email%20groups&state=keycape-recovery&code_challenge=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA&code_challenge_method=S256')" +[[ "$authorize_status" == "302" ]] || { + echo "OpenBao authorization handoff returned HTTP $authorize_status" >&2 + exit 1 +} + +after_jwks="$(jwks_fingerprint)" +after_keycape_rv="$(kubectl get secret keycape-config -n sso -o jsonpath='{.metadata.resourceVersion}')" +after_pi_rv="$(kubectl get secret privacyidea-config -n mfa -o jsonpath='{.metadata.resourceVersion}')" +after_lldap_rv="$(kubectl get secret lldap-secrets -n sso -o jsonpath='{.metadata.resourceVersion}')" +after_authelia_rv="$(kubectl get secret authelia-secrets -n sso -o jsonpath='{.metadata.resourceVersion}')" + +[[ "$after_jwks" != "$before_jwks" ]] || { + echo "public JWKS fingerprint did not change" >&2 + exit 1 +} +[[ "$after_keycape_rv" != "$before_keycape_rv" && + "$after_pi_rv" != "$before_pi_rv" && + "$after_lldap_rv" != "$before_lldap_rv" && + "$after_authelia_rv" != "$before_authelia_rv" ]] || { + echo "one or more Secret resource versions did not change" >&2 + exit 1 +} + +echo "verification: discovery, OpenBao callback admission, readiness, and JWKS rollover passed" +echo "verification: new public JWKS fingerprint: $after_jwks" +echo "verification: new resource versions: keycape=$after_keycape_rv privacyidea=$after_pi_rv lldap=$after_lldap_rv authelia=$after_authelia_rv" +echo "recovery complete" diff --git a/workplans/KEY-WP-0011-live-secret-exposure-recovery.md b/workplans/KEY-WP-0011-live-secret-exposure-recovery.md index bb51810..f7b906f 100644 --- a/workplans/KEY-WP-0011-live-secret-exposure-recovery.md +++ b/workplans/KEY-WP-0011-live-secret-exposure-recovery.md @@ -4,7 +4,7 @@ type: workplan title: "Recover from live KeyCape Secret output exposure" domain: infotech repo: key-cape -status: blocked +status: finished owner: codex topic_slug: netkingdom created: "2026-08-23" @@ -38,29 +38,31 @@ mutation was attempted. ```task id: KEY-WP-0011-T02 -status: wait +status: done priority: high state_hub_task_id: "9d60569c-a0f1-53a9-b4dc-01687ab92bcc" ``` -Requires an approved owner-coordinated rotation plan. KeyCape signing-key -rollover must account for JWT/JWKS continuity or deliberate invalidation; -backend credential rotations must update each issuing system and every -consumer atomically enough to avoid an authentication outage. Secret values -must use approved custody transports and never enter Git, State Hub, chat, -arguments, or ordinary logs. +Approved by the operator on 2026-08-23, accepting deliberate invalidation of +active sessions and a brief SSO interruption. The RS256 key, LLDAP bind/admin +credential, Authelia-KeyCape client credential, and privacyIDEA application +signing secret were replaced without rendering values. The exposed +privacyIDEA admin JWT was removed from live use. Replacement material is +recoverable only through the NetKingdom age recipient in the encrypted +recovery overlay under `history/`. ## Verify recovery without values ```task id: KEY-WP-0011-T03 -status: wait +status: done priority: high state_hub_task_id: "a079573d-97fd-5c6c-b1f9-c0af65e2933a" ``` -After rotation, verify new resource versions and non-secret fingerprints, -KeyCape readiness/discovery/JWKS, LLDAP lookup, Authelia authorization handoff, -privacyIDEA fail-closed MFA, and invalidation of the superseded credentials. -Verification must not render any Kubernetes Secret data or authentication -value. +Verified changed resource versions and public JWKS fingerprints, four Ready +deployments, successful replacement LLDAP and Authelia authentication, +wrong-secret rejection, explicit privacyIDEA fail-closed denial, issuer +discovery, and the OpenBao callback handoff to Authelia. The temporary LLDAP +force-reset flag was removed. No Kubernetes Secret payload or authentication +value was rendered during recovery.