security: rotate exposed KeyCape credentials

Assistant: codex
Assistant-Model: gpt-5.6-sol
Assistant-Session: 01a02e3f-7301-7622-9be1-12e5f352881c

history/KEY-WP-0011-live-secret-exposure-recovery.md

@@ -0,0 +1,71 @@
+# KEY-WP-0011 — Live Secret exposure recovery evidence
+
+Date: 2026-08-23
+
+## Outcome
+
+The accidental rendering of the `sso/keycape-config` Secret was treated as a
+live credential exposure. With explicit operator approval for session
+invalidation and brief SSO disruption, recovery replaced:
+
+- the KeyCape RS256 signing key;
+- the shared LLDAP admin/bind credential used by LLDAP, Authelia, and KeyCape;
+- the Authelia-KeyCape OIDC client credential and verifier; and
+- the privacyIDEA application signing secret, invalidating the exposed admin
+  JWT.
+
+KeyCape now carries the non-credential marker
+`ROTATED_DISABLED_ADMIN_TOKEN` instead of a privacyIDEA admin JWT. The current
+client set has no enrollment handoff URL, and `requireForAll: true` keeps the
+normal MFA path fail-closed. A durable, least-privilege privacyIDEA service
+credential remains a follow-up architecture gap.
+
+## Custody
+
+The authoritative replacement material is encrypted to the NetKingdom age
+recipient in
+`KEY-WP-0011-rotated-credentials-2026-08-23-final.tar.age` (mode `0600`).
+It contains no pre-exposure value. The temporary first-cutover archive was
+securely removed after its contents were superseded by the final rotation.
+The final archive was not decrypted during verification.
+
+## Non-secret verification evidence
+
+- Pre-recovery public JWKS fingerprint:
+  `4172efd1109e2a07b728c59b28558bf6e0171b22afb95d1692f87fccf284039b`
+- First-cutover public JWKS fingerprint:
+  `8ec2651168468caa595f185bb7021cfdec228eb4992f1f6c1521c6e62762c73e`
+- Final public JWKS fingerprint:
+  `c6faac5dfeef2453daf9cfc14671f62b535dd321befdf6014e3ee5c2cf1f1156`
+- Final Secret resource versions:
+  - `sso/keycape-config`: `51346058`
+  - `sso/keycape-pi-token`: `51344120`
+  - `sso/authelia-secrets`: `51346049`
+  - `sso/lldap-secrets`: `51345775`
+  - `mfa/privacyidea-config`: `51345601`
+- KeyCape, Authelia, LLDAP, and privacyIDEA each reported one Ready, updated,
+  available replica.
+- LLDAP accepted the replacement credential before and after removal of
+  `LLDAP_FORCE_LDAP_USER_PASS_RESET`; the final deployment has zero copies of
+  that emergency setting.
+- Authelia accepted the replacement KeyCape client credential and rejected a
+  deliberately wrong credential. Its active ConfigMap verifier matches the
+  copy held in `authelia-secrets`.
+- privacyIDEA rejected the synthetic invalid MFA proof with HTTP 400,
+  `result.status=false`, and error code 907.
+- OIDC discovery returned issuer `https://kc.coulomb.social`.
+- The registered OpenBao local callback returned HTTP 302 to
+  `auth.coulomb.social`.
+
+## Discovered operational gap
+
+The NetKingdom manifest currently code-defines the Authelia client verifier in
+`authelia-config`, while its legacy rotation script updates only
+`authelia-secrets/keycape_client_secret_hash`. The live recovery synchronized
+both sources. NetKingdom should migrate the client verifier to an actually
+consumed secret-file setting or update its rotation procedure so a later
+manifest apply cannot restore a stale verifier.
+
+The credential-routing catalog also has no concrete native rotation entries
+for these legacy KeyCape backend credentials; generic and interactive-login
+routes were insufficient for incident execution.