key-cape/history/KEY-WP-0011-live-secret-exposure-recovery.md

# KEY-WP-0011 — Live Secret exposure recovery evidence

Date: 2026-08-23

## Outcome

The accidental rendering of the `sso/keycape-config` Secret was treated as a
live credential exposure. With explicit operator approval for session
invalidation and brief SSO disruption, recovery replaced:

- the KeyCape RS256 signing key;
- the shared LLDAP admin/bind credential used by LLDAP, Authelia, and KeyCape;
- the Authelia-KeyCape OIDC client credential and verifier; and
- the privacyIDEA application signing secret, invalidating the exposed admin
  JWT.

KeyCape now carries the non-credential marker
`ROTATED_DISABLED_ADMIN_TOKEN` instead of a privacyIDEA admin JWT. The current
client set has no enrollment handoff URL, and `requireForAll: true` keeps the
normal MFA path fail-closed. A durable, least-privilege privacyIDEA service
credential remains a follow-up architecture gap.

## Custody

The authoritative replacement material is encrypted to the NetKingdom age
recipient in
`KEY-WP-0011-rotated-credentials-2026-08-23-final.tar.age` (mode `0600`).
It contains no pre-exposure value. The temporary first-cutover archive was
securely removed after its contents were superseded by the final rotation.
The final archive was not decrypted during verification.

## Non-secret verification evidence

- Pre-recovery public JWKS fingerprint:
  `4172efd1109e2a07b728c59b28558bf6e0171b22afb95d1692f87fccf284039b`
- First-cutover public JWKS fingerprint:
  `8ec2651168468caa595f185bb7021cfdec228eb4992f1f6c1521c6e62762c73e`
- Final public JWKS fingerprint:
  `c6faac5dfeef2453daf9cfc14671f62b535dd321befdf6014e3ee5c2cf1f1156`
- Final Secret resource versions:
  - `sso/keycape-config`: `51346058`
  - `sso/keycape-pi-token`: `51344120`
  - `sso/authelia-secrets`: `51346049`
  - `sso/lldap-secrets`: `51345775`
  - `mfa/privacyidea-config`: `51345601`
- KeyCape, Authelia, LLDAP, privacyIDEA, and identity-provisioner each reported
  one Ready, updated, available replica.
- LLDAP accepted the replacement credential before and after removal of
  `LLDAP_FORCE_LDAP_USER_PASS_RESET`; the final deployment has zero copies of
  that emergency setting.
- Authelia accepted the replacement KeyCape client credential and rejected a
  deliberately wrong credential. Its active ConfigMap verifier matches the
  copy held in `authelia-secrets`.
- privacyIDEA rejected the synthetic invalid MFA proof with HTTP 400,
  `result.status=false`, and error code 907.
- OIDC discovery returned issuer `https://kc.coulomb.social`.
- The registered OpenBao local callback returned HTTP 302 to
  `auth.coulomb.social`.

## Follow-up disposition

NetKingdom resolved the manifest mismatch in revision `c956ceb`: Authelia's
OIDC client list now uses the supported template filter to read the verifier
from the Secret-mounted file. The revision was applied live and Authelia
returned Ready with startup complete.

The credential-routing catalog also has no concrete native rotation entries
for these legacy KeyCape backend credentials; generic and interactive-login
routes were insufficient for incident execution.

The remaining cross-repository follow-up is the privacyIDEA
`lldap-coulomb` resolver. It needs an attended provider-admin operation to
replace its persisted LLDAP bind password and prove a user lookup; no
unattended provider-admin credential was retained after the exposed JWT was
disabled.