railiance-platform/docs/keycape-live-secret-exposure-recovery.md

KeyCape live Secret exposure recovery

Incident: KEYCAPE-EXPOSURE-20260823-01

This is the value-safe recovery contract for the accidental rendering of the base64 data map of Kubernetes Secret sso/keycape-config. Treat every value in that map as exposed even though no value was decoded or repeated. This document authorizes planning and metadata-only checks; it is not a live rotation GO.

Affected classes

• KeyCape RS256 signing private key (key.pem).
• LLDAP bind/admin credential embedded in config.yaml.
• Authelia confidential-client credential embedded in config.yaml and its provider-side hash.
• privacyIDEA admin token embedded in config.yaml and held by its provider.

The Secret also carries non-secret client registrations. Those registrations must come from one reviewed source revision during reconstruction; replaying a stale bootstrap bundle can silently remove a new client or reintroduce an already replaced credential.

Containment rules

• Never use kubectl get secret ... -o yaml, -o json, .data, stringData, describe, or a command that can render the Secret payload.
• Never run NetKingdom's legacy creds-rotate.sh; it prints generated values.
• Never put plaintext, base64 payloads, hashes derived from low-entropy passwords, private-key PEM, tokens, or complete Secret manifests in Git, State Hub, chat, command arguments, ordinary logs, or captured agent output.
• Create a private workspace with mktemp -d, mode 0700; keep each value in a separate mode-0600 file and pass it to tools by file or protected stdin. Install an idempotent cleanup trap before generating anything.
• One attended driver owns all writes. A second named operator owns abort. No background job or independent per-credential helper may write during the window.
• The recovery is forward-only. The exposed bundle is not a rollback artifact and must never be restored.

Safe evidence is limited to identifiers, Kubernetes UIDs/resource versions, Git revisions, public-key/JWKS digests, credential fingerprints that cannot be used for authentication, timestamps, status codes, boolean pass/fail results, and cleanup outcomes.

Current metadata anchor

Observed value-safely on 2026-08-23:

• cluster context: default;
• Secret: sso/keycape-config;
• Secret UID: 2e94519d-1550-41c7-9701-2efe47fe1fd3;
• Secret resource version: 51326531;
• KeyCape Deployment generation/observed generation: 25/25;
• KeyCape image: key-cape:mfa-client-20260809;
• KeyCape Ready/Available: 1/1;
• discovery issuer: https://kc.coulomb.social;
• public JWKS kid: key-1;
• public JWKS SHA-256: 8e3237da6030c6af91c5005d0112a149204bc549f67d2234939c5581fdb95a32.

Recheck these fields before a window using metadata-only columns and public OIDC endpoints. A changed UID, resource version, source revision, image, or JWKS digest invalidates the prepared approval receipt and requires review.

Owner receipt update (2026-08-23)

KeyCape reports that its emergency signing-key rotation completed with deliberate JWT invalidation; consumers must refresh JWKS/re-authenticate. The receipt contains no secret values, but it does not yet identify the reviewed non-secret client-config revision or the post-rotation public JWKS digest. NetKingdom published the value-safe dependency and provider sequence at c24d67b, noting that the LLDAP admin account is persistent and that the privacyIDEA admin token is an expiring session JWT without individual revocation. Use expiry-based predecessor denial unless a separate global signing-secret invalidation is explicitly approved. These acknowledgements do not constitute a live GO; the approval template remains pending.

A fresh public JWKS read on 2026-08-23 found kid key-1 and SHA-256 3d46c07b649432eb41112b9e5f5a929460027766127d1eb419ebac8eb9858c06. Treat this as an observation only until KeyCape binds it to the source revision and confirms downstream cache refresh.

KeyCape subsequently supplied source revision 93704fd2424503007c20b458b62a7f7d994bb288, final public JWKS kid key-1, and SHA-256 c6faac5dfeef2453daf9cfc14671f62b535dd321befdf6014e3ee5c2cf1f1156. KeyCape, Authelia, LLDAP, privacyIDEA, and identity-provisioner were reported Ready 1/1, with replacement/negative checks recorded. The persistent privacyIDEA lldap-coulomb resolver remains an attended provider-admin follow-up and is explicitly not complete; do not declare the incident closed or perform further bundle mutation until that owner action is separately authorized and evidenced.

Ownership

Boundary	Owner	Required contribution
Bundle custody and guarded apply	railiance-platform	Private workspace, exact Secret UID/resource-version guard, single bundle apply, fingerprints, cleanup receipt.
KeyCape config/JWT/JWKS behavior	key-cape	Pin the complete non-secret client configuration and the signing-key strategy; verify discovery, new JWTs, JWKS, and old-token invalidation.
LLDAP provider and Authelia consumer	net-kingdom	Replace the LLDAP credential at its issuer, update Authelia's consumer binding, and prove predecessor denial.
Authelia client provider	net-kingdom	Install the replacement client-secret hash and prove old-secret rejection.
privacyIDEA token provider	net-kingdom	Issue a replacement token with overlap, verify it, then revoke the predecessor.
Downstream JWT consumers	service owners coordinated by key-cape	Refresh JWKS caches or restart as declared and prove new-token acceptance plus old-token rejection.

warden route show openbao-api-key confirms railiance-platform owns the non-automatable rotation mechanics; ops-warden routes but does not execute.

Required decisions and receipts

Before any value exists, record one revision-pinned receipt containing:

• this incident id and exact Secret UID/resource version;
• KeyCape, NetKingdom, and railiance-platform Git revisions;
• a named start/end window of at most 30 minutes;
• the attended driver and independent abort operator;
• owner acknowledgements for all four credential classes;
• the exact non-secret KeyCape client-config source;
• the chosen signing strategy and downstream cache behavior;
• explicit permission for provider writes, the one Secret apply, rollouts, predecessor revocation, and protected negative verification.

KeyCape currently signs and publishes only hard-coded kid=key-1. Its KEY-WP-0011 records operator acceptance of deliberate session/token invalidation, but that is not a live GO for this platform plan. Before the window, KeyCape must pin one of these concrete implementations:

1. a reviewed unique-kid/multi-key overlap revision; or
2. deliberate immediate invalidation, including explicit downstream JWKS-cache refresh/restart steps and proof that the old controlled token is rejected.

Do not rotate a key behind the same cached kid=key-1 without the declared cache invalidation procedure.

Attended sequence

1. Preflight and hold

1. Recheck the metadata anchor without reading Secret data.
2. Verify all pinned repositories are clean and at the approved revisions.
3. Verify KeyCape, Authelia, LLDAP, and privacyIDEA are healthy.
4. Create one controlled pre-cutover OIDC token and keep it only in the private workspace for the post-cutover negative test.
5. Verify the cleanup trap and abort operator, then stop at an exact human GO.

2. Prepare one replacement bundle

1. Generate the new signing key and selected unique kid/rollover metadata.
2. Generate the replacement LLDAP and Authelia credentials.
3. Ask privacyIDEA to issue the replacement admin token while the predecessor remains valid.
4. Build the complete KeyCape config.yaml once from the pinned non-secret client configuration plus the three replacement provider credentials.
5. Build the Authelia replacement configuration/hash from the same LLDAP and Authelia values.
6. Validate syntax and derive only safe fingerprints. Do not print either file.

3. Provider and consumer cutover

1. Install the replacement Authelia client hash and replacement Authelia LLDAP binding without using a stale full-bundle generator.
2. Replace the LLDAP provider credential. From this point the prior LLDAP credential must fail and the bounded SSO interruption clock starts.
3. Apply exactly one reconstructed sso/keycape-config bundle guarded by the anchored Secret UID/resource version. The apply consumes config.yaml and key.pem from files and must not emit a Secret manifest.
4. Roll Authelia and KeyCape, wait for observed generations and readiness, and abort forward if either fails. Never restore the exposed bundle.
5. Keep the predecessor privacyIDEA token only until positive KeyCape MFA proof completes, then revoke it.

4. Positive and negative proof

Record only pass/fail and safe identifiers:

• discovery and JWKS are reachable; JWKS digest/kid match the approved result;
• a newly issued controlled JWT validates at a downstream consumer;
• the pre-cutover controlled JWT is rejected after the declared invalidation or overlap expiry;
• KeyCape can perform an LLDAP lookup with the replacement and the predecessor bind credential is rejected;
• Authelia accepts the replacement KeyCape client credential and rejects the predecessor;
• KeyCape completes privacyIDEA MFA with the replacement token and the predecessor token is rejected after revocation;
• KeyCape, Authelia, LLDAP, privacyIDEA, and affected downstreams are Ready.

Any ambiguous result is a fail-closed abort. Do not extend the window by restoring or re-enabling an exposed predecessor.

5. Cleanup and evidence

1. Revoke every predecessor not intrinsically invalidated by its provider.
2. Securely remove the private workspace and verify all files are absent.
3. Record the new Secret resource version, public JWKS digest/kid, provider receipt identifiers, boolean predecessor-denial results, rollout status, start/end timestamps, and cleanup outcome.
4. Notify KeyCape and NetKingdom with identifiers/fingerprints/status only.

Abort conditions

Abort before mutation on revision/metadata drift, a missing owner, absent provider access, unsafe helper output, an unverified cleanup trap, or no exact human GO. Abort forward after mutation on failed readiness, missing negative proof, an unexpected JWKS result, any captured value, or elapsed window. The abort operator may stop the sequence at any time.