Plan KeyCape exposure recovery

Assistant: codex
Assistant-Model: gpt-5.6-sol
Assistant-Session: 01a02e56-e4ad-71a2-b3e2-b6193e0d8093

docs/keycape-live-secret-exposure-recovery.md

@@ -0,0 +1,177 @@
+# KeyCape live Secret exposure recovery
+
+Incident: `KEYCAPE-EXPOSURE-20260823-01`
+
+This is the value-safe recovery contract for the accidental rendering of the
+base64 data map of Kubernetes Secret `sso/keycape-config`. Treat every value in
+that map as exposed even though no value was decoded or repeated. This document
+authorizes planning and metadata-only checks; it is not a live rotation GO.
+
+## Affected classes
+
+- KeyCape RS256 signing private key (`key.pem`).
+- LLDAP bind/admin credential embedded in `config.yaml`.
+- Authelia confidential-client credential embedded in `config.yaml` and its
+  provider-side hash.
+- privacyIDEA admin token embedded in `config.yaml` and held by its provider.
+
+The Secret also carries non-secret client registrations. Those registrations
+must come from one reviewed source revision during reconstruction; replaying a
+stale bootstrap bundle can silently remove a new client or reintroduce an
+already replaced credential.
+
+## Containment rules
+
+- Never use `kubectl get secret ... -o yaml`, `-o json`, `.data`, `stringData`,
+  `describe`, or a command that can render the Secret payload.
+- Never run NetKingdom's legacy `creds-rotate.sh`; it prints generated values.
+- Never put plaintext, base64 payloads, hashes derived from low-entropy
+  passwords, private-key PEM, tokens, or complete Secret manifests in Git,
+  State Hub, chat, command arguments, ordinary logs, or captured agent output.
+- Create a private workspace with `mktemp -d`, mode `0700`; keep each value in
+  a separate mode-`0600` file and pass it to tools by file or protected stdin.
+  Install an idempotent cleanup trap before generating anything.
+- One attended driver owns all writes. A second named operator owns abort. No
+  background job or independent per-credential helper may write during the
+  window.
+- The recovery is forward-only. The exposed bundle is not a rollback artifact
+  and must never be restored.
+
+Safe evidence is limited to identifiers, Kubernetes UIDs/resource versions,
+Git revisions, public-key/JWKS digests, credential fingerprints that cannot be
+used for authentication, timestamps, status codes, boolean pass/fail results,
+and cleanup outcomes.
+
+## Current metadata anchor
+
+Observed value-safely on 2026-08-23:
+
+- cluster context: `default`;
+- Secret: `sso/keycape-config`;
+- Secret UID: `2e94519d-1550-41c7-9701-2efe47fe1fd3`;
+- Secret resource version: `51326531`;
+- KeyCape Deployment generation/observed generation: `25/25`;
+- KeyCape image: `key-cape:mfa-client-20260809`;
+- KeyCape Ready/Available: `1/1`;
+- discovery issuer: `https://kc.coulomb.social`;
+- public JWKS kid: `key-1`;
+- public JWKS SHA-256:
+  `8e3237da6030c6af91c5005d0112a149204bc549f67d2234939c5581fdb95a32`.
+
+Recheck these fields before a window using metadata-only columns and public
+OIDC endpoints. A changed UID, resource version, source revision, image, or
+JWKS digest invalidates the prepared approval receipt and requires review.
+
+## Ownership
+
+| Boundary | Owner | Required contribution |
+| --- | --- | --- |
+| Bundle custody and guarded apply | railiance-platform | Private workspace, exact Secret UID/resource-version guard, single bundle apply, fingerprints, cleanup receipt. |
+| KeyCape config/JWT/JWKS behavior | key-cape | Pin the complete non-secret client configuration and the signing-key strategy; verify discovery, new JWTs, JWKS, and old-token invalidation. |
+| LLDAP provider and Authelia consumer | net-kingdom | Replace the LLDAP credential at its issuer, update Authelia's consumer binding, and prove predecessor denial. |
+| Authelia client provider | net-kingdom | Install the replacement client-secret hash and prove old-secret rejection. |
+| privacyIDEA token provider | net-kingdom | Issue a replacement token with overlap, verify it, then revoke the predecessor. |
+| Downstream JWT consumers | service owners coordinated by key-cape | Refresh JWKS caches or restart as declared and prove new-token acceptance plus old-token rejection. |
+
+`warden route show openbao-api-key` confirms railiance-platform owns the
+non-automatable rotation mechanics; ops-warden routes but does not execute.
+
+## Required decisions and receipts
+
+Before any value exists, record one revision-pinned receipt containing:
+
+- this incident id and exact Secret UID/resource version;
+- KeyCape, NetKingdom, and railiance-platform Git revisions;
+- a named start/end window of at most 30 minutes;
+- the attended driver and independent abort operator;
+- owner acknowledgements for all four credential classes;
+- the exact non-secret KeyCape client-config source;
+- the chosen signing strategy and downstream cache behavior;
+- explicit permission for provider writes, the one Secret apply, rollouts,
+  predecessor revocation, and protected negative verification.
+
+KeyCape currently signs and publishes only hard-coded `kid=key-1`. Its
+`KEY-WP-0011` records operator acceptance of deliberate session/token
+invalidation, but that is not a live GO for this platform plan. Before the
+window, KeyCape must pin one of these concrete implementations:
+
+1. a reviewed unique-kid/multi-key overlap revision; or
+2. deliberate immediate invalidation, including explicit downstream JWKS-cache
+   refresh/restart steps and proof that the old controlled token is rejected.
+
+Do not rotate a key behind the same cached `kid=key-1` without the declared
+cache invalidation procedure.
+
+## Attended sequence
+
+### 1. Preflight and hold
+
+1. Recheck the metadata anchor without reading Secret data.
+2. Verify all pinned repositories are clean and at the approved revisions.
+3. Verify KeyCape, Authelia, LLDAP, and privacyIDEA are healthy.
+4. Create one controlled pre-cutover OIDC token and keep it only in the private
+   workspace for the post-cutover negative test.
+5. Verify the cleanup trap and abort operator, then stop at an exact human GO.
+
+### 2. Prepare one replacement bundle
+
+1. Generate the new signing key and selected unique kid/rollover metadata.
+2. Generate the replacement LLDAP and Authelia credentials.
+3. Ask privacyIDEA to issue the replacement admin token while the predecessor
+   remains valid.
+4. Build the complete KeyCape `config.yaml` once from the pinned non-secret
+   client configuration plus the three replacement provider credentials.
+5. Build the Authelia replacement configuration/hash from the same LLDAP and
+   Authelia values.
+6. Validate syntax and derive only safe fingerprints. Do not print either file.
+
+### 3. Provider and consumer cutover
+
+1. Install the replacement Authelia client hash and replacement Authelia LLDAP
+   binding without using a stale full-bundle generator.
+2. Replace the LLDAP provider credential. From this point the prior LLDAP
+   credential must fail and the bounded SSO interruption clock starts.
+3. Apply exactly one reconstructed `sso/keycape-config` bundle guarded by the
+   anchored Secret UID/resource version. The apply consumes `config.yaml` and
+   `key.pem` from files and must not emit a Secret manifest.
+4. Roll Authelia and KeyCape, wait for observed generations and readiness, and
+   abort forward if either fails. Never restore the exposed bundle.
+5. Keep the predecessor privacyIDEA token only until positive KeyCape MFA proof
+   completes, then revoke it.
+
+### 4. Positive and negative proof
+
+Record only pass/fail and safe identifiers:
+
+- discovery and JWKS are reachable; JWKS digest/kid match the approved result;
+- a newly issued controlled JWT validates at a downstream consumer;
+- the pre-cutover controlled JWT is rejected after the declared invalidation
+  or overlap expiry;
+- KeyCape can perform an LLDAP lookup with the replacement and the predecessor
+  bind credential is rejected;
+- Authelia accepts the replacement KeyCape client credential and rejects the
+  predecessor;
+- KeyCape completes privacyIDEA MFA with the replacement token and the
+  predecessor token is rejected after revocation;
+- KeyCape, Authelia, LLDAP, privacyIDEA, and affected downstreams are Ready.
+
+Any ambiguous result is a fail-closed abort. Do not extend the window by
+restoring or re-enabling an exposed predecessor.
+
+### 5. Cleanup and evidence
+
+1. Revoke every predecessor not intrinsically invalidated by its provider.
+2. Securely remove the private workspace and verify all files are absent.
+3. Record the new Secret resource version, public JWKS digest/kid, provider
+   receipt identifiers, boolean predecessor-denial results, rollout status,
+   start/end timestamps, and cleanup outcome.
+4. Notify KeyCape and NetKingdom with identifiers/fingerprints/status only.
+
+## Abort conditions
+
+Abort before mutation on revision/metadata drift, a missing owner, absent
+provider access, unsafe helper output, an unverified cleanup trap, or no exact
+human GO. Abort forward after mutation on failed readiness, missing negative
+proof, an unexpected JWKS result, any captured value, or elapsed window. The
+abort operator may stop the sequence at any time.
+