railiance-platform/docs/keycape-live-secret-exposure-recovery.md

# KeyCape live Secret exposure recovery

Incident: `KEYCAPE-EXPOSURE-20260823-01`

This is the value-safe recovery contract for the accidental rendering of the
base64 data map of Kubernetes Secret `sso/keycape-config`. Treat every value in
that map as exposed even though no value was decoded or repeated. This document
authorizes planning and metadata-only checks; it is not a live rotation GO.

## Affected classes

- KeyCape RS256 signing private key (`key.pem`).
- LLDAP bind/admin credential embedded in `config.yaml`.
- Authelia confidential-client credential embedded in `config.yaml` and its
  provider-side hash.
- privacyIDEA admin token embedded in `config.yaml` and held by its provider.

The Secret also carries non-secret client registrations. Those registrations
must come from one reviewed source revision during reconstruction; replaying a
stale bootstrap bundle can silently remove a new client or reintroduce an
already replaced credential.

## Containment rules

- Never use `kubectl get secret ... -o yaml`, `-o json`, `.data`, `stringData`,
  `describe`, or a command that can render the Secret payload.
- Never run NetKingdom's legacy `creds-rotate.sh`; it prints generated values.
- Never put plaintext, base64 payloads, hashes derived from low-entropy
  passwords, private-key PEM, tokens, or complete Secret manifests in Git,
  State Hub, chat, command arguments, ordinary logs, or captured agent output.
- Create a private workspace with `mktemp -d`, mode `0700`; keep each value in
  a separate mode-`0600` file and pass it to tools by file or protected stdin.
  Install an idempotent cleanup trap before generating anything.
- One attended driver owns all writes. A second named operator owns abort. No
  background job or independent per-credential helper may write during the
  window.
- The recovery is forward-only. The exposed bundle is not a rollback artifact
  and must never be restored.

Safe evidence is limited to identifiers, Kubernetes UIDs/resource versions,
Git revisions, public-key/JWKS digests, credential fingerprints that cannot be
used for authentication, timestamps, status codes, boolean pass/fail results,
and cleanup outcomes.

## Current metadata anchor

Observed value-safely on 2026-08-23:

- cluster context: `default`;
- Secret: `sso/keycape-config`;
- Secret UID: `2e94519d-1550-41c7-9701-2efe47fe1fd3`;
- Secret resource version: `51326531`;
- KeyCape Deployment generation/observed generation: `25/25`;
- KeyCape image: `key-cape:mfa-client-20260809`;
- KeyCape Ready/Available: `1/1`;
- discovery issuer: `https://kc.coulomb.social`;
- public JWKS kid: `key-1`;
- public JWKS SHA-256:
  `8e3237da6030c6af91c5005d0112a149204bc549f67d2234939c5581fdb95a32`.

Recheck these fields before a window using metadata-only columns and public
OIDC endpoints. A changed UID, resource version, source revision, image, or
JWKS digest invalidates the prepared approval receipt and requires review.

## Owner receipt update (2026-08-23)

KeyCape reports that its emergency signing-key rotation completed with
deliberate JWT invalidation; consumers must refresh JWKS/re-authenticate. The
receipt contains no secret values, but it does not yet identify the reviewed
non-secret client-config revision or the post-rotation public JWKS digest.
NetKingdom published the value-safe dependency and provider sequence at
`c24d67b`, noting that the LLDAP admin account is persistent and that the
privacyIDEA admin token is an expiring session JWT without individual
revocation. Use expiry-based predecessor denial unless a separate global
signing-secret invalidation is explicitly approved. These acknowledgements do
not constitute a live GO; the approval template remains pending.

A fresh public JWKS read on 2026-08-23 found kid `key-1` and SHA-256
`3d46c07b649432eb41112b9e5f5a929460027766127d1eb419ebac8eb9858c06`.
Treat this as an observation only until KeyCape binds it to the source
revision and confirms downstream cache refresh.

KeyCape subsequently supplied source revision
`93704fd2424503007c20b458b62a7f7d994bb288`, final public JWKS kid `key-1`,
and SHA-256
`c6faac5dfeef2453daf9cfc14671f62b535dd321befdf6014e3ee5c2cf1f1156`.
KeyCape, Authelia, LLDAP, privacyIDEA, and identity-provisioner were reported
Ready 1/1, with replacement/negative checks recorded. The persistent
privacyIDEA `lldap-coulomb` resolver remains an attended provider-admin
follow-up and is explicitly not complete; do not declare the incident closed
or perform further bundle mutation until that owner action is separately
authorized and evidenced.

## Ownership

| Boundary | Owner | Required contribution |
| --- | --- | --- |
| Bundle custody and guarded apply | railiance-platform | Private workspace, exact Secret UID/resource-version guard, single bundle apply, fingerprints, cleanup receipt. |
| KeyCape config/JWT/JWKS behavior | key-cape | Pin the complete non-secret client configuration and the signing-key strategy; verify discovery, new JWTs, JWKS, and old-token invalidation. |
| LLDAP provider and Authelia consumer | net-kingdom | Replace the LLDAP credential at its issuer, update Authelia's consumer binding, and prove predecessor denial. |
| Authelia client provider | net-kingdom | Install the replacement client-secret hash and prove old-secret rejection. |
| privacyIDEA token provider | net-kingdom | Issue a replacement token with overlap, verify it, then revoke the predecessor. |
| Downstream JWT consumers | service owners coordinated by key-cape | Refresh JWKS caches or restart as declared and prove new-token acceptance plus old-token rejection. |

`warden route show openbao-api-key` confirms railiance-platform owns the
non-automatable rotation mechanics; ops-warden routes but does not execute.

## Required decisions and receipts

Before any value exists, record one revision-pinned receipt containing:

- this incident id and exact Secret UID/resource version;
- KeyCape, NetKingdom, and railiance-platform Git revisions;
- a named start/end window of at most 30 minutes;
- the attended driver and independent abort operator;
- owner acknowledgements for all four credential classes;
- the exact non-secret KeyCape client-config source;
- the chosen signing strategy and downstream cache behavior;
- explicit permission for provider writes, the one Secret apply, rollouts,
  predecessor revocation, and protected negative verification.

KeyCape currently signs and publishes only hard-coded `kid=key-1`. Its
`KEY-WP-0011` records operator acceptance of deliberate session/token
invalidation, but that is not a live GO for this platform plan. Before the
window, KeyCape must pin one of these concrete implementations:

1. a reviewed unique-kid/multi-key overlap revision; or
2. deliberate immediate invalidation, including explicit downstream JWKS-cache
   refresh/restart steps and proof that the old controlled token is rejected.

Do not rotate a key behind the same cached `kid=key-1` without the declared
cache invalidation procedure.

## Attended sequence

### 1. Preflight and hold

1. Recheck the metadata anchor without reading Secret data.
2. Verify all pinned repositories are clean and at the approved revisions.
3. Verify KeyCape, Authelia, LLDAP, and privacyIDEA are healthy.
4. Create one controlled pre-cutover OIDC token and keep it only in the private
   workspace for the post-cutover negative test.
5. Verify the cleanup trap and abort operator, then stop at an exact human GO.

### 2. Prepare one replacement bundle

1. Generate the new signing key and selected unique kid/rollover metadata.
2. Generate the replacement LLDAP and Authelia credentials.
3. Ask privacyIDEA to issue the replacement admin token while the predecessor
   remains valid.
4. Build the complete KeyCape `config.yaml` once from the pinned non-secret
   client configuration plus the three replacement provider credentials.
5. Build the Authelia replacement configuration/hash from the same LLDAP and
   Authelia values.
6. Validate syntax and derive only safe fingerprints. Do not print either file.

### 3. Provider and consumer cutover

1. Install the replacement Authelia client hash and replacement Authelia LLDAP
   binding without using a stale full-bundle generator.
2. Replace the LLDAP provider credential. From this point the prior LLDAP
   credential must fail and the bounded SSO interruption clock starts.
3. Apply exactly one reconstructed `sso/keycape-config` bundle guarded by the
   anchored Secret UID/resource version. The apply consumes `config.yaml` and
   `key.pem` from files and must not emit a Secret manifest.
4. Roll Authelia and KeyCape, wait for observed generations and readiness, and
   abort forward if either fails. Never restore the exposed bundle.
5. Keep the predecessor privacyIDEA token only until positive KeyCape MFA proof
   completes, then revoke it.

### 4. Positive and negative proof

Record only pass/fail and safe identifiers:

- discovery and JWKS are reachable; JWKS digest/kid match the approved result;
- a newly issued controlled JWT validates at a downstream consumer;
- the pre-cutover controlled JWT is rejected after the declared invalidation
  or overlap expiry;
- KeyCape can perform an LLDAP lookup with the replacement and the predecessor
  bind credential is rejected;
- Authelia accepts the replacement KeyCape client credential and rejects the
  predecessor;
- KeyCape completes privacyIDEA MFA with the replacement token and the
  predecessor token is rejected after revocation;
- KeyCape, Authelia, LLDAP, privacyIDEA, and affected downstreams are Ready.

Any ambiguous result is a fail-closed abort. Do not extend the window by
restoring or re-enabling an exposed predecessor.

### 5. Cleanup and evidence

1. Revoke every predecessor not intrinsically invalidated by its provider.
2. Securely remove the private workspace and verify all files are absent.
3. Record the new Secret resource version, public JWKS digest/kid, provider
   receipt identifiers, boolean predecessor-denial results, rollout status,
   start/end timestamps, and cleanup outcome.
4. Notify KeyCape and NetKingdom with identifiers/fingerprints/status only.

## Abort conditions

Abort before mutation on revision/metadata drift, a missing owner, absent
provider access, unsafe helper output, an unverified cleanup trap, or no exact
human GO. Abort forward after mutation on failed readiness, missing negative
proof, an unexpected JWKS result, any captured value, or elapsed window. The
abort operator may stop the sequence at any time.