From 453fed3b2d47c5520fb0398045da6eb54af5ddd5 Mon Sep 17 00:00:00 2001 From: codex Date: Sun, 23 Aug 2026 14:10:06 +0200 Subject: [PATCH] Plan KeyCape exposure recovery Assistant: codex Assistant-Model: gpt-5.6-sol Assistant-Session: 01a02e56-e4ad-71a2-b3e2-b6193e0d8093 --- WORK-RECORDS.md | 6 + docs/keycape-live-secret-exposure-recovery.md | 177 ++++++++++++++++++ ...9-keycape-live-secret-exposure-recovery.md | 88 +++++++++ 3 files changed, 271 insertions(+) create mode 100644 docs/keycape-live-secret-exposure-recovery.md create mode 100644 workplans/RAILIANCE-WP-0029-keycape-live-secret-exposure-recovery.md diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index f6f9180..cf0c039 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -28,6 +28,7 @@ | workplan | RAILIANCE-WP-0026 | finished | — | workplans/RAILIANCE-WP-0026-attended-login-output-containment.md | | workplan | RAILIANCE-WP-0027 | blocked | — | workplans/RAILIANCE-WP-0027-openbao-operator-only-access.md | | workplan | RAILIANCE-WP-0028 | finished | — | workplans/RAILIANCE-WP-0028-flex-auth-credential-grant-integration.md | +| workplan | RAILIANCE-WP-0029 | active | — | workplans/RAILIANCE-WP-0029-keycape-live-secret-exposure-recovery.md | | workplan | RPF-WP-0018 | finished | — | workplans/RPF-WP-0018-policy-surface-alignment.md | | workplan | RPF-WP-0019 | finished | — | workplans/RPF-WP-0019-apps-pg-recoverability-and-controls.md | | workplan | RPF-WP-0020 | finished | — | workplans/RPF-WP-0020-ccr-schema-drift.md | @@ -121,6 +122,11 @@ | task | RAILIANCE-WP-0028-T01 | done | — | workplans/RAILIANCE-WP-0028-flex-auth-credential-grant-integration.md | | task | RAILIANCE-WP-0028-T02 | done | — | workplans/RAILIANCE-WP-0028-flex-auth-credential-grant-integration.md | | task | RAILIANCE-WP-0028-T03 | done | — | workplans/RAILIANCE-WP-0028-flex-auth-credential-grant-integration.md | +| task | RAILIANCE-WP-0029-T01 | done | — | workplans/RAILIANCE-WP-0029-keycape-live-secret-exposure-recovery.md | +| task | RAILIANCE-WP-0029-T02 | done | — | workplans/RAILIANCE-WP-0029-keycape-live-secret-exposure-recovery.md | +| task | RAILIANCE-WP-0029-T03 | progress | — | workplans/RAILIANCE-WP-0029-keycape-live-secret-exposure-recovery.md | +| task | RAILIANCE-WP-0029-T04 | wait | — | workplans/RAILIANCE-WP-0029-keycape-live-secret-exposure-recovery.md | +| task | RAILIANCE-WP-0029-T05 | wait | — | workplans/RAILIANCE-WP-0029-keycape-live-secret-exposure-recovery.md | | task | RPF-WP-0018-T01 | done | — | workplans/RPF-WP-0018-policy-surface-alignment.md | | task | RPF-WP-0018-T02 | done | — | workplans/RPF-WP-0018-policy-surface-alignment.md | | task | RPF-WP-0018-T03 | done | — | workplans/RPF-WP-0018-policy-surface-alignment.md | diff --git a/docs/keycape-live-secret-exposure-recovery.md b/docs/keycape-live-secret-exposure-recovery.md new file mode 100644 index 0000000..d44aaa2 --- /dev/null +++ b/docs/keycape-live-secret-exposure-recovery.md @@ -0,0 +1,177 @@ +# KeyCape live Secret exposure recovery + +Incident: `KEYCAPE-EXPOSURE-20260823-01` + +This is the value-safe recovery contract for the accidental rendering of the +base64 data map of Kubernetes Secret `sso/keycape-config`. Treat every value in +that map as exposed even though no value was decoded or repeated. This document +authorizes planning and metadata-only checks; it is not a live rotation GO. + +## Affected classes + +- KeyCape RS256 signing private key (`key.pem`). +- LLDAP bind/admin credential embedded in `config.yaml`. +- Authelia confidential-client credential embedded in `config.yaml` and its + provider-side hash. +- privacyIDEA admin token embedded in `config.yaml` and held by its provider. + +The Secret also carries non-secret client registrations. Those registrations +must come from one reviewed source revision during reconstruction; replaying a +stale bootstrap bundle can silently remove a new client or reintroduce an +already replaced credential. + +## Containment rules + +- Never use `kubectl get secret ... -o yaml`, `-o json`, `.data`, `stringData`, + `describe`, or a command that can render the Secret payload. +- Never run NetKingdom's legacy `creds-rotate.sh`; it prints generated values. +- Never put plaintext, base64 payloads, hashes derived from low-entropy + passwords, private-key PEM, tokens, or complete Secret manifests in Git, + State Hub, chat, command arguments, ordinary logs, or captured agent output. +- Create a private workspace with `mktemp -d`, mode `0700`; keep each value in + a separate mode-`0600` file and pass it to tools by file or protected stdin. + Install an idempotent cleanup trap before generating anything. +- One attended driver owns all writes. A second named operator owns abort. No + background job or independent per-credential helper may write during the + window. +- The recovery is forward-only. The exposed bundle is not a rollback artifact + and must never be restored. + +Safe evidence is limited to identifiers, Kubernetes UIDs/resource versions, +Git revisions, public-key/JWKS digests, credential fingerprints that cannot be +used for authentication, timestamps, status codes, boolean pass/fail results, +and cleanup outcomes. + +## Current metadata anchor + +Observed value-safely on 2026-08-23: + +- cluster context: `default`; +- Secret: `sso/keycape-config`; +- Secret UID: `2e94519d-1550-41c7-9701-2efe47fe1fd3`; +- Secret resource version: `51326531`; +- KeyCape Deployment generation/observed generation: `25/25`; +- KeyCape image: `key-cape:mfa-client-20260809`; +- KeyCape Ready/Available: `1/1`; +- discovery issuer: `https://kc.coulomb.social`; +- public JWKS kid: `key-1`; +- public JWKS SHA-256: + `8e3237da6030c6af91c5005d0112a149204bc549f67d2234939c5581fdb95a32`. + +Recheck these fields before a window using metadata-only columns and public +OIDC endpoints. A changed UID, resource version, source revision, image, or +JWKS digest invalidates the prepared approval receipt and requires review. + +## Ownership + +| Boundary | Owner | Required contribution | +| --- | --- | --- | +| Bundle custody and guarded apply | railiance-platform | Private workspace, exact Secret UID/resource-version guard, single bundle apply, fingerprints, cleanup receipt. | +| KeyCape config/JWT/JWKS behavior | key-cape | Pin the complete non-secret client configuration and the signing-key strategy; verify discovery, new JWTs, JWKS, and old-token invalidation. | +| LLDAP provider and Authelia consumer | net-kingdom | Replace the LLDAP credential at its issuer, update Authelia's consumer binding, and prove predecessor denial. | +| Authelia client provider | net-kingdom | Install the replacement client-secret hash and prove old-secret rejection. | +| privacyIDEA token provider | net-kingdom | Issue a replacement token with overlap, verify it, then revoke the predecessor. | +| Downstream JWT consumers | service owners coordinated by key-cape | Refresh JWKS caches or restart as declared and prove new-token acceptance plus old-token rejection. | + +`warden route show openbao-api-key` confirms railiance-platform owns the +non-automatable rotation mechanics; ops-warden routes but does not execute. + +## Required decisions and receipts + +Before any value exists, record one revision-pinned receipt containing: + +- this incident id and exact Secret UID/resource version; +- KeyCape, NetKingdom, and railiance-platform Git revisions; +- a named start/end window of at most 30 minutes; +- the attended driver and independent abort operator; +- owner acknowledgements for all four credential classes; +- the exact non-secret KeyCape client-config source; +- the chosen signing strategy and downstream cache behavior; +- explicit permission for provider writes, the one Secret apply, rollouts, + predecessor revocation, and protected negative verification. + +KeyCape currently signs and publishes only hard-coded `kid=key-1`. Its +`KEY-WP-0011` records operator acceptance of deliberate session/token +invalidation, but that is not a live GO for this platform plan. Before the +window, KeyCape must pin one of these concrete implementations: + +1. a reviewed unique-kid/multi-key overlap revision; or +2. deliberate immediate invalidation, including explicit downstream JWKS-cache + refresh/restart steps and proof that the old controlled token is rejected. + +Do not rotate a key behind the same cached `kid=key-1` without the declared +cache invalidation procedure. + +## Attended sequence + +### 1. Preflight and hold + +1. Recheck the metadata anchor without reading Secret data. +2. Verify all pinned repositories are clean and at the approved revisions. +3. Verify KeyCape, Authelia, LLDAP, and privacyIDEA are healthy. +4. Create one controlled pre-cutover OIDC token and keep it only in the private + workspace for the post-cutover negative test. +5. Verify the cleanup trap and abort operator, then stop at an exact human GO. + +### 2. Prepare one replacement bundle + +1. Generate the new signing key and selected unique kid/rollover metadata. +2. Generate the replacement LLDAP and Authelia credentials. +3. Ask privacyIDEA to issue the replacement admin token while the predecessor + remains valid. +4. Build the complete KeyCape `config.yaml` once from the pinned non-secret + client configuration plus the three replacement provider credentials. +5. Build the Authelia replacement configuration/hash from the same LLDAP and + Authelia values. +6. Validate syntax and derive only safe fingerprints. Do not print either file. + +### 3. Provider and consumer cutover + +1. Install the replacement Authelia client hash and replacement Authelia LLDAP + binding without using a stale full-bundle generator. +2. Replace the LLDAP provider credential. From this point the prior LLDAP + credential must fail and the bounded SSO interruption clock starts. +3. Apply exactly one reconstructed `sso/keycape-config` bundle guarded by the + anchored Secret UID/resource version. The apply consumes `config.yaml` and + `key.pem` from files and must not emit a Secret manifest. +4. Roll Authelia and KeyCape, wait for observed generations and readiness, and + abort forward if either fails. Never restore the exposed bundle. +5. Keep the predecessor privacyIDEA token only until positive KeyCape MFA proof + completes, then revoke it. + +### 4. Positive and negative proof + +Record only pass/fail and safe identifiers: + +- discovery and JWKS are reachable; JWKS digest/kid match the approved result; +- a newly issued controlled JWT validates at a downstream consumer; +- the pre-cutover controlled JWT is rejected after the declared invalidation + or overlap expiry; +- KeyCape can perform an LLDAP lookup with the replacement and the predecessor + bind credential is rejected; +- Authelia accepts the replacement KeyCape client credential and rejects the + predecessor; +- KeyCape completes privacyIDEA MFA with the replacement token and the + predecessor token is rejected after revocation; +- KeyCape, Authelia, LLDAP, privacyIDEA, and affected downstreams are Ready. + +Any ambiguous result is a fail-closed abort. Do not extend the window by +restoring or re-enabling an exposed predecessor. + +### 5. Cleanup and evidence + +1. Revoke every predecessor not intrinsically invalidated by its provider. +2. Securely remove the private workspace and verify all files are absent. +3. Record the new Secret resource version, public JWKS digest/kid, provider + receipt identifiers, boolean predecessor-denial results, rollout status, + start/end timestamps, and cleanup outcome. +4. Notify KeyCape and NetKingdom with identifiers/fingerprints/status only. + +## Abort conditions + +Abort before mutation on revision/metadata drift, a missing owner, absent +provider access, unsafe helper output, an unverified cleanup trap, or no exact +human GO. Abort forward after mutation on failed readiness, missing negative +proof, an unexpected JWKS result, any captured value, or elapsed window. The +abort operator may stop the sequence at any time. + diff --git a/workplans/RAILIANCE-WP-0029-keycape-live-secret-exposure-recovery.md b/workplans/RAILIANCE-WP-0029-keycape-live-secret-exposure-recovery.md new file mode 100644 index 0000000..0756360 --- /dev/null +++ b/workplans/RAILIANCE-WP-0029-keycape-live-secret-exposure-recovery.md @@ -0,0 +1,88 @@ +--- +id: RAILIANCE-WP-0029 +type: workplan +title: "Coordinate KeyCape live Secret exposure recovery" +domain: financials +repo: railiance-platform +status: active +owner: codex +topic_slug: railiance +created: "2026-08-23" +updated: "2026-08-23" +related: + - KEY-WP-0011 +origin: routed +origin_ref: "State Hub messages e88abb61-e393-4a82-817c-5ac378a2ee3d and acf98be3-ff6b-4270-bd21-0193bebd806b" +--- + +# RAILIANCE-WP-0029 — KeyCape live Secret exposure recovery + +## Goal + +Coordinate a forward-only, value-safe rotation of every credential class in +the exposed `sso/keycape-config` bundle. Never reproduce or decode the exposed +payload and never treat repository access as live mutation authority. + +## T01 — Contain and establish the recovery boundary + +```task +id: RAILIANCE-WP-0029-T01 +status: done +priority: high +``` + +Accepted the KeyCape/NetKingdom incident reports, stopped payload inspection, +and routed custody through `warden route show openbao-api-key`. Metadata-only +preflight pinned Secret UID/resource version, Deployment generation/image, and +the public JWKS digest/kid. The legacy value-printing rotation helper is banned. + +## T02 — Publish the governed bundle cutover + +```task +id: RAILIANCE-WP-0029-T02 +status: done +priority: high +``` + +`docs/keycape-live-secret-exposure-recovery.md` defines owners, required +revision/window/operator receipts, private-file handling, one guarded bundle +apply, provider/consumer ordering, forward-only abort, positive/negative proof, +predecessor revocation, and sanitized evidence. + +## T03 — Collect exact owner acknowledgements + +```task +id: RAILIANCE-WP-0029-T03 +status: progress +priority: high +``` + +KeyCape must pin the non-secret client-config revision and either a unique-kid +overlap implementation or the exact immediate-invalidation/cache-refresh +procedure. NetKingdom must pin the LLDAP, Authelia, and privacyIDEA provider +steps. Railiance-platform must then issue one digest-bound approval template. + +## T04 — Execute the attended rotation + +```task +id: RAILIANCE-WP-0029-T04 +status: wait +priority: high +``` + +Requires a fresh exact human GO, an at-most-30-minute window, named driver and +abort operator, approved revisions, provider access, private workspace cleanup, +and all T03 acknowledgements. No value may enter captured output. + +## T05 — Prove predecessor denial and close + +```task +id: RAILIANCE-WP-0029-T05 +status: wait +priority: high +``` + +Verify replacement operation and predecessor rejection for the signing key, +LLDAP binding, Authelia client, and privacyIDEA token. Retain only safe +fingerprints, resource versions, public JWKS metadata, boolean results, rollout +status, timestamps, and cleanup receipts.