tegwick d9557ba90e

CI Smoke / host-smoke (push) Successful in 0s

Details

CI Smoke / container-smoke (push) Successful in 1s

Details

Harden remaining bootstrap custody callers and prepare isolated recovery

Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a13eab-1e6a-7b33-9c37-372e90ada3a2

2026-10-15 22:57:14 +02:00

5.3 KiB

Raw Blame History

Disaster Recovery Runbook — net-kingdom SSO/MFA Platform

Procedure status: legacy examples were withdrawn in the 2026-10-15 custody review. Use current owner procedures below. Historical exercise receipts do not prove current recovery or independent custody. See the procedure inventory and NK-WP-0049 recovery ceremony.

Stack: LLDAP + Authelia + KeyCape (sso namespace) + privacyIDEA (mfa namespace) PostgreSQL: Managed separately by CNPG (postgresql/scheduled-backup.yaml)

Recovery scenarios

Scenario	Impact	Recovery
Pod crash / OOM	Stateless pods (KeyCape) recover automatically. Stateful pods (LLDAP, Authelia, PI) restart and reload from PVC.	K8s self-heals. Verify with `verify-t05.sh`.
PVC data corruption	Users/sessions/tokens lost.	Restore from SQLite backup (see below).
Node failure (single-node K3s)	All pods lost; PVC survival must be checked.	Owner-reviewed host/workload recovery with exact retained data and current declarations.
Node total loss (disk gone)	Everything lost.	Full restore from backup + KeePassXC.
Stack locked out (SSO broken, can't log in)	No user access to OIDC-protected apps.	Use break-glass account.
enckey lost (privacyIDEA)	Enrolled token secrets cannot be decrypted without the matching key.	Restore the original encryption/audit key set and prove native token/audit continuity; never auto-generate a substitute.

Independent operator access

Confirm an existing provider console/rescue path and independently held custody before any recovery change. An LLDAP account or kubectl exec depends on the service or cluster being reachable; neither alone is independent recovery. Preserve existing break-glass credentials. Use the IAM owner's approved procedure and record actual login/MFA results, without passwords, tokens or OTP values. Do not rerun the retired account/password helpers or disable MFA as a test.

Restore order

CRITICAL: Always restore in this order. Components depend on each other at startup: privacyIDEA needs PostgreSQL, KeyCape needs all three.

1. PostgreSQL (databases ns)    — CNPG operator handles restore
2. privacyIDEA (mfa ns)        — needs PG + enckey PVC
3. LLDAP (sso ns)              — standalone
4. Authelia (sso ns)           — needs LLDAP (LDAP bind at startup check)
5. KeyCape (sso ns)            — needs Authelia + LLDAP + privacyIDEA

SQLite recovery

The former unpinned sqlite3:latest examples piped an in-pod dump into a local shell's SQLite process; they were not a valid in-PVC restore procedure. Do not run them. Under the exact owner-reviewed isolated ceremony, retrieve an identified off-host database copy, verify its checksum and SQLite integrity in private staging, stop only an explicitly approved target, and restore into its exact pinned helper/PVC. Preserve the original files and their UIDs/ownership for reconciliation. Restart and verify actual service behavior. Production restore is a separate maintenance operation, not part of the local transport check.

privacyIDEA encryption and audit keys

Use the privacyIDEA key custody/PVC recovery procedure. The authoritative paths are on privacyidea-data, not a mounted Secret. Restore all three files from the matching independent backup into that PVC with the app stopped, then perform native database/token/audit acceptance. The DR Secret field is enckey; PI_ENCFILE names the application config setting and historic safe entry. The previous Secret-only restart instructions are withdrawn. Never generate a fresh encryption key as an automatic response to a missing recovery key.

Full node restore (new host)

Prepare with the Railiance infrastructure, database and IAM owners. Obtain an exact target inventory, immutable recovery images independent of the failed registry, matching off-host database/key/config snapshots and retained original state. Follow current owner deployment declarations, restore order above and privacyIDEA key recovery. Do not recursively apply this historical tree or use fresh generated values in place of matching recovery material. KeyCape's clients, issuer/signing keys and scoped factor reader delivery must be preserved. Current package boundaries are recorded in the deployment guide.

The whole-node restart proof and historical migration restore are supporting receipts with their own scope; neither proves a fresh full identity restore.

Backup offsite export

Use the existing Platform/backup owner's reviewed encrypted transport with the registered public recipient. No private age key is needed to encrypt. The former example read a permanent HOME key just to find its public recipient, wrote plaintext databases into predictable /tmp paths, and encrypted only one of its two exports; it is withdrawn. Verify actual transfer and retrieval for every required asset, preserve prior good points and independently held key custody. Scaleway and Storage Share provider presence alone does not establish coverage of LLDAP, Authelia, privacyIDEA keys/database, KeyCape configuration or a current bootstrap bundle. Do not use an upload-only share as a retrieval credential.

5.3 KiB Raw Blame History