tegwick
d9557ba90e
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a13eab-1e6a-7b33-9c37-372e90ada3a2
101 lines
5.3 KiB
Markdown
101 lines
5.3 KiB
Markdown
# Disaster Recovery Runbook — net-kingdom SSO/MFA Platform
|
|
|
|
Procedure status: legacy examples were withdrawn in the 2026-10-15 custody
|
|
review. Use current owner procedures below. Historical exercise receipts do not
|
|
prove current recovery or independent custody. See the
|
|
[procedure inventory](../../../docs/attended-procedure-inventory.md) and
|
|
[NK-WP-0049 recovery ceremony](../../../docs/bootstrap-recovery-ceremony.md).
|
|
|
|
**Stack:** LLDAP + Authelia + KeyCape (sso namespace) + privacyIDEA (mfa namespace)
|
|
**PostgreSQL:** Managed separately by CNPG (`postgresql/scheduled-backup.yaml`)
|
|
|
|
---
|
|
|
|
## Recovery scenarios
|
|
|
|
| Scenario | Impact | Recovery |
|
|
|----------|--------|----------|
|
|
| Pod crash / OOM | Stateless pods (KeyCape) recover automatically. Stateful pods (LLDAP, Authelia, PI) restart and reload from PVC. | K8s self-heals. Verify with `verify-t05.sh`. |
|
|
| PVC data corruption | Users/sessions/tokens lost. | Restore from SQLite backup (see below). |
|
|
| Node failure (single-node K3s) | All pods lost; PVC survival must be checked. | Owner-reviewed host/workload recovery with exact retained data and current declarations. |
|
|
| Node total loss (disk gone) | Everything lost. | Full restore from backup + KeePassXC. |
|
|
| Stack locked out (SSO broken, can't log in) | No user access to OIDC-protected apps. | Use break-glass account. |
|
|
| enckey lost (privacyIDEA) | Enrolled token secrets cannot be decrypted without the matching key. | Restore the original encryption/audit key set and prove native token/audit continuity; never auto-generate a substitute. |
|
|
|
|
---
|
|
|
|
## Independent operator access
|
|
|
|
Confirm an existing provider console/rescue path and independently held custody
|
|
before any recovery change. An LLDAP account or `kubectl exec` depends on the
|
|
service or cluster being reachable; neither alone is independent recovery.
|
|
Preserve existing break-glass credentials. Use the IAM owner's approved procedure
|
|
and record actual login/MFA results, without passwords, tokens or OTP values.
|
|
Do not rerun the retired account/password helpers or disable MFA as a test.
|
|
|
|
---
|
|
|
|
## Restore order
|
|
|
|
**CRITICAL: Always restore in this order.** Components depend on each other
|
|
at startup: privacyIDEA needs PostgreSQL, KeyCape needs all three.
|
|
|
|
```
|
|
1. PostgreSQL (databases ns) — CNPG operator handles restore
|
|
2. privacyIDEA (mfa ns) — needs PG + enckey PVC
|
|
3. LLDAP (sso ns) — standalone
|
|
4. Authelia (sso ns) — needs LLDAP (LDAP bind at startup check)
|
|
5. KeyCape (sso ns) — needs Authelia + LLDAP + privacyIDEA
|
|
```
|
|
|
|
---
|
|
|
|
## SQLite recovery
|
|
|
|
The former unpinned `sqlite3:latest` examples piped an in-pod dump into a local
|
|
shell's SQLite process; they were not a valid in-PVC restore procedure. Do not
|
|
run them. Under the exact owner-reviewed isolated ceremony, retrieve an identified
|
|
off-host database copy, verify its checksum and SQLite integrity in private
|
|
staging, stop only an explicitly approved target, and restore into its exact
|
|
pinned helper/PVC. Preserve the original files and their UIDs/ownership for
|
|
reconciliation. Restart and verify actual service behavior. Production restore
|
|
is a separate maintenance operation, not part of the local transport check.
|
|
|
|
### privacyIDEA encryption and audit keys
|
|
|
|
Use the [privacyIDEA key custody/PVC recovery procedure](../privacyidea/KEY-RECOVERY.md).
|
|
The authoritative paths are on `privacyidea-data`, not a mounted Secret. Restore
|
|
all three files from the matching independent backup into that PVC with the app
|
|
stopped, then perform native database/token/audit acceptance. The DR Secret field
|
|
is `enckey`; `PI_ENCFILE` names the application config setting and historic safe
|
|
entry. The previous Secret-only restart instructions are withdrawn. Never generate
|
|
a fresh encryption key as an automatic response to a missing recovery key.
|
|
|
|
---
|
|
|
|
## Full node restore (new host)
|
|
|
|
Prepare with the Railiance infrastructure, database and IAM owners. Obtain an
|
|
exact target inventory, immutable recovery images independent of the failed
|
|
registry, matching off-host database/key/config snapshots and retained original
|
|
state. Follow current owner deployment declarations, restore order above and
|
|
[privacyIDEA key recovery](../privacyidea/KEY-RECOVERY.md). Do not recursively
|
|
apply this historical tree or use fresh generated values in place of matching
|
|
recovery material. KeyCape's clients, issuer/signing keys and scoped factor
|
|
reader delivery must be preserved. Current package boundaries are recorded in
|
|
[the deployment guide](../README.md).
|
|
|
|
The whole-node restart proof and historical migration restore are supporting
|
|
receipts with their own scope; neither proves a fresh full identity restore.
|
|
|
|
## Backup offsite export
|
|
|
|
Use the existing Platform/backup owner's reviewed encrypted transport with the
|
|
registered public recipient. No private age key is needed to encrypt. The former
|
|
example read a permanent HOME key just to find its public recipient, wrote
|
|
plaintext databases into predictable `/tmp` paths, and encrypted only one of its
|
|
two exports; it is withdrawn. Verify actual transfer and retrieval for every
|
|
required asset, preserve prior good points and independently held key custody.
|
|
Scaleway and Storage Share provider presence alone does not establish coverage
|
|
of LLDAP, Authelia, privacyIDEA keys/database, KeyCape configuration or a current
|
|
bootstrap bundle. Do not use an upload-only share as a retrieval credential.
|