net-kingdom/sso-mfa/k8s/backup/DR-RUNBOOK.md

Disaster Recovery Runbook — net-kingdom SSO/MFA Platform

Exercise status: unknown for the procedures in this runbook; no per-procedure successful-run receipt with operator attribution was established in the 2026-09-05 review. See procedure inventory.

Stack: LLDAP + Authelia + KeyCape (sso namespace) + privacyIDEA (mfa namespace) PostgreSQL: Managed separately by CNPG (postgresql/scheduled-backup.yaml)

---

Recovery scenarios

Scenario	Impact	Recovery
Pod crash / OOM	Stateless pods (KeyCape) recover automatically. Stateful pods (LLDAP, Authelia, PI) restart and reload from PVC.	K8s self-heals. Verify with verify-t05.sh.
PVC data corruption	Users/sessions/tokens lost.	Restore from SQLite backup (see below).
Node failure (single-node K3s)	All pods lost. PVCs intact on host.	Re-apply all manifests (idempotent). Pods re-attach to PVCs.
Node total loss (disk gone)	Everything lost.	Full restore from backup + KeePassXC.
Stack locked out (SSO broken, can't log in)	No user access to OIDC-protected apps.	Use break-glass account.
enckey lost (privacyIDEA)	All enrolled MFA tokens invalid. Users must re-enroll.	Restore from enckey backup or re-enroll all tokens.

---

Break-glass access

When the SSO stack is broken and no user can authenticate:

# 1. Access LLDAP admin UI directly (requires VPN / IP-allowlisted access)
#    URL: https://lldap.coulomb.social
#    Username: break-glass
#    Password: from KeePassXC → net-kingdom/Break-glass/break-glass
#
# 2. Or access LLDAP via kubectl exec (no network required)
kubectl exec -n sso deployment/lldap -- /bin/sh
# Inside container: use ldapwhoami / ldapsearch to verify directory state

# 3. Access privacyIDEA admin UI
#    URL: https://pink.coulomb.social
#    Username: pi-admin
#    Password: from KeePassXC → net-kingdom/privacyIDEA/pi-admin
#    NOTE: pi-admin has MFA enrolled — if privacyIDEA MFA is down, use:
kubectl exec -n mfa deployment/privacyidea -- pi-manage admin list

---

Restore order

CRITICAL: Always restore in this order. Components depend on each other at startup: privacyIDEA needs PostgreSQL, KeyCape needs all three.

1. PostgreSQL (databases ns)    — CNPG operator handles restore
2. privacyIDEA (mfa ns)        — needs PG + enckey PVC
3. LLDAP (sso ns)              — standalone
4. Authelia (sso ns)           — needs LLDAP (LDAP bind at startup check)
5. KeyCape (sso ns)            — needs Authelia + LLDAP + privacyIDEA

---

Restore from SQLite backup (PVC data corruption)

LLDAP

# 1. Scale down LLDAP
kubectl scale deployment/lldap -n sso --replicas=0

# 2. Start a restore pod on the lldap-data PVC
kubectl run -n sso lldap-restore --image=nouchka/sqlite3:latest \
  --restart=Never \
  --overrides='{"spec":{"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"lldap-data"}}],"containers":[{"name":"lldap-restore","image":"nouchka/sqlite3:latest","command":["sleep","3600"],"volumeMounts":[{"name":"data","mountPath":"/data"}]}]}}'

# 3. Copy backup file into the pod (or it's already on the PVC under /data/backups/)
kubectl exec -n sso lldap-restore -- ls /data/backups/

# 4. Restore from the chosen backup
kubectl exec -n sso lldap-restore -- \
  sqlite3 /data/backups/users.backup.YYYY-MM-DD ".dump" | \
  sqlite3 /data/users.db

# 5. Clean up and restart
kubectl delete pod -n sso lldap-restore
kubectl scale deployment/lldap -n sso --replicas=1
kubectl rollout status deployment/lldap -n sso --timeout=120s

Authelia

# On single-node k3s (local-path PVCs are hostPath-backed), a restore pod can mount
# authelia-data alongside the running Authelia pod. Scale down only if you need to
# replace the live db.sqlite3 in-place (Authelia must be stopped to avoid corruption).
kubectl scale deployment/authelia -n sso --replicas=0
kubectl run -n sso authelia-restore --image=nouchka/sqlite3:latest \
  --restart=Never \
  --overrides='{"spec":{"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"authelia-data"}}],"containers":[{"name":"authelia-restore","image":"nouchka/sqlite3:latest","command":["sleep","3600"],"volumeMounts":[{"name":"data","mountPath":"/data"}]}]}}'
kubectl exec -n sso authelia-restore -- ls /data/backups/
kubectl exec -n sso authelia-restore -- \
  sqlite3 /data/backups/authelia.backup.YYYY-MM-DD ".dump" | \
  sqlite3 /data/db.sqlite3
kubectl delete pod -n sso authelia-restore
kubectl scale deployment/authelia -n sso --replicas=1
kubectl rollout status deployment/authelia -n sso --timeout=120s

privacyIDEA encryption and audit keys

Use the privacyIDEA key custody/PVC recovery procedure. The authoritative paths are on privacyidea-data, not a mounted Secret. Restore all three files from the matching independent backup into that PVC with the app stopped, then perform native database/token/audit acceptance. The DR Secret field is enckey; PI_ENCFILE names the application config setting and historic safe entry. The previous Secret-only restart instructions are withdrawn. Never generate a fresh encryption key as an automatic response to a missing recovery key.

---

Full node restore (new host)

# Prerequisites on new host:
#   - K3s installed
#   - Traefik ingress (bundled with K3s)
#   - cert-manager installed (helm install cert-manager ...)
#   - DNS records pointing to new node IP
#   - KeePassXC vault accessible (offline copy or age-encrypted bundle)

# 1. Restore PostgreSQL from CNPG backup
#    (See CNPG documentation for cluster restore from barmanObjectStore)

# 2. Re-apply all manifests in order
cd sso-mfa/k8s
kubectl apply -f namespaces/namespaces.yaml
kubectl apply -f network-policies/
kubectl apply -f cert-manager/issuers.yaml

# 3. Restore secrets from KeePassXC
#    Run each create-secrets.sh in order:
cd postgresql   && ./create-secrets.sh  && cd ..
cd privacyidea  && ./create-secrets.sh  && cd ..
cd lldap        && ./create-secrets.sh  && cd ..
cd authelia     && ./create-secrets.sh  && cd ..
# Restore owner-managed RPF-WP-0040 factor delivery before first KeyCape creation.
# If keycape-config already exists, use the reviewed configuration-preserving restore.
cd keycape      && ./create-secrets.sh  && cd ..

# 4. Apply workloads in restore order
kubectl apply -f postgresql/cluster.yaml
kubectl apply -f privacyidea/{pvc.yaml,configmap.yaml,deployment.yaml,middleware.yaml,ingress.yaml}
kubectl apply -f lldap/{pvc.yaml,deployment.yaml,middleware.yaml,ingress.yaml}
kubectl apply -f authelia/{pvc.yaml,configmap.yaml,deployment.yaml,ingress.yaml}
kubectl apply -f keycape/{deployment.yaml,middleware.yaml,ingress.yaml}

# 5. Wait for everything to be Ready
kubectl rollout status deployment/privacyidea -n mfa --timeout=300s
kubectl rollout status deployment/lldap       -n sso --timeout=120s
kubectl rollout status deployment/authelia    -n sso --timeout=120s
kubectl rollout status deployment/keycape     -n sso --timeout=60s

# 6. Re-run bootstrap scripts if PVC data was lost
# Restore the original key set first using privacyidea/KEY-RECOVERY.md.
# Only then follow the reviewed account/realm reconciliation procedure.
cd privacyidea && ./enckey-bootstrap.sh /absolute/private/custody
# Restore and verify existing LLDAP identities from the owner-reviewed backup.
# Legacy break-glass creation is retired; do not overwrite existing passwords.
# For group preview/apply and structural diagnostics, follow lldap/GROUP-MAINTENANCE.md.
# Those checks do not prove native recovery or broader T07 completion.
# See ../../../docs/legacy-lldap-lifecycle-retirement.md for recovery limits.
# Verify owner-managed factor renewal, ESO delivery and mounted reload (RPF-WP-0040).
# Do not mint a personal admin token or replace existing KeyCape configuration.

# 7. Verify
./verify-t04.sh && ./verify-t05.sh && ./verify-t06.sh --user platform-root && ./verify-t08.sh
# Separately, owner-invoked LDAP structure observation (exit 2 means incomplete acceptance):
./verify-t07.sh --root /absolute/private/custody
# Actual lifecycle, login and independent recovery receipts remain required.

---

Backup offsite export

The SQLite backup files land on the PVCs but are not offsite until exported. Run this on the node host to pull them out and encrypt for offsite storage:

# Pull backup files from pods
kubectl exec -n sso deployment/lldap -- \
  cat /data/backups/users.backup.$(date +%Y-%m-%d) > /tmp/lldap-backup.db
kubectl exec -n sso deployment/authelia -- \
  cat /data/backups/authelia.backup.$(date +%Y-%m-%d) > /tmp/authelia-backup.db

# Encrypt with age and send offsite (same key as the ops bundle)
age -r "$(cat ~/net-kingdom-ops-bundle.key | grep 'public key' | awk '{print $NF}')" \
  -o /tmp/lldap-backup.db.age /tmp/lldap-backup.db

# Shred plaintext copies
shred -u /tmp/lldap-backup.db /tmp/authelia-backup.db