net-kingdom/docs/disaster-recovery-storage-share.md

NetKingdom disaster recovery — start here

Updated 2026-10-27. Owner: NetKingdom with Railiance Platform and the individual identity service owners. This guide covers the identity backup, not whole-host, OpenBao, Forgejo, User Engine or application recovery.

If the identity services are down

1. Use a trusted recovery workstation. Obtain Storage Share access independently from your password safe/offline custody, and the independently held railiance-backup age private key (crisis list item 1; not the custodian age key, item 1b, which only opens the repository's SOPS files). Never paste either into chat or a ticket. The private key must match this public recipient: age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz. The provider password and encryption key are separate requirements.

2. Open https://nx4069.your-storageshare.de using the independent provider account, not NetKingdom SSO. In the Backup account, navigate to railiance-backups/net-kingdom-identity/. Recovery documentation and a small versioned recovery-tools ZIP are placed beside the encrypted archives. Keep a copy of this guide and ZIP outside Storage Share as well.

3. Choose an intact recovery generation. The full identity filename starts net-kingdom-identity- and ends .tar.age. A privacyidea_db-*.dump.age database-only backup cannot replace the matching service keys and configs. Each generation has a .receipt.json index. Preserve the original objects.

4. Check timestamp, size and SHA256 against a separately trusted receipt. The index is useful metadata, not a signed independent trust anchor. For the first actually recovered, protected generation the trusted values are:

   • Artifact: net-kingdom-identity-20261016T063159Z-5882771aa2ce.tar.age
   • Bytes: 109738344
   • SHA256: 7117b3fe1e4840c86b301c94c85e6b16709e5626bb0884bd0580bde630e30e30
   • Strong ETag at publication: "28ec1f7014e492de4c40e1ae75502f08"

   This is a known good fallback, not a claim that it is the latest backup. For a newer generation, reconcile its index with the independently retained publisher receipt before constructing a new download input. Never edit a receipt just to make a failed integrity check pass. ETag changes require owner reconciliation even if the filename is unchanged.

5. On Linux with Python 3, unpack the recovery-tools ZIP into a private working directory and enter it. Verify its SHA256SUMS against a trusted repository copy or independently held kit digest before executing code. A checksum stored only alongside the download detects damage, not a malicious provider. Alternatively use an independently available trusted net-kingdom checkout.

   python3 tools/bootstrap-recovery/retrieve-identity-offline.py \
     --source-receipt docs/evidence/2026-10-16-identity-backup-live/actual-offsite.json \
     --auth account --receipt /tmp/nk49-outage-transfer.json
   

   In your own terminal, enter the independently held Backup account app password at the hidden prompt. The fixed account name is Backup. If your independently held credential is a read-capable public share token for this exact identity folder, use --auth folder-share instead. An upload-only token, another account's password or a share for the backup root is insufficient for this command. A 401/403/404 stops; do not widen permissions or substitute another token silently. Download performs only HEAD/GET against the fixed provider, rejects redirects/proxies, and verifies ETag/size/hash. Failure removes its incomplete local download. Success leaves ciphertext in a private /tmp/nk49-identity-fetched-* directory named in the receipt.

6. Verify and decrypt through the existing hidden key helper. Requirements: age, age-keygen, openssl, Python 3 and Docker; the following PostgreSQL image must already be present on the recovery host: postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70. Stage and verify this image before a drill. The tools ZIP contains no images. Allow enough private disk and RAM for a roughly 110 MB ciphertext, a bounded 512 MiB archive and the disposable PostgreSQL container (2 GiB memory limit).

   bash sso-mfa/bootstrap/sops-custody-unlock.sh \
     --expected-recipient age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz \
     --prompt -- \
     python3 tools/bootstrap-recovery/verify-identity.py \
       --transfer-receipt /tmp/nk49-outage-transfer.json \
       --receipt /tmp/nk49-outage-archive-recovery.json
   

   Use new receipt names on every run. The helper removes its temporary private key on exit. The verifier checks the encrypted manifest and all file hashes, original key pairs, SQLite integrity, wrong-key refusal and an isolated PostgreSQL restore. It removes decrypted staging and the disposable database. It deliberately does not install restored secrets or alter production.

7. Preserve the sanitized receipts and original ciphertext. Remove your local downloaded ciphertext directory when no longer needed. If the verifier fails, stop and check the bounded receipt with the service owner; do not regenerate encryption/signing keys, reset user passwords or delete the provider backup.

What the backup contains

Component	Recovery material
LLDAP	Native SQLite snapshot, remaining durable data, deployment and referenced secrets
Authelia	Native SQLite snapshot, encryption/session/OIDC material, current configuration and referenced secrets
KeyCape	Original signing/configuration material, client configuration, authentication policies and receipts
privacyIDEA	Fresh PostgreSQL dump, original encryption key, matching original audit RSA key pair, current configuration and required durable files

The encrypted manifest records exact files and capture intervals. This is not a global atomic snapshot of all services. Historical backups and selected notification/log files are excluded as documented by the capture profile. The backup contains sensitive service credentials; only ciphertext belongs on ordinary persistent recovery media. The age private key is never in the bundle.

Restoring working services

Archive recovery passed on 2026-10-16. Isolated restored-service recovery passed for LLDAP, Authelia, privacyIDEA (MFA and audit), KeyCape (issuer side) and OpenBao by 2026-10-24, and was owner-accepted in builder mode on 2026-10-27 (NK-WP-0049). A joint restore with a live consumer login is still a pre-production drill (NK-WP-0055-T02/T03). The archive verifier deletes its plaintext and is not a service importer. Follow docs/identity-service-restore-packet.md and the identity restore drill for a service drill.

Restore the exact matching data/config/original keys into an isolated recovery environment first. Keep production ingress, public DNS, notifications, external provider writes and real consumer traffic disconnected. Establish directory and database readiness; prove LLDAP existing-user authentication; then privacyIDEA retained-token MFA/audit continuity, Authelia storage/OIDC, and KeyCape/current consumer continuity according to their actual dependencies. Existing user/MFA inputs stay in operator custody. Public issuer identity must be preserved while test traffic is routed privately; do not change production DNS for a drill.

Only after owner acceptance of native positive/negative tests should the owners approve a production cutover, including target identity, rollback, DNS/ingress, current consumers, credential dispositions and monitoring. This guide does not authorize a production restore or automatic credential rotation.

Reaching the platform when SSO and OpenBao are down

The normal admin paths need the services being recovered: KeyCape OIDC with privacyIDEA MFA for OpenBao and portals, and ops-warden SSH certificates, which are signed by OpenBao. With those down, only paths built on offline material work:

• Cluster: the attended admin kubeconfig (~/.kube/config-railiance01, client certificate) through the k3s-api-railiance01 tunnel, which uses a static SSH key. Today both are on the founder workstation only.
• OpenBao: unseal with the shares (crisis list item 3). If an admin token is needed while OIDC is down, the shares mint a short-lived root token with the generate-root ceremony in railiance-platform/docs/openbao.md ("Break-glass: OpenBao with SSO down"). On production it runs against the loopback-only listener 127.0.0.1:8210, reached with kubectl port-forward (live since 2026-10-28, RAPP-OPENBAO-WP-0003). Revoke the token afterwards.
• Host: console and rescue access through the Host Europe provider interface, which is independent of the platform.
• Workstation lost: an age-encrypted copy of the cluster break-glass material is kept on the Storage Share Backup account under railiance-backups/net-kingdom-kingpath/ (tools/bootstrap-recovery/kingpath-offsite.py, NK-WP-0055-T13); decrypting it needs only crisis list items 1 and 2.

Every use is a break-glass event: record what was done and why for review. Item-level custody and open gaps are in the crisis custody list; drilling this path is NK-WP-0055-T13.

Who and what unlocks recovery

The crisis custody list names every item that unlocks recovery, who holds it and whether it still works with SSO and OpenBao down. It holds no secret values.

Schedule, protection and limits

Daily capture is scheduled at 03:45 UTC with up to five minutes jitter. Hourly host-local checks flag a backup older than 25 hours. Retention protects the independently recovered pinned generation, newest generation, seven distinct daily and two weekly points; unknown/unindexed objects are not auto-deleted.

At this revision the full identity backup has one provider copy on Storage Share. Existing Scaleway database backups do not establish a second full identity copy. Host-local health cannot report a dead host. Full service login/MFA/audit proofs, an independently executed outage-path retrieval drill, a second provider copy and external host-loss monitoring must not be assumed from an upload or this guide. Track remaining drills in NK-WP-0053 and production readiness in NK-WP-0055; current task state lives in the repository and may be newer than this offline guide.