net-kingdom/docs/disaster-recovery-storage-share.md

# NetKingdom disaster recovery — start here

Updated 2026-10-27. Owner: NetKingdom with Railiance Platform and the individual
identity service owners. This guide covers the identity backup, not whole-host,
OpenBao, Forgejo, User Engine or application recovery.

## If the identity services are down

1. Use a trusted recovery workstation. Obtain Storage Share access independently
   from your password safe/offline custody, and the independently held
   `railiance-backup` age private key (crisis list item 1; **not** the custodian
   age key, item 1b, which only opens the repository's SOPS files). Never paste either into chat or a ticket.
   The private key must match this **public** recipient:
   `age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz`.
   The provider password and encryption key are separate requirements.
2. Open <https://nx4069.your-storageshare.de> using the independent provider
   account, not NetKingdom SSO. In the Backup account, navigate to
   `railiance-backups/net-kingdom-identity/`. Recovery documentation and a small
   versioned recovery-tools ZIP are placed beside the encrypted archives.
   Keep a copy of this guide and ZIP outside Storage Share as well.
3. Choose an intact recovery generation. The full identity filename starts
   `net-kingdom-identity-` and ends `.tar.age`. A `privacyidea_db-*.dump.age`
   database-only backup cannot replace the matching service keys and configs.
   Each generation has a `.receipt.json` index. Preserve the original objects.
4. Check timestamp, size and SHA256 against a separately trusted receipt. The
   index is useful metadata, not a signed independent trust anchor. For the
   first actually recovered, protected generation the trusted values are:

   - Artifact: `net-kingdom-identity-20261016T063159Z-5882771aa2ce.tar.age`
   - Bytes: `109738344`
   - SHA256: `7117b3fe1e4840c86b301c94c85e6b16709e5626bb0884bd0580bde630e30e30`
   - Strong ETag at publication: `"28ec1f7014e492de4c40e1ae75502f08"`

   This is a known good fallback, not a claim that it is the latest backup.
   For a newer generation, reconcile its index with the independently retained
   publisher receipt before constructing a new download input. Never edit a
   receipt just to make a failed integrity check pass. ETag changes require
   owner reconciliation even if the filename is unchanged.
5. On Linux with Python 3, unpack the recovery-tools ZIP into a private working
   directory and enter it. Verify its `SHA256SUMS` against a trusted repository
   copy or independently held kit digest before executing code. A checksum
   stored only alongside the download detects damage, not a malicious provider.
   Alternatively use an independently available trusted net-kingdom checkout.

   ```bash
   python3 tools/bootstrap-recovery/retrieve-identity-offline.py \
     --source-receipt docs/evidence/2026-10-16-identity-backup-live/actual-offsite.json \
     --auth account --receipt /tmp/nk49-outage-transfer.json
   ```

   In your own terminal, enter the independently held **Backup** account app
   password at the hidden prompt. The fixed account name is `Backup`.
   If your independently held credential is a read-capable public share token
   for this exact identity folder, use `--auth folder-share` instead. An
   upload-only token, another account's password or a share for the backup root
   is insufficient for this command. A 401/403/404 stops; do not widen permissions
   or substitute another token silently. Download performs only HEAD/GET against
   the fixed provider, rejects redirects/proxies, and verifies ETag/size/hash.
   Failure removes its incomplete local download. Success leaves ciphertext
   in a private `/tmp/nk49-identity-fetched-*` directory named in the receipt.
6. Verify and decrypt through the existing hidden key helper. Requirements:
   `age`, `age-keygen`, `openssl`, Python 3 and Docker; the following PostgreSQL
   image must already be present on the recovery host:
   `postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70`.
   Stage and verify this image before a drill. The tools ZIP contains no images.
   Allow enough private disk and RAM for a roughly 110 MB ciphertext, a bounded
   512 MiB archive and the disposable PostgreSQL container (2 GiB memory limit).

   ```bash
   bash sso-mfa/bootstrap/sops-custody-unlock.sh \
     --expected-recipient age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz \
     --prompt -- \
     python3 tools/bootstrap-recovery/verify-identity.py \
       --transfer-receipt /tmp/nk49-outage-transfer.json \
       --receipt /tmp/nk49-outage-archive-recovery.json
   ```

   Use new receipt names on every run. The helper removes its temporary private
   key on exit. The verifier checks the encrypted manifest and all file hashes,
   original key pairs, SQLite integrity, wrong-key refusal and an isolated
   PostgreSQL restore. It removes decrypted staging and the disposable database.
   It deliberately does not install restored secrets or alter production.
7. Preserve the sanitized receipts and original ciphertext. Remove your local
   downloaded ciphertext directory when no longer needed. If the verifier fails,
   stop and check the bounded receipt with the service owner; do not regenerate
   encryption/signing keys, reset user passwords or delete the provider backup.

## What the backup contains

| Component | Recovery material |
| --- | --- |
| LLDAP | Native SQLite snapshot, remaining durable data, deployment and referenced secrets |
| Authelia | Native SQLite snapshot, encryption/session/OIDC material, current configuration and referenced secrets |
| KeyCape | Original signing/configuration material, client configuration, authentication policies and receipts |
| privacyIDEA | Fresh PostgreSQL dump, original encryption key, matching original audit RSA key pair, current configuration and required durable files |

The encrypted manifest records exact files and capture intervals. This is not
a global atomic snapshot of all services. Historical backups and selected
notification/log files are excluded as documented by the capture profile.
The backup contains sensitive service credentials; only ciphertext belongs on
ordinary persistent recovery media. The age private key is never in the bundle.

## Restoring working services

**Archive recovery passed on 2026-10-16. Isolated restored-service recovery passed
for LLDAP, Authelia, privacyIDEA (MFA and audit), KeyCape (issuer side) and OpenBao
by 2026-10-24, and was owner-accepted in builder mode on 2026-10-27 (NK-WP-0049).**
A joint restore with a live consumer login is still a pre-production drill
(NK-WP-0055-T02/T03). The archive verifier deletes its plaintext and is not a
service importer. Follow `docs/identity-service-restore-packet.md` and the
[identity restore drill](identity-restore-drill.md) for a service drill.

Restore the exact matching data/config/original keys into an isolated recovery
environment first. Keep production ingress, public DNS, notifications, external
provider writes and real consumer traffic disconnected. Establish directory and
database readiness; prove LLDAP existing-user authentication; then privacyIDEA
retained-token MFA/audit continuity, Authelia storage/OIDC, and KeyCape/current
consumer continuity according to their actual dependencies. Existing user/MFA
inputs stay in operator custody. Public issuer identity must be preserved while
test traffic is routed privately; do not change production DNS for a drill.

Only after owner acceptance of native positive/negative tests should the owners
approve a production cutover, including target identity, rollback, DNS/ingress,
current consumers, credential dispositions and monitoring. This guide does not
authorize a production restore or automatic credential rotation.

## Reaching the platform when SSO and OpenBao are down

The normal admin paths need the services being recovered: KeyCape OIDC with
privacyIDEA MFA for OpenBao and portals, and ops-warden SSH certificates, which
are signed by OpenBao. With those down, only paths built on offline material work:

- **Cluster:** the attended admin kubeconfig (`~/.kube/config-railiance01`,
  client certificate) through the `k3s-api-railiance01` tunnel, which uses a
  static SSH key. Today both are on the founder workstation only.
- **OpenBao:** unseal with the shares (crisis list item 3). If an admin token is
  needed while OIDC is down, the shares mint a short-lived root token with the
  `generate-root` ceremony in `railiance-platform/docs/openbao.md` ("Break-glass:
  OpenBao with SSO down"). On production it runs against the loopback-only
  listener `127.0.0.1:8210`, reached with `kubectl port-forward` (live since
  2026-10-28, RAPP-OPENBAO-WP-0003). Revoke the token afterwards.
- **Host:** console and rescue access through the Host Europe provider
  interface, which is independent of the platform.
- **Workstation lost:** an age-encrypted copy of the cluster break-glass
  material is kept on the Storage Share `Backup` account under
  `railiance-backups/net-kingdom-kingpath/` (`tools/bootstrap-recovery/kingpath-offsite.py`,
  NK-WP-0055-T13); decrypting it needs only crisis list items 1 and 2.

Every use is a break-glass event: record what was done and why for review.
Item-level custody and open gaps are in the crisis custody list; drilling this
path is NK-WP-0055-T13.

## Who and what unlocks recovery

The [crisis custody list](crisis-custody-list.md) names every item that unlocks recovery, who holds
it and whether it still works with SSO and OpenBao down. It holds no secret values.

## Schedule, protection and limits

Daily capture is scheduled at 03:45 UTC with up to five minutes jitter. Hourly
host-local checks flag a backup older than 25 hours. Retention protects the
independently recovered pinned generation, newest generation, seven distinct
daily and two weekly points; unknown/unindexed objects are not auto-deleted.

At this revision the full identity backup has one provider copy on Storage Share.
Existing Scaleway database backups do not establish a second full identity copy.
Host-local health cannot report a dead host. Full service login/MFA/audit proofs,
an independently executed outage-path retrieval drill, a second provider copy
and external host-loss monitoring must not be assumed from an upload or this
guide. Track remaining drills in NK-WP-0053 and production readiness in
NK-WP-0055; current task state lives in the repository and may be newer than
this offline guide.