disaster-control/RestoreRunbooks/01-full-server-loss.md

Restore Runbook: Full Server Loss

Trigger: disk failure, VPS deletion, hardware loss, unrecoverable OS corruption.
RTO target: ≤ 1 day full recovery; ≤ 4–8h for code/content/control plane.
Owner: Bernd

Do not restore into a compromised machine. This runbook assumes hardware loss, not active breach (see IncidentPlaybooks/ransomware.md for breach).

---

Prerequisites

• Password-manager copy of age backup keys
• OpenBao unseal material (if restoring railiance01 secrets)
• Nextcloud admin or local backup cache access
• IONOS DNS access
• warden sign SSH access to new host
• This repo cloned (from mirror once T01 complete; until then: any copy)

---

Phase 1 — Provision clean host

1. Provision new Ubuntu server (or recover WSL2 workstation for dev lane only).
2. Apply SSH hardening via Ansible (railiance-infra).
3. Install tooling: age, docker, k3s, helm, kubectl, ansible.
4. Record new host identity in AssetRegister.yaml after recovery.

---

Phase 2 — Secret recovery

1. Retrieve OpenBao unseal keys from sealed recovery vault (password manager).
2. If railiance01: follow railiance-platform OpenBao deploy/unseal runbooks.
3. Retrieve backup upload credentials from OpenBao if re-wiring backup jobs.
4. Do not copy plaintext secrets into Git, chat, or State Hub.

---

Phase 3 — Data restore (by lane)

Dev workstation (bnt-lap001)

1. Download latest db-*.sql.age and config-*.tar.gz.age from Nextcloud.
2. age -d -i ~/.config/age/railiance-backup.key -o dump.sql db-*.sql.age
3. Restore PostgreSQL custodian database into infra-postgres-1.
4. Decrypt and unpack config tar to $HOME.
5. Verify: bin/railiance preflight exits 0.

Reference: railiance-cluster/docs/backup-restore.md

railiance01 — S1 OS config (optional)

1. Decrypt os-config-*.tar.gz.age from /opt/backup/railiance/infra/ (if disk survived) or offsite copy when wired.
2. Restore files or re-apply Ansible playbooks (preferred).

railiance01 — S2 k8s runtime

1. If rebuilding k3s: install k3s on clean host.
2. Decrypt helm-values-*.tar.gz.age — redeploy each release with saved values.
3. Decrypt kubeconfig-*.yaml.age if needed for admin access.
4. k3s state.db hot backup: prefer redeploy from Git/Helm over raw DB restore.

Reference: railiance-cluster/tools/cmd/railiance-backup-s2

Gitea (coulombcore)

1. If coulombcore lost: provision host, install Gitea via Ansible.
2. Restore from latest gitea dump or git mirrors of critical repos.
3. Verify: clone coulomb/disaster-control, API health.

Forgejo (railiance01)

1. If automated backup wired: download latest dump from Nextcloud.
2. Run railiance-infra/tools/forgejo-restore-drill.sh (adapt for production namespace).
3. Verify: GET /api/v1/version, org repos visible.

Evidence: TestEvidence/restore-drill-2026-07-04-forgejo.md

Platform databases

Blocked until BackupPickupQueue.md items 3–4 close (platform tool missing).

---

Phase 4 — Validate services

Check	Command / endpoint
k3s nodes	kubectl get nodes
Forgejo health	curl -fsS https://forgejo.coulomb.social/api/v1/version
Gitea health	curl -fsS https://gitea.coulomb.social/api/v1/version
State Hub	curl -fsS http://127.0.0.1:8000/state/summary
Representative app	App-specific smoke (e.g. vergabe health endpoint)

---

Phase 5 — DNS cutover

1. Confirm all services healthy on new infrastructure.
2. Update IONOS A records per PanicSheet.md §5.
3. Wait for TTL propagation; verify externally (dig @8.8.8.8).
4. Monitor for 24h.

---

Phase 6 — Evidence report

Write TestEvidence/restore-drill-YYYY-MM-DD-full-server-loss.md:

• Start/end times (RTO measured)
• Backup artifacts used (RPO achieved)
• Systems restored vs redeployed
• Issues and follow-up actions
• Update AssetRegister.yaml if host identities changed

---

References

• PanicSheet.md
• BackupPolicy.md
• AssetRegister.yaml
• RTO-RPO-Matrix.md