tegwick
d011073e16
Expand AssetRegister and DataRegister with T0/T4 stack assets. Add ResiliencePolicy, PanicSheet, full-server-loss runbook, and five incident playbooks. Update README with mirror status. Mark T02-T05, T07-T08 done; T01 and T06 in progress.
126 lines
No EOL
4 KiB
Markdown
126 lines
No EOL
4 KiB
Markdown
# Restore Runbook: Full Server Loss
|
||
|
||
**Trigger:** disk failure, VPS deletion, hardware loss, unrecoverable OS corruption.
|
||
**RTO target:** ≤ 1 day full recovery; ≤ 4–8h for code/content/control plane.
|
||
**Owner:** Bernd
|
||
|
||
Do **not** restore into a compromised machine. This runbook assumes hardware
|
||
loss, not active breach (see `IncidentPlaybooks/ransomware.md` for breach).
|
||
|
||
---
|
||
|
||
## Prerequisites
|
||
|
||
- [ ] Password-manager copy of age backup keys
|
||
- [ ] OpenBao unseal material (if restoring railiance01 secrets)
|
||
- [ ] Nextcloud admin or local backup cache access
|
||
- [ ] IONOS DNS access
|
||
- [ ] `warden sign` SSH access to new host
|
||
- [ ] This repo cloned (from mirror once T01 complete; until then: any copy)
|
||
|
||
---
|
||
|
||
## Phase 1 — Provision clean host
|
||
|
||
1. Provision new Ubuntu server (or recover WSL2 workstation for dev lane only).
|
||
2. Apply SSH hardening via Ansible (`railiance-infra`).
|
||
3. Install tooling: `age`, `docker`, `k3s`, `helm`, `kubectl`, `ansible`.
|
||
4. Record new host identity in `AssetRegister.yaml` after recovery.
|
||
|
||
---
|
||
|
||
## Phase 2 — Secret recovery
|
||
|
||
1. Retrieve OpenBao unseal keys from sealed recovery vault (password manager).
|
||
2. If railiance01: follow `railiance-platform` OpenBao deploy/unseal runbooks.
|
||
3. Retrieve backup upload credentials from OpenBao if re-wiring backup jobs.
|
||
4. **Do not** copy plaintext secrets into Git, chat, or State Hub.
|
||
|
||
---
|
||
|
||
## Phase 3 — Data restore (by lane)
|
||
|
||
### Dev workstation (bnt-lap001)
|
||
|
||
1. Download latest `db-*.sql.age` and `config-*.tar.gz.age` from Nextcloud.
|
||
2. `age -d -i ~/.config/age/railiance-backup.key -o dump.sql db-*.sql.age`
|
||
3. Restore PostgreSQL `custodian` database into `infra-postgres-1`.
|
||
4. Decrypt and unpack config tar to `$HOME`.
|
||
5. Verify: `bin/railiance preflight` exits 0.
|
||
|
||
Reference: `railiance-cluster/docs/backup-restore.md`
|
||
|
||
### railiance01 — S1 OS config (optional)
|
||
|
||
1. Decrypt `os-config-*.tar.gz.age` from `/opt/backup/railiance/infra/` (if disk survived)
|
||
or offsite copy when wired.
|
||
2. Restore files or re-apply Ansible playbooks (preferred).
|
||
|
||
### railiance01 — S2 k8s runtime
|
||
|
||
1. If rebuilding k3s: install k3s on clean host.
|
||
2. Decrypt `helm-values-*.tar.gz.age` — redeploy each release with saved values.
|
||
3. Decrypt `kubeconfig-*.yaml.age` if needed for admin access.
|
||
4. k3s `state.db` hot backup: prefer **redeploy from Git/Helm** over raw DB restore.
|
||
|
||
Reference: `railiance-cluster/tools/cmd/railiance-backup-s2`
|
||
|
||
### Gitea (coulombcore)
|
||
|
||
1. If coulombcore lost: provision host, install Gitea via Ansible.
|
||
2. Restore from latest `gitea dump` or git mirrors of critical repos.
|
||
3. Verify: clone `coulomb/disaster-control`, API health.
|
||
|
||
### Forgejo (railiance01)
|
||
|
||
1. If automated backup wired: download latest dump from Nextcloud.
|
||
2. Run `railiance-infra/tools/forgejo-restore-drill.sh` (adapt for production namespace).
|
||
3. Verify: `GET /api/v1/version`, org repos visible.
|
||
|
||
Evidence: `TestEvidence/restore-drill-2026-07-04-forgejo.md`
|
||
|
||
### Platform databases
|
||
|
||
Blocked until `BackupPickupQueue.md` items 3–4 close (platform tool missing).
|
||
|
||
---
|
||
|
||
## Phase 4 — Validate services
|
||
|
||
| Check | Command / endpoint |
|
||
| --- | --- |
|
||
| k3s nodes | `kubectl get nodes` |
|
||
| Forgejo health | `curl -fsS https://forgejo.coulomb.social/api/v1/version` |
|
||
| Gitea health | `curl -fsS https://gitea.coulomb.social/api/v1/version` |
|
||
| State Hub | `curl -fsS http://127.0.0.1:8000/state/summary` |
|
||
| Representative app | App-specific smoke (e.g. vergabe health endpoint) |
|
||
|
||
---
|
||
|
||
## Phase 5 — DNS cutover
|
||
|
||
1. Confirm all services healthy on new infrastructure.
|
||
2. Update IONOS A records per `PanicSheet.md` §5.
|
||
3. Wait for TTL propagation; verify externally (`dig @8.8.8.8`).
|
||
4. Monitor for 24h.
|
||
|
||
---
|
||
|
||
## Phase 6 — Evidence report
|
||
|
||
Write `TestEvidence/restore-drill-YYYY-MM-DD-full-server-loss.md`:
|
||
|
||
- Start/end times (RTO measured)
|
||
- Backup artifacts used (RPO achieved)
|
||
- Systems restored vs redeployed
|
||
- Issues and follow-up actions
|
||
- Update `AssetRegister.yaml` if host identities changed
|
||
|
||
---
|
||
|
||
## References
|
||
|
||
- `PanicSheet.md`
|
||
- `BackupPolicy.md`
|
||
- `AssetRegister.yaml`
|
||
- `RTO-RPO-Matrix.md` |