tegwick
d011073e16
Expand AssetRegister and DataRegister with T0/T4 stack assets. Add ResiliencePolicy, PanicSheet, full-server-loss runbook, and five incident playbooks. Update README with mirror status. Mark T02-T05, T07-T08 done; T01 and T06 in progress.
96 lines
No EOL
3.3 KiB
Markdown
96 lines
No EOL
3.3 KiB
Markdown
# Panic Sheet — Coulomb / Helix Recovery
|
|
|
|
**One page. Read this first in an incident.**
|
|
**Updated:** 2026-07-07 · **Owner:** Bernd
|
|
|
|
`no_secret_material_recorded: true` — no passwords or tokens in this document.
|
|
|
|
---
|
|
|
|
## 1. Where are the backups?
|
|
|
|
| What | Where | Encrypted | Off-node |
|
|
| --- | --- | --- | --- |
|
|
| Custodian DB + dev config | Nextcloud file drop + `~/.cache/railiance/backups/` | age | Yes (Nextcloud) |
|
|
| k3s state + Helm values | `/opt/backup/railiance/cluster/` on railiance01 | age | **No** |
|
|
| OS config (S1) | `/opt/backup/railiance/infra/` on railiance01 | age | **No** |
|
|
| Forgejo / platform (intended) | Nextcloud file drop | age | Adopted, **not wired** |
|
|
| Gitea | Manual dump only | varies | **No** |
|
|
|
|
Details: `BackupPolicy.md`, `AssetRegister.yaml`.
|
|
|
|
Decrypt keys: password manager first; operational keys on lost machine are gone.
|
|
|
|
---
|
|
|
|
## 2. Who has access?
|
|
|
|
| System | Admin | Recovery |
|
|
| --- | --- | --- |
|
|
| DNS / domains | Bernd | IONOS panel — backup admin **not recorded** |
|
|
| Gitea | Bernd | coulombcore |
|
|
| Forgejo | Bernd | railiance01 k3s |
|
|
| Stripe | Bernd | Dashboard — MFA recovery **not recorded** |
|
|
| Backups (offsite) | Bernd | Nextcloud admin read; upload token in OpenBao |
|
|
| Secrets | Bernd | OpenBao unseal — see platform runbooks |
|
|
| SSH to fleet | Bernd | `warden sign` (ops-warden) |
|
|
|
|
---
|
|
|
|
## 3. How to rebuild (full server loss)
|
|
|
|
1. **Stop** — do not patch a compromised host; build clean.
|
|
2. **Provision** new machine (Ubuntu).
|
|
3. **Ansible** baseline from `railiance-infra`.
|
|
4. **Secrets** — recover OpenBao unseal material; unseal; rotate if breach.
|
|
5. **Data** — restore from age-decrypted backups (see `RestoreRunbooks/01-full-server-loss.md`).
|
|
6. **k3s** — redeploy Helm releases from values backup or GitOps.
|
|
7. **Validate** — health checks, smoke tests, representative workflow.
|
|
8. **DNS** — cutover A records in IONOS when confident.
|
|
9. **Evidence** — write `TestEvidence/restore-drill-YYYY-MM-DD.md`.
|
|
|
|
---
|
|
|
|
## 4. How to rotate secrets (breach or leak)
|
|
|
|
1. Isolate affected systems.
|
|
2. Revoke exposed credentials immediately.
|
|
3. Build **clean** replacement environment.
|
|
4. Restore data from backup **before compromise time**.
|
|
5. Rotate: OpenBao paths, Stripe keys, API keys, Gitea/Forgejo tokens, backup tokens.
|
|
6. Force session/password reset where applicable.
|
|
7. Log in State Hub + `TestEvidence/`.
|
|
|
|
Playbook: `IncidentPlaybooks/credential-leak.md`, `IncidentPlaybooks/ransomware.md`.
|
|
|
|
---
|
|
|
|
## 5. How to switch DNS
|
|
|
|
| Hostname | Production IP (2026-07-07) | Notes |
|
|
| --- | --- | --- |
|
|
| `*.coulomb.social` (railiance01 services) | `92.205.62.239` | forgejo, reuse, hub target |
|
|
| CoulombCore legacy | `92.205.130.254` | gitea until cutover |
|
|
|
|
1. Export current zone from IONOS before changes.
|
|
2. Lower TTL ahead of planned cutover if possible.
|
|
3. Update A records; verify with `dig +short <host> A`.
|
|
4. Validate HTTPS: `curl -fsS https://<host>/health` or equivalent.
|
|
5. Roll back to previous A record if validation fails.
|
|
|
|
---
|
|
|
|
## 6. Before any infrastructure work
|
|
|
|
```bash
|
|
bin/railiance preflight # on dev workstation — must exit 0
|
|
```
|
|
|
|
---
|
|
|
|
## 7. Escalation and evidence
|
|
|
|
- Full runbook: `RestoreRunbooks/01-full-server-loss.md`
|
|
- Registers: `AssetRegister.yaml`, `DataRegister.yaml`
|
|
- Pickup queue (open backup work): `BackupPickupQueue.md`
|
|
- Forgejo restore: `railiance-infra/tools/forgejo-restore-drill.sh` |