# Panic Sheet — Coulomb / Helix Recovery

**One page. Read this first in an incident.**  
**Updated:** 2026-07-07 · **Owner:** Bernd

`no_secret_material_recorded: true` — no passwords or tokens in this document.

---

## 1. Where are the backups?

| What | Where | Encrypted | Off-node |
| --- | --- | --- | --- |
| Custodian DB + dev config | Nextcloud file drop + `~/.cache/railiance/backups/` | age | Yes (Nextcloud) |
| k3s state + Helm values | `/opt/backup/railiance/cluster/` on railiance01 | age | **No** |
| OS config (S1) | `/opt/backup/railiance/infra/` on railiance01 | age | **No** |
| Forgejo / platform (intended) | Nextcloud file drop | age | Adopted, **not wired** |
| Gitea | Manual dump only | varies | **No** |

Details: `BackupPolicy.md`, `AssetRegister.yaml`.

Decrypt keys: password manager first; operational keys on lost machine are gone.

---

## 2. Who has access?

| System | Admin | Recovery |
| --- | --- | --- |
| DNS / domains | Bernd | IONOS panel — backup admin **not recorded** |
| Gitea | Bernd | coulombcore |
| Forgejo | Bernd | railiance01 k3s |
| Stripe | Bernd | Dashboard — MFA recovery **not recorded** |
| Backups (offsite) | Bernd | Nextcloud admin read; upload token in OpenBao |
| Secrets | Bernd | OpenBao unseal — see platform runbooks |
| SSH to fleet | Bernd | `warden sign` (ops-warden) |

---

## 3. How to rebuild (full server loss)

1. **Stop** — do not patch a compromised host; build clean.
2. **Provision** new machine (Ubuntu).
3. **Ansible** baseline from `railiance-infra`.
4. **Secrets** — recover OpenBao unseal material; unseal; rotate if breach.
5. **Data** — restore from age-decrypted backups (see `RestoreRunbooks/01-full-server-loss.md`).
6. **k3s** — redeploy Helm releases from values backup or GitOps.
7. **Validate** — health checks, smoke tests, representative workflow.
8. **DNS** — cutover A records in IONOS when confident.
9. **Evidence** — write `TestEvidence/restore-drill-YYYY-MM-DD.md`.

---

## 4. How to rotate secrets (breach or leak)

1. Isolate affected systems.
2. Revoke exposed credentials immediately.
3. Build **clean** replacement environment.
4. Restore data from backup **before compromise time**.
5. Rotate: OpenBao paths, Stripe keys, API keys, Gitea/Forgejo tokens, backup tokens.
6. Force session/password reset where applicable.
7. Log in State Hub + `TestEvidence/`.

Playbook: `IncidentPlaybooks/credential-leak.md`, `IncidentPlaybooks/ransomware.md`.

---

## 5. How to switch DNS

| Hostname | Production IP (2026-07-07) | Notes |
| --- | --- | --- |
| `*.coulomb.social` (railiance01 services) | `92.205.62.239` | forgejo, reuse, hub target |
| CoulombCore legacy | `92.205.130.254` | gitea until cutover |

1. Export current zone from IONOS before changes.
2. Lower TTL ahead of planned cutover if possible.
3. Update A records; verify with `dig +short <host> A`.
4. Validate HTTPS: `curl -fsS https://<host>/health` or equivalent.
5. Roll back to previous A record if validation fails.

---

## 6. Before any infrastructure work

```bash
bin/railiance preflight   # on dev workstation — must exit 0
```

---

## 7. Escalation and evidence

- Full runbook: `RestoreRunbooks/01-full-server-loss.md`
- Registers: `AssetRegister.yaml`, `DataRegister.yaml`
- Pickup queue (open backup work): `BackupPickupQueue.md`
- Forgejo restore: `railiance-infra/tools/forgejo-restore-drill.sh`