coulomb/disaster-control
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
disaster-control/PanicSheet.md
tegwick d011073e16 Start DISCTL-WP-0002: policies, registers, runbooks, playbooks 
Expand AssetRegister and DataRegister with T0/T4 stack assets.
Add ResiliencePolicy, PanicSheet, full-server-loss runbook, and five
incident playbooks. Update README with mirror status. Mark T02-T05,
T07-T08 done; T01 and T06 in progress.
2026-07-07 20:04:54 +02:00

3.3 KiB
Raw Permalink Blame History

Panic Sheet — Coulomb / Helix Recovery

One page. Read this first in an incident.
Updated: 2026-07-07 · Owner: Bernd

no_secret_material_recorded: true — no passwords or tokens in this document.

1. Where are the backups?

What Where Encrypted Off-node
Custodian DB + dev config Nextcloud file drop + ~/.cache/railiance/backups/ age Yes (Nextcloud)
k3s state + Helm values /opt/backup/railiance/cluster/ on railiance01 age No
OS config (S1) /opt/backup/railiance/infra/ on railiance01 age No
Forgejo / platform (intended) Nextcloud file drop age Adopted, not wired
Gitea Manual dump only varies No

Details: BackupPolicy.md, AssetRegister.yaml.

Decrypt keys: password manager first; operational keys on lost machine are gone.

2. Who has access?

System Admin Recovery
DNS / domains Bernd IONOS panel — backup admin not recorded
Gitea Bernd coulombcore
Forgejo Bernd railiance01 k3s
Stripe Bernd Dashboard — MFA recovery not recorded
Backups (offsite) Bernd Nextcloud admin read; upload token in OpenBao
Secrets Bernd OpenBao unseal — see platform runbooks
SSH to fleet Bernd warden sign (ops-warden)

3. How to rebuild (full server loss)

  1. Stop — do not patch a compromised host; build clean.
  2. Provision new machine (Ubuntu).
  3. Ansible baseline from railiance-infra.
  4. Secrets — recover OpenBao unseal material; unseal; rotate if breach.
  5. Data — restore from age-decrypted backups (see RestoreRunbooks/01-full-server-loss.md).
  6. k3s — redeploy Helm releases from values backup or GitOps.
  7. Validate — health checks, smoke tests, representative workflow.
  8. DNS — cutover A records in IONOS when confident.
  9. Evidence — write TestEvidence/restore-drill-YYYY-MM-DD.md.

4. How to rotate secrets (breach or leak)

  1. Isolate affected systems.
  2. Revoke exposed credentials immediately.
  3. Build clean replacement environment.
  4. Restore data from backup before compromise time.
  5. Rotate: OpenBao paths, Stripe keys, API keys, Gitea/Forgejo tokens, backup tokens.
  6. Force session/password reset where applicable.
  7. Log in State Hub + TestEvidence/.

Playbook: IncidentPlaybooks/credential-leak.md, IncidentPlaybooks/ransomware.md.

5. How to switch DNS

Hostname Production IP (2026-07-07) Notes
*.coulomb.social (railiance01 services) 92.205.62.239 forgejo, reuse, hub target
CoulombCore legacy 92.205.130.254 gitea until cutover
  1. Export current zone from IONOS before changes.
  2. Lower TTL ahead of planned cutover if possible.
  3. Update A records; verify with dig +short <host> A.
  4. Validate HTTPS: curl -fsS https://<host>/health or equivalent.
  5. Roll back to previous A record if validation fails.

6. Before any infrastructure work

bin/railiance preflight   # on dev workstation — must exit 0

7. Escalation and evidence

  • Full runbook: RestoreRunbooks/01-full-server-loss.md
  • Registers: AssetRegister.yaml, DataRegister.yaml
  • Pickup queue (open backup work): BackupPickupQueue.md
  • Forgejo restore: railiance-infra/tools/forgejo-restore-drill.sh