Start DISCTL-WP-0002: policies, registers, runbooks, playbooks
Expand AssetRegister and DataRegister with T0/T4 stack assets. Add ResiliencePolicy, PanicSheet, full-server-loss runbook, and five incident playbooks. Update README with mirror status. Mark T02-T05, T07-T08 done; T01 and T06 in progress.
This commit is contained in:
parent
4288ad21a9
commit
d011073e16
13 changed files with 814 additions and 29 deletions
96
PanicSheet.md
Normal file
96
PanicSheet.md
Normal file
|
|
@ -0,0 +1,96 @@
|
|||
# Panic Sheet — Coulomb / Helix Recovery
|
||||
|
||||
**One page. Read this first in an incident.**
|
||||
**Updated:** 2026-07-07 · **Owner:** Bernd
|
||||
|
||||
`no_secret_material_recorded: true` — no passwords or tokens in this document.
|
||||
|
||||
---
|
||||
|
||||
## 1. Where are the backups?
|
||||
|
||||
| What | Where | Encrypted | Off-node |
|
||||
| --- | --- | --- | --- |
|
||||
| Custodian DB + dev config | Nextcloud file drop + `~/.cache/railiance/backups/` | age | Yes (Nextcloud) |
|
||||
| k3s state + Helm values | `/opt/backup/railiance/cluster/` on railiance01 | age | **No** |
|
||||
| OS config (S1) | `/opt/backup/railiance/infra/` on railiance01 | age | **No** |
|
||||
| Forgejo / platform (intended) | Nextcloud file drop | age | Adopted, **not wired** |
|
||||
| Gitea | Manual dump only | varies | **No** |
|
||||
|
||||
Details: `BackupPolicy.md`, `AssetRegister.yaml`.
|
||||
|
||||
Decrypt keys: password manager first; operational keys on lost machine are gone.
|
||||
|
||||
---
|
||||
|
||||
## 2. Who has access?
|
||||
|
||||
| System | Admin | Recovery |
|
||||
| --- | --- | --- |
|
||||
| DNS / domains | Bernd | IONOS panel — backup admin **not recorded** |
|
||||
| Gitea | Bernd | coulombcore |
|
||||
| Forgejo | Bernd | railiance01 k3s |
|
||||
| Stripe | Bernd | Dashboard — MFA recovery **not recorded** |
|
||||
| Backups (offsite) | Bernd | Nextcloud admin read; upload token in OpenBao |
|
||||
| Secrets | Bernd | OpenBao unseal — see platform runbooks |
|
||||
| SSH to fleet | Bernd | `warden sign` (ops-warden) |
|
||||
|
||||
---
|
||||
|
||||
## 3. How to rebuild (full server loss)
|
||||
|
||||
1. **Stop** — do not patch a compromised host; build clean.
|
||||
2. **Provision** new machine (Ubuntu).
|
||||
3. **Ansible** baseline from `railiance-infra`.
|
||||
4. **Secrets** — recover OpenBao unseal material; unseal; rotate if breach.
|
||||
5. **Data** — restore from age-decrypted backups (see `RestoreRunbooks/01-full-server-loss.md`).
|
||||
6. **k3s** — redeploy Helm releases from values backup or GitOps.
|
||||
7. **Validate** — health checks, smoke tests, representative workflow.
|
||||
8. **DNS** — cutover A records in IONOS when confident.
|
||||
9. **Evidence** — write `TestEvidence/restore-drill-YYYY-MM-DD.md`.
|
||||
|
||||
---
|
||||
|
||||
## 4. How to rotate secrets (breach or leak)
|
||||
|
||||
1. Isolate affected systems.
|
||||
2. Revoke exposed credentials immediately.
|
||||
3. Build **clean** replacement environment.
|
||||
4. Restore data from backup **before compromise time**.
|
||||
5. Rotate: OpenBao paths, Stripe keys, API keys, Gitea/Forgejo tokens, backup tokens.
|
||||
6. Force session/password reset where applicable.
|
||||
7. Log in State Hub + `TestEvidence/`.
|
||||
|
||||
Playbook: `IncidentPlaybooks/credential-leak.md`, `IncidentPlaybooks/ransomware.md`.
|
||||
|
||||
---
|
||||
|
||||
## 5. How to switch DNS
|
||||
|
||||
| Hostname | Production IP (2026-07-07) | Notes |
|
||||
| --- | --- | --- |
|
||||
| `*.coulomb.social` (railiance01 services) | `92.205.62.239` | forgejo, reuse, hub target |
|
||||
| CoulombCore legacy | `92.205.130.254` | gitea until cutover |
|
||||
|
||||
1. Export current zone from IONOS before changes.
|
||||
2. Lower TTL ahead of planned cutover if possible.
|
||||
3. Update A records; verify with `dig +short <host> A`.
|
||||
4. Validate HTTPS: `curl -fsS https://<host>/health` or equivalent.
|
||||
5. Roll back to previous A record if validation fails.
|
||||
|
||||
---
|
||||
|
||||
## 6. Before any infrastructure work
|
||||
|
||||
```bash
|
||||
bin/railiance preflight # on dev workstation — must exit 0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Escalation and evidence
|
||||
|
||||
- Full runbook: `RestoreRunbooks/01-full-server-loss.md`
|
||||
- Registers: `AssetRegister.yaml`, `DataRegister.yaml`
|
||||
- Pickup queue (open backup work): `BackupPickupQueue.md`
|
||||
- Forgejo restore: `railiance-infra/tools/forgejo-restore-drill.sh`
|
||||
Loading…
Add table
Add a link
Reference in a new issue