--- id: DISCTL-WP-0002 type: workplan title: "Resilience foundation: registers, policy, first tested restore" domain: infotech repo: disaster-control status: finished owner: grok topic_slug: custodian created: "2026-07-04" updated: "2026-07-07" state_hub_workstream_id: "73d2ecb3-e594-4a7c-920d-8753198eb6ec" --- # Resilience foundation: registers, policy, first tested restore Stand up the minimum viable Resilience Control Plane for Coulomb Social / Helix Forge: know what exists (registers), what must survive (policy, RTO/RPO), how it comes back (runbook, panic sheet), and prove it once (drill + evidence). Driven by `history/2026-07-04-initial-risk-analysis.md` — priority order R2 (untested restores), R9 (silent backup failure), R3 (backup-deleting breach), R4/R5 (identity & secret recovery), R10 (control plane single-homed), R1 (full server loss). Design source: `specs/ResilienceControlSetup.md`. The same-day Forgejo/Railiance assessment (`history/2026-07-04-forgejo-backup-strategy-assessment.md`) proved the Forgejo restore path (RAIL-HO-WP-0005-T09) and handed this repo a pickup queue: its items 7–8 are T03/T04/T06 below; its cross-repo items are tracked by T08. ## Mirror disaster-control off the primary Gitea (R10) ```task id: DISCTL-WP-0002-T01 status: done priority: high state_hub_task_id: "f2f8648e-110c-455e-9745-6e2aa7f4fdfa" ``` This repo must survive the disasters it describes. Add at least one mirror on infrastructure independent of CoulombCore/Gitea (e.g. GitHub private mirror or secondary machine), document the mirror location in README, and note how the mirror is refreshed. ## Asset and data registers with Tier-0 recovery paths (R4, R7, R8, R12) ```task id: DISCTL-WP-0002-T02 status: done priority: high state_hub_task_id: "99bb7f1b-3ab4-4cee-b7ff-f55f0c2b287a" ``` Write `AssetRegister.yaml` and `DataRegister.yaml` seeded with the current stack: domains/DNS/registrar, email, Stripe, Bubble.io, OpenRouter, Gitea, CoulombCore content, k3s, databases, backup storage, secrets/OpenBao. Every T0 entry must record its recovery path (backup admin, MFA recovery, break-glass). Every T1 entry must name its source of truth and backup mechanism (or mark it `backup: none` honestly). Use the tier model from the spec (§2). ## ResiliencePolicy and RTO/RPO matrix (R1, R6) ```task id: DISCTL-WP-0002-T03 status: done priority: high state_hub_task_id: "a6f87f9d-5fbd-4586-bb97-62e0ce99cb27" ``` Write `ResiliencePolicy.md` (operating principle, tier definitions, drill cadence, evidence requirement) and `RTO-RPO-Matrix.md` with explicit per-system targets, starting from the spec's suggested values (§3): RPO ≤ 24h everywhere, RTO ≤ 1 day full recovery, RTO ≤ 4–8h for code/content/control plane. ## BackupPolicy with credential separation and monitoring (R3, R9) ```task id: DISCTL-WP-0002-T04 status: done priority: high state_hub_task_id: "75ce1886-e53e-4382-9a39-07f638cb59a8" ``` Write `BackupPolicy.md`: 3-2-1 + immutable/offline model, per-stream mechanisms (gitea dump + mirrors, restic, k3s etcd snapshots, Bubble/Stripe exports, SOPS/age secret vault), and two hard rules with verification steps — (a) backup-storage credentials never live on the production host with delete permission; (b) backup job success and repository integrity are checked on a defined schedule with a named alerting path. ## Full-server-loss restore runbook and panic sheet (R1, R11) ```task id: DISCTL-WP-0002-T05 status: done priority: medium state_hub_task_id: "da73d8fd-13d0-4549-a196-c29e9f418784" ``` Write `RestoreRunbooks/01-full-server-loss.md` (new machine → Ansible → secret recovery → data restore → k3s redeploy → validate → DNS cutover → evidence report) and the one-page panic sheet: where backups are, who has access, how to rebuild, how to rotate secrets, how to switch DNS — written so a competent stranger could execute it. ## First restore drill with evidence file (R2) ```task id: DISCTL-WP-0002-T06 status: done priority: high state_hub_task_id: "b08f1b02-4937-42aa-9989-042f50cc2e01" ``` The 2026-07-04 Forgejo drill (evidence in `railiance-infra`) proved that path; this task covers the next untested system: restore one Gitea repo/dump or decrypt the secret vault from cold, onto a clean target. Write `TestEvidence/restore-drill-2026-MM-DD.md` with measured RTO/RPO, result, issues found, and follow-up actions; also copy or link the Forgejo drill evidence so this repo holds the complete evidence trail. Define the recurring drill cadence (pickup item 7). This task is the exit criterion for the workplan: no drill evidence here, not finished. ## Incident playbooks (R3, R4, R5, R6, R7) ```task id: DISCTL-WP-0002-T07 status: done priority: medium state_hub_task_id: "b1b9df65-1dc9-45a2-89fb-c6e0655be85d" ``` Write the five playbooks under `IncidentPlaybooks/`: ransomware/hostile access, credential leak, accidental delete / data corruption, provider outage (Bubble/Stripe/OpenRouter/DNS/email), loss of admin identity. Keep each to one page of numbered steps per the spec (§6). ## Track cross-repo backup pickup queue (R2, R9) ```task id: DISCTL-WP-0002-T08 status: done priority: medium state_hub_task_id: "2c8f432e-dc63-4a6b-a838-e60eb514d848" ``` Drive closure of the cross-repo items from the Forgejo assessment's pickup queue (items 1–6: backup destination decision #7, CNPG ScheduledBackup, scheduled `forgejo dump`, `railiance-backup` tool, Gitea pre-cutover snapshot, `apps-pg` coverage). Implementation lives in `railiance-platform` / `railiance-apps` / `railiance-infra`; this repo tracks status and records the outcome in the registers and BackupPolicy as each item lands.