disaster-control/workplans/DISCTL-WP-0002-resilience-foundation.md

---
id: DISCTL-WP-0002
type: workplan
title: "Resilience foundation: registers, policy, first tested restore"
domain: infotech
repo: disaster-control
status: finished
owner: grok
topic_slug: custodian
created: "2026-07-04"
updated: "2026-07-07"
state_hub_workstream_id: "73d2ecb3-e594-4a7c-920d-8753198eb6ec"
---

# Resilience foundation: registers, policy, first tested restore

Stand up the minimum viable Resilience Control Plane for Coulomb Social /
Helix Forge: know what exists (registers), what must survive (policy, RTO/RPO),
how it comes back (runbook, panic sheet), and prove it once (drill + evidence).

Driven by `history/2026-07-04-initial-risk-analysis.md` — priority order R2
(untested restores), R9 (silent backup failure), R3 (backup-deleting breach),
R4/R5 (identity & secret recovery), R10 (control plane single-homed), R1
(full server loss). Design source: `specs/ResilienceControlSetup.md`.

The same-day Forgejo/Railiance assessment
(`history/2026-07-04-forgejo-backup-strategy-assessment.md`) proved the Forgejo
restore path (RAIL-HO-WP-0005-T09) and handed this repo a pickup queue: its
items 7–8 are T03/T04/T06 below; its cross-repo items are tracked by T08.

## Mirror disaster-control off the primary Gitea (R10)

```task
id: DISCTL-WP-0002-T01
status: done
priority: high
state_hub_task_id: "f2f8648e-110c-455e-9745-6e2aa7f4fdfa"
```

This repo must survive the disasters it describes. Add at least one mirror on
infrastructure independent of CoulombCore/Gitea (e.g. GitHub private mirror or
secondary machine), document the mirror location in README, and note how the
mirror is refreshed.

## Asset and data registers with Tier-0 recovery paths (R4, R7, R8, R12)

```task
id: DISCTL-WP-0002-T02
status: done
priority: high
state_hub_task_id: "99bb7f1b-3ab4-4cee-b7ff-f55f0c2b287a"
```

Write `AssetRegister.yaml` and `DataRegister.yaml` seeded with the current
stack: domains/DNS/registrar, email, Stripe, Bubble.io, OpenRouter, Gitea,
CoulombCore content, k3s, databases, backup storage, secrets/OpenBao. Every T0
entry must record its recovery path (backup admin, MFA recovery, break-glass).
Every T1 entry must name its source of truth and backup mechanism (or mark it
`backup: none` honestly). Use the tier model from the spec (§2).

## ResiliencePolicy and RTO/RPO matrix (R1, R6)

```task
id: DISCTL-WP-0002-T03
status: done
priority: high
state_hub_task_id: "a6f87f9d-5fbd-4586-bb97-62e0ce99cb27"
```

Write `ResiliencePolicy.md` (operating principle, tier definitions, drill
cadence, evidence requirement) and `RTO-RPO-Matrix.md` with explicit per-system
targets, starting from the spec's suggested values (§3): RPO ≤ 24h everywhere,
RTO ≤ 1 day full recovery, RTO ≤ 4–8h for code/content/control plane.

## BackupPolicy with credential separation and monitoring (R3, R9)

```task
id: DISCTL-WP-0002-T04
status: done
priority: high
state_hub_task_id: "75ce1886-e53e-4382-9a39-07f638cb59a8"
```

Write `BackupPolicy.md`: 3-2-1 + immutable/offline model, per-stream mechanisms
(gitea dump + mirrors, restic, k3s etcd snapshots, Bubble/Stripe exports,
SOPS/age secret vault), and two hard rules with verification steps —
(a) backup-storage credentials never live on the production host with delete
permission; (b) backup job success and repository integrity are checked on a
defined schedule with a named alerting path.

## Full-server-loss restore runbook and panic sheet (R1, R11)

```task
id: DISCTL-WP-0002-T05
status: done
priority: medium
state_hub_task_id: "da73d8fd-13d0-4549-a196-c29e9f418784"
```

Write `RestoreRunbooks/01-full-server-loss.md` (new machine → Ansible → secret
recovery → data restore → k3s redeploy → validate → DNS cutover → evidence
report) and the one-page panic sheet: where backups are, who has access, how to
rebuild, how to rotate secrets, how to switch DNS — written so a competent
stranger could execute it.

## First restore drill with evidence file (R2)

```task
id: DISCTL-WP-0002-T06
status: done
priority: high
state_hub_task_id: "b08f1b02-4937-42aa-9989-042f50cc2e01"
```

The 2026-07-04 Forgejo drill (evidence in `railiance-infra`) proved that path;
this task covers the next untested system: restore one Gitea repo/dump or
decrypt the secret vault from cold, onto a clean target. Write
`TestEvidence/restore-drill-2026-MM-DD.md` with measured RTO/RPO, result,
issues found, and follow-up actions; also copy or link the Forgejo drill
evidence so this repo holds the complete evidence trail. Define the recurring
drill cadence (pickup item 7). This task is the exit criterion for the
workplan: no drill evidence here, not finished.

## Incident playbooks (R3, R4, R5, R6, R7)

```task
id: DISCTL-WP-0002-T07
status: done
priority: medium
state_hub_task_id: "b1b9df65-1dc9-45a2-89fb-c6e0655be85d"
```

Write the five playbooks under `IncidentPlaybooks/`: ransomware/hostile access,
credential leak, accidental delete / data corruption, provider outage
(Bubble/Stripe/OpenRouter/DNS/email), loss of admin identity. Keep each to one
page of numbered steps per the spec (§6).

## Track cross-repo backup pickup queue (R2, R9)

```task
id: DISCTL-WP-0002-T08
status: done
priority: medium
state_hub_task_id: "2c8f432e-dc63-4a6b-a838-e60eb514d848"
```

Drive closure of the cross-repo items from the Forgejo assessment's pickup
queue (items 1–6: backup destination decision #7, CNPG ScheduledBackup,
scheduled `forgejo dump`, `railiance-backup` tool, Gitea pre-cutover snapshot,
`apps-pg` coverage). Implementation lives in `railiance-platform` /
`railiance-apps` / `railiance-infra`; this repo tracks status and records the
outcome in the registers and BackupPolicy as each item lands.