diff --git a/INTENT.md b/INTENT.md new file mode 100644 index 0000000..8782e94 --- /dev/null +++ b/INTENT.md @@ -0,0 +1,29 @@ +# INTENT + +## Purpose + +`disaster-control` is the **Resilience Control Plane** for Coulomb Social / Helix Forge and adjacent endeavors. It is the single source of truth for recovery — independent of any one server — answering: what exists, what must survive, how it is rebuilt, and whether restore actually works. + +The framing is **business continuity + disaster recovery + cyber recovery + evidence**, informed by ISO 22301, NIST CSF 2.0, NIST SP 800-34, and CISA backup guidance. + +## What lives here + +- **Registers** — `AssetRegister.yaml`, `DataRegister.yaml`, service dependency map +- **Policy** — `ResiliencePolicy.md`, `BackupPolicy.md`, RTO/RPO matrix +- **Restore runbooks** — full server loss, Gitea restore, k3s restore, Bubble export restore, breach recovery, secret rotation +- **Incident playbooks** — ransomware, credential leak, accidental delete, provider outage, data corruption +- **Test evidence** — dated restore-drill reports proving RTO/RPO are actually met + +## Operating principle + +> Every capability that becomes valuable must declare its source of truth, recovery method, backup schedule, RTO, RPO, owner, and restore test evidence. + +Servers are disposable; data, identity, secrets, evidence, and payment/account records are not. Rebuild (Ansible/IaC + data restore + secret recovery + DNS cutover) is the default recovery pattern, not server resurrection. + +## Boundaries + +This repo documents, plans, and evidences resilience for the Coulomb stack. It does not host application runtime code, and it does not own the backup tooling itself (restic, Velero, `gitea dump`, etc. run where the infrastructure lives). Extracting the control plane into a standalone product is explicitly deferred. + +## Current state + +Early exploration: `specs/ResilienceControlSetup.md` outlines the full design (asset tiers, RTO/RPO targets, 3-2-1 + immutable backup architecture, playbooks, drill cadence). The registers, policies, runbooks, and evidence loop are not yet written. diff --git a/history/2026-07-04-initial-risk-analysis.md b/history/2026-07-04-initial-risk-analysis.md new file mode 100644 index 0000000..eaf09b5 --- /dev/null +++ b/history/2026-07-04-initial-risk-analysis.md @@ -0,0 +1,149 @@ +# Initial Risk Analysis — Failure Modes for disaster-control + +Date: 2026-07-04 +Author: claude-code (session with Bernd) +Status: baseline — to be revisited after AssetRegister and first restore drill +Source: `specs/ResilienceControlSetup.md`, `INTENT.md`, current Coulomb/Helix stack +(Gitea, k3s, CoulombCore, Bubble.io, Stripe, OpenRouter, DNS, secrets, Ansible) + +## Purpose + +Enumerate the failure modes the Resilience Control Plane must address, rank them, +and map each to the artifact (register, policy, runbook, playbook, drill) that +mitigates it. This is the justification layer for the initial workplans. + +## Scoring + +- **Likelihood** L1 (rare) – L5 (expected within a year) +- **Impact** I1 (nuisance) – I5 (existential: money, identity, or irreplaceable data lost) +- **Exposure today** = how well the current setup handles it *before* this repo does its job + +## Failure modes + +### R1 — Full production server loss (CoulombCore) + +Disk failure, VPS deletion, provider account closure, fat-fingered teardown. +**L3 / I4.** Exposure today: high — rebuild path exists conceptually (Ansible) +but is not documented end-to-end and has never been drilled. +**Mitigation:** restore runbook `01-full-server-loss`, Ansible baseline as the +recovery asset, offsite encrypted backups, DNS cutover procedure, quarterly rebuild drill. + +### R2 — Backup exists but restore fails (untested backups) + +The classic silent failure: dumps run for months, first real restore reveals +corruption, missing volumes, undumped databases, or an unknown passphrase. +**L4 / I5** (it converts every other risk into a total loss). Exposure today: +critical — no restore has ever been evidenced. +**Mitigation:** the TestEvidence loop itself — monthly restore drills with dated +reports measuring actual RTO/RPO. This is the highest-leverage artifact in the repo. + +### R3 — Ransomware / hostile access deletes backups too + +Attacker with production admin credentials encrypts data *and* deletes reachable +backups. **L2 / I5.** Exposure today: high if backup credentials live on the +production host with delete permission. +**Mitigation:** backup credential separation (BackupPolicy §credentials), +immutable/append-only offsite copy, offline weekly copy, breach-recovery runbook +(`never restore into the compromised machine`), secret rotation runbook. + +### R4 — Loss of admin identity (registrar, Stripe, Gitea, email, cloud) + +Locked out via lost MFA device, expired recovery email, account takeover, or +provider suspension. **L2 / I5** — identity is Tier 0; without it nothing else +can be recovered. Exposure today: unknown — no inventory of recovery codes, +backup admins, or break-glass paths exists. +**Mitigation:** AssetRegister T0 section with recovery paths per account, +offline recovery codes, break-glass process, loss-of-admin-identity playbook. + +### R5 — Secrets lost or leaked + +Two directions: (a) recovery material lost → backups undecryptable, services +unrebuildable; (b) secret leaked → every dependent system compromised. +**L3 / I5.** Exposure today: medium-high — OpenBao/warden infrastructure exists +but sealed *recovery* material for disaster scenarios is not evidenced. +**Mitigation:** encrypted (SOPS/age) sealed recovery vault backed up offline, +secret-rotation runbook, credential-leak playbook. Restore drills must include +"can we actually decrypt the vault from cold". + +### R6 — Accidental deletion / bad migration / data corruption + +Deleted repo, destructive script, corrupted DB, bad Ansible run. **L4 / I3** +(bounded if backups have granularity and short RPO). Exposure today: medium. +**Mitigation:** accidental-delete and data-corruption playbooks (stop writes → +staging restore → identify last good point → selective restore → diff), local +snapshots for fast rollback, RPO targets per system in the RTO/RPO matrix. + +### R7 — External provider outage or exit (Bubble, Stripe, OpenRouter, DNS, email) + +Outage is L4/I2 (degradation); *forced exit* (provider shutdown, account ban, +pricing change) is L2/I4 — especially Bubble, which holds member data on a +platform we don't control. Exposure today: high for Bubble (export cadence not +established), low-medium for others. +**Mitigation:** T4 section of AssetRegister documenting export paths and limits, +scheduled Bubble and Stripe exports, provider-outage playbook with graceful +degradation, alternate-LLM path for OpenRouter. + +### R8 — DNS / domain hijack or lapse + +Expired domain, registrar compromise, hostile transfer. **L1 / I5.** +Exposure today: unknown (lock status, MFA, backup contacts uninventoried). +**Mitigation:** registrar lock + MFA + backup admin recorded in AssetRegister, +zone-record exports after changes, 2h RTO cutover procedure. + +### R9 — Silent backup failure (job stops, nobody notices) + +Cron dies, disk fills, token expires; discovered only at restore time. +**L4 / I4** (degenerates into R2). Exposure today: high — no backup success +monitoring is defined. +**Mitigation:** monitoring & evidence checks (daily job-success check, weekly +integrity check) defined in BackupPolicy with an explicit alerting owner. + +### R10 — The control plane itself is lost or stale + +Two sub-modes: (a) this repo lives on the same Gitea it protects — a full-loss +event takes the recovery instructions down with it; (b) docs drift from reality +and runbooks silently rot. **L3 / I4.** Exposure today: high — repo is +single-homed as of this writing. +**Mitigation:** mirror disaster-control to at least one independent location +(and one offline/printable panic sheet), monthly asset-register review, drills +double as doc-verification. + +### R11 — Single-operator dependency (bus factor = 1) + +Bernd is the only person who knows where backups are, how to rotate secrets, +and holds the MFA devices. **L2 / I5.** Exposure today: structural. +**Mitigation:** the panic sheet ("where backups are, who has access, how to +rebuild") written so a competent stranger could execute it; offline recovery +material reachable by a designated second person. + +### R12 — Payment/financial record loss (Stripe divergence) + +Stripe holds the authoritative customer/payment state; account loss or +membership-state divergence with Bubble breaks the business's money loop. +**L2 / I4.** Exposure today: medium. +**Mitigation:** scheduled Stripe exports (CSV or Data Pipeline) into backed-up +storage, reconciliation step in provider-outage playbook. + +## Priority ranking (exposure × impact) + +1. **R2** untested restores — invalidates everything else; fix via evidence loop +2. **R9** silent backup failure — feeds R2 directly +3. **R3** backup-deleting breach — credential separation is cheap and decisive +4. **R4/R5** identity & secrets recovery — T0, low effort to inventory, existential if wrong +5. **R10** control plane single-homed — mirror this repo immediately (near-zero cost) +6. **R1** full server loss — the flagship runbook, exercised by drills +7. **R7/R12** provider exports (Bubble, Stripe) — schedule and verify +8. **R6, R8, R11** — covered incrementally by playbooks and the panic sheet + +## Consequences for the workplan + +- First workplan must deliver: RTO/RPO matrix + ResiliencePolicy (targets), + AssetRegister/DataRegister (inventory incl. T0 recovery paths), BackupPolicy + with credential-separation rules, the full-server-loss runbook, a panic sheet, + and **one real restore drill with an evidence file** — because until R2 is + exercised once, everything else is theory. +- Mirroring this repo off the primary Gitea (R10) is a day-one ad hoc task, + not a workplan phase. +- Backup *automation* (restic, gitea dump, etcd snapshots, Bubble/Stripe export + jobs) runs where the infrastructure lives; this repo tracks its policy, + schedule, and evidence — see INTENT.md boundaries. diff --git a/specs/ResilienceControlSetup.md b/specs/ResilienceControlSetup.md new file mode 100644 index 0000000..a214564 --- /dev/null +++ b/specs/ResilienceControlSetup.md @@ -0,0 +1,294 @@ +Resilience Control Setup Exploration + +## Introduction + +This is a rough outline for how we will secure Coulomb Social / Helix Forge against disasters. + +We will **not** start with “buy a backup tool.” instead we will set up a small **Resilience Control Plane**: +a documented, automated, tested system that knows what exists, what must survive, how it is rebuilt, and whether restore actually works. + +The useful framing is: **business continuity + disaster recovery + cyber recovery + evidence**. ISO 22301 is the standard family for business-continuity management, while NIST CSF 2.0 frames cyber resilience around Govern, Identify, Protect, Detect, Respond, and Recover. NIST SP 800-34 remains a practical reference for IT contingency planning, and CISA explicitly recommends offline, encrypted backups that are regularly tested. ([ISO][1]) + +Remark: The Resilience Control Plane might evolve into a product of it's own later. The repository disaster-control for now is about resilience for the coulomb project and adjacent endevors. Extracting a product is for a later day. + +## 1. Treat Coulomb / Helix as a recoverable organization + +Given your current shape — Bubble.io legacy app, Stripe memberships, OpenRouter usage, CoulombCore on Ubuntu, Gitea, Decap CMS, k3s, Ansible, AI/webhook services, many repos — create one repo, for example: + +`disaster-control` + +Inside it: + +```text +disaster-control/ + INTENT.md + ResiliencePolicy.md + AssetRegister.yaml + DataRegister.yaml + ServiceDependencyMap.md + RTO-RPO-Matrix.md + BackupPolicy.md + RestoreRunbooks/ + 01-full-server-loss.md + 02-gitea-restore.md + 03-k3s-restore.md + 04-bubble-export-restore.md + 05-security-breach-recovery.md + 06-secret-rotation.md + IncidentPlaybooks/ + ransomware.md + credential-leak.md + accidental-delete.md + provider-outage.md + data-corruption.md + TestEvidence/ + restore-drill-YYYY-MM-DD.md +``` + +This becomes the “single source of truth” for recovery, independent of any one server. + +## 2. Classify assets by recovery importance + +Use tiers. This prevents over-engineering everything. + +| Tier | Asset type | Examples | Target | +| ------------------------------------ | ----------------------------------------------------------------------------------------- | ------------------------------------------ | ------ | +| **T0 Identity / trust / money** | Domains, DNS, registrar, email, Stripe, OAuth, admin accounts, secrets, SSH keys | Must be recoverable even after breach | | +| **T1 Irreplaceable source of truth** | Gitea repos, CoulombCore content, user/member data, product registry, paid customer state | Strong backups, tested restore | | +| **T2 Runtime platforms** | Ubuntu server, k3s, Gitea instance, Decap CMS, webhooks, databases | Rebuild from IaC + restore data | | +| **T3 Derived / rebuildable** | caches, generated artifacts, embeddings, logs beyond retention, local model outputs | Recompute where possible | | +| **T4 External dependencies** | Bubble.io, Stripe, OpenRouter, domain/DNS provider, email provider | Export, monitor, fallback, document limits | | + +The key idea: **servers are disposable; data, identity, secrets, evidence, and payment/account records are not.** + +## 3. Define RTO and RPO per system + +Use explicit recovery objectives: + +| System | Suggested RPO | Suggested RTO | Notes | +| ------------------------------ | -----------------------------------------: | ------------: | ------------------------------------------------------------------------------------------------------ | +| Gitea repos | 1h or better | 4–8h | Mirror repos plus Gitea dump | +| CoulombCore content | 1h–6h | 4–8h | Back up content store + DB | +| k3s cluster state | 1h–24h | 8h | Prefer rebuildable cluster + snapshots | +| Bubble.io database | 24h initially | 24h | Export regularly; Bubble has database restore/export capabilities ([manual.bubble.io][2]) | +| Stripe financial/customer data | 24h–7d | 24h | Stripe can export/sync data to storage/warehouse destinations ([Stripe Dokumentation][3]) | +| Secrets | immediate rotation plan | 2–4h | Back up encrypted recovery material, not plaintext secrets | +| DNS/domains | manual export after changes | 2h | Registrar lock + MFA + backup admin | +| OpenRouter dependency | no data RPO unless storing prompts/results | 1–4h fallback | Track API keys, quotas, usage; OpenRouter exposes key/limit/credit info via API/docs ([OpenRouter][4]) | + +For an early-stage environment, I would optimize for **RPO ≤ 24h everywhere, RTO ≤ 1 day for full recovery, and RTO ≤ 4–8h for code/content/control-plane recovery**. + +## 4. Implement the backup architecture + +Use a practical **3-2-1 + immutable/offline + tested restore** model: at least three copies, on two storage types, one offsite; plus one copy that an attacker cannot easily delete or encrypt. The UK NCSC describes 3-2-1 as a common resilient-backup pattern, and CISA emphasizes offline, encrypted, regularly tested backups for ransomware resilience. ([National Cyber Security Centre][5]) + +For Coulomb / Helix, I would set it up like this: + +```text +Production server + ├─ local snapshots for fast rollback + ├─ encrypted daily backup to offsite object storage + ├─ immutable/offline weekly backup + └─ monthly restore drill to clean machine +``` + +Concrete backup streams: + +| Stream | Mechanism | +| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| **Gitea** | Scheduled `gitea dump` plus repo mirrors. Gitea’s official docs describe `dump` as creating a ZIP backup usable for restore. ([Gitea Dokumentation][6]) | +| **Git repos** | Mirror clone to separate location/provider. GitHub’s docs recommend `git clone --mirror` for repository backup including history. ([GitHub Docs][7]) | +| **k3s** | k3s datastore snapshots. k3s documents `etcd-snapshot` for managing and restoring embedded etcd snapshots. ([K3s][8]) | +| **Kubernetes resources/PVs** | Velero for cluster resources and persistent volumes. Velero is designed for Kubernetes backup, restore, migration, and cluster replication. ([Velero][9]) | +| **Files / content / configs** | Restic or Borg-style encrypted backups. Restic supports encrypted backups to local, self-hosted, and online storage. ([Restic][10]) | +| **Bubble.io** | Scheduled app/data exports and documented restore workflow. Bubble documents database restore and export/import tooling. ([manual.bubble.io][2]) | +| **Stripe** | Periodic CSV exports or Stripe Data Pipeline/cloud storage export for accounting/customer continuity. ([Stripe Dokumentation][3]) | +| **Secrets** | SOPS/age-encrypted sealed backup of recovery secrets, plus printed/offline recovery material where appropriate. After breach: rotate, don’t blindly restore. | +| **DNS / domains** | Export zone records, registrar settings, recovery codes, admin contacts, and MFA recovery process. | + +## 5. Make rebuild the default, not server resurrection + +For the Ubuntu/k3s/Gitea environment, the best recovery pattern is: + +```text +New clean machine + → install base OS + → apply Ansible + → restore secrets from encrypted recovery vault + → restore Gitea / DB / content + → restore k3s state or redeploy manifests + → validate services + → switch DNS +``` + +This means your real disaster-recovery asset is not the server image. It is: + +1. **Ansible/IaC** +2. **data backups** +3. **secret recovery** +4. **DNS cutover** +5. **tested runbooks** + +For a security breach, never restore into the compromised machine. Build a clean environment, restore only known-good data, rotate credentials, and preserve evidence. + +## 6. Create incident playbooks + +You need at least five. + +### A. Full server loss + +Trigger: disk failure, VPS deletion, hardware loss. + +Recovery: + +1. Provision new server. +2. Apply Ansible baseline. +3. Restore Gitea dump and repo mirrors. +4. Restore databases/content. +5. Restore k3s from snapshot or redeploy workloads. +6. Validate health checks. +7. Move DNS. +8. Write evidence report. + +### B. Accidental deletion or bad migration + +Trigger: deleted repo, corrupted database, bad script. + +Recovery: + +1. Stop writes. +2. Clone latest backup into staging. +3. Identify last good point. +4. Restore selectively. +5. Diff recovered data against production. +6. Resume writes. + +### C. Ransomware / hostile access + +Trigger: suspicious encryption, unknown admin access, altered repos, leaked token. + +Recovery: + +1. Isolate systems. +2. Preserve logs/snapshots for evidence. +3. Revoke exposed credentials. +4. Build clean replacement environment. +5. Restore from backup before compromise time. +6. Rotate all secrets. +7. Force password/session reset where needed. +8. Publish internal incident note. + +### D. Provider outage + +Trigger: Bubble, Stripe, OpenRouter, DNS provider, email provider unavailable. + +Recovery: + +1. Switch status page / static fallback. +2. Disable dependent features gracefully. +3. Queue work where possible. +4. Use alternate LLM provider path where feasible. +5. Reconcile once provider returns. + +### E. Loss of admin identity + +Trigger: locked out of registrar, Gitea, Stripe, Bubble, email, cloud. + +Recovery: + +1. Use documented break-glass account. +2. Use offline recovery codes. +3. Verify ownership records. +4. Rotate compromised accounts. +5. Update evidence log. + +## 7. Separate backup credentials from production credentials + +A common failure mode: the attacker gets production admin access and deletes the backups too. So: + +* Backup storage credentials must **not** live on the main server with delete permissions. +* Use append-only or immutable storage where possible. +* Use separate admin accounts for production, backup, registrar, and payment systems. +* Enable MFA everywhere. +* Keep offline recovery codes. +* Keep a minimal “break-glass” process documented. + +## 8. Add monitoring and evidence + +You want the system to prove itself. + +Minimum checks: + +| Check | Frequency | +| --------------------------------- | --------------------------: | +| Backup job succeeded | daily | +| Backup repository integrity check | weekly | +| Restore one repo to staging | weekly | +| Restore Gitea dump to staging | monthly | +| Rebuild full server from Ansible | quarterly | +| Rotate critical secrets | quarterly or after incident | +| Review asset register | monthly | +| Review external dependencies | monthly | + +Every drill should produce a short file: + +```text +TestEvidence/restore-drill-2026-07-04.md +``` + +With: + +```markdown +# Restore Drill: Gitea + +Date: 2026-07-04 +Backup used: gitea-dump-... +Target: clean staging server +Result: success / partial / failed +RTO measured: ... +RPO measured: ... +Issues found: +- ... +Actions: +- ... +``` + +This turns resilience into a learning loop. + +## 9. Minimal viable setup + +For the first version, I would implement this: + +1. **Create `coulomb-resilience-control` repo.** +2. **Write AssetRegister.yaml** for domains, Gitea, Bubble, Stripe, OpenRouter, k3s, databases, storage, secrets. +3. **Automate Gitea backup** using `gitea dump` plus repo mirrors. +4. **Automate server/content backups** with encrypted restic/Borg-style offsite backup. +5. **Add k3s etcd snapshots** if k3s state matters; otherwise prefer full redeploy from manifests/Ansible. +6. **Export Bubble data** on a fixed schedule while Bubble remains in the architecture. +7. **Export Stripe/accounting data** to storage/warehouse or at least scheduled CSV snapshots. +8. **Create one clean restore machine** and perform a monthly restore drill. +9. **Document secret rotation and breach recovery.** +10. **Create a one-page “panic sheet”**: where backups are, who has access, how to rebuild, how to rotate secrets, how to switch DNS. + +## 10. Suggested operating principle + +For Coulomb / Helix Forge, I would use this rule: + +> **Every capability that becomes valuable must declare its source of truth, recovery method, backup schedule, RTO, RPO, owner, and restore test evidence.** + +That fits naturally with your capability-registry thinking. A repo or service is not “production-worthy” until its recovery story is known. + +A good next artifact would be a `ResiliencePolicy.md` plus `AssetRegister.yaml` seed for your current Coulomb/Helix stack. + +[1]: https://www.iso.org/standard/75106.html?utm_source=chatgpt.com "ISO 22301:2019 - Business continuity management systems" +[2]: https://manual.bubble.io/help-guides/maintaining-an-application/database-maintenance/database-copy-and-backups?utm_source=chatgpt.com "Restoring database backups" +[3]: https://docs.stripe.com/data/data-pipeline?utm_source=chatgpt.com "Data Pipeline | Stripe Documentation" +[4]: https://openrouter.ai/docs/api/reference/limits?utm_source=chatgpt.com "API Rate Limits - Manage Model Usage and Quotas" +[5]: https://www.ncsc.gov.uk/blog-post/offline-backups-in-an-online-world?utm_source=chatgpt.com "Offline backups in an online world" +[6]: https://docs.gitea.com/administration/backup-and-restore?utm_source=chatgpt.com "Backup and Restore" +[7]: https://docs.github.com/en/repositories/archiving-a-github-repository/backing-up-a-repository?utm_source=chatgpt.com "Backing up a repository" +[8]: https://docs.k3s.io/cli/etcd-snapshot?utm_source=chatgpt.com "k3s etcd-snapshot" +[9]: https://velero.io/docs/main/?utm_source=chatgpt.com "Velero Docs - Overview" +[10]: https://restic.net/?utm_source=chatgpt.com "restic · Backups done right!" + +xxx diff --git a/workplans/DISCTL-WP-0001-statehub-bootstrap.md b/workplans/DISCTL-WP-0001-statehub-bootstrap.md new file mode 100644 index 0000000..662f4af --- /dev/null +++ b/workplans/DISCTL-WP-0001-statehub-bootstrap.md @@ -0,0 +1,58 @@ +--- +id: DISCTL-WP-0001 +type: workplan +title: "Bootstrap State Hub integration" +domain: infotech +repo: disaster-control +status: ready +owner: codex +topic_slug: custodian +created: "2026-07-04" +updated: "2026-07-04" +--- + +# Bootstrap State Hub integration + +Resilience control plane for Coulomb Social / Helix Forge: the single source of truth for business continuity, disaster recovery, cyber recovery, and restore-test evidence covering the Coulomb infrastructure (Gitea, k3s, CoulombCore, Bubble.io, Stripe, DNS, secrets). + +## Review Generated Integration Files + +```task +id: DISCTL-WP-0001-T01 +status: todo +priority: high +``` + +Review `INTENT.md`, `SCOPE.md`, `AGENTS.md`, and `.custodian-brief.md`. +Replace generated placeholders with repo-specific facts where needed. + +## Verify Local Developer Workflow + +```task +id: DISCTL-WP-0001-T02 +status: todo +priority: high +``` + +Identify the repo's install, test, lint, build, and run commands. Add or refine +those commands in the agent instructions so future coding sessions can verify +changes confidently. + +## Seed First Real Workplan + +```task +id: DISCTL-WP-0001-T03 +status: done +priority: medium +``` + +Result 2026-07-04: Created `DISCTL-WP-0002-resilience-foundation.md`, derived +from the initial risk analysis in `history/2026-07-04-initial-risk-analysis.md`. + +Create the first implementation workplan for the repository's most important +next change. After workplan file updates, run the sync locally from this repo +checkout: + +```bash +statehub fix-consistency +``` diff --git a/workplans/DISCTL-WP-0002-resilience-foundation.md b/workplans/DISCTL-WP-0002-resilience-foundation.md new file mode 100644 index 0000000..b91d174 --- /dev/null +++ b/workplans/DISCTL-WP-0002-resilience-foundation.md @@ -0,0 +1,120 @@ +--- +id: DISCTL-WP-0002 +type: workplan +title: "Resilience foundation: registers, policy, first tested restore" +domain: infotech +repo: disaster-control +status: ready +owner: claude-code +topic_slug: custodian +created: "2026-07-04" +updated: "2026-07-04" +--- + +# Resilience foundation: registers, policy, first tested restore + +Stand up the minimum viable Resilience Control Plane for Coulomb Social / +Helix Forge: know what exists (registers), what must survive (policy, RTO/RPO), +how it comes back (runbook, panic sheet), and prove it once (drill + evidence). + +Driven by `history/2026-07-04-initial-risk-analysis.md` — priority order R2 +(untested restores), R9 (silent backup failure), R3 (backup-deleting breach), +R4/R5 (identity & secret recovery), R10 (control plane single-homed), R1 +(full server loss). Design source: `specs/ResilienceControlSetup.md`. + +## Mirror disaster-control off the primary Gitea (R10) + +```task +id: DISCTL-WP-0002-T01 +status: todo +priority: high +``` + +This repo must survive the disasters it describes. Add at least one mirror on +infrastructure independent of CoulombCore/Gitea (e.g. GitHub private mirror or +secondary machine), document the mirror location in README, and note how the +mirror is refreshed. + +## Asset and data registers with Tier-0 recovery paths (R4, R7, R8, R12) + +```task +id: DISCTL-WP-0002-T02 +status: todo +priority: high +``` + +Write `AssetRegister.yaml` and `DataRegister.yaml` seeded with the current +stack: domains/DNS/registrar, email, Stripe, Bubble.io, OpenRouter, Gitea, +CoulombCore content, k3s, databases, backup storage, secrets/OpenBao. Every T0 +entry must record its recovery path (backup admin, MFA recovery, break-glass). +Every T1 entry must name its source of truth and backup mechanism (or mark it +`backup: none` honestly). Use the tier model from the spec (§2). + +## ResiliencePolicy and RTO/RPO matrix (R1, R6) + +```task +id: DISCTL-WP-0002-T03 +status: todo +priority: high +``` + +Write `ResiliencePolicy.md` (operating principle, tier definitions, drill +cadence, evidence requirement) and `RTO-RPO-Matrix.md` with explicit per-system +targets, starting from the spec's suggested values (§3): RPO ≤ 24h everywhere, +RTO ≤ 1 day full recovery, RTO ≤ 4–8h for code/content/control plane. + +## BackupPolicy with credential separation and monitoring (R3, R9) + +```task +id: DISCTL-WP-0002-T04 +status: todo +priority: high +``` + +Write `BackupPolicy.md`: 3-2-1 + immutable/offline model, per-stream mechanisms +(gitea dump + mirrors, restic, k3s etcd snapshots, Bubble/Stripe exports, +SOPS/age secret vault), and two hard rules with verification steps — +(a) backup-storage credentials never live on the production host with delete +permission; (b) backup job success and repository integrity are checked on a +defined schedule with a named alerting path. + +## Full-server-loss restore runbook and panic sheet (R1, R11) + +```task +id: DISCTL-WP-0002-T05 +status: todo +priority: medium +``` + +Write `RestoreRunbooks/01-full-server-loss.md` (new machine → Ansible → secret +recovery → data restore → k3s redeploy → validate → DNS cutover → evidence +report) and the one-page panic sheet: where backups are, who has access, how to +rebuild, how to rotate secrets, how to switch DNS — written so a competent +stranger could execute it. + +## First restore drill with evidence file (R2) + +```task +id: DISCTL-WP-0002-T06 +status: todo +priority: high +``` + +Perform one real restore to a clean target — smallest meaningful scope (e.g. +one Gitea repo from a `gitea dump`, or the secret vault decrypted from cold) — +and write `TestEvidence/restore-drill-2026-MM-DD.md` with measured RTO/RPO, +result, issues found, and follow-up actions. This task is the exit criterion +for the workplan: no drill evidence, not finished. + +## Incident playbooks (R3, R4, R5, R6, R7) + +```task +id: DISCTL-WP-0002-T07 +status: todo +priority: medium +``` + +Write the five playbooks under `IncidentPlaybooks/`: ransomware/hostile access, +credential leak, accidental delete / data corruption, provider outage +(Bubble/Stripe/OpenRouter/DNS/email), loss of admin identity. Keep each to one +page of numbered steps per the spec (§6).