disaster-control/specs/ResilienceControlSetup.md

Resilience Control Setup Exploration

Introduction

This is a rough outline for how we will secure Coulomb Social / Helix Forge against disasters.

We will not start with “buy a backup tool.” instead we will set up a small Resilience Control Plane: a documented, automated, tested system that knows what exists, what must survive, how it is rebuilt, and whether restore actually works.

The useful framing is: business continuity + disaster recovery + cyber recovery + evidence. ISO 22301 is the standard family for business-continuity management, while NIST CSF 2.0 frames cyber resilience around Govern, Identify, Protect, Detect, Respond, and Recover. NIST SP 800-34 remains a practical reference for IT contingency planning, and CISA explicitly recommends offline, encrypted backups that are regularly tested. (ISO)

Remark: The Resilience Control Plane might evolve into a product of it's own later. The repository disaster-control for now is about resilience for the coulomb project and adjacent endevors. Extracting a product is for a later day.

1. Treat Coulomb / Helix as a recoverable organization

Given your current shape — Bubble.io legacy app, Stripe memberships, OpenRouter usage, CoulombCore on Ubuntu, Gitea, Decap CMS, k3s, Ansible, AI/webhook services, many repos — create one repo, for example:

disaster-control

Inside it:

disaster-control/
  INTENT.md
  ResiliencePolicy.md
  AssetRegister.yaml
  DataRegister.yaml
  ServiceDependencyMap.md
  RTO-RPO-Matrix.md
  BackupPolicy.md
  RestoreRunbooks/
    01-full-server-loss.md
    02-gitea-restore.md
    03-k3s-restore.md
    04-bubble-export-restore.md
    05-security-breach-recovery.md
    06-secret-rotation.md
  IncidentPlaybooks/
    ransomware.md
    credential-leak.md
    accidental-delete.md
    provider-outage.md
    data-corruption.md
  TestEvidence/
    restore-drill-YYYY-MM-DD.md

This becomes the “single source of truth” for recovery, independent of any one server.

2. Classify assets by recovery importance

Use tiers. This prevents over-engineering everything.

Tier	Asset type	Examples	Target
T0 Identity / trust / money	Domains, DNS, registrar, email, Stripe, OAuth, admin accounts, secrets, SSH keys	Must be recoverable even after breach
T1 Irreplaceable source of truth	Gitea repos, CoulombCore content, user/member data, product registry, paid customer state	Strong backups, tested restore
T2 Runtime platforms	Ubuntu server, k3s, Gitea instance, Decap CMS, webhooks, databases	Rebuild from IaC + restore data
T3 Derived / rebuildable	caches, generated artifacts, embeddings, logs beyond retention, local model outputs	Recompute where possible
T4 External dependencies	Bubble.io, Stripe, OpenRouter, domain/DNS provider, email provider	Export, monitor, fallback, document limits

The key idea: servers are disposable; data, identity, secrets, evidence, and payment/account records are not.

3. Define RTO and RPO per system

Use explicit recovery objectives:

System	Suggested RPO	Suggested RTO	Notes
Gitea repos	1h or better	4–8h	Mirror repos plus Gitea dump
CoulombCore content	1h–6h	4–8h	Back up content store + DB
k3s cluster state	1h–24h	8h	Prefer rebuildable cluster + snapshots
Bubble.io database	24h initially	24h	Export regularly; Bubble has database restore/export capabilities (manual.bubble.io)
Stripe financial/customer data	24h–7d	24h	Stripe can export/sync data to storage/warehouse destinations (Stripe Dokumentation)
Secrets	immediate rotation plan	2–4h	Back up encrypted recovery material, not plaintext secrets
DNS/domains	manual export after changes	2h	Registrar lock + MFA + backup admin
OpenRouter dependency	no data RPO unless storing prompts/results	1–4h fallback	Track API keys, quotas, usage; OpenRouter exposes key/limit/credit info via API/docs (OpenRouter)

For an early-stage environment, I would optimize for RPO ≤ 24h everywhere, RTO ≤ 1 day for full recovery, and RTO ≤ 4–8h for code/content/control-plane recovery.

4. Implement the backup architecture

Use a practical 3-2-1 + immutable/offline + tested restore model: at least three copies, on two storage types, one offsite; plus one copy that an attacker cannot easily delete or encrypt. The UK NCSC describes 3-2-1 as a common resilient-backup pattern, and CISA emphasizes offline, encrypted, regularly tested backups for ransomware resilience. (National Cyber Security Centre)

For Coulomb / Helix, I would set it up like this:

Production server
  ├─ local snapshots for fast rollback
  ├─ encrypted daily backup to offsite object storage
  ├─ immutable/offline weekly backup
  └─ monthly restore drill to clean machine

Concrete backup streams:

Stream	Mechanism
Gitea	Scheduled gitea dump plus repo mirrors. Gitea’s official docs describe dump as creating a ZIP backup usable for restore. (Gitea Dokumentation)
Git repos	Mirror clone to separate location/provider. GitHub’s docs recommend git clone --mirror for repository backup including history. (GitHub Docs)
k3s	k3s datastore snapshots. k3s documents etcd-snapshot for managing and restoring embedded etcd snapshots. (K3s)
Kubernetes resources/PVs	Velero for cluster resources and persistent volumes. Velero is designed for Kubernetes backup, restore, migration, and cluster replication. (Velero)
Files / content / configs	Restic or Borg-style encrypted backups. Restic supports encrypted backups to local, self-hosted, and online storage. (Restic)
Bubble.io	Scheduled app/data exports and documented restore workflow. Bubble documents database restore and export/import tooling. (manual.bubble.io)
Stripe	Periodic CSV exports or Stripe Data Pipeline/cloud storage export for accounting/customer continuity. (Stripe Dokumentation)
Secrets	SOPS/age-encrypted sealed backup of recovery secrets, plus printed/offline recovery material where appropriate. After breach: rotate, don’t blindly restore.
DNS / domains	Export zone records, registrar settings, recovery codes, admin contacts, and MFA recovery process.

5. Make rebuild the default, not server resurrection

For the Ubuntu/k3s/Gitea environment, the best recovery pattern is:

New clean machine
  → install base OS
  → apply Ansible
  → restore secrets from encrypted recovery vault
  → restore Gitea / DB / content
  → restore k3s state or redeploy manifests
  → validate services
  → switch DNS

This means your real disaster-recovery asset is not the server image. It is:

1. Ansible/IaC
2. data backups
3. secret recovery
4. DNS cutover
5. tested runbooks

For a security breach, never restore into the compromised machine. Build a clean environment, restore only known-good data, rotate credentials, and preserve evidence.

6. Create incident playbooks

You need at least five.

A. Full server loss

Trigger: disk failure, VPS deletion, hardware loss.

Recovery:

1. Provision new server.
2. Apply Ansible baseline.
3. Restore Gitea dump and repo mirrors.
4. Restore databases/content.
5. Restore k3s from snapshot or redeploy workloads.
6. Validate health checks.
7. Move DNS.
8. Write evidence report.

B. Accidental deletion or bad migration

Trigger: deleted repo, corrupted database, bad script.

Recovery:

1. Stop writes.
2. Clone latest backup into staging.
3. Identify last good point.
4. Restore selectively.
5. Diff recovered data against production.
6. Resume writes.

C. Ransomware / hostile access

Trigger: suspicious encryption, unknown admin access, altered repos, leaked token.

Recovery:

1. Isolate systems.
2. Preserve logs/snapshots for evidence.
3. Revoke exposed credentials.
4. Build clean replacement environment.
5. Restore from backup before compromise time.
6. Rotate all secrets.
7. Force password/session reset where needed.
8. Publish internal incident note.

D. Provider outage

Trigger: Bubble, Stripe, OpenRouter, DNS provider, email provider unavailable.

Recovery:

1. Switch status page / static fallback.
2. Disable dependent features gracefully.
3. Queue work where possible.
4. Use alternate LLM provider path where feasible.
5. Reconcile once provider returns.

E. Loss of admin identity

Trigger: locked out of registrar, Gitea, Stripe, Bubble, email, cloud.

Recovery:

1. Use documented break-glass account.
2. Use offline recovery codes.
3. Verify ownership records.
4. Rotate compromised accounts.
5. Update evidence log.

7. Separate backup credentials from production credentials

A common failure mode: the attacker gets production admin access and deletes the backups too. So:

• Backup storage credentials must not live on the main server with delete permissions.
• Use append-only or immutable storage where possible.
• Use separate admin accounts for production, backup, registrar, and payment systems.
• Enable MFA everywhere.
• Keep offline recovery codes.
• Keep a minimal “break-glass” process documented.

8. Add monitoring and evidence

You want the system to prove itself.

Minimum checks:

Check	Frequency
Backup job succeeded	daily
Backup repository integrity check	weekly
Restore one repo to staging	weekly
Restore Gitea dump to staging	monthly
Rebuild full server from Ansible	quarterly
Rotate critical secrets	quarterly or after incident
Review asset register	monthly
Review external dependencies	monthly

Every drill should produce a short file:

TestEvidence/restore-drill-2026-07-04.md

With:

# Restore Drill: Gitea

Date: 2026-07-04
Backup used: gitea-dump-...
Target: clean staging server
Result: success / partial / failed
RTO measured: ...
RPO measured: ...
Issues found:
- ...
Actions:
- ...

This turns resilience into a learning loop.

9. Minimal viable setup

For the first version, I would implement this:

1. Create coulomb-resilience-control repo.
2. Write AssetRegister.yaml for domains, Gitea, Bubble, Stripe, OpenRouter, k3s, databases, storage, secrets.
3. Automate Gitea backup using gitea dump plus repo mirrors.
4. Automate server/content backups with encrypted restic/Borg-style offsite backup.
5. Add k3s etcd snapshots if k3s state matters; otherwise prefer full redeploy from manifests/Ansible.
6. Export Bubble data on a fixed schedule while Bubble remains in the architecture.
7. Export Stripe/accounting data to storage/warehouse or at least scheduled CSV snapshots.
8. Create one clean restore machine and perform a monthly restore drill.
9. Document secret rotation and breach recovery.
10. Create a one-page “panic sheet”: where backups are, who has access, how to rebuild, how to rotate secrets, how to switch DNS.

10. Suggested operating principle

For Coulomb / Helix Forge, I would use this rule:

Every capability that becomes valuable must declare its source of truth, recovery method, backup schedule, RTO, RPO, owner, and restore test evidence.

That fits naturally with your capability-registry thinking. A repo or service is not “production-worthy” until its recovery story is known.

A good next artifact would be a ResiliencePolicy.md plus AssetRegister.yaml seed for your current Coulomb/Helix stack.

xxx