Bootstrap resilience control plane: intent, risk analysis, first workplan
- INTENT.md: purpose and boundaries of the resilience control plane - specs/ResilienceControlSetup.md: design exploration (tiers, RTO/RPO, 3-2-1) - history/2026-07-04-initial-risk-analysis.md: 12 failure modes ranked, mapped to artifacts - workplans/DISCTL-WP-0002-resilience-foundation.md: registers, policy, first tested restore - DISCTL-WP-0001 T03 (seed first real workplan) done Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
96afc72b8a
commit
2f6f489289
5 changed files with 650 additions and 0 deletions
29
INTENT.md
Normal file
29
INTENT.md
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
# INTENT
|
||||
|
||||
## Purpose
|
||||
|
||||
`disaster-control` is the **Resilience Control Plane** for Coulomb Social / Helix Forge and adjacent endeavors. It is the single source of truth for recovery — independent of any one server — answering: what exists, what must survive, how it is rebuilt, and whether restore actually works.
|
||||
|
||||
The framing is **business continuity + disaster recovery + cyber recovery + evidence**, informed by ISO 22301, NIST CSF 2.0, NIST SP 800-34, and CISA backup guidance.
|
||||
|
||||
## What lives here
|
||||
|
||||
- **Registers** — `AssetRegister.yaml`, `DataRegister.yaml`, service dependency map
|
||||
- **Policy** — `ResiliencePolicy.md`, `BackupPolicy.md`, RTO/RPO matrix
|
||||
- **Restore runbooks** — full server loss, Gitea restore, k3s restore, Bubble export restore, breach recovery, secret rotation
|
||||
- **Incident playbooks** — ransomware, credential leak, accidental delete, provider outage, data corruption
|
||||
- **Test evidence** — dated restore-drill reports proving RTO/RPO are actually met
|
||||
|
||||
## Operating principle
|
||||
|
||||
> Every capability that becomes valuable must declare its source of truth, recovery method, backup schedule, RTO, RPO, owner, and restore test evidence.
|
||||
|
||||
Servers are disposable; data, identity, secrets, evidence, and payment/account records are not. Rebuild (Ansible/IaC + data restore + secret recovery + DNS cutover) is the default recovery pattern, not server resurrection.
|
||||
|
||||
## Boundaries
|
||||
|
||||
This repo documents, plans, and evidences resilience for the Coulomb stack. It does not host application runtime code, and it does not own the backup tooling itself (restic, Velero, `gitea dump`, etc. run where the infrastructure lives). Extracting the control plane into a standalone product is explicitly deferred.
|
||||
|
||||
## Current state
|
||||
|
||||
Early exploration: `specs/ResilienceControlSetup.md` outlines the full design (asset tiers, RTO/RPO targets, 3-2-1 + immutable backup architecture, playbooks, drill cadence). The registers, policies, runbooks, and evidence loop are not yet written.
|
||||
149
history/2026-07-04-initial-risk-analysis.md
Normal file
149
history/2026-07-04-initial-risk-analysis.md
Normal file
|
|
@ -0,0 +1,149 @@
|
|||
# Initial Risk Analysis — Failure Modes for disaster-control
|
||||
|
||||
Date: 2026-07-04
|
||||
Author: claude-code (session with Bernd)
|
||||
Status: baseline — to be revisited after AssetRegister and first restore drill
|
||||
Source: `specs/ResilienceControlSetup.md`, `INTENT.md`, current Coulomb/Helix stack
|
||||
(Gitea, k3s, CoulombCore, Bubble.io, Stripe, OpenRouter, DNS, secrets, Ansible)
|
||||
|
||||
## Purpose
|
||||
|
||||
Enumerate the failure modes the Resilience Control Plane must address, rank them,
|
||||
and map each to the artifact (register, policy, runbook, playbook, drill) that
|
||||
mitigates it. This is the justification layer for the initial workplans.
|
||||
|
||||
## Scoring
|
||||
|
||||
- **Likelihood** L1 (rare) – L5 (expected within a year)
|
||||
- **Impact** I1 (nuisance) – I5 (existential: money, identity, or irreplaceable data lost)
|
||||
- **Exposure today** = how well the current setup handles it *before* this repo does its job
|
||||
|
||||
## Failure modes
|
||||
|
||||
### R1 — Full production server loss (CoulombCore)
|
||||
|
||||
Disk failure, VPS deletion, provider account closure, fat-fingered teardown.
|
||||
**L3 / I4.** Exposure today: high — rebuild path exists conceptually (Ansible)
|
||||
but is not documented end-to-end and has never been drilled.
|
||||
**Mitigation:** restore runbook `01-full-server-loss`, Ansible baseline as the
|
||||
recovery asset, offsite encrypted backups, DNS cutover procedure, quarterly rebuild drill.
|
||||
|
||||
### R2 — Backup exists but restore fails (untested backups)
|
||||
|
||||
The classic silent failure: dumps run for months, first real restore reveals
|
||||
corruption, missing volumes, undumped databases, or an unknown passphrase.
|
||||
**L4 / I5** (it converts every other risk into a total loss). Exposure today:
|
||||
critical — no restore has ever been evidenced.
|
||||
**Mitigation:** the TestEvidence loop itself — monthly restore drills with dated
|
||||
reports measuring actual RTO/RPO. This is the highest-leverage artifact in the repo.
|
||||
|
||||
### R3 — Ransomware / hostile access deletes backups too
|
||||
|
||||
Attacker with production admin credentials encrypts data *and* deletes reachable
|
||||
backups. **L2 / I5.** Exposure today: high if backup credentials live on the
|
||||
production host with delete permission.
|
||||
**Mitigation:** backup credential separation (BackupPolicy §credentials),
|
||||
immutable/append-only offsite copy, offline weekly copy, breach-recovery runbook
|
||||
(`never restore into the compromised machine`), secret rotation runbook.
|
||||
|
||||
### R4 — Loss of admin identity (registrar, Stripe, Gitea, email, cloud)
|
||||
|
||||
Locked out via lost MFA device, expired recovery email, account takeover, or
|
||||
provider suspension. **L2 / I5** — identity is Tier 0; without it nothing else
|
||||
can be recovered. Exposure today: unknown — no inventory of recovery codes,
|
||||
backup admins, or break-glass paths exists.
|
||||
**Mitigation:** AssetRegister T0 section with recovery paths per account,
|
||||
offline recovery codes, break-glass process, loss-of-admin-identity playbook.
|
||||
|
||||
### R5 — Secrets lost or leaked
|
||||
|
||||
Two directions: (a) recovery material lost → backups undecryptable, services
|
||||
unrebuildable; (b) secret leaked → every dependent system compromised.
|
||||
**L3 / I5.** Exposure today: medium-high — OpenBao/warden infrastructure exists
|
||||
but sealed *recovery* material for disaster scenarios is not evidenced.
|
||||
**Mitigation:** encrypted (SOPS/age) sealed recovery vault backed up offline,
|
||||
secret-rotation runbook, credential-leak playbook. Restore drills must include
|
||||
"can we actually decrypt the vault from cold".
|
||||
|
||||
### R6 — Accidental deletion / bad migration / data corruption
|
||||
|
||||
Deleted repo, destructive script, corrupted DB, bad Ansible run. **L4 / I3**
|
||||
(bounded if backups have granularity and short RPO). Exposure today: medium.
|
||||
**Mitigation:** accidental-delete and data-corruption playbooks (stop writes →
|
||||
staging restore → identify last good point → selective restore → diff), local
|
||||
snapshots for fast rollback, RPO targets per system in the RTO/RPO matrix.
|
||||
|
||||
### R7 — External provider outage or exit (Bubble, Stripe, OpenRouter, DNS, email)
|
||||
|
||||
Outage is L4/I2 (degradation); *forced exit* (provider shutdown, account ban,
|
||||
pricing change) is L2/I4 — especially Bubble, which holds member data on a
|
||||
platform we don't control. Exposure today: high for Bubble (export cadence not
|
||||
established), low-medium for others.
|
||||
**Mitigation:** T4 section of AssetRegister documenting export paths and limits,
|
||||
scheduled Bubble and Stripe exports, provider-outage playbook with graceful
|
||||
degradation, alternate-LLM path for OpenRouter.
|
||||
|
||||
### R8 — DNS / domain hijack or lapse
|
||||
|
||||
Expired domain, registrar compromise, hostile transfer. **L1 / I5.**
|
||||
Exposure today: unknown (lock status, MFA, backup contacts uninventoried).
|
||||
**Mitigation:** registrar lock + MFA + backup admin recorded in AssetRegister,
|
||||
zone-record exports after changes, 2h RTO cutover procedure.
|
||||
|
||||
### R9 — Silent backup failure (job stops, nobody notices)
|
||||
|
||||
Cron dies, disk fills, token expires; discovered only at restore time.
|
||||
**L4 / I4** (degenerates into R2). Exposure today: high — no backup success
|
||||
monitoring is defined.
|
||||
**Mitigation:** monitoring & evidence checks (daily job-success check, weekly
|
||||
integrity check) defined in BackupPolicy with an explicit alerting owner.
|
||||
|
||||
### R10 — The control plane itself is lost or stale
|
||||
|
||||
Two sub-modes: (a) this repo lives on the same Gitea it protects — a full-loss
|
||||
event takes the recovery instructions down with it; (b) docs drift from reality
|
||||
and runbooks silently rot. **L3 / I4.** Exposure today: high — repo is
|
||||
single-homed as of this writing.
|
||||
**Mitigation:** mirror disaster-control to at least one independent location
|
||||
(and one offline/printable panic sheet), monthly asset-register review, drills
|
||||
double as doc-verification.
|
||||
|
||||
### R11 — Single-operator dependency (bus factor = 1)
|
||||
|
||||
Bernd is the only person who knows where backups are, how to rotate secrets,
|
||||
and holds the MFA devices. **L2 / I5.** Exposure today: structural.
|
||||
**Mitigation:** the panic sheet ("where backups are, who has access, how to
|
||||
rebuild") written so a competent stranger could execute it; offline recovery
|
||||
material reachable by a designated second person.
|
||||
|
||||
### R12 — Payment/financial record loss (Stripe divergence)
|
||||
|
||||
Stripe holds the authoritative customer/payment state; account loss or
|
||||
membership-state divergence with Bubble breaks the business's money loop.
|
||||
**L2 / I4.** Exposure today: medium.
|
||||
**Mitigation:** scheduled Stripe exports (CSV or Data Pipeline) into backed-up
|
||||
storage, reconciliation step in provider-outage playbook.
|
||||
|
||||
## Priority ranking (exposure × impact)
|
||||
|
||||
1. **R2** untested restores — invalidates everything else; fix via evidence loop
|
||||
2. **R9** silent backup failure — feeds R2 directly
|
||||
3. **R3** backup-deleting breach — credential separation is cheap and decisive
|
||||
4. **R4/R5** identity & secrets recovery — T0, low effort to inventory, existential if wrong
|
||||
5. **R10** control plane single-homed — mirror this repo immediately (near-zero cost)
|
||||
6. **R1** full server loss — the flagship runbook, exercised by drills
|
||||
7. **R7/R12** provider exports (Bubble, Stripe) — schedule and verify
|
||||
8. **R6, R8, R11** — covered incrementally by playbooks and the panic sheet
|
||||
|
||||
## Consequences for the workplan
|
||||
|
||||
- First workplan must deliver: RTO/RPO matrix + ResiliencePolicy (targets),
|
||||
AssetRegister/DataRegister (inventory incl. T0 recovery paths), BackupPolicy
|
||||
with credential-separation rules, the full-server-loss runbook, a panic sheet,
|
||||
and **one real restore drill with an evidence file** — because until R2 is
|
||||
exercised once, everything else is theory.
|
||||
- Mirroring this repo off the primary Gitea (R10) is a day-one ad hoc task,
|
||||
not a workplan phase.
|
||||
- Backup *automation* (restic, gitea dump, etcd snapshots, Bubble/Stripe export
|
||||
jobs) runs where the infrastructure lives; this repo tracks its policy,
|
||||
schedule, and evidence — see INTENT.md boundaries.
|
||||
294
specs/ResilienceControlSetup.md
Normal file
294
specs/ResilienceControlSetup.md
Normal file
|
|
@ -0,0 +1,294 @@
|
|||
Resilience Control Setup Exploration
|
||||
|
||||
## Introduction
|
||||
|
||||
This is a rough outline for how we will secure Coulomb Social / Helix Forge against disasters.
|
||||
|
||||
We will **not** start with “buy a backup tool.” instead we will set up a small **Resilience Control Plane**:
|
||||
a documented, automated, tested system that knows what exists, what must survive, how it is rebuilt, and whether restore actually works.
|
||||
|
||||
The useful framing is: **business continuity + disaster recovery + cyber recovery + evidence**. ISO 22301 is the standard family for business-continuity management, while NIST CSF 2.0 frames cyber resilience around Govern, Identify, Protect, Detect, Respond, and Recover. NIST SP 800-34 remains a practical reference for IT contingency planning, and CISA explicitly recommends offline, encrypted backups that are regularly tested. ([ISO][1])
|
||||
|
||||
Remark: The Resilience Control Plane might evolve into a product of it's own later. The repository disaster-control for now is about resilience for the coulomb project and adjacent endevors. Extracting a product is for a later day.
|
||||
|
||||
## 1. Treat Coulomb / Helix as a recoverable organization
|
||||
|
||||
Given your current shape — Bubble.io legacy app, Stripe memberships, OpenRouter usage, CoulombCore on Ubuntu, Gitea, Decap CMS, k3s, Ansible, AI/webhook services, many repos — create one repo, for example:
|
||||
|
||||
`disaster-control`
|
||||
|
||||
Inside it:
|
||||
|
||||
```text
|
||||
disaster-control/
|
||||
INTENT.md
|
||||
ResiliencePolicy.md
|
||||
AssetRegister.yaml
|
||||
DataRegister.yaml
|
||||
ServiceDependencyMap.md
|
||||
RTO-RPO-Matrix.md
|
||||
BackupPolicy.md
|
||||
RestoreRunbooks/
|
||||
01-full-server-loss.md
|
||||
02-gitea-restore.md
|
||||
03-k3s-restore.md
|
||||
04-bubble-export-restore.md
|
||||
05-security-breach-recovery.md
|
||||
06-secret-rotation.md
|
||||
IncidentPlaybooks/
|
||||
ransomware.md
|
||||
credential-leak.md
|
||||
accidental-delete.md
|
||||
provider-outage.md
|
||||
data-corruption.md
|
||||
TestEvidence/
|
||||
restore-drill-YYYY-MM-DD.md
|
||||
```
|
||||
|
||||
This becomes the “single source of truth” for recovery, independent of any one server.
|
||||
|
||||
## 2. Classify assets by recovery importance
|
||||
|
||||
Use tiers. This prevents over-engineering everything.
|
||||
|
||||
| Tier | Asset type | Examples | Target |
|
||||
| ------------------------------------ | ----------------------------------------------------------------------------------------- | ------------------------------------------ | ------ |
|
||||
| **T0 Identity / trust / money** | Domains, DNS, registrar, email, Stripe, OAuth, admin accounts, secrets, SSH keys | Must be recoverable even after breach | |
|
||||
| **T1 Irreplaceable source of truth** | Gitea repos, CoulombCore content, user/member data, product registry, paid customer state | Strong backups, tested restore | |
|
||||
| **T2 Runtime platforms** | Ubuntu server, k3s, Gitea instance, Decap CMS, webhooks, databases | Rebuild from IaC + restore data | |
|
||||
| **T3 Derived / rebuildable** | caches, generated artifacts, embeddings, logs beyond retention, local model outputs | Recompute where possible | |
|
||||
| **T4 External dependencies** | Bubble.io, Stripe, OpenRouter, domain/DNS provider, email provider | Export, monitor, fallback, document limits | |
|
||||
|
||||
The key idea: **servers are disposable; data, identity, secrets, evidence, and payment/account records are not.**
|
||||
|
||||
## 3. Define RTO and RPO per system
|
||||
|
||||
Use explicit recovery objectives:
|
||||
|
||||
| System | Suggested RPO | Suggested RTO | Notes |
|
||||
| ------------------------------ | -----------------------------------------: | ------------: | ------------------------------------------------------------------------------------------------------ |
|
||||
| Gitea repos | 1h or better | 4–8h | Mirror repos plus Gitea dump |
|
||||
| CoulombCore content | 1h–6h | 4–8h | Back up content store + DB |
|
||||
| k3s cluster state | 1h–24h | 8h | Prefer rebuildable cluster + snapshots |
|
||||
| Bubble.io database | 24h initially | 24h | Export regularly; Bubble has database restore/export capabilities ([manual.bubble.io][2]) |
|
||||
| Stripe financial/customer data | 24h–7d | 24h | Stripe can export/sync data to storage/warehouse destinations ([Stripe Dokumentation][3]) |
|
||||
| Secrets | immediate rotation plan | 2–4h | Back up encrypted recovery material, not plaintext secrets |
|
||||
| DNS/domains | manual export after changes | 2h | Registrar lock + MFA + backup admin |
|
||||
| OpenRouter dependency | no data RPO unless storing prompts/results | 1–4h fallback | Track API keys, quotas, usage; OpenRouter exposes key/limit/credit info via API/docs ([OpenRouter][4]) |
|
||||
|
||||
For an early-stage environment, I would optimize for **RPO ≤ 24h everywhere, RTO ≤ 1 day for full recovery, and RTO ≤ 4–8h for code/content/control-plane recovery**.
|
||||
|
||||
## 4. Implement the backup architecture
|
||||
|
||||
Use a practical **3-2-1 + immutable/offline + tested restore** model: at least three copies, on two storage types, one offsite; plus one copy that an attacker cannot easily delete or encrypt. The UK NCSC describes 3-2-1 as a common resilient-backup pattern, and CISA emphasizes offline, encrypted, regularly tested backups for ransomware resilience. ([National Cyber Security Centre][5])
|
||||
|
||||
For Coulomb / Helix, I would set it up like this:
|
||||
|
||||
```text
|
||||
Production server
|
||||
├─ local snapshots for fast rollback
|
||||
├─ encrypted daily backup to offsite object storage
|
||||
├─ immutable/offline weekly backup
|
||||
└─ monthly restore drill to clean machine
|
||||
```
|
||||
|
||||
Concrete backup streams:
|
||||
|
||||
| Stream | Mechanism |
|
||||
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| **Gitea** | Scheduled `gitea dump` plus repo mirrors. Gitea’s official docs describe `dump` as creating a ZIP backup usable for restore. ([Gitea Dokumentation][6]) |
|
||||
| **Git repos** | Mirror clone to separate location/provider. GitHub’s docs recommend `git clone --mirror` for repository backup including history. ([GitHub Docs][7]) |
|
||||
| **k3s** | k3s datastore snapshots. k3s documents `etcd-snapshot` for managing and restoring embedded etcd snapshots. ([K3s][8]) |
|
||||
| **Kubernetes resources/PVs** | Velero for cluster resources and persistent volumes. Velero is designed for Kubernetes backup, restore, migration, and cluster replication. ([Velero][9]) |
|
||||
| **Files / content / configs** | Restic or Borg-style encrypted backups. Restic supports encrypted backups to local, self-hosted, and online storage. ([Restic][10]) |
|
||||
| **Bubble.io** | Scheduled app/data exports and documented restore workflow. Bubble documents database restore and export/import tooling. ([manual.bubble.io][2]) |
|
||||
| **Stripe** | Periodic CSV exports or Stripe Data Pipeline/cloud storage export for accounting/customer continuity. ([Stripe Dokumentation][3]) |
|
||||
| **Secrets** | SOPS/age-encrypted sealed backup of recovery secrets, plus printed/offline recovery material where appropriate. After breach: rotate, don’t blindly restore. |
|
||||
| **DNS / domains** | Export zone records, registrar settings, recovery codes, admin contacts, and MFA recovery process. |
|
||||
|
||||
## 5. Make rebuild the default, not server resurrection
|
||||
|
||||
For the Ubuntu/k3s/Gitea environment, the best recovery pattern is:
|
||||
|
||||
```text
|
||||
New clean machine
|
||||
→ install base OS
|
||||
→ apply Ansible
|
||||
→ restore secrets from encrypted recovery vault
|
||||
→ restore Gitea / DB / content
|
||||
→ restore k3s state or redeploy manifests
|
||||
→ validate services
|
||||
→ switch DNS
|
||||
```
|
||||
|
||||
This means your real disaster-recovery asset is not the server image. It is:
|
||||
|
||||
1. **Ansible/IaC**
|
||||
2. **data backups**
|
||||
3. **secret recovery**
|
||||
4. **DNS cutover**
|
||||
5. **tested runbooks**
|
||||
|
||||
For a security breach, never restore into the compromised machine. Build a clean environment, restore only known-good data, rotate credentials, and preserve evidence.
|
||||
|
||||
## 6. Create incident playbooks
|
||||
|
||||
You need at least five.
|
||||
|
||||
### A. Full server loss
|
||||
|
||||
Trigger: disk failure, VPS deletion, hardware loss.
|
||||
|
||||
Recovery:
|
||||
|
||||
1. Provision new server.
|
||||
2. Apply Ansible baseline.
|
||||
3. Restore Gitea dump and repo mirrors.
|
||||
4. Restore databases/content.
|
||||
5. Restore k3s from snapshot or redeploy workloads.
|
||||
6. Validate health checks.
|
||||
7. Move DNS.
|
||||
8. Write evidence report.
|
||||
|
||||
### B. Accidental deletion or bad migration
|
||||
|
||||
Trigger: deleted repo, corrupted database, bad script.
|
||||
|
||||
Recovery:
|
||||
|
||||
1. Stop writes.
|
||||
2. Clone latest backup into staging.
|
||||
3. Identify last good point.
|
||||
4. Restore selectively.
|
||||
5. Diff recovered data against production.
|
||||
6. Resume writes.
|
||||
|
||||
### C. Ransomware / hostile access
|
||||
|
||||
Trigger: suspicious encryption, unknown admin access, altered repos, leaked token.
|
||||
|
||||
Recovery:
|
||||
|
||||
1. Isolate systems.
|
||||
2. Preserve logs/snapshots for evidence.
|
||||
3. Revoke exposed credentials.
|
||||
4. Build clean replacement environment.
|
||||
5. Restore from backup before compromise time.
|
||||
6. Rotate all secrets.
|
||||
7. Force password/session reset where needed.
|
||||
8. Publish internal incident note.
|
||||
|
||||
### D. Provider outage
|
||||
|
||||
Trigger: Bubble, Stripe, OpenRouter, DNS provider, email provider unavailable.
|
||||
|
||||
Recovery:
|
||||
|
||||
1. Switch status page / static fallback.
|
||||
2. Disable dependent features gracefully.
|
||||
3. Queue work where possible.
|
||||
4. Use alternate LLM provider path where feasible.
|
||||
5. Reconcile once provider returns.
|
||||
|
||||
### E. Loss of admin identity
|
||||
|
||||
Trigger: locked out of registrar, Gitea, Stripe, Bubble, email, cloud.
|
||||
|
||||
Recovery:
|
||||
|
||||
1. Use documented break-glass account.
|
||||
2. Use offline recovery codes.
|
||||
3. Verify ownership records.
|
||||
4. Rotate compromised accounts.
|
||||
5. Update evidence log.
|
||||
|
||||
## 7. Separate backup credentials from production credentials
|
||||
|
||||
A common failure mode: the attacker gets production admin access and deletes the backups too. So:
|
||||
|
||||
* Backup storage credentials must **not** live on the main server with delete permissions.
|
||||
* Use append-only or immutable storage where possible.
|
||||
* Use separate admin accounts for production, backup, registrar, and payment systems.
|
||||
* Enable MFA everywhere.
|
||||
* Keep offline recovery codes.
|
||||
* Keep a minimal “break-glass” process documented.
|
||||
|
||||
## 8. Add monitoring and evidence
|
||||
|
||||
You want the system to prove itself.
|
||||
|
||||
Minimum checks:
|
||||
|
||||
| Check | Frequency |
|
||||
| --------------------------------- | --------------------------: |
|
||||
| Backup job succeeded | daily |
|
||||
| Backup repository integrity check | weekly |
|
||||
| Restore one repo to staging | weekly |
|
||||
| Restore Gitea dump to staging | monthly |
|
||||
| Rebuild full server from Ansible | quarterly |
|
||||
| Rotate critical secrets | quarterly or after incident |
|
||||
| Review asset register | monthly |
|
||||
| Review external dependencies | monthly |
|
||||
|
||||
Every drill should produce a short file:
|
||||
|
||||
```text
|
||||
TestEvidence/restore-drill-2026-07-04.md
|
||||
```
|
||||
|
||||
With:
|
||||
|
||||
```markdown
|
||||
# Restore Drill: Gitea
|
||||
|
||||
Date: 2026-07-04
|
||||
Backup used: gitea-dump-...
|
||||
Target: clean staging server
|
||||
Result: success / partial / failed
|
||||
RTO measured: ...
|
||||
RPO measured: ...
|
||||
Issues found:
|
||||
- ...
|
||||
Actions:
|
||||
- ...
|
||||
```
|
||||
|
||||
This turns resilience into a learning loop.
|
||||
|
||||
## 9. Minimal viable setup
|
||||
|
||||
For the first version, I would implement this:
|
||||
|
||||
1. **Create `coulomb-resilience-control` repo.**
|
||||
2. **Write AssetRegister.yaml** for domains, Gitea, Bubble, Stripe, OpenRouter, k3s, databases, storage, secrets.
|
||||
3. **Automate Gitea backup** using `gitea dump` plus repo mirrors.
|
||||
4. **Automate server/content backups** with encrypted restic/Borg-style offsite backup.
|
||||
5. **Add k3s etcd snapshots** if k3s state matters; otherwise prefer full redeploy from manifests/Ansible.
|
||||
6. **Export Bubble data** on a fixed schedule while Bubble remains in the architecture.
|
||||
7. **Export Stripe/accounting data** to storage/warehouse or at least scheduled CSV snapshots.
|
||||
8. **Create one clean restore machine** and perform a monthly restore drill.
|
||||
9. **Document secret rotation and breach recovery.**
|
||||
10. **Create a one-page “panic sheet”**: where backups are, who has access, how to rebuild, how to rotate secrets, how to switch DNS.
|
||||
|
||||
## 10. Suggested operating principle
|
||||
|
||||
For Coulomb / Helix Forge, I would use this rule:
|
||||
|
||||
> **Every capability that becomes valuable must declare its source of truth, recovery method, backup schedule, RTO, RPO, owner, and restore test evidence.**
|
||||
|
||||
That fits naturally with your capability-registry thinking. A repo or service is not “production-worthy” until its recovery story is known.
|
||||
|
||||
A good next artifact would be a `ResiliencePolicy.md` plus `AssetRegister.yaml` seed for your current Coulomb/Helix stack.
|
||||
|
||||
[1]: https://www.iso.org/standard/75106.html?utm_source=chatgpt.com "ISO 22301:2019 - Business continuity management systems"
|
||||
[2]: https://manual.bubble.io/help-guides/maintaining-an-application/database-maintenance/database-copy-and-backups?utm_source=chatgpt.com "Restoring database backups"
|
||||
[3]: https://docs.stripe.com/data/data-pipeline?utm_source=chatgpt.com "Data Pipeline | Stripe Documentation"
|
||||
[4]: https://openrouter.ai/docs/api/reference/limits?utm_source=chatgpt.com "API Rate Limits - Manage Model Usage and Quotas"
|
||||
[5]: https://www.ncsc.gov.uk/blog-post/offline-backups-in-an-online-world?utm_source=chatgpt.com "Offline backups in an online world"
|
||||
[6]: https://docs.gitea.com/administration/backup-and-restore?utm_source=chatgpt.com "Backup and Restore"
|
||||
[7]: https://docs.github.com/en/repositories/archiving-a-github-repository/backing-up-a-repository?utm_source=chatgpt.com "Backing up a repository"
|
||||
[8]: https://docs.k3s.io/cli/etcd-snapshot?utm_source=chatgpt.com "k3s etcd-snapshot"
|
||||
[9]: https://velero.io/docs/main/?utm_source=chatgpt.com "Velero Docs - Overview"
|
||||
[10]: https://restic.net/?utm_source=chatgpt.com "restic · Backups done right!"
|
||||
|
||||
xxx
|
||||
58
workplans/DISCTL-WP-0001-statehub-bootstrap.md
Normal file
58
workplans/DISCTL-WP-0001-statehub-bootstrap.md
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
---
|
||||
id: DISCTL-WP-0001
|
||||
type: workplan
|
||||
title: "Bootstrap State Hub integration"
|
||||
domain: infotech
|
||||
repo: disaster-control
|
||||
status: ready
|
||||
owner: codex
|
||||
topic_slug: custodian
|
||||
created: "2026-07-04"
|
||||
updated: "2026-07-04"
|
||||
---
|
||||
|
||||
# Bootstrap State Hub integration
|
||||
|
||||
Resilience control plane for Coulomb Social / Helix Forge: the single source of truth for business continuity, disaster recovery, cyber recovery, and restore-test evidence covering the Coulomb infrastructure (Gitea, k3s, CoulombCore, Bubble.io, Stripe, DNS, secrets).
|
||||
|
||||
## Review Generated Integration Files
|
||||
|
||||
```task
|
||||
id: DISCTL-WP-0001-T01
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Review `INTENT.md`, `SCOPE.md`, `AGENTS.md`, and `.custodian-brief.md`.
|
||||
Replace generated placeholders with repo-specific facts where needed.
|
||||
|
||||
## Verify Local Developer Workflow
|
||||
|
||||
```task
|
||||
id: DISCTL-WP-0001-T02
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Identify the repo's install, test, lint, build, and run commands. Add or refine
|
||||
those commands in the agent instructions so future coding sessions can verify
|
||||
changes confidently.
|
||||
|
||||
## Seed First Real Workplan
|
||||
|
||||
```task
|
||||
id: DISCTL-WP-0001-T03
|
||||
status: done
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Result 2026-07-04: Created `DISCTL-WP-0002-resilience-foundation.md`, derived
|
||||
from the initial risk analysis in `history/2026-07-04-initial-risk-analysis.md`.
|
||||
|
||||
Create the first implementation workplan for the repository's most important
|
||||
next change. After workplan file updates, run the sync locally from this repo
|
||||
checkout:
|
||||
|
||||
```bash
|
||||
statehub fix-consistency
|
||||
```
|
||||
120
workplans/DISCTL-WP-0002-resilience-foundation.md
Normal file
120
workplans/DISCTL-WP-0002-resilience-foundation.md
Normal file
|
|
@ -0,0 +1,120 @@
|
|||
---
|
||||
id: DISCTL-WP-0002
|
||||
type: workplan
|
||||
title: "Resilience foundation: registers, policy, first tested restore"
|
||||
domain: infotech
|
||||
repo: disaster-control
|
||||
status: ready
|
||||
owner: claude-code
|
||||
topic_slug: custodian
|
||||
created: "2026-07-04"
|
||||
updated: "2026-07-04"
|
||||
---
|
||||
|
||||
# Resilience foundation: registers, policy, first tested restore
|
||||
|
||||
Stand up the minimum viable Resilience Control Plane for Coulomb Social /
|
||||
Helix Forge: know what exists (registers), what must survive (policy, RTO/RPO),
|
||||
how it comes back (runbook, panic sheet), and prove it once (drill + evidence).
|
||||
|
||||
Driven by `history/2026-07-04-initial-risk-analysis.md` — priority order R2
|
||||
(untested restores), R9 (silent backup failure), R3 (backup-deleting breach),
|
||||
R4/R5 (identity & secret recovery), R10 (control plane single-homed), R1
|
||||
(full server loss). Design source: `specs/ResilienceControlSetup.md`.
|
||||
|
||||
## Mirror disaster-control off the primary Gitea (R10)
|
||||
|
||||
```task
|
||||
id: DISCTL-WP-0002-T01
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
This repo must survive the disasters it describes. Add at least one mirror on
|
||||
infrastructure independent of CoulombCore/Gitea (e.g. GitHub private mirror or
|
||||
secondary machine), document the mirror location in README, and note how the
|
||||
mirror is refreshed.
|
||||
|
||||
## Asset and data registers with Tier-0 recovery paths (R4, R7, R8, R12)
|
||||
|
||||
```task
|
||||
id: DISCTL-WP-0002-T02
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Write `AssetRegister.yaml` and `DataRegister.yaml` seeded with the current
|
||||
stack: domains/DNS/registrar, email, Stripe, Bubble.io, OpenRouter, Gitea,
|
||||
CoulombCore content, k3s, databases, backup storage, secrets/OpenBao. Every T0
|
||||
entry must record its recovery path (backup admin, MFA recovery, break-glass).
|
||||
Every T1 entry must name its source of truth and backup mechanism (or mark it
|
||||
`backup: none` honestly). Use the tier model from the spec (§2).
|
||||
|
||||
## ResiliencePolicy and RTO/RPO matrix (R1, R6)
|
||||
|
||||
```task
|
||||
id: DISCTL-WP-0002-T03
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Write `ResiliencePolicy.md` (operating principle, tier definitions, drill
|
||||
cadence, evidence requirement) and `RTO-RPO-Matrix.md` with explicit per-system
|
||||
targets, starting from the spec's suggested values (§3): RPO ≤ 24h everywhere,
|
||||
RTO ≤ 1 day full recovery, RTO ≤ 4–8h for code/content/control plane.
|
||||
|
||||
## BackupPolicy with credential separation and monitoring (R3, R9)
|
||||
|
||||
```task
|
||||
id: DISCTL-WP-0002-T04
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Write `BackupPolicy.md`: 3-2-1 + immutable/offline model, per-stream mechanisms
|
||||
(gitea dump + mirrors, restic, k3s etcd snapshots, Bubble/Stripe exports,
|
||||
SOPS/age secret vault), and two hard rules with verification steps —
|
||||
(a) backup-storage credentials never live on the production host with delete
|
||||
permission; (b) backup job success and repository integrity are checked on a
|
||||
defined schedule with a named alerting path.
|
||||
|
||||
## Full-server-loss restore runbook and panic sheet (R1, R11)
|
||||
|
||||
```task
|
||||
id: DISCTL-WP-0002-T05
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Write `RestoreRunbooks/01-full-server-loss.md` (new machine → Ansible → secret
|
||||
recovery → data restore → k3s redeploy → validate → DNS cutover → evidence
|
||||
report) and the one-page panic sheet: where backups are, who has access, how to
|
||||
rebuild, how to rotate secrets, how to switch DNS — written so a competent
|
||||
stranger could execute it.
|
||||
|
||||
## First restore drill with evidence file (R2)
|
||||
|
||||
```task
|
||||
id: DISCTL-WP-0002-T06
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Perform one real restore to a clean target — smallest meaningful scope (e.g.
|
||||
one Gitea repo from a `gitea dump`, or the secret vault decrypted from cold) —
|
||||
and write `TestEvidence/restore-drill-2026-MM-DD.md` with measured RTO/RPO,
|
||||
result, issues found, and follow-up actions. This task is the exit criterion
|
||||
for the workplan: no drill evidence, not finished.
|
||||
|
||||
## Incident playbooks (R3, R4, R5, R6, R7)
|
||||
|
||||
```task
|
||||
id: DISCTL-WP-0002-T07
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Write the five playbooks under `IncidentPlaybooks/`: ransomware/hostile access,
|
||||
credential leak, accidental delete / data corruption, provider outage
|
||||
(Bubble/Stripe/OpenRouter/DNS/email), loss of admin identity. Keep each to one
|
||||
page of numbered steps per the spec (§6).
|
||||
Loading…
Add table
Add a link
Reference in a new issue