Bootstrap resilience control plane: intent, risk analysis, first workplan
- INTENT.md: purpose and boundaries of the resilience control plane - specs/ResilienceControlSetup.md: design exploration (tiers, RTO/RPO, 3-2-1) - history/2026-07-04-initial-risk-analysis.md: 12 failure modes ranked, mapped to artifacts - workplans/DISCTL-WP-0002-resilience-foundation.md: registers, policy, first tested restore - DISCTL-WP-0001 T03 (seed first real workplan) done Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
96afc72b8a
commit
2f6f489289
5 changed files with 650 additions and 0 deletions
149
history/2026-07-04-initial-risk-analysis.md
Normal file
149
history/2026-07-04-initial-risk-analysis.md
Normal file
|
|
@ -0,0 +1,149 @@
|
|||
# Initial Risk Analysis — Failure Modes for disaster-control
|
||||
|
||||
Date: 2026-07-04
|
||||
Author: claude-code (session with Bernd)
|
||||
Status: baseline — to be revisited after AssetRegister and first restore drill
|
||||
Source: `specs/ResilienceControlSetup.md`, `INTENT.md`, current Coulomb/Helix stack
|
||||
(Gitea, k3s, CoulombCore, Bubble.io, Stripe, OpenRouter, DNS, secrets, Ansible)
|
||||
|
||||
## Purpose
|
||||
|
||||
Enumerate the failure modes the Resilience Control Plane must address, rank them,
|
||||
and map each to the artifact (register, policy, runbook, playbook, drill) that
|
||||
mitigates it. This is the justification layer for the initial workplans.
|
||||
|
||||
## Scoring
|
||||
|
||||
- **Likelihood** L1 (rare) – L5 (expected within a year)
|
||||
- **Impact** I1 (nuisance) – I5 (existential: money, identity, or irreplaceable data lost)
|
||||
- **Exposure today** = how well the current setup handles it *before* this repo does its job
|
||||
|
||||
## Failure modes
|
||||
|
||||
### R1 — Full production server loss (CoulombCore)
|
||||
|
||||
Disk failure, VPS deletion, provider account closure, fat-fingered teardown.
|
||||
**L3 / I4.** Exposure today: high — rebuild path exists conceptually (Ansible)
|
||||
but is not documented end-to-end and has never been drilled.
|
||||
**Mitigation:** restore runbook `01-full-server-loss`, Ansible baseline as the
|
||||
recovery asset, offsite encrypted backups, DNS cutover procedure, quarterly rebuild drill.
|
||||
|
||||
### R2 — Backup exists but restore fails (untested backups)
|
||||
|
||||
The classic silent failure: dumps run for months, first real restore reveals
|
||||
corruption, missing volumes, undumped databases, or an unknown passphrase.
|
||||
**L4 / I5** (it converts every other risk into a total loss). Exposure today:
|
||||
critical — no restore has ever been evidenced.
|
||||
**Mitigation:** the TestEvidence loop itself — monthly restore drills with dated
|
||||
reports measuring actual RTO/RPO. This is the highest-leverage artifact in the repo.
|
||||
|
||||
### R3 — Ransomware / hostile access deletes backups too
|
||||
|
||||
Attacker with production admin credentials encrypts data *and* deletes reachable
|
||||
backups. **L2 / I5.** Exposure today: high if backup credentials live on the
|
||||
production host with delete permission.
|
||||
**Mitigation:** backup credential separation (BackupPolicy §credentials),
|
||||
immutable/append-only offsite copy, offline weekly copy, breach-recovery runbook
|
||||
(`never restore into the compromised machine`), secret rotation runbook.
|
||||
|
||||
### R4 — Loss of admin identity (registrar, Stripe, Gitea, email, cloud)
|
||||
|
||||
Locked out via lost MFA device, expired recovery email, account takeover, or
|
||||
provider suspension. **L2 / I5** — identity is Tier 0; without it nothing else
|
||||
can be recovered. Exposure today: unknown — no inventory of recovery codes,
|
||||
backup admins, or break-glass paths exists.
|
||||
**Mitigation:** AssetRegister T0 section with recovery paths per account,
|
||||
offline recovery codes, break-glass process, loss-of-admin-identity playbook.
|
||||
|
||||
### R5 — Secrets lost or leaked
|
||||
|
||||
Two directions: (a) recovery material lost → backups undecryptable, services
|
||||
unrebuildable; (b) secret leaked → every dependent system compromised.
|
||||
**L3 / I5.** Exposure today: medium-high — OpenBao/warden infrastructure exists
|
||||
but sealed *recovery* material for disaster scenarios is not evidenced.
|
||||
**Mitigation:** encrypted (SOPS/age) sealed recovery vault backed up offline,
|
||||
secret-rotation runbook, credential-leak playbook. Restore drills must include
|
||||
"can we actually decrypt the vault from cold".
|
||||
|
||||
### R6 — Accidental deletion / bad migration / data corruption
|
||||
|
||||
Deleted repo, destructive script, corrupted DB, bad Ansible run. **L4 / I3**
|
||||
(bounded if backups have granularity and short RPO). Exposure today: medium.
|
||||
**Mitigation:** accidental-delete and data-corruption playbooks (stop writes →
|
||||
staging restore → identify last good point → selective restore → diff), local
|
||||
snapshots for fast rollback, RPO targets per system in the RTO/RPO matrix.
|
||||
|
||||
### R7 — External provider outage or exit (Bubble, Stripe, OpenRouter, DNS, email)
|
||||
|
||||
Outage is L4/I2 (degradation); *forced exit* (provider shutdown, account ban,
|
||||
pricing change) is L2/I4 — especially Bubble, which holds member data on a
|
||||
platform we don't control. Exposure today: high for Bubble (export cadence not
|
||||
established), low-medium for others.
|
||||
**Mitigation:** T4 section of AssetRegister documenting export paths and limits,
|
||||
scheduled Bubble and Stripe exports, provider-outage playbook with graceful
|
||||
degradation, alternate-LLM path for OpenRouter.
|
||||
|
||||
### R8 — DNS / domain hijack or lapse
|
||||
|
||||
Expired domain, registrar compromise, hostile transfer. **L1 / I5.**
|
||||
Exposure today: unknown (lock status, MFA, backup contacts uninventoried).
|
||||
**Mitigation:** registrar lock + MFA + backup admin recorded in AssetRegister,
|
||||
zone-record exports after changes, 2h RTO cutover procedure.
|
||||
|
||||
### R9 — Silent backup failure (job stops, nobody notices)
|
||||
|
||||
Cron dies, disk fills, token expires; discovered only at restore time.
|
||||
**L4 / I4** (degenerates into R2). Exposure today: high — no backup success
|
||||
monitoring is defined.
|
||||
**Mitigation:** monitoring & evidence checks (daily job-success check, weekly
|
||||
integrity check) defined in BackupPolicy with an explicit alerting owner.
|
||||
|
||||
### R10 — The control plane itself is lost or stale
|
||||
|
||||
Two sub-modes: (a) this repo lives on the same Gitea it protects — a full-loss
|
||||
event takes the recovery instructions down with it; (b) docs drift from reality
|
||||
and runbooks silently rot. **L3 / I4.** Exposure today: high — repo is
|
||||
single-homed as of this writing.
|
||||
**Mitigation:** mirror disaster-control to at least one independent location
|
||||
(and one offline/printable panic sheet), monthly asset-register review, drills
|
||||
double as doc-verification.
|
||||
|
||||
### R11 — Single-operator dependency (bus factor = 1)
|
||||
|
||||
Bernd is the only person who knows where backups are, how to rotate secrets,
|
||||
and holds the MFA devices. **L2 / I5.** Exposure today: structural.
|
||||
**Mitigation:** the panic sheet ("where backups are, who has access, how to
|
||||
rebuild") written so a competent stranger could execute it; offline recovery
|
||||
material reachable by a designated second person.
|
||||
|
||||
### R12 — Payment/financial record loss (Stripe divergence)
|
||||
|
||||
Stripe holds the authoritative customer/payment state; account loss or
|
||||
membership-state divergence with Bubble breaks the business's money loop.
|
||||
**L2 / I4.** Exposure today: medium.
|
||||
**Mitigation:** scheduled Stripe exports (CSV or Data Pipeline) into backed-up
|
||||
storage, reconciliation step in provider-outage playbook.
|
||||
|
||||
## Priority ranking (exposure × impact)
|
||||
|
||||
1. **R2** untested restores — invalidates everything else; fix via evidence loop
|
||||
2. **R9** silent backup failure — feeds R2 directly
|
||||
3. **R3** backup-deleting breach — credential separation is cheap and decisive
|
||||
4. **R4/R5** identity & secrets recovery — T0, low effort to inventory, existential if wrong
|
||||
5. **R10** control plane single-homed — mirror this repo immediately (near-zero cost)
|
||||
6. **R1** full server loss — the flagship runbook, exercised by drills
|
||||
7. **R7/R12** provider exports (Bubble, Stripe) — schedule and verify
|
||||
8. **R6, R8, R11** — covered incrementally by playbooks and the panic sheet
|
||||
|
||||
## Consequences for the workplan
|
||||
|
||||
- First workplan must deliver: RTO/RPO matrix + ResiliencePolicy (targets),
|
||||
AssetRegister/DataRegister (inventory incl. T0 recovery paths), BackupPolicy
|
||||
with credential-separation rules, the full-server-loss runbook, a panic sheet,
|
||||
and **one real restore drill with an evidence file** — because until R2 is
|
||||
exercised once, everything else is theory.
|
||||
- Mirroring this repo off the primary Gitea (R10) is a day-one ad hoc task,
|
||||
not a workplan phase.
|
||||
- Backup *automation* (restic, gitea dump, etcd snapshots, Bubble/Stripe export
|
||||
jobs) runs where the infrastructure lives; this repo tracks its policy,
|
||||
schedule, and evidence — see INTENT.md boundaries.
|
||||
Loading…
Add table
Add a link
Reference in a new issue