disaster-control/history/2026-07-04-initial-risk-analysis.md

# Initial Risk Analysis — Failure Modes for disaster-control

Date: 2026-07-04
Author: claude-code (session with Bernd)
Status: baseline — to be revisited after AssetRegister and first restore drill
Source: `specs/ResilienceControlSetup.md`, `INTENT.md`, current Coulomb/Helix stack
(Gitea, k3s, CoulombCore, Bubble.io, Stripe, OpenRouter, DNS, secrets, Ansible)
Companion: `2026-07-04-forgejo-backup-strategy-assessment.md` — same-day
Forgejo/Railiance deep-dive with a concrete pickup queue for this repo

## Purpose

Enumerate the failure modes the Resilience Control Plane must address, rank them,
and map each to the artifact (register, policy, runbook, playbook, drill) that
mitigates it. This is the justification layer for the initial workplans.

## Scoring

- **Likelihood** L1 (rare) – L5 (expected within a year)
- **Impact** I1 (nuisance) – I5 (existential: money, identity, or irreplaceable data lost)
- **Exposure today** = how well the current setup handles it *before* this repo does its job

## Failure modes

### R1 — Full production server loss (CoulombCore)

Disk failure, VPS deletion, provider account closure, fat-fingered teardown.
**L3 / I4.** Exposure today: high — rebuild path exists conceptually (Ansible)
but is not documented end-to-end and has never been drilled.
**Mitigation:** restore runbook `01-full-server-loss`, Ansible baseline as the
recovery asset, offsite encrypted backups, DNS cutover procedure, quarterly rebuild drill.

### R2 — Backup exists but restore fails (untested backups)

The classic silent failure: dumps run for months, first real restore reveals
corruption, missing volumes, undumped databases, or an unknown passphrase.
**L4 / I5** (it converts every other risk into a total loss). Exposure today:
high — exactly one restore has been evidenced: the 2026-07-04 Forgejo isolated
restore drill (RAIL-HO-WP-0005-T09, evidence in `railiance-infra`). Gitea (the
canonical source forge today), CoulombCore content, databases, Bubble exports,
and the secret vault have never been restore-tested.
**Mitigation:** the TestEvidence loop itself — recurring restore drills with dated
reports measuring actual RTO/RPO. This is the highest-leverage artifact in the repo.

### R3 — Ransomware / hostile access deletes backups too

Attacker with production admin credentials encrypts data *and* deletes reachable
backups. **L2 / I5.** Exposure today: high if backup credentials live on the
production host with delete permission.
**Mitigation:** backup credential separation (BackupPolicy §credentials),
immutable/append-only offsite copy, offline weekly copy, breach-recovery runbook
(`never restore into the compromised machine`), secret rotation runbook.

### R4 — Loss of admin identity (registrar, Stripe, Gitea, email, cloud)

Locked out via lost MFA device, expired recovery email, account takeover, or
provider suspension. **L2 / I5** — identity is Tier 0; without it nothing else
can be recovered. Exposure today: unknown — no inventory of recovery codes,
backup admins, or break-glass paths exists.
**Mitigation:** AssetRegister T0 section with recovery paths per account,
offline recovery codes, break-glass process, loss-of-admin-identity playbook.

### R5 — Secrets lost or leaked

Two directions: (a) recovery material lost → backups undecryptable, services
unrebuildable; (b) secret leaked → every dependent system compromised.
**L3 / I5.** Exposure today: medium-high — OpenBao/warden infrastructure exists
but sealed *recovery* material for disaster scenarios is not evidenced.
**Mitigation:** encrypted (SOPS/age) sealed recovery vault backed up offline,
secret-rotation runbook, credential-leak playbook. Restore drills must include
"can we actually decrypt the vault from cold".

### R6 — Accidental deletion / bad migration / data corruption

Deleted repo, destructive script, corrupted DB, bad Ansible run. **L4 / I3**
(bounded if backups have granularity and short RPO). Exposure today: medium.
**Mitigation:** accidental-delete and data-corruption playbooks (stop writes →
staging restore → identify last good point → selective restore → diff), local
snapshots for fast rollback, RPO targets per system in the RTO/RPO matrix.

### R7 — External provider outage or exit (Bubble, Stripe, OpenRouter, DNS, email)

Outage is L4/I2 (degradation); *forced exit* (provider shutdown, account ban,
pricing change) is L2/I4 — especially Bubble, which holds member data on a
platform we don't control. Exposure today: high for Bubble (export cadence not
established), low-medium for others.
**Mitigation:** T4 section of AssetRegister documenting export paths and limits,
scheduled Bubble and Stripe exports, provider-outage playbook with graceful
degradation, alternate-LLM path for OpenRouter.

### R8 — DNS / domain hijack or lapse

Expired domain, registrar compromise, hostile transfer. **L1 / I5.**
Exposure today: unknown (lock status, MFA, backup contacts uninventoried).
**Mitigation:** registrar lock + MFA + backup admin recorded in AssetRegister,
zone-record exports after changes, 2h RTO cutover procedure.

### R9 — Silent backup failure (job stops, nobody notices)

Cron dies, disk fills, token expires; discovered only at restore time.
**L4 / I4** (degenerates into R2). Exposure today: high — no backup success
monitoring is defined.
**Mitigation:** monitoring & evidence checks (daily job-success check, weekly
integrity check) defined in BackupPolicy with an explicit alerting owner.

### R10 — The control plane itself is lost or stale

Two sub-modes: (a) this repo lives on the same Gitea it protects — a full-loss
event takes the recovery instructions down with it; (b) docs drift from reality
and runbooks silently rot. **L3 / I4.** Exposure today: high — repo is
single-homed as of this writing.
**Mitigation:** mirror disaster-control to at least one independent location
(and one offline/printable panic sheet), monthly asset-register review, drills
double as doc-verification.

### R11 — Single-operator dependency (bus factor = 1)

Bernd is the only person who knows where backups are, how to rotate secrets,
and holds the MFA devices. **L2 / I5.** Exposure today: structural.
**Mitigation:** the panic sheet ("where backups are, who has access, how to
rebuild") written so a competent stranger could execute it; offline recovery
material reachable by a designated second person.

### R12 — Payment/financial record loss (Stripe divergence)

Stripe holds the authoritative customer/payment state; account loss or
membership-state divergence with Bubble breaks the business's money loop.
**L2 / I4.** Exposure today: medium.
**Mitigation:** scheduled Stripe exports (CSV or Data Pipeline) into backed-up
storage, reconciliation step in provider-outage playbook.

## Priority ranking (exposure × impact)

1. **R2** untested restores — invalidates everything else; fix via evidence loop
2. **R9** silent backup failure — feeds R2 directly
3. **R3** backup-deleting breach — credential separation is cheap and decisive
4. **R4/R5** identity & secrets recovery — T0, low effort to inventory, existential if wrong
5. **R10** control plane single-homed — mirror this repo immediately (near-zero cost)
6. **R1** full server loss — the flagship runbook, exercised by drills
7. **R7/R12** provider exports (Bubble, Stripe) — schedule and verify
8. **R6, R8, R11** — covered incrementally by playbooks and the panic sheet

## Consequences for the workplan

- First workplan must deliver: RTO/RPO matrix + ResiliencePolicy (targets),
  AssetRegister/DataRegister (inventory incl. T0 recovery paths), BackupPolicy
  with credential-separation rules, the full-server-loss runbook, a panic sheet,
  and **one real restore drill with an evidence file** — because until R2 is
  exercised once, everything else is theory.
- Mirroring this repo off the primary Gitea (R10) is a day-one ad hoc task,
  not a workplan phase.
- Backup *automation* (restic, gitea dump, etcd snapshots, Bubble/Stripe export
  jobs) runs where the infrastructure lives; this repo tracks its policy,
  schedule, and evidence — see INTENT.md boundaries.