disaster-control/history/2026-07-04-initial-risk-analysis.md

Initial Risk Analysis — Failure Modes for disaster-control

Date: 2026-07-04 Author: claude-code (session with Bernd) Status: baseline — to be revisited after AssetRegister and first restore drill Source: specs/ResilienceControlSetup.md, INTENT.md, current Coulomb/Helix stack (Gitea, k3s, CoulombCore, Bubble.io, Stripe, OpenRouter, DNS, secrets, Ansible) Companion: 2026-07-04-forgejo-backup-strategy-assessment.md — same-day Forgejo/Railiance deep-dive with a concrete pickup queue for this repo

Purpose

Enumerate the failure modes the Resilience Control Plane must address, rank them, and map each to the artifact (register, policy, runbook, playbook, drill) that mitigates it. This is the justification layer for the initial workplans.

Scoring

• Likelihood L1 (rare) – L5 (expected within a year)
• Impact I1 (nuisance) – I5 (existential: money, identity, or irreplaceable data lost)
• Exposure today = how well the current setup handles it before this repo does its job

Failure modes

R1 — Full production server loss (CoulombCore)

Disk failure, VPS deletion, provider account closure, fat-fingered teardown. L3 / I4. Exposure today: high — rebuild path exists conceptually (Ansible) but is not documented end-to-end and has never been drilled. Mitigation: restore runbook 01-full-server-loss, Ansible baseline as the recovery asset, offsite encrypted backups, DNS cutover procedure, quarterly rebuild drill.

R2 — Backup exists but restore fails (untested backups)

The classic silent failure: dumps run for months, first real restore reveals corruption, missing volumes, undumped databases, or an unknown passphrase. L4 / I5 (it converts every other risk into a total loss). Exposure today: high — exactly one restore has been evidenced: the 2026-07-04 Forgejo isolated restore drill (RAIL-HO-WP-0005-T09, evidence in railiance-infra). Gitea (the canonical source forge today), CoulombCore content, databases, Bubble exports, and the secret vault have never been restore-tested. Mitigation: the TestEvidence loop itself — recurring restore drills with dated reports measuring actual RTO/RPO. This is the highest-leverage artifact in the repo.

R3 — Ransomware / hostile access deletes backups too

Attacker with production admin credentials encrypts data and deletes reachable backups. L2 / I5. Exposure today: high if backup credentials live on the production host with delete permission. Mitigation: backup credential separation (BackupPolicy §credentials), immutable/append-only offsite copy, offline weekly copy, breach-recovery runbook (never restore into the compromised machine), secret rotation runbook.

R4 — Loss of admin identity (registrar, Stripe, Gitea, email, cloud)

Locked out via lost MFA device, expired recovery email, account takeover, or provider suspension. L2 / I5 — identity is Tier 0; without it nothing else can be recovered. Exposure today: unknown — no inventory of recovery codes, backup admins, or break-glass paths exists. Mitigation: AssetRegister T0 section with recovery paths per account, offline recovery codes, break-glass process, loss-of-admin-identity playbook.

R5 — Secrets lost or leaked

Two directions: (a) recovery material lost → backups undecryptable, services unrebuildable; (b) secret leaked → every dependent system compromised. L3 / I5. Exposure today: medium-high — OpenBao/warden infrastructure exists but sealed recovery material for disaster scenarios is not evidenced. Mitigation: encrypted (SOPS/age) sealed recovery vault backed up offline, secret-rotation runbook, credential-leak playbook. Restore drills must include "can we actually decrypt the vault from cold".

R6 — Accidental deletion / bad migration / data corruption

Deleted repo, destructive script, corrupted DB, bad Ansible run. L4 / I3 (bounded if backups have granularity and short RPO). Exposure today: medium. Mitigation: accidental-delete and data-corruption playbooks (stop writes → staging restore → identify last good point → selective restore → diff), local snapshots for fast rollback, RPO targets per system in the RTO/RPO matrix.

R7 — External provider outage or exit (Bubble, Stripe, OpenRouter, DNS, email)

Outage is L4/I2 (degradation); forced exit (provider shutdown, account ban, pricing change) is L2/I4 — especially Bubble, which holds member data on a platform we don't control. Exposure today: high for Bubble (export cadence not established), low-medium for others. Mitigation: T4 section of AssetRegister documenting export paths and limits, scheduled Bubble and Stripe exports, provider-outage playbook with graceful degradation, alternate-LLM path for OpenRouter.

R8 — DNS / domain hijack or lapse

Expired domain, registrar compromise, hostile transfer. L1 / I5. Exposure today: unknown (lock status, MFA, backup contacts uninventoried). Mitigation: registrar lock + MFA + backup admin recorded in AssetRegister, zone-record exports after changes, 2h RTO cutover procedure.

R9 — Silent backup failure (job stops, nobody notices)

Cron dies, disk fills, token expires; discovered only at restore time. L4 / I4 (degenerates into R2). Exposure today: high — no backup success monitoring is defined. Mitigation: monitoring & evidence checks (daily job-success check, weekly integrity check) defined in BackupPolicy with an explicit alerting owner.

R10 — The control plane itself is lost or stale

Two sub-modes: (a) this repo lives on the same Gitea it protects — a full-loss event takes the recovery instructions down with it; (b) docs drift from reality and runbooks silently rot. L3 / I4. Exposure today: high — repo is single-homed as of this writing. Mitigation: mirror disaster-control to at least one independent location (and one offline/printable panic sheet), monthly asset-register review, drills double as doc-verification.

R11 — Single-operator dependency (bus factor = 1)

Bernd is the only person who knows where backups are, how to rotate secrets, and holds the MFA devices. L2 / I5. Exposure today: structural. Mitigation: the panic sheet ("where backups are, who has access, how to rebuild") written so a competent stranger could execute it; offline recovery material reachable by a designated second person.

R12 — Payment/financial record loss (Stripe divergence)

Stripe holds the authoritative customer/payment state; account loss or membership-state divergence with Bubble breaks the business's money loop. L2 / I4. Exposure today: medium. Mitigation: scheduled Stripe exports (CSV or Data Pipeline) into backed-up storage, reconciliation step in provider-outage playbook.

Priority ranking (exposure × impact)

1. R2 untested restores — invalidates everything else; fix via evidence loop
2. R9 silent backup failure — feeds R2 directly
3. R3 backup-deleting breach — credential separation is cheap and decisive
4. R4/R5 identity & secrets recovery — T0, low effort to inventory, existential if wrong
5. R10 control plane single-homed — mirror this repo immediately (near-zero cost)
6. R1 full server loss — the flagship runbook, exercised by drills
7. R7/R12 provider exports (Bubble, Stripe) — schedule and verify
8. R6, R8, R11 — covered incrementally by playbooks and the panic sheet

Consequences for the workplan

• First workplan must deliver: RTO/RPO matrix + ResiliencePolicy (targets), AssetRegister/DataRegister (inventory incl. T0 recovery paths), BackupPolicy with credential-separation rules, the full-server-loss runbook, a panic sheet, and one real restore drill with an evidence file — because until R2 is exercised once, everything else is theory.
• Mirroring this repo off the primary Gitea (R10) is a day-one ad hoc task, not a workplan phase.
• Backup automation (restic, gitea dump, etcd snapshots, Bubble/Stripe export jobs) runs where the infrastructure lives; this repo tracks its policy, schedule, and evidence — see INTENT.md boundaries.