tegwick
e7aa485c9f
Create private coulomb/disaster-control on forgejo.coulomb.social, add mirror sync docs and tools/sync-mirror.sh, update registers and README. Mark T01 done.
3.7 KiB
Resilience Policy
Status: adopted (initial)
Updated: 2026-07-07
Owner: Bernd
Scope: Coulomb Social / Helix Forge
Operating principle
Every capability that becomes valuable must declare its source of truth, recovery method, backup schedule, RTO, RPO, owner, and restore test evidence.
Servers are disposable. Data, identity, secrets, evidence, and payment records are not. Recovery means rebuild (Ansible/IaC + data restore + secret recovery + DNS cutover), not server resurrection.
After a security breach: never restore into the compromised machine. Build clean, restore known-good data, rotate credentials, preserve evidence.
Asset tiers
| Tier | Definition | Requirement |
|---|---|---|
| T0 | Identity / trust / money | Recoverable even after breach; recovery path documented |
| T1 | Irreplaceable source of truth | Strong backups, tested restore, offsite copy where feasible |
| T2 | Runtime platforms | Rebuild from IaC + restore data |
| T3 | Derived / rebuildable | Recompute where possible |
| T4 | External dependencies | Export, monitor, fallback, document limits |
Canonical inventory: AssetRegister.yaml, DataRegister.yaml.
Recovery objectives
Global baseline:
- RPO ≤ 24h everywhere (unless vendor-limited)
- RTO ≤ 1 day for full organizational recovery
- RTO ≤ 4–8h for code, content, and control-plane recovery
Per-system targets: RTO-RPO-Matrix.md.
Backup mechanisms: BackupPolicy.md.
Drill and evidence cadence
| Activity | Frequency | Output |
|---|---|---|
| Backup job success check (M-1) | Daily | Operator review / log check |
| Backup integrity check (M-3) | Weekly | age-decrypt sample per lane |
| T1 restore drill | Quarterly | TestEvidence/restore-drill-YYYY-MM-DD-<system>.md |
| T0 cold decrypt drill | Quarterly | Evidence file + register update |
| Full server rebuild drill | Quarterly | RestoreRunbooks/01-full-server-loss.md + evidence |
| Asset register review | Monthly | Update AssetRegister.yaml statuses |
| Critical secret rotation | Quarterly or after incident | Evidence note in State Hub |
| Cross-repo pickup queue review | Monthly | Update BackupPickupQueue.md |
A workplan is not finished until at least one restore drill evidence file
exists in TestEvidence/ for an untested T1 system.
Roles
| Role | Holder | Responsibilities |
|---|---|---|
| Policy owner | Bernd | Approve targets, run drills, update registers |
| Alerting owner | Bernd | Respond to backup failures (M-1–M-3) until on-call defined |
| Implementation | Owner repos | Backup automation, CNPG, cron, platform tools |
Control plane independence (R10)
disaster-control must survive the disasters it describes:
- At least one git mirror off primary Gitea —
forgejo.coulomb.social/coulomb/disaster-control(seedocs/mirror-sync.md). - Registers, policies, and evidence readable without production infrastructure.
- Panic sheet (
PanicSheet.md) executable by a competent stranger.
Incident response
Five playbooks under IncidentPlaybooks/:
- Ransomware / hostile access
- Credential leak
- Accidental delete / data corruption
- Provider outage
- Loss of admin identity
Each playbook: numbered steps, one page, links to runbooks and registers.
Related artifacts
| Artifact | Purpose |
|---|---|
BackupPolicy.md |
Backup architecture, encryption, monitoring |
RTO-RPO-Matrix.md |
Per-system RTO/RPO targets and measured drills |
BackupPickupQueue.md |
Cross-repo implementation tracker |
PanicSheet.md |
One-page emergency reference |
RestoreRunbooks/ |
Step-by-step recovery procedures |
TestEvidence/ |
Dated drill reports |