coulomb/disaster-control
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
disaster-control/IncidentPlaybooks/accidental-delete.md
tegwick d011073e16 Start DISCTL-WP-0002: policies, registers, runbooks, playbooks 
Expand AssetRegister and DataRegister with T0/T4 stack assets.
Add ResiliencePolicy, PanicSheet, full-server-loss runbook, and five
incident playbooks. Update README with mirror status. Mark T02-T05,
T07-T08 done; T01 and T06 in progress.
2026-07-07 20:04:54 +02:00

1.1 KiB
Raw Blame History

Playbook: Accidental Delete / Data Corruption

Triggers: deleted repo, dropped table, bad migration, destructive script, bad Ansible run.

Steps

  1. Stop writes — scale deployment to zero, enable maintenance mode, or revoke DB write access.
  2. Assess scope — what was deleted/corrupted, when, last known good state.
  3. Clone backup to staging — never restore directly over production first.
  4. Identify last good point — backup timestamp, git ref, DB snapshot.
  5. Restore selectively — one repo, one table, or one PVC — not full blast restore unless required.
  6. Diff — compare staging restore against current production; list deltas.
  7. Apply to production — merge recovered data; re-run migrations if needed.
  8. Resume writes — remove maintenance mode; monitor.
  9. Document — TestEvidence/ note with RPO achieved; update registers if backup gap contributed.

Fast rollback

If local snapshot or recent git push exists, prefer that over full restore.

References

RTO-RPO-Matrix.md · BackupPolicy.md · AssetRegister.yaml