coulomb/disaster-control
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
disaster-control/IncidentPlaybooks/provider-outage.md
tegwick d011073e16 Start DISCTL-WP-0002: policies, registers, runbooks, playbooks 
Expand AssetRegister and DataRegister with T0/T4 stack assets.
Add ResiliencePolicy, PanicSheet, full-server-loss runbook, and five
incident playbooks. Update README with mirror status. Mark T02-T05,
T07-T08 done; T01 and T06 in progress.
2026-07-07 20:04:54 +02:00

1.1 KiB
Raw Blame History

Playbook: Provider Outage

Triggers: Bubble, Stripe, OpenRouter, IONOS DNS/email, or other vendor unavailable.

Steps

  1. Confirm outage — vendor status page, dig, curl, third-party monitor; rule out local DNS/SSH issue.
  2. Communicate — internal status note; disable affected features in UI if possible.
  3. Graceful degradation — queue writes; serve static fallback; read-only mode.
  4. Bubble.io — document member-facing impact; no local DB to restore; wait for vendor.
  5. Stripe — pause checkout if webhooks failing; queue events; reconcile on return.
  6. OpenRouter — switch to alternate LLM provider if configured; reduce non-critical inference.
  7. DNS/email — if IONOS down, existing TTLs may mask briefly; avoid panic changes until vendor confirms.
  8. Reconcile — after recovery, replay webhooks, verify Stripe/Bubble state, run smoke tests.
  9. Document — outage window, impact, follow-up (export cadence, alternate provider).

References

AssetRegister.yaml T4 entries · ResiliencePolicy.md · Bubble/Stripe export gaps in DataRegister.yaml