disaster-control/IncidentPlaybooks/provider-outage.md

19 lines
1.1 KiB
Markdown
Raw Normal View History

# Playbook: Provider Outage
**Triggers:** Bubble, Stripe, OpenRouter, IONOS DNS/email, or other vendor unavailable.
## Steps
1. **Confirm outage** — vendor status page, `dig`, `curl`, third-party monitor; rule out local DNS/SSH issue.
2. **Communicate** — internal status note; disable affected features in UI if possible.
3. **Graceful degradation** — queue writes; serve static fallback; read-only mode.
4. **Bubble.io** — document member-facing impact; no local DB to restore; wait for vendor.
5. **Stripe** — pause checkout if webhooks failing; queue events; reconcile on return.
6. **OpenRouter** — switch to alternate LLM provider if configured; reduce non-critical inference.
7. **DNS/email** — if IONOS down, existing TTLs may mask briefly; avoid panic changes until vendor confirms.
8. **Reconcile** — after recovery, replay webhooks, verify Stripe/Bubble state, run smoke tests.
9. **Document** — outage window, impact, follow-up (export cadence, alternate provider).
## References
`AssetRegister.yaml` T4 entries · `ResiliencePolicy.md` · Bubble/Stripe export gaps in `DataRegister.yaml`