diff --git a/workplans/RAILIANCE-WP-0015-cnpg-backup-scheduledbackup-coverage.md b/workplans/RAILIANCE-WP-0015-cnpg-backup-scheduledbackup-coverage.md new file mode 100644 index 0000000..23a1bdf --- /dev/null +++ b/workplans/RAILIANCE-WP-0015-cnpg-backup-scheduledbackup-coverage.md @@ -0,0 +1,116 @@ +--- +id: RAILIANCE-WP-0015 +type: workplan +title: "CNPG backup ScheduledBackup coverage — drive cnpg-backup-status to healthy" +domain: financials +repo: railiance-apps +status: ready +owner: codex +topic_slug: railiance +created: "2026-07-14" +updated: "2026-07-14" +--- + +# CNPG backup ScheduledBackup coverage + +**Parent / origin:** descoped out of `RAIL-HO-WP-0005` (Forgejo Production +Migration, finished 2026-07-14) T04/T09. The migration is complete; this is the +residual **DB-backup operational hardening**, not a migration blocker. + +## Goal + +Drive `make cnpg-backup-status` (railiance-apps) from **degraded → healthy**: +scheduled, unattended, encrypted, off-cluster backups with a passing restore +drill for all four production CNPG clusters — `apps-pg`, `gitea-db`, +`net-kingdom-pg`, `state-hub-db`. + +## Current state (2026-07-14) + +- `cnpg-backup-status`: **degraded — 4 clusters lack ScheduledBackup coverage** + (all show `backup spec = null`, `ScheduledBackup = none`; clusters otherwise + healthy, 1 instance each). +- Decided approach (RAIL-HO-WP-0005 Key Decision 5, Option A, 2026-07-09): + daily **age-encrypted `pg_dump`/`forgejo dump` → Nextcloud WebDAV**, 14 daily + + 4 weekly retention, RPO 24h / RTO 4h. A one-off encrypted `forgejo-db` dump was + produced + restore-drilled (RAILIANCE-WP-0013, finished 2026-07-11). +- Tooling already present: `tools/cnpg-backup-status.sh`, `manifests/cnpg-backup-readiness.yaml` + (commented template), `make apps-pg-backup-dry-run`, `platform forgejo-backup`, + `docs/app-data-backup-restore-handoff.md`. + +## Blocking gate (human / operator) + +The offsite credential lane **`CCR-2026-0004`** (railiance-backup-offsite-lane) +is `status: active`, `decision: approved`, but **`resolvable: false`** +(`readiness: applied-pending-verify`). No backup Secret exists in the `databases` +namespace. Provisioning `NC_WEBDAV_TOKEN` / `NC_WEBDAV_URL` / `AGE_PRIVATE_KEY` +into OpenBao (making `warden access railiance-backup-offsite-lane --fetch …` +resolve) is an **operator/OpenBao action** — out of scope for agent credential +handling per credential-routing. Everything downstream is gated on it. + +## Task: Provision CCR-2026-0004 offsite credentials (operator) + +```task +id: RAILIANCE-WP-0015-T01 +status: wait +priority: high +needs_human: true +``` + +Operator provisions `NC_WEBDAV_TOKEN`, `NC_WEBDAV_URL`, `AGE_PRIVATE_KEY` per +`CCR-2026-0004` so the lane becomes resolvable. **Done when:** +`warden access railiance-backup-offsite-lane --fetch NC_WEBDAV_TOKEN` succeeds and +the CCR flips to `resolvable: true`. + +## Task: Create offsite backup Secret in databases namespace + +```task +id: RAILIANCE-WP-0015-T02 +status: wait +priority: high +``` + +Once T01 resolves, materialize the `databases` namespace Secret from the lane +(no plaintext in Git — sourced via OpenBao/operator path). **Done when:** the +Secret exists and `pg_dump`/WebDAV upload authenticates. + +## Task: Wire scheduled backup for all four CNPG clusters + +```task +id: RAILIANCE-WP-0015-T03 +status: wait +priority: high +``` + +Implement the decided Option A lane as an unattended schedule for `apps-pg`, +`gitea-db`, `net-kingdom-pg`, `state-hub-db`: age-encrypted logical dump → +Nextcloud WebDAV, 14 daily + 4 weekly retention. Uncomment/parameterize +`manifests/cnpg-backup-readiness.yaml` (or CronJob equivalent). **Done when:** +each cluster has a running scheduled backup producing encrypted off-cluster +artifacts without manual intervention. + +## Task: Reconcile cnpg-backup-status health definition with Option A + +```task +id: RAILIANCE-WP-0015-T04 +status: wait +priority: medium +``` + +`cnpg-backup-status.sh` currently reports `degraded` whenever a CNPG-native +`ScheduledBackup` resource is absent, but the decided approach is logical-dump +CronJobs, not barman `ScheduledBackup`. Update the health check so a passing +Option-A schedule reports **healthy** (or adopt barman ObjectStore if chosen +instead). **Done when:** `make cnpg-backup-status` reflects the real, decided +posture. + +## Task: Restore drill + RPO/RTO evidence, mark healthy + +```task +id: RAILIANCE-WP-0015-T05 +status: wait +priority: high +``` + +Run an isolated-namespace restore drill from a scheduled artifact for at least +one cluster; record RPO 24h / RTO 4h evidence. **Done when:** `make +cnpg-backup-status` → **healthy** and restore evidence is recorded.