RAILIANCE-WP-0015: CNPG backup ScheduledBackup coverage followup
Descoped out of RAIL-HO-WP-0005 T04/T09. Drives cnpg-backup-status degraded→healthy. Blocked on CCR-2026-0004 offsite credential provisioning (operator/OpenBao). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
a307041cac
commit
deef13e8a5
1 changed files with 116 additions and 0 deletions
|
|
@ -0,0 +1,116 @@
|
|||
---
|
||||
id: RAILIANCE-WP-0015
|
||||
type: workplan
|
||||
title: "CNPG backup ScheduledBackup coverage — drive cnpg-backup-status to healthy"
|
||||
domain: financials
|
||||
repo: railiance-apps
|
||||
status: ready
|
||||
owner: codex
|
||||
topic_slug: railiance
|
||||
created: "2026-07-14"
|
||||
updated: "2026-07-14"
|
||||
---
|
||||
|
||||
# CNPG backup ScheduledBackup coverage
|
||||
|
||||
**Parent / origin:** descoped out of `RAIL-HO-WP-0005` (Forgejo Production
|
||||
Migration, finished 2026-07-14) T04/T09. The migration is complete; this is the
|
||||
residual **DB-backup operational hardening**, not a migration blocker.
|
||||
|
||||
## Goal
|
||||
|
||||
Drive `make cnpg-backup-status` (railiance-apps) from **degraded → healthy**:
|
||||
scheduled, unattended, encrypted, off-cluster backups with a passing restore
|
||||
drill for all four production CNPG clusters — `apps-pg`, `gitea-db`,
|
||||
`net-kingdom-pg`, `state-hub-db`.
|
||||
|
||||
## Current state (2026-07-14)
|
||||
|
||||
- `cnpg-backup-status`: **degraded — 4 clusters lack ScheduledBackup coverage**
|
||||
(all show `backup spec = null`, `ScheduledBackup = none`; clusters otherwise
|
||||
healthy, 1 instance each).
|
||||
- Decided approach (RAIL-HO-WP-0005 Key Decision 5, Option A, 2026-07-09):
|
||||
daily **age-encrypted `pg_dump`/`forgejo dump` → Nextcloud WebDAV**, 14 daily +
|
||||
4 weekly retention, RPO 24h / RTO 4h. A one-off encrypted `forgejo-db` dump was
|
||||
produced + restore-drilled (RAILIANCE-WP-0013, finished 2026-07-11).
|
||||
- Tooling already present: `tools/cnpg-backup-status.sh`, `manifests/cnpg-backup-readiness.yaml`
|
||||
(commented template), `make apps-pg-backup-dry-run`, `platform forgejo-backup`,
|
||||
`docs/app-data-backup-restore-handoff.md`.
|
||||
|
||||
## Blocking gate (human / operator)
|
||||
|
||||
The offsite credential lane **`CCR-2026-0004`** (railiance-backup-offsite-lane)
|
||||
is `status: active`, `decision: approved`, but **`resolvable: false`**
|
||||
(`readiness: applied-pending-verify`). No backup Secret exists in the `databases`
|
||||
namespace. Provisioning `NC_WEBDAV_TOKEN` / `NC_WEBDAV_URL` / `AGE_PRIVATE_KEY`
|
||||
into OpenBao (making `warden access railiance-backup-offsite-lane --fetch …`
|
||||
resolve) is an **operator/OpenBao action** — out of scope for agent credential
|
||||
handling per credential-routing. Everything downstream is gated on it.
|
||||
|
||||
## Task: Provision CCR-2026-0004 offsite credentials (operator)
|
||||
|
||||
```task
|
||||
id: RAILIANCE-WP-0015-T01
|
||||
status: wait
|
||||
priority: high
|
||||
needs_human: true
|
||||
```
|
||||
|
||||
Operator provisions `NC_WEBDAV_TOKEN`, `NC_WEBDAV_URL`, `AGE_PRIVATE_KEY` per
|
||||
`CCR-2026-0004` so the lane becomes resolvable. **Done when:**
|
||||
`warden access railiance-backup-offsite-lane --fetch NC_WEBDAV_TOKEN` succeeds and
|
||||
the CCR flips to `resolvable: true`.
|
||||
|
||||
## Task: Create offsite backup Secret in databases namespace
|
||||
|
||||
```task
|
||||
id: RAILIANCE-WP-0015-T02
|
||||
status: wait
|
||||
priority: high
|
||||
```
|
||||
|
||||
Once T01 resolves, materialize the `databases` namespace Secret from the lane
|
||||
(no plaintext in Git — sourced via OpenBao/operator path). **Done when:** the
|
||||
Secret exists and `pg_dump`/WebDAV upload authenticates.
|
||||
|
||||
## Task: Wire scheduled backup for all four CNPG clusters
|
||||
|
||||
```task
|
||||
id: RAILIANCE-WP-0015-T03
|
||||
status: wait
|
||||
priority: high
|
||||
```
|
||||
|
||||
Implement the decided Option A lane as an unattended schedule for `apps-pg`,
|
||||
`gitea-db`, `net-kingdom-pg`, `state-hub-db`: age-encrypted logical dump →
|
||||
Nextcloud WebDAV, 14 daily + 4 weekly retention. Uncomment/parameterize
|
||||
`manifests/cnpg-backup-readiness.yaml` (or CronJob equivalent). **Done when:**
|
||||
each cluster has a running scheduled backup producing encrypted off-cluster
|
||||
artifacts without manual intervention.
|
||||
|
||||
## Task: Reconcile cnpg-backup-status health definition with Option A
|
||||
|
||||
```task
|
||||
id: RAILIANCE-WP-0015-T04
|
||||
status: wait
|
||||
priority: medium
|
||||
```
|
||||
|
||||
`cnpg-backup-status.sh` currently reports `degraded` whenever a CNPG-native
|
||||
`ScheduledBackup` resource is absent, but the decided approach is logical-dump
|
||||
CronJobs, not barman `ScheduledBackup`. Update the health check so a passing
|
||||
Option-A schedule reports **healthy** (or adopt barman ObjectStore if chosen
|
||||
instead). **Done when:** `make cnpg-backup-status` reflects the real, decided
|
||||
posture.
|
||||
|
||||
## Task: Restore drill + RPO/RTO evidence, mark healthy
|
||||
|
||||
```task
|
||||
id: RAILIANCE-WP-0015-T05
|
||||
status: wait
|
||||
priority: high
|
||||
```
|
||||
|
||||
Run an isolated-namespace restore drill from a scheduled artifact for at least
|
||||
one cluster; record RPO 24h / RTO 4h evidence. **Done when:** `make
|
||||
cnpg-backup-status` → **healthy** and restore evidence is recorded.
|
||||
Loading…
Add table
Add a link
Reference in a new issue