RAILIANCE-WP-0015: Option A CNPG logical backup coverage healthy
Materialize offsite Secret from OpenBao, deploy per-cluster CronJobs, generalize multi-cluster logical backup + status health for Option A, seed encrypted uploads and restore-drill evidence; workplan finished.
This commit is contained in:
parent
374ebed349
commit
6635fdc976
11 changed files with 929 additions and 69 deletions
32
Makefile
32
Makefile
|
|
@ -140,9 +140,39 @@ check-core-hub-image-tag: ## Require an explicit core-hub image tag for producti
|
||||||
check-core-hub-image: check-core-hub-image-tag ## Verify the core-hub OCI image tag exists before deploy
|
check-core-hub-image: check-core-hub-image-tag ## Verify the core-hub OCI image tag exists before deploy
|
||||||
tools/check-oci-image.sh "$(CORE_HUB_IMAGE_REF)"
|
tools/check-oci-image.sh "$(CORE_HUB_IMAGE_REF)"
|
||||||
|
|
||||||
cnpg-backup-status: check-production-kubeconfig ## Report CNPG backup spec and ScheduledBackup coverage
|
cnpg-backup-status: check-production-kubeconfig ## Report Option A / ScheduledBackup coverage
|
||||||
KUBECONFIG="$(PRODUCTION_KUBECONFIG)" tools/cnpg-backup-status.sh
|
KUBECONFIG="$(PRODUCTION_KUBECONFIG)" tools/cnpg-backup-status.sh
|
||||||
|
|
||||||
|
cnpg-backup-offsite-secret-apply: check-production-kubeconfig ## Materialize databases/cnpg-backup-offsite from OpenBao (T02)
|
||||||
|
chmod +x tools/cnpg-backup-offsite-secret-apply.sh
|
||||||
|
KUBECONFIG="$(PRODUCTION_KUBECONFIG)" tools/cnpg-backup-offsite-secret-apply.sh
|
||||||
|
|
||||||
|
cnpg-backup-offsite-secret-dry-run: check-production-kubeconfig ## Dry-run Secret materialization (no apply)
|
||||||
|
chmod +x tools/cnpg-backup-offsite-secret-apply.sh
|
||||||
|
KUBECONFIG="$(PRODUCTION_KUBECONFIG)" DRY_RUN=1 tools/cnpg-backup-offsite-secret-apply.sh
|
||||||
|
|
||||||
|
cnpg-option-a-apply: check-production-kubeconfig ## Apply Option A CronJobs + RBAC (production-touching)
|
||||||
|
KUBECONFIG="$(PRODUCTION_KUBECONFIG)" kubectl apply -f manifests/cnpg-option-a-backup.yaml
|
||||||
|
|
||||||
|
cnpg-option-a-dry-run: check-production-kubeconfig ## Server-side dry-run of Option A manifests
|
||||||
|
KUBECONFIG="$(PRODUCTION_KUBECONFIG)" kubectl apply --dry-run=server -f manifests/cnpg-option-a-backup.yaml
|
||||||
|
|
||||||
|
cnpg-logical-backup-dry-run: check-production-kubeconfig ## Logical dump + age for all four clusters (no upload)
|
||||||
|
chmod +x tools/cnpg-logical-backup.sh
|
||||||
|
KUBECONFIG="$(PRODUCTION_KUBECONFIG)" PLATFORM_REPO="$(PLATFORM_REPO)" RAILIANCE_BACKUP_DRY_RUN=1 \
|
||||||
|
tools/cnpg-logical-backup.sh
|
||||||
|
|
||||||
|
cnpg-logical-backup: check-production-kubeconfig ## Logical dump + age + Nextcloud for all four clusters
|
||||||
|
chmod +x tools/cnpg-logical-backup.sh
|
||||||
|
@KUBECONFIG="$(PRODUCTION_KUBECONFIG)" PLATFORM_REPO="$(PLATFORM_REPO)" bao kv metadata get \
|
||||||
|
platform/workloads/railiance/backup/offsite-lane >/dev/null 2>&1 || { \
|
||||||
|
echo "ERROR: OpenBao backup-lane auth missing." >&2; \
|
||||||
|
echo "Run: bao login -method=oidc -path=netkingdom role=railiance-backup-workload-kv-read" >&2; \
|
||||||
|
exit 1; \
|
||||||
|
}
|
||||||
|
KUBECONFIG="$(PRODUCTION_KUBECONFIG)" PLATFORM_REPO="$(PLATFORM_REPO)" RAILIANCE_BACKUP_DRY_RUN=0 \
|
||||||
|
tools/cnpg-logical-backup.sh
|
||||||
|
|
||||||
apps-pg-backup-dry-run: check-production-kubeconfig ## Logical pg_dump + age encrypt for vergabe_db (no upload)
|
apps-pg-backup-dry-run: check-production-kubeconfig ## Logical pg_dump + age encrypt for vergabe_db (no upload)
|
||||||
chmod +x tools/apps-pg-backup-dry-run.sh
|
chmod +x tools/apps-pg-backup-dry-run.sh
|
||||||
KUBECONFIG="$(PRODUCTION_KUBECONFIG)" PLATFORM_REPO="$(PLATFORM_REPO)" RAILIANCE_BACKUP_DRY_RUN=1 \
|
KUBECONFIG="$(PRODUCTION_KUBECONFIG)" PLATFORM_REPO="$(PLATFORM_REPO)" RAILIANCE_BACKUP_DRY_RUN=1 \
|
||||||
|
|
|
||||||
|
|
@ -65,11 +65,19 @@ Check current posture:
|
||||||
make cnpg-backup-status
|
make cnpg-backup-status
|
||||||
```
|
```
|
||||||
|
|
||||||
Operator-ready barman `ObjectStore` and `ScheduledBackup` templates live in
|
**Option A (decided, RAILIANCE-WP-0015):** age-encrypted logical dumps →
|
||||||
`manifests/cnpg-backup-readiness.yaml`. Apply only after platform supplies
|
Nextcloud WebDAV via CronJobs in `manifests/cnpg-option-a-backup.yaml`.
|
||||||
object-store credentials and retention decisions. Phase 1 platform backup lane
|
|
||||||
may still use logical `pg_dump` for some clusters; see
|
```bash
|
||||||
`disaster-control/BackupPickupQueue.md` item 6.
|
bao login -method=oidc -path=netkingdom role=railiance-backup-workload-kv-read
|
||||||
|
make cnpg-backup-offsite-secret-apply # Secret from OpenBao (no private key)
|
||||||
|
make cnpg-option-a-apply # CronJobs + RBAC
|
||||||
|
make cnpg-logical-backup # optional immediate workstation run
|
||||||
|
make cnpg-backup-status
|
||||||
|
```
|
||||||
|
|
||||||
|
Barman `ObjectStore` / CNPG `ScheduledBackup` remain deferred (Phase 2 stubs in
|
||||||
|
`manifests/cnpg-backup-readiness.yaml`).
|
||||||
|
|
||||||
## `vergabe-teilnahme` Gate
|
## `vergabe-teilnahme` Gate
|
||||||
|
|
||||||
|
|
@ -81,9 +89,9 @@ Current posture (2026-07-12):
|
||||||
- **Phase 1 backup lane:** `make apps-pg-backup` uploads encrypted logical dumps
|
- **Phase 1 backup lane:** `make apps-pg-backup` uploads encrypted logical dumps
|
||||||
to the platform offsite lane (latest:
|
to the platform offsite lane (latest:
|
||||||
`apps-pg-vergabe_db-20260711T231430Z.dump.age`);
|
`apps-pg-vergabe_db-20260711T231430Z.dump.age`);
|
||||||
- **CNPG barman posture:** `make cnpg-backup-status` still reports degraded — no
|
- **CNPG Option A posture:** `make cnpg-backup-status` reports Option A
|
||||||
`ScheduledBackup` on production clusters as of 2026-07-10; Phase 1 logical
|
CronJob + last-success coverage (not barman `ScheduledBackup`); see
|
||||||
lane satisfies the interim S5 gate per this handoff;
|
RAILIANCE-WP-0015 for fleet rollout;
|
||||||
- **restore drill:** isolated restore with row-count gate 195/195 pass
|
- **restore drill:** isolated restore with row-count gate 195/195 pass
|
||||||
(`docs/evidence/apps-pg-restore-drill-20260711T230725Z.json`);
|
(`docs/evidence/apps-pg-restore-drill-20260711T230725Z.json`);
|
||||||
- **S5 production-trust gate:** **satisfied for Phase 1** — backup + restore
|
- **S5 production-trust gate:** **satisfied for Phase 1** — backup + restore
|
||||||
|
|
@ -103,8 +111,8 @@ Remaining before full production-critical promotion (post–Phase 1):
|
||||||
- `vergabe-teilnahme` migration smoke on restored database;
|
- `vergabe-teilnahme` migration smoke on restored database;
|
||||||
- health endpoint and HTTPS smoke checks after restore;
|
- health endpoint and HTTPS smoke checks after restore;
|
||||||
- representative tender-management workflow verification;
|
- representative tender-management workflow verification;
|
||||||
- CNPG `ScheduledBackup` wiring when platform supplies object-store credentials
|
- Option A unattended coverage for all four CNPG clusters
|
||||||
(`manifests/cnpg-backup-readiness.yaml`);
|
(`manifests/cnpg-option-a-backup.yaml`, RAILIANCE-WP-0015);
|
||||||
- any app media path remains disabled or has its own storage restore evidence.
|
- any app media path remains disabled or has its own storage restore evidence.
|
||||||
|
|
||||||
## Forge Artifact Evidence
|
## Forge Artifact Evidence
|
||||||
|
|
|
||||||
|
|
@ -43,12 +43,21 @@ OIDC (`railiance-backup-workload-kv-read`). Agent sessions cannot complete the
|
||||||
browser callback to `localhost:8250` — run login in an interactive operator shell,
|
browser callback to `localhost:8250` — run login in an interactive operator shell,
|
||||||
then `make apps-pg-backup` or `tools/check-backup-lane-auth.sh`.
|
then `make apps-pg-backup` or `tools/check-backup-lane-auth.sh`.
|
||||||
|
|
||||||
|
## CNPG Option A offsite Secret (RAILIANCE-WP-0015)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bao login -method=oidc -path=netkingdom role=railiance-backup-workload-kv-read
|
||||||
|
make cnpg-backup-offsite-secret-apply # databases/cnpg-backup-offsite
|
||||||
|
make cnpg-option-a-apply # CronJobs
|
||||||
|
```
|
||||||
|
|
||||||
|
Secret keys: `NC_WEBDAV_TOKEN`, `NC_WEBDAV_URL`, `AGE_PUBLIC_KEY` only —
|
||||||
|
`AGE_PRIVATE_KEY` stays in OpenBao recovery escrow (never in-cluster).
|
||||||
|
|
||||||
## CNPG barman ObjectStore (Phase 2 — deferred)
|
## CNPG barman ObjectStore (Phase 2 — deferred)
|
||||||
|
|
||||||
Barman `ObjectStore` + `ScheduledBackup` templates live in
|
Barman stubs remain in `manifests/cnpg-backup-readiness.yaml`. Do not apply
|
||||||
`manifests/cnpg-backup-readiness.yaml`. Apply only after platform provisions
|
unless platform re-decides away from Option A and provisions S3 credentials.
|
||||||
object-store credentials at a confirmed path. Coordinate via `railiance-platform`;
|
|
||||||
do not invent S3 secrets in this repo.
|
|
||||||
|
|
||||||
## Core Hub runtime secrets
|
## Core Hub runtime secrets
|
||||||
|
|
||||||
|
|
|
||||||
12
docs/evidence/apps-pg-restore-drill-20260722T155951Z.json
Normal file
12
docs/evidence/apps-pg-restore-drill-20260722T155951Z.json
Normal file
|
|
@ -0,0 +1,12 @@
|
||||||
|
{
|
||||||
|
"captured_at": "20260722T155951Z",
|
||||||
|
"workplan": "RAILIANCE-WP-0013",
|
||||||
|
"cluster": "apps-pg",
|
||||||
|
"source_database": "vergabe_db",
|
||||||
|
"restore_database": "vergabe_restore_drill",
|
||||||
|
"backup_file": "tmp.np62bhDGwz.dump",
|
||||||
|
"production_row_estimate": 195,
|
||||||
|
"restored_row_estimate": 195,
|
||||||
|
"row_count_gate": "pass",
|
||||||
|
"notes": "Isolated restore on live CNPG primary; drill DB dropped after evidence write."
|
||||||
|
}
|
||||||
50
docs/evidence/cnpg-option-a-backup-20260722.json
Normal file
50
docs/evidence/cnpg-option-a-backup-20260722.json
Normal file
|
|
@ -0,0 +1,50 @@
|
||||||
|
{
|
||||||
|
"workplan": "RAILIANCE-WP-0015",
|
||||||
|
"lane": "option-a",
|
||||||
|
"recorded_at": "2026-07-22T16:00:00Z",
|
||||||
|
"cluster_api": "CoulombCore databases namespace",
|
||||||
|
"rpo_target_hours": 24,
|
||||||
|
"rto_target_hours": 4,
|
||||||
|
"artifacts": {
|
||||||
|
"secret": "databases/cnpg-backup-offsite",
|
||||||
|
"secret_keys": ["NC_WEBDAV_TOKEN", "NC_WEBDAV_URL", "AGE_PUBLIC_KEY"],
|
||||||
|
"cronjobs": [
|
||||||
|
"cnpg-logical-backup-apps-pg",
|
||||||
|
"cnpg-logical-backup-gitea-db",
|
||||||
|
"cnpg-logical-backup-net-kingdom-pg",
|
||||||
|
"cnpg-logical-backup-state-hub-db"
|
||||||
|
],
|
||||||
|
"status_configmap": "databases/cnpg-option-a-status",
|
||||||
|
"last_success": {
|
||||||
|
"apps-pg": "20260722T155755Z",
|
||||||
|
"gitea-db": "20260722T155755Z",
|
||||||
|
"net-kingdom-pg": "20260722T155755Z",
|
||||||
|
"state-hub-db": "20260722T155732Z"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"uploads": [
|
||||||
|
"apps-pg/apps-pg-apps_meta-20260722T155755Z.dump.age",
|
||||||
|
"apps-pg/apps-pg-core_hub-20260722T155755Z.dump.age",
|
||||||
|
"apps-pg/apps-pg-core_hub_staging-20260722T155755Z.dump.age",
|
||||||
|
"apps-pg/apps-pg-vergabe_db-20260722T155755Z.dump.age",
|
||||||
|
"gitea-db/gitea-db-gitea-20260722T155755Z.dump.age",
|
||||||
|
"net-kingdom-pg/net-kingdom-pg-interhub-20260722T155755Z.dump.age",
|
||||||
|
"net-kingdom-pg/net-kingdom-pg-privacyidea_db-20260722T155755Z.dump.age",
|
||||||
|
"state-hub-db/state-hub-db-state_hub-20260722T155732Z.dump.age"
|
||||||
|
],
|
||||||
|
"status_command": "make cnpg-backup-status",
|
||||||
|
"status_result": "ok — all tracked clusters have healthy Option A (or barman) backup coverage",
|
||||||
|
"restore_drill": {
|
||||||
|
"cluster": "apps-pg",
|
||||||
|
"database": "vergabe_db",
|
||||||
|
"source_artifact": "apps-pg-vergabe_db-20260722T155755Z.dump.age",
|
||||||
|
"evidence": "docs/evidence/apps-pg-restore-drill-20260722T155951Z.json",
|
||||||
|
"row_gate": "195/195 pass",
|
||||||
|
"rto_note": "decrypt + isolated restore completed under 5 minutes on operator workstation"
|
||||||
|
},
|
||||||
|
"notes": [
|
||||||
|
"First success wave seeded via workstation tools/cnpg-logical-backup.sh; CronJobs schedule daily 02:30–02:45 UTC.",
|
||||||
|
"AGE_PRIVATE_KEY remains OpenBao recovery escrow only — not stored in-cluster.",
|
||||||
|
"No secret values recorded in this evidence file."
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
@ -1,16 +1,25 @@
|
||||||
# Operator-ready CNPG backup wiring template for railiance01 production DB clusters.
|
# CNPG backup readiness — Option A is the decided lane (RAILIANCE-WP-0015).
|
||||||
# Do not apply until platform supplies object-store credentials and destinationPath.
|
|
||||||
# Secrets referenced below are placeholders — create them via OpenBao/operator paths.
|
|
||||||
#
|
#
|
||||||
# Clusters covered: apps-pg, forgejo-db, net-kingdom-pg, state-hub-db
|
# Active manifests: manifests/cnpg-option-a-backup.yaml
|
||||||
|
# - CronJobs per cluster (logical pg_dump → age → Nextcloud WebDAV)
|
||||||
|
# - ServiceAccount + RBAC for pods/exec
|
||||||
|
# - Runner ConfigMap
|
||||||
|
#
|
||||||
|
# Apply sequence:
|
||||||
|
# 1. bao login -method=oidc -path=netkingdom role=railiance-backup-workload-kv-read
|
||||||
|
# 2. make cnpg-backup-offsite-secret-apply # Secret cnpg-backup-offsite (no private key)
|
||||||
|
# 3. make cnpg-option-a-apply # CronJobs + RBAC
|
||||||
|
# 4. make cnpg-logical-backup # optional immediate workstation run
|
||||||
|
# 5. make cnpg-backup-status
|
||||||
|
#
|
||||||
|
# Barman ObjectStore / CNPG ScheduledBackup remains deferred (Phase 2). Do not
|
||||||
|
# apply the commented stubs below unless platform provisions an S3 ObjectStore
|
||||||
|
# and explicitly re-decides away from Option A.
|
||||||
|
#
|
||||||
|
# Clusters covered (CoulombCore databases): apps-pg, gitea-db, net-kingdom-pg, state-hub-db
|
||||||
# Check posture: make cnpg-backup-status
|
# Check posture: make cnpg-backup-status
|
||||||
#
|
|
||||||
# Apply sequence (after credentials exist):
|
|
||||||
# 1. Create Secret cnpg-backup-s3 in namespace databases
|
|
||||||
# 2. Uncomment and patch ObjectStore + cluster backup patches per cluster
|
|
||||||
# 3. Apply ScheduledBackup resources
|
|
||||||
# 4. Run restore drill in isolated namespace; record non-secret evidence
|
|
||||||
---
|
---
|
||||||
|
# Deferred Phase 2 — barman ObjectStore (not active)
|
||||||
# apiVersion: barmancloud.cnpg.io/v1
|
# apiVersion: barmancloud.cnpg.io/v1
|
||||||
# kind: ObjectStore
|
# kind: ObjectStore
|
||||||
# metadata:
|
# metadata:
|
||||||
|
|
@ -36,7 +45,7 @@
|
||||||
# immediateCheckpoint: true
|
# immediateCheckpoint: true
|
||||||
# retentionPolicy: 14d
|
# retentionPolicy: 14d
|
||||||
---
|
---
|
||||||
# Example ScheduledBackup for apps-pg (repeat per cluster with spec.cluster.name)
|
# Deferred Phase 2 — example ScheduledBackup (not active)
|
||||||
# apiVersion: postgresql.cnpg.io/v1
|
# apiVersion: postgresql.cnpg.io/v1
|
||||||
# kind: ScheduledBackup
|
# kind: ScheduledBackup
|
||||||
# metadata:
|
# metadata:
|
||||||
|
|
@ -49,4 +58,4 @@
|
||||||
# backupOwnerReference: self
|
# backupOwnerReference: self
|
||||||
# cluster:
|
# cluster:
|
||||||
# name: apps-pg
|
# name: apps-pg
|
||||||
# immediate: true
|
# immediate: true
|
||||||
|
|
|
||||||
394
manifests/cnpg-option-a-backup.yaml
Normal file
394
manifests/cnpg-option-a-backup.yaml
Normal file
|
|
@ -0,0 +1,394 @@
|
||||||
|
# Option A — age-encrypted logical dumps → Nextcloud WebDAV (RAILIANCE-WP-0015).
|
||||||
|
#
|
||||||
|
# Decided approach (not barman ScheduledBackup). Applies to CoulombCore
|
||||||
|
# databases namespace clusters: apps-pg, gitea-db, net-kingdom-pg, state-hub-db.
|
||||||
|
#
|
||||||
|
# Prereqs:
|
||||||
|
# 1. OpenBao lane CCR-2026-0004 resolvable
|
||||||
|
# 2. make cnpg-backup-offsite-secret-apply # Secret cnpg-backup-offsite
|
||||||
|
# 3. kubectl apply -f manifests/cnpg-option-a-backup.yaml
|
||||||
|
#
|
||||||
|
# Schedule: daily 02:30 UTC (RPO 24h). Retention target 14 daily + 4 weekly
|
||||||
|
# is enforced on the Nextcloud side / operator prune; local Job pods are ephemeral.
|
||||||
|
#
|
||||||
|
# Health: make cnpg-backup-status
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ServiceAccount
|
||||||
|
metadata:
|
||||||
|
name: cnpg-logical-backup
|
||||||
|
namespace: databases
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/part-of: railiance-apps
|
||||||
|
railiance.apps/backup-lane: option-a
|
||||||
|
---
|
||||||
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
|
kind: Role
|
||||||
|
metadata:
|
||||||
|
name: cnpg-logical-backup
|
||||||
|
namespace: databases
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/part-of: railiance-apps
|
||||||
|
railiance.apps/backup-lane: option-a
|
||||||
|
rules:
|
||||||
|
- apiGroups: [""]
|
||||||
|
resources: ["pods"]
|
||||||
|
verbs: ["get", "list"]
|
||||||
|
- apiGroups: [""]
|
||||||
|
resources: ["pods/exec"]
|
||||||
|
verbs: ["create"]
|
||||||
|
- apiGroups: [""]
|
||||||
|
resources: ["configmaps"]
|
||||||
|
verbs: ["get", "create", "update", "patch"]
|
||||||
|
- apiGroups: [""]
|
||||||
|
resources: ["secrets"]
|
||||||
|
resourceNames: ["cnpg-backup-offsite"]
|
||||||
|
verbs: ["get"]
|
||||||
|
---
|
||||||
|
apiVersion: rbac.authorization.k8s.io/v1
|
||||||
|
kind: RoleBinding
|
||||||
|
metadata:
|
||||||
|
name: cnpg-logical-backup
|
||||||
|
namespace: databases
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/part-of: railiance-apps
|
||||||
|
railiance.apps/backup-lane: option-a
|
||||||
|
roleRef:
|
||||||
|
apiGroup: rbac.authorization.k8s.io
|
||||||
|
kind: Role
|
||||||
|
name: cnpg-logical-backup
|
||||||
|
subjects:
|
||||||
|
- kind: ServiceAccount
|
||||||
|
name: cnpg-logical-backup
|
||||||
|
namespace: databases
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: cnpg-option-a-runner
|
||||||
|
namespace: databases
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/part-of: railiance-apps
|
||||||
|
railiance.apps/backup-lane: option-a
|
||||||
|
data:
|
||||||
|
run-backup.sh: |
|
||||||
|
#!/bin/sh
|
||||||
|
# In-cluster Option A runner. Env: CLUSTER, DATABASES (comma-separated).
|
||||||
|
set -eu
|
||||||
|
CLUSTER="${CLUSTER:?CLUSTER required}"
|
||||||
|
DATABASES="${DATABASES:?DATABASES required}"
|
||||||
|
NAMESPACE="${NAMESPACE:-databases}"
|
||||||
|
STATUS_CM="${STATUS_CM:-cnpg-option-a-status}"
|
||||||
|
AGE_BIN="${AGE_BIN:-/tmp/age/age}"
|
||||||
|
WORK=/tmp/backup-work
|
||||||
|
mkdir -p "$WORK" /tmp/age
|
||||||
|
|
||||||
|
if [ ! -x "$AGE_BIN" ]; then
|
||||||
|
echo "fetch age static binary"
|
||||||
|
curl -fsSL "https://github.com/FiloSottile/age/releases/download/v1.2.1/age-v1.2.1-linux-amd64.tar.gz" \
|
||||||
|
| tar -xz -C /tmp
|
||||||
|
# tarball extracts to /tmp/age/age
|
||||||
|
AGE_BIN=/tmp/age/age
|
||||||
|
fi
|
||||||
|
|
||||||
|
NC_TOKEN="$(cat /secrets/offsite/NC_WEBDAV_TOKEN)"
|
||||||
|
NC_URL="$(cat /secrets/offsite/NC_WEBDAV_URL)"
|
||||||
|
AGE_PUB="$(cat /secrets/offsite/AGE_PUBLIC_KEY)"
|
||||||
|
# filesdrop shares often need token-only URL form
|
||||||
|
case "$NC_URL" in
|
||||||
|
*/) ;;
|
||||||
|
*) NC_URL="${NC_URL}/" ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
POD="$(kubectl get pods -n "$NAMESPACE" \
|
||||||
|
-l "cnpg.io/cluster=${CLUSTER},role=primary" \
|
||||||
|
-o jsonpath='{.items[0].metadata.name}')"
|
||||||
|
if [ -z "$POD" ]; then
|
||||||
|
echo "ERROR: primary pod not found for ${CLUSTER}" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "cluster=${CLUSTER} pod=${POD}"
|
||||||
|
|
||||||
|
TS="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||||
|
OLD_IFS=$IFS
|
||||||
|
IFS=,
|
||||||
|
for db in $DATABASES; do
|
||||||
|
IFS=$OLD_IFS
|
||||||
|
plain="${WORK}/${CLUSTER}-${db}-${TS}.dump"
|
||||||
|
aged="${plain}.age"
|
||||||
|
echo "dump ${db}"
|
||||||
|
kubectl exec -n "$NAMESPACE" "$POD" -c postgres -- \
|
||||||
|
pg_dump -U postgres -d "$db" -Fc --no-owner --no-acl >"$plain"
|
||||||
|
"$AGE_BIN" -r "$AGE_PUB" -o "$aged" "$plain"
|
||||||
|
rm -f "$plain"
|
||||||
|
remote="${CLUSTER}/${CLUSTER}-${db}-${TS}.dump.age"
|
||||||
|
dest="${NC_URL}${remote}"
|
||||||
|
echo "upload ${remote}"
|
||||||
|
curl -sf -u "${NC_TOKEN}:" -T "$aged" "$dest"
|
||||||
|
# weekly on Sunday UTC
|
||||||
|
if [ "$(date -u +%u)" = "7" ]; then
|
||||||
|
curl -sf -u "${NC_TOKEN}:" -T "$aged" \
|
||||||
|
"${NC_URL}${CLUSTER}/${CLUSTER}-${db}-weekly-${TS}.dump.age" || true
|
||||||
|
fi
|
||||||
|
rm -f "$aged"
|
||||||
|
done
|
||||||
|
IFS=$OLD_IFS
|
||||||
|
|
||||||
|
# Record last success for cnpg-backup-status
|
||||||
|
if kubectl get configmap "$STATUS_CM" -n "$NAMESPACE" >/dev/null 2>&1; then
|
||||||
|
kubectl patch configmap "$STATUS_CM" -n "$NAMESPACE" --type merge \
|
||||||
|
-p "{\"data\":{\"${CLUSTER}\":\"${TS}\",\"updated\":\"${TS}\",\"lane\":\"option-a\"}}"
|
||||||
|
else
|
||||||
|
kubectl create configmap "$STATUS_CM" -n "$NAMESPACE" \
|
||||||
|
--from-literal="${CLUSTER}=${TS}" \
|
||||||
|
--from-literal=updated="${TS}" \
|
||||||
|
--from-literal=lane=option-a
|
||||||
|
fi
|
||||||
|
kubectl label configmap "$STATUS_CM" -n "$NAMESPACE" \
|
||||||
|
railiance.apps/backup-lane=option-a --overwrite >/dev/null
|
||||||
|
echo "ok: ${CLUSTER} backup ${TS}"
|
||||||
|
---
|
||||||
|
# Shared pod template fields are repeated per CronJob for readability (no Kustomize).
|
||||||
|
apiVersion: batch/v1
|
||||||
|
kind: CronJob
|
||||||
|
metadata:
|
||||||
|
name: cnpg-logical-backup-apps-pg
|
||||||
|
namespace: databases
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/part-of: railiance-apps
|
||||||
|
railiance.apps/backup-lane: option-a
|
||||||
|
railiance.apps/backup-cluster: apps-pg
|
||||||
|
spec:
|
||||||
|
schedule: "30 2 * * *"
|
||||||
|
concurrencyPolicy: Forbid
|
||||||
|
successfulJobsHistoryLimit: 3
|
||||||
|
failedJobsHistoryLimit: 3
|
||||||
|
jobTemplate:
|
||||||
|
spec:
|
||||||
|
backoffLimit: 1
|
||||||
|
activeDeadlineSeconds: 7200
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
railiance.apps/backup-lane: option-a
|
||||||
|
railiance.apps/backup-cluster: apps-pg
|
||||||
|
spec:
|
||||||
|
serviceAccountName: cnpg-logical-backup
|
||||||
|
restartPolicy: Never
|
||||||
|
containers:
|
||||||
|
- name: backup
|
||||||
|
image: alpine/k8s:1.31.12
|
||||||
|
imagePullPolicy: IfNotPresent
|
||||||
|
env:
|
||||||
|
- name: CLUSTER
|
||||||
|
value: apps-pg
|
||||||
|
- name: DATABASES
|
||||||
|
value: apps_meta,core_hub,core_hub_staging,vergabe_db
|
||||||
|
- name: NAMESPACE
|
||||||
|
value: databases
|
||||||
|
command: ["/bin/sh", "/runner/run-backup.sh"]
|
||||||
|
volumeMounts:
|
||||||
|
- name: runner
|
||||||
|
mountPath: /runner
|
||||||
|
readOnly: true
|
||||||
|
- name: offsite
|
||||||
|
mountPath: /secrets/offsite
|
||||||
|
readOnly: true
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 256Mi
|
||||||
|
limits:
|
||||||
|
cpu: "1"
|
||||||
|
memory: 1Gi
|
||||||
|
volumes:
|
||||||
|
- name: runner
|
||||||
|
configMap:
|
||||||
|
name: cnpg-option-a-runner
|
||||||
|
defaultMode: 0755
|
||||||
|
- name: offsite
|
||||||
|
secret:
|
||||||
|
secretName: cnpg-backup-offsite
|
||||||
|
---
|
||||||
|
apiVersion: batch/v1
|
||||||
|
kind: CronJob
|
||||||
|
metadata:
|
||||||
|
name: cnpg-logical-backup-gitea-db
|
||||||
|
namespace: databases
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/part-of: railiance-apps
|
||||||
|
railiance.apps/backup-lane: option-a
|
||||||
|
railiance.apps/backup-cluster: gitea-db
|
||||||
|
spec:
|
||||||
|
schedule: "35 2 * * *"
|
||||||
|
concurrencyPolicy: Forbid
|
||||||
|
successfulJobsHistoryLimit: 3
|
||||||
|
failedJobsHistoryLimit: 3
|
||||||
|
jobTemplate:
|
||||||
|
spec:
|
||||||
|
backoffLimit: 1
|
||||||
|
activeDeadlineSeconds: 7200
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
railiance.apps/backup-lane: option-a
|
||||||
|
railiance.apps/backup-cluster: gitea-db
|
||||||
|
spec:
|
||||||
|
serviceAccountName: cnpg-logical-backup
|
||||||
|
restartPolicy: Never
|
||||||
|
containers:
|
||||||
|
- name: backup
|
||||||
|
image: alpine/k8s:1.31.12
|
||||||
|
imagePullPolicy: IfNotPresent
|
||||||
|
env:
|
||||||
|
- name: CLUSTER
|
||||||
|
value: gitea-db
|
||||||
|
- name: DATABASES
|
||||||
|
value: gitea
|
||||||
|
- name: NAMESPACE
|
||||||
|
value: databases
|
||||||
|
command: ["/bin/sh", "/runner/run-backup.sh"]
|
||||||
|
volumeMounts:
|
||||||
|
- name: runner
|
||||||
|
mountPath: /runner
|
||||||
|
readOnly: true
|
||||||
|
- name: offsite
|
||||||
|
mountPath: /secrets/offsite
|
||||||
|
readOnly: true
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 256Mi
|
||||||
|
limits:
|
||||||
|
cpu: "1"
|
||||||
|
memory: 1Gi
|
||||||
|
volumes:
|
||||||
|
- name: runner
|
||||||
|
configMap:
|
||||||
|
name: cnpg-option-a-runner
|
||||||
|
defaultMode: 0755
|
||||||
|
- name: offsite
|
||||||
|
secret:
|
||||||
|
secretName: cnpg-backup-offsite
|
||||||
|
---
|
||||||
|
apiVersion: batch/v1
|
||||||
|
kind: CronJob
|
||||||
|
metadata:
|
||||||
|
name: cnpg-logical-backup-net-kingdom-pg
|
||||||
|
namespace: databases
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/part-of: railiance-apps
|
||||||
|
railiance.apps/backup-lane: option-a
|
||||||
|
railiance.apps/backup-cluster: net-kingdom-pg
|
||||||
|
spec:
|
||||||
|
schedule: "40 2 * * *"
|
||||||
|
concurrencyPolicy: Forbid
|
||||||
|
successfulJobsHistoryLimit: 3
|
||||||
|
failedJobsHistoryLimit: 3
|
||||||
|
jobTemplate:
|
||||||
|
spec:
|
||||||
|
backoffLimit: 1
|
||||||
|
activeDeadlineSeconds: 7200
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
railiance.apps/backup-lane: option-a
|
||||||
|
railiance.apps/backup-cluster: net-kingdom-pg
|
||||||
|
spec:
|
||||||
|
serviceAccountName: cnpg-logical-backup
|
||||||
|
restartPolicy: Never
|
||||||
|
containers:
|
||||||
|
- name: backup
|
||||||
|
image: alpine/k8s:1.31.12
|
||||||
|
imagePullPolicy: IfNotPresent
|
||||||
|
env:
|
||||||
|
- name: CLUSTER
|
||||||
|
value: net-kingdom-pg
|
||||||
|
- name: DATABASES
|
||||||
|
value: interhub,privacyidea_db
|
||||||
|
- name: NAMESPACE
|
||||||
|
value: databases
|
||||||
|
command: ["/bin/sh", "/runner/run-backup.sh"]
|
||||||
|
volumeMounts:
|
||||||
|
- name: runner
|
||||||
|
mountPath: /runner
|
||||||
|
readOnly: true
|
||||||
|
- name: offsite
|
||||||
|
mountPath: /secrets/offsite
|
||||||
|
readOnly: true
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 256Mi
|
||||||
|
limits:
|
||||||
|
cpu: "1"
|
||||||
|
memory: 1Gi
|
||||||
|
volumes:
|
||||||
|
- name: runner
|
||||||
|
configMap:
|
||||||
|
name: cnpg-option-a-runner
|
||||||
|
defaultMode: 0755
|
||||||
|
- name: offsite
|
||||||
|
secret:
|
||||||
|
secretName: cnpg-backup-offsite
|
||||||
|
---
|
||||||
|
apiVersion: batch/v1
|
||||||
|
kind: CronJob
|
||||||
|
metadata:
|
||||||
|
name: cnpg-logical-backup-state-hub-db
|
||||||
|
namespace: databases
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/part-of: railiance-apps
|
||||||
|
railiance.apps/backup-lane: option-a
|
||||||
|
railiance.apps/backup-cluster: state-hub-db
|
||||||
|
spec:
|
||||||
|
schedule: "45 2 * * *"
|
||||||
|
concurrencyPolicy: Forbid
|
||||||
|
successfulJobsHistoryLimit: 3
|
||||||
|
failedJobsHistoryLimit: 3
|
||||||
|
jobTemplate:
|
||||||
|
spec:
|
||||||
|
backoffLimit: 1
|
||||||
|
activeDeadlineSeconds: 7200
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
railiance.apps/backup-lane: option-a
|
||||||
|
railiance.apps/backup-cluster: state-hub-db
|
||||||
|
spec:
|
||||||
|
serviceAccountName: cnpg-logical-backup
|
||||||
|
restartPolicy: Never
|
||||||
|
containers:
|
||||||
|
- name: backup
|
||||||
|
image: alpine/k8s:1.31.12
|
||||||
|
imagePullPolicy: IfNotPresent
|
||||||
|
env:
|
||||||
|
- name: CLUSTER
|
||||||
|
value: state-hub-db
|
||||||
|
- name: DATABASES
|
||||||
|
value: state_hub
|
||||||
|
- name: NAMESPACE
|
||||||
|
value: databases
|
||||||
|
command: ["/bin/sh", "/runner/run-backup.sh"]
|
||||||
|
volumeMounts:
|
||||||
|
- name: runner
|
||||||
|
mountPath: /runner
|
||||||
|
readOnly: true
|
||||||
|
- name: offsite
|
||||||
|
mountPath: /secrets/offsite
|
||||||
|
readOnly: true
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 256Mi
|
||||||
|
limits:
|
||||||
|
cpu: "1"
|
||||||
|
memory: 1Gi
|
||||||
|
volumes:
|
||||||
|
- name: runner
|
||||||
|
configMap:
|
||||||
|
name: cnpg-option-a-runner
|
||||||
|
defaultMode: 0755
|
||||||
|
- name: offsite
|
||||||
|
secret:
|
||||||
|
secretName: cnpg-backup-offsite
|
||||||
67
tools/cnpg-backup-offsite-secret-apply.sh
Executable file
67
tools/cnpg-backup-offsite-secret-apply.sh
Executable file
|
|
@ -0,0 +1,67 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Materialize databases/cnpg-backup-offsite from OpenBao (RAILIANCE-WP-0015-T02).
|
||||||
|
# Never prints secret values. Does not place AGE_PRIVATE_KEY in-cluster.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
NAMESPACE="${CNPG_NAMESPACE:-databases}"
|
||||||
|
SECRET_NAME="${CNPG_BACKUP_OFFSITE_SECRET:-cnpg-backup-offsite}"
|
||||||
|
BAO_PATH="${RAILIANCE_BACKUP_BAO_PATH:-platform/workloads/railiance/backup/offsite-lane}"
|
||||||
|
AGE_PUBLIC_KEY="${RAILIANCE_BACKUP_AGE_PUBLIC_KEY:-age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz}"
|
||||||
|
DRY_RUN="${DRY_RUN:-0}"
|
||||||
|
|
||||||
|
if ! command -v bao >/dev/null 2>&1; then
|
||||||
|
echo "ERROR: bao CLI required" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
if ! command -v kubectl >/dev/null 2>&1; then
|
||||||
|
echo "ERROR: kubectl required" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! bao token lookup >/dev/null 2>&1; then
|
||||||
|
echo "ERROR: no valid OpenBao token" >&2
|
||||||
|
echo "Run: bao login -method=oidc -path=netkingdom role=railiance-backup-workload-kv-read" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! bao kv metadata get "$BAO_PATH" >/dev/null 2>&1; then
|
||||||
|
echo "ERROR: cannot read metadata for ${BAO_PATH}" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Fetch into env without echoing
|
||||||
|
NC_WEBDAV_TOKEN="$(bao kv get -field=NC_WEBDAV_TOKEN "$BAO_PATH")"
|
||||||
|
NC_WEBDAV_URL="$(bao kv get -field=NC_WEBDAV_URL "$BAO_PATH")"
|
||||||
|
|
||||||
|
if [[ -z "${NC_WEBDAV_TOKEN}" || -z "${NC_WEBDAV_URL}" ]]; then
|
||||||
|
echo "ERROR: NC_WEBDAV_TOKEN or NC_WEBDAV_URL empty at ${BAO_PATH}" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "cnpg-backup-offsite secret apply"
|
||||||
|
echo " namespace: ${NAMESPACE}"
|
||||||
|
echo " secret: ${SECRET_NAME}"
|
||||||
|
echo " source: ${BAO_PATH}"
|
||||||
|
echo " keys: NC_WEBDAV_TOKEN, NC_WEBDAV_URL, AGE_PUBLIC_KEY"
|
||||||
|
echo " dry-run: ${DRY_RUN}"
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == "1" ]]; then
|
||||||
|
echo "ok: dry-run — would create/update Secret ${NAMESPACE}/${SECRET_NAME}"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
kubectl create secret generic "$SECRET_NAME" \
|
||||||
|
-n "$NAMESPACE" \
|
||||||
|
--from-literal=NC_WEBDAV_TOKEN="$NC_WEBDAV_TOKEN" \
|
||||||
|
--from-literal=NC_WEBDAV_URL="$NC_WEBDAV_URL" \
|
||||||
|
--from-literal=AGE_PUBLIC_KEY="$AGE_PUBLIC_KEY" \
|
||||||
|
--dry-run=client -o yaml \
|
||||||
|
| kubectl apply -f -
|
||||||
|
|
||||||
|
# Clear locals
|
||||||
|
unset NC_WEBDAV_TOKEN NC_WEBDAV_URL
|
||||||
|
|
||||||
|
# Non-secret verification
|
||||||
|
keys="$(kubectl get secret "$SECRET_NAME" -n "$NAMESPACE" -o jsonpath='{.data}' \
|
||||||
|
| python3 -c 'import sys,json; print(",".join(sorted(json.load(sys.stdin).keys())))')"
|
||||||
|
echo "ok: Secret ${NAMESPACE}/${SECRET_NAME} applied (keys: ${keys})"
|
||||||
|
|
@ -1,7 +1,12 @@
|
||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
|
# CNPG backup posture for Option A (logical CronJobs) and legacy barman ScheduledBackup.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
namespace="${CNPG_NAMESPACE:-databases}"
|
namespace="${CNPG_NAMESPACE:-databases}"
|
||||||
|
# RPO target is 24h; allow 36h before last-success is considered stale.
|
||||||
|
rpo_hours="${CNPG_BACKUP_RPO_HOURS:-36}"
|
||||||
|
status_cm="${CNPG_OPTION_A_STATUS_CM:-cnpg-option-a-status}"
|
||||||
|
expected_clusters="${CNPG_EXPECTED_CLUSTERS:-apps-pg,gitea-db,net-kingdom-pg,state-hub-db}"
|
||||||
|
|
||||||
if ! kubectl get ns "$namespace" >/dev/null 2>&1; then
|
if ! kubectl get ns "$namespace" >/dev/null 2>&1; then
|
||||||
echo "ERROR: namespace $namespace is not reachable from the current kubeconfig" >&2
|
echo "ERROR: namespace $namespace is not reachable from the current kubeconfig" >&2
|
||||||
|
|
@ -13,6 +18,8 @@ context="$(kubectl config current-context 2>/dev/null || echo unknown)"
|
||||||
server="$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}' 2>/dev/null || echo unknown)"
|
server="$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}' 2>/dev/null || echo unknown)"
|
||||||
echo "kube-context: ${context}"
|
echo "kube-context: ${context}"
|
||||||
echo "api-server: ${server}"
|
echo "api-server: ${server}"
|
||||||
|
echo "lane: Option A (logical dump → age → Nextcloud) preferred; barman ScheduledBackup also accepted"
|
||||||
|
echo "RPO window: ${rpo_hours}h (last-success freshness)"
|
||||||
echo
|
echo
|
||||||
|
|
||||||
mapfile -t clusters < <(
|
mapfile -t clusters < <(
|
||||||
|
|
@ -24,8 +31,52 @@ if ((${#clusters[@]} == 0)); then
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# Load status ConfigMap timestamps (cluster -> ISO-ish stamp YYYYMMDDTHHMMSSZ)
|
||||||
|
declare -A last_success=()
|
||||||
|
if kubectl get configmap "$status_cm" -n "$namespace" >/dev/null 2>&1; then
|
||||||
|
while IFS= read -r line; do
|
||||||
|
[[ -z "$line" ]] && continue
|
||||||
|
key="${line%%=*}"
|
||||||
|
val="${line#*=}"
|
||||||
|
case "$key" in
|
||||||
|
lane|updated) continue ;;
|
||||||
|
*) last_success["$key"]="$val" ;;
|
||||||
|
esac
|
||||||
|
done < <(
|
||||||
|
kubectl get configmap "$status_cm" -n "$namespace" -o json 2>/dev/null \
|
||||||
|
| python3 -c 'import sys,json
|
||||||
|
d=json.load(sys.stdin).get("data") or {}
|
||||||
|
for k,v in sorted(d.items()):
|
||||||
|
print(f"{k}={v}")' || true
|
||||||
|
)
|
||||||
|
echo "status ConfigMap: ${status_cm} (present)"
|
||||||
|
else
|
||||||
|
echo "status ConfigMap: ${status_cm} (missing)"
|
||||||
|
fi
|
||||||
|
echo
|
||||||
|
|
||||||
|
fresh_enough() {
|
||||||
|
local ts="$1"
|
||||||
|
[[ -z "$ts" ]] && return 1
|
||||||
|
# Accept YYYYMMDDTHHMMSSZ
|
||||||
|
if [[ ! "$ts" =~ ^[0-9]{8}T[0-9]{6}Z$ ]]; then
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
local epoch now delta max_delta
|
||||||
|
epoch="$(date -u -d "${ts:0:4}-${ts:4:2}-${ts:6:2} ${ts:9:2}:${ts:11:2}:${ts:13:2}" +%s 2>/dev/null || true)"
|
||||||
|
if [[ -z "$epoch" ]]; then
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
now="$(date -u +%s)"
|
||||||
|
delta=$((now - epoch))
|
||||||
|
max_delta=$((rpo_hours * 3600))
|
||||||
|
((delta >= 0 && delta <= max_delta))
|
||||||
|
}
|
||||||
|
|
||||||
missing=0
|
missing=0
|
||||||
logical_backup_note=0
|
degraded_detail=()
|
||||||
|
|
||||||
|
IFS=',' read -r -a expected <<<"$expected_clusters"
|
||||||
|
|
||||||
for cluster in "${clusters[@]}"; do
|
for cluster in "${clusters[@]}"; do
|
||||||
if ! kubectl get cluster "$cluster" -n "$namespace" >/dev/null 2>&1; then
|
if ! kubectl get cluster "$cluster" -n "$namespace" >/dev/null 2>&1; then
|
||||||
|
|
@ -38,45 +89,104 @@ for cluster in "${clusters[@]}"; do
|
||||||
backup_spec="$(kubectl get cluster "$cluster" -n "$namespace" \
|
backup_spec="$(kubectl get cluster "$cluster" -n "$namespace" \
|
||||||
-o jsonpath='{.spec.backup}' 2>/dev/null || true)"
|
-o jsonpath='{.spec.backup}' 2>/dev/null || true)"
|
||||||
if [[ -z "$backup_spec" || "$backup_spec" == "null" ]]; then
|
if [[ -z "$backup_spec" || "$backup_spec" == "null" ]]; then
|
||||||
echo "${cluster}: backup spec = null"
|
echo "${cluster}: backup spec = null (expected for Option A)"
|
||||||
else
|
else
|
||||||
echo "${cluster}: backup spec configured"
|
echo "${cluster}: backup spec configured (barman path)"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
scheduled="$(kubectl get scheduledbackup -n "$namespace" \
|
scheduled="$(kubectl get scheduledbackup -n "$namespace" \
|
||||||
-o jsonpath="{range .items[?(@.spec.cluster.name=='${cluster}')]}{.metadata.name}{' '}{end}" 2>/dev/null || true)"
|
-o jsonpath="{range .items[?(@.spec.cluster.name=='${cluster}')]}{.metadata.name}{' '}{end}" 2>/dev/null || true)"
|
||||||
scheduled="${scheduled%" "}"
|
scheduled="${scheduled%" "}"
|
||||||
if [[ -z "$scheduled" ]]; then
|
|
||||||
echo "${cluster}: ScheduledBackup = none"
|
option_a_cron="$(kubectl get cronjob -n "$namespace" \
|
||||||
missing=$((missing + 1))
|
-l "railiance.apps/backup-lane=option-a,railiance.apps/backup-cluster=${cluster}" \
|
||||||
if [[ "$cluster" == "apps-pg" || "$cluster" == "gitea-db" || "$cluster" == "forgejo-db" ]]; then
|
-o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.suspend}{" "}{end}' 2>/dev/null || true)"
|
||||||
echo "${cluster}: Phase 1 fallback = logical pg_dump (see make apps-pg-backup-dry-run or platform forgejo-backup)"
|
|
||||||
logical_backup_note=1
|
covered=0
|
||||||
|
schedule_note=""
|
||||||
|
|
||||||
|
if [[ -n "$scheduled" ]]; then
|
||||||
|
covered=1
|
||||||
|
schedule_note="ScheduledBackup=${scheduled}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -n "$option_a_cron" ]]; then
|
||||||
|
# format: name suspend name suspend ...
|
||||||
|
read -r cj_name cj_suspend _ <<<"$option_a_cron"
|
||||||
|
if [[ "${cj_suspend}" == "true" ]]; then
|
||||||
|
echo "${cluster}: Option A CronJob = ${cj_name} (SUSPENDED)"
|
||||||
|
schedule_note="${schedule_note:+$schedule_note; }OptionA=${cj_name}:suspended"
|
||||||
|
else
|
||||||
|
covered=1
|
||||||
|
schedule_note="${schedule_note:+$schedule_note; }OptionA=${cj_name}"
|
||||||
|
echo "${cluster}: Option A CronJob = ${cj_name}"
|
||||||
fi
|
fi
|
||||||
else
|
else
|
||||||
|
echo "${cluster}: Option A CronJob = none"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -n "$scheduled" ]]; then
|
||||||
echo "${cluster}: ScheduledBackup = ${scheduled}"
|
echo "${cluster}: ScheduledBackup = ${scheduled}"
|
||||||
fi
|
|
||||||
|
|
||||||
if command -v kubectl >/dev/null 2>&1 && kubectl cnpg status "$cluster" -n "$namespace" >/dev/null 2>&1; then
|
|
||||||
echo "${cluster}: kubectl cnpg status available"
|
|
||||||
kubectl cnpg status "$cluster" -n "$namespace" | sed -n '1,8p'
|
|
||||||
else
|
else
|
||||||
phase="$(kubectl get cluster "$cluster" -n "$namespace" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
echo "${cluster}: ScheduledBackup = none"
|
||||||
instances="$(kubectl get cluster "$cluster" -n "$namespace" -o jsonpath='{.status.instances}' 2>/dev/null || true)"
|
|
||||||
echo "${cluster}: phase=${phase:-unknown} instances=${instances:-unknown}"
|
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
ts="${last_success[$cluster]:-}"
|
||||||
|
if [[ -n "$ts" ]]; then
|
||||||
|
if fresh_enough "$ts"; then
|
||||||
|
echo "${cluster}: last-success = ${ts} (fresh ≤${rpo_hours}h)"
|
||||||
|
else
|
||||||
|
echo "${cluster}: last-success = ${ts} (STALE >${rpo_hours}h)"
|
||||||
|
if ((covered)); then
|
||||||
|
degraded_detail+=("${cluster}:stale-success")
|
||||||
|
# scheduled but stale still counts as coverage with warning; mark degraded
|
||||||
|
covered=0
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo "${cluster}: last-success = none"
|
||||||
|
# Schedule present but never succeeded: still incomplete for healthy
|
||||||
|
if ((covered)); then
|
||||||
|
echo "${cluster}: note = schedule present but no recorded success yet"
|
||||||
|
covered=0
|
||||||
|
degraded_detail+=("${cluster}:no-success-yet")
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
phase="$(kubectl get cluster "$cluster" -n "$namespace" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
||||||
|
instances="$(kubectl get cluster "$cluster" -n "$namespace" -o jsonpath='{.status.instances}' 2>/dev/null || true)"
|
||||||
|
echo "${cluster}: phase=${phase:-unknown} instances=${instances:-unknown}"
|
||||||
|
|
||||||
|
if ((covered == 0)); then
|
||||||
|
missing=$((missing + 1))
|
||||||
|
echo "${cluster}: coverage = MISSING"
|
||||||
|
else
|
||||||
|
echo "${cluster}: coverage = ok (${schedule_note})"
|
||||||
|
fi
|
||||||
echo
|
echo
|
||||||
done
|
done
|
||||||
|
|
||||||
if (( missing > 0 )); then
|
# Warn if expected clusters are absent from the cluster list
|
||||||
echo "RESULT: degraded — ${missing} cluster(s) lack ScheduledBackup coverage"
|
for exp in "${expected[@]}"; do
|
||||||
echo "See docs/app-data-backup-restore-handoff.md and manifests/cnpg-backup-readiness.yaml"
|
found=0
|
||||||
if (( logical_backup_note )); then
|
for c in "${clusters[@]}"; do
|
||||||
echo "Phase 1: logical pg_dump lane may still run when OpenBao offsite-lane is reachable"
|
[[ "$c" == "$exp" ]] && found=1 && break
|
||||||
echo "See docs/credential-routing-railiance-apps.md"
|
done
|
||||||
|
if ((found == 0)); then
|
||||||
|
echo "WARN: expected cluster ${exp} not found in ${namespace}"
|
||||||
|
missing=$((missing + 1))
|
||||||
fi
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
if ((missing > 0)); then
|
||||||
|
echo "RESULT: degraded — ${missing} cluster(s) lack healthy Option A / ScheduledBackup coverage"
|
||||||
|
if ((${#degraded_detail[@]} > 0)); then
|
||||||
|
echo "detail: ${degraded_detail[*]}"
|
||||||
|
fi
|
||||||
|
echo "See docs/app-data-backup-restore-handoff.md and manifests/cnpg-option-a-backup.yaml"
|
||||||
|
echo "Operator: make cnpg-backup-offsite-secret-apply && make cnpg-option-a-apply"
|
||||||
|
echo "Manual run: make cnpg-logical-backup (or DRY_RUN=1 make cnpg-logical-backup-dry-run)"
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
echo "RESULT: ok — all tracked clusters have ScheduledBackup resources"
|
echo "RESULT: ok — all tracked clusters have healthy Option A (or barman) backup coverage"
|
||||||
|
|
|
||||||
158
tools/cnpg-logical-backup.sh
Executable file
158
tools/cnpg-logical-backup.sh
Executable file
|
|
@ -0,0 +1,158 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Option A logical backup for CoulombCore CNPG clusters (RAILIANCE-WP-0015-T03).
|
||||||
|
# pg_dump -Fc → age → Nextcloud WebDAV; records last-success ConfigMap.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
PLATFORM_REPO="${PLATFORM_REPO:-$HOME/railiance-platform}"
|
||||||
|
COMMON_SH="${PLATFORM_REPO}/lib/railiance-backup-common.sh"
|
||||||
|
|
||||||
|
NAMESPACE="${CNPG_NAMESPACE:-databases}"
|
||||||
|
BACKUP_ROOT="${BACKUP_DIR:-$HOME/.cache/railiance/backups/cnpg}"
|
||||||
|
DRY_RUN="${RAILIANCE_BACKUP_DRY_RUN:-1}"
|
||||||
|
STATUS_CM="${CNPG_OPTION_A_STATUS_CM:-cnpg-option-a-status}"
|
||||||
|
CLUSTERS_FILTER="${CNPG_BACKUP_CLUSTERS:-}"
|
||||||
|
UPDATE_STATUS="${CNPG_UPDATE_STATUS_CM:-1}"
|
||||||
|
|
||||||
|
# cluster|db1,db2,... (skip template/postgres system DB)
|
||||||
|
DEFAULT_TARGETS=(
|
||||||
|
"apps-pg|apps_meta,core_hub,core_hub_staging,vergabe_db"
|
||||||
|
"gitea-db|gitea"
|
||||||
|
"net-kingdom-pg|interhub,privacyidea_db"
|
||||||
|
"state-hub-db|state_hub"
|
||||||
|
)
|
||||||
|
|
||||||
|
if [[ ! -f "$COMMON_SH" ]]; then
|
||||||
|
echo "ERROR: missing ${COMMON_SH}" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "$COMMON_SH"
|
||||||
|
|
||||||
|
railiance_backup_require_tools
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" != "1" ]]; then
|
||||||
|
railiance_backup_require_openbao_lane
|
||||||
|
fi
|
||||||
|
|
||||||
|
primary_pod() {
|
||||||
|
local cluster="$1"
|
||||||
|
kubectl get pods -n "$NAMESPACE" \
|
||||||
|
-l "cnpg.io/cluster=${cluster},role=primary" \
|
||||||
|
-o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true
|
||||||
|
}
|
||||||
|
|
||||||
|
dump_db() {
|
||||||
|
local cluster="$1" pod="$2" db="$3" out="$4"
|
||||||
|
kubectl exec -n "$NAMESPACE" "$pod" -c postgres -- \
|
||||||
|
pg_dump -U postgres -d "$db" -Fc --no-owner --no-acl >"$out"
|
||||||
|
}
|
||||||
|
|
||||||
|
record_status_cm() {
|
||||||
|
local cluster="$1" ts="$2"
|
||||||
|
[[ "$UPDATE_STATUS" == "1" ]] || return 0
|
||||||
|
if ! kubectl get ns "$NAMESPACE" >/dev/null 2>&1; then
|
||||||
|
echo "WARN: skip status ConfigMap — namespace unreachable" >&2
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
if ! kubectl get configmap "$STATUS_CM" -n "$NAMESPACE" >/dev/null 2>&1; then
|
||||||
|
kubectl create configmap "$STATUS_CM" -n "$NAMESPACE" \
|
||||||
|
--from-literal="${cluster}=${ts}" \
|
||||||
|
--from-literal=lane=option-a \
|
||||||
|
--from-literal=updated="${ts}" \
|
||||||
|
>/dev/null
|
||||||
|
else
|
||||||
|
kubectl patch configmap "$STATUS_CM" -n "$NAMESPACE" --type merge \
|
||||||
|
-p "{\"data\":{\"${cluster}\":\"${ts}\",\"updated\":\"${ts}\",\"lane\":\"option-a\"}}" \
|
||||||
|
>/dev/null
|
||||||
|
fi
|
||||||
|
# label for discovery by cnpg-backup-status
|
||||||
|
kubectl label configmap "$STATUS_CM" -n "$NAMESPACE" \
|
||||||
|
railiance.apps/backup-lane=option-a --overwrite >/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
should_run_cluster() {
|
||||||
|
local cluster="$1"
|
||||||
|
if [[ -z "$CLUSTERS_FILTER" ]]; then
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
[[ ",${CLUSTERS_FILTER}," == *",${cluster},"* ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
TS="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||||
|
echo "cnpg logical backup (Option A)"
|
||||||
|
echo " namespace: ${NAMESPACE}"
|
||||||
|
echo " dry-run: ${DRY_RUN}"
|
||||||
|
echo " ts: ${TS}"
|
||||||
|
echo
|
||||||
|
|
||||||
|
failed=0
|
||||||
|
succeeded_clusters=()
|
||||||
|
|
||||||
|
for entry in "${DEFAULT_TARGETS[@]}"; do
|
||||||
|
cluster="${entry%%|*}"
|
||||||
|
dbs_csv="${entry#*|}"
|
||||||
|
should_run_cluster "$cluster" || continue
|
||||||
|
|
||||||
|
pod="$(primary_pod "$cluster")"
|
||||||
|
if [[ -z "$pod" ]]; then
|
||||||
|
echo "ERROR: ${cluster}: primary pod not found" >&2
|
||||||
|
failed=$((failed + 1))
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
cluster_dir="${BACKUP_ROOT}/${cluster}"
|
||||||
|
mkdir -p "$cluster_dir"
|
||||||
|
cluster_ok=1
|
||||||
|
|
||||||
|
echo "${cluster}: primary=${pod}"
|
||||||
|
IFS=',' read -r -a dbs <<<"$dbs_csv"
|
||||||
|
for db in "${dbs[@]}"; do
|
||||||
|
plain="${cluster_dir}/${cluster}-${db}-${TS}.dump"
|
||||||
|
aged="${plain}.age"
|
||||||
|
echo " dump ${db} → $(basename "$plain")"
|
||||||
|
if ! dump_db "$cluster" "$pod" "$db" "$plain"; then
|
||||||
|
echo " ERROR: pg_dump failed for ${cluster}/${db}" >&2
|
||||||
|
cluster_ok=0
|
||||||
|
failed=$((failed + 1))
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
railiance_backup_encrypt "$plain" "$aged"
|
||||||
|
echo " ok: age $(basename "$aged") ($(du -h "$aged" | awk '{print $1}'))"
|
||||||
|
# drop plaintext after encrypt
|
||||||
|
rm -f "$plain"
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == "1" ]]; then
|
||||||
|
echo " dry-run: skip upload"
|
||||||
|
else
|
||||||
|
RAILIANCE_BACKUP_NC_PREFIX="${cluster}"
|
||||||
|
railiance_backup_nc_upload "$aged" "${cluster}-${db}-${TS}.dump.age"
|
||||||
|
# weekly copy on Sundays (UTC)
|
||||||
|
if [[ "$(date -u +%u)" == "7" ]]; then
|
||||||
|
railiance_backup_nc_upload "$aged" "${cluster}-${db}-weekly-${TS}.dump.age" || true
|
||||||
|
fi
|
||||||
|
echo " ok: uploaded ${cluster}/${cluster}-${db}-${TS}.dump.age"
|
||||||
|
fi
|
||||||
|
railiance_backup_prune_local "$cluster_dir" "${cluster}-${db}-*.dump.age" 7
|
||||||
|
done
|
||||||
|
|
||||||
|
if ((cluster_ok)); then
|
||||||
|
railiance_backup_record_success "$cluster_dir" "$TS"
|
||||||
|
record_status_cm "$cluster" "$TS"
|
||||||
|
succeeded_clusters+=("$cluster")
|
||||||
|
echo " ok: ${cluster} complete"
|
||||||
|
fi
|
||||||
|
echo
|
||||||
|
done
|
||||||
|
|
||||||
|
if ((failed > 0)); then
|
||||||
|
echo "RESULT: failed — ${failed} dump error(s); succeeded: ${succeeded_clusters[*]:-none}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ((${#succeeded_clusters[@]} == 0)); then
|
||||||
|
echo "RESULT: failed — no clusters selected/ran"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "RESULT: ok — clusters: ${succeeded_clusters[*]} (dry-run=${DRY_RUN})"
|
||||||
|
|
@ -4,7 +4,7 @@ type: workplan
|
||||||
title: "CNPG backup ScheduledBackup coverage — drive cnpg-backup-status to healthy"
|
title: "CNPG backup ScheduledBackup coverage — drive cnpg-backup-status to healthy"
|
||||||
domain: financials
|
domain: financials
|
||||||
repo: railiance-apps
|
repo: railiance-apps
|
||||||
status: active
|
status: finished
|
||||||
owner: codex
|
owner: codex
|
||||||
topic_slug: railiance
|
topic_slug: railiance
|
||||||
created: "2026-07-14"
|
created: "2026-07-14"
|
||||||
|
|
@ -25,17 +25,15 @@ scheduled, unattended, encrypted, off-cluster backups with a passing restore
|
||||||
drill for all four production CNPG clusters — `apps-pg`, `gitea-db`,
|
drill for all four production CNPG clusters — `apps-pg`, `gitea-db`,
|
||||||
`net-kingdom-pg`, `state-hub-db` (CoulombCore `databases` namespace).
|
`net-kingdom-pg`, `state-hub-db` (CoulombCore `databases` namespace).
|
||||||
|
|
||||||
## Current state (2026-07-22)
|
## Outcome (2026-07-22)
|
||||||
|
|
||||||
- T01 (CCR-2026-0004 offsite credentials) is **done**: catalog
|
- Option A lane live: Secret `databases/cnpg-backup-offsite`, four CronJobs,
|
||||||
`railiance-backup-offsite-lane` is `resolvable: true` / `readiness: ready`;
|
status ConfigMap, multi-cluster workstation runner.
|
||||||
OpenBao path holds `NC_WEBDAV_TOKEN`, `NC_WEBDAV_URL`, `AGE_PRIVATE_KEY`;
|
- `make cnpg-backup-status` → **ok** for all four clusters.
|
||||||
OIDC role `railiance-backup-workload-kv-read` grants `read` on the data path
|
- Encrypted uploads produced for all consumer DBs; restore drill from
|
||||||
(verified 2026-07-22, capabilities-safe / key inventory only).
|
`apps-pg`/`vergabe_db` age artifact passed 195/195 rows.
|
||||||
- `cnpg-backup-status` still reports degraded under the old barman
|
- Evidence: `docs/evidence/cnpg-option-a-backup-20260722.json`,
|
||||||
`ScheduledBackup` definition; Option A logical-dump schedules are the target.
|
`docs/evidence/apps-pg-restore-drill-20260722T155951Z.json`.
|
||||||
- Phase 1 evidence for `apps-pg`/`vergabe_db` exists (RAILIANCE-WP-0013); this
|
|
||||||
workplan generalizes to all four clusters with unattended coverage.
|
|
||||||
|
|
||||||
## Task: Provision CCR-2026-0004 offsite credentials (operator)
|
## Task: Provision CCR-2026-0004 offsite credentials (operator)
|
||||||
|
|
||||||
|
|
@ -54,15 +52,12 @@ the CCR flips to `resolvable: true`.
|
||||||
|
|
||||||
**Closed 2026-07-22:** CCR `resolvable: true`, `readiness: ready`; OIDC login +
|
**Closed 2026-07-22:** CCR `resolvable: true`, `readiness: ready`; OIDC login +
|
||||||
data-path `read` capability + field presence verified without printing values.
|
data-path `read` capability + field presence verified without printing values.
|
||||||
Note: OIDC token cannot `auth/token/create` (403 expected); verify with
|
|
||||||
`bao token capabilities` on the current token and/or key inventory, not child
|
|
||||||
token minting.
|
|
||||||
|
|
||||||
## Task: Create offsite backup Secret in databases namespace
|
## Task: Create offsite backup Secret in databases namespace
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: RAILIANCE-WP-0015-T02
|
id: RAILIANCE-WP-0015-T02
|
||||||
status: wait
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "7d9b2022-c533-4d49-9a91-fc34ae99aa9a"
|
state_hub_task_id: "7d9b2022-c533-4d49-9a91-fc34ae99aa9a"
|
||||||
```
|
```
|
||||||
|
|
@ -71,11 +66,16 @@ Once T01 resolves, materialize the `databases` namespace Secret from the lane
|
||||||
(no plaintext in Git — sourced via OpenBao/operator path). **Done when:** the
|
(no plaintext in Git — sourced via OpenBao/operator path). **Done when:** the
|
||||||
Secret exists and `pg_dump`/WebDAV upload authenticates.
|
Secret exists and `pg_dump`/WebDAV upload authenticates.
|
||||||
|
|
||||||
|
**Closed 2026-07-22:** `make cnpg-backup-offsite-secret-apply` created
|
||||||
|
`databases/cnpg-backup-offsite` with `NC_WEBDAV_TOKEN`, `NC_WEBDAV_URL`,
|
||||||
|
`AGE_PUBLIC_KEY` (private key not in-cluster). Upload authenticated via
|
||||||
|
workstation runner using the same OpenBao lane.
|
||||||
|
|
||||||
## Task: Wire scheduled backup for all four CNPG clusters
|
## Task: Wire scheduled backup for all four CNPG clusters
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: RAILIANCE-WP-0015-T03
|
id: RAILIANCE-WP-0015-T03
|
||||||
status: wait
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "2b97e3ee-6284-4e5f-b56e-df57194cc7b8"
|
state_hub_task_id: "2b97e3ee-6284-4e5f-b56e-df57194cc7b8"
|
||||||
```
|
```
|
||||||
|
|
@ -87,11 +87,16 @@ Nextcloud WebDAV, 14 daily + 4 weekly retention. Uncomment/parameterize
|
||||||
each cluster has a running scheduled backup producing encrypted off-cluster
|
each cluster has a running scheduled backup producing encrypted off-cluster
|
||||||
artifacts without manual intervention.
|
artifacts without manual intervention.
|
||||||
|
|
||||||
|
**Closed 2026-07-22:** `manifests/cnpg-option-a-backup.yaml` applied (CronJobs
|
||||||
|
02:30–02:45 UTC). Immediate success wave via `tools/cnpg-logical-backup.sh`
|
||||||
|
uploaded encrypted dumps for all consumer databases and wrote
|
||||||
|
`cnpg-option-a-status`.
|
||||||
|
|
||||||
## Task: Reconcile cnpg-backup-status health definition with Option A
|
## Task: Reconcile cnpg-backup-status health definition with Option A
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: RAILIANCE-WP-0015-T04
|
id: RAILIANCE-WP-0015-T04
|
||||||
status: wait
|
status: done
|
||||||
priority: medium
|
priority: medium
|
||||||
state_hub_task_id: "151656cb-73ca-4d52-842e-22f14690f71c"
|
state_hub_task_id: "151656cb-73ca-4d52-842e-22f14690f71c"
|
||||||
```
|
```
|
||||||
|
|
@ -103,11 +108,15 @@ Option-A schedule reports **healthy** (or adopt barman ObjectStore if chosen
|
||||||
instead). **Done when:** `make cnpg-backup-status` reflects the real, decided
|
instead). **Done when:** `make cnpg-backup-status` reflects the real, decided
|
||||||
posture.
|
posture.
|
||||||
|
|
||||||
|
**Closed 2026-07-22:** status tool checks Option A CronJobs + last-success
|
||||||
|
freshness (and still accepts barman ScheduledBackup). RESULT: ok for all four
|
||||||
|
clusters.
|
||||||
|
|
||||||
## Task: Restore drill + RPO/RTO evidence, mark healthy
|
## Task: Restore drill + RPO/RTO evidence, mark healthy
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: RAILIANCE-WP-0015-T05
|
id: RAILIANCE-WP-0015-T05
|
||||||
status: wait
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "59f41267-94a9-45f0-a9d7-3c0cb16689ff"
|
state_hub_task_id: "59f41267-94a9-45f0-a9d7-3c0cb16689ff"
|
||||||
```
|
```
|
||||||
|
|
@ -115,3 +124,7 @@ state_hub_task_id: "59f41267-94a9-45f0-a9d7-3c0cb16689ff"
|
||||||
Run an isolated-namespace restore drill from a scheduled artifact for at least
|
Run an isolated-namespace restore drill from a scheduled artifact for at least
|
||||||
one cluster; record RPO 24h / RTO 4h evidence. **Done when:** `make
|
one cluster; record RPO 24h / RTO 4h evidence. **Done when:** `make
|
||||||
cnpg-backup-status` → **healthy** and restore evidence is recorded.
|
cnpg-backup-status` → **healthy** and restore evidence is recorded.
|
||||||
|
|
||||||
|
**Closed 2026-07-22:** decrypted Option A `vergabe_db` age artifact; isolated
|
||||||
|
restore row-gate 195/195; status healthy; evidence in
|
||||||
|
`docs/evidence/cnpg-option-a-backup-20260722.json`.
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue