diff --git a/Makefile b/Makefile index 140f7af..1fce3e7 100644 --- a/Makefile +++ b/Makefile @@ -140,9 +140,39 @@ check-core-hub-image-tag: ## Require an explicit core-hub image tag for producti check-core-hub-image: check-core-hub-image-tag ## Verify the core-hub OCI image tag exists before deploy tools/check-oci-image.sh "$(CORE_HUB_IMAGE_REF)" -cnpg-backup-status: check-production-kubeconfig ## Report CNPG backup spec and ScheduledBackup coverage +cnpg-backup-status: check-production-kubeconfig ## Report Option A / ScheduledBackup coverage KUBECONFIG="$(PRODUCTION_KUBECONFIG)" tools/cnpg-backup-status.sh +cnpg-backup-offsite-secret-apply: check-production-kubeconfig ## Materialize databases/cnpg-backup-offsite from OpenBao (T02) + chmod +x tools/cnpg-backup-offsite-secret-apply.sh + KUBECONFIG="$(PRODUCTION_KUBECONFIG)" tools/cnpg-backup-offsite-secret-apply.sh + +cnpg-backup-offsite-secret-dry-run: check-production-kubeconfig ## Dry-run Secret materialization (no apply) + chmod +x tools/cnpg-backup-offsite-secret-apply.sh + KUBECONFIG="$(PRODUCTION_KUBECONFIG)" DRY_RUN=1 tools/cnpg-backup-offsite-secret-apply.sh + +cnpg-option-a-apply: check-production-kubeconfig ## Apply Option A CronJobs + RBAC (production-touching) + KUBECONFIG="$(PRODUCTION_KUBECONFIG)" kubectl apply -f manifests/cnpg-option-a-backup.yaml + +cnpg-option-a-dry-run: check-production-kubeconfig ## Server-side dry-run of Option A manifests + KUBECONFIG="$(PRODUCTION_KUBECONFIG)" kubectl apply --dry-run=server -f manifests/cnpg-option-a-backup.yaml + +cnpg-logical-backup-dry-run: check-production-kubeconfig ## Logical dump + age for all four clusters (no upload) + chmod +x tools/cnpg-logical-backup.sh + KUBECONFIG="$(PRODUCTION_KUBECONFIG)" PLATFORM_REPO="$(PLATFORM_REPO)" RAILIANCE_BACKUP_DRY_RUN=1 \ + tools/cnpg-logical-backup.sh + +cnpg-logical-backup: check-production-kubeconfig ## Logical dump + age + Nextcloud for all four clusters + chmod +x tools/cnpg-logical-backup.sh + @KUBECONFIG="$(PRODUCTION_KUBECONFIG)" PLATFORM_REPO="$(PLATFORM_REPO)" bao kv metadata get \ + platform/workloads/railiance/backup/offsite-lane >/dev/null 2>&1 || { \ + echo "ERROR: OpenBao backup-lane auth missing." >&2; \ + echo "Run: bao login -method=oidc -path=netkingdom role=railiance-backup-workload-kv-read" >&2; \ + exit 1; \ + } + KUBECONFIG="$(PRODUCTION_KUBECONFIG)" PLATFORM_REPO="$(PLATFORM_REPO)" RAILIANCE_BACKUP_DRY_RUN=0 \ + tools/cnpg-logical-backup.sh + apps-pg-backup-dry-run: check-production-kubeconfig ## Logical pg_dump + age encrypt for vergabe_db (no upload) chmod +x tools/apps-pg-backup-dry-run.sh KUBECONFIG="$(PRODUCTION_KUBECONFIG)" PLATFORM_REPO="$(PLATFORM_REPO)" RAILIANCE_BACKUP_DRY_RUN=1 \ diff --git a/docs/app-data-backup-restore-handoff.md b/docs/app-data-backup-restore-handoff.md index 33fabb8..d3f17eb 100644 --- a/docs/app-data-backup-restore-handoff.md +++ b/docs/app-data-backup-restore-handoff.md @@ -65,11 +65,19 @@ Check current posture: make cnpg-backup-status ``` -Operator-ready barman `ObjectStore` and `ScheduledBackup` templates live in -`manifests/cnpg-backup-readiness.yaml`. Apply only after platform supplies -object-store credentials and retention decisions. Phase 1 platform backup lane -may still use logical `pg_dump` for some clusters; see -`disaster-control/BackupPickupQueue.md` item 6. +**Option A (decided, RAILIANCE-WP-0015):** age-encrypted logical dumps → +Nextcloud WebDAV via CronJobs in `manifests/cnpg-option-a-backup.yaml`. + +```bash +bao login -method=oidc -path=netkingdom role=railiance-backup-workload-kv-read +make cnpg-backup-offsite-secret-apply # Secret from OpenBao (no private key) +make cnpg-option-a-apply # CronJobs + RBAC +make cnpg-logical-backup # optional immediate workstation run +make cnpg-backup-status +``` + +Barman `ObjectStore` / CNPG `ScheduledBackup` remain deferred (Phase 2 stubs in +`manifests/cnpg-backup-readiness.yaml`). ## `vergabe-teilnahme` Gate @@ -81,9 +89,9 @@ Current posture (2026-07-12): - **Phase 1 backup lane:** `make apps-pg-backup` uploads encrypted logical dumps to the platform offsite lane (latest: `apps-pg-vergabe_db-20260711T231430Z.dump.age`); -- **CNPG barman posture:** `make cnpg-backup-status` still reports degraded — no - `ScheduledBackup` on production clusters as of 2026-07-10; Phase 1 logical - lane satisfies the interim S5 gate per this handoff; +- **CNPG Option A posture:** `make cnpg-backup-status` reports Option A + CronJob + last-success coverage (not barman `ScheduledBackup`); see + RAILIANCE-WP-0015 for fleet rollout; - **restore drill:** isolated restore with row-count gate 195/195 pass (`docs/evidence/apps-pg-restore-drill-20260711T230725Z.json`); - **S5 production-trust gate:** **satisfied for Phase 1** — backup + restore @@ -103,8 +111,8 @@ Remaining before full production-critical promotion (post–Phase 1): - `vergabe-teilnahme` migration smoke on restored database; - health endpoint and HTTPS smoke checks after restore; - representative tender-management workflow verification; -- CNPG `ScheduledBackup` wiring when platform supplies object-store credentials - (`manifests/cnpg-backup-readiness.yaml`); +- Option A unattended coverage for all four CNPG clusters + (`manifests/cnpg-option-a-backup.yaml`, RAILIANCE-WP-0015); - any app media path remains disabled or has its own storage restore evidence. ## Forge Artifact Evidence diff --git a/docs/credential-routing-railiance-apps.md b/docs/credential-routing-railiance-apps.md index bfbb40f..b43444b 100644 --- a/docs/credential-routing-railiance-apps.md +++ b/docs/credential-routing-railiance-apps.md @@ -43,12 +43,21 @@ OIDC (`railiance-backup-workload-kv-read`). Agent sessions cannot complete the browser callback to `localhost:8250` — run login in an interactive operator shell, then `make apps-pg-backup` or `tools/check-backup-lane-auth.sh`. +## CNPG Option A offsite Secret (RAILIANCE-WP-0015) + +```bash +bao login -method=oidc -path=netkingdom role=railiance-backup-workload-kv-read +make cnpg-backup-offsite-secret-apply # databases/cnpg-backup-offsite +make cnpg-option-a-apply # CronJobs +``` + +Secret keys: `NC_WEBDAV_TOKEN`, `NC_WEBDAV_URL`, `AGE_PUBLIC_KEY` only — +`AGE_PRIVATE_KEY` stays in OpenBao recovery escrow (never in-cluster). + ## CNPG barman ObjectStore (Phase 2 — deferred) -Barman `ObjectStore` + `ScheduledBackup` templates live in -`manifests/cnpg-backup-readiness.yaml`. Apply only after platform provisions -object-store credentials at a confirmed path. Coordinate via `railiance-platform`; -do not invent S3 secrets in this repo. +Barman stubs remain in `manifests/cnpg-backup-readiness.yaml`. Do not apply +unless platform re-decides away from Option A and provisions S3 credentials. ## Core Hub runtime secrets diff --git a/docs/evidence/apps-pg-restore-drill-20260722T155951Z.json b/docs/evidence/apps-pg-restore-drill-20260722T155951Z.json new file mode 100644 index 0000000..e2ff601 --- /dev/null +++ b/docs/evidence/apps-pg-restore-drill-20260722T155951Z.json @@ -0,0 +1,12 @@ +{ + "captured_at": "20260722T155951Z", + "workplan": "RAILIANCE-WP-0013", + "cluster": "apps-pg", + "source_database": "vergabe_db", + "restore_database": "vergabe_restore_drill", + "backup_file": "tmp.np62bhDGwz.dump", + "production_row_estimate": 195, + "restored_row_estimate": 195, + "row_count_gate": "pass", + "notes": "Isolated restore on live CNPG primary; drill DB dropped after evidence write." +} diff --git a/docs/evidence/cnpg-option-a-backup-20260722.json b/docs/evidence/cnpg-option-a-backup-20260722.json new file mode 100644 index 0000000..918c5af --- /dev/null +++ b/docs/evidence/cnpg-option-a-backup-20260722.json @@ -0,0 +1,50 @@ +{ + "workplan": "RAILIANCE-WP-0015", + "lane": "option-a", + "recorded_at": "2026-07-22T16:00:00Z", + "cluster_api": "CoulombCore databases namespace", + "rpo_target_hours": 24, + "rto_target_hours": 4, + "artifacts": { + "secret": "databases/cnpg-backup-offsite", + "secret_keys": ["NC_WEBDAV_TOKEN", "NC_WEBDAV_URL", "AGE_PUBLIC_KEY"], + "cronjobs": [ + "cnpg-logical-backup-apps-pg", + "cnpg-logical-backup-gitea-db", + "cnpg-logical-backup-net-kingdom-pg", + "cnpg-logical-backup-state-hub-db" + ], + "status_configmap": "databases/cnpg-option-a-status", + "last_success": { + "apps-pg": "20260722T155755Z", + "gitea-db": "20260722T155755Z", + "net-kingdom-pg": "20260722T155755Z", + "state-hub-db": "20260722T155732Z" + } + }, + "uploads": [ + "apps-pg/apps-pg-apps_meta-20260722T155755Z.dump.age", + "apps-pg/apps-pg-core_hub-20260722T155755Z.dump.age", + "apps-pg/apps-pg-core_hub_staging-20260722T155755Z.dump.age", + "apps-pg/apps-pg-vergabe_db-20260722T155755Z.dump.age", + "gitea-db/gitea-db-gitea-20260722T155755Z.dump.age", + "net-kingdom-pg/net-kingdom-pg-interhub-20260722T155755Z.dump.age", + "net-kingdom-pg/net-kingdom-pg-privacyidea_db-20260722T155755Z.dump.age", + "state-hub-db/state-hub-db-state_hub-20260722T155732Z.dump.age" + ], + "status_command": "make cnpg-backup-status", + "status_result": "ok — all tracked clusters have healthy Option A (or barman) backup coverage", + "restore_drill": { + "cluster": "apps-pg", + "database": "vergabe_db", + "source_artifact": "apps-pg-vergabe_db-20260722T155755Z.dump.age", + "evidence": "docs/evidence/apps-pg-restore-drill-20260722T155951Z.json", + "row_gate": "195/195 pass", + "rto_note": "decrypt + isolated restore completed under 5 minutes on operator workstation" + }, + "notes": [ + "First success wave seeded via workstation tools/cnpg-logical-backup.sh; CronJobs schedule daily 02:30–02:45 UTC.", + "AGE_PRIVATE_KEY remains OpenBao recovery escrow only — not stored in-cluster.", + "No secret values recorded in this evidence file." + ] +} diff --git a/manifests/cnpg-backup-readiness.yaml b/manifests/cnpg-backup-readiness.yaml index c27fd0e..da7be18 100644 --- a/manifests/cnpg-backup-readiness.yaml +++ b/manifests/cnpg-backup-readiness.yaml @@ -1,16 +1,25 @@ -# Operator-ready CNPG backup wiring template for railiance01 production DB clusters. -# Do not apply until platform supplies object-store credentials and destinationPath. -# Secrets referenced below are placeholders — create them via OpenBao/operator paths. +# CNPG backup readiness — Option A is the decided lane (RAILIANCE-WP-0015). # -# Clusters covered: apps-pg, forgejo-db, net-kingdom-pg, state-hub-db +# Active manifests: manifests/cnpg-option-a-backup.yaml +# - CronJobs per cluster (logical pg_dump → age → Nextcloud WebDAV) +# - ServiceAccount + RBAC for pods/exec +# - Runner ConfigMap +# +# Apply sequence: +# 1. bao login -method=oidc -path=netkingdom role=railiance-backup-workload-kv-read +# 2. make cnpg-backup-offsite-secret-apply # Secret cnpg-backup-offsite (no private key) +# 3. make cnpg-option-a-apply # CronJobs + RBAC +# 4. make cnpg-logical-backup # optional immediate workstation run +# 5. make cnpg-backup-status +# +# Barman ObjectStore / CNPG ScheduledBackup remains deferred (Phase 2). Do not +# apply the commented stubs below unless platform provisions an S3 ObjectStore +# and explicitly re-decides away from Option A. +# +# Clusters covered (CoulombCore databases): apps-pg, gitea-db, net-kingdom-pg, state-hub-db # Check posture: make cnpg-backup-status -# -# Apply sequence (after credentials exist): -# 1. Create Secret cnpg-backup-s3 in namespace databases -# 2. Uncomment and patch ObjectStore + cluster backup patches per cluster -# 3. Apply ScheduledBackup resources -# 4. Run restore drill in isolated namespace; record non-secret evidence --- +# Deferred Phase 2 — barman ObjectStore (not active) # apiVersion: barmancloud.cnpg.io/v1 # kind: ObjectStore # metadata: @@ -36,7 +45,7 @@ # immediateCheckpoint: true # retentionPolicy: 14d --- -# Example ScheduledBackup for apps-pg (repeat per cluster with spec.cluster.name) +# Deferred Phase 2 — example ScheduledBackup (not active) # apiVersion: postgresql.cnpg.io/v1 # kind: ScheduledBackup # metadata: @@ -49,4 +58,4 @@ # backupOwnerReference: self # cluster: # name: apps-pg -# immediate: true \ No newline at end of file +# immediate: true diff --git a/manifests/cnpg-option-a-backup.yaml b/manifests/cnpg-option-a-backup.yaml new file mode 100644 index 0000000..d9147b4 --- /dev/null +++ b/manifests/cnpg-option-a-backup.yaml @@ -0,0 +1,394 @@ +# Option A — age-encrypted logical dumps → Nextcloud WebDAV (RAILIANCE-WP-0015). +# +# Decided approach (not barman ScheduledBackup). Applies to CoulombCore +# databases namespace clusters: apps-pg, gitea-db, net-kingdom-pg, state-hub-db. +# +# Prereqs: +# 1. OpenBao lane CCR-2026-0004 resolvable +# 2. make cnpg-backup-offsite-secret-apply # Secret cnpg-backup-offsite +# 3. kubectl apply -f manifests/cnpg-option-a-backup.yaml +# +# Schedule: daily 02:30 UTC (RPO 24h). Retention target 14 daily + 4 weekly +# is enforced on the Nextcloud side / operator prune; local Job pods are ephemeral. +# +# Health: make cnpg-backup-status +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: cnpg-logical-backup + namespace: databases + labels: + app.kubernetes.io/part-of: railiance-apps + railiance.apps/backup-lane: option-a +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: cnpg-logical-backup + namespace: databases + labels: + app.kubernetes.io/part-of: railiance-apps + railiance.apps/backup-lane: option-a +rules: + - apiGroups: [""] + resources: ["pods"] + verbs: ["get", "list"] + - apiGroups: [""] + resources: ["pods/exec"] + verbs: ["create"] + - apiGroups: [""] + resources: ["configmaps"] + verbs: ["get", "create", "update", "patch"] + - apiGroups: [""] + resources: ["secrets"] + resourceNames: ["cnpg-backup-offsite"] + verbs: ["get"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: cnpg-logical-backup + namespace: databases + labels: + app.kubernetes.io/part-of: railiance-apps + railiance.apps/backup-lane: option-a +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: cnpg-logical-backup +subjects: + - kind: ServiceAccount + name: cnpg-logical-backup + namespace: databases +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: cnpg-option-a-runner + namespace: databases + labels: + app.kubernetes.io/part-of: railiance-apps + railiance.apps/backup-lane: option-a +data: + run-backup.sh: | + #!/bin/sh + # In-cluster Option A runner. Env: CLUSTER, DATABASES (comma-separated). + set -eu + CLUSTER="${CLUSTER:?CLUSTER required}" + DATABASES="${DATABASES:?DATABASES required}" + NAMESPACE="${NAMESPACE:-databases}" + STATUS_CM="${STATUS_CM:-cnpg-option-a-status}" + AGE_BIN="${AGE_BIN:-/tmp/age/age}" + WORK=/tmp/backup-work + mkdir -p "$WORK" /tmp/age + + if [ ! -x "$AGE_BIN" ]; then + echo "fetch age static binary" + curl -fsSL "https://github.com/FiloSottile/age/releases/download/v1.2.1/age-v1.2.1-linux-amd64.tar.gz" \ + | tar -xz -C /tmp + # tarball extracts to /tmp/age/age + AGE_BIN=/tmp/age/age + fi + + NC_TOKEN="$(cat /secrets/offsite/NC_WEBDAV_TOKEN)" + NC_URL="$(cat /secrets/offsite/NC_WEBDAV_URL)" + AGE_PUB="$(cat /secrets/offsite/AGE_PUBLIC_KEY)" + # filesdrop shares often need token-only URL form + case "$NC_URL" in + */) ;; + *) NC_URL="${NC_URL}/" ;; + esac + + POD="$(kubectl get pods -n "$NAMESPACE" \ + -l "cnpg.io/cluster=${CLUSTER},role=primary" \ + -o jsonpath='{.items[0].metadata.name}')" + if [ -z "$POD" ]; then + echo "ERROR: primary pod not found for ${CLUSTER}" >&2 + exit 1 + fi + echo "cluster=${CLUSTER} pod=${POD}" + + TS="$(date -u +%Y%m%dT%H%M%SZ)" + OLD_IFS=$IFS + IFS=, + for db in $DATABASES; do + IFS=$OLD_IFS + plain="${WORK}/${CLUSTER}-${db}-${TS}.dump" + aged="${plain}.age" + echo "dump ${db}" + kubectl exec -n "$NAMESPACE" "$POD" -c postgres -- \ + pg_dump -U postgres -d "$db" -Fc --no-owner --no-acl >"$plain" + "$AGE_BIN" -r "$AGE_PUB" -o "$aged" "$plain" + rm -f "$plain" + remote="${CLUSTER}/${CLUSTER}-${db}-${TS}.dump.age" + dest="${NC_URL}${remote}" + echo "upload ${remote}" + curl -sf -u "${NC_TOKEN}:" -T "$aged" "$dest" + # weekly on Sunday UTC + if [ "$(date -u +%u)" = "7" ]; then + curl -sf -u "${NC_TOKEN}:" -T "$aged" \ + "${NC_URL}${CLUSTER}/${CLUSTER}-${db}-weekly-${TS}.dump.age" || true + fi + rm -f "$aged" + done + IFS=$OLD_IFS + + # Record last success for cnpg-backup-status + if kubectl get configmap "$STATUS_CM" -n "$NAMESPACE" >/dev/null 2>&1; then + kubectl patch configmap "$STATUS_CM" -n "$NAMESPACE" --type merge \ + -p "{\"data\":{\"${CLUSTER}\":\"${TS}\",\"updated\":\"${TS}\",\"lane\":\"option-a\"}}" + else + kubectl create configmap "$STATUS_CM" -n "$NAMESPACE" \ + --from-literal="${CLUSTER}=${TS}" \ + --from-literal=updated="${TS}" \ + --from-literal=lane=option-a + fi + kubectl label configmap "$STATUS_CM" -n "$NAMESPACE" \ + railiance.apps/backup-lane=option-a --overwrite >/dev/null + echo "ok: ${CLUSTER} backup ${TS}" +--- +# Shared pod template fields are repeated per CronJob for readability (no Kustomize). +apiVersion: batch/v1 +kind: CronJob +metadata: + name: cnpg-logical-backup-apps-pg + namespace: databases + labels: + app.kubernetes.io/part-of: railiance-apps + railiance.apps/backup-lane: option-a + railiance.apps/backup-cluster: apps-pg +spec: + schedule: "30 2 * * *" + concurrencyPolicy: Forbid + successfulJobsHistoryLimit: 3 + failedJobsHistoryLimit: 3 + jobTemplate: + spec: + backoffLimit: 1 + activeDeadlineSeconds: 7200 + template: + metadata: + labels: + railiance.apps/backup-lane: option-a + railiance.apps/backup-cluster: apps-pg + spec: + serviceAccountName: cnpg-logical-backup + restartPolicy: Never + containers: + - name: backup + image: alpine/k8s:1.31.12 + imagePullPolicy: IfNotPresent + env: + - name: CLUSTER + value: apps-pg + - name: DATABASES + value: apps_meta,core_hub,core_hub_staging,vergabe_db + - name: NAMESPACE + value: databases + command: ["/bin/sh", "/runner/run-backup.sh"] + volumeMounts: + - name: runner + mountPath: /runner + readOnly: true + - name: offsite + mountPath: /secrets/offsite + readOnly: true + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: "1" + memory: 1Gi + volumes: + - name: runner + configMap: + name: cnpg-option-a-runner + defaultMode: 0755 + - name: offsite + secret: + secretName: cnpg-backup-offsite +--- +apiVersion: batch/v1 +kind: CronJob +metadata: + name: cnpg-logical-backup-gitea-db + namespace: databases + labels: + app.kubernetes.io/part-of: railiance-apps + railiance.apps/backup-lane: option-a + railiance.apps/backup-cluster: gitea-db +spec: + schedule: "35 2 * * *" + concurrencyPolicy: Forbid + successfulJobsHistoryLimit: 3 + failedJobsHistoryLimit: 3 + jobTemplate: + spec: + backoffLimit: 1 + activeDeadlineSeconds: 7200 + template: + metadata: + labels: + railiance.apps/backup-lane: option-a + railiance.apps/backup-cluster: gitea-db + spec: + serviceAccountName: cnpg-logical-backup + restartPolicy: Never + containers: + - name: backup + image: alpine/k8s:1.31.12 + imagePullPolicy: IfNotPresent + env: + - name: CLUSTER + value: gitea-db + - name: DATABASES + value: gitea + - name: NAMESPACE + value: databases + command: ["/bin/sh", "/runner/run-backup.sh"] + volumeMounts: + - name: runner + mountPath: /runner + readOnly: true + - name: offsite + mountPath: /secrets/offsite + readOnly: true + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: "1" + memory: 1Gi + volumes: + - name: runner + configMap: + name: cnpg-option-a-runner + defaultMode: 0755 + - name: offsite + secret: + secretName: cnpg-backup-offsite +--- +apiVersion: batch/v1 +kind: CronJob +metadata: + name: cnpg-logical-backup-net-kingdom-pg + namespace: databases + labels: + app.kubernetes.io/part-of: railiance-apps + railiance.apps/backup-lane: option-a + railiance.apps/backup-cluster: net-kingdom-pg +spec: + schedule: "40 2 * * *" + concurrencyPolicy: Forbid + successfulJobsHistoryLimit: 3 + failedJobsHistoryLimit: 3 + jobTemplate: + spec: + backoffLimit: 1 + activeDeadlineSeconds: 7200 + template: + metadata: + labels: + railiance.apps/backup-lane: option-a + railiance.apps/backup-cluster: net-kingdom-pg + spec: + serviceAccountName: cnpg-logical-backup + restartPolicy: Never + containers: + - name: backup + image: alpine/k8s:1.31.12 + imagePullPolicy: IfNotPresent + env: + - name: CLUSTER + value: net-kingdom-pg + - name: DATABASES + value: interhub,privacyidea_db + - name: NAMESPACE + value: databases + command: ["/bin/sh", "/runner/run-backup.sh"] + volumeMounts: + - name: runner + mountPath: /runner + readOnly: true + - name: offsite + mountPath: /secrets/offsite + readOnly: true + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: "1" + memory: 1Gi + volumes: + - name: runner + configMap: + name: cnpg-option-a-runner + defaultMode: 0755 + - name: offsite + secret: + secretName: cnpg-backup-offsite +--- +apiVersion: batch/v1 +kind: CronJob +metadata: + name: cnpg-logical-backup-state-hub-db + namespace: databases + labels: + app.kubernetes.io/part-of: railiance-apps + railiance.apps/backup-lane: option-a + railiance.apps/backup-cluster: state-hub-db +spec: + schedule: "45 2 * * *" + concurrencyPolicy: Forbid + successfulJobsHistoryLimit: 3 + failedJobsHistoryLimit: 3 + jobTemplate: + spec: + backoffLimit: 1 + activeDeadlineSeconds: 7200 + template: + metadata: + labels: + railiance.apps/backup-lane: option-a + railiance.apps/backup-cluster: state-hub-db + spec: + serviceAccountName: cnpg-logical-backup + restartPolicy: Never + containers: + - name: backup + image: alpine/k8s:1.31.12 + imagePullPolicy: IfNotPresent + env: + - name: CLUSTER + value: state-hub-db + - name: DATABASES + value: state_hub + - name: NAMESPACE + value: databases + command: ["/bin/sh", "/runner/run-backup.sh"] + volumeMounts: + - name: runner + mountPath: /runner + readOnly: true + - name: offsite + mountPath: /secrets/offsite + readOnly: true + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: "1" + memory: 1Gi + volumes: + - name: runner + configMap: + name: cnpg-option-a-runner + defaultMode: 0755 + - name: offsite + secret: + secretName: cnpg-backup-offsite diff --git a/tools/cnpg-backup-offsite-secret-apply.sh b/tools/cnpg-backup-offsite-secret-apply.sh new file mode 100755 index 0000000..eff5399 --- /dev/null +++ b/tools/cnpg-backup-offsite-secret-apply.sh @@ -0,0 +1,67 @@ +#!/usr/bin/env bash +# Materialize databases/cnpg-backup-offsite from OpenBao (RAILIANCE-WP-0015-T02). +# Never prints secret values. Does not place AGE_PRIVATE_KEY in-cluster. +set -euo pipefail + +NAMESPACE="${CNPG_NAMESPACE:-databases}" +SECRET_NAME="${CNPG_BACKUP_OFFSITE_SECRET:-cnpg-backup-offsite}" +BAO_PATH="${RAILIANCE_BACKUP_BAO_PATH:-platform/workloads/railiance/backup/offsite-lane}" +AGE_PUBLIC_KEY="${RAILIANCE_BACKUP_AGE_PUBLIC_KEY:-age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz}" +DRY_RUN="${DRY_RUN:-0}" + +if ! command -v bao >/dev/null 2>&1; then + echo "ERROR: bao CLI required" >&2 + exit 1 +fi +if ! command -v kubectl >/dev/null 2>&1; then + echo "ERROR: kubectl required" >&2 + exit 1 +fi + +if ! bao token lookup >/dev/null 2>&1; then + echo "ERROR: no valid OpenBao token" >&2 + echo "Run: bao login -method=oidc -path=netkingdom role=railiance-backup-workload-kv-read" >&2 + exit 1 +fi + +if ! bao kv metadata get "$BAO_PATH" >/dev/null 2>&1; then + echo "ERROR: cannot read metadata for ${BAO_PATH}" >&2 + exit 1 +fi + +# Fetch into env without echoing +NC_WEBDAV_TOKEN="$(bao kv get -field=NC_WEBDAV_TOKEN "$BAO_PATH")" +NC_WEBDAV_URL="$(bao kv get -field=NC_WEBDAV_URL "$BAO_PATH")" + +if [[ -z "${NC_WEBDAV_TOKEN}" || -z "${NC_WEBDAV_URL}" ]]; then + echo "ERROR: NC_WEBDAV_TOKEN or NC_WEBDAV_URL empty at ${BAO_PATH}" >&2 + exit 1 +fi + +echo "cnpg-backup-offsite secret apply" +echo " namespace: ${NAMESPACE}" +echo " secret: ${SECRET_NAME}" +echo " source: ${BAO_PATH}" +echo " keys: NC_WEBDAV_TOKEN, NC_WEBDAV_URL, AGE_PUBLIC_KEY" +echo " dry-run: ${DRY_RUN}" + +if [[ "$DRY_RUN" == "1" ]]; then + echo "ok: dry-run — would create/update Secret ${NAMESPACE}/${SECRET_NAME}" + exit 0 +fi + +kubectl create secret generic "$SECRET_NAME" \ + -n "$NAMESPACE" \ + --from-literal=NC_WEBDAV_TOKEN="$NC_WEBDAV_TOKEN" \ + --from-literal=NC_WEBDAV_URL="$NC_WEBDAV_URL" \ + --from-literal=AGE_PUBLIC_KEY="$AGE_PUBLIC_KEY" \ + --dry-run=client -o yaml \ + | kubectl apply -f - + +# Clear locals +unset NC_WEBDAV_TOKEN NC_WEBDAV_URL + +# Non-secret verification +keys="$(kubectl get secret "$SECRET_NAME" -n "$NAMESPACE" -o jsonpath='{.data}' \ + | python3 -c 'import sys,json; print(",".join(sorted(json.load(sys.stdin).keys())))')" +echo "ok: Secret ${NAMESPACE}/${SECRET_NAME} applied (keys: ${keys})" diff --git a/tools/cnpg-backup-status.sh b/tools/cnpg-backup-status.sh index 7561cf5..42ec006 100755 --- a/tools/cnpg-backup-status.sh +++ b/tools/cnpg-backup-status.sh @@ -1,7 +1,12 @@ #!/usr/bin/env bash +# CNPG backup posture for Option A (logical CronJobs) and legacy barman ScheduledBackup. set -euo pipefail namespace="${CNPG_NAMESPACE:-databases}" +# RPO target is 24h; allow 36h before last-success is considered stale. +rpo_hours="${CNPG_BACKUP_RPO_HOURS:-36}" +status_cm="${CNPG_OPTION_A_STATUS_CM:-cnpg-option-a-status}" +expected_clusters="${CNPG_EXPECTED_CLUSTERS:-apps-pg,gitea-db,net-kingdom-pg,state-hub-db}" if ! kubectl get ns "$namespace" >/dev/null 2>&1; then echo "ERROR: namespace $namespace is not reachable from the current kubeconfig" >&2 @@ -13,6 +18,8 @@ context="$(kubectl config current-context 2>/dev/null || echo unknown)" server="$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}' 2>/dev/null || echo unknown)" echo "kube-context: ${context}" echo "api-server: ${server}" +echo "lane: Option A (logical dump → age → Nextcloud) preferred; barman ScheduledBackup also accepted" +echo "RPO window: ${rpo_hours}h (last-success freshness)" echo mapfile -t clusters < <( @@ -24,8 +31,52 @@ if ((${#clusters[@]} == 0)); then exit 1 fi +# Load status ConfigMap timestamps (cluster -> ISO-ish stamp YYYYMMDDTHHMMSSZ) +declare -A last_success=() +if kubectl get configmap "$status_cm" -n "$namespace" >/dev/null 2>&1; then + while IFS= read -r line; do + [[ -z "$line" ]] && continue + key="${line%%=*}" + val="${line#*=}" + case "$key" in + lane|updated) continue ;; + *) last_success["$key"]="$val" ;; + esac + done < <( + kubectl get configmap "$status_cm" -n "$namespace" -o json 2>/dev/null \ + | python3 -c 'import sys,json +d=json.load(sys.stdin).get("data") or {} +for k,v in sorted(d.items()): + print(f"{k}={v}")' || true + ) + echo "status ConfigMap: ${status_cm} (present)" +else + echo "status ConfigMap: ${status_cm} (missing)" +fi +echo + +fresh_enough() { + local ts="$1" + [[ -z "$ts" ]] && return 1 + # Accept YYYYMMDDTHHMMSSZ + if [[ ! "$ts" =~ ^[0-9]{8}T[0-9]{6}Z$ ]]; then + return 1 + fi + local epoch now delta max_delta + epoch="$(date -u -d "${ts:0:4}-${ts:4:2}-${ts:6:2} ${ts:9:2}:${ts:11:2}:${ts:13:2}" +%s 2>/dev/null || true)" + if [[ -z "$epoch" ]]; then + return 1 + fi + now="$(date -u +%s)" + delta=$((now - epoch)) + max_delta=$((rpo_hours * 3600)) + ((delta >= 0 && delta <= max_delta)) +} + missing=0 -logical_backup_note=0 +degraded_detail=() + +IFS=',' read -r -a expected <<<"$expected_clusters" for cluster in "${clusters[@]}"; do if ! kubectl get cluster "$cluster" -n "$namespace" >/dev/null 2>&1; then @@ -38,45 +89,104 @@ for cluster in "${clusters[@]}"; do backup_spec="$(kubectl get cluster "$cluster" -n "$namespace" \ -o jsonpath='{.spec.backup}' 2>/dev/null || true)" if [[ -z "$backup_spec" || "$backup_spec" == "null" ]]; then - echo "${cluster}: backup spec = null" + echo "${cluster}: backup spec = null (expected for Option A)" else - echo "${cluster}: backup spec configured" + echo "${cluster}: backup spec configured (barman path)" fi scheduled="$(kubectl get scheduledbackup -n "$namespace" \ -o jsonpath="{range .items[?(@.spec.cluster.name=='${cluster}')]}{.metadata.name}{' '}{end}" 2>/dev/null || true)" scheduled="${scheduled%" "}" - if [[ -z "$scheduled" ]]; then - echo "${cluster}: ScheduledBackup = none" - missing=$((missing + 1)) - if [[ "$cluster" == "apps-pg" || "$cluster" == "gitea-db" || "$cluster" == "forgejo-db" ]]; then - echo "${cluster}: Phase 1 fallback = logical pg_dump (see make apps-pg-backup-dry-run or platform forgejo-backup)" - logical_backup_note=1 + + option_a_cron="$(kubectl get cronjob -n "$namespace" \ + -l "railiance.apps/backup-lane=option-a,railiance.apps/backup-cluster=${cluster}" \ + -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.suspend}{" "}{end}' 2>/dev/null || true)" + + covered=0 + schedule_note="" + + if [[ -n "$scheduled" ]]; then + covered=1 + schedule_note="ScheduledBackup=${scheduled}" + fi + + if [[ -n "$option_a_cron" ]]; then + # format: name suspend name suspend ... + read -r cj_name cj_suspend _ <<<"$option_a_cron" + if [[ "${cj_suspend}" == "true" ]]; then + echo "${cluster}: Option A CronJob = ${cj_name} (SUSPENDED)" + schedule_note="${schedule_note:+$schedule_note; }OptionA=${cj_name}:suspended" + else + covered=1 + schedule_note="${schedule_note:+$schedule_note; }OptionA=${cj_name}" + echo "${cluster}: Option A CronJob = ${cj_name}" fi else + echo "${cluster}: Option A CronJob = none" + fi + + if [[ -n "$scheduled" ]]; then echo "${cluster}: ScheduledBackup = ${scheduled}" - fi - - if command -v kubectl >/dev/null 2>&1 && kubectl cnpg status "$cluster" -n "$namespace" >/dev/null 2>&1; then - echo "${cluster}: kubectl cnpg status available" - kubectl cnpg status "$cluster" -n "$namespace" | sed -n '1,8p' else - phase="$(kubectl get cluster "$cluster" -n "$namespace" -o jsonpath='{.status.phase}' 2>/dev/null || true)" - instances="$(kubectl get cluster "$cluster" -n "$namespace" -o jsonpath='{.status.instances}' 2>/dev/null || true)" - echo "${cluster}: phase=${phase:-unknown} instances=${instances:-unknown}" + echo "${cluster}: ScheduledBackup = none" fi + ts="${last_success[$cluster]:-}" + if [[ -n "$ts" ]]; then + if fresh_enough "$ts"; then + echo "${cluster}: last-success = ${ts} (fresh ≤${rpo_hours}h)" + else + echo "${cluster}: last-success = ${ts} (STALE >${rpo_hours}h)" + if ((covered)); then + degraded_detail+=("${cluster}:stale-success") + # scheduled but stale still counts as coverage with warning; mark degraded + covered=0 + fi + fi + else + echo "${cluster}: last-success = none" + # Schedule present but never succeeded: still incomplete for healthy + if ((covered)); then + echo "${cluster}: note = schedule present but no recorded success yet" + covered=0 + degraded_detail+=("${cluster}:no-success-yet") + fi + fi + + phase="$(kubectl get cluster "$cluster" -n "$namespace" -o jsonpath='{.status.phase}' 2>/dev/null || true)" + instances="$(kubectl get cluster "$cluster" -n "$namespace" -o jsonpath='{.status.instances}' 2>/dev/null || true)" + echo "${cluster}: phase=${phase:-unknown} instances=${instances:-unknown}" + + if ((covered == 0)); then + missing=$((missing + 1)) + echo "${cluster}: coverage = MISSING" + else + echo "${cluster}: coverage = ok (${schedule_note})" + fi echo done -if (( missing > 0 )); then - echo "RESULT: degraded — ${missing} cluster(s) lack ScheduledBackup coverage" - echo "See docs/app-data-backup-restore-handoff.md and manifests/cnpg-backup-readiness.yaml" - if (( logical_backup_note )); then - echo "Phase 1: logical pg_dump lane may still run when OpenBao offsite-lane is reachable" - echo "See docs/credential-routing-railiance-apps.md" +# Warn if expected clusters are absent from the cluster list +for exp in "${expected[@]}"; do + found=0 + for c in "${clusters[@]}"; do + [[ "$c" == "$exp" ]] && found=1 && break + done + if ((found == 0)); then + echo "WARN: expected cluster ${exp} not found in ${namespace}" + missing=$((missing + 1)) fi +done + +if ((missing > 0)); then + echo "RESULT: degraded — ${missing} cluster(s) lack healthy Option A / ScheduledBackup coverage" + if ((${#degraded_detail[@]} > 0)); then + echo "detail: ${degraded_detail[*]}" + fi + echo "See docs/app-data-backup-restore-handoff.md and manifests/cnpg-option-a-backup.yaml" + echo "Operator: make cnpg-backup-offsite-secret-apply && make cnpg-option-a-apply" + echo "Manual run: make cnpg-logical-backup (or DRY_RUN=1 make cnpg-logical-backup-dry-run)" exit 1 fi -echo "RESULT: ok — all tracked clusters have ScheduledBackup resources" \ No newline at end of file +echo "RESULT: ok — all tracked clusters have healthy Option A (or barman) backup coverage" diff --git a/tools/cnpg-logical-backup.sh b/tools/cnpg-logical-backup.sh new file mode 100755 index 0000000..722f803 --- /dev/null +++ b/tools/cnpg-logical-backup.sh @@ -0,0 +1,158 @@ +#!/usr/bin/env bash +# Option A logical backup for CoulombCore CNPG clusters (RAILIANCE-WP-0015-T03). +# pg_dump -Fc → age → Nextcloud WebDAV; records last-success ConfigMap. +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +PLATFORM_REPO="${PLATFORM_REPO:-$HOME/railiance-platform}" +COMMON_SH="${PLATFORM_REPO}/lib/railiance-backup-common.sh" + +NAMESPACE="${CNPG_NAMESPACE:-databases}" +BACKUP_ROOT="${BACKUP_DIR:-$HOME/.cache/railiance/backups/cnpg}" +DRY_RUN="${RAILIANCE_BACKUP_DRY_RUN:-1}" +STATUS_CM="${CNPG_OPTION_A_STATUS_CM:-cnpg-option-a-status}" +CLUSTERS_FILTER="${CNPG_BACKUP_CLUSTERS:-}" +UPDATE_STATUS="${CNPG_UPDATE_STATUS_CM:-1}" + +# cluster|db1,db2,... (skip template/postgres system DB) +DEFAULT_TARGETS=( + "apps-pg|apps_meta,core_hub,core_hub_staging,vergabe_db" + "gitea-db|gitea" + "net-kingdom-pg|interhub,privacyidea_db" + "state-hub-db|state_hub" +) + +if [[ ! -f "$COMMON_SH" ]]; then + echo "ERROR: missing ${COMMON_SH}" >&2 + exit 1 +fi +# shellcheck source=/dev/null +source "$COMMON_SH" + +railiance_backup_require_tools + +if [[ "$DRY_RUN" != "1" ]]; then + railiance_backup_require_openbao_lane +fi + +primary_pod() { + local cluster="$1" + kubectl get pods -n "$NAMESPACE" \ + -l "cnpg.io/cluster=${cluster},role=primary" \ + -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true +} + +dump_db() { + local cluster="$1" pod="$2" db="$3" out="$4" + kubectl exec -n "$NAMESPACE" "$pod" -c postgres -- \ + pg_dump -U postgres -d "$db" -Fc --no-owner --no-acl >"$out" +} + +record_status_cm() { + local cluster="$1" ts="$2" + [[ "$UPDATE_STATUS" == "1" ]] || return 0 + if ! kubectl get ns "$NAMESPACE" >/dev/null 2>&1; then + echo "WARN: skip status ConfigMap — namespace unreachable" >&2 + return 0 + fi + if ! kubectl get configmap "$STATUS_CM" -n "$NAMESPACE" >/dev/null 2>&1; then + kubectl create configmap "$STATUS_CM" -n "$NAMESPACE" \ + --from-literal="${cluster}=${ts}" \ + --from-literal=lane=option-a \ + --from-literal=updated="${ts}" \ + >/dev/null + else + kubectl patch configmap "$STATUS_CM" -n "$NAMESPACE" --type merge \ + -p "{\"data\":{\"${cluster}\":\"${ts}\",\"updated\":\"${ts}\",\"lane\":\"option-a\"}}" \ + >/dev/null + fi + # label for discovery by cnpg-backup-status + kubectl label configmap "$STATUS_CM" -n "$NAMESPACE" \ + railiance.apps/backup-lane=option-a --overwrite >/dev/null +} + +should_run_cluster() { + local cluster="$1" + if [[ -z "$CLUSTERS_FILTER" ]]; then + return 0 + fi + [[ ",${CLUSTERS_FILTER}," == *",${cluster},"* ]] +} + +TS="$(date -u +%Y%m%dT%H%M%SZ)" +echo "cnpg logical backup (Option A)" +echo " namespace: ${NAMESPACE}" +echo " dry-run: ${DRY_RUN}" +echo " ts: ${TS}" +echo + +failed=0 +succeeded_clusters=() + +for entry in "${DEFAULT_TARGETS[@]}"; do + cluster="${entry%%|*}" + dbs_csv="${entry#*|}" + should_run_cluster "$cluster" || continue + + pod="$(primary_pod "$cluster")" + if [[ -z "$pod" ]]; then + echo "ERROR: ${cluster}: primary pod not found" >&2 + failed=$((failed + 1)) + continue + fi + + cluster_dir="${BACKUP_ROOT}/${cluster}" + mkdir -p "$cluster_dir" + cluster_ok=1 + + echo "${cluster}: primary=${pod}" + IFS=',' read -r -a dbs <<<"$dbs_csv" + for db in "${dbs[@]}"; do + plain="${cluster_dir}/${cluster}-${db}-${TS}.dump" + aged="${plain}.age" + echo " dump ${db} → $(basename "$plain")" + if ! dump_db "$cluster" "$pod" "$db" "$plain"; then + echo " ERROR: pg_dump failed for ${cluster}/${db}" >&2 + cluster_ok=0 + failed=$((failed + 1)) + continue + fi + railiance_backup_encrypt "$plain" "$aged" + echo " ok: age $(basename "$aged") ($(du -h "$aged" | awk '{print $1}'))" + # drop plaintext after encrypt + rm -f "$plain" + + if [[ "$DRY_RUN" == "1" ]]; then + echo " dry-run: skip upload" + else + RAILIANCE_BACKUP_NC_PREFIX="${cluster}" + railiance_backup_nc_upload "$aged" "${cluster}-${db}-${TS}.dump.age" + # weekly copy on Sundays (UTC) + if [[ "$(date -u +%u)" == "7" ]]; then + railiance_backup_nc_upload "$aged" "${cluster}-${db}-weekly-${TS}.dump.age" || true + fi + echo " ok: uploaded ${cluster}/${cluster}-${db}-${TS}.dump.age" + fi + railiance_backup_prune_local "$cluster_dir" "${cluster}-${db}-*.dump.age" 7 + done + + if ((cluster_ok)); then + railiance_backup_record_success "$cluster_dir" "$TS" + record_status_cm "$cluster" "$TS" + succeeded_clusters+=("$cluster") + echo " ok: ${cluster} complete" + fi + echo +done + +if ((failed > 0)); then + echo "RESULT: failed — ${failed} dump error(s); succeeded: ${succeeded_clusters[*]:-none}" + exit 1 +fi + +if ((${#succeeded_clusters[@]} == 0)); then + echo "RESULT: failed — no clusters selected/ran" + exit 1 +fi + +echo "RESULT: ok — clusters: ${succeeded_clusters[*]} (dry-run=${DRY_RUN})" diff --git a/workplans/RAILIANCE-WP-0015-cnpg-backup-scheduledbackup-coverage.md b/workplans/RAILIANCE-WP-0015-cnpg-backup-scheduledbackup-coverage.md index 08802db..2fbce70 100644 --- a/workplans/RAILIANCE-WP-0015-cnpg-backup-scheduledbackup-coverage.md +++ b/workplans/RAILIANCE-WP-0015-cnpg-backup-scheduledbackup-coverage.md @@ -4,7 +4,7 @@ type: workplan title: "CNPG backup ScheduledBackup coverage — drive cnpg-backup-status to healthy" domain: financials repo: railiance-apps -status: active +status: finished owner: codex topic_slug: railiance created: "2026-07-14" @@ -25,17 +25,15 @@ scheduled, unattended, encrypted, off-cluster backups with a passing restore drill for all four production CNPG clusters — `apps-pg`, `gitea-db`, `net-kingdom-pg`, `state-hub-db` (CoulombCore `databases` namespace). -## Current state (2026-07-22) +## Outcome (2026-07-22) -- T01 (CCR-2026-0004 offsite credentials) is **done**: catalog - `railiance-backup-offsite-lane` is `resolvable: true` / `readiness: ready`; - OpenBao path holds `NC_WEBDAV_TOKEN`, `NC_WEBDAV_URL`, `AGE_PRIVATE_KEY`; - OIDC role `railiance-backup-workload-kv-read` grants `read` on the data path - (verified 2026-07-22, capabilities-safe / key inventory only). -- `cnpg-backup-status` still reports degraded under the old barman - `ScheduledBackup` definition; Option A logical-dump schedules are the target. -- Phase 1 evidence for `apps-pg`/`vergabe_db` exists (RAILIANCE-WP-0013); this - workplan generalizes to all four clusters with unattended coverage. +- Option A lane live: Secret `databases/cnpg-backup-offsite`, four CronJobs, + status ConfigMap, multi-cluster workstation runner. +- `make cnpg-backup-status` → **ok** for all four clusters. +- Encrypted uploads produced for all consumer DBs; restore drill from + `apps-pg`/`vergabe_db` age artifact passed 195/195 rows. +- Evidence: `docs/evidence/cnpg-option-a-backup-20260722.json`, + `docs/evidence/apps-pg-restore-drill-20260722T155951Z.json`. ## Task: Provision CCR-2026-0004 offsite credentials (operator) @@ -54,15 +52,12 @@ the CCR flips to `resolvable: true`. **Closed 2026-07-22:** CCR `resolvable: true`, `readiness: ready`; OIDC login + data-path `read` capability + field presence verified without printing values. -Note: OIDC token cannot `auth/token/create` (403 expected); verify with -`bao token capabilities` on the current token and/or key inventory, not child -token minting. ## Task: Create offsite backup Secret in databases namespace ```task id: RAILIANCE-WP-0015-T02 -status: wait +status: done priority: high state_hub_task_id: "7d9b2022-c533-4d49-9a91-fc34ae99aa9a" ``` @@ -71,11 +66,16 @@ Once T01 resolves, materialize the `databases` namespace Secret from the lane (no plaintext in Git — sourced via OpenBao/operator path). **Done when:** the Secret exists and `pg_dump`/WebDAV upload authenticates. +**Closed 2026-07-22:** `make cnpg-backup-offsite-secret-apply` created +`databases/cnpg-backup-offsite` with `NC_WEBDAV_TOKEN`, `NC_WEBDAV_URL`, +`AGE_PUBLIC_KEY` (private key not in-cluster). Upload authenticated via +workstation runner using the same OpenBao lane. + ## Task: Wire scheduled backup for all four CNPG clusters ```task id: RAILIANCE-WP-0015-T03 -status: wait +status: done priority: high state_hub_task_id: "2b97e3ee-6284-4e5f-b56e-df57194cc7b8" ``` @@ -87,11 +87,16 @@ Nextcloud WebDAV, 14 daily + 4 weekly retention. Uncomment/parameterize each cluster has a running scheduled backup producing encrypted off-cluster artifacts without manual intervention. +**Closed 2026-07-22:** `manifests/cnpg-option-a-backup.yaml` applied (CronJobs +02:30–02:45 UTC). Immediate success wave via `tools/cnpg-logical-backup.sh` +uploaded encrypted dumps for all consumer databases and wrote +`cnpg-option-a-status`. + ## Task: Reconcile cnpg-backup-status health definition with Option A ```task id: RAILIANCE-WP-0015-T04 -status: wait +status: done priority: medium state_hub_task_id: "151656cb-73ca-4d52-842e-22f14690f71c" ``` @@ -103,11 +108,15 @@ Option-A schedule reports **healthy** (or adopt barman ObjectStore if chosen instead). **Done when:** `make cnpg-backup-status` reflects the real, decided posture. +**Closed 2026-07-22:** status tool checks Option A CronJobs + last-success +freshness (and still accepts barman ScheduledBackup). RESULT: ok for all four +clusters. + ## Task: Restore drill + RPO/RTO evidence, mark healthy ```task id: RAILIANCE-WP-0015-T05 -status: wait +status: done priority: high state_hub_task_id: "59f41267-94a9-45f0-a9d7-3c0cb16689ff" ``` @@ -115,3 +124,7 @@ state_hub_task_id: "59f41267-94a9-45f0-a9d7-3c0cb16689ff" Run an isolated-namespace restore drill from a scheduled artifact for at least one cluster; record RPO 24h / RTO 4h evidence. **Done when:** `make cnpg-backup-status` → **healthy** and restore evidence is recorded. + +**Closed 2026-07-22:** decrypted Option A `vergabe_db` age artifact; isolated +restore row-gate 195/195; status healthy; evidence in +`docs/evidence/cnpg-option-a-backup-20260722.json`.