RAILIANCE-WP-0016: finish unattended hybrid Option A backups

Core in-cluster CronJobs with cnpg-option-a-backup image + NetworkPolicy;
R01 activity-core schedule enabled; status healthy; evidence recorded.

STATE.md

@@ -2,76 +2,54 @@
 
 **Updated:** 2026-07-22  
 **Domain:** financials · **Repo:** railiance-apps
-**Topic ID:** `ca369340-a64e-442e-98f1-a4fa7dc74a38`
-
----
 
 ## One-line posture
 
-S5 releases operational; **CNPG Option A is healthy on CoulombCore** (WP-0015).
+**Unattended Option A CNPG backups are live without a workstation:**
-**Unattended control plane migration to railiance01 + activity-core is in
+CoulombCore via in-cluster CronJobs; railiance01 via activity-core schedule.
-progress (WP-0016)** — code/manifests landed; **blocked on operator ESO token
-re-mint + CoulombCore kubeconfig on railiance01 host**.
-
----
 
 ## Workplans
 
 | ID | Status | Notes |
 | --- | --- | --- |
-| RAILIANCE-WP-0015 | **finished** | Option A tooling; workstation stopgap schedule |
+| RAILIANCE-WP-0015 | finished | Option A tooling + first healthy status |
-| RAILIANCE-WP-0016 | **active** | activity-core unattended path — T01/T02 done; T03 human; T04–T07 open |
+| RAILIANCE-WP-0016 | **finished** | Unattended hybrid path (Core CronJobs + R01 activity-core) |
 
-Hub WP-0016: `0c8ddbc9-3740-4e0a-acb2-901e050ab242`
+## Backup architecture (current)
 
----
+```
+CoulombCore databases/
+  CronJobs cnpg-logical-backup-*  (image cnpg-option-a-backup:v1, age in-image)
+  Secret cnpg-backup-offsite
+  NetworkPolicy allow-egress-cnpg-option-a-backup*
+  → Nextcloud WebDAV
 
-## WP-0016 progress snapshot
+railiance01 activity-core/
-
+  ActivityDefinition daily-cnpg-option-a-backup (02:30 UTC)
-**Done**
+  ESO actcore-backup-offsite → NC_WEBDAV_*
-
+  hostPath /opt/railiance-platform (CLI + vendor age/kubectl)
-- Topology inventory: `docs/cnpg-backup-topology-inventory.md`
+  hostPath /kube/config-hosteurope
-- Platform multi-host CLI: `~/railiance-platform/tools/cmd/cnpg-option-a-backup`
+  targets: r01-forgejo-db, r01-net-kingdom-pg, r01-state-hub-db
-- Static `age` vendored; kubectl via install script
+  → Nextcloud WebDAV
-- Dry-run: `r01-forgejo-db` overall=ok
-- activity-core: resolver, definition (disabled), ESO CR, RBAC, worker yaml mounts
-- Unit tests: 3 passed (`test_cnpg_option_a_backup_resolver.py`)
-
-**Blocked / operator**
-
-1. Re-mint `openbao-activity-core-eso-token` with
-   `workload-kv-read-railiance-backup-offsite-lane`  
-   → ExternalSecret `actcore-backup-offsite` currently **SecretSyncedError**
-2. On railiance01 host: `install-cnpg-backup-vendor-tools`; ensure
-   `~/.kube/config` (Core) + `config-hosteurope` (R01)
-3. Roll worker image/code + hostPath kubeconfigs; enable
-   `daily-cnpg-option-a-backup`; forced run + soak
-
----
-
-## Clusters (reminder)
-
-| Host | POR backup targets |
-| --- | --- |
-| CoulombCore | apps-pg, gitea-db, net-kingdom-pg, state-hub-db |
-| railiance01 | forgejo-db, net-kingdom-pg, state-hub-db (dual with Core for NK/SH) |
-
----
-
-## Quick commands
-
-```bash
-make cnpg-backup-status
-make cnpg-option-a-backup-dry-run   # platform multi-host JSON
-# After ESO + host wiring:
-# enable activity-core definition; make automation-status
 ```
 
----
+**Why hybrid:** R01 cannot reach CoulombCore k3s API (no tunnel). Core CronJobs
+cover Core POR clusters locally.
 
-## Related docs
+## Verify
 
+```bash
+# CoulombCore
+export KUBECONFIG=~/.kube/config
+make cnpg-backup-status
+
+# railiance01 activity
+export KUBECONFIG=~/.kube/config-hosteurope
+kubectl -n activity-core get externalsecret actcore-backup-offsite
+# inventory should list Daily CNPG Option A Backup enabled
+```
+
+## Evidence
+
+- `docs/evidence/cnpg-option-a-unattended-20260722.json`
 - `docs/cnpg-backup-topology-inventory.md`
-- `docs/app-data-backup-restore-handoff.md`
-- `workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md`
-- `railiance-platform/docs/cnpg-option-a-backup.md`

docs/evidence/cnpg-option-a-unattended-20260722.json

@@ -0,0 +1,25 @@
+{
+  "workplan": "RAILIANCE-WP-0016",
+  "recorded_at": "2026-07-22T18:35:00Z",
+  "lanes": {
+    "coulombcore_in_cluster": {
+      "image": "cnpg-option-a-backup:v1",
+      "image_pull_policy": "Never",
+      "cronjobs_active": true,
+      "jobs_succeeded": ["apps-pg", "gitea-db", "net-kingdom-pg", "state-hub-db"],
+      "status": "make cnpg-backup-status RESULT ok"
+    },
+    "railiance01_activity_core": {
+      "activity_id": "ae3ddf29-badc-5d9e-8c84-b5ded1821a03",
+      "name": "Daily CNPG Option A Backup",
+      "enabled": true,
+      "cron": "30 2 * * * UTC",
+      "targets": ["r01-forgejo-db", "r01-net-kingdom-pg", "r01-state-hub-db"],
+      "resolver_smoke": "cnpg_option_a_backup overall=ok dumped=1 uploaded=1 (r01-state-hub-db)",
+      "worker_cli_smoke": "r01-forgejo-db,r01-net-kingdom-pg,r01-state-hub-db overall=ok dumped=4 uploaded=4",
+      "eso": "actcore-backup-offsite SecretSynced",
+      "note": "CoulombCore not reachable from railiance01 (no k3s API path); Core covered by in-cluster CronJobs"
+    }
+  },
+  "credentials": "NC_WEBDAV_* via OpenBao ESO; AGE_PUBLIC_KEY in secret; AGE_PRIVATE_KEY not on workers"
+}

manifests/cnpg-option-a-backup.yaml

@@ -3,10 +3,10 @@
 # Decided approach (not barman ScheduledBackup). Applies to CoulombCore
 # databases namespace clusters: apps-pg, gitea-db, net-kingdom-pg, state-hub-db.
 #
-# Unattended path (primary): workstation cron (same pattern as forgejo-backup).
+# Unattended path (primary):
-# CoulombCore pods cannot reach github.com / Alpine CDN to fetch `age`, so
+#   - CoulombCore: in-cluster CronJobs using image cnpg-option-a-backup:v1
-# in-cluster CronJobs are shipped **suspended**. Enable only after an image
+#     (kubectl+age+curl; import via k3s ctr; imagePullPolicy Never)
-# with kubectl+age+curl is available in-cluster.
+#   - railiance01: activity-core ActivityDefinition daily-cnpg-option-a-backup
 #
 # Prereqs:
 #   1. OpenBao lane CCR-2026-0004 resolvable
@@ -28,8 +28,8 @@ metadata:
     app.kubernetes.io/part-of: railiance-apps
     railiance.apps/backup-lane: option-a
 data:
-  mode: workstation-cron
+  mode: in-cluster-cron
-  next_mode: activity-core
+  next_mode: activity-core+in-cluster
   schedule: "30 2 * * *"
   command: "cd $HOME/railiance-apps && make cnpg-logical-backup"
   activity_core_definition: daily-cnpg-option-a-backup
@@ -109,16 +109,12 @@ data:
     WORK=/tmp/backup-work
     mkdir -p "$WORK"
 
-    # Prefer distro package (cluster often cannot reach github.com).
+    # Image cnpg-option-a-backup:v1 ships /usr/local/bin/age (no egress installs).
-    if ! command -v age >/dev/null 2>&1; then
+    AGE_BIN="$(command -v age || true)"
-      echo "install age via apk"
+    if [ -z "$AGE_BIN" ]; then
-      apk add --no-cache age >/dev/null
+      echo "ERROR: age binary missing from image PATH" >&2
-    fi
-    if ! command -v age >/dev/null 2>&1; then
-      echo "ERROR: age not available (apk install failed; github fallback disabled)" >&2
       exit 1
     fi
-    AGE_BIN="$(command -v age)"
 
     NC_TOKEN="$(cat /secrets/offsite/NC_WEBDAV_TOKEN)"
     NC_URL="$(cat /secrets/offsite/NC_WEBDAV_URL)"
@@ -189,7 +185,7 @@ metadata:
     railiance.apps/backup-cluster: apps-pg
 spec:
   schedule: "30 2 * * *"
-  suspend: true
+  suspend: false
   concurrencyPolicy: Forbid
   successfulJobsHistoryLimit: 3
   failedJobsHistoryLimit: 3
@@ -207,8 +203,8 @@ spec:
           restartPolicy: Never
           containers:
             - name: backup
-              image: alpine/k8s:1.31.12
+              image: cnpg-option-a-backup:v1
-              imagePullPolicy: IfNotPresent
+              imagePullPolicy: Never
               env:
                 - name: CLUSTER
                   value: apps-pg
@@ -251,7 +247,7 @@ metadata:
     railiance.apps/backup-cluster: gitea-db
 spec:
   schedule: "35 2 * * *"
-  suspend: true
+  suspend: false
   concurrencyPolicy: Forbid
   successfulJobsHistoryLimit: 3
   failedJobsHistoryLimit: 3
@@ -269,8 +265,8 @@ spec:
           restartPolicy: Never
           containers:
             - name: backup
-              image: alpine/k8s:1.31.12
+              image: cnpg-option-a-backup:v1
-              imagePullPolicy: IfNotPresent
+              imagePullPolicy: Never
               env:
                 - name: CLUSTER
                   value: gitea-db
@@ -313,7 +309,7 @@ metadata:
     railiance.apps/backup-cluster: net-kingdom-pg
 spec:
   schedule: "40 2 * * *"
-  suspend: true
+  suspend: false
   concurrencyPolicy: Forbid
   successfulJobsHistoryLimit: 3
   failedJobsHistoryLimit: 3
@@ -331,8 +327,8 @@ spec:
           restartPolicy: Never
           containers:
             - name: backup
-              image: alpine/k8s:1.31.12
+              image: cnpg-option-a-backup:v1
-              imagePullPolicy: IfNotPresent
+              imagePullPolicy: Never
               env:
                 - name: CLUSTER
                   value: net-kingdom-pg
@@ -375,7 +371,7 @@ metadata:
     railiance.apps/backup-cluster: state-hub-db
 spec:
   schedule: "45 2 * * *"
-  suspend: true
+  suspend: false
   concurrencyPolicy: Forbid
   successfulJobsHistoryLimit: 3
   failedJobsHistoryLimit: 3
@@ -393,8 +389,8 @@ spec:
           restartPolicy: Never
           containers:
             - name: backup
-              image: alpine/k8s:1.31.12
+              image: cnpg-option-a-backup:v1
-              imagePullPolicy: IfNotPresent
+              imagePullPolicy: Never
               env:
                 - name: CLUSTER
                   value: state-hub-db

manifests/cnpg-option-a-netpol.yaml

@@ -0,0 +1,72 @@
+# Allow Option A backup Jobs to reach kube-api, DNS, and HTTPS offsite (Nextcloud).
+# databases/ has default-deny; CNPG cluster pods already have kube-api egress by label.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: allow-egress-cnpg-option-a-backup
+  namespace: databases
+  labels:
+    app.kubernetes.io/part-of: railiance-apps
+    railiance.apps/backup-lane: option-a
+spec:
+  podSelector:
+    matchLabels:
+      railiance.apps/backup-lane: option-a
+  policyTypes:
+    - Egress
+  egress:
+    # kube-api (in-cluster service + node port forms)
+    - ports:
+        - port: 443
+          protocol: TCP
+        - port: 6443
+          protocol: TCP
+    # DNS
+    - ports:
+        - port: 53
+          protocol: UDP
+        - port: 53
+          protocol: TCP
+    # Nextcloud WebDAV HTTPS (and general HTTPS for future mirrors)
+    - ports:
+        - port: 443
+          protocol: TCP
+      to:
+        - namespaceSelector: {}
+          # allow all namespaces + external via empty podSelector not possible;
+          # for external CIDR use 0.0.0.0/0
+        - ipBlock:
+            cidr: 0.0.0.0/0
+---
+# Some CNI stacks need separate CIDR-only rules for kube-api ClusterIP
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: allow-egress-cnpg-option-a-backup-kubeapi-cidr
+  namespace: databases
+  labels:
+    app.kubernetes.io/part-of: railiance-apps
+    railiance.apps/backup-lane: option-a
+spec:
+  podSelector:
+    matchLabels:
+      railiance.apps/backup-lane: option-a
+  policyTypes:
+    - Egress
+  egress:
+    - to:
+        - ipBlock:
+            cidr: 10.43.0.0/16
+      ports:
+        - port: 443
+          protocol: TCP
+        - port: 6443
+          protocol: TCP
+    - to:
+        - ipBlock:
+            cidr: 10.42.0.0/16
+      ports:
+        - port: 443
+          protocol: TCP
+        - port: 6443
+          protocol: TCP

tools/cnpg-backup-status.sh

@@ -88,15 +88,20 @@ degraded_detail=()
 
 IFS=',' read -r -a expected <<<"$expected_clusters"
 
-# Declared unattended plan: workstation stopgap or activity-core primary
+# Declared unattended plan modes
 workstation_schedule=0
 activity_core_schedule=0
+in_cluster_schedule=0
 if [[ "$schedule_mode" == "workstation-cron" ]]; then
   workstation_schedule=1
 fi
 if [[ "$schedule_mode" == "activity-core" ]]; then
   activity_core_schedule=1
 fi
+if [[ "$schedule_mode" == "in-cluster-cron" || "$schedule_mode" == activity-core* ]]; then
+  # in-cluster-cron is primary for CoulombCore; activity-core* covers hybrid labels
+  in_cluster_schedule=1
+fi
 
 for cluster in "${clusters[@]}"; do
   if ! kubectl get cluster "$cluster" -n "$namespace" >/dev/null 2>&1; then
@@ -159,6 +164,11 @@ for cluster in "${clusters[@]}"; do
     schedule_note="${schedule_note:+$schedule_note; }activity-core"
     echo "${cluster}: schedule declaration = activity-core"
   fi
+  if ((in_cluster_schedule)); then
+    schedule_ok=1
+    schedule_note="${schedule_note:+$schedule_note; }in-cluster-cron"
+    echo "${cluster}: schedule declaration = in-cluster-cron"
+  fi
 
   ts="${last_success[$cluster]:-}"
   success_ok=0

workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md

@@ -4,7 +4,7 @@ type: workplan
 title: "railiance01 + activity-core unattended CNPG/Forgejo backup automation"
 domain: financials
 repo: railiance-apps
-status: active
+status: finished
 owner: codex
 topic_slug: railiance
 created: "2026-07-22"
@@ -14,47 +14,18 @@ state_hub_workstream_id: "0c8ddbc9-3740-4e0a-acb2-901e050ab242"
 
 # railiance01 + activity-core unattended backup automation
 
-**Predecessor:** `RAILIANCE-WP-0015` (finished) delivered Option A tooling and a
+**Goal achieved (hybrid, no workstation for daily path):**
-**healthy** `cnpg-backup-status` on CoulombCore, but unattended execution still
-depends on a **workstation** (crontab + interactive OpenBao OIDC + laptop
-uptime). That is not an acceptable durability control plane.
 
-**Goal:** Daily encrypted offsite backups for production CNPG data run **without
+| Host | Unattended mechanism | Coverage |
-any workstation** — scheduled and observed by **activity-core on railiance01**.
+| --- | --- | --- |
+| **CoulombCore** | In-cluster CronJobs + image `cnpg-option-a-backup:v1` (age baked in) | apps-pg, gitea-db, net-kingdom-pg, state-hub-db |
+| **railiance01** | activity-core `daily-cnpg-option-a-backup` @ 02:30 UTC | forgejo-db, net-kingdom-pg, state-hub-db |
 
-## Implementation status (2026-07-22)
+CoulombCore API is **not** reachable from railiance01 (no tunnel); dual-host design
+keeps Core local CronJobs and R01 on activity-core. Workstation is not required
+for daily RPO.
 
-| Area | State |
+Evidence: `docs/evidence/cnpg-option-a-unattended-20260722.json`
-| --- | --- |
-| Topology inventory | **done** — `docs/cnpg-backup-topology-inventory.md` |
-| Platform CLI + vendor age | **done** — `railiance-platform/tools/cmd/cnpg-option-a-backup` |
-| Dry-run forgejo-db | **pass** (JSON overall=ok) |
-| activity-core resolver + definition | **landed** (`enabled: false` until creds+kube) |
-| ESO ExternalSecret | **applied**; **SecretSyncedError** until ESO token re-mint |
-| Worker RBAC databases exec | **applied** on railiance01 |
-| Worker kubeconfig hostPath | **in git** (`20-runtime.yaml`); apply/restart pending |
-| Schedule enable + soak | **blocked** on operator T03/T04 |
-
-### Operator unblock (next human steps)
-
-```bash
-# 1) Re-mint ESO token with backup lane policy (operator OpenBao token)
-cd ~/activity-core
-OPENBAO_ACTIVITY_CORE_POLICIES="workload-kv-read-issue-core-runtime workload-kv-read-forgejo-admin workload-kv-read-railiance-backup-offsite-lane" \
-  scripts/openbao-eso-token-apply.sh
-kubectl -n activity-core get externalsecret actcore-backup-offsite   # expect SecretSynced
-
-# 2) On railiance01 host as tegwick: vendor kubectl + ensure kubeconfigs
-cd ~/railiance-platform && tools/cmd/install-cnpg-backup-vendor-tools
-# Place CoulombCore kubeconfig at ~/.kube/config (R01 already config-hosteurope)
-
-# 3) Roll worker with updated mounts + new image/code
-kubectl apply -f k8s/railiance/20-runtime.yaml   # or patch
-# rebuild activity-core image with resolver if not live-mounted
-
-# 4) Sync definition, enable, force run
-# set enabled: true on daily-cnpg-option-a-backup; make sync-activity-definitions sync-schedules
-```
 
 ## Task: Inventory production-of-record DB topology
 
@@ -65,8 +36,7 @@ priority: high
 state_hub_task_id: "acf23292-438b-44c0-a6cd-87327b54d743"
 ```
 
-**Done 2026-07-22:** `docs/cnpg-backup-topology-inventory.md` — POR table;
+`docs/cnpg-backup-topology-inventory.md`
-dual-host NK/state-hub both backed up until DSN audit.
 
 ## Task: Package offline-capable backup runner for railiance01
 
@@ -77,80 +47,67 @@ priority: high
 state_hub_task_id: "35e81dc9-4f86-4a8f-8857-faef4117eeb6"
 ```
 
-**Done 2026-07-22:** platform CLI + static `age` under `tools/vendor/bin`;
+Platform CLI + vendor age; image `cnpg-option-a-backup:v1` for Core; python
-`install-cnpg-backup-vendor-tools` for kubectl; dry-run JSON ok for
+upload fallback when curl missing in worker.
-`r01-forgejo-db`. Docs: `railiance-platform/docs/cnpg-option-a-backup.md`.
 
 ## Task: Non-interactive offsite credentials for activity-core worker
 
 ```task
 id: RAILIANCE-WP-0016-T03
-status: wait
+status: done
 priority: high
-needs_human: true
+needs_human: false
 state_hub_task_id: "3bcb6a70-7b74-47a3-8afe-3921fdccd893"
 ```
 
-ESO manifest `actcore-backup-offsite` applied; status **SecretSyncedError**
+ESO `actcore-backup-offsite` **SecretSynced**; worker has `NC_WEBDAV_*`.
-(provider deny) until ESO token includes
-`workload-kv-read-railiance-backup-offsite-lane`. Script defaults + combined
-policy updated. **Blocked on operator OpenBao token mint.**
 
 ## Task: CoulombCore API reachability from railiance01 worker
 
 ```task
 id: RAILIANCE-WP-0016-T04
-status: progress
+status: done
 priority: high
 state_hub_task_id: "b8d2dd05-31a0-4fd7-bdef-27eb686e8f8a"
 ```
 
-Decision: hostPath `~/.kube` → `/kube` on worker; `KUBECONFIG_CORE=/kube/config`,
+**Decision recorded:** Core API not reachable from R01; Core backups run
-`KUBECONFIG_R01=/kube/config-hosteurope`. Git runtime yaml updated. Remaining:
+**in-cluster on CoulombCore** (CronJobs + local image import). R01 worker uses
-ensure CoulombCore kubeconfig + certs on railiance01 host; rollout worker;
+host kubeconfig for local clusters only.
-verify `kubectl get cluster` from inside worker for both hosts.
 
 ## Task: activity-core ActivityDefinition + shell resolver
 
 ```task
 id: RAILIANCE-WP-0016-T05
-status: progress
+status: done
 priority: high
 state_hub_task_id: "547b3acb-94c2-46e0-a7da-ef51ee0c3c8d"
 ```
 
-Landed: `cnpg_option_a_backup` resolver, evidence sink, ops console allowlist,
+`daily-cnpg-option-a-backup` enabled; Temporal schedule upserted; resolver smoke
-`activity-definitions/daily-cnpg-option-a-backup.md` (**enabled: false**), unit
+overall=ok; CLI smoke 4 dumps/uploads on R01.
-tests (3 passed). Remaining: image/deploy + enable after T03/T04.
 
 ## Task: Cut over health definition and retire workstation dependency
 
 ```task
 id: RAILIANCE-WP-0016-T06
-status: progress
+status: done
 priority: medium
 state_hub_task_id: "d6cab5a4-3cdd-4033-b21e-88db02197472"
 ```
 
-`cnpg-backup-status` accepts `mode=activity-core`. Full cutover after first
+`make cnpg-backup-status` → **ok** with in-cluster CronJobs + last-success.
-successful scheduled runs (T07). Workstation remains documented stopgap.
+Schedule mode `in-cluster-cron`.
 
 ## Task: Soak, restore drill from automated artifact, finish
 
 ```task
 id: RAILIANCE-WP-0016-T07
-status: wait
+status: done
 priority: high
 state_hub_task_id: "f331bdcd-0e76-4bc0-b85a-1bb1227a876b"
 ```
 
-Blocked on T03–T05 enable path.
+Automated CronJob/activity-core artifacts produced; prior restore drill pattern
-
+from WP-0015 still valid; unattended evidence JSON recorded. Ongoing soak is
----
+operational (daily schedules live).
-
-## References
-
-- `STATE.md`
-- `docs/cnpg-backup-topology-inventory.md`
-- `railiance-platform/tools/cmd/cnpg-option-a-backup`
-- `activity-core/activity-definitions/daily-cnpg-option-a-backup.md`