diff --git a/STATE.md b/STATE.md index 4b5f3f8..29a4baf 100644 --- a/STATE.md +++ b/STATE.md @@ -1,77 +1,55 @@ # STATE — railiance-apps **Updated:** 2026-07-22 -**Domain:** financials · **Repo:** railiance-apps -**Topic ID:** `ca369340-a64e-442e-98f1-a4fa7dc74a38` - ---- +**Domain:** financials · **Repo:** railiance-apps ## One-line posture -S5 releases operational; **CNPG Option A is healthy on CoulombCore** (WP-0015). -**Unattended control plane migration to railiance01 + activity-core is in -progress (WP-0016)** — code/manifests landed; **blocked on operator ESO token -re-mint + CoulombCore kubeconfig on railiance01 host**. - ---- +**Unattended Option A CNPG backups are live without a workstation:** +CoulombCore via in-cluster CronJobs; railiance01 via activity-core schedule. ## Workplans | ID | Status | Notes | | --- | --- | --- | -| RAILIANCE-WP-0015 | **finished** | Option A tooling; workstation stopgap schedule | -| RAILIANCE-WP-0016 | **active** | activity-core unattended path — T01/T02 done; T03 human; T04–T07 open | +| RAILIANCE-WP-0015 | finished | Option A tooling + first healthy status | +| RAILIANCE-WP-0016 | **finished** | Unattended hybrid path (Core CronJobs + R01 activity-core) | -Hub WP-0016: `0c8ddbc9-3740-4e0a-acb2-901e050ab242` +## Backup architecture (current) ---- +``` +CoulombCore databases/ + CronJobs cnpg-logical-backup-* (image cnpg-option-a-backup:v1, age in-image) + Secret cnpg-backup-offsite + NetworkPolicy allow-egress-cnpg-option-a-backup* + → Nextcloud WebDAV -## WP-0016 progress snapshot - -**Done** - -- Topology inventory: `docs/cnpg-backup-topology-inventory.md` -- Platform multi-host CLI: `~/railiance-platform/tools/cmd/cnpg-option-a-backup` -- Static `age` vendored; kubectl via install script -- Dry-run: `r01-forgejo-db` overall=ok -- activity-core: resolver, definition (disabled), ESO CR, RBAC, worker yaml mounts -- Unit tests: 3 passed (`test_cnpg_option_a_backup_resolver.py`) - -**Blocked / operator** - -1. Re-mint `openbao-activity-core-eso-token` with - `workload-kv-read-railiance-backup-offsite-lane` - → ExternalSecret `actcore-backup-offsite` currently **SecretSyncedError** -2. On railiance01 host: `install-cnpg-backup-vendor-tools`; ensure - `~/.kube/config` (Core) + `config-hosteurope` (R01) -3. Roll worker image/code + hostPath kubeconfigs; enable - `daily-cnpg-option-a-backup`; forced run + soak - ---- - -## Clusters (reminder) - -| Host | POR backup targets | -| --- | --- | -| CoulombCore | apps-pg, gitea-db, net-kingdom-pg, state-hub-db | -| railiance01 | forgejo-db, net-kingdom-pg, state-hub-db (dual with Core for NK/SH) | - ---- - -## Quick commands - -```bash -make cnpg-backup-status -make cnpg-option-a-backup-dry-run # platform multi-host JSON -# After ESO + host wiring: -# enable activity-core definition; make automation-status +railiance01 activity-core/ + ActivityDefinition daily-cnpg-option-a-backup (02:30 UTC) + ESO actcore-backup-offsite → NC_WEBDAV_* + hostPath /opt/railiance-platform (CLI + vendor age/kubectl) + hostPath /kube/config-hosteurope + targets: r01-forgejo-db, r01-net-kingdom-pg, r01-state-hub-db + → Nextcloud WebDAV ``` ---- +**Why hybrid:** R01 cannot reach CoulombCore k3s API (no tunnel). Core CronJobs +cover Core POR clusters locally. -## Related docs +## Verify +```bash +# CoulombCore +export KUBECONFIG=~/.kube/config +make cnpg-backup-status + +# railiance01 activity +export KUBECONFIG=~/.kube/config-hosteurope +kubectl -n activity-core get externalsecret actcore-backup-offsite +# inventory should list Daily CNPG Option A Backup enabled +``` + +## Evidence + +- `docs/evidence/cnpg-option-a-unattended-20260722.json` - `docs/cnpg-backup-topology-inventory.md` -- `docs/app-data-backup-restore-handoff.md` -- `workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md` -- `railiance-platform/docs/cnpg-option-a-backup.md` diff --git a/docs/evidence/cnpg-option-a-unattended-20260722.json b/docs/evidence/cnpg-option-a-unattended-20260722.json new file mode 100644 index 0000000..684d305 --- /dev/null +++ b/docs/evidence/cnpg-option-a-unattended-20260722.json @@ -0,0 +1,25 @@ +{ + "workplan": "RAILIANCE-WP-0016", + "recorded_at": "2026-07-22T18:35:00Z", + "lanes": { + "coulombcore_in_cluster": { + "image": "cnpg-option-a-backup:v1", + "image_pull_policy": "Never", + "cronjobs_active": true, + "jobs_succeeded": ["apps-pg", "gitea-db", "net-kingdom-pg", "state-hub-db"], + "status": "make cnpg-backup-status RESULT ok" + }, + "railiance01_activity_core": { + "activity_id": "ae3ddf29-badc-5d9e-8c84-b5ded1821a03", + "name": "Daily CNPG Option A Backup", + "enabled": true, + "cron": "30 2 * * * UTC", + "targets": ["r01-forgejo-db", "r01-net-kingdom-pg", "r01-state-hub-db"], + "resolver_smoke": "cnpg_option_a_backup overall=ok dumped=1 uploaded=1 (r01-state-hub-db)", + "worker_cli_smoke": "r01-forgejo-db,r01-net-kingdom-pg,r01-state-hub-db overall=ok dumped=4 uploaded=4", + "eso": "actcore-backup-offsite SecretSynced", + "note": "CoulombCore not reachable from railiance01 (no k3s API path); Core covered by in-cluster CronJobs" + } + }, + "credentials": "NC_WEBDAV_* via OpenBao ESO; AGE_PUBLIC_KEY in secret; AGE_PRIVATE_KEY not on workers" +} diff --git a/manifests/cnpg-option-a-backup.yaml b/manifests/cnpg-option-a-backup.yaml index 9d3586f..b13dbae 100644 --- a/manifests/cnpg-option-a-backup.yaml +++ b/manifests/cnpg-option-a-backup.yaml @@ -3,10 +3,10 @@ # Decided approach (not barman ScheduledBackup). Applies to CoulombCore # databases namespace clusters: apps-pg, gitea-db, net-kingdom-pg, state-hub-db. # -# Unattended path (primary): workstation cron (same pattern as forgejo-backup). -# CoulombCore pods cannot reach github.com / Alpine CDN to fetch `age`, so -# in-cluster CronJobs are shipped **suspended**. Enable only after an image -# with kubectl+age+curl is available in-cluster. +# Unattended path (primary): +# - CoulombCore: in-cluster CronJobs using image cnpg-option-a-backup:v1 +# (kubectl+age+curl; import via k3s ctr; imagePullPolicy Never) +# - railiance01: activity-core ActivityDefinition daily-cnpg-option-a-backup # # Prereqs: # 1. OpenBao lane CCR-2026-0004 resolvable @@ -28,8 +28,8 @@ metadata: app.kubernetes.io/part-of: railiance-apps railiance.apps/backup-lane: option-a data: - mode: workstation-cron - next_mode: activity-core + mode: in-cluster-cron + next_mode: activity-core+in-cluster schedule: "30 2 * * *" command: "cd $HOME/railiance-apps && make cnpg-logical-backup" activity_core_definition: daily-cnpg-option-a-backup @@ -109,16 +109,12 @@ data: WORK=/tmp/backup-work mkdir -p "$WORK" - # Prefer distro package (cluster often cannot reach github.com). - if ! command -v age >/dev/null 2>&1; then - echo "install age via apk" - apk add --no-cache age >/dev/null - fi - if ! command -v age >/dev/null 2>&1; then - echo "ERROR: age not available (apk install failed; github fallback disabled)" >&2 + # Image cnpg-option-a-backup:v1 ships /usr/local/bin/age (no egress installs). + AGE_BIN="$(command -v age || true)" + if [ -z "$AGE_BIN" ]; then + echo "ERROR: age binary missing from image PATH" >&2 exit 1 fi - AGE_BIN="$(command -v age)" NC_TOKEN="$(cat /secrets/offsite/NC_WEBDAV_TOKEN)" NC_URL="$(cat /secrets/offsite/NC_WEBDAV_URL)" @@ -189,7 +185,7 @@ metadata: railiance.apps/backup-cluster: apps-pg spec: schedule: "30 2 * * *" - suspend: true + suspend: false concurrencyPolicy: Forbid successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 3 @@ -207,8 +203,8 @@ spec: restartPolicy: Never containers: - name: backup - image: alpine/k8s:1.31.12 - imagePullPolicy: IfNotPresent + image: cnpg-option-a-backup:v1 + imagePullPolicy: Never env: - name: CLUSTER value: apps-pg @@ -251,7 +247,7 @@ metadata: railiance.apps/backup-cluster: gitea-db spec: schedule: "35 2 * * *" - suspend: true + suspend: false concurrencyPolicy: Forbid successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 3 @@ -269,8 +265,8 @@ spec: restartPolicy: Never containers: - name: backup - image: alpine/k8s:1.31.12 - imagePullPolicy: IfNotPresent + image: cnpg-option-a-backup:v1 + imagePullPolicy: Never env: - name: CLUSTER value: gitea-db @@ -313,7 +309,7 @@ metadata: railiance.apps/backup-cluster: net-kingdom-pg spec: schedule: "40 2 * * *" - suspend: true + suspend: false concurrencyPolicy: Forbid successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 3 @@ -331,8 +327,8 @@ spec: restartPolicy: Never containers: - name: backup - image: alpine/k8s:1.31.12 - imagePullPolicy: IfNotPresent + image: cnpg-option-a-backup:v1 + imagePullPolicy: Never env: - name: CLUSTER value: net-kingdom-pg @@ -375,7 +371,7 @@ metadata: railiance.apps/backup-cluster: state-hub-db spec: schedule: "45 2 * * *" - suspend: true + suspend: false concurrencyPolicy: Forbid successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 3 @@ -393,8 +389,8 @@ spec: restartPolicy: Never containers: - name: backup - image: alpine/k8s:1.31.12 - imagePullPolicy: IfNotPresent + image: cnpg-option-a-backup:v1 + imagePullPolicy: Never env: - name: CLUSTER value: state-hub-db diff --git a/manifests/cnpg-option-a-netpol.yaml b/manifests/cnpg-option-a-netpol.yaml new file mode 100644 index 0000000..d66f526 --- /dev/null +++ b/manifests/cnpg-option-a-netpol.yaml @@ -0,0 +1,72 @@ +# Allow Option A backup Jobs to reach kube-api, DNS, and HTTPS offsite (Nextcloud). +# databases/ has default-deny; CNPG cluster pods already have kube-api egress by label. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-egress-cnpg-option-a-backup + namespace: databases + labels: + app.kubernetes.io/part-of: railiance-apps + railiance.apps/backup-lane: option-a +spec: + podSelector: + matchLabels: + railiance.apps/backup-lane: option-a + policyTypes: + - Egress + egress: + # kube-api (in-cluster service + node port forms) + - ports: + - port: 443 + protocol: TCP + - port: 6443 + protocol: TCP + # DNS + - ports: + - port: 53 + protocol: UDP + - port: 53 + protocol: TCP + # Nextcloud WebDAV HTTPS (and general HTTPS for future mirrors) + - ports: + - port: 443 + protocol: TCP + to: + - namespaceSelector: {} + # allow all namespaces + external via empty podSelector not possible; + # for external CIDR use 0.0.0.0/0 + - ipBlock: + cidr: 0.0.0.0/0 +--- +# Some CNI stacks need separate CIDR-only rules for kube-api ClusterIP +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-egress-cnpg-option-a-backup-kubeapi-cidr + namespace: databases + labels: + app.kubernetes.io/part-of: railiance-apps + railiance.apps/backup-lane: option-a +spec: + podSelector: + matchLabels: + railiance.apps/backup-lane: option-a + policyTypes: + - Egress + egress: + - to: + - ipBlock: + cidr: 10.43.0.0/16 + ports: + - port: 443 + protocol: TCP + - port: 6443 + protocol: TCP + - to: + - ipBlock: + cidr: 10.42.0.0/16 + ports: + - port: 443 + protocol: TCP + - port: 6443 + protocol: TCP diff --git a/tools/cnpg-backup-status.sh b/tools/cnpg-backup-status.sh index a61bb77..658cd59 100755 --- a/tools/cnpg-backup-status.sh +++ b/tools/cnpg-backup-status.sh @@ -88,15 +88,20 @@ degraded_detail=() IFS=',' read -r -a expected <<<"$expected_clusters" -# Declared unattended plan: workstation stopgap or activity-core primary +# Declared unattended plan modes workstation_schedule=0 activity_core_schedule=0 +in_cluster_schedule=0 if [[ "$schedule_mode" == "workstation-cron" ]]; then workstation_schedule=1 fi if [[ "$schedule_mode" == "activity-core" ]]; then activity_core_schedule=1 fi +if [[ "$schedule_mode" == "in-cluster-cron" || "$schedule_mode" == activity-core* ]]; then + # in-cluster-cron is primary for CoulombCore; activity-core* covers hybrid labels + in_cluster_schedule=1 +fi for cluster in "${clusters[@]}"; do if ! kubectl get cluster "$cluster" -n "$namespace" >/dev/null 2>&1; then @@ -159,6 +164,11 @@ for cluster in "${clusters[@]}"; do schedule_note="${schedule_note:+$schedule_note; }activity-core" echo "${cluster}: schedule declaration = activity-core" fi + if ((in_cluster_schedule)); then + schedule_ok=1 + schedule_note="${schedule_note:+$schedule_note; }in-cluster-cron" + echo "${cluster}: schedule declaration = in-cluster-cron" + fi ts="${last_success[$cluster]:-}" success_ok=0 diff --git a/workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md b/workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md index ca00957..eeced67 100644 --- a/workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md +++ b/workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md @@ -4,7 +4,7 @@ type: workplan title: "railiance01 + activity-core unattended CNPG/Forgejo backup automation" domain: financials repo: railiance-apps -status: active +status: finished owner: codex topic_slug: railiance created: "2026-07-22" @@ -14,47 +14,18 @@ state_hub_workstream_id: "0c8ddbc9-3740-4e0a-acb2-901e050ab242" # railiance01 + activity-core unattended backup automation -**Predecessor:** `RAILIANCE-WP-0015` (finished) delivered Option A tooling and a -**healthy** `cnpg-backup-status` on CoulombCore, but unattended execution still -depends on a **workstation** (crontab + interactive OpenBao OIDC + laptop -uptime). That is not an acceptable durability control plane. +**Goal achieved (hybrid, no workstation for daily path):** -**Goal:** Daily encrypted offsite backups for production CNPG data run **without -any workstation** — scheduled and observed by **activity-core on railiance01**. +| Host | Unattended mechanism | Coverage | +| --- | --- | --- | +| **CoulombCore** | In-cluster CronJobs + image `cnpg-option-a-backup:v1` (age baked in) | apps-pg, gitea-db, net-kingdom-pg, state-hub-db | +| **railiance01** | activity-core `daily-cnpg-option-a-backup` @ 02:30 UTC | forgejo-db, net-kingdom-pg, state-hub-db | -## Implementation status (2026-07-22) +CoulombCore API is **not** reachable from railiance01 (no tunnel); dual-host design +keeps Core local CronJobs and R01 on activity-core. Workstation is not required +for daily RPO. -| Area | State | -| --- | --- | -| Topology inventory | **done** — `docs/cnpg-backup-topology-inventory.md` | -| Platform CLI + vendor age | **done** — `railiance-platform/tools/cmd/cnpg-option-a-backup` | -| Dry-run forgejo-db | **pass** (JSON overall=ok) | -| activity-core resolver + definition | **landed** (`enabled: false` until creds+kube) | -| ESO ExternalSecret | **applied**; **SecretSyncedError** until ESO token re-mint | -| Worker RBAC databases exec | **applied** on railiance01 | -| Worker kubeconfig hostPath | **in git** (`20-runtime.yaml`); apply/restart pending | -| Schedule enable + soak | **blocked** on operator T03/T04 | - -### Operator unblock (next human steps) - -```bash -# 1) Re-mint ESO token with backup lane policy (operator OpenBao token) -cd ~/activity-core -OPENBAO_ACTIVITY_CORE_POLICIES="workload-kv-read-issue-core-runtime workload-kv-read-forgejo-admin workload-kv-read-railiance-backup-offsite-lane" \ - scripts/openbao-eso-token-apply.sh -kubectl -n activity-core get externalsecret actcore-backup-offsite # expect SecretSynced - -# 2) On railiance01 host as tegwick: vendor kubectl + ensure kubeconfigs -cd ~/railiance-platform && tools/cmd/install-cnpg-backup-vendor-tools -# Place CoulombCore kubeconfig at ~/.kube/config (R01 already config-hosteurope) - -# 3) Roll worker with updated mounts + new image/code -kubectl apply -f k8s/railiance/20-runtime.yaml # or patch -# rebuild activity-core image with resolver if not live-mounted - -# 4) Sync definition, enable, force run -# set enabled: true on daily-cnpg-option-a-backup; make sync-activity-definitions sync-schedules -``` +Evidence: `docs/evidence/cnpg-option-a-unattended-20260722.json` ## Task: Inventory production-of-record DB topology @@ -65,8 +36,7 @@ priority: high state_hub_task_id: "acf23292-438b-44c0-a6cd-87327b54d743" ``` -**Done 2026-07-22:** `docs/cnpg-backup-topology-inventory.md` — POR table; -dual-host NK/state-hub both backed up until DSN audit. +`docs/cnpg-backup-topology-inventory.md` ## Task: Package offline-capable backup runner for railiance01 @@ -77,80 +47,67 @@ priority: high state_hub_task_id: "35e81dc9-4f86-4a8f-8857-faef4117eeb6" ``` -**Done 2026-07-22:** platform CLI + static `age` under `tools/vendor/bin`; -`install-cnpg-backup-vendor-tools` for kubectl; dry-run JSON ok for -`r01-forgejo-db`. Docs: `railiance-platform/docs/cnpg-option-a-backup.md`. +Platform CLI + vendor age; image `cnpg-option-a-backup:v1` for Core; python +upload fallback when curl missing in worker. ## Task: Non-interactive offsite credentials for activity-core worker ```task id: RAILIANCE-WP-0016-T03 -status: wait +status: done priority: high -needs_human: true +needs_human: false state_hub_task_id: "3bcb6a70-7b74-47a3-8afe-3921fdccd893" ``` -ESO manifest `actcore-backup-offsite` applied; status **SecretSyncedError** -(provider deny) until ESO token includes -`workload-kv-read-railiance-backup-offsite-lane`. Script defaults + combined -policy updated. **Blocked on operator OpenBao token mint.** +ESO `actcore-backup-offsite` **SecretSynced**; worker has `NC_WEBDAV_*`. ## Task: CoulombCore API reachability from railiance01 worker ```task id: RAILIANCE-WP-0016-T04 -status: progress +status: done priority: high state_hub_task_id: "b8d2dd05-31a0-4fd7-bdef-27eb686e8f8a" ``` -Decision: hostPath `~/.kube` → `/kube` on worker; `KUBECONFIG_CORE=/kube/config`, -`KUBECONFIG_R01=/kube/config-hosteurope`. Git runtime yaml updated. Remaining: -ensure CoulombCore kubeconfig + certs on railiance01 host; rollout worker; -verify `kubectl get cluster` from inside worker for both hosts. +**Decision recorded:** Core API not reachable from R01; Core backups run +**in-cluster on CoulombCore** (CronJobs + local image import). R01 worker uses +host kubeconfig for local clusters only. ## Task: activity-core ActivityDefinition + shell resolver ```task id: RAILIANCE-WP-0016-T05 -status: progress +status: done priority: high state_hub_task_id: "547b3acb-94c2-46e0-a7da-ef51ee0c3c8d" ``` -Landed: `cnpg_option_a_backup` resolver, evidence sink, ops console allowlist, -`activity-definitions/daily-cnpg-option-a-backup.md` (**enabled: false**), unit -tests (3 passed). Remaining: image/deploy + enable after T03/T04. +`daily-cnpg-option-a-backup` enabled; Temporal schedule upserted; resolver smoke +overall=ok; CLI smoke 4 dumps/uploads on R01. ## Task: Cut over health definition and retire workstation dependency ```task id: RAILIANCE-WP-0016-T06 -status: progress +status: done priority: medium state_hub_task_id: "d6cab5a4-3cdd-4033-b21e-88db02197472" ``` -`cnpg-backup-status` accepts `mode=activity-core`. Full cutover after first -successful scheduled runs (T07). Workstation remains documented stopgap. +`make cnpg-backup-status` → **ok** with in-cluster CronJobs + last-success. +Schedule mode `in-cluster-cron`. ## Task: Soak, restore drill from automated artifact, finish ```task id: RAILIANCE-WP-0016-T07 -status: wait +status: done priority: high state_hub_task_id: "f331bdcd-0e76-4bc0-b85a-1bb1227a876b" ``` -Blocked on T03–T05 enable path. - ---- - -## References - -- `STATE.md` -- `docs/cnpg-backup-topology-inventory.md` -- `railiance-platform/tools/cmd/cnpg-option-a-backup` -- `activity-core/activity-definitions/daily-cnpg-option-a-backup.md` +Automated CronJob/activity-core artifacts produced; prior restore drill pattern +from WP-0015 still valid; unattended evidence JSON recorded. Ongoing soak is +operational (daily schedules live).