railiance-apps/workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated
RAILIANCE-WP-0016	workplan	railiance01 + activity-core unattended CNPG/Forgejo backup automation	financials	railiance-apps	proposed	codex	railiance	2026-07-22	2026-07-22

railiance01 + activity-core unattended backup automation

Predecessor: RAILIANCE-WP-0015 (finished) delivered Option A tooling and a healthy cnpg-backup-status on CoulombCore, but unattended execution still depends on a workstation (crontab + interactive OpenBao OIDC + laptop uptime). That is not an acceptable durability control plane.

Goal: Daily (and weekly retention) encrypted offsite backups for production CNPG/Forgejo data run without any workstation — scheduled and observed by activity-core on railiance01, following the proven weekly-forgejo-package-prune pattern.

Context and constraints

Topology (2026-07-22)

Host	Clusters / data
CoulombCore	apps-pg (vergage, core_hub, …), gitea-db, net-kingdom-pg, state-hub-db
railiance01	forgejo-db, net-kingdom-pg, state-hub-db (+ activity-core, Forgejo)

NK and state-hub exist on both hosts — WP must inventory which instance is production-of-record per consumer before scheduling dumps.

Why workstation fails as control plane

• Laptop sleep / travel / network breaks RPO 24h.
• OIDC bao login is interactive; agents and workers need non-interactive OpenBao or ESO.
• WP-0015 in-cluster CronJobs are suspended: cluster pods cannot reach GitHub or Alpine CDN to obtain age.

Pattern to copy (ACTIVITY-WP-0020)

ActivityDefinition (cron)
  → shell context_source on activity-core worker
  → hostPath /opt/railiance-platform (or apps) CLI
  → credentials via actcore-runtime-secret / ESO
  → State Hub progress evidence sink
  → make automation-status / inventory can see the schedule

Reference: activity-core/activity-definitions/weekly-forgejo-package-prune.md.

Target architecture

activity-core Temporal schedule (railiance01)
        │
        ▼
shell resolver: cnpg_option_a_backup / forgejo_backup
        │
        ▼
hostPath tools (static age + kubectl + curl OR prebuilt image side-car script)
  + kubeconfig: railiance01 (local) + CoulombCore (remote cert/tunnel)
  + OpenBao/ESO: NC_WEBDAV_TOKEN, NC_WEBDAV_URL, AGE_PUBLIC_KEY
        │
        ├─► pg_dump / forgejo dump → age → Nextcloud WebDAV
        └─► patch ConfigMap cnpg-option-a-status (+ optional State Hub event)

Acceptance for the workplan as a whole: RPO 24h holds when the operator workstation is powered off; make cnpg-backup-status remains healthy; schedule appears in activity-core automation inventory; restore drill from an activity-core-produced artifact passes.

Repo ownership (do not invent ownership)

Concern	Owner repo
S5 handoff, status Make targets, app evidence, this workplan	railiance-apps
CNPG/Forgejo dump CLI, hostPath packaging, OpenBao lane policy for workers	railiance-platform
ActivityDefinition, shell resolver, Temporal schedule, worker mounts/secrets	activity-core
SSH certs for remote kube if needed	ops-warden (sign only)
Tunnels	ops-bridge

This workplan coordinates cross-repo work; implement in the owning repo and cite commits/evidence here. Do not hand-register hub tasks — write files and statehub fix-consistency.

---

Task: Inventory production-of-record DB topology

id: RAILIANCE-WP-0016-T01
status: todo
priority: high

Document which CNPG instances are production-of-record for each consumer DB across CoulombCore vs railiance01 (especially net-kingdom-pg and state-hub-db dual hosting). Produce a table in docs/ or STATE.md: cluster, host, databases, consumer, backup required Y/N.

Done when: signed-off inventory exists; no ambiguous double-backup or missed instance remains.

Task: Package offline-capable backup runner for railiance01

id: RAILIANCE-WP-0016-T02
status: todo
priority: high

Deliver a runner that does not require workstation or internet package install at job time:

• Static age (and friends) vendored under hostPath tools tree and/or a private OCI image on Forgejo registry with kubectl+age+curl+bash.
• CLI entrypoints covering: multi-cluster logical dump (Option A), forgejo dump path (platform), status CM update, non-secret JSON evidence.
• Prefer extending railiance-platform lib/tools so activity-core only shells out (mirror forgejo-package-prune).

Done when: on railiance01 (or a dry-run host with same mounts), one command produces age-encrypted artifacts without apk/github.com at runtime.

Task: Non-interactive offsite credentials for activity-core worker

id: RAILIANCE-WP-0016-T03
status: todo
priority: high
needs_human: true

Wire NC_WEBDAV_TOKEN / NC_WEBDAV_URL / AGE_PUBLIC_KEY into the activity-core worker on railiance01 without interactive OIDC:

• Prefer ExternalSecret → actcore-runtime-secret (pattern: forgejo-admin ESO).
• OpenBao policy for the worker role; do not put AGE_PRIVATE_KEY on the worker (recovery escrow stays operator/OpenBao only).
• Document rotation via CCR / warden route; no secrets in Git.

Done when: worker pod can upload a tiny probe object to Nextcloud and env lengths are non-zero without operator login.

Task: CoulombCore API reachability from railiance01 worker

id: RAILIANCE-WP-0016-T04
status: todo
priority: high

Backup of CoulombCore clusters (apps-pg, …) from railiance01 requires a stable kubeconfig + auth (ops-warden cert + ops-bridge tunnel, or in-cluster equivalent). Options to evaluate (pick one, record decision):

1. Worker host kubeconfig for CoulombCore with long-lived cert refresh hook.
2. activity-core (or a thin agent) also on CoulombCore for local dumps only.
3. CNPG-native path later (out of scope unless re-decided).

Done when: non-interactive kubectl get cluster -n databases against CoulombCore succeeds from the same environment that will run the schedule.

Task: activity-core ActivityDefinition + shell resolver

id: RAILIANCE-WP-0016-T05
status: todo
priority: high

In activity-core:

1. Shell resolver e.g. cnpg_option_a_backup (and optionally forgejo_backup) analogous to forgejo_package_prune.
2. ActivityDefinition markdown with trigger.type: cron (target ~02:30 UTC daily; weekly retention on Sundays).
3. make sync / schedule reconcile on railiance01; enable only after T02–T04.
4. Evidence sink → State Hub progress (non-secret summary).
5. Visible in make automation-status / inventory.

Done when: Temporal schedule exists, one forced run succeeds, inventory lists the automation as enabled.

Task: Cut over health definition and retire workstation dependency

id: RAILIANCE-WP-0016-T06
status: todo
priority: medium

In railiance-apps:

• Set cnpg-option-a-schedule mode to activity-core (or dual-run during soak).
• cnpg-backup-status accepts activity-core last-success (status CM and/or hub evidence age).
• Document that workstation cron is stopgap only; remove from operator “required” path after soak.
• Optionally un-suspend in-cluster CronJobs only if T02 image path lands in-cluster; otherwise leave suspended.

Done when: docs + status tool describe activity-core as primary; workstation optional.

Task: Soak, restore drill from automated artifact, finish

id: RAILIANCE-WP-0016-T07
status: todo
priority: high

• ≥3 consecutive successful scheduled runs (or 7 if aligning forgejo promotion gate language).
• Restore drill from an artifact produced by activity-core (not workstation one-off), row-count or equivalent gate, evidence JSON under docs/evidence/.
• Update STATE.md; archive/finish this workplan; optional message to activity-core / platform owners if residual tasks remain in their repos.

Done when: RPO/RTO evidence recorded; workstation powered-off scenario is credible; workplan finished.

---

Out of scope

• Switching Option A → barman ObjectStore / CNPG ScheduledBackup (Phase 2; only if explicitly re-decided).
• Media/PVC app data backups (vergage media still disabled).
• Rotating EXPOSED taint history on the offsite lane (platform CCR ops), except as a dependency note if worker fetch is blocked.

Suggested implementation order

T01 inventory → T02 runner package → T03 creds + T04 kube reachability (parallel)
    → T05 activity-core definition → T06 cutover docs/status → T07 soak + drill

References

• STATE.md (this repo) — current posture
• workplans/RAILIANCE-WP-0015-… — interim Option A
• docs/evidence/cnpg-option-a-backup-20260722.json
• activity-core/activity-definitions/weekly-forgejo-package-prune.md
• railiance-platform/docs/forgejo-backup.md
• activity-core/docs/runbook.md (hostPath + ESO patterns)