coulomb/railiance-apps
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
railiance-apps/workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md
tegwick 04626b6335
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
Details
CI Smoke / container-smoke (push) Successful in 1s
Details
chore(consistency): write state-hub IDs into RAILIANCE-WP-0016 
C-06 registration IDs from fix-consistency.
2026-07-22 19:36:36 +02:00

9.4 KiB
Raw Blame History

id type title domain repo status owner topic_slug created updated state_hub_workstream_id
RAILIANCE-WP-0016 workplan railiance01 + activity-core unattended CNPG/Forgejo backup automation financials railiance-apps proposed codex railiance 2026-07-22 2026-07-22 0c8ddbc9-3740-4e0a-acb2-901e050ab242

railiance01 + activity-core unattended backup automation

Predecessor: RAILIANCE-WP-0015 (finished) delivered Option A tooling and a healthy cnpg-backup-status on CoulombCore, but unattended execution still depends on a workstation (crontab + interactive OpenBao OIDC + laptop uptime). That is not an acceptable durability control plane.

Goal: Daily (and weekly retention) encrypted offsite backups for production CNPG/Forgejo data run without any workstation — scheduled and observed by activity-core on railiance01, following the proven weekly-forgejo-package-prune pattern.

Context and constraints

Topology (2026-07-22)

Host Clusters / data
CoulombCore apps-pg (vergage, core_hub, …), gitea-db, net-kingdom-pg, state-hub-db
railiance01 forgejo-db, net-kingdom-pg, state-hub-db (+ activity-core, Forgejo)

NK and state-hub exist on both hosts — WP must inventory which instance is production-of-record per consumer before scheduling dumps.

Why workstation fails as control plane

  • Laptop sleep / travel / network breaks RPO 24h.
  • OIDC bao login is interactive; agents and workers need non-interactive OpenBao or ESO.
  • WP-0015 in-cluster CronJobs are suspended: cluster pods cannot reach GitHub or Alpine CDN to obtain age.

Pattern to copy (ACTIVITY-WP-0020)

ActivityDefinition (cron)
  → shell context_source on activity-core worker
  → hostPath /opt/railiance-platform (or apps) CLI
  → credentials via actcore-runtime-secret / ESO
  → State Hub progress evidence sink
  → make automation-status / inventory can see the schedule

Reference: activity-core/activity-definitions/weekly-forgejo-package-prune.md.

Target architecture

activity-core Temporal schedule (railiance01)
        │
        ▼
shell resolver: cnpg_option_a_backup / forgejo_backup
        │
        ▼
hostPath tools (static age + kubectl + curl OR prebuilt image side-car script)
  + kubeconfig: railiance01 (local) + CoulombCore (remote cert/tunnel)
  + OpenBao/ESO: NC_WEBDAV_TOKEN, NC_WEBDAV_URL, AGE_PUBLIC_KEY
        │
        ├─► pg_dump / forgejo dump → age → Nextcloud WebDAV
        └─► patch ConfigMap cnpg-option-a-status (+ optional State Hub event)

Acceptance for the workplan as a whole: RPO 24h holds when the operator workstation is powered off; make cnpg-backup-status remains healthy; schedule appears in activity-core automation inventory; restore drill from an activity-core-produced artifact passes.

Repo ownership (do not invent ownership)

Concern Owner repo
S5 handoff, status Make targets, app evidence, this workplan railiance-apps
CNPG/Forgejo dump CLI, hostPath packaging, OpenBao lane policy for workers railiance-platform
ActivityDefinition, shell resolver, Temporal schedule, worker mounts/secrets activity-core
SSH certs for remote kube if needed ops-warden (sign only)
Tunnels ops-bridge

This workplan coordinates cross-repo work; implement in the owning repo and cite commits/evidence here. Do not hand-register hub tasks — write files and statehub fix-consistency.

Task: Inventory production-of-record DB topology

id: RAILIANCE-WP-0016-T01
status: todo
priority: high
state_hub_task_id: "acf23292-438b-44c0-a6cd-87327b54d743"

Document which CNPG instances are production-of-record for each consumer DB across CoulombCore vs railiance01 (especially net-kingdom-pg and state-hub-db dual hosting). Produce a table in docs/ or STATE.md: cluster, host, databases, consumer, backup required Y/N.

Done when: signed-off inventory exists; no ambiguous double-backup or missed instance remains.

Task: Package offline-capable backup runner for railiance01

id: RAILIANCE-WP-0016-T02
status: todo
priority: high
state_hub_task_id: "35e81dc9-4f86-4a8f-8857-faef4117eeb6"

Deliver a runner that does not require workstation or internet package install at job time:

  • Static age (and friends) vendored under hostPath tools tree and/or a private OCI image on Forgejo registry with kubectl+age+curl+bash.
  • CLI entrypoints covering: multi-cluster logical dump (Option A), forgejo dump path (platform), status CM update, non-secret JSON evidence.
  • Prefer extending railiance-platform lib/tools so activity-core only shells out (mirror forgejo-package-prune).

Done when: on railiance01 (or a dry-run host with same mounts), one command produces age-encrypted artifacts without apk/github.com at runtime.

Task: Non-interactive offsite credentials for activity-core worker

id: RAILIANCE-WP-0016-T03
status: todo
priority: high
needs_human: true
state_hub_task_id: "3bcb6a70-7b74-47a3-8afe-3921fdccd893"

Wire NC_WEBDAV_TOKEN / NC_WEBDAV_URL / AGE_PUBLIC_KEY into the activity-core worker on railiance01 without interactive OIDC:

  • Prefer ExternalSecret → actcore-runtime-secret (pattern: forgejo-admin ESO).
  • OpenBao policy for the worker role; do not put AGE_PRIVATE_KEY on the worker (recovery escrow stays operator/OpenBao only).
  • Document rotation via CCR / warden route; no secrets in Git.

Done when: worker pod can upload a tiny probe object to Nextcloud and env lengths are non-zero without operator login.

Task: CoulombCore API reachability from railiance01 worker

id: RAILIANCE-WP-0016-T04
status: todo
priority: high
state_hub_task_id: "b8d2dd05-31a0-4fd7-bdef-27eb686e8f8a"

Backup of CoulombCore clusters (apps-pg, …) from railiance01 requires a stable kubeconfig + auth (ops-warden cert + ops-bridge tunnel, or in-cluster equivalent). Options to evaluate (pick one, record decision):

  1. Worker host kubeconfig for CoulombCore with long-lived cert refresh hook.
  2. activity-core (or a thin agent) also on CoulombCore for local dumps only.
  3. CNPG-native path later (out of scope unless re-decided).

Done when: non-interactive kubectl get cluster -n databases against CoulombCore succeeds from the same environment that will run the schedule.

Task: activity-core ActivityDefinition + shell resolver

id: RAILIANCE-WP-0016-T05
status: todo
priority: high
state_hub_task_id: "547b3acb-94c2-46e0-a7da-ef51ee0c3c8d"

In activity-core:

  1. Shell resolver e.g. cnpg_option_a_backup (and optionally forgejo_backup) analogous to forgejo_package_prune.
  2. ActivityDefinition markdown with trigger.type: cron (target ~02:30 UTC daily; weekly retention on Sundays).
  3. make sync / schedule reconcile on railiance01; enable only after T02–T04.
  4. Evidence sink → State Hub progress (non-secret summary).
  5. Visible in make automation-status / inventory.

Done when: Temporal schedule exists, one forced run succeeds, inventory lists the automation as enabled.

Task: Cut over health definition and retire workstation dependency

id: RAILIANCE-WP-0016-T06
status: todo
priority: medium
state_hub_task_id: "d6cab5a4-3cdd-4033-b21e-88db02197472"

In railiance-apps:

  • Set cnpg-option-a-schedule mode to activity-core (or dual-run during soak).
  • cnpg-backup-status accepts activity-core last-success (status CM and/or hub evidence age).
  • Document that workstation cron is stopgap only; remove from operator “required” path after soak.
  • Optionally un-suspend in-cluster CronJobs only if T02 image path lands in-cluster; otherwise leave suspended.

Done when: docs + status tool describe activity-core as primary; workstation optional.

Task: Soak, restore drill from automated artifact, finish

id: RAILIANCE-WP-0016-T07
status: todo
priority: high
state_hub_task_id: "f331bdcd-0e76-4bc0-b85a-1bb1227a876b"
  • ≥3 consecutive successful scheduled runs (or 7 if aligning forgejo promotion gate language).
  • Restore drill from an artifact produced by activity-core (not workstation one-off), row-count or equivalent gate, evidence JSON under docs/evidence/.
  • Update STATE.md; archive/finish this workplan; optional message to activity-core / platform owners if residual tasks remain in their repos.

Done when: RPO/RTO evidence recorded; workstation powered-off scenario is credible; workplan finished.

Out of scope

  • Switching Option A → barman ObjectStore / CNPG ScheduledBackup (Phase 2; only if explicitly re-decided).
  • Media/PVC app data backups (vergage media still disabled).
  • Rotating EXPOSED taint history on the offsite lane (platform CCR ops), except as a dependency note if worker fetch is blocked.

Suggested implementation order

T01 inventory → T02 runner package → T03 creds + T04 kube reachability (parallel)
    → T05 activity-core definition → T06 cutover docs/status → T07 soak + drill

References

  • STATE.md (this repo) — current posture
  • workplans/RAILIANCE-WP-0015-… — interim Option A
  • docs/evidence/cnpg-option-a-backup-20260722.json
  • activity-core/activity-definitions/weekly-forgejo-package-prune.md
  • railiance-platform/docs/forgejo-backup.md
  • activity-core/docs/runbook.md (hostPath + ESO patterns)