railiance-apps/workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md

---
id: RAILIANCE-WP-0016
type: workplan
title: "railiance01 + activity-core unattended CNPG/Forgejo backup automation"
domain: financials
repo: railiance-apps
status: proposed
owner: codex
topic_slug: railiance
created: "2026-07-22"
updated: "2026-07-22"
---

# railiance01 + activity-core unattended backup automation

**Predecessor:** `RAILIANCE-WP-0015` (finished) delivered Option A tooling and a
**healthy** `cnpg-backup-status` on CoulombCore, but unattended execution still
depends on a **workstation** (crontab + interactive OpenBao OIDC + laptop
uptime). That is not an acceptable durability control plane.

**Goal:** Daily (and weekly retention) encrypted offsite backups for production
CNPG/Forgejo data run **without any workstation** — scheduled and observed by
**activity-core on railiance01**, following the proven
`weekly-forgejo-package-prune` pattern.

## Context and constraints

### Topology (2026-07-22)

| Host | Clusters / data |
| --- | --- |
| CoulombCore | `apps-pg` (vergage, core_hub, …), `gitea-db`, `net-kingdom-pg`, `state-hub-db` |
| railiance01 | `forgejo-db`, `net-kingdom-pg`, `state-hub-db` (+ activity-core, Forgejo) |

NK and state-hub exist on **both** hosts — WP must inventory which instance is
production-of-record per consumer before scheduling dumps.

### Why workstation fails as control plane

- Laptop sleep / travel / network breaks RPO 24h.
- OIDC `bao login` is interactive; agents and workers need non-interactive
  OpenBao or ESO.
- WP-0015 in-cluster CronJobs are **suspended**: cluster pods cannot reach
  GitHub or Alpine CDN to obtain `age`.

### Pattern to copy (ACTIVITY-WP-0020)

```text
ActivityDefinition (cron)
  → shell context_source on activity-core worker
  → hostPath /opt/railiance-platform (or apps) CLI
  → credentials via actcore-runtime-secret / ESO
  → State Hub progress evidence sink
  → make automation-status / inventory can see the schedule
```

Reference: `activity-core/activity-definitions/weekly-forgejo-package-prune.md`.

### Target architecture

```text
activity-core Temporal schedule (railiance01)
        │
        ▼
shell resolver: cnpg_option_a_backup / forgejo_backup
        │
        ▼
hostPath tools (static age + kubectl + curl OR prebuilt image side-car script)
  + kubeconfig: railiance01 (local) + CoulombCore (remote cert/tunnel)
  + OpenBao/ESO: NC_WEBDAV_TOKEN, NC_WEBDAV_URL, AGE_PUBLIC_KEY
        │
        ├─► pg_dump / forgejo dump → age → Nextcloud WebDAV
        └─► patch ConfigMap cnpg-option-a-status (+ optional State Hub event)
```

**Acceptance for the workplan as a whole:** RPO 24h holds when the operator
workstation is powered off; `make cnpg-backup-status` remains healthy; schedule
appears in activity-core automation inventory; restore drill from an
activity-core-produced artifact passes.

### Repo ownership (do not invent ownership)

| Concern | Owner repo |
| --- | --- |
| S5 handoff, status Make targets, app evidence, this workplan | **railiance-apps** |
| CNPG/Forgejo dump CLI, hostPath packaging, OpenBao lane policy for workers | **railiance-platform** |
| ActivityDefinition, shell resolver, Temporal schedule, worker mounts/secrets | **activity-core** |
| SSH certs for remote kube if needed | **ops-warden** (sign only) |
| Tunnels | **ops-bridge** |

This workplan **coordinates** cross-repo work; implement in the owning repo and
cite commits/evidence here. Do **not** hand-register hub tasks — write files and
`statehub fix-consistency`.

---

## Task: Inventory production-of-record DB topology

```task
id: RAILIANCE-WP-0016-T01
status: todo
priority: high
```

Document which CNPG instances are production-of-record for each consumer DB
across CoulombCore vs railiance01 (especially `net-kingdom-pg` and
`state-hub-db` dual hosting). Produce a table in `docs/` or STATE.md: cluster,
host, databases, consumer, backup required Y/N.

**Done when:** signed-off inventory exists; no ambiguous double-backup or missed
instance remains.

## Task: Package offline-capable backup runner for railiance01

```task
id: RAILIANCE-WP-0016-T02
status: todo
priority: high
```

Deliver a runner that does **not** require workstation or internet package
install at job time:

- Static `age` (and friends) vendored under hostPath tools tree and/or a
  private OCI image on Forgejo registry with `kubectl`+`age`+`curl`+`bash`.
- CLI entrypoints covering: multi-cluster logical dump (Option A), forgejo dump
  path (platform), status CM update, non-secret JSON evidence.
- Prefer extending `railiance-platform` lib/tools so activity-core only shells out
  (mirror `forgejo-package-prune`).

**Done when:** on railiance01 (or a dry-run host with same mounts), one command
produces age-encrypted artifacts **without** `apk`/`github.com` at runtime.

## Task: Non-interactive offsite credentials for activity-core worker

```task
id: RAILIANCE-WP-0016-T03
status: todo
priority: high
needs_human: true
```

Wire `NC_WEBDAV_TOKEN` / `NC_WEBDAV_URL` / `AGE_PUBLIC_KEY` into the activity-core
worker on railiance01 without interactive OIDC:

- Prefer ExternalSecret → `actcore-runtime-secret` (pattern: forgejo-admin ESO).
- OpenBao policy for the worker role; **do not** put `AGE_PRIVATE_KEY` on the
  worker (recovery escrow stays operator/OpenBao only).
- Document rotation via CCR / warden route; no secrets in Git.

**Done when:** worker pod can upload a tiny probe object to Nextcloud and env
lengths are non-zero without operator login.

## Task: CoulombCore API reachability from railiance01 worker

```task
id: RAILIANCE-WP-0016-T04
status: todo
priority: high
```

Backup of CoulombCore clusters (`apps-pg`, …) from railiance01 requires a
stable kubeconfig + auth (ops-warden cert + ops-bridge tunnel, or in-cluster
equivalent). Options to evaluate (pick one, record decision):

1. Worker host kubeconfig for CoulombCore with long-lived cert refresh hook.
2. activity-core (or a thin agent) also on CoulombCore for local dumps only.
3. CNPG-native path later (out of scope unless re-decided).

**Done when:** non-interactive `kubectl get cluster -n databases` against
CoulombCore succeeds from the same environment that will run the schedule.

## Task: activity-core ActivityDefinition + shell resolver

```task
id: RAILIANCE-WP-0016-T05
status: todo
priority: high
```

In **activity-core**:

1. Shell resolver e.g. `cnpg_option_a_backup` (and optionally `forgejo_backup`)
   analogous to `forgejo_package_prune`.
2. ActivityDefinition markdown with `trigger.type: cron` (target ~02:30 UTC
   daily; weekly retention on Sundays).
3. `make sync` / schedule reconcile on railiance01; enable only after T02–T04.
4. Evidence sink → State Hub progress (non-secret summary).
5. Visible in `make automation-status` / inventory.

**Done when:** Temporal schedule exists, one forced run succeeds, inventory lists
the automation as enabled.

## Task: Cut over health definition and retire workstation dependency

```task
id: RAILIANCE-WP-0016-T06
status: todo
priority: medium
```

In **railiance-apps**:

- Set `cnpg-option-a-schedule` `mode` to `activity-core` (or dual-run during
  soak).
- `cnpg-backup-status` accepts activity-core last-success (status CM and/or
  hub evidence age).
- Document that workstation cron is **stopgap only**; remove from operator
  “required” path after soak.
- Optionally un-suspend in-cluster CronJobs **only if** T02 image path lands
  in-cluster; otherwise leave suspended.

**Done when:** docs + status tool describe activity-core as primary; workstation
optional.

## Task: Soak, restore drill from automated artifact, finish

```task
id: RAILIANCE-WP-0016-T07
status: todo
priority: high
```

- ≥3 consecutive successful scheduled runs (or 7 if aligning forgejo promotion
  gate language).
- Restore drill from an artifact produced by activity-core (not workstation
  one-off), row-count or equivalent gate, evidence JSON under `docs/evidence/`.
- Update `STATE.md`; archive/finish this workplan; optional message to
  activity-core / platform owners if residual tasks remain in their repos.

**Done when:** RPO/RTO evidence recorded; workstation powered-off scenario is
credible; workplan `finished`.

---

## Out of scope

- Switching Option A → barman ObjectStore / CNPG `ScheduledBackup` (Phase 2;
  only if explicitly re-decided).
- Media/PVC app data backups (vergage media still disabled).
- Rotating EXPOSED taint history on the offsite lane (platform CCR ops), except
  as a dependency note if worker fetch is blocked.

## Suggested implementation order

```text
T01 inventory → T02 runner package → T03 creds + T04 kube reachability (parallel)
    → T05 activity-core definition → T06 cutover docs/status → T07 soak + drill
```

## References

- `STATE.md` (this repo) — current posture
- `workplans/RAILIANCE-WP-0015-…` — interim Option A
- `docs/evidence/cnpg-option-a-backup-20260722.json`
- `activity-core/activity-definitions/weekly-forgejo-package-prune.md`
- `railiance-platform/docs/forgejo-backup.md`
- `activity-core/docs/runbook.md` (hostPath + ESO patterns)