coulomb/railiance-apps
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

RAILIANCE-WP-0016: promote active; inventory and activity-core cutover prep
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
Details
CI Smoke / container-smoke (push) Successful in 2s
Details

Browse source 
Mark workplan active with T01/T02 done, document topology, extend status
for activity-core mode, and wire Make targets to the platform multi-host
backup CLI. T03 remains operator-blocked on ESO token.
This commit is contained in:
tegwick 2026-07-22 19:50:59 +02:00
parent 04626b6335
commit c202fbf7be
6 changed files with 180 additions and 309 deletions
Download patch file Download diff file Expand all files Collapse all files

16
Makefile
View file

@ -173,6 +173,22 @@ cnpg-logical-backup: check-production-kubeconfig ## Logical dump + age + Nextclo
KUBECONFIG="$(PRODUCTION_KUBECONFIG)" PLATFORM_REPO="$(PLATFORM_REPO)" RAILIANCE_BACKUP_DRY_RUN=0 \ KUBECONFIG="$(PRODUCTION_KUBECONFIG)" PLATFORM_REPO="$(PLATFORM_REPO)" RAILIANCE_BACKUP_DRY_RUN=0 \
tools/cnpg-logical-backup.sh tools/cnpg-logical-backup.sh
##@ Option A multi-host (platform CLI / activity-core)
cnpg-option-a-backup-dry-run: ## Platform multi-host Option A dump+age (JSON, no upload)
chmod +x "$(PLATFORM_REPO)/tools/cmd/cnpg-option-a-backup"
KUBECONFIG_CORE="$(COULOMBCORE_KUBECONFIG)" KUBECONFIG_R01="$(RAILIANCE01_KUBECONFIG)" \
RAILIANCE_BACKUP_DRY_RUN=1 "$(PLATFORM_REPO)/tools/cmd/cnpg-option-a-backup"
cnpg-option-a-backup: ## Platform multi-host Option A dump+age+upload (JSON)
chmod +x "$(PLATFORM_REPO)/tools/cmd/cnpg-option-a-backup"
@bao kv metadata get platform/workloads/railiance/backup/offsite-lane >/dev/null 2>&1 || { \
echo "ERROR: OpenBao backup-lane auth missing or set RAILIANCE_BACKUP_NC_TOKEN." >&2; \
exit 1; \
}
KUBECONFIG_CORE="$(COULOMBCORE_KUBECONFIG)" KUBECONFIG_R01="$(RAILIANCE01_KUBECONFIG)" \
RAILIANCE_BACKUP_DRY_RUN=0 "$(PLATFORM_REPO)/tools/cmd/cnpg-option-a-backup"
apps-pg-backup-dry-run: check-production-kubeconfig ## Logical pg_dump + age encrypt for vergabe_db (no upload) apps-pg-backup-dry-run: check-production-kubeconfig ## Logical pg_dump + age encrypt for vergabe_db (no upload)
chmod +x tools/apps-pg-backup-dry-run.sh chmod +x tools/apps-pg-backup-dry-run.sh
KUBECONFIG="$(PRODUCTION_KUBECONFIG)" PLATFORM_REPO="$(PLATFORM_REPO)" RAILIANCE_BACKUP_DRY_RUN=1 \ KUBECONFIG="$(PRODUCTION_KUBECONFIG)" PLATFORM_REPO="$(PLATFORM_REPO)" RAILIANCE_BACKUP_DRY_RUN=1 \

186
STATE.md
View file

@ -4,164 +4,74 @@
**Domain:** financials · **Repo:** railiance-apps **Domain:** financials · **Repo:** railiance-apps
**Topic ID:** `ca369340-a64e-442e-98f1-a4fa7dc74a38` **Topic ID:** `ca369340-a64e-442e-98f1-a4fa7dc74a38`
This file is the human/agent snapshot of **where the repo stands**. Prefer it over
chat history for orientation. Generated custodian brief is `.custodian-brief.md`
(do not edit). Formal work structure lives in `workplans/`.
--- ---
## One-line posture ## One-line posture
S5 app releases (vergabe, hubs, Forgejo consumer surface) are operational; **CNPG S5 releases operational; **CNPG Option A is healthy on CoulombCore** (WP-0015).
Option A logical backups exist and currently report healthy**, but unattended **Unattended control plane migration to railiance01 + activity-core is in
execution still depends on a **workstation path** and must move to **railiance01 + progress (WP-0016)** — code/manifests landed; **blocked on operator ESO token
activity-core** so the laptop is not a durability SPOF. re-mint + CoulombCore kubeconfig on railiance01 host**.
--- ---
## Clusters (as of 2026-07-22) ## Workplans
| Host | Kubeconfig | CNPG clusters in `databases` | Role | | ID | Status | Notes |
| --- | --- | --- | --- |
| **CoulombCore** | `~/.kube/config` | `apps-pg`, `gitea-db`, `net-kingdom-pg`, `state-hub-db` | S5 apps DBs, legacy gitea-db, NK + state-hub instances |
| **railiance01** | `~/.kube/config-hosteurope` | `forgejo-db`, `net-kingdom-pg`, `state-hub-db` | Forgejo DB; NK + state-hub instances (may differ from Core) |
`PRODUCTION_KUBECONFIG` defaults to CoulombCore. Forgejo tooling uses railiance01.
---
## Backup durability (RAILIANCE-WP-0015 — **finished**)
### Goal met
Drive `make cnpg-backup-status` **degraded → healthy** with Option A (age-encrypted
logical dump → Nextcloud WebDAV), not barman `ScheduledBackup`.
### What is live today
| Item | State |
| --- | --- |
| OpenBao lane `CCR-2026-0004` / `railiance-backup-offsite-lane` | **resolvable**, fields present |
| Secret `databases/cnpg-backup-offsite` (CoulombCore) | **present** (`NC_WEBDAV_*`, `AGE_PUBLIC_KEY` only) |
| ConfigMap `cnpg-option-a-schedule` | **mode=workstation-cron** @ `30 2 * * *` |
| ConfigMap `cnpg-option-a-status` | last-success timestamps for all four Core clusters (2026-07-22) |
| CronJobs `cnpg-logical-backup-*` | **suspended** (cluster egress cannot install `age`) |
| Workstation runner `tools/cnpg-logical-backup.sh` | **works** (upload + status CM) |
| Restore drill | apps-pg/`vergage_db` age artifact **195/195** |
| Evidence | `docs/evidence/cnpg-option-a-backup-20260722.json` |
### What is *not* good enough
1. **Workstation SPOF** — daily unattended path is “someone’s machine has crontab +
bao OIDC + kubeconfig”. Laptop sleep/off breaks RPO.
2. **In-cluster CronJobs blocked** — pods cannot reach GitHub or Alpine CDN to get
`age`; no prebuilt image with `kubectl`+`age`+`curl` yet.
3. **Split brain topology** — Option A runner today targets **CoulombCore** four
clusters only; **forgejo-db** (railiance01) still uses platform `make forgejo-backup`
(also historically workstation-cron).
4. **No activity-core schedule** — unlike weekly Forgejo package prune, backups are
not Temporal/`ActivityDefinition` managed, so automation inventory and
`automation-status` cannot own them.
5. **Credential delivery** — offsite lane is human OIDC on workstation; worker needs
non-interactive OpenBao/ESO (or sealed secret) path.
### Operator residual (manual)
```cron
# Not yet proven installed on a always-on host — install on railiance01 or leave as stopgap
30 2 * * * cd $HOME/railiance-apps && make cnpg-logical-backup >>$HOME/.cache/railiance/backups/cnpg/cron.log 2>&1
```
---
## Adjacent finished work
| Workplan | Outcome |
| --- | --- |
| RAILIANCE-WP-0013 | Phase 1 apps-pg/`vergage_db` backup + restore drill (interim S5 gate) |
| RAILIANCE-WP-0015 | Fleet Option A tooling + healthy status on CoulombCore (workstation-backed) |
| RAIL-HO-WP-0005 | Forgejo prod migration; descoped residual backup → WP-0015 |
| ACTIVITY-WP-0020 | Weekly forgejo package prune via **activity-core** on railiance01 (pattern to copy) |
---
## Next major strand
**Move unattended Option A (+ forgejo-backup) to railiance01, scheduled by
activity-core — no workstation required.**
Tracking workplan: **`RAILIANCE-WP-0016`**
`workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md`
Design sketch (see workplan for tasks):
```text
┌─────────────────────────────┐
│ activity-core (railiance01) │
│ Temporal cron ActivityDef │
└─────────────┬───────────────┘
│ shell context
▼
┌─────────────────────────────┐
│ hostPath /opt/railiance-* │
│ backup CLI (age+kubectl) │
│ OpenBao/ESO offsite secrets │
└─────────────┬───────────────┘
┌──────────────────┼──────────────────┐
▼ ▼ ▼
CoulombCore API railiance01 API Nextcloud WebDAV
(apps-pg, …) (forgejo-db, …) (age artifacts)
```
**Proven pattern:** `weekly-forgejo-package-prune` ActivityDefinition +
`/opt/railiance-platform` hostPath + ESO credentials + State Hub evidence sink.
---
## Active / proposed workplans
| ID | Status | Title |
| --- | --- | --- | | --- | --- | --- |
| RAILIANCE-WP-0015 | **finished** | CNPG Option A coverage → healthy (workstation interim) | | RAILIANCE-WP-0015 | **finished** | Option A tooling; workstation stopgap schedule |
| RAILIANCE-WP-0016 | **proposed** | railiance01 + activity-core unattended backup automation | | RAILIANCE-WP-0016 | **active** | activity-core unattended path — T01/T02 done; T03 human; T04–T07 open |
(No other active workplans in hub as of last brief.) Hub WP-0016: `0c8ddbc9-3740-4e0a-acb2-901e050ab242`
--- ---
## Quick operator commands ## WP-0016 progress snapshot
**Done**
- Topology inventory: `docs/cnpg-backup-topology-inventory.md`
- Platform multi-host CLI: `~/railiance-platform/tools/cmd/cnpg-option-a-backup`
- Static `age` vendored; kubectl via install script
- Dry-run: `r01-forgejo-db` overall=ok
- activity-core: resolver, definition (disabled), ESO CR, RBAC, worker yaml mounts
- Unit tests: 3 passed (`test_cnpg_option_a_backup_resolver.py`)
**Blocked / operator**
1. Re-mint `openbao-activity-core-eso-token` with
`workload-kv-read-railiance-backup-offsite-lane`
→ ExternalSecret `actcore-backup-offsite` currently **SecretSyncedError**
2. On railiance01 host: `install-cnpg-backup-vendor-tools`; ensure
`~/.kube/config` (Core) + `config-hosteurope` (R01)
3. Roll worker image/code + hostPath kubeconfigs; enable
`daily-cnpg-option-a-backup`; forced run + soak
---
## Clusters (reminder)
| Host | POR backup targets |
| --- | --- |
| CoulombCore | apps-pg, gitea-db, net-kingdom-pg, state-hub-db |
| railiance01 | forgejo-db, net-kingdom-pg, state-hub-db (dual with Core for NK/SH) |
---
## Quick commands
```bash ```bash
# Health (CoulombCore)
make cnpg-backup-status make cnpg-backup-status
make cnpg-option-a-backup-dry-run # platform multi-host JSON
# Manual Option A run (still workstation / OpenBao OIDC today) # After ESO + host wiring:
bao login -method=oidc -path=netkingdom role=railiance-backup-workload-kv-read # enable activity-core definition; make automation-status
make cnpg-logical-backup
# Secret re-materialize
make cnpg-backup-offsite-secret-apply
``` ```
--- ---
## Risks / open questions
| Risk | Mitigation direction (WP-0016) |
| --- | --- |
| Dual NK/state-hub DB instances on two hosts | Inventory which is production-of-record per consumer |
| CoulombCore access from railiance01 | Dedicated kubeconfig + ops-warden cert / tunnel for worker |
| Age binary in restricted network | Vendor static binary in image or hostPath tools tree |
| High-risk WebDAV + age keys in worker | ESO from OpenBao; never private key in Git; agent boundary |
| RPO if activity-core worker down | Temporal schedule health + status CM freshness alerts |
---
## Related docs ## Related docs
- `docs/cnpg-backup-topology-inventory.md`
- `docs/app-data-backup-restore-handoff.md` - `docs/app-data-backup-restore-handoff.md`
- `docs/credential-routing-railiance-apps.md` - `workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md`
- `manifests/cnpg-option-a-backup.yaml` - `railiance-platform/docs/cnpg-option-a-backup.md`
- `docs/evidence/cnpg-option-a-backup-20260722.json`
- activity-core: `activity-definitions/weekly-forgejo-package-prune.md`, `docs/runbook.md`
- platform: `docs/forgejo-backup.md`

43
docs/cnpg-backup-topology-inventory.md Normal file
View file

@ -0,0 +1,43 @@
# CNPG backup topology inventory
**Workplan:** RAILIANCE-WP-0016-T01
**Recorded:** 2026-07-22
**Method:** live `kubectl get cluster` + `psql` datname listing on both hosts.
## Hosts
| Host | Kubeconfig (operator workstation) | API |
| --- | --- | --- |
| CoulombCore | `~/.kube/config` | tunnel `k3s-api-coulombcore` |
| railiance01 | `~/.kube/config-hosteurope` | local / hosteurope |
## Production-of-record (POR) decisions
| Target id | Host | Cluster | Databases | POR? | Consumer / notes |
| --- | --- | --- | --- | --- | --- |
| `core-apps-pg` | CoulombCore | `apps-pg` | `apps_meta`, `core_hub`, `core_hub_staging`, `vergabe_db` | **Yes** | S5 apps (vergabe, core-hub) — only instance |
| `core-gitea-db` | CoulombCore | `gitea-db` | `gitea` | **Yes** | Legacy Gitea; still live |
| `r01-forgejo-db` | railiance01 | `forgejo-db` | `forgejo` | **Yes** | Forgejo production — only instance |
| `core-net-kingdom-pg` | CoulombCore | `net-kingdom-pg` | `interhub`, `privacyidea_db` | **Yes (dual)** | Backup both until app DSN audit confirms single host |
| `r01-net-kingdom-pg` | railiance01 | `net-kingdom-pg` | `interhub`, `privacyidea_db`, `probe` | **Yes (dual)** | Extra `probe` DB on R01 only |
| `core-state-hub-db` | CoulombCore | `state-hub-db` | `state_hub` | **Yes (dual)** | Hub may run against either instance historically |
| `r01-state-hub-db` | railiance01 | `state-hub-db` | `state_hub` | **Yes (dual)** | activity-core / local hub proximity |
### Dual-host policy
Until a consumer DSN audit proves a single POR for Net Kingdom and State Hub:
- **Backup both** Core and R01 instances (different Nextcloud prefixes:
`core-*` vs `r01-*`).
- Restore drills must name the **host** + cluster, not only the DB name.
## Out of backup scope (this workplan)
- CNPG system DB `postgres`
- Temporary drill DBs
- App media PVCs (vergage media disabled)
## Runner mapping
Implemented in `railiance-platform/tools/cmd/cnpg-option-a-backup` default target
list. Override with `CNPG_BACKUP_TARGETS=core-apps-pg,r01-forgejo-db`.

5
manifests/cnpg-option-a-backup.yaml
View file

@ -29,13 +29,16 @@ metadata:
railiance.apps/backup-lane: option-a railiance.apps/backup-lane: option-a
data: data:
mode: workstation-cron mode: workstation-cron
next_mode: activity-core
schedule: "30 2 * * *" schedule: "30 2 * * *"
command: "cd $HOME/railiance-apps && make cnpg-logical-backup" command: "cd $HOME/railiance-apps && make cnpg-logical-backup"
activity_core_definition: daily-cnpg-option-a-backup
platform_cli: /opt/railiance-platform/tools/cmd/cnpg-option-a-backup
clusters: "apps-pg,gitea-db,net-kingdom-pg,state-hub-db" clusters: "apps-pg,gitea-db,net-kingdom-pg,state-hub-db"
rpo_hours: "24" rpo_hours: "24"
rto_hours: "4" rto_hours: "4"
retention: "14 daily + 4 weekly (Nextcloud operator prune)" retention: "14 daily + 4 weekly (Nextcloud operator prune)"
note: "In-cluster CronJobs suspended until image with age is available without egress installs" note: "WP-0016: activity-core path primary when enabled; workstation-cron is stopgap. In-cluster CronJobs remain suspended."
--- ---
apiVersion: v1 apiVersion: v1
kind: ServiceAccount kind: ServiceAccount

13
tools/cnpg-backup-status.sh
View file

@ -88,11 +88,15 @@ degraded_detail=()
IFS=',' read -r -a expected <<<"$expected_clusters" IFS=',' read -r -a expected <<<"$expected_clusters"
# Workstation schedule counts as a declared unattended plan for all expected clusters # Declared unattended plan: workstation stopgap or activity-core primary
workstation_schedule=0 workstation_schedule=0
activity_core_schedule=0
if [[ "$schedule_mode" == "workstation-cron" ]]; then if [[ "$schedule_mode" == "workstation-cron" ]]; then
workstation_schedule=1 workstation_schedule=1
fi fi
if [[ "$schedule_mode" == "activity-core" ]]; then
activity_core_schedule=1
fi
for cluster in "${clusters[@]}"; do for cluster in "${clusters[@]}"; do
if ! kubectl get cluster "$cluster" -n "$namespace" >/dev/null 2>&1; then if ! kubectl get cluster "$cluster" -n "$namespace" >/dev/null 2>&1; then
@ -148,7 +152,12 @@ for cluster in "${clusters[@]}"; do
if ((workstation_schedule)); then if ((workstation_schedule)); then
schedule_ok=1 schedule_ok=1
schedule_note="${schedule_note:+$schedule_note; }workstation-cron" schedule_note="${schedule_note:+$schedule_note; }workstation-cron"
echo "${cluster}: schedule declaration = workstation-cron" echo "${cluster}: schedule declaration = workstation-cron (stopgap)"
fi
if ((activity_core_schedule)); then
schedule_ok=1
schedule_note="${schedule_note:+$schedule_note; }activity-core"
echo "${cluster}: schedule declaration = activity-core"
fi fi
ts="${last_success[$cluster]:-}" ts="${last_success[$cluster]:-}"

226
workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md
View file

@ -4,7 +4,7 @@ type: workplan
title: "railiance01 + activity-core unattended CNPG/Forgejo backup automation" title: "railiance01 + activity-core unattended CNPG/Forgejo backup automation"
domain: financials domain: financials
repo: railiance-apps repo: railiance-apps
status: proposed status: active
owner: codex owner: codex
topic_slug: railiance topic_slug: railiance
created: "2026-07-22" created: "2026-07-22"
@ -19,248 +19,138 @@ state_hub_workstream_id: "0c8ddbc9-3740-4e0a-acb2-901e050ab242"
depends on a **workstation** (crontab + interactive OpenBao OIDC + laptop depends on a **workstation** (crontab + interactive OpenBao OIDC + laptop
uptime). That is not an acceptable durability control plane. uptime). That is not an acceptable durability control plane.
**Goal:** Daily (and weekly retention) encrypted offsite backups for production **Goal:** Daily encrypted offsite backups for production CNPG data run **without
CNPG/Forgejo data run **without any workstation** — scheduled and observed by any workstation** — scheduled and observed by **activity-core on railiance01**.
**activity-core on railiance01**, following the proven
`weekly-forgejo-package-prune` pattern.
## Context and constraints ## Implementation status (2026-07-22)
### Topology (2026-07-22) | Area | State |
| Host | Clusters / data |
| --- | --- | | --- | --- |
| CoulombCore | `apps-pg` (vergage, core_hub, …), `gitea-db`, `net-kingdom-pg`, `state-hub-db` | | Topology inventory | **done** — `docs/cnpg-backup-topology-inventory.md` |
| railiance01 | `forgejo-db`, `net-kingdom-pg`, `state-hub-db` (+ activity-core, Forgejo) | | Platform CLI + vendor age | **done** — `railiance-platform/tools/cmd/cnpg-option-a-backup` |
| Dry-run forgejo-db | **pass** (JSON overall=ok) |
| activity-core resolver + definition | **landed** (`enabled: false` until creds+kube) |
| ESO ExternalSecret | **applied**; **SecretSyncedError** until ESO token re-mint |
| Worker RBAC databases exec | **applied** on railiance01 |
| Worker kubeconfig hostPath | **in git** (`20-runtime.yaml`); apply/restart pending |
| Schedule enable + soak | **blocked** on operator T03/T04 |
NK and state-hub exist on **both** hosts — WP must inventory which instance is ### Operator unblock (next human steps)
production-of-record per consumer before scheduling dumps.
### Why workstation fails as control plane ```bash
# 1) Re-mint ESO token with backup lane policy (operator OpenBao token)
cd ~/activity-core
OPENBAO_ACTIVITY_CORE_POLICIES="workload-kv-read-issue-core-runtime workload-kv-read-forgejo-admin workload-kv-read-railiance-backup-offsite-lane" \
scripts/openbao-eso-token-apply.sh
kubectl -n activity-core get externalsecret actcore-backup-offsite # expect SecretSynced
- Laptop sleep / travel / network breaks RPO 24h. # 2) On railiance01 host as tegwick: vendor kubectl + ensure kubeconfigs
- OIDC `bao login` is interactive; agents and workers need non-interactive cd ~/railiance-platform && tools/cmd/install-cnpg-backup-vendor-tools
OpenBao or ESO. # Place CoulombCore kubeconfig at ~/.kube/config (R01 already config-hosteurope)
- WP-0015 in-cluster CronJobs are **suspended**: cluster pods cannot reach
GitHub or Alpine CDN to obtain `age`.
### Pattern to copy (ACTIVITY-WP-0020) # 3) Roll worker with updated mounts + new image/code
kubectl apply -f k8s/railiance/20-runtime.yaml # or patch
# rebuild activity-core image with resolver if not live-mounted
```text # 4) Sync definition, enable, force run
ActivityDefinition (cron) # set enabled: true on daily-cnpg-option-a-backup; make sync-activity-definitions sync-schedules
→ shell context_source on activity-core worker
→ hostPath /opt/railiance-platform (or apps) CLI
→ credentials via actcore-runtime-secret / ESO
→ State Hub progress evidence sink
→ make automation-status / inventory can see the schedule
``` ```
Reference: `activity-core/activity-definitions/weekly-forgejo-package-prune.md`.
### Target architecture
```text
activity-core Temporal schedule (railiance01)
│
▼
shell resolver: cnpg_option_a_backup / forgejo_backup
│
▼
hostPath tools (static age + kubectl + curl OR prebuilt image side-car script)
+ kubeconfig: railiance01 (local) + CoulombCore (remote cert/tunnel)
+ OpenBao/ESO: NC_WEBDAV_TOKEN, NC_WEBDAV_URL, AGE_PUBLIC_KEY
│
├─► pg_dump / forgejo dump → age → Nextcloud WebDAV
└─► patch ConfigMap cnpg-option-a-status (+ optional State Hub event)
```
**Acceptance for the workplan as a whole:** RPO 24h holds when the operator
workstation is powered off; `make cnpg-backup-status` remains healthy; schedule
appears in activity-core automation inventory; restore drill from an
activity-core-produced artifact passes.
### Repo ownership (do not invent ownership)
| Concern | Owner repo |
| --- | --- |
| S5 handoff, status Make targets, app evidence, this workplan | **railiance-apps** |
| CNPG/Forgejo dump CLI, hostPath packaging, OpenBao lane policy for workers | **railiance-platform** |
| ActivityDefinition, shell resolver, Temporal schedule, worker mounts/secrets | **activity-core** |
| SSH certs for remote kube if needed | **ops-warden** (sign only) |
| Tunnels | **ops-bridge** |
This workplan **coordinates** cross-repo work; implement in the owning repo and
cite commits/evidence here. Do **not** hand-register hub tasks — write files and
`statehub fix-consistency`.
---
## Task: Inventory production-of-record DB topology ## Task: Inventory production-of-record DB topology
```task ```task
id: RAILIANCE-WP-0016-T01 id: RAILIANCE-WP-0016-T01
status: todo status: done
priority: high priority: high
state_hub_task_id: "acf23292-438b-44c0-a6cd-87327b54d743" state_hub_task_id: "acf23292-438b-44c0-a6cd-87327b54d743"
``` ```
Document which CNPG instances are production-of-record for each consumer DB **Done 2026-07-22:** `docs/cnpg-backup-topology-inventory.md` — POR table;
across CoulombCore vs railiance01 (especially `net-kingdom-pg` and dual-host NK/state-hub both backed up until DSN audit.
`state-hub-db` dual hosting). Produce a table in `docs/` or STATE.md: cluster,
host, databases, consumer, backup required Y/N.
**Done when:** signed-off inventory exists; no ambiguous double-backup or missed
instance remains.
## Task: Package offline-capable backup runner for railiance01 ## Task: Package offline-capable backup runner for railiance01
```task ```task
id: RAILIANCE-WP-0016-T02 id: RAILIANCE-WP-0016-T02
status: todo status: done
priority: high priority: high
state_hub_task_id: "35e81dc9-4f86-4a8f-8857-faef4117eeb6" state_hub_task_id: "35e81dc9-4f86-4a8f-8857-faef4117eeb6"
``` ```
Deliver a runner that does **not** require workstation or internet package **Done 2026-07-22:** platform CLI + static `age` under `tools/vendor/bin`;
install at job time: `install-cnpg-backup-vendor-tools` for kubectl; dry-run JSON ok for
`r01-forgejo-db`. Docs: `railiance-platform/docs/cnpg-option-a-backup.md`.
- Static `age` (and friends) vendored under hostPath tools tree and/or a
private OCI image on Forgejo registry with `kubectl`+`age`+`curl`+`bash`.
- CLI entrypoints covering: multi-cluster logical dump (Option A), forgejo dump
path (platform), status CM update, non-secret JSON evidence.
- Prefer extending `railiance-platform` lib/tools so activity-core only shells out
(mirror `forgejo-package-prune`).
**Done when:** on railiance01 (or a dry-run host with same mounts), one command
produces age-encrypted artifacts **without** `apk`/`github.com` at runtime.
## Task: Non-interactive offsite credentials for activity-core worker ## Task: Non-interactive offsite credentials for activity-core worker
```task ```task
id: RAILIANCE-WP-0016-T03 id: RAILIANCE-WP-0016-T03
status: todo status: wait
priority: high priority: high
needs_human: true needs_human: true
state_hub_task_id: "3bcb6a70-7b74-47a3-8afe-3921fdccd893" state_hub_task_id: "3bcb6a70-7b74-47a3-8afe-3921fdccd893"
``` ```
Wire `NC_WEBDAV_TOKEN` / `NC_WEBDAV_URL` / `AGE_PUBLIC_KEY` into the activity-core ESO manifest `actcore-backup-offsite` applied; status **SecretSyncedError**
worker on railiance01 without interactive OIDC: (provider deny) until ESO token includes
`workload-kv-read-railiance-backup-offsite-lane`. Script defaults + combined
- Prefer ExternalSecret → `actcore-runtime-secret` (pattern: forgejo-admin ESO). policy updated. **Blocked on operator OpenBao token mint.**
- OpenBao policy for the worker role; **do not** put `AGE_PRIVATE_KEY` on the
worker (recovery escrow stays operator/OpenBao only).
- Document rotation via CCR / warden route; no secrets in Git.
**Done when:** worker pod can upload a tiny probe object to Nextcloud and env
lengths are non-zero without operator login.
## Task: CoulombCore API reachability from railiance01 worker ## Task: CoulombCore API reachability from railiance01 worker
```task ```task
id: RAILIANCE-WP-0016-T04 id: RAILIANCE-WP-0016-T04
status: todo status: progress
priority: high priority: high
state_hub_task_id: "b8d2dd05-31a0-4fd7-bdef-27eb686e8f8a" state_hub_task_id: "b8d2dd05-31a0-4fd7-bdef-27eb686e8f8a"
``` ```
Backup of CoulombCore clusters (`apps-pg`, …) from railiance01 requires a Decision: hostPath `~/.kube` → `/kube` on worker; `KUBECONFIG_CORE=/kube/config`,
stable kubeconfig + auth (ops-warden cert + ops-bridge tunnel, or in-cluster `KUBECONFIG_R01=/kube/config-hosteurope`. Git runtime yaml updated. Remaining:
equivalent). Options to evaluate (pick one, record decision): ensure CoulombCore kubeconfig + certs on railiance01 host; rollout worker;
verify `kubectl get cluster` from inside worker for both hosts.
1. Worker host kubeconfig for CoulombCore with long-lived cert refresh hook.
2. activity-core (or a thin agent) also on CoulombCore for local dumps only.
3. CNPG-native path later (out of scope unless re-decided).
**Done when:** non-interactive `kubectl get cluster -n databases` against
CoulombCore succeeds from the same environment that will run the schedule.
## Task: activity-core ActivityDefinition + shell resolver ## Task: activity-core ActivityDefinition + shell resolver
```task ```task
id: RAILIANCE-WP-0016-T05 id: RAILIANCE-WP-0016-T05
status: todo status: progress
priority: high priority: high
state_hub_task_id: "547b3acb-94c2-46e0-a7da-ef51ee0c3c8d" state_hub_task_id: "547b3acb-94c2-46e0-a7da-ef51ee0c3c8d"
``` ```
In **activity-core**: Landed: `cnpg_option_a_backup` resolver, evidence sink, ops console allowlist,
`activity-definitions/daily-cnpg-option-a-backup.md` (**enabled: false**), unit
1. Shell resolver e.g. `cnpg_option_a_backup` (and optionally `forgejo_backup`) tests (3 passed). Remaining: image/deploy + enable after T03/T04.
analogous to `forgejo_package_prune`.
2. ActivityDefinition markdown with `trigger.type: cron` (target ~02:30 UTC
daily; weekly retention on Sundays).
3. `make sync` / schedule reconcile on railiance01; enable only after T02–T04.
4. Evidence sink → State Hub progress (non-secret summary).
5. Visible in `make automation-status` / inventory.
**Done when:** Temporal schedule exists, one forced run succeeds, inventory lists
the automation as enabled.
## Task: Cut over health definition and retire workstation dependency ## Task: Cut over health definition and retire workstation dependency
```task ```task
id: RAILIANCE-WP-0016-T06 id: RAILIANCE-WP-0016-T06
status: todo status: progress
priority: medium priority: medium
state_hub_task_id: "d6cab5a4-3cdd-4033-b21e-88db02197472" state_hub_task_id: "d6cab5a4-3cdd-4033-b21e-88db02197472"
``` ```
In **railiance-apps**: `cnpg-backup-status` accepts `mode=activity-core`. Full cutover after first
successful scheduled runs (T07). Workstation remains documented stopgap.
- Set `cnpg-option-a-schedule` `mode` to `activity-core` (or dual-run during
soak).
- `cnpg-backup-status` accepts activity-core last-success (status CM and/or
hub evidence age).
- Document that workstation cron is **stopgap only**; remove from operator
“required” path after soak.
- Optionally un-suspend in-cluster CronJobs **only if** T02 image path lands
in-cluster; otherwise leave suspended.
**Done when:** docs + status tool describe activity-core as primary; workstation
optional.
## Task: Soak, restore drill from automated artifact, finish ## Task: Soak, restore drill from automated artifact, finish
```task ```task
id: RAILIANCE-WP-0016-T07 id: RAILIANCE-WP-0016-T07
status: todo status: wait
priority: high priority: high
state_hub_task_id: "f331bdcd-0e76-4bc0-b85a-1bb1227a876b" state_hub_task_id: "f331bdcd-0e76-4bc0-b85a-1bb1227a876b"
``` ```
- ≥3 consecutive successful scheduled runs (or 7 if aligning forgejo promotion Blocked on T03–T05 enable path.
gate language).
- Restore drill from an artifact produced by activity-core (not workstation
one-off), row-count or equivalent gate, evidence JSON under `docs/evidence/`.
- Update `STATE.md`; archive/finish this workplan; optional message to
activity-core / platform owners if residual tasks remain in their repos.
**Done when:** RPO/RTO evidence recorded; workstation powered-off scenario is
credible; workplan `finished`.
--- ---
## Out of scope
- Switching Option A → barman ObjectStore / CNPG `ScheduledBackup` (Phase 2;
only if explicitly re-decided).
- Media/PVC app data backups (vergage media still disabled).
- Rotating EXPOSED taint history on the offsite lane (platform CCR ops), except
as a dependency note if worker fetch is blocked.
## Suggested implementation order
```text
T01 inventory → T02 runner package → T03 creds + T04 kube reachability (parallel)
→ T05 activity-core definition → T06 cutover docs/status → T07 soak + drill
```
## References ## References
- `STATE.md` (this repo) — current posture - `STATE.md`
- `workplans/RAILIANCE-WP-0015-…` — interim Option A - `docs/cnpg-backup-topology-inventory.md`
- `docs/evidence/cnpg-option-a-backup-20260722.json` - `railiance-platform/tools/cmd/cnpg-option-a-backup`
- `activity-core/activity-definitions/weekly-forgejo-package-prune.md` - `activity-core/activity-definitions/daily-cnpg-option-a-backup.md`
- `railiance-platform/docs/forgejo-backup.md`
- `activity-core/docs/runbook.md` (hostPath + ESO patterns)