tegwick
c202fbf7be
Mark workplan active with T01/T02 done, document topology, extend status for activity-core mode, and wire Make targets to the platform multi-host backup CLI. T03 remains operator-blocked on ESO token.
156 lines
5.2 KiB
Markdown
156 lines
5.2 KiB
Markdown
---
|
||
id: RAILIANCE-WP-0016
|
||
type: workplan
|
||
title: "railiance01 + activity-core unattended CNPG/Forgejo backup automation"
|
||
domain: financials
|
||
repo: railiance-apps
|
||
status: active
|
||
owner: codex
|
||
topic_slug: railiance
|
||
created: "2026-07-22"
|
||
updated: "2026-07-22"
|
||
state_hub_workstream_id: "0c8ddbc9-3740-4e0a-acb2-901e050ab242"
|
||
---
|
||
|
||
# railiance01 + activity-core unattended backup automation
|
||
|
||
**Predecessor:** `RAILIANCE-WP-0015` (finished) delivered Option A tooling and a
|
||
**healthy** `cnpg-backup-status` on CoulombCore, but unattended execution still
|
||
depends on a **workstation** (crontab + interactive OpenBao OIDC + laptop
|
||
uptime). That is not an acceptable durability control plane.
|
||
|
||
**Goal:** Daily encrypted offsite backups for production CNPG data run **without
|
||
any workstation** — scheduled and observed by **activity-core on railiance01**.
|
||
|
||
## Implementation status (2026-07-22)
|
||
|
||
| Area | State |
|
||
| --- | --- |
|
||
| Topology inventory | **done** — `docs/cnpg-backup-topology-inventory.md` |
|
||
| Platform CLI + vendor age | **done** — `railiance-platform/tools/cmd/cnpg-option-a-backup` |
|
||
| Dry-run forgejo-db | **pass** (JSON overall=ok) |
|
||
| activity-core resolver + definition | **landed** (`enabled: false` until creds+kube) |
|
||
| ESO ExternalSecret | **applied**; **SecretSyncedError** until ESO token re-mint |
|
||
| Worker RBAC databases exec | **applied** on railiance01 |
|
||
| Worker kubeconfig hostPath | **in git** (`20-runtime.yaml`); apply/restart pending |
|
||
| Schedule enable + soak | **blocked** on operator T03/T04 |
|
||
|
||
### Operator unblock (next human steps)
|
||
|
||
```bash
|
||
# 1) Re-mint ESO token with backup lane policy (operator OpenBao token)
|
||
cd ~/activity-core
|
||
OPENBAO_ACTIVITY_CORE_POLICIES="workload-kv-read-issue-core-runtime workload-kv-read-forgejo-admin workload-kv-read-railiance-backup-offsite-lane" \
|
||
scripts/openbao-eso-token-apply.sh
|
||
kubectl -n activity-core get externalsecret actcore-backup-offsite # expect SecretSynced
|
||
|
||
# 2) On railiance01 host as tegwick: vendor kubectl + ensure kubeconfigs
|
||
cd ~/railiance-platform && tools/cmd/install-cnpg-backup-vendor-tools
|
||
# Place CoulombCore kubeconfig at ~/.kube/config (R01 already config-hosteurope)
|
||
|
||
# 3) Roll worker with updated mounts + new image/code
|
||
kubectl apply -f k8s/railiance/20-runtime.yaml # or patch
|
||
# rebuild activity-core image with resolver if not live-mounted
|
||
|
||
# 4) Sync definition, enable, force run
|
||
# set enabled: true on daily-cnpg-option-a-backup; make sync-activity-definitions sync-schedules
|
||
```
|
||
|
||
## Task: Inventory production-of-record DB topology
|
||
|
||
```task
|
||
id: RAILIANCE-WP-0016-T01
|
||
status: done
|
||
priority: high
|
||
state_hub_task_id: "acf23292-438b-44c0-a6cd-87327b54d743"
|
||
```
|
||
|
||
**Done 2026-07-22:** `docs/cnpg-backup-topology-inventory.md` — POR table;
|
||
dual-host NK/state-hub both backed up until DSN audit.
|
||
|
||
## Task: Package offline-capable backup runner for railiance01
|
||
|
||
```task
|
||
id: RAILIANCE-WP-0016-T02
|
||
status: done
|
||
priority: high
|
||
state_hub_task_id: "35e81dc9-4f86-4a8f-8857-faef4117eeb6"
|
||
```
|
||
|
||
**Done 2026-07-22:** platform CLI + static `age` under `tools/vendor/bin`;
|
||
`install-cnpg-backup-vendor-tools` for kubectl; dry-run JSON ok for
|
||
`r01-forgejo-db`. Docs: `railiance-platform/docs/cnpg-option-a-backup.md`.
|
||
|
||
## Task: Non-interactive offsite credentials for activity-core worker
|
||
|
||
```task
|
||
id: RAILIANCE-WP-0016-T03
|
||
status: wait
|
||
priority: high
|
||
needs_human: true
|
||
state_hub_task_id: "3bcb6a70-7b74-47a3-8afe-3921fdccd893"
|
||
```
|
||
|
||
ESO manifest `actcore-backup-offsite` applied; status **SecretSyncedError**
|
||
(provider deny) until ESO token includes
|
||
`workload-kv-read-railiance-backup-offsite-lane`. Script defaults + combined
|
||
policy updated. **Blocked on operator OpenBao token mint.**
|
||
|
||
## Task: CoulombCore API reachability from railiance01 worker
|
||
|
||
```task
|
||
id: RAILIANCE-WP-0016-T04
|
||
status: progress
|
||
priority: high
|
||
state_hub_task_id: "b8d2dd05-31a0-4fd7-bdef-27eb686e8f8a"
|
||
```
|
||
|
||
Decision: hostPath `~/.kube` → `/kube` on worker; `KUBECONFIG_CORE=/kube/config`,
|
||
`KUBECONFIG_R01=/kube/config-hosteurope`. Git runtime yaml updated. Remaining:
|
||
ensure CoulombCore kubeconfig + certs on railiance01 host; rollout worker;
|
||
verify `kubectl get cluster` from inside worker for both hosts.
|
||
|
||
## Task: activity-core ActivityDefinition + shell resolver
|
||
|
||
```task
|
||
id: RAILIANCE-WP-0016-T05
|
||
status: progress
|
||
priority: high
|
||
state_hub_task_id: "547b3acb-94c2-46e0-a7da-ef51ee0c3c8d"
|
||
```
|
||
|
||
Landed: `cnpg_option_a_backup` resolver, evidence sink, ops console allowlist,
|
||
`activity-definitions/daily-cnpg-option-a-backup.md` (**enabled: false**), unit
|
||
tests (3 passed). Remaining: image/deploy + enable after T03/T04.
|
||
|
||
## Task: Cut over health definition and retire workstation dependency
|
||
|
||
```task
|
||
id: RAILIANCE-WP-0016-T06
|
||
status: progress
|
||
priority: medium
|
||
state_hub_task_id: "d6cab5a4-3cdd-4033-b21e-88db02197472"
|
||
```
|
||
|
||
`cnpg-backup-status` accepts `mode=activity-core`. Full cutover after first
|
||
successful scheduled runs (T07). Workstation remains documented stopgap.
|
||
|
||
## Task: Soak, restore drill from automated artifact, finish
|
||
|
||
```task
|
||
id: RAILIANCE-WP-0016-T07
|
||
status: wait
|
||
priority: high
|
||
state_hub_task_id: "f331bdcd-0e76-4bc0-b85a-1bb1227a876b"
|
||
```
|
||
|
||
Blocked on T03–T05 enable path.
|
||
|
||
---
|
||
|
||
## References
|
||
|
||
- `STATE.md`
|
||
- `docs/cnpg-backup-topology-inventory.md`
|
||
- `railiance-platform/tools/cmd/cnpg-option-a-backup`
|
||
- `activity-core/activity-definitions/daily-cnpg-option-a-backup.md`
|