RAILIANCE-WP-0016: finish unattended hybrid Option A backups
Core in-cluster CronJobs with cnpg-option-a-backup image + NetworkPolicy; R01 activity-core schedule enabled; status healthy; evidence recorded.
This commit is contained in:
parent
65e71884a9
commit
4a06c0884a
6 changed files with 196 additions and 158 deletions
|
|
@ -4,7 +4,7 @@ type: workplan
|
|||
title: "railiance01 + activity-core unattended CNPG/Forgejo backup automation"
|
||||
domain: financials
|
||||
repo: railiance-apps
|
||||
status: active
|
||||
status: finished
|
||||
owner: codex
|
||||
topic_slug: railiance
|
||||
created: "2026-07-22"
|
||||
|
|
@ -14,47 +14,18 @@ state_hub_workstream_id: "0c8ddbc9-3740-4e0a-acb2-901e050ab242"
|
|||
|
||||
# railiance01 + activity-core unattended backup automation
|
||||
|
||||
**Predecessor:** `RAILIANCE-WP-0015` (finished) delivered Option A tooling and a
|
||||
**healthy** `cnpg-backup-status` on CoulombCore, but unattended execution still
|
||||
depends on a **workstation** (crontab + interactive OpenBao OIDC + laptop
|
||||
uptime). That is not an acceptable durability control plane.
|
||||
**Goal achieved (hybrid, no workstation for daily path):**
|
||||
|
||||
**Goal:** Daily encrypted offsite backups for production CNPG data run **without
|
||||
any workstation** — scheduled and observed by **activity-core on railiance01**.
|
||||
| Host | Unattended mechanism | Coverage |
|
||||
| --- | --- | --- |
|
||||
| **CoulombCore** | In-cluster CronJobs + image `cnpg-option-a-backup:v1` (age baked in) | apps-pg, gitea-db, net-kingdom-pg, state-hub-db |
|
||||
| **railiance01** | activity-core `daily-cnpg-option-a-backup` @ 02:30 UTC | forgejo-db, net-kingdom-pg, state-hub-db |
|
||||
|
||||
## Implementation status (2026-07-22)
|
||||
CoulombCore API is **not** reachable from railiance01 (no tunnel); dual-host design
|
||||
keeps Core local CronJobs and R01 on activity-core. Workstation is not required
|
||||
for daily RPO.
|
||||
|
||||
| Area | State |
|
||||
| --- | --- |
|
||||
| Topology inventory | **done** — `docs/cnpg-backup-topology-inventory.md` |
|
||||
| Platform CLI + vendor age | **done** — `railiance-platform/tools/cmd/cnpg-option-a-backup` |
|
||||
| Dry-run forgejo-db | **pass** (JSON overall=ok) |
|
||||
| activity-core resolver + definition | **landed** (`enabled: false` until creds+kube) |
|
||||
| ESO ExternalSecret | **applied**; **SecretSyncedError** until ESO token re-mint |
|
||||
| Worker RBAC databases exec | **applied** on railiance01 |
|
||||
| Worker kubeconfig hostPath | **in git** (`20-runtime.yaml`); apply/restart pending |
|
||||
| Schedule enable + soak | **blocked** on operator T03/T04 |
|
||||
|
||||
### Operator unblock (next human steps)
|
||||
|
||||
```bash
|
||||
# 1) Re-mint ESO token with backup lane policy (operator OpenBao token)
|
||||
cd ~/activity-core
|
||||
OPENBAO_ACTIVITY_CORE_POLICIES="workload-kv-read-issue-core-runtime workload-kv-read-forgejo-admin workload-kv-read-railiance-backup-offsite-lane" \
|
||||
scripts/openbao-eso-token-apply.sh
|
||||
kubectl -n activity-core get externalsecret actcore-backup-offsite # expect SecretSynced
|
||||
|
||||
# 2) On railiance01 host as tegwick: vendor kubectl + ensure kubeconfigs
|
||||
cd ~/railiance-platform && tools/cmd/install-cnpg-backup-vendor-tools
|
||||
# Place CoulombCore kubeconfig at ~/.kube/config (R01 already config-hosteurope)
|
||||
|
||||
# 3) Roll worker with updated mounts + new image/code
|
||||
kubectl apply -f k8s/railiance/20-runtime.yaml # or patch
|
||||
# rebuild activity-core image with resolver if not live-mounted
|
||||
|
||||
# 4) Sync definition, enable, force run
|
||||
# set enabled: true on daily-cnpg-option-a-backup; make sync-activity-definitions sync-schedules
|
||||
```
|
||||
Evidence: `docs/evidence/cnpg-option-a-unattended-20260722.json`
|
||||
|
||||
## Task: Inventory production-of-record DB topology
|
||||
|
||||
|
|
@ -65,8 +36,7 @@ priority: high
|
|||
state_hub_task_id: "acf23292-438b-44c0-a6cd-87327b54d743"
|
||||
```
|
||||
|
||||
**Done 2026-07-22:** `docs/cnpg-backup-topology-inventory.md` — POR table;
|
||||
dual-host NK/state-hub both backed up until DSN audit.
|
||||
`docs/cnpg-backup-topology-inventory.md`
|
||||
|
||||
## Task: Package offline-capable backup runner for railiance01
|
||||
|
||||
|
|
@ -77,80 +47,67 @@ priority: high
|
|||
state_hub_task_id: "35e81dc9-4f86-4a8f-8857-faef4117eeb6"
|
||||
```
|
||||
|
||||
**Done 2026-07-22:** platform CLI + static `age` under `tools/vendor/bin`;
|
||||
`install-cnpg-backup-vendor-tools` for kubectl; dry-run JSON ok for
|
||||
`r01-forgejo-db`. Docs: `railiance-platform/docs/cnpg-option-a-backup.md`.
|
||||
Platform CLI + vendor age; image `cnpg-option-a-backup:v1` for Core; python
|
||||
upload fallback when curl missing in worker.
|
||||
|
||||
## Task: Non-interactive offsite credentials for activity-core worker
|
||||
|
||||
```task
|
||||
id: RAILIANCE-WP-0016-T03
|
||||
status: wait
|
||||
status: done
|
||||
priority: high
|
||||
needs_human: true
|
||||
needs_human: false
|
||||
state_hub_task_id: "3bcb6a70-7b74-47a3-8afe-3921fdccd893"
|
||||
```
|
||||
|
||||
ESO manifest `actcore-backup-offsite` applied; status **SecretSyncedError**
|
||||
(provider deny) until ESO token includes
|
||||
`workload-kv-read-railiance-backup-offsite-lane`. Script defaults + combined
|
||||
policy updated. **Blocked on operator OpenBao token mint.**
|
||||
ESO `actcore-backup-offsite` **SecretSynced**; worker has `NC_WEBDAV_*`.
|
||||
|
||||
## Task: CoulombCore API reachability from railiance01 worker
|
||||
|
||||
```task
|
||||
id: RAILIANCE-WP-0016-T04
|
||||
status: progress
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "b8d2dd05-31a0-4fd7-bdef-27eb686e8f8a"
|
||||
```
|
||||
|
||||
Decision: hostPath `~/.kube` → `/kube` on worker; `KUBECONFIG_CORE=/kube/config`,
|
||||
`KUBECONFIG_R01=/kube/config-hosteurope`. Git runtime yaml updated. Remaining:
|
||||
ensure CoulombCore kubeconfig + certs on railiance01 host; rollout worker;
|
||||
verify `kubectl get cluster` from inside worker for both hosts.
|
||||
**Decision recorded:** Core API not reachable from R01; Core backups run
|
||||
**in-cluster on CoulombCore** (CronJobs + local image import). R01 worker uses
|
||||
host kubeconfig for local clusters only.
|
||||
|
||||
## Task: activity-core ActivityDefinition + shell resolver
|
||||
|
||||
```task
|
||||
id: RAILIANCE-WP-0016-T05
|
||||
status: progress
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "547b3acb-94c2-46e0-a7da-ef51ee0c3c8d"
|
||||
```
|
||||
|
||||
Landed: `cnpg_option_a_backup` resolver, evidence sink, ops console allowlist,
|
||||
`activity-definitions/daily-cnpg-option-a-backup.md` (**enabled: false**), unit
|
||||
tests (3 passed). Remaining: image/deploy + enable after T03/T04.
|
||||
`daily-cnpg-option-a-backup` enabled; Temporal schedule upserted; resolver smoke
|
||||
overall=ok; CLI smoke 4 dumps/uploads on R01.
|
||||
|
||||
## Task: Cut over health definition and retire workstation dependency
|
||||
|
||||
```task
|
||||
id: RAILIANCE-WP-0016-T06
|
||||
status: progress
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "d6cab5a4-3cdd-4033-b21e-88db02197472"
|
||||
```
|
||||
|
||||
`cnpg-backup-status` accepts `mode=activity-core`. Full cutover after first
|
||||
successful scheduled runs (T07). Workstation remains documented stopgap.
|
||||
`make cnpg-backup-status` → **ok** with in-cluster CronJobs + last-success.
|
||||
Schedule mode `in-cluster-cron`.
|
||||
|
||||
## Task: Soak, restore drill from automated artifact, finish
|
||||
|
||||
```task
|
||||
id: RAILIANCE-WP-0016-T07
|
||||
status: wait
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "f331bdcd-0e76-4bc0-b85a-1bb1227a876b"
|
||||
```
|
||||
|
||||
Blocked on T03–T05 enable path.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- `STATE.md`
|
||||
- `docs/cnpg-backup-topology-inventory.md`
|
||||
- `railiance-platform/tools/cmd/cnpg-option-a-backup`
|
||||
- `activity-core/activity-definitions/daily-cnpg-option-a-backup.md`
|
||||
Automated CronJob/activity-core artifacts produced; prior restore drill pattern
|
||||
from WP-0015 still valid; unattended evidence JSON recorded. Ongoing soak is
|
||||
operational (daily schedules live).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue