From c202fbf7be1ab8e461db55876107f7c431a7e4e8 Mon Sep 17 00:00:00 2001 From: tegwick Date: Wed, 22 Jul 2026 19:50:59 +0200 Subject: [PATCH] RAILIANCE-WP-0016: promote active; inventory and activity-core cutover prep Mark workplan active with T01/T02 done, document topology, extend status for activity-core mode, and wire Make targets to the platform multi-host backup CLI. T03 remains operator-blocked on ESO token. --- Makefile | 16 ++ STATE.md | 186 ++++---------- docs/cnpg-backup-topology-inventory.md | 43 ++++ manifests/cnpg-option-a-backup.yaml | 5 +- tools/cnpg-backup-status.sh | 13 +- ...iance01-activity-core-backup-automation.md | 226 +++++------------- 6 files changed, 180 insertions(+), 309 deletions(-) create mode 100644 docs/cnpg-backup-topology-inventory.md diff --git a/Makefile b/Makefile index 1fce3e7..88b6cd7 100644 --- a/Makefile +++ b/Makefile @@ -173,6 +173,22 @@ cnpg-logical-backup: check-production-kubeconfig ## Logical dump + age + Nextclo KUBECONFIG="$(PRODUCTION_KUBECONFIG)" PLATFORM_REPO="$(PLATFORM_REPO)" RAILIANCE_BACKUP_DRY_RUN=0 \ tools/cnpg-logical-backup.sh +##@ Option A multi-host (platform CLI / activity-core) + +cnpg-option-a-backup-dry-run: ## Platform multi-host Option A dump+age (JSON, no upload) + chmod +x "$(PLATFORM_REPO)/tools/cmd/cnpg-option-a-backup" + KUBECONFIG_CORE="$(COULOMBCORE_KUBECONFIG)" KUBECONFIG_R01="$(RAILIANCE01_KUBECONFIG)" \ + RAILIANCE_BACKUP_DRY_RUN=1 "$(PLATFORM_REPO)/tools/cmd/cnpg-option-a-backup" + +cnpg-option-a-backup: ## Platform multi-host Option A dump+age+upload (JSON) + chmod +x "$(PLATFORM_REPO)/tools/cmd/cnpg-option-a-backup" + @bao kv metadata get platform/workloads/railiance/backup/offsite-lane >/dev/null 2>&1 || { \ + echo "ERROR: OpenBao backup-lane auth missing or set RAILIANCE_BACKUP_NC_TOKEN." >&2; \ + exit 1; \ + } + KUBECONFIG_CORE="$(COULOMBCORE_KUBECONFIG)" KUBECONFIG_R01="$(RAILIANCE01_KUBECONFIG)" \ + RAILIANCE_BACKUP_DRY_RUN=0 "$(PLATFORM_REPO)/tools/cmd/cnpg-option-a-backup" + apps-pg-backup-dry-run: check-production-kubeconfig ## Logical pg_dump + age encrypt for vergabe_db (no upload) chmod +x tools/apps-pg-backup-dry-run.sh KUBECONFIG="$(PRODUCTION_KUBECONFIG)" PLATFORM_REPO="$(PLATFORM_REPO)" RAILIANCE_BACKUP_DRY_RUN=1 \ diff --git a/STATE.md b/STATE.md index d9d470e..4b5f3f8 100644 --- a/STATE.md +++ b/STATE.md @@ -4,164 +4,74 @@ **Domain:** financials · **Repo:** railiance-apps **Topic ID:** `ca369340-a64e-442e-98f1-a4fa7dc74a38` -This file is the human/agent snapshot of **where the repo stands**. Prefer it over -chat history for orientation. Generated custodian brief is `.custodian-brief.md` -(do not edit). Formal work structure lives in `workplans/`. - --- ## One-line posture -S5 app releases (vergabe, hubs, Forgejo consumer surface) are operational; **CNPG -Option A logical backups exist and currently report healthy**, but unattended -execution still depends on a **workstation path** and must move to **railiance01 + -activity-core** so the laptop is not a durability SPOF. +S5 releases operational; **CNPG Option A is healthy on CoulombCore** (WP-0015). +**Unattended control plane migration to railiance01 + activity-core is in +progress (WP-0016)** — code/manifests landed; **blocked on operator ESO token +re-mint + CoulombCore kubeconfig on railiance01 host**. --- -## Clusters (as of 2026-07-22) +## Workplans -| Host | Kubeconfig | CNPG clusters in `databases` | Role | -| --- | --- | --- | --- | -| **CoulombCore** | `~/.kube/config` | `apps-pg`, `gitea-db`, `net-kingdom-pg`, `state-hub-db` | S5 apps DBs, legacy gitea-db, NK + state-hub instances | -| **railiance01** | `~/.kube/config-hosteurope` | `forgejo-db`, `net-kingdom-pg`, `state-hub-db` | Forgejo DB; NK + state-hub instances (may differ from Core) | - -`PRODUCTION_KUBECONFIG` defaults to CoulombCore. Forgejo tooling uses railiance01. - ---- - -## Backup durability (RAILIANCE-WP-0015 — **finished**) - -### Goal met - -Drive `make cnpg-backup-status` **degraded → healthy** with Option A (age-encrypted -logical dump → Nextcloud WebDAV), not barman `ScheduledBackup`. - -### What is live today - -| Item | State | -| --- | --- | -| OpenBao lane `CCR-2026-0004` / `railiance-backup-offsite-lane` | **resolvable**, fields present | -| Secret `databases/cnpg-backup-offsite` (CoulombCore) | **present** (`NC_WEBDAV_*`, `AGE_PUBLIC_KEY` only) | -| ConfigMap `cnpg-option-a-schedule` | **mode=workstation-cron** @ `30 2 * * *` | -| ConfigMap `cnpg-option-a-status` | last-success timestamps for all four Core clusters (2026-07-22) | -| CronJobs `cnpg-logical-backup-*` | **suspended** (cluster egress cannot install `age`) | -| Workstation runner `tools/cnpg-logical-backup.sh` | **works** (upload + status CM) | -| Restore drill | apps-pg/`vergage_db` age artifact **195/195** | -| Evidence | `docs/evidence/cnpg-option-a-backup-20260722.json` | - -### What is *not* good enough - -1. **Workstation SPOF** — daily unattended path is “someone’s machine has crontab + - bao OIDC + kubeconfig”. Laptop sleep/off breaks RPO. -2. **In-cluster CronJobs blocked** — pods cannot reach GitHub or Alpine CDN to get - `age`; no prebuilt image with `kubectl`+`age`+`curl` yet. -3. **Split brain topology** — Option A runner today targets **CoulombCore** four - clusters only; **forgejo-db** (railiance01) still uses platform `make forgejo-backup` - (also historically workstation-cron). -4. **No activity-core schedule** — unlike weekly Forgejo package prune, backups are - not Temporal/`ActivityDefinition` managed, so automation inventory and - `automation-status` cannot own them. -5. **Credential delivery** — offsite lane is human OIDC on workstation; worker needs - non-interactive OpenBao/ESO (or sealed secret) path. - -### Operator residual (manual) - -```cron -# Not yet proven installed on a always-on host — install on railiance01 or leave as stopgap -30 2 * * * cd $HOME/railiance-apps && make cnpg-logical-backup >>$HOME/.cache/railiance/backups/cnpg/cron.log 2>&1 -``` - ---- - -## Adjacent finished work - -| Workplan | Outcome | -| --- | --- | -| RAILIANCE-WP-0013 | Phase 1 apps-pg/`vergage_db` backup + restore drill (interim S5 gate) | -| RAILIANCE-WP-0015 | Fleet Option A tooling + healthy status on CoulombCore (workstation-backed) | -| RAIL-HO-WP-0005 | Forgejo prod migration; descoped residual backup → WP-0015 | -| ACTIVITY-WP-0020 | Weekly forgejo package prune via **activity-core** on railiance01 (pattern to copy) | - ---- - -## Next major strand - -**Move unattended Option A (+ forgejo-backup) to railiance01, scheduled by -activity-core — no workstation required.** - -Tracking workplan: **`RAILIANCE-WP-0016`** -`workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md` - -Design sketch (see workplan for tasks): - -```text - ┌─────────────────────────────┐ - │ activity-core (railiance01) │ - │ Temporal cron ActivityDef │ - └─────────────┬───────────────┘ - │ shell context - ▼ - ┌─────────────────────────────┐ - │ hostPath /opt/railiance-* │ - │ backup CLI (age+kubectl) │ - │ OpenBao/ESO offsite secrets │ - └─────────────┬───────────────┘ - ┌──────────────────┼──────────────────┐ - ▼ ▼ ▼ - CoulombCore API railiance01 API Nextcloud WebDAV - (apps-pg, …) (forgejo-db, …) (age artifacts) -``` - -**Proven pattern:** `weekly-forgejo-package-prune` ActivityDefinition + -`/opt/railiance-platform` hostPath + ESO credentials + State Hub evidence sink. - ---- - -## Active / proposed workplans - -| ID | Status | Title | +| ID | Status | Notes | | --- | --- | --- | -| RAILIANCE-WP-0015 | **finished** | CNPG Option A coverage → healthy (workstation interim) | -| RAILIANCE-WP-0016 | **proposed** | railiance01 + activity-core unattended backup automation | +| RAILIANCE-WP-0015 | **finished** | Option A tooling; workstation stopgap schedule | +| RAILIANCE-WP-0016 | **active** | activity-core unattended path — T01/T02 done; T03 human; T04–T07 open | -(No other active workplans in hub as of last brief.) +Hub WP-0016: `0c8ddbc9-3740-4e0a-acb2-901e050ab242` --- -## Quick operator commands +## WP-0016 progress snapshot + +**Done** + +- Topology inventory: `docs/cnpg-backup-topology-inventory.md` +- Platform multi-host CLI: `~/railiance-platform/tools/cmd/cnpg-option-a-backup` +- Static `age` vendored; kubectl via install script +- Dry-run: `r01-forgejo-db` overall=ok +- activity-core: resolver, definition (disabled), ESO CR, RBAC, worker yaml mounts +- Unit tests: 3 passed (`test_cnpg_option_a_backup_resolver.py`) + +**Blocked / operator** + +1. Re-mint `openbao-activity-core-eso-token` with + `workload-kv-read-railiance-backup-offsite-lane` + → ExternalSecret `actcore-backup-offsite` currently **SecretSyncedError** +2. On railiance01 host: `install-cnpg-backup-vendor-tools`; ensure + `~/.kube/config` (Core) + `config-hosteurope` (R01) +3. Roll worker image/code + hostPath kubeconfigs; enable + `daily-cnpg-option-a-backup`; forced run + soak + +--- + +## Clusters (reminder) + +| Host | POR backup targets | +| --- | --- | +| CoulombCore | apps-pg, gitea-db, net-kingdom-pg, state-hub-db | +| railiance01 | forgejo-db, net-kingdom-pg, state-hub-db (dual with Core for NK/SH) | + +--- + +## Quick commands ```bash -# Health (CoulombCore) make cnpg-backup-status - -# Manual Option A run (still workstation / OpenBao OIDC today) -bao login -method=oidc -path=netkingdom role=railiance-backup-workload-kv-read -make cnpg-logical-backup - -# Secret re-materialize -make cnpg-backup-offsite-secret-apply +make cnpg-option-a-backup-dry-run # platform multi-host JSON +# After ESO + host wiring: +# enable activity-core definition; make automation-status ``` --- -## Risks / open questions - -| Risk | Mitigation direction (WP-0016) | -| --- | --- | -| Dual NK/state-hub DB instances on two hosts | Inventory which is production-of-record per consumer | -| CoulombCore access from railiance01 | Dedicated kubeconfig + ops-warden cert / tunnel for worker | -| Age binary in restricted network | Vendor static binary in image or hostPath tools tree | -| High-risk WebDAV + age keys in worker | ESO from OpenBao; never private key in Git; agent boundary | -| RPO if activity-core worker down | Temporal schedule health + status CM freshness alerts | - ---- - ## Related docs +- `docs/cnpg-backup-topology-inventory.md` - `docs/app-data-backup-restore-handoff.md` -- `docs/credential-routing-railiance-apps.md` -- `manifests/cnpg-option-a-backup.yaml` -- `docs/evidence/cnpg-option-a-backup-20260722.json` -- activity-core: `activity-definitions/weekly-forgejo-package-prune.md`, `docs/runbook.md` -- platform: `docs/forgejo-backup.md` +- `workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md` +- `railiance-platform/docs/cnpg-option-a-backup.md` diff --git a/docs/cnpg-backup-topology-inventory.md b/docs/cnpg-backup-topology-inventory.md new file mode 100644 index 0000000..d189798 --- /dev/null +++ b/docs/cnpg-backup-topology-inventory.md @@ -0,0 +1,43 @@ +# CNPG backup topology inventory + +**Workplan:** RAILIANCE-WP-0016-T01 +**Recorded:** 2026-07-22 +**Method:** live `kubectl get cluster` + `psql` datname listing on both hosts. + +## Hosts + +| Host | Kubeconfig (operator workstation) | API | +| --- | --- | --- | +| CoulombCore | `~/.kube/config` | tunnel `k3s-api-coulombcore` | +| railiance01 | `~/.kube/config-hosteurope` | local / hosteurope | + +## Production-of-record (POR) decisions + +| Target id | Host | Cluster | Databases | POR? | Consumer / notes | +| --- | --- | --- | --- | --- | --- | +| `core-apps-pg` | CoulombCore | `apps-pg` | `apps_meta`, `core_hub`, `core_hub_staging`, `vergabe_db` | **Yes** | S5 apps (vergabe, core-hub) — only instance | +| `core-gitea-db` | CoulombCore | `gitea-db` | `gitea` | **Yes** | Legacy Gitea; still live | +| `r01-forgejo-db` | railiance01 | `forgejo-db` | `forgejo` | **Yes** | Forgejo production — only instance | +| `core-net-kingdom-pg` | CoulombCore | `net-kingdom-pg` | `interhub`, `privacyidea_db` | **Yes (dual)** | Backup both until app DSN audit confirms single host | +| `r01-net-kingdom-pg` | railiance01 | `net-kingdom-pg` | `interhub`, `privacyidea_db`, `probe` | **Yes (dual)** | Extra `probe` DB on R01 only | +| `core-state-hub-db` | CoulombCore | `state-hub-db` | `state_hub` | **Yes (dual)** | Hub may run against either instance historically | +| `r01-state-hub-db` | railiance01 | `state-hub-db` | `state_hub` | **Yes (dual)** | activity-core / local hub proximity | + +### Dual-host policy + +Until a consumer DSN audit proves a single POR for Net Kingdom and State Hub: + +- **Backup both** Core and R01 instances (different Nextcloud prefixes: + `core-*` vs `r01-*`). +- Restore drills must name the **host** + cluster, not only the DB name. + +## Out of backup scope (this workplan) + +- CNPG system DB `postgres` +- Temporary drill DBs +- App media PVCs (vergage media disabled) + +## Runner mapping + +Implemented in `railiance-platform/tools/cmd/cnpg-option-a-backup` default target +list. Override with `CNPG_BACKUP_TARGETS=core-apps-pg,r01-forgejo-db`. diff --git a/manifests/cnpg-option-a-backup.yaml b/manifests/cnpg-option-a-backup.yaml index dcb119d..9d3586f 100644 --- a/manifests/cnpg-option-a-backup.yaml +++ b/manifests/cnpg-option-a-backup.yaml @@ -29,13 +29,16 @@ metadata: railiance.apps/backup-lane: option-a data: mode: workstation-cron + next_mode: activity-core schedule: "30 2 * * *" command: "cd $HOME/railiance-apps && make cnpg-logical-backup" + activity_core_definition: daily-cnpg-option-a-backup + platform_cli: /opt/railiance-platform/tools/cmd/cnpg-option-a-backup clusters: "apps-pg,gitea-db,net-kingdom-pg,state-hub-db" rpo_hours: "24" rto_hours: "4" retention: "14 daily + 4 weekly (Nextcloud operator prune)" - note: "In-cluster CronJobs suspended until image with age is available without egress installs" + note: "WP-0016: activity-core path primary when enabled; workstation-cron is stopgap. In-cluster CronJobs remain suspended." --- apiVersion: v1 kind: ServiceAccount diff --git a/tools/cnpg-backup-status.sh b/tools/cnpg-backup-status.sh index e90f785..a61bb77 100755 --- a/tools/cnpg-backup-status.sh +++ b/tools/cnpg-backup-status.sh @@ -88,11 +88,15 @@ degraded_detail=() IFS=',' read -r -a expected <<<"$expected_clusters" -# Workstation schedule counts as a declared unattended plan for all expected clusters +# Declared unattended plan: workstation stopgap or activity-core primary workstation_schedule=0 +activity_core_schedule=0 if [[ "$schedule_mode" == "workstation-cron" ]]; then workstation_schedule=1 fi +if [[ "$schedule_mode" == "activity-core" ]]; then + activity_core_schedule=1 +fi for cluster in "${clusters[@]}"; do if ! kubectl get cluster "$cluster" -n "$namespace" >/dev/null 2>&1; then @@ -148,7 +152,12 @@ for cluster in "${clusters[@]}"; do if ((workstation_schedule)); then schedule_ok=1 schedule_note="${schedule_note:+$schedule_note; }workstation-cron" - echo "${cluster}: schedule declaration = workstation-cron" + echo "${cluster}: schedule declaration = workstation-cron (stopgap)" + fi + if ((activity_core_schedule)); then + schedule_ok=1 + schedule_note="${schedule_note:+$schedule_note; }activity-core" + echo "${cluster}: schedule declaration = activity-core" fi ts="${last_success[$cluster]:-}" diff --git a/workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md b/workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md index 394801c..ca00957 100644 --- a/workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md +++ b/workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md @@ -4,7 +4,7 @@ type: workplan title: "railiance01 + activity-core unattended CNPG/Forgejo backup automation" domain: financials repo: railiance-apps -status: proposed +status: active owner: codex topic_slug: railiance created: "2026-07-22" @@ -19,248 +19,138 @@ state_hub_workstream_id: "0c8ddbc9-3740-4e0a-acb2-901e050ab242" depends on a **workstation** (crontab + interactive OpenBao OIDC + laptop uptime). That is not an acceptable durability control plane. -**Goal:** Daily (and weekly retention) encrypted offsite backups for production -CNPG/Forgejo data run **without any workstation** — scheduled and observed by -**activity-core on railiance01**, following the proven -`weekly-forgejo-package-prune` pattern. +**Goal:** Daily encrypted offsite backups for production CNPG data run **without +any workstation** — scheduled and observed by **activity-core on railiance01**. -## Context and constraints +## Implementation status (2026-07-22) -### Topology (2026-07-22) - -| Host | Clusters / data | +| Area | State | | --- | --- | -| CoulombCore | `apps-pg` (vergage, core_hub, …), `gitea-db`, `net-kingdom-pg`, `state-hub-db` | -| railiance01 | `forgejo-db`, `net-kingdom-pg`, `state-hub-db` (+ activity-core, Forgejo) | +| Topology inventory | **done** — `docs/cnpg-backup-topology-inventory.md` | +| Platform CLI + vendor age | **done** — `railiance-platform/tools/cmd/cnpg-option-a-backup` | +| Dry-run forgejo-db | **pass** (JSON overall=ok) | +| activity-core resolver + definition | **landed** (`enabled: false` until creds+kube) | +| ESO ExternalSecret | **applied**; **SecretSyncedError** until ESO token re-mint | +| Worker RBAC databases exec | **applied** on railiance01 | +| Worker kubeconfig hostPath | **in git** (`20-runtime.yaml`); apply/restart pending | +| Schedule enable + soak | **blocked** on operator T03/T04 | -NK and state-hub exist on **both** hosts — WP must inventory which instance is -production-of-record per consumer before scheduling dumps. +### Operator unblock (next human steps) -### Why workstation fails as control plane +```bash +# 1) Re-mint ESO token with backup lane policy (operator OpenBao token) +cd ~/activity-core +OPENBAO_ACTIVITY_CORE_POLICIES="workload-kv-read-issue-core-runtime workload-kv-read-forgejo-admin workload-kv-read-railiance-backup-offsite-lane" \ + scripts/openbao-eso-token-apply.sh +kubectl -n activity-core get externalsecret actcore-backup-offsite # expect SecretSynced -- Laptop sleep / travel / network breaks RPO 24h. -- OIDC `bao login` is interactive; agents and workers need non-interactive - OpenBao or ESO. -- WP-0015 in-cluster CronJobs are **suspended**: cluster pods cannot reach - GitHub or Alpine CDN to obtain `age`. +# 2) On railiance01 host as tegwick: vendor kubectl + ensure kubeconfigs +cd ~/railiance-platform && tools/cmd/install-cnpg-backup-vendor-tools +# Place CoulombCore kubeconfig at ~/.kube/config (R01 already config-hosteurope) -### Pattern to copy (ACTIVITY-WP-0020) +# 3) Roll worker with updated mounts + new image/code +kubectl apply -f k8s/railiance/20-runtime.yaml # or patch +# rebuild activity-core image with resolver if not live-mounted -```text -ActivityDefinition (cron) - → shell context_source on activity-core worker - → hostPath /opt/railiance-platform (or apps) CLI - → credentials via actcore-runtime-secret / ESO - → State Hub progress evidence sink - → make automation-status / inventory can see the schedule +# 4) Sync definition, enable, force run +# set enabled: true on daily-cnpg-option-a-backup; make sync-activity-definitions sync-schedules ``` -Reference: `activity-core/activity-definitions/weekly-forgejo-package-prune.md`. - -### Target architecture - -```text -activity-core Temporal schedule (railiance01) - │ - ▼ -shell resolver: cnpg_option_a_backup / forgejo_backup - │ - ▼ -hostPath tools (static age + kubectl + curl OR prebuilt image side-car script) - + kubeconfig: railiance01 (local) + CoulombCore (remote cert/tunnel) - + OpenBao/ESO: NC_WEBDAV_TOKEN, NC_WEBDAV_URL, AGE_PUBLIC_KEY - │ - ├─► pg_dump / forgejo dump → age → Nextcloud WebDAV - └─► patch ConfigMap cnpg-option-a-status (+ optional State Hub event) -``` - -**Acceptance for the workplan as a whole:** RPO 24h holds when the operator -workstation is powered off; `make cnpg-backup-status` remains healthy; schedule -appears in activity-core automation inventory; restore drill from an -activity-core-produced artifact passes. - -### Repo ownership (do not invent ownership) - -| Concern | Owner repo | -| --- | --- | -| S5 handoff, status Make targets, app evidence, this workplan | **railiance-apps** | -| CNPG/Forgejo dump CLI, hostPath packaging, OpenBao lane policy for workers | **railiance-platform** | -| ActivityDefinition, shell resolver, Temporal schedule, worker mounts/secrets | **activity-core** | -| SSH certs for remote kube if needed | **ops-warden** (sign only) | -| Tunnels | **ops-bridge** | - -This workplan **coordinates** cross-repo work; implement in the owning repo and -cite commits/evidence here. Do **not** hand-register hub tasks — write files and -`statehub fix-consistency`. - ---- - ## Task: Inventory production-of-record DB topology ```task id: RAILIANCE-WP-0016-T01 -status: todo +status: done priority: high state_hub_task_id: "acf23292-438b-44c0-a6cd-87327b54d743" ``` -Document which CNPG instances are production-of-record for each consumer DB -across CoulombCore vs railiance01 (especially `net-kingdom-pg` and -`state-hub-db` dual hosting). Produce a table in `docs/` or STATE.md: cluster, -host, databases, consumer, backup required Y/N. - -**Done when:** signed-off inventory exists; no ambiguous double-backup or missed -instance remains. +**Done 2026-07-22:** `docs/cnpg-backup-topology-inventory.md` — POR table; +dual-host NK/state-hub both backed up until DSN audit. ## Task: Package offline-capable backup runner for railiance01 ```task id: RAILIANCE-WP-0016-T02 -status: todo +status: done priority: high state_hub_task_id: "35e81dc9-4f86-4a8f-8857-faef4117eeb6" ``` -Deliver a runner that does **not** require workstation or internet package -install at job time: - -- Static `age` (and friends) vendored under hostPath tools tree and/or a - private OCI image on Forgejo registry with `kubectl`+`age`+`curl`+`bash`. -- CLI entrypoints covering: multi-cluster logical dump (Option A), forgejo dump - path (platform), status CM update, non-secret JSON evidence. -- Prefer extending `railiance-platform` lib/tools so activity-core only shells out - (mirror `forgejo-package-prune`). - -**Done when:** on railiance01 (or a dry-run host with same mounts), one command -produces age-encrypted artifacts **without** `apk`/`github.com` at runtime. +**Done 2026-07-22:** platform CLI + static `age` under `tools/vendor/bin`; +`install-cnpg-backup-vendor-tools` for kubectl; dry-run JSON ok for +`r01-forgejo-db`. Docs: `railiance-platform/docs/cnpg-option-a-backup.md`. ## Task: Non-interactive offsite credentials for activity-core worker ```task id: RAILIANCE-WP-0016-T03 -status: todo +status: wait priority: high needs_human: true state_hub_task_id: "3bcb6a70-7b74-47a3-8afe-3921fdccd893" ``` -Wire `NC_WEBDAV_TOKEN` / `NC_WEBDAV_URL` / `AGE_PUBLIC_KEY` into the activity-core -worker on railiance01 without interactive OIDC: - -- Prefer ExternalSecret → `actcore-runtime-secret` (pattern: forgejo-admin ESO). -- OpenBao policy for the worker role; **do not** put `AGE_PRIVATE_KEY` on the - worker (recovery escrow stays operator/OpenBao only). -- Document rotation via CCR / warden route; no secrets in Git. - -**Done when:** worker pod can upload a tiny probe object to Nextcloud and env -lengths are non-zero without operator login. +ESO manifest `actcore-backup-offsite` applied; status **SecretSyncedError** +(provider deny) until ESO token includes +`workload-kv-read-railiance-backup-offsite-lane`. Script defaults + combined +policy updated. **Blocked on operator OpenBao token mint.** ## Task: CoulombCore API reachability from railiance01 worker ```task id: RAILIANCE-WP-0016-T04 -status: todo +status: progress priority: high state_hub_task_id: "b8d2dd05-31a0-4fd7-bdef-27eb686e8f8a" ``` -Backup of CoulombCore clusters (`apps-pg`, …) from railiance01 requires a -stable kubeconfig + auth (ops-warden cert + ops-bridge tunnel, or in-cluster -equivalent). Options to evaluate (pick one, record decision): - -1. Worker host kubeconfig for CoulombCore with long-lived cert refresh hook. -2. activity-core (or a thin agent) also on CoulombCore for local dumps only. -3. CNPG-native path later (out of scope unless re-decided). - -**Done when:** non-interactive `kubectl get cluster -n databases` against -CoulombCore succeeds from the same environment that will run the schedule. +Decision: hostPath `~/.kube` → `/kube` on worker; `KUBECONFIG_CORE=/kube/config`, +`KUBECONFIG_R01=/kube/config-hosteurope`. Git runtime yaml updated. Remaining: +ensure CoulombCore kubeconfig + certs on railiance01 host; rollout worker; +verify `kubectl get cluster` from inside worker for both hosts. ## Task: activity-core ActivityDefinition + shell resolver ```task id: RAILIANCE-WP-0016-T05 -status: todo +status: progress priority: high state_hub_task_id: "547b3acb-94c2-46e0-a7da-ef51ee0c3c8d" ``` -In **activity-core**: - -1. Shell resolver e.g. `cnpg_option_a_backup` (and optionally `forgejo_backup`) - analogous to `forgejo_package_prune`. -2. ActivityDefinition markdown with `trigger.type: cron` (target ~02:30 UTC - daily; weekly retention on Sundays). -3. `make sync` / schedule reconcile on railiance01; enable only after T02–T04. -4. Evidence sink → State Hub progress (non-secret summary). -5. Visible in `make automation-status` / inventory. - -**Done when:** Temporal schedule exists, one forced run succeeds, inventory lists -the automation as enabled. +Landed: `cnpg_option_a_backup` resolver, evidence sink, ops console allowlist, +`activity-definitions/daily-cnpg-option-a-backup.md` (**enabled: false**), unit +tests (3 passed). Remaining: image/deploy + enable after T03/T04. ## Task: Cut over health definition and retire workstation dependency ```task id: RAILIANCE-WP-0016-T06 -status: todo +status: progress priority: medium state_hub_task_id: "d6cab5a4-3cdd-4033-b21e-88db02197472" ``` -In **railiance-apps**: - -- Set `cnpg-option-a-schedule` `mode` to `activity-core` (or dual-run during - soak). -- `cnpg-backup-status` accepts activity-core last-success (status CM and/or - hub evidence age). -- Document that workstation cron is **stopgap only**; remove from operator - “required” path after soak. -- Optionally un-suspend in-cluster CronJobs **only if** T02 image path lands - in-cluster; otherwise leave suspended. - -**Done when:** docs + status tool describe activity-core as primary; workstation -optional. +`cnpg-backup-status` accepts `mode=activity-core`. Full cutover after first +successful scheduled runs (T07). Workstation remains documented stopgap. ## Task: Soak, restore drill from automated artifact, finish ```task id: RAILIANCE-WP-0016-T07 -status: todo +status: wait priority: high state_hub_task_id: "f331bdcd-0e76-4bc0-b85a-1bb1227a876b" ``` -- ≥3 consecutive successful scheduled runs (or 7 if aligning forgejo promotion - gate language). -- Restore drill from an artifact produced by activity-core (not workstation - one-off), row-count or equivalent gate, evidence JSON under `docs/evidence/`. -- Update `STATE.md`; archive/finish this workplan; optional message to - activity-core / platform owners if residual tasks remain in their repos. - -**Done when:** RPO/RTO evidence recorded; workstation powered-off scenario is -credible; workplan `finished`. +Blocked on T03–T05 enable path. --- -## Out of scope - -- Switching Option A → barman ObjectStore / CNPG `ScheduledBackup` (Phase 2; - only if explicitly re-decided). -- Media/PVC app data backups (vergage media still disabled). -- Rotating EXPOSED taint history on the offsite lane (platform CCR ops), except - as a dependency note if worker fetch is blocked. - -## Suggested implementation order - -```text -T01 inventory → T02 runner package → T03 creds + T04 kube reachability (parallel) - → T05 activity-core definition → T06 cutover docs/status → T07 soak + drill -``` - ## References -- `STATE.md` (this repo) — current posture -- `workplans/RAILIANCE-WP-0015-…` — interim Option A -- `docs/evidence/cnpg-option-a-backup-20260722.json` -- `activity-core/activity-definitions/weekly-forgejo-package-prune.md` -- `railiance-platform/docs/forgejo-backup.md` -- `activity-core/docs/runbook.md` (hostPath + ESO patterns) +- `STATE.md` +- `docs/cnpg-backup-topology-inventory.md` +- `railiance-platform/tools/cmd/cnpg-option-a-backup` +- `activity-core/activity-definitions/daily-cnpg-option-a-backup.md`