RAILIANCE-WP-0016: promote active; inventory and activity-core cutover prep
Mark workplan active with T01/T02 done, document topology, extend status for activity-core mode, and wire Make targets to the platform multi-host backup CLI. T03 remains operator-blocked on ESO token.
This commit is contained in:
parent
04626b6335
commit
c202fbf7be
6 changed files with 180 additions and 309 deletions
16
Makefile
16
Makefile
|
|
@ -173,6 +173,22 @@ cnpg-logical-backup: check-production-kubeconfig ## Logical dump + age + Nextclo
|
|||
KUBECONFIG="$(PRODUCTION_KUBECONFIG)" PLATFORM_REPO="$(PLATFORM_REPO)" RAILIANCE_BACKUP_DRY_RUN=0 \
|
||||
tools/cnpg-logical-backup.sh
|
||||
|
||||
##@ Option A multi-host (platform CLI / activity-core)
|
||||
|
||||
cnpg-option-a-backup-dry-run: ## Platform multi-host Option A dump+age (JSON, no upload)
|
||||
chmod +x "$(PLATFORM_REPO)/tools/cmd/cnpg-option-a-backup"
|
||||
KUBECONFIG_CORE="$(COULOMBCORE_KUBECONFIG)" KUBECONFIG_R01="$(RAILIANCE01_KUBECONFIG)" \
|
||||
RAILIANCE_BACKUP_DRY_RUN=1 "$(PLATFORM_REPO)/tools/cmd/cnpg-option-a-backup"
|
||||
|
||||
cnpg-option-a-backup: ## Platform multi-host Option A dump+age+upload (JSON)
|
||||
chmod +x "$(PLATFORM_REPO)/tools/cmd/cnpg-option-a-backup"
|
||||
@bao kv metadata get platform/workloads/railiance/backup/offsite-lane >/dev/null 2>&1 || { \
|
||||
echo "ERROR: OpenBao backup-lane auth missing or set RAILIANCE_BACKUP_NC_TOKEN." >&2; \
|
||||
exit 1; \
|
||||
}
|
||||
KUBECONFIG_CORE="$(COULOMBCORE_KUBECONFIG)" KUBECONFIG_R01="$(RAILIANCE01_KUBECONFIG)" \
|
||||
RAILIANCE_BACKUP_DRY_RUN=0 "$(PLATFORM_REPO)/tools/cmd/cnpg-option-a-backup"
|
||||
|
||||
apps-pg-backup-dry-run: check-production-kubeconfig ## Logical pg_dump + age encrypt for vergabe_db (no upload)
|
||||
chmod +x tools/apps-pg-backup-dry-run.sh
|
||||
KUBECONFIG="$(PRODUCTION_KUBECONFIG)" PLATFORM_REPO="$(PLATFORM_REPO)" RAILIANCE_BACKUP_DRY_RUN=1 \
|
||||
|
|
|
|||
186
STATE.md
186
STATE.md
|
|
@ -4,164 +4,74 @@
|
|||
**Domain:** financials · **Repo:** railiance-apps
|
||||
**Topic ID:** `ca369340-a64e-442e-98f1-a4fa7dc74a38`
|
||||
|
||||
This file is the human/agent snapshot of **where the repo stands**. Prefer it over
|
||||
chat history for orientation. Generated custodian brief is `.custodian-brief.md`
|
||||
(do not edit). Formal work structure lives in `workplans/`.
|
||||
|
||||
---
|
||||
|
||||
## One-line posture
|
||||
|
||||
S5 app releases (vergabe, hubs, Forgejo consumer surface) are operational; **CNPG
|
||||
Option A logical backups exist and currently report healthy**, but unattended
|
||||
execution still depends on a **workstation path** and must move to **railiance01 +
|
||||
activity-core** so the laptop is not a durability SPOF.
|
||||
S5 releases operational; **CNPG Option A is healthy on CoulombCore** (WP-0015).
|
||||
**Unattended control plane migration to railiance01 + activity-core is in
|
||||
progress (WP-0016)** — code/manifests landed; **blocked on operator ESO token
|
||||
re-mint + CoulombCore kubeconfig on railiance01 host**.
|
||||
|
||||
---
|
||||
|
||||
## Clusters (as of 2026-07-22)
|
||||
## Workplans
|
||||
|
||||
| Host | Kubeconfig | CNPG clusters in `databases` | Role |
|
||||
| --- | --- | --- | --- |
|
||||
| **CoulombCore** | `~/.kube/config` | `apps-pg`, `gitea-db`, `net-kingdom-pg`, `state-hub-db` | S5 apps DBs, legacy gitea-db, NK + state-hub instances |
|
||||
| **railiance01** | `~/.kube/config-hosteurope` | `forgejo-db`, `net-kingdom-pg`, `state-hub-db` | Forgejo DB; NK + state-hub instances (may differ from Core) |
|
||||
|
||||
`PRODUCTION_KUBECONFIG` defaults to CoulombCore. Forgejo tooling uses railiance01.
|
||||
|
||||
---
|
||||
|
||||
## Backup durability (RAILIANCE-WP-0015 — **finished**)
|
||||
|
||||
### Goal met
|
||||
|
||||
Drive `make cnpg-backup-status` **degraded → healthy** with Option A (age-encrypted
|
||||
logical dump → Nextcloud WebDAV), not barman `ScheduledBackup`.
|
||||
|
||||
### What is live today
|
||||
|
||||
| Item | State |
|
||||
| --- | --- |
|
||||
| OpenBao lane `CCR-2026-0004` / `railiance-backup-offsite-lane` | **resolvable**, fields present |
|
||||
| Secret `databases/cnpg-backup-offsite` (CoulombCore) | **present** (`NC_WEBDAV_*`, `AGE_PUBLIC_KEY` only) |
|
||||
| ConfigMap `cnpg-option-a-schedule` | **mode=workstation-cron** @ `30 2 * * *` |
|
||||
| ConfigMap `cnpg-option-a-status` | last-success timestamps for all four Core clusters (2026-07-22) |
|
||||
| CronJobs `cnpg-logical-backup-*` | **suspended** (cluster egress cannot install `age`) |
|
||||
| Workstation runner `tools/cnpg-logical-backup.sh` | **works** (upload + status CM) |
|
||||
| Restore drill | apps-pg/`vergage_db` age artifact **195/195** |
|
||||
| Evidence | `docs/evidence/cnpg-option-a-backup-20260722.json` |
|
||||
|
||||
### What is *not* good enough
|
||||
|
||||
1. **Workstation SPOF** — daily unattended path is “someone’s machine has crontab +
|
||||
bao OIDC + kubeconfig”. Laptop sleep/off breaks RPO.
|
||||
2. **In-cluster CronJobs blocked** — pods cannot reach GitHub or Alpine CDN to get
|
||||
`age`; no prebuilt image with `kubectl`+`age`+`curl` yet.
|
||||
3. **Split brain topology** — Option A runner today targets **CoulombCore** four
|
||||
clusters only; **forgejo-db** (railiance01) still uses platform `make forgejo-backup`
|
||||
(also historically workstation-cron).
|
||||
4. **No activity-core schedule** — unlike weekly Forgejo package prune, backups are
|
||||
not Temporal/`ActivityDefinition` managed, so automation inventory and
|
||||
`automation-status` cannot own them.
|
||||
5. **Credential delivery** — offsite lane is human OIDC on workstation; worker needs
|
||||
non-interactive OpenBao/ESO (or sealed secret) path.
|
||||
|
||||
### Operator residual (manual)
|
||||
|
||||
```cron
|
||||
# Not yet proven installed on a always-on host — install on railiance01 or leave as stopgap
|
||||
30 2 * * * cd $HOME/railiance-apps && make cnpg-logical-backup >>$HOME/.cache/railiance/backups/cnpg/cron.log 2>&1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adjacent finished work
|
||||
|
||||
| Workplan | Outcome |
|
||||
| --- | --- |
|
||||
| RAILIANCE-WP-0013 | Phase 1 apps-pg/`vergage_db` backup + restore drill (interim S5 gate) |
|
||||
| RAILIANCE-WP-0015 | Fleet Option A tooling + healthy status on CoulombCore (workstation-backed) |
|
||||
| RAIL-HO-WP-0005 | Forgejo prod migration; descoped residual backup → WP-0015 |
|
||||
| ACTIVITY-WP-0020 | Weekly forgejo package prune via **activity-core** on railiance01 (pattern to copy) |
|
||||
|
||||
---
|
||||
|
||||
## Next major strand
|
||||
|
||||
**Move unattended Option A (+ forgejo-backup) to railiance01, scheduled by
|
||||
activity-core — no workstation required.**
|
||||
|
||||
Tracking workplan: **`RAILIANCE-WP-0016`**
|
||||
`workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md`
|
||||
|
||||
Design sketch (see workplan for tasks):
|
||||
|
||||
```text
|
||||
┌─────────────────────────────┐
|
||||
│ activity-core (railiance01) │
|
||||
│ Temporal cron ActivityDef │
|
||||
└─────────────┬───────────────┘
|
||||
│ shell context
|
||||
▼
|
||||
┌─────────────────────────────┐
|
||||
│ hostPath /opt/railiance-* │
|
||||
│ backup CLI (age+kubectl) │
|
||||
│ OpenBao/ESO offsite secrets │
|
||||
└─────────────┬───────────────┘
|
||||
┌──────────────────┼──────────────────┐
|
||||
▼ ▼ ▼
|
||||
CoulombCore API railiance01 API Nextcloud WebDAV
|
||||
(apps-pg, …) (forgejo-db, …) (age artifacts)
|
||||
```
|
||||
|
||||
**Proven pattern:** `weekly-forgejo-package-prune` ActivityDefinition +
|
||||
`/opt/railiance-platform` hostPath + ESO credentials + State Hub evidence sink.
|
||||
|
||||
---
|
||||
|
||||
## Active / proposed workplans
|
||||
|
||||
| ID | Status | Title |
|
||||
| ID | Status | Notes |
|
||||
| --- | --- | --- |
|
||||
| RAILIANCE-WP-0015 | **finished** | CNPG Option A coverage → healthy (workstation interim) |
|
||||
| RAILIANCE-WP-0016 | **proposed** | railiance01 + activity-core unattended backup automation |
|
||||
| RAILIANCE-WP-0015 | **finished** | Option A tooling; workstation stopgap schedule |
|
||||
| RAILIANCE-WP-0016 | **active** | activity-core unattended path — T01/T02 done; T03 human; T04–T07 open |
|
||||
|
||||
(No other active workplans in hub as of last brief.)
|
||||
Hub WP-0016: `0c8ddbc9-3740-4e0a-acb2-901e050ab242`
|
||||
|
||||
---
|
||||
|
||||
## Quick operator commands
|
||||
## WP-0016 progress snapshot
|
||||
|
||||
**Done**
|
||||
|
||||
- Topology inventory: `docs/cnpg-backup-topology-inventory.md`
|
||||
- Platform multi-host CLI: `~/railiance-platform/tools/cmd/cnpg-option-a-backup`
|
||||
- Static `age` vendored; kubectl via install script
|
||||
- Dry-run: `r01-forgejo-db` overall=ok
|
||||
- activity-core: resolver, definition (disabled), ESO CR, RBAC, worker yaml mounts
|
||||
- Unit tests: 3 passed (`test_cnpg_option_a_backup_resolver.py`)
|
||||
|
||||
**Blocked / operator**
|
||||
|
||||
1. Re-mint `openbao-activity-core-eso-token` with
|
||||
`workload-kv-read-railiance-backup-offsite-lane`
|
||||
→ ExternalSecret `actcore-backup-offsite` currently **SecretSyncedError**
|
||||
2. On railiance01 host: `install-cnpg-backup-vendor-tools`; ensure
|
||||
`~/.kube/config` (Core) + `config-hosteurope` (R01)
|
||||
3. Roll worker image/code + hostPath kubeconfigs; enable
|
||||
`daily-cnpg-option-a-backup`; forced run + soak
|
||||
|
||||
---
|
||||
|
||||
## Clusters (reminder)
|
||||
|
||||
| Host | POR backup targets |
|
||||
| --- | --- |
|
||||
| CoulombCore | apps-pg, gitea-db, net-kingdom-pg, state-hub-db |
|
||||
| railiance01 | forgejo-db, net-kingdom-pg, state-hub-db (dual with Core for NK/SH) |
|
||||
|
||||
---
|
||||
|
||||
## Quick commands
|
||||
|
||||
```bash
|
||||
# Health (CoulombCore)
|
||||
make cnpg-backup-status
|
||||
|
||||
# Manual Option A run (still workstation / OpenBao OIDC today)
|
||||
bao login -method=oidc -path=netkingdom role=railiance-backup-workload-kv-read
|
||||
make cnpg-logical-backup
|
||||
|
||||
# Secret re-materialize
|
||||
make cnpg-backup-offsite-secret-apply
|
||||
make cnpg-option-a-backup-dry-run # platform multi-host JSON
|
||||
# After ESO + host wiring:
|
||||
# enable activity-core definition; make automation-status
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Risks / open questions
|
||||
|
||||
| Risk | Mitigation direction (WP-0016) |
|
||||
| --- | --- |
|
||||
| Dual NK/state-hub DB instances on two hosts | Inventory which is production-of-record per consumer |
|
||||
| CoulombCore access from railiance01 | Dedicated kubeconfig + ops-warden cert / tunnel for worker |
|
||||
| Age binary in restricted network | Vendor static binary in image or hostPath tools tree |
|
||||
| High-risk WebDAV + age keys in worker | ESO from OpenBao; never private key in Git; agent boundary |
|
||||
| RPO if activity-core worker down | Temporal schedule health + status CM freshness alerts |
|
||||
|
||||
---
|
||||
|
||||
## Related docs
|
||||
|
||||
- `docs/cnpg-backup-topology-inventory.md`
|
||||
- `docs/app-data-backup-restore-handoff.md`
|
||||
- `docs/credential-routing-railiance-apps.md`
|
||||
- `manifests/cnpg-option-a-backup.yaml`
|
||||
- `docs/evidence/cnpg-option-a-backup-20260722.json`
|
||||
- activity-core: `activity-definitions/weekly-forgejo-package-prune.md`, `docs/runbook.md`
|
||||
- platform: `docs/forgejo-backup.md`
|
||||
- `workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md`
|
||||
- `railiance-platform/docs/cnpg-option-a-backup.md`
|
||||
|
|
|
|||
43
docs/cnpg-backup-topology-inventory.md
Normal file
43
docs/cnpg-backup-topology-inventory.md
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
# CNPG backup topology inventory
|
||||
|
||||
**Workplan:** RAILIANCE-WP-0016-T01
|
||||
**Recorded:** 2026-07-22
|
||||
**Method:** live `kubectl get cluster` + `psql` datname listing on both hosts.
|
||||
|
||||
## Hosts
|
||||
|
||||
| Host | Kubeconfig (operator workstation) | API |
|
||||
| --- | --- | --- |
|
||||
| CoulombCore | `~/.kube/config` | tunnel `k3s-api-coulombcore` |
|
||||
| railiance01 | `~/.kube/config-hosteurope` | local / hosteurope |
|
||||
|
||||
## Production-of-record (POR) decisions
|
||||
|
||||
| Target id | Host | Cluster | Databases | POR? | Consumer / notes |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `core-apps-pg` | CoulombCore | `apps-pg` | `apps_meta`, `core_hub`, `core_hub_staging`, `vergabe_db` | **Yes** | S5 apps (vergabe, core-hub) — only instance |
|
||||
| `core-gitea-db` | CoulombCore | `gitea-db` | `gitea` | **Yes** | Legacy Gitea; still live |
|
||||
| `r01-forgejo-db` | railiance01 | `forgejo-db` | `forgejo` | **Yes** | Forgejo production — only instance |
|
||||
| `core-net-kingdom-pg` | CoulombCore | `net-kingdom-pg` | `interhub`, `privacyidea_db` | **Yes (dual)** | Backup both until app DSN audit confirms single host |
|
||||
| `r01-net-kingdom-pg` | railiance01 | `net-kingdom-pg` | `interhub`, `privacyidea_db`, `probe` | **Yes (dual)** | Extra `probe` DB on R01 only |
|
||||
| `core-state-hub-db` | CoulombCore | `state-hub-db` | `state_hub` | **Yes (dual)** | Hub may run against either instance historically |
|
||||
| `r01-state-hub-db` | railiance01 | `state-hub-db` | `state_hub` | **Yes (dual)** | activity-core / local hub proximity |
|
||||
|
||||
### Dual-host policy
|
||||
|
||||
Until a consumer DSN audit proves a single POR for Net Kingdom and State Hub:
|
||||
|
||||
- **Backup both** Core and R01 instances (different Nextcloud prefixes:
|
||||
`core-*` vs `r01-*`).
|
||||
- Restore drills must name the **host** + cluster, not only the DB name.
|
||||
|
||||
## Out of backup scope (this workplan)
|
||||
|
||||
- CNPG system DB `postgres`
|
||||
- Temporary drill DBs
|
||||
- App media PVCs (vergage media disabled)
|
||||
|
||||
## Runner mapping
|
||||
|
||||
Implemented in `railiance-platform/tools/cmd/cnpg-option-a-backup` default target
|
||||
list. Override with `CNPG_BACKUP_TARGETS=core-apps-pg,r01-forgejo-db`.
|
||||
|
|
@ -29,13 +29,16 @@ metadata:
|
|||
railiance.apps/backup-lane: option-a
|
||||
data:
|
||||
mode: workstation-cron
|
||||
next_mode: activity-core
|
||||
schedule: "30 2 * * *"
|
||||
command: "cd $HOME/railiance-apps && make cnpg-logical-backup"
|
||||
activity_core_definition: daily-cnpg-option-a-backup
|
||||
platform_cli: /opt/railiance-platform/tools/cmd/cnpg-option-a-backup
|
||||
clusters: "apps-pg,gitea-db,net-kingdom-pg,state-hub-db"
|
||||
rpo_hours: "24"
|
||||
rto_hours: "4"
|
||||
retention: "14 daily + 4 weekly (Nextcloud operator prune)"
|
||||
note: "In-cluster CronJobs suspended until image with age is available without egress installs"
|
||||
note: "WP-0016: activity-core path primary when enabled; workstation-cron is stopgap. In-cluster CronJobs remain suspended."
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
|
|
|
|||
|
|
@ -88,11 +88,15 @@ degraded_detail=()
|
|||
|
||||
IFS=',' read -r -a expected <<<"$expected_clusters"
|
||||
|
||||
# Workstation schedule counts as a declared unattended plan for all expected clusters
|
||||
# Declared unattended plan: workstation stopgap or activity-core primary
|
||||
workstation_schedule=0
|
||||
activity_core_schedule=0
|
||||
if [[ "$schedule_mode" == "workstation-cron" ]]; then
|
||||
workstation_schedule=1
|
||||
fi
|
||||
if [[ "$schedule_mode" == "activity-core" ]]; then
|
||||
activity_core_schedule=1
|
||||
fi
|
||||
|
||||
for cluster in "${clusters[@]}"; do
|
||||
if ! kubectl get cluster "$cluster" -n "$namespace" >/dev/null 2>&1; then
|
||||
|
|
@ -148,7 +152,12 @@ for cluster in "${clusters[@]}"; do
|
|||
if ((workstation_schedule)); then
|
||||
schedule_ok=1
|
||||
schedule_note="${schedule_note:+$schedule_note; }workstation-cron"
|
||||
echo "${cluster}: schedule declaration = workstation-cron"
|
||||
echo "${cluster}: schedule declaration = workstation-cron (stopgap)"
|
||||
fi
|
||||
if ((activity_core_schedule)); then
|
||||
schedule_ok=1
|
||||
schedule_note="${schedule_note:+$schedule_note; }activity-core"
|
||||
echo "${cluster}: schedule declaration = activity-core"
|
||||
fi
|
||||
|
||||
ts="${last_success[$cluster]:-}"
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ type: workplan
|
|||
title: "railiance01 + activity-core unattended CNPG/Forgejo backup automation"
|
||||
domain: financials
|
||||
repo: railiance-apps
|
||||
status: proposed
|
||||
status: active
|
||||
owner: codex
|
||||
topic_slug: railiance
|
||||
created: "2026-07-22"
|
||||
|
|
@ -19,248 +19,138 @@ state_hub_workstream_id: "0c8ddbc9-3740-4e0a-acb2-901e050ab242"
|
|||
depends on a **workstation** (crontab + interactive OpenBao OIDC + laptop
|
||||
uptime). That is not an acceptable durability control plane.
|
||||
|
||||
**Goal:** Daily (and weekly retention) encrypted offsite backups for production
|
||||
CNPG/Forgejo data run **without any workstation** — scheduled and observed by
|
||||
**activity-core on railiance01**, following the proven
|
||||
`weekly-forgejo-package-prune` pattern.
|
||||
**Goal:** Daily encrypted offsite backups for production CNPG data run **without
|
||||
any workstation** — scheduled and observed by **activity-core on railiance01**.
|
||||
|
||||
## Context and constraints
|
||||
## Implementation status (2026-07-22)
|
||||
|
||||
### Topology (2026-07-22)
|
||||
|
||||
| Host | Clusters / data |
|
||||
| Area | State |
|
||||
| --- | --- |
|
||||
| CoulombCore | `apps-pg` (vergage, core_hub, …), `gitea-db`, `net-kingdom-pg`, `state-hub-db` |
|
||||
| railiance01 | `forgejo-db`, `net-kingdom-pg`, `state-hub-db` (+ activity-core, Forgejo) |
|
||||
| Topology inventory | **done** — `docs/cnpg-backup-topology-inventory.md` |
|
||||
| Platform CLI + vendor age | **done** — `railiance-platform/tools/cmd/cnpg-option-a-backup` |
|
||||
| Dry-run forgejo-db | **pass** (JSON overall=ok) |
|
||||
| activity-core resolver + definition | **landed** (`enabled: false` until creds+kube) |
|
||||
| ESO ExternalSecret | **applied**; **SecretSyncedError** until ESO token re-mint |
|
||||
| Worker RBAC databases exec | **applied** on railiance01 |
|
||||
| Worker kubeconfig hostPath | **in git** (`20-runtime.yaml`); apply/restart pending |
|
||||
| Schedule enable + soak | **blocked** on operator T03/T04 |
|
||||
|
||||
NK and state-hub exist on **both** hosts — WP must inventory which instance is
|
||||
production-of-record per consumer before scheduling dumps.
|
||||
### Operator unblock (next human steps)
|
||||
|
||||
### Why workstation fails as control plane
|
||||
```bash
|
||||
# 1) Re-mint ESO token with backup lane policy (operator OpenBao token)
|
||||
cd ~/activity-core
|
||||
OPENBAO_ACTIVITY_CORE_POLICIES="workload-kv-read-issue-core-runtime workload-kv-read-forgejo-admin workload-kv-read-railiance-backup-offsite-lane" \
|
||||
scripts/openbao-eso-token-apply.sh
|
||||
kubectl -n activity-core get externalsecret actcore-backup-offsite # expect SecretSynced
|
||||
|
||||
- Laptop sleep / travel / network breaks RPO 24h.
|
||||
- OIDC `bao login` is interactive; agents and workers need non-interactive
|
||||
OpenBao or ESO.
|
||||
- WP-0015 in-cluster CronJobs are **suspended**: cluster pods cannot reach
|
||||
GitHub or Alpine CDN to obtain `age`.
|
||||
# 2) On railiance01 host as tegwick: vendor kubectl + ensure kubeconfigs
|
||||
cd ~/railiance-platform && tools/cmd/install-cnpg-backup-vendor-tools
|
||||
# Place CoulombCore kubeconfig at ~/.kube/config (R01 already config-hosteurope)
|
||||
|
||||
### Pattern to copy (ACTIVITY-WP-0020)
|
||||
# 3) Roll worker with updated mounts + new image/code
|
||||
kubectl apply -f k8s/railiance/20-runtime.yaml # or patch
|
||||
# rebuild activity-core image with resolver if not live-mounted
|
||||
|
||||
```text
|
||||
ActivityDefinition (cron)
|
||||
→ shell context_source on activity-core worker
|
||||
→ hostPath /opt/railiance-platform (or apps) CLI
|
||||
→ credentials via actcore-runtime-secret / ESO
|
||||
→ State Hub progress evidence sink
|
||||
→ make automation-status / inventory can see the schedule
|
||||
# 4) Sync definition, enable, force run
|
||||
# set enabled: true on daily-cnpg-option-a-backup; make sync-activity-definitions sync-schedules
|
||||
```
|
||||
|
||||
Reference: `activity-core/activity-definitions/weekly-forgejo-package-prune.md`.
|
||||
|
||||
### Target architecture
|
||||
|
||||
```text
|
||||
activity-core Temporal schedule (railiance01)
|
||||
│
|
||||
▼
|
||||
shell resolver: cnpg_option_a_backup / forgejo_backup
|
||||
│
|
||||
▼
|
||||
hostPath tools (static age + kubectl + curl OR prebuilt image side-car script)
|
||||
+ kubeconfig: railiance01 (local) + CoulombCore (remote cert/tunnel)
|
||||
+ OpenBao/ESO: NC_WEBDAV_TOKEN, NC_WEBDAV_URL, AGE_PUBLIC_KEY
|
||||
│
|
||||
├─► pg_dump / forgejo dump → age → Nextcloud WebDAV
|
||||
└─► patch ConfigMap cnpg-option-a-status (+ optional State Hub event)
|
||||
```
|
||||
|
||||
**Acceptance for the workplan as a whole:** RPO 24h holds when the operator
|
||||
workstation is powered off; `make cnpg-backup-status` remains healthy; schedule
|
||||
appears in activity-core automation inventory; restore drill from an
|
||||
activity-core-produced artifact passes.
|
||||
|
||||
### Repo ownership (do not invent ownership)
|
||||
|
||||
| Concern | Owner repo |
|
||||
| --- | --- |
|
||||
| S5 handoff, status Make targets, app evidence, this workplan | **railiance-apps** |
|
||||
| CNPG/Forgejo dump CLI, hostPath packaging, OpenBao lane policy for workers | **railiance-platform** |
|
||||
| ActivityDefinition, shell resolver, Temporal schedule, worker mounts/secrets | **activity-core** |
|
||||
| SSH certs for remote kube if needed | **ops-warden** (sign only) |
|
||||
| Tunnels | **ops-bridge** |
|
||||
|
||||
This workplan **coordinates** cross-repo work; implement in the owning repo and
|
||||
cite commits/evidence here. Do **not** hand-register hub tasks — write files and
|
||||
`statehub fix-consistency`.
|
||||
|
||||
---
|
||||
|
||||
## Task: Inventory production-of-record DB topology
|
||||
|
||||
```task
|
||||
id: RAILIANCE-WP-0016-T01
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "acf23292-438b-44c0-a6cd-87327b54d743"
|
||||
```
|
||||
|
||||
Document which CNPG instances are production-of-record for each consumer DB
|
||||
across CoulombCore vs railiance01 (especially `net-kingdom-pg` and
|
||||
`state-hub-db` dual hosting). Produce a table in `docs/` or STATE.md: cluster,
|
||||
host, databases, consumer, backup required Y/N.
|
||||
|
||||
**Done when:** signed-off inventory exists; no ambiguous double-backup or missed
|
||||
instance remains.
|
||||
**Done 2026-07-22:** `docs/cnpg-backup-topology-inventory.md` — POR table;
|
||||
dual-host NK/state-hub both backed up until DSN audit.
|
||||
|
||||
## Task: Package offline-capable backup runner for railiance01
|
||||
|
||||
```task
|
||||
id: RAILIANCE-WP-0016-T02
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "35e81dc9-4f86-4a8f-8857-faef4117eeb6"
|
||||
```
|
||||
|
||||
Deliver a runner that does **not** require workstation or internet package
|
||||
install at job time:
|
||||
|
||||
- Static `age` (and friends) vendored under hostPath tools tree and/or a
|
||||
private OCI image on Forgejo registry with `kubectl`+`age`+`curl`+`bash`.
|
||||
- CLI entrypoints covering: multi-cluster logical dump (Option A), forgejo dump
|
||||
path (platform), status CM update, non-secret JSON evidence.
|
||||
- Prefer extending `railiance-platform` lib/tools so activity-core only shells out
|
||||
(mirror `forgejo-package-prune`).
|
||||
|
||||
**Done when:** on railiance01 (or a dry-run host with same mounts), one command
|
||||
produces age-encrypted artifacts **without** `apk`/`github.com` at runtime.
|
||||
**Done 2026-07-22:** platform CLI + static `age` under `tools/vendor/bin`;
|
||||
`install-cnpg-backup-vendor-tools` for kubectl; dry-run JSON ok for
|
||||
`r01-forgejo-db`. Docs: `railiance-platform/docs/cnpg-option-a-backup.md`.
|
||||
|
||||
## Task: Non-interactive offsite credentials for activity-core worker
|
||||
|
||||
```task
|
||||
id: RAILIANCE-WP-0016-T03
|
||||
status: todo
|
||||
status: wait
|
||||
priority: high
|
||||
needs_human: true
|
||||
state_hub_task_id: "3bcb6a70-7b74-47a3-8afe-3921fdccd893"
|
||||
```
|
||||
|
||||
Wire `NC_WEBDAV_TOKEN` / `NC_WEBDAV_URL` / `AGE_PUBLIC_KEY` into the activity-core
|
||||
worker on railiance01 without interactive OIDC:
|
||||
|
||||
- Prefer ExternalSecret → `actcore-runtime-secret` (pattern: forgejo-admin ESO).
|
||||
- OpenBao policy for the worker role; **do not** put `AGE_PRIVATE_KEY` on the
|
||||
worker (recovery escrow stays operator/OpenBao only).
|
||||
- Document rotation via CCR / warden route; no secrets in Git.
|
||||
|
||||
**Done when:** worker pod can upload a tiny probe object to Nextcloud and env
|
||||
lengths are non-zero without operator login.
|
||||
ESO manifest `actcore-backup-offsite` applied; status **SecretSyncedError**
|
||||
(provider deny) until ESO token includes
|
||||
`workload-kv-read-railiance-backup-offsite-lane`. Script defaults + combined
|
||||
policy updated. **Blocked on operator OpenBao token mint.**
|
||||
|
||||
## Task: CoulombCore API reachability from railiance01 worker
|
||||
|
||||
```task
|
||||
id: RAILIANCE-WP-0016-T04
|
||||
status: todo
|
||||
status: progress
|
||||
priority: high
|
||||
state_hub_task_id: "b8d2dd05-31a0-4fd7-bdef-27eb686e8f8a"
|
||||
```
|
||||
|
||||
Backup of CoulombCore clusters (`apps-pg`, …) from railiance01 requires a
|
||||
stable kubeconfig + auth (ops-warden cert + ops-bridge tunnel, or in-cluster
|
||||
equivalent). Options to evaluate (pick one, record decision):
|
||||
|
||||
1. Worker host kubeconfig for CoulombCore with long-lived cert refresh hook.
|
||||
2. activity-core (or a thin agent) also on CoulombCore for local dumps only.
|
||||
3. CNPG-native path later (out of scope unless re-decided).
|
||||
|
||||
**Done when:** non-interactive `kubectl get cluster -n databases` against
|
||||
CoulombCore succeeds from the same environment that will run the schedule.
|
||||
Decision: hostPath `~/.kube` → `/kube` on worker; `KUBECONFIG_CORE=/kube/config`,
|
||||
`KUBECONFIG_R01=/kube/config-hosteurope`. Git runtime yaml updated. Remaining:
|
||||
ensure CoulombCore kubeconfig + certs on railiance01 host; rollout worker;
|
||||
verify `kubectl get cluster` from inside worker for both hosts.
|
||||
|
||||
## Task: activity-core ActivityDefinition + shell resolver
|
||||
|
||||
```task
|
||||
id: RAILIANCE-WP-0016-T05
|
||||
status: todo
|
||||
status: progress
|
||||
priority: high
|
||||
state_hub_task_id: "547b3acb-94c2-46e0-a7da-ef51ee0c3c8d"
|
||||
```
|
||||
|
||||
In **activity-core**:
|
||||
|
||||
1. Shell resolver e.g. `cnpg_option_a_backup` (and optionally `forgejo_backup`)
|
||||
analogous to `forgejo_package_prune`.
|
||||
2. ActivityDefinition markdown with `trigger.type: cron` (target ~02:30 UTC
|
||||
daily; weekly retention on Sundays).
|
||||
3. `make sync` / schedule reconcile on railiance01; enable only after T02–T04.
|
||||
4. Evidence sink → State Hub progress (non-secret summary).
|
||||
5. Visible in `make automation-status` / inventory.
|
||||
|
||||
**Done when:** Temporal schedule exists, one forced run succeeds, inventory lists
|
||||
the automation as enabled.
|
||||
Landed: `cnpg_option_a_backup` resolver, evidence sink, ops console allowlist,
|
||||
`activity-definitions/daily-cnpg-option-a-backup.md` (**enabled: false**), unit
|
||||
tests (3 passed). Remaining: image/deploy + enable after T03/T04.
|
||||
|
||||
## Task: Cut over health definition and retire workstation dependency
|
||||
|
||||
```task
|
||||
id: RAILIANCE-WP-0016-T06
|
||||
status: todo
|
||||
status: progress
|
||||
priority: medium
|
||||
state_hub_task_id: "d6cab5a4-3cdd-4033-b21e-88db02197472"
|
||||
```
|
||||
|
||||
In **railiance-apps**:
|
||||
|
||||
- Set `cnpg-option-a-schedule` `mode` to `activity-core` (or dual-run during
|
||||
soak).
|
||||
- `cnpg-backup-status` accepts activity-core last-success (status CM and/or
|
||||
hub evidence age).
|
||||
- Document that workstation cron is **stopgap only**; remove from operator
|
||||
“required” path after soak.
|
||||
- Optionally un-suspend in-cluster CronJobs **only if** T02 image path lands
|
||||
in-cluster; otherwise leave suspended.
|
||||
|
||||
**Done when:** docs + status tool describe activity-core as primary; workstation
|
||||
optional.
|
||||
`cnpg-backup-status` accepts `mode=activity-core`. Full cutover after first
|
||||
successful scheduled runs (T07). Workstation remains documented stopgap.
|
||||
|
||||
## Task: Soak, restore drill from automated artifact, finish
|
||||
|
||||
```task
|
||||
id: RAILIANCE-WP-0016-T07
|
||||
status: todo
|
||||
status: wait
|
||||
priority: high
|
||||
state_hub_task_id: "f331bdcd-0e76-4bc0-b85a-1bb1227a876b"
|
||||
```
|
||||
|
||||
- ≥3 consecutive successful scheduled runs (or 7 if aligning forgejo promotion
|
||||
gate language).
|
||||
- Restore drill from an artifact produced by activity-core (not workstation
|
||||
one-off), row-count or equivalent gate, evidence JSON under `docs/evidence/`.
|
||||
- Update `STATE.md`; archive/finish this workplan; optional message to
|
||||
activity-core / platform owners if residual tasks remain in their repos.
|
||||
|
||||
**Done when:** RPO/RTO evidence recorded; workstation powered-off scenario is
|
||||
credible; workplan `finished`.
|
||||
Blocked on T03–T05 enable path.
|
||||
|
||||
---
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Switching Option A → barman ObjectStore / CNPG `ScheduledBackup` (Phase 2;
|
||||
only if explicitly re-decided).
|
||||
- Media/PVC app data backups (vergage media still disabled).
|
||||
- Rotating EXPOSED taint history on the offsite lane (platform CCR ops), except
|
||||
as a dependency note if worker fetch is blocked.
|
||||
|
||||
## Suggested implementation order
|
||||
|
||||
```text
|
||||
T01 inventory → T02 runner package → T03 creds + T04 kube reachability (parallel)
|
||||
→ T05 activity-core definition → T06 cutover docs/status → T07 soak + drill
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- `STATE.md` (this repo) — current posture
|
||||
- `workplans/RAILIANCE-WP-0015-…` — interim Option A
|
||||
- `docs/evidence/cnpg-option-a-backup-20260722.json`
|
||||
- `activity-core/activity-definitions/weekly-forgejo-package-prune.md`
|
||||
- `railiance-platform/docs/forgejo-backup.md`
|
||||
- `activity-core/docs/runbook.md` (hostPath + ESO patterns)
|
||||
- `STATE.md`
|
||||
- `docs/cnpg-backup-topology-inventory.md`
|
||||
- `railiance-platform/tools/cmd/cnpg-option-a-backup`
|
||||
- `activity-core/activity-definitions/daily-cnpg-option-a-backup.md`
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue