RAILIANCE-WP-0016: promote active; inventory and activity-core cutover prep
Mark workplan active with T01/T02 done, document topology, extend status for activity-core mode, and wire Make targets to the platform multi-host backup CLI. T03 remains operator-blocked on ESO token.
This commit is contained in:
parent
04626b6335
commit
c202fbf7be
6 changed files with 180 additions and 309 deletions
186
STATE.md
186
STATE.md
|
|
@ -4,164 +4,74 @@
|
|||
**Domain:** financials · **Repo:** railiance-apps
|
||||
**Topic ID:** `ca369340-a64e-442e-98f1-a4fa7dc74a38`
|
||||
|
||||
This file is the human/agent snapshot of **where the repo stands**. Prefer it over
|
||||
chat history for orientation. Generated custodian brief is `.custodian-brief.md`
|
||||
(do not edit). Formal work structure lives in `workplans/`.
|
||||
|
||||
---
|
||||
|
||||
## One-line posture
|
||||
|
||||
S5 app releases (vergabe, hubs, Forgejo consumer surface) are operational; **CNPG
|
||||
Option A logical backups exist and currently report healthy**, but unattended
|
||||
execution still depends on a **workstation path** and must move to **railiance01 +
|
||||
activity-core** so the laptop is not a durability SPOF.
|
||||
S5 releases operational; **CNPG Option A is healthy on CoulombCore** (WP-0015).
|
||||
**Unattended control plane migration to railiance01 + activity-core is in
|
||||
progress (WP-0016)** — code/manifests landed; **blocked on operator ESO token
|
||||
re-mint + CoulombCore kubeconfig on railiance01 host**.
|
||||
|
||||
---
|
||||
|
||||
## Clusters (as of 2026-07-22)
|
||||
## Workplans
|
||||
|
||||
| Host | Kubeconfig | CNPG clusters in `databases` | Role |
|
||||
| --- | --- | --- | --- |
|
||||
| **CoulombCore** | `~/.kube/config` | `apps-pg`, `gitea-db`, `net-kingdom-pg`, `state-hub-db` | S5 apps DBs, legacy gitea-db, NK + state-hub instances |
|
||||
| **railiance01** | `~/.kube/config-hosteurope` | `forgejo-db`, `net-kingdom-pg`, `state-hub-db` | Forgejo DB; NK + state-hub instances (may differ from Core) |
|
||||
|
||||
`PRODUCTION_KUBECONFIG` defaults to CoulombCore. Forgejo tooling uses railiance01.
|
||||
|
||||
---
|
||||
|
||||
## Backup durability (RAILIANCE-WP-0015 — **finished**)
|
||||
|
||||
### Goal met
|
||||
|
||||
Drive `make cnpg-backup-status` **degraded → healthy** with Option A (age-encrypted
|
||||
logical dump → Nextcloud WebDAV), not barman `ScheduledBackup`.
|
||||
|
||||
### What is live today
|
||||
|
||||
| Item | State |
|
||||
| --- | --- |
|
||||
| OpenBao lane `CCR-2026-0004` / `railiance-backup-offsite-lane` | **resolvable**, fields present |
|
||||
| Secret `databases/cnpg-backup-offsite` (CoulombCore) | **present** (`NC_WEBDAV_*`, `AGE_PUBLIC_KEY` only) |
|
||||
| ConfigMap `cnpg-option-a-schedule` | **mode=workstation-cron** @ `30 2 * * *` |
|
||||
| ConfigMap `cnpg-option-a-status` | last-success timestamps for all four Core clusters (2026-07-22) |
|
||||
| CronJobs `cnpg-logical-backup-*` | **suspended** (cluster egress cannot install `age`) |
|
||||
| Workstation runner `tools/cnpg-logical-backup.sh` | **works** (upload + status CM) |
|
||||
| Restore drill | apps-pg/`vergage_db` age artifact **195/195** |
|
||||
| Evidence | `docs/evidence/cnpg-option-a-backup-20260722.json` |
|
||||
|
||||
### What is *not* good enough
|
||||
|
||||
1. **Workstation SPOF** — daily unattended path is “someone’s machine has crontab +
|
||||
bao OIDC + kubeconfig”. Laptop sleep/off breaks RPO.
|
||||
2. **In-cluster CronJobs blocked** — pods cannot reach GitHub or Alpine CDN to get
|
||||
`age`; no prebuilt image with `kubectl`+`age`+`curl` yet.
|
||||
3. **Split brain topology** — Option A runner today targets **CoulombCore** four
|
||||
clusters only; **forgejo-db** (railiance01) still uses platform `make forgejo-backup`
|
||||
(also historically workstation-cron).
|
||||
4. **No activity-core schedule** — unlike weekly Forgejo package prune, backups are
|
||||
not Temporal/`ActivityDefinition` managed, so automation inventory and
|
||||
`automation-status` cannot own them.
|
||||
5. **Credential delivery** — offsite lane is human OIDC on workstation; worker needs
|
||||
non-interactive OpenBao/ESO (or sealed secret) path.
|
||||
|
||||
### Operator residual (manual)
|
||||
|
||||
```cron
|
||||
# Not yet proven installed on a always-on host — install on railiance01 or leave as stopgap
|
||||
30 2 * * * cd $HOME/railiance-apps && make cnpg-logical-backup >>$HOME/.cache/railiance/backups/cnpg/cron.log 2>&1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adjacent finished work
|
||||
|
||||
| Workplan | Outcome |
|
||||
| --- | --- |
|
||||
| RAILIANCE-WP-0013 | Phase 1 apps-pg/`vergage_db` backup + restore drill (interim S5 gate) |
|
||||
| RAILIANCE-WP-0015 | Fleet Option A tooling + healthy status on CoulombCore (workstation-backed) |
|
||||
| RAIL-HO-WP-0005 | Forgejo prod migration; descoped residual backup → WP-0015 |
|
||||
| ACTIVITY-WP-0020 | Weekly forgejo package prune via **activity-core** on railiance01 (pattern to copy) |
|
||||
|
||||
---
|
||||
|
||||
## Next major strand
|
||||
|
||||
**Move unattended Option A (+ forgejo-backup) to railiance01, scheduled by
|
||||
activity-core — no workstation required.**
|
||||
|
||||
Tracking workplan: **`RAILIANCE-WP-0016`**
|
||||
`workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md`
|
||||
|
||||
Design sketch (see workplan for tasks):
|
||||
|
||||
```text
|
||||
┌─────────────────────────────┐
|
||||
│ activity-core (railiance01) │
|
||||
│ Temporal cron ActivityDef │
|
||||
└─────────────┬───────────────┘
|
||||
│ shell context
|
||||
▼
|
||||
┌─────────────────────────────┐
|
||||
│ hostPath /opt/railiance-* │
|
||||
│ backup CLI (age+kubectl) │
|
||||
│ OpenBao/ESO offsite secrets │
|
||||
└─────────────┬───────────────┘
|
||||
┌──────────────────┼──────────────────┐
|
||||
▼ ▼ ▼
|
||||
CoulombCore API railiance01 API Nextcloud WebDAV
|
||||
(apps-pg, …) (forgejo-db, …) (age artifacts)
|
||||
```
|
||||
|
||||
**Proven pattern:** `weekly-forgejo-package-prune` ActivityDefinition +
|
||||
`/opt/railiance-platform` hostPath + ESO credentials + State Hub evidence sink.
|
||||
|
||||
---
|
||||
|
||||
## Active / proposed workplans
|
||||
|
||||
| ID | Status | Title |
|
||||
| ID | Status | Notes |
|
||||
| --- | --- | --- |
|
||||
| RAILIANCE-WP-0015 | **finished** | CNPG Option A coverage → healthy (workstation interim) |
|
||||
| RAILIANCE-WP-0016 | **proposed** | railiance01 + activity-core unattended backup automation |
|
||||
| RAILIANCE-WP-0015 | **finished** | Option A tooling; workstation stopgap schedule |
|
||||
| RAILIANCE-WP-0016 | **active** | activity-core unattended path — T01/T02 done; T03 human; T04–T07 open |
|
||||
|
||||
(No other active workplans in hub as of last brief.)
|
||||
Hub WP-0016: `0c8ddbc9-3740-4e0a-acb2-901e050ab242`
|
||||
|
||||
---
|
||||
|
||||
## Quick operator commands
|
||||
## WP-0016 progress snapshot
|
||||
|
||||
**Done**
|
||||
|
||||
- Topology inventory: `docs/cnpg-backup-topology-inventory.md`
|
||||
- Platform multi-host CLI: `~/railiance-platform/tools/cmd/cnpg-option-a-backup`
|
||||
- Static `age` vendored; kubectl via install script
|
||||
- Dry-run: `r01-forgejo-db` overall=ok
|
||||
- activity-core: resolver, definition (disabled), ESO CR, RBAC, worker yaml mounts
|
||||
- Unit tests: 3 passed (`test_cnpg_option_a_backup_resolver.py`)
|
||||
|
||||
**Blocked / operator**
|
||||
|
||||
1. Re-mint `openbao-activity-core-eso-token` with
|
||||
`workload-kv-read-railiance-backup-offsite-lane`
|
||||
→ ExternalSecret `actcore-backup-offsite` currently **SecretSyncedError**
|
||||
2. On railiance01 host: `install-cnpg-backup-vendor-tools`; ensure
|
||||
`~/.kube/config` (Core) + `config-hosteurope` (R01)
|
||||
3. Roll worker image/code + hostPath kubeconfigs; enable
|
||||
`daily-cnpg-option-a-backup`; forced run + soak
|
||||
|
||||
---
|
||||
|
||||
## Clusters (reminder)
|
||||
|
||||
| Host | POR backup targets |
|
||||
| --- | --- |
|
||||
| CoulombCore | apps-pg, gitea-db, net-kingdom-pg, state-hub-db |
|
||||
| railiance01 | forgejo-db, net-kingdom-pg, state-hub-db (dual with Core for NK/SH) |
|
||||
|
||||
---
|
||||
|
||||
## Quick commands
|
||||
|
||||
```bash
|
||||
# Health (CoulombCore)
|
||||
make cnpg-backup-status
|
||||
|
||||
# Manual Option A run (still workstation / OpenBao OIDC today)
|
||||
bao login -method=oidc -path=netkingdom role=railiance-backup-workload-kv-read
|
||||
make cnpg-logical-backup
|
||||
|
||||
# Secret re-materialize
|
||||
make cnpg-backup-offsite-secret-apply
|
||||
make cnpg-option-a-backup-dry-run # platform multi-host JSON
|
||||
# After ESO + host wiring:
|
||||
# enable activity-core definition; make automation-status
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Risks / open questions
|
||||
|
||||
| Risk | Mitigation direction (WP-0016) |
|
||||
| --- | --- |
|
||||
| Dual NK/state-hub DB instances on two hosts | Inventory which is production-of-record per consumer |
|
||||
| CoulombCore access from railiance01 | Dedicated kubeconfig + ops-warden cert / tunnel for worker |
|
||||
| Age binary in restricted network | Vendor static binary in image or hostPath tools tree |
|
||||
| High-risk WebDAV + age keys in worker | ESO from OpenBao; never private key in Git; agent boundary |
|
||||
| RPO if activity-core worker down | Temporal schedule health + status CM freshness alerts |
|
||||
|
||||
---
|
||||
|
||||
## Related docs
|
||||
|
||||
- `docs/cnpg-backup-topology-inventory.md`
|
||||
- `docs/app-data-backup-restore-handoff.md`
|
||||
- `docs/credential-routing-railiance-apps.md`
|
||||
- `manifests/cnpg-option-a-backup.yaml`
|
||||
- `docs/evidence/cnpg-option-a-backup-20260722.json`
|
||||
- activity-core: `activity-definitions/weekly-forgejo-package-prune.md`, `docs/runbook.md`
|
||||
- platform: `docs/forgejo-backup.md`
|
||||
- `workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md`
|
||||
- `railiance-platform/docs/cnpg-option-a-backup.md`
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue