railiance-apps/STATE.md

STATE — railiance-apps

Updated: 2026-07-22
Domain: financials · Repo: railiance-apps
Topic ID: ca369340-a64e-442e-98f1-a4fa7dc74a38

This file is the human/agent snapshot of where the repo stands. Prefer it over chat history for orientation. Generated custodian brief is .custodian-brief.md (do not edit). Formal work structure lives in workplans/.

---

One-line posture

S5 app releases (vergabe, hubs, Forgejo consumer surface) are operational; CNPG Option A logical backups exist and currently report healthy, but unattended execution still depends on a workstation path and must move to railiance01 + activity-core so the laptop is not a durability SPOF.

---

Clusters (as of 2026-07-22)

Host	Kubeconfig	CNPG clusters in databases	Role
CoulombCore	~/.kube/config	apps-pg, gitea-db, net-kingdom-pg, state-hub-db	S5 apps DBs, legacy gitea-db, NK + state-hub instances
railiance01	~/.kube/config-hosteurope	forgejo-db, net-kingdom-pg, state-hub-db	Forgejo DB; NK + state-hub instances (may differ from Core)

PRODUCTION_KUBECONFIG defaults to CoulombCore. Forgejo tooling uses railiance01.

---

Backup durability (RAILIANCE-WP-0015 — finished)

Goal met

Drive make cnpg-backup-status degraded → healthy with Option A (age-encrypted logical dump → Nextcloud WebDAV), not barman ScheduledBackup.

What is live today

Item	State
OpenBao lane CCR-2026-0004 / railiance-backup-offsite-lane	resolvable, fields present
Secret databases/cnpg-backup-offsite (CoulombCore)	present (NC_WEBDAV_*, AGE_PUBLIC_KEY only)
ConfigMap cnpg-option-a-schedule	mode=workstation-cron @ 30 2 * * *
ConfigMap cnpg-option-a-status	last-success timestamps for all four Core clusters (2026-07-22)
CronJobs cnpg-logical-backup-*	suspended (cluster egress cannot install age)
Workstation runner tools/cnpg-logical-backup.sh	works (upload + status CM)
Restore drill	apps-pg/vergage_db age artifact 195/195
Evidence	docs/evidence/cnpg-option-a-backup-20260722.json

What is not good enough

1. Workstation SPOF — daily unattended path is “someone’s machine has crontab + bao OIDC + kubeconfig”. Laptop sleep/off breaks RPO.
2. In-cluster CronJobs blocked — pods cannot reach GitHub or Alpine CDN to get age; no prebuilt image with kubectl+age+curl yet.
3. Split brain topology — Option A runner today targets CoulombCore four clusters only; forgejo-db (railiance01) still uses platform make forgejo-backup (also historically workstation-cron).
4. No activity-core schedule — unlike weekly Forgejo package prune, backups are not Temporal/ActivityDefinition managed, so automation inventory and automation-status cannot own them.
5. Credential delivery — offsite lane is human OIDC on workstation; worker needs non-interactive OpenBao/ESO (or sealed secret) path.

Operator residual (manual)

# Not yet proven installed on a always-on host — install on railiance01 or leave as stopgap
30 2 * * * cd $HOME/railiance-apps && make cnpg-logical-backup >>$HOME/.cache/railiance/backups/cnpg/cron.log 2>&1

---

Adjacent finished work

Workplan	Outcome
RAILIANCE-WP-0013	Phase 1 apps-pg/vergage_db backup + restore drill (interim S5 gate)
RAILIANCE-WP-0015	Fleet Option A tooling + healthy status on CoulombCore (workstation-backed)
RAIL-HO-WP-0005	Forgejo prod migration; descoped residual backup → WP-0015
ACTIVITY-WP-0020	Weekly forgejo package prune via activity-core on railiance01 (pattern to copy)

---

Next major strand

Move unattended Option A (+ forgejo-backup) to railiance01, scheduled by activity-core — no workstation required.

Tracking workplan: RAILIANCE-WP-0016
workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md

Design sketch (see workplan for tasks):

                    ┌─────────────────────────────┐
                    │ activity-core (railiance01) │
                    │ Temporal cron ActivityDef   │
                    └─────────────┬───────────────┘
                                  │ shell context
                                  ▼
                    ┌─────────────────────────────┐
                    │ hostPath /opt/railiance-*   │
                    │ backup CLI (age+kubectl)    │
                    │ OpenBao/ESO offsite secrets │
                    └─────────────┬───────────────┘
               ┌──────────────────┼──────────────────┐
               ▼                  ▼                  ▼
        CoulombCore API    railiance01 API     Nextcloud WebDAV
        (apps-pg, …)       (forgejo-db, …)     (age artifacts)

Proven pattern: weekly-forgejo-package-prune ActivityDefinition + /opt/railiance-platform hostPath + ESO credentials + State Hub evidence sink.

---

Active / proposed workplans

ID	Status	Title
RAILIANCE-WP-0015	finished	CNPG Option A coverage → healthy (workstation interim)
RAILIANCE-WP-0016	proposed	railiance01 + activity-core unattended backup automation

(No other active workplans in hub as of last brief.)

---

Quick operator commands

# Health (CoulombCore)
make cnpg-backup-status

# Manual Option A run (still workstation / OpenBao OIDC today)
bao login -method=oidc -path=netkingdom role=railiance-backup-workload-kv-read
make cnpg-logical-backup

# Secret re-materialize
make cnpg-backup-offsite-secret-apply

---

Risks / open questions

Risk	Mitigation direction (WP-0016)
Dual NK/state-hub DB instances on two hosts	Inventory which is production-of-record per consumer
CoulombCore access from railiance01	Dedicated kubeconfig + ops-warden cert / tunnel for worker
Age binary in restricted network	Vendor static binary in image or hostPath tools tree
High-risk WebDAV + age keys in worker	ESO from OpenBao; never private key in Git; agent boundary
RPO if activity-core worker down	Temporal schedule health + status CM freshness alerts

---

Related docs

• docs/app-data-backup-restore-handoff.md
• docs/credential-routing-railiance-apps.md
• manifests/cnpg-option-a-backup.yaml
• docs/evidence/cnpg-option-a-backup-20260722.json
• activity-core: activity-definitions/weekly-forgejo-package-prune.md, docs/runbook.md
• platform: docs/forgejo-backup.md