168 lines
6.8 KiB
Markdown
168 lines
6.8 KiB
Markdown
|
# STATE — railiance-apps
|
|||
|
|
|
|||
|
|
**Updated:** 2026-07-22
|
|||
|
|
**Domain:** financials · **Repo:** railiance-apps
|
|||
|
|
**Topic ID:** `ca369340-a64e-442e-98f1-a4fa7dc74a38`
|
|||
|
|
|
|||
|
|
This file is the human/agent snapshot of **where the repo stands**. Prefer it over
|
|||
|
|
chat history for orientation. Generated custodian brief is `.custodian-brief.md`
|
|||
|
|
(do not edit). Formal work structure lives in `workplans/`.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## One-line posture
|
|||
|
|
|
|||
|
|
S5 app releases (vergabe, hubs, Forgejo consumer surface) are operational; **CNPG
|
|||
|
|
Option A logical backups exist and currently report healthy**, but unattended
|
|||
|
|
execution still depends on a **workstation path** and must move to **railiance01 +
|
|||
|
|
activity-core** so the laptop is not a durability SPOF.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Clusters (as of 2026-07-22)
|
|||
|
|
|
|||
|
|
| Host | Kubeconfig | CNPG clusters in `databases` | Role |
|
|||
|
|
| --- | --- | --- | --- |
|
|||
|
|
| **CoulombCore** | `~/.kube/config` | `apps-pg`, `gitea-db`, `net-kingdom-pg`, `state-hub-db` | S5 apps DBs, legacy gitea-db, NK + state-hub instances |
|
|||
|
|
| **railiance01** | `~/.kube/config-hosteurope` | `forgejo-db`, `net-kingdom-pg`, `state-hub-db` | Forgejo DB; NK + state-hub instances (may differ from Core) |
|
|||
|
|
|
|||
|
|
`PRODUCTION_KUBECONFIG` defaults to CoulombCore. Forgejo tooling uses railiance01.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Backup durability (RAILIANCE-WP-0015 — **finished**)
|
|||
|
|
|
|||
|
|
### Goal met
|
|||
|
|
|
|||
|
|
Drive `make cnpg-backup-status` **degraded → healthy** with Option A (age-encrypted
|
|||
|
|
logical dump → Nextcloud WebDAV), not barman `ScheduledBackup`.
|
|||
|
|
|
|||
|
|
### What is live today
|
|||
|
|
|
|||
|
|
| Item | State |
|
|||
|
|
| --- | --- |
|
|||
|
|
| OpenBao lane `CCR-2026-0004` / `railiance-backup-offsite-lane` | **resolvable**, fields present |
|
|||
|
|
| Secret `databases/cnpg-backup-offsite` (CoulombCore) | **present** (`NC_WEBDAV_*`, `AGE_PUBLIC_KEY` only) |
|
|||
|
|
| ConfigMap `cnpg-option-a-schedule` | **mode=workstation-cron** @ `30 2 * * *` |
|
|||
|
|
| ConfigMap `cnpg-option-a-status` | last-success timestamps for all four Core clusters (2026-07-22) |
|
|||
|
|
| CronJobs `cnpg-logical-backup-*` | **suspended** (cluster egress cannot install `age`) |
|
|||
|
|
| Workstation runner `tools/cnpg-logical-backup.sh` | **works** (upload + status CM) |
|
|||
|
|
| Restore drill | apps-pg/`vergage_db` age artifact **195/195** |
|
|||
|
|
| Evidence | `docs/evidence/cnpg-option-a-backup-20260722.json` |
|
|||
|
|
|
|||
|
|
### What is *not* good enough
|
|||
|
|
|
|||
|
|
1. **Workstation SPOF** — daily unattended path is “someone’s machine has crontab +
|
|||
|
|
bao OIDC + kubeconfig”. Laptop sleep/off breaks RPO.
|
|||
|
|
2. **In-cluster CronJobs blocked** — pods cannot reach GitHub or Alpine CDN to get
|
|||
|
|
`age`; no prebuilt image with `kubectl`+`age`+`curl` yet.
|
|||
|
|
3. **Split brain topology** — Option A runner today targets **CoulombCore** four
|
|||
|
|
clusters only; **forgejo-db** (railiance01) still uses platform `make forgejo-backup`
|
|||
|
|
(also historically workstation-cron).
|
|||
|
|
4. **No activity-core schedule** — unlike weekly Forgejo package prune, backups are
|
|||
|
|
not Temporal/`ActivityDefinition` managed, so automation inventory and
|
|||
|
|
`automation-status` cannot own them.
|
|||
|
|
5. **Credential delivery** — offsite lane is human OIDC on workstation; worker needs
|
|||
|
|
non-interactive OpenBao/ESO (or sealed secret) path.
|
|||
|
|
|
|||
|
|
### Operator residual (manual)
|
|||
|
|
|
|||
|
|
```cron
|
|||
|
|
# Not yet proven installed on a always-on host — install on railiance01 or leave as stopgap
|
|||
|
|
30 2 * * * cd $HOME/railiance-apps && make cnpg-logical-backup >>$HOME/.cache/railiance/backups/cnpg/cron.log 2>&1
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Adjacent finished work
|
|||
|
|
|
|||
|
|
| Workplan | Outcome |
|
|||
|
|
| --- | --- |
|
|||
|
|
| RAILIANCE-WP-0013 | Phase 1 apps-pg/`vergage_db` backup + restore drill (interim S5 gate) |
|
|||
|
|
| RAILIANCE-WP-0015 | Fleet Option A tooling + healthy status on CoulombCore (workstation-backed) |
|
|||
|
|
| RAIL-HO-WP-0005 | Forgejo prod migration; descoped residual backup → WP-0015 |
|
|||
|
|
| ACTIVITY-WP-0020 | Weekly forgejo package prune via **activity-core** on railiance01 (pattern to copy) |
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Next major strand
|
|||
|
|
|
|||
|
|
**Move unattended Option A (+ forgejo-backup) to railiance01, scheduled by
|
|||
|
|
activity-core — no workstation required.**
|
|||
|
|
|
|||
|
|
Tracking workplan: **`RAILIANCE-WP-0016`**
|
|||
|
|
`workplans/RAILIANCE-WP-0016-railiance01-activity-core-backup-automation.md`
|
|||
|
|
|
|||
|
|
Design sketch (see workplan for tasks):
|
|||
|
|
|
|||
|
|
```text
|
|||
|
|
┌─────────────────────────────┐
|
|||
|
|
│ activity-core (railiance01) │
|
|||
|
|
│ Temporal cron ActivityDef │
|
|||
|
|
└─────────────┬───────────────┘
|
|||
|
|
│ shell context
|
|||
|
|
▼
|
|||
|
|
┌─────────────────────────────┐
|
|||
|
|
│ hostPath /opt/railiance-* │
|
|||
|
|
│ backup CLI (age+kubectl) │
|
|||
|
|
│ OpenBao/ESO offsite secrets │
|
|||
|
|
└─────────────┬───────────────┘
|
|||
|
|
┌──────────────────┼──────────────────┐
|
|||
|
|
▼ ▼ ▼
|
|||
|
|
CoulombCore API railiance01 API Nextcloud WebDAV
|
|||
|
|
(apps-pg, …) (forgejo-db, …) (age artifacts)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**Proven pattern:** `weekly-forgejo-package-prune` ActivityDefinition +
|
|||
|
|
`/opt/railiance-platform` hostPath + ESO credentials + State Hub evidence sink.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Active / proposed workplans
|
|||
|
|
|
|||
|
|
| ID | Status | Title |
|
|||
|
|
| --- | --- | --- |
|
|||
|
|
| RAILIANCE-WP-0015 | **finished** | CNPG Option A coverage → healthy (workstation interim) |
|
|||
|
|
| RAILIANCE-WP-0016 | **proposed** | railiance01 + activity-core unattended backup automation |
|
|||
|
|
|
|||
|
|
(No other active workplans in hub as of last brief.)
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Quick operator commands
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
# Health (CoulombCore)
|
|||
|
|
make cnpg-backup-status
|
|||
|
|
|
|||
|
|
# Manual Option A run (still workstation / OpenBao OIDC today)
|
|||
|
|
bao login -method=oidc -path=netkingdom role=railiance-backup-workload-kv-read
|
|||
|
|
make cnpg-logical-backup
|
|||
|
|
|
|||
|
|
# Secret re-materialize
|
|||
|
|
make cnpg-backup-offsite-secret-apply
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Risks / open questions
|
|||
|
|
|
|||
|
|
| Risk | Mitigation direction (WP-0016) |
|
|||
|
|
| --- | --- |
|
|||
|
|
| Dual NK/state-hub DB instances on two hosts | Inventory which is production-of-record per consumer |
|
|||
|
|
| CoulombCore access from railiance01 | Dedicated kubeconfig + ops-warden cert / tunnel for worker |
|
|||
|
|
| Age binary in restricted network | Vendor static binary in image or hostPath tools tree |
|
|||
|
|
| High-risk WebDAV + age keys in worker | ESO from OpenBao; never private key in Git; agent boundary |
|
|||
|
|
| RPO if activity-core worker down | Temporal schedule health + status CM freshness alerts |
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Related docs
|
|||
|
|
|
|||
|
|
- `docs/app-data-backup-restore-handoff.md`
|
|||
|
|
- `docs/credential-routing-railiance-apps.md`
|
|||
|
|
- `manifests/cnpg-option-a-backup.yaml`
|
|||
|
|
- `docs/evidence/cnpg-option-a-backup-20260722.json`
|
|||
|
|
- activity-core: `activity-definitions/weekly-forgejo-package-prune.md`, `docs/runbook.md`
|
|||
|
|
- platform: `docs/forgejo-backup.md`
|