diff --git a/history/2026-07-04-forgejo-backup-strategy-assessment.md b/history/2026-07-04-forgejo-backup-strategy-assessment.md new file mode 100644 index 0000000..e08192b --- /dev/null +++ b/history/2026-07-04-forgejo-backup-strategy-assessment.md @@ -0,0 +1,151 @@ +# Forgejo / Railiance Backup Strategy Assessment + +Date: 2026-07-04 +Author: grok (session on `the-custodian`) +`no_secret_material_recorded: true` +Status: **assessment recorded — pickup from disaster-control** + +## Context + +Assessment performed after **RAIL-HO-WP-0005-T09** isolated restore drill (2026-07-04). +Restore path is proven; scheduled backups and off-cluster durability are not. + +Related workplans and evidence: + +| Artifact | Location | +| --- | --- | +| Forgejo migration umbrella | `railiance-infra/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md` | +| Restore drill evidence | `railiance-infra/docs/forgejo-restore-drill-evidence.md` | +| Restore orchestration | `railiance-infra/tools/forgejo-restore-drill.sh` | +| Open production decisions | `the-custodian/docs/forgejo-production-decisions.md` (decision #7 TBD) | +| App/platform handoff | `railiance-apps/docs/app-data-backup-restore-handoff.md` | + +## Executive summary + +Backup posture is **layered on paper, thin in practice** for Forgejo and most +platform data. We can recover Forgejo if an operator manually runs `forgejo dump` +and knows the restore script. We do **not** yet have a production-grade backup +strategy: scheduled jobs, encrypted off-cluster storage, retention policy, and +Gitea parity are all open before tier-3 (`state-hub`) or production cutover. + +**Honest grade:** restore procedure **B+**; backup automation **D**; +off-cluster durability **F**; Gitea (canonical source forge) **at risk**. + +## Layer inventory + +| Layer | What it should cover | Current state | +| --- | --- | --- | +| **Restore drill** | Prove Forgejo recovery | **Done** (2026-07-04) — manual `forgejo dump` restored in `forgejo-restore-drill`; health 200; `glas-harness`, `key-cape`, 3 org repos verified | +| **Forgejo production** | DB + git repos + packages + attachments | **Manual only** — no schedule, no off-cluster target | +| **CNPG (`forgejo-db`, `apps-pg`)** | DB base backups + WAL | **Not configured** — no `Backup` / `ScheduledBackup` CRs on railiance01 | +| **Cluster runtime (S2)** | k3s state, Helm values, kubeconfig | **Works locally** — `railiance-cluster/tools/cmd/railiance-backup-s2` → age-encrypted files in `/opt/backup/railiance/cluster/`, keep 7; no network upload | +| **Platform (S3)** | Legacy PostgreSQL HA, Valkey → Nextcloud | **Documented, not wired** — `railiance-platform/Makefile` `backup` target references `tools/cmd/railiance-backup`, but that tool is absent from the repo (relocation planned/cancelled) | +| **Gitea (coulombcore)** | Canonical source forge today | **Weak** — bundled PG + `gitea-shared-storage` PVC; prior audit found no Gitea backup CronJobs; package blob backup not evidenced | +| **Adjacent SSO/MFA** | privacyidea, authelia, lldap | **Scheduled** — daily backup CronJobs exist on railiance01 | + +Live cluster check (2026-07-04): `kubectl get backup,scheduledbackup -A` → no +resources. Backup-related CronJobs: `privacyidea-backup`, `authelia-backup`, +`lldap-backup` only. + +## What T09 proved (restore, not backup) + +The drill validated **recovery mechanics**, not operational backup coverage: + +1. `forgejo dump` from production pod (~11.7 MiB: `repos/`, `data/`, + `forgejo-db.sql`, `app.ini`) +2. Manual unzip + file copy + `psql` import (Forgejo 11.0.3 has no + `forgejo restore` CLI) +3. Isolated Helm deploy in `forgejo-restore-drill` +4. API checks: health 200; `coulomb/glas-harness` and `coulomb/key-cape` on + `main`; 3 org repos visible + +| Metric | Observed | +| --- | --- | +| RPO (operational today) | Point-in-time of last manual `forgejo dump` | +| RTO (isolated restore) | ~3–5 minutes (CNPG ready + import + Helm on railiance01) | +| Production impact during drill | None — read-only dump; separate namespace | + +Re-run hygiene: `DRILL_CLEAN=1 ./tools/forgejo-restore-drill.sh` (SQL import is +not idempotent). + +## Gaps blocking production cutover + +1. **No scheduled Forgejo backups** — no CronJob/operator for `forgejo dump` +2. **No CNPG-native backups** for `forgejo-db` — cluster manifest has storage + only (`railiance-platform/helm/forgejo-db-cluster.yaml`) +3. **No approved off-cluster target** — drill archive on workstation `/tmp` + only; encryption/retention undecided (T02 decision #7) +4. **Platform backup tool missing** — S3 `make backup` target without + implementation +5. **Gitea canonical data** — safety contract requires fresh Gitea backup before + each migration wave; no automated Gitea backup on coulombcore +6. **Split-brain restore risk** — restoring only CNPG DB would miss git blobs on + the Forgejo app PVC; full `forgejo dump` is the coherent unit today + +## Target architecture (intended, not implemented) + +Per `RAIL-HO-WP-0005` repo boundaries: + +```text +S2 railiance-cluster + etcd/state, helm values, kubeconfig + local age-encrypted, 7-day retention + +S3 railiance-platform + CNPG ScheduledBackup → off-cluster target (MinIO/S3/Nextcloud — TBD) + implement or restore railiance-backup tool + +S5 railiance-apps + scheduled forgejo dump (repos + data + SQL + app.ini) + include OCI package blobs; age-encrypt; upload to approved target + +Evidence loop (this repo) + quarterly restore drill → isolated namespace + dated evidence under history/ or TestEvidence/ +``` + +Ownership: `railiance-platform` owns backup mechanism; `railiance-apps` owns +readiness gates and restore evidence per +`docs/app-data-backup-restore-handoff.md`. + +## Pickup queue (for disaster-control) + +Track and drive closure of these items; implementation spans multiple repos. + +| # | Item | Owner repo | Blocker | +| --- | --- | --- | --- | +| 1 | Operator decision: backup destination + retention | `railiance-infra` / T02 | Human — decision #7 in `forgejo-production-decisions.md` | +| 2 | CNPG `ScheduledBackup` on `forgejo-db` | `railiance-platform` | #1 + backup target wiring | +| 3 | Scheduled `forgejo dump` → age-encrypt → upload | `railiance-apps` | #1 | +| 4 | Implement or restore `tools/cmd/railiance-backup` | `railiance-platform` | Relocation never completed | +| 5 | Gitea pre-cutover snapshot runbook + evidence | `railiance-infra` / `railiance-apps` | No automated Gitea backup today | +| 6 | `apps-pg` backup coverage | `railiance-platform` | Documented follow-up in `docs/apps-pg.md` | +| 7 | Quarterly restore drill cadence + evidence | `disaster-control` + `railiance-infra` | Calendar + RTO/RPO matrix not written | +| 8 | Populate `RTO-RPO-Matrix.md`, `BackupPolicy.md`, registers | `disaster-control` | Per `INTENT.md` / `specs/ResilienceControlSetup.md` | + +**Gate:** tier-3 production repos (`state-hub`, etc.) remain blocked until T09 +scheduled automation and T02 backup target are closed (`RAIL-HO-WP-0005`, +`CUST-WP-0054`). + +## Suggested RTO/RPO targets (draft — not adopted) + +Aligns with `specs/ResilienceControlSetup.md` tier **T1** (irreplaceable source +of truth) until operator confirms: + +| System | Draft RPO | Draft RTO | Notes | +| --- | --- | --- | --- | +| Forgejo git + packages | 24h | 4–8h | Daily `forgejo dump` + CNPG backup | +| Gitea (canonical until cutover) | 24h | 4–8h | Mirror + dump before each migration wave | +| `forgejo-db` CNPG | 1h–24h | 2–4h | WAL + base backup to off-cluster target | +| k3s cluster state (S2) | 24h | 4–8h | Existing `railiance-backup-s2` local retention | + +## References + +- `railiance-infra/docs/forgejo-restore-drill-evidence.md` +- `railiance-infra/tools/forgejo-restore-drill.sh` +- `railiance-platform/helm/forgejo-db-cluster.yaml` +- `railiance-platform/docs/apps-pg.md` (backup follow-up note) +- `railiance-platform/docs/postgresql-ha.md` (intended platform backup) +- `the-custodian/docs/forgejo-production-decisions.md` +- `disaster-control/specs/ResilienceControlSetup.md` +- `disaster-control/INTENT.md` \ No newline at end of file diff --git a/history/README.md b/history/README.md new file mode 100644 index 0000000..6d2e1e4 --- /dev/null +++ b/history/README.md @@ -0,0 +1,7 @@ +# History + +Dated assessments, audits, and decision snapshots for the Resilience Control +Plane. Active runbooks and policies live at repo root when written; `history/` +preserves point-in-time findings that inform later work. + +Naming: `YYYY-MM-DD-.md` \ No newline at end of file