Record Forgejo backup strategy assessment in history/
Persist 2026-07-04 post-drill findings: restore proven, scheduled backups and off-cluster durability open. Establishes history/ for dated resilience snapshots.
This commit is contained in:
parent
651029a5cf
commit
96afc72b8a
2 changed files with 158 additions and 0 deletions
151
history/2026-07-04-forgejo-backup-strategy-assessment.md
Normal file
151
history/2026-07-04-forgejo-backup-strategy-assessment.md
Normal file
|
|
@ -0,0 +1,151 @@
|
|||
# Forgejo / Railiance Backup Strategy Assessment
|
||||
|
||||
Date: 2026-07-04
|
||||
Author: grok (session on `the-custodian`)
|
||||
`no_secret_material_recorded: true`
|
||||
Status: **assessment recorded — pickup from disaster-control**
|
||||
|
||||
## Context
|
||||
|
||||
Assessment performed after **RAIL-HO-WP-0005-T09** isolated restore drill (2026-07-04).
|
||||
Restore path is proven; scheduled backups and off-cluster durability are not.
|
||||
|
||||
Related workplans and evidence:
|
||||
|
||||
| Artifact | Location |
|
||||
| --- | --- |
|
||||
| Forgejo migration umbrella | `railiance-infra/workplans/RAIL-HO-WP-0005-forgejo-production-migration.md` |
|
||||
| Restore drill evidence | `railiance-infra/docs/forgejo-restore-drill-evidence.md` |
|
||||
| Restore orchestration | `railiance-infra/tools/forgejo-restore-drill.sh` |
|
||||
| Open production decisions | `the-custodian/docs/forgejo-production-decisions.md` (decision #7 TBD) |
|
||||
| App/platform handoff | `railiance-apps/docs/app-data-backup-restore-handoff.md` |
|
||||
|
||||
## Executive summary
|
||||
|
||||
Backup posture is **layered on paper, thin in practice** for Forgejo and most
|
||||
platform data. We can recover Forgejo if an operator manually runs `forgejo dump`
|
||||
and knows the restore script. We do **not** yet have a production-grade backup
|
||||
strategy: scheduled jobs, encrypted off-cluster storage, retention policy, and
|
||||
Gitea parity are all open before tier-3 (`state-hub`) or production cutover.
|
||||
|
||||
**Honest grade:** restore procedure **B+**; backup automation **D**;
|
||||
off-cluster durability **F**; Gitea (canonical source forge) **at risk**.
|
||||
|
||||
## Layer inventory
|
||||
|
||||
| Layer | What it should cover | Current state |
|
||||
| --- | --- | --- |
|
||||
| **Restore drill** | Prove Forgejo recovery | **Done** (2026-07-04) — manual `forgejo dump` restored in `forgejo-restore-drill`; health 200; `glas-harness`, `key-cape`, 3 org repos verified |
|
||||
| **Forgejo production** | DB + git repos + packages + attachments | **Manual only** — no schedule, no off-cluster target |
|
||||
| **CNPG (`forgejo-db`, `apps-pg`)** | DB base backups + WAL | **Not configured** — no `Backup` / `ScheduledBackup` CRs on railiance01 |
|
||||
| **Cluster runtime (S2)** | k3s state, Helm values, kubeconfig | **Works locally** — `railiance-cluster/tools/cmd/railiance-backup-s2` → age-encrypted files in `/opt/backup/railiance/cluster/`, keep 7; no network upload |
|
||||
| **Platform (S3)** | Legacy PostgreSQL HA, Valkey → Nextcloud | **Documented, not wired** — `railiance-platform/Makefile` `backup` target references `tools/cmd/railiance-backup`, but that tool is absent from the repo (relocation planned/cancelled) |
|
||||
| **Gitea (coulombcore)** | Canonical source forge today | **Weak** — bundled PG + `gitea-shared-storage` PVC; prior audit found no Gitea backup CronJobs; package blob backup not evidenced |
|
||||
| **Adjacent SSO/MFA** | privacyidea, authelia, lldap | **Scheduled** — daily backup CronJobs exist on railiance01 |
|
||||
|
||||
Live cluster check (2026-07-04): `kubectl get backup,scheduledbackup -A` → no
|
||||
resources. Backup-related CronJobs: `privacyidea-backup`, `authelia-backup`,
|
||||
`lldap-backup` only.
|
||||
|
||||
## What T09 proved (restore, not backup)
|
||||
|
||||
The drill validated **recovery mechanics**, not operational backup coverage:
|
||||
|
||||
1. `forgejo dump` from production pod (~11.7 MiB: `repos/`, `data/`,
|
||||
`forgejo-db.sql`, `app.ini`)
|
||||
2. Manual unzip + file copy + `psql` import (Forgejo 11.0.3 has no
|
||||
`forgejo restore` CLI)
|
||||
3. Isolated Helm deploy in `forgejo-restore-drill`
|
||||
4. API checks: health 200; `coulomb/glas-harness` and `coulomb/key-cape` on
|
||||
`main`; 3 org repos visible
|
||||
|
||||
| Metric | Observed |
|
||||
| --- | --- |
|
||||
| RPO (operational today) | Point-in-time of last manual `forgejo dump` |
|
||||
| RTO (isolated restore) | ~3–5 minutes (CNPG ready + import + Helm on railiance01) |
|
||||
| Production impact during drill | None — read-only dump; separate namespace |
|
||||
|
||||
Re-run hygiene: `DRILL_CLEAN=1 ./tools/forgejo-restore-drill.sh` (SQL import is
|
||||
not idempotent).
|
||||
|
||||
## Gaps blocking production cutover
|
||||
|
||||
1. **No scheduled Forgejo backups** — no CronJob/operator for `forgejo dump`
|
||||
2. **No CNPG-native backups** for `forgejo-db` — cluster manifest has storage
|
||||
only (`railiance-platform/helm/forgejo-db-cluster.yaml`)
|
||||
3. **No approved off-cluster target** — drill archive on workstation `/tmp`
|
||||
only; encryption/retention undecided (T02 decision #7)
|
||||
4. **Platform backup tool missing** — S3 `make backup` target without
|
||||
implementation
|
||||
5. **Gitea canonical data** — safety contract requires fresh Gitea backup before
|
||||
each migration wave; no automated Gitea backup on coulombcore
|
||||
6. **Split-brain restore risk** — restoring only CNPG DB would miss git blobs on
|
||||
the Forgejo app PVC; full `forgejo dump` is the coherent unit today
|
||||
|
||||
## Target architecture (intended, not implemented)
|
||||
|
||||
Per `RAIL-HO-WP-0005` repo boundaries:
|
||||
|
||||
```text
|
||||
S2 railiance-cluster
|
||||
etcd/state, helm values, kubeconfig
|
||||
local age-encrypted, 7-day retention
|
||||
|
||||
S3 railiance-platform
|
||||
CNPG ScheduledBackup → off-cluster target (MinIO/S3/Nextcloud — TBD)
|
||||
implement or restore railiance-backup tool
|
||||
|
||||
S5 railiance-apps
|
||||
scheduled forgejo dump (repos + data + SQL + app.ini)
|
||||
include OCI package blobs; age-encrypt; upload to approved target
|
||||
|
||||
Evidence loop (this repo)
|
||||
quarterly restore drill → isolated namespace
|
||||
dated evidence under history/ or TestEvidence/
|
||||
```
|
||||
|
||||
Ownership: `railiance-platform` owns backup mechanism; `railiance-apps` owns
|
||||
readiness gates and restore evidence per
|
||||
`docs/app-data-backup-restore-handoff.md`.
|
||||
|
||||
## Pickup queue (for disaster-control)
|
||||
|
||||
Track and drive closure of these items; implementation spans multiple repos.
|
||||
|
||||
| # | Item | Owner repo | Blocker |
|
||||
| --- | --- | --- | --- |
|
||||
| 1 | Operator decision: backup destination + retention | `railiance-infra` / T02 | Human — decision #7 in `forgejo-production-decisions.md` |
|
||||
| 2 | CNPG `ScheduledBackup` on `forgejo-db` | `railiance-platform` | #1 + backup target wiring |
|
||||
| 3 | Scheduled `forgejo dump` → age-encrypt → upload | `railiance-apps` | #1 |
|
||||
| 4 | Implement or restore `tools/cmd/railiance-backup` | `railiance-platform` | Relocation never completed |
|
||||
| 5 | Gitea pre-cutover snapshot runbook + evidence | `railiance-infra` / `railiance-apps` | No automated Gitea backup today |
|
||||
| 6 | `apps-pg` backup coverage | `railiance-platform` | Documented follow-up in `docs/apps-pg.md` |
|
||||
| 7 | Quarterly restore drill cadence + evidence | `disaster-control` + `railiance-infra` | Calendar + RTO/RPO matrix not written |
|
||||
| 8 | Populate `RTO-RPO-Matrix.md`, `BackupPolicy.md`, registers | `disaster-control` | Per `INTENT.md` / `specs/ResilienceControlSetup.md` |
|
||||
|
||||
**Gate:** tier-3 production repos (`state-hub`, etc.) remain blocked until T09
|
||||
scheduled automation and T02 backup target are closed (`RAIL-HO-WP-0005`,
|
||||
`CUST-WP-0054`).
|
||||
|
||||
## Suggested RTO/RPO targets (draft — not adopted)
|
||||
|
||||
Aligns with `specs/ResilienceControlSetup.md` tier **T1** (irreplaceable source
|
||||
of truth) until operator confirms:
|
||||
|
||||
| System | Draft RPO | Draft RTO | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| Forgejo git + packages | 24h | 4–8h | Daily `forgejo dump` + CNPG backup |
|
||||
| Gitea (canonical until cutover) | 24h | 4–8h | Mirror + dump before each migration wave |
|
||||
| `forgejo-db` CNPG | 1h–24h | 2–4h | WAL + base backup to off-cluster target |
|
||||
| k3s cluster state (S2) | 24h | 4–8h | Existing `railiance-backup-s2` local retention |
|
||||
|
||||
## References
|
||||
|
||||
- `railiance-infra/docs/forgejo-restore-drill-evidence.md`
|
||||
- `railiance-infra/tools/forgejo-restore-drill.sh`
|
||||
- `railiance-platform/helm/forgejo-db-cluster.yaml`
|
||||
- `railiance-platform/docs/apps-pg.md` (backup follow-up note)
|
||||
- `railiance-platform/docs/postgresql-ha.md` (intended platform backup)
|
||||
- `the-custodian/docs/forgejo-production-decisions.md`
|
||||
- `disaster-control/specs/ResilienceControlSetup.md`
|
||||
- `disaster-control/INTENT.md`
|
||||
7
history/README.md
Normal file
7
history/README.md
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
# History
|
||||
|
||||
Dated assessments, audits, and decision snapshots for the Resilience Control
|
||||
Plane. Active runbooks and policies live at repo root when written; `history/`
|
||||
preserves point-in-time findings that inform later work.
|
||||
|
||||
Naming: `YYYY-MM-DD-<slug>.md`
|
||||
Loading…
Add table
Add a link
Reference in a new issue