coulomb/railiance-infra
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
railiance-infra/docs/forgejo-restore-drill-evidence.md
tegwick 040ccccc8e WP-0005: T12 prep checklist, backup gate 4/7, doc refresh 
Update workplan with Jul 9 cron success (4/7 gate, ETA Jul 12), add Gitea
retirement dependency scan and operator checklist, and align deploy-stack,
restore-drill evidence, and migration inventory with post-cutover state.
2026-07-09 10:34:44 +02:00

4.9 KiB
Raw Blame History

Forgejo Backup/Restore Drill Evidence

Date: 2026-07-04
Workplan: RAIL-HO-WP-0005
Task: RAIL-HO-WP-0005-T09
no_secret_material_recorded: true

Purpose

Prove that a production forgejo dump can be restored into an isolated namespace and serve repository metadata without touching production Forgejo or Gitea.

Backup source

Field Value
Method forgejo dump from production pod
Production pod forgejo-gitea-64c5b57684-ph9vt (namespace forgejo)
Archive path (workstation) /tmp/forgejo-drill/forgejo-drill-backup.zip
Archive size 12,284,847 bytes (~11.7 MiB)
Archive timestamp 2026-07-04 11:20 +0200
Archive contents (top-level) repos/, data/, forgejo-db.sql, app.ini

Repos present in dump: forgejo-actions-probe, glas-harness, key-cape (all under repos/coulomb/).

Restore target

Field Value
Namespace forgejo-restore-drill
Database CNPG cluster forgejo-db-restore (isolated, 1 instance)
App data PVC forgejo-restore-data (local-path, 10Gi)
Helm release forgejo-restore (gitea-charts/gitea 12.5.0)
Orchestration tools/forgejo-restore-drill.sh

Restore path (Forgejo 11.0.3 has no forgejo restore CLI):

  1. Unzip dump into import pod staging area.
  2. Copy repos/ → /data/git/gitea-repositories/.
  3. Copy data/ → /data/ (packages, attachments, avatars).
  4. Import forgejo-db.sql via psql into forgejo-db-restore.
  5. Deploy isolated Helm release bound to restored PVC + restore DB host.

Post-restore checks (2026-07-04)

Port-forward: svc/forgejo-restore-gitea-http → 127.0.0.1:13000

Check Result
GET / health HTTP 200
GET /api/v1/repos/coulomb/glas-harness full_name=coulomb/glas-harness, default_branch=main
GET /api/v1/repos/coulomb/key-cape full_name=coulomb/key-cape, default_branch=main
GET /api/v1/orgs/coulomb/repos 3 repos: forgejo-actions-probe, glas-harness, key-cape

Script exit marker: restore-drill-complete

RPO / RTO (drill scope)

Metric Observed / assumed
RPO (manual dump) Point-in-time of forgejo dump execution; no scheduled backup yet
RTO (isolated restore) ~3–5 minutes for CNPG ready + import + Helm deploy on railiance01
Production impact None — read-only dump from running pod; separate namespace

Nextcloud backup restore (2026-07-08)

Drill using production backup artifact 20260707T180844Z (Option A path).

Field Value
Offsite object forgejo/forgejo-dump-20260707T180844Z.zip.age (uploaded by make forgejo-backup)
Workstation decrypt ~/.config/age/railiance-backup.key → 686 MiB zip
Nextcloud GET Not supported on file-drop share (Only PUT, MKCOL and MOVE are allowed)
Recovery path used Decrypt local copy of uploaded .age artifact (same bytes as offsite)
Archive contents 14 repos under repos/coulomb/, forgejo-db.sql, data/ (packages, attachments)
Copy to pod Chunked kubectl cp (48 MiB parts) — single cp fails on ~686 MiB websocket EOF
Post-restore org repos 14 (includes state-hub, ops-warden, railiance stack)

Port-forward checks (127.0.0.1:13000):

Check Result
GET / health HTTP 200
coulomb/glas-harness main
coulomb/key-cape main
coulomb/state-hub main
GET /api/v1/orgs/coulomb/repos 14 repos

Script fixes applied: tools/forgejo-restore-drill.sh — chunked copy, mailer stub secret, mailer.ENABLED=false, Helm --timeout=20m.

Exit marker: restore-drill-complete

Gaps (remaining)

  • Scheduled backups: railiance-platform make forgejo-backup — daily cron 02:15 UTC; 7-day gate 4/7 as of 2026-07-09 (ETA 2026-07-12). CNPG Backup CRs deferred (logical pg_dump via forgejo-backup tool).
  • Offsite download: Nextcloud file-drop is upload-only; add a read-capable recovery lane or operator UI download for true disaster recovery without workstation cache.
  • Re-run hygiene: concurrent or repeat runs require DRILL_CLEAN=1 to wipe forgejo-restore-drill before import (SQL import is not idempotent).

Closed since initial drill (2026-07-04)

  • Encryption + offsite target: age-encrypted artifacts uploaded to Nextcloud forgejo/ (Option A, T02).
  • Automation tooling: railiance-platform/tools/cmd/forgejo-backup + docs/forgejo-backup.md.
  • Nextcloud restore drill: passed 2026-07-08 using artifact 20260707T180844Z.

Cleanup

After evidence capture, delete the drill namespace:

kubectl delete namespace forgejo-restore-drill --wait=true

Production Forgejo (forgejo namespace) and Gitea remain unchanged.

References

  • infra/forgejo-restore-drill/forgejo-db-restore-cluster.yaml
  • infra/forgejo-restore-drill/restore-job.yaml
  • tools/forgejo-restore-drill.sh
  • workplans/RAIL-HO-WP-0005-forgejo-production-migration.md (T09)