coulomb/railiance-infra
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
railiance-infra/docs/forgejo-restore-drill-evidence.md
tegwick 1da9c3269b
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
Details
CI Smoke / container-smoke (push) Successful in 9s
Details
Pass Nextcloud backup restore drill for Forgejo (T09) 
Decrypt production forgejo-dump artifact, restore 14 repos in isolated
namespace, and harden restore script for large chunked copy and drill mailer stub.
2026-07-07 22:44:55 +02:00

4.7 KiB
Raw Blame History

Forgejo Backup/Restore Drill Evidence

Date: 2026-07-04
Workplan: RAIL-HO-WP-0005
Task: RAIL-HO-WP-0005-T09
no_secret_material_recorded: true

Purpose

Prove that a production forgejo dump can be restored into an isolated namespace and serve repository metadata without touching production Forgejo or Gitea.

Backup source

Field Value
Method forgejo dump from production pod
Production pod forgejo-gitea-64c5b57684-ph9vt (namespace forgejo)
Archive path (workstation) /tmp/forgejo-drill/forgejo-drill-backup.zip
Archive size 12,284,847 bytes (~11.7 MiB)
Archive timestamp 2026-07-04 11:20 +0200
Archive contents (top-level) repos/, data/, forgejo-db.sql, app.ini

Repos present in dump: forgejo-actions-probe, glas-harness, key-cape (all under repos/coulomb/).

Restore target

Field Value
Namespace forgejo-restore-drill
Database CNPG cluster forgejo-db-restore (isolated, 1 instance)
App data PVC forgejo-restore-data (local-path, 10Gi)
Helm release forgejo-restore (gitea-charts/gitea 12.5.0)
Orchestration tools/forgejo-restore-drill.sh

Restore path (Forgejo 11.0.3 has no forgejo restore CLI):

  1. Unzip dump into import pod staging area.
  2. Copy repos/ → /data/git/gitea-repositories/.
  3. Copy data/ → /data/ (packages, attachments, avatars).
  4. Import forgejo-db.sql via psql into forgejo-db-restore.
  5. Deploy isolated Helm release bound to restored PVC + restore DB host.

Post-restore checks (2026-07-04)

Port-forward: svc/forgejo-restore-gitea-http → 127.0.0.1:13000

Check Result
GET / health HTTP 200
GET /api/v1/repos/coulomb/glas-harness full_name=coulomb/glas-harness, default_branch=main
GET /api/v1/repos/coulomb/key-cape full_name=coulomb/key-cape, default_branch=main
GET /api/v1/orgs/coulomb/repos 3 repos: forgejo-actions-probe, glas-harness, key-cape

Script exit marker: restore-drill-complete

RPO / RTO (drill scope)

Metric Observed / assumed
RPO (manual dump) Point-in-time of forgejo dump execution; no scheduled backup yet
RTO (isolated restore) ~3–5 minutes for CNPG ready + import + Helm deploy on railiance01
Production impact None — read-only dump from running pod; separate namespace

Nextcloud backup restore (2026-07-08)

Drill using production backup artifact 20260707T180844Z (Option A path).

Field Value
Offsite object forgejo/forgejo-dump-20260707T180844Z.zip.age (uploaded by make forgejo-backup)
Workstation decrypt ~/.config/age/railiance-backup.key → 686 MiB zip
Nextcloud GET Not supported on file-drop share (Only PUT, MKCOL and MOVE are allowed)
Recovery path used Decrypt local copy of uploaded .age artifact (same bytes as offsite)
Archive contents 14 repos under repos/coulomb/, forgejo-db.sql, data/ (packages, attachments)
Copy to pod Chunked kubectl cp (48 MiB parts) — single cp fails on ~686 MiB websocket EOF
Post-restore org repos 14 (includes state-hub, ops-warden, railiance stack)

Port-forward checks (127.0.0.1:13000):

Check Result
GET / health HTTP 200
coulomb/glas-harness main
coulomb/key-cape main
coulomb/state-hub main
GET /api/v1/orgs/coulomb/repos 14 repos

Script fixes applied: tools/forgejo-restore-drill.sh — chunked copy, mailer stub secret, mailer.ENABLED=false, Helm --timeout=20m.

Exit marker: restore-drill-complete

Gaps (not closed by this drill)

  • Scheduled backups: CNPG Backup CRs and off-cluster target not configured (kubectl cnpg plugin absent on workstation).
  • Encryption at rest: dump stored locally on workstation for drill only; no approved backup target wired.
  • Automation: railiance-platform make forgejo-backup — daily cron 02:15 UTC; 7-day gate still counting.
  • Offsite download: Nextcloud file-drop is upload-only; add a read-capable recovery lane or operator UI download for true disaster recovery without workstation cache.
  • Re-run hygiene: concurrent or repeat runs require DRILL_CLEAN=1 to wipe forgejo-restore-drill before import (SQL import is not idempotent).

Cleanup

After evidence capture, delete the drill namespace:

kubectl delete namespace forgejo-restore-drill --wait=true

Production Forgejo (forgejo namespace) and Gitea remain unchanged.

References

  • infra/forgejo-restore-drill/forgejo-db-restore-cluster.yaml
  • infra/forgejo-restore-drill/restore-job.yaml
  • tools/forgejo-restore-drill.sh
  • workplans/RAIL-HO-WP-0005-forgejo-production-migration.md (T09)