coulomb/disaster-control
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
disaster-control/TestEvidence/restore-drill-2026-07-04-forgejo.md
tegwick b59b9a565c Complete DISCTL-WP-0003-T03 through T05 
Add RTO-RPO matrix, backup pickup queue tracker, and TestEvidence
with Forgejo drill import. Mark T03, T04, T05 done.
2026-07-07 19:58:21 +02:00

3.5 KiB
Raw Permalink Blame History

Restore Drill: Forgejo production

Date: 2026-07-04
System: backup-forgejo-production (Forgejo git + packages + forgejo-db)
Operator: Bernd (via grok session, railiance-infra)
Result: success
Workplan: RAIL-HO-WP-0005-T09

no_secret_material_recorded: true

Summary

Proved that a production forgejo dump can be restored into an isolated namespace and serve repository metadata without touching production Forgejo or Gitea. This validates recovery mechanics — not operational backup coverage.

Backup artifact

Field Value
Method forgejo dump from production pod (manual)
Production pod forgejo-gitea-64c5b57684-ph9vt (namespace forgejo)
Archive path /tmp/forgejo-drill/forgejo-drill-backup.zip (workstation)
Archive size 12,284,847 bytes (~11.7 MiB)
Archive timestamp 2026-07-04 11:20 +0200
Contents repos/, data/, forgejo-db.sql, app.ini
Repos in dump forgejo-actions-probe, glas-harness, key-cape (coulomb/)
Automated / offsite No — workstation /tmp only; not from Nextcloud

Restore target

Field Value
Namespace forgejo-restore-drill
Database CNPG forgejo-db-restore (isolated, 1 instance)
App data PVC forgejo-restore-data (local-path, 10Gi)
Helm release forgejo-restore (gitea-charts/gitea 12.5.0)
Orchestration railiance-infra/tools/forgejo-restore-drill.sh

Restore steps (Forgejo 11.0.3 — no forgejo restore CLI):

  1. Unzip dump into import pod staging area.
  2. Copy repos/ → /data/git/gitea-repositories/.
  3. Copy data/ → /data/ (packages, attachments, avatars).
  4. Import forgejo-db.sql via psql into forgejo-db-restore.
  5. Deploy isolated Helm release bound to restored PVC + restore DB host.

Validation checks

Port-forward: svc/forgejo-restore-gitea-http → 127.0.0.1:13000

Check Result
GET / health HTTP 200
GET /api/v1/repos/coulomb/glas-harness full_name=coulomb/glas-harness, default_branch=main
GET /api/v1/repos/coulomb/key-cape full_name=coulomb/key-cape, default_branch=main
GET /api/v1/orgs/coulomb/repos 3 repos visible

Script exit marker: restore-drill-complete

RPO / RTO measured

Metric Target (adopted) Achieved
RPO 24h Point-in-time of manual dump — no scheduled backup
RTO 4h ~3–5 min (isolated namespace on railiance01)
Production impact None Read-only dump; separate namespace

Posture: RTO target met. RPO target not operational until daily automated backups run (see BackupPickupQueue.md items 3–4).

Issues found

  • No scheduled backups; CNPG Backup CRs not configured.
  • Dump stored on workstation only — no encryption-at-rest on approved offsite target.
  • SQL import not idempotent — re-runs need DRILL_CLEAN=1.

Follow-up actions

Action Owner Queue / workplan
Wire daily forgejo dump + Nextcloud upload railiance-platform Pickup #3
Restore platform railiance-backup tool railiance-platform Pickup #4
Re-drill from Nextcloud artifact railiance-infra Promotion gate (BackupPolicy.md §7)
7 consecutive daily backup successes railiance-platform Promotion gate

Canonical evidence

Full drill record (owner repo):

railiance-infra/docs/forgejo-restore-drill-evidence.md

Supporting artifacts:

  • railiance-infra/tools/forgejo-restore-drill.sh
  • railiance-infra/infra/forgejo-restore-drill/