tegwick
040ccccc8e
Update workplan with Jul 9 cron success (4/7 gate, ETA Jul 12), add Gitea retirement dependency scan and operator checklist, and align deploy-stack, restore-drill evidence, and migration inventory with post-cutover state.
130 lines
No EOL
4.9 KiB
Markdown
130 lines
No EOL
4.9 KiB
Markdown
# Forgejo Backup/Restore Drill Evidence
|
||
|
||
Date: 2026-07-04
|
||
Workplan: RAIL-HO-WP-0005
|
||
Task: RAIL-HO-WP-0005-T09
|
||
`no_secret_material_recorded: true`
|
||
|
||
## Purpose
|
||
|
||
Prove that a production `forgejo dump` can be restored into an isolated
|
||
namespace and serve repository metadata without touching production Forgejo or
|
||
Gitea.
|
||
|
||
## Backup source
|
||
|
||
| Field | Value |
|
||
| --- | --- |
|
||
| Method | `forgejo dump` from production pod |
|
||
| Production pod | `forgejo-gitea-64c5b57684-ph9vt` (namespace `forgejo`) |
|
||
| Archive path (workstation) | `/tmp/forgejo-drill/forgejo-drill-backup.zip` |
|
||
| Archive size | 12,284,847 bytes (~11.7 MiB) |
|
||
| Archive timestamp | 2026-07-04 11:20 +0200 |
|
||
| Archive contents (top-level) | `repos/`, `data/`, `forgejo-db.sql`, `app.ini` |
|
||
|
||
Repos present in dump: `forgejo-actions-probe`, `glas-harness`, `key-cape`
|
||
(all under `repos/coulomb/`).
|
||
|
||
## Restore target
|
||
|
||
| Field | Value |
|
||
| --- | --- |
|
||
| Namespace | `forgejo-restore-drill` |
|
||
| Database | CNPG cluster `forgejo-db-restore` (isolated, 1 instance) |
|
||
| App data PVC | `forgejo-restore-data` (`local-path`, 10Gi) |
|
||
| Helm release | `forgejo-restore` (`gitea-charts/gitea` 12.5.0) |
|
||
| Orchestration | `tools/forgejo-restore-drill.sh` |
|
||
|
||
Restore path (Forgejo 11.0.3 has no `forgejo restore` CLI):
|
||
|
||
1. Unzip dump into import pod staging area.
|
||
2. Copy `repos/` → `/data/git/gitea-repositories/`.
|
||
3. Copy `data/` → `/data/` (packages, attachments, avatars).
|
||
4. Import `forgejo-db.sql` via `psql` into `forgejo-db-restore`.
|
||
5. Deploy isolated Helm release bound to restored PVC + restore DB host.
|
||
|
||
## Post-restore checks (2026-07-04)
|
||
|
||
Port-forward: `svc/forgejo-restore-gitea-http` → `127.0.0.1:13000`
|
||
|
||
| Check | Result |
|
||
| --- | --- |
|
||
| `GET /` health | HTTP 200 |
|
||
| `GET /api/v1/repos/coulomb/glas-harness` | `full_name=coulomb/glas-harness`, `default_branch=main` |
|
||
| `GET /api/v1/repos/coulomb/key-cape` | `full_name=coulomb/key-cape`, `default_branch=main` |
|
||
| `GET /api/v1/orgs/coulomb/repos` | 3 repos: `forgejo-actions-probe`, `glas-harness`, `key-cape` |
|
||
|
||
Script exit marker: `restore-drill-complete`
|
||
|
||
## RPO / RTO (drill scope)
|
||
|
||
| Metric | Observed / assumed |
|
||
| --- | --- |
|
||
| RPO (manual dump) | Point-in-time of `forgejo dump` execution; no scheduled backup yet |
|
||
| RTO (isolated restore) | ~3–5 minutes for CNPG ready + import + Helm deploy on railiance01 |
|
||
| Production impact | None — read-only dump from running pod; separate namespace |
|
||
|
||
## Nextcloud backup restore (2026-07-08)
|
||
|
||
Drill using production backup artifact `20260707T180844Z` (Option A path).
|
||
|
||
| Field | Value |
|
||
| --- | --- |
|
||
| Offsite object | `forgejo/forgejo-dump-20260707T180844Z.zip.age` (uploaded by `make forgejo-backup`) |
|
||
| Workstation decrypt | `~/.config/age/railiance-backup.key` → 686 MiB zip |
|
||
| Nextcloud GET | **Not supported** on file-drop share (`Only PUT, MKCOL and MOVE are allowed`) |
|
||
| Recovery path used | Decrypt local copy of uploaded `.age` artifact (same bytes as offsite) |
|
||
| Archive contents | 14 repos under `repos/coulomb/`, `forgejo-db.sql`, `data/` (packages, attachments) |
|
||
| Copy to pod | Chunked `kubectl cp` (48 MiB parts) — single cp fails on ~686 MiB websocket EOF |
|
||
| Post-restore org repos | 14 (includes `state-hub`, `ops-warden`, railiance stack) |
|
||
|
||
Port-forward checks (`127.0.0.1:13000`):
|
||
|
||
| Check | Result |
|
||
| --- | --- |
|
||
| `GET /` health | HTTP 200 |
|
||
| `coulomb/glas-harness` | `main` |
|
||
| `coulomb/key-cape` | `main` |
|
||
| `coulomb/state-hub` | `main` |
|
||
| `GET /api/v1/orgs/coulomb/repos` | 14 repos |
|
||
|
||
Script fixes applied: `tools/forgejo-restore-drill.sh` — chunked copy, mailer stub
|
||
secret, `mailer.ENABLED=false`, Helm `--timeout=20m`.
|
||
|
||
Exit marker: `restore-drill-complete`
|
||
|
||
## Gaps (remaining)
|
||
|
||
- **Scheduled backups:** `railiance-platform` `make forgejo-backup` — daily cron
|
||
02:15 UTC; 7-day gate **4/7** as of 2026-07-09 (ETA 2026-07-12). CNPG `Backup`
|
||
CRs deferred (logical `pg_dump` via `forgejo-backup` tool).
|
||
- **Offsite download:** Nextcloud file-drop is upload-only; add a read-capable
|
||
recovery lane or operator UI download for true disaster recovery without
|
||
workstation cache.
|
||
- **Re-run hygiene:** concurrent or repeat runs require `DRILL_CLEAN=1` to wipe
|
||
`forgejo-restore-drill` before import (SQL import is not idempotent).
|
||
|
||
## Closed since initial drill (2026-07-04)
|
||
|
||
- **Encryption + offsite target:** age-encrypted artifacts uploaded to Nextcloud
|
||
`forgejo/` (Option A, T02).
|
||
- **Automation tooling:** `railiance-platform/tools/cmd/forgejo-backup` +
|
||
`docs/forgejo-backup.md`.
|
||
- **Nextcloud restore drill:** passed 2026-07-08 using artifact `20260707T180844Z`.
|
||
|
||
## Cleanup
|
||
|
||
After evidence capture, delete the drill namespace:
|
||
|
||
```bash
|
||
kubectl delete namespace forgejo-restore-drill --wait=true
|
||
```
|
||
|
||
Production Forgejo (`forgejo` namespace) and Gitea remain unchanged.
|
||
|
||
## References
|
||
|
||
- `infra/forgejo-restore-drill/forgejo-db-restore-cluster.yaml`
|
||
- `infra/forgejo-restore-drill/restore-job.yaml`
|
||
- `tools/forgejo-restore-drill.sh`
|
||
- `workplans/RAIL-HO-WP-0005-forgejo-production-migration.md` (T09) |