coulomb/railiance-platform
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
railiance-platform/docs/forgejo-backup.md
codex 195a51a62c
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
Details
CI Smoke / container-smoke (push) Successful in 1s
Details
Record rejected historical backup, verified fresh capture and worker activation 
Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a06ecb-456a-71c2-b41e-0755d336e883
2026-09-05 22:35:20 +02:00

4.3 KiB
Raw Blame History

Forgejo backup (railiance01)

Workplan: RAIL-HO-WP-0005 T04/T09 · Decision: Option A (Nextcloud + age)

What is backed up

Artifact Source Format
Blob state forgejo dump in production pod zip → age
PostgreSQL pg_dump --no-owner --no-acl from CNPG forgejo-db SQL → age

Covers git repos, packages (OCI/npm/generic), attachments, LFS, avatars, and DB metadata.

Operator commands

cd ~/railiance-platform
make forgejo-backup-dry-run    # local encrypt only, no upload
make forgejo-backup            # encrypt + upload to Nextcloud forgejo/
make forgejo-backup-status     # last success + 7-day gate hint

Requires: kubectl, age, Python 3, and curl or the Python upload backend. Set KUBECONFIG to the currently verified railiance01 access path.

Preflight: forgejo-db pod must be Ready (make forgejo-db-status). The validated 2026-09-05 full archive is 5,351,717,533 bytes before age encryption. Allow for substantial transfer time and check the account quota before upload. This size supersedes the historical ~670 MiB estimate.

Uploads require the governed CCR-2026-0004 OpenBao offsite lane or explicitly provided RAILIANCE_BACKUP_NC_TOKEN / RAILIANCE_BACKUP_NC_WEBDAV_URL inputs. There is no built-in credential fallback. Missing credentials stop execution before any cluster dump. Local encryption dry-runs skip upload authentication. Provider rotation and replacement upload/restore proof are tracked separately in RPF-WP-0029; removing the source default does not prove revocation. The concrete owner procedure is docs/backup-credential-recovery.md. Upload credentials and credential-bearing URLs are passed to curl through stdin, and backend errors never print those values. Redirects are refused.

Decrypt: ~/.config/age/railiance-backup.key (same key as other Railiance backups).

Nextcloud layout

Since 2026-09-05, uploads go to user Backup, folder /railiance-backups, through a create-only share delivered from OpenBao. The account has 10 GiB total capacity; the retention target below is subject to available space. Operator credentials stay in operators/nextcloud/backup; workload jobs receive only the derived upload grant. See account and recovery procedure.

forgejo/forgejo-dump-<timestamp>.zip.age
forgejo/forgejo-db-<timestamp>.sql.age
forgejo/forgejo-dump-weekly-<timestamp>.zip.age   # Sundays only
forgejo/forgejo-db-weekly-<timestamp>.sql.age

Retention target: 14 daily + 4 weekly on Nextcloud (operator may prune old objects in the WebDAV folder; local cache keeps 7 per type).

Cron (workstation)

# Daily 02:15 UTC — Forgejo backup (RPO 24h)
15 2 * * * cd $HOME/railiance-platform && make forgejo-backup >>$HOME/.cache/railiance/backups/forgejo/cron.log 2>&1

Promotion gate (tier-3 cutover)

Do not promote further production repos until:

  1. make forgejo-backup succeeds 7 consecutive days (check success-log).
  2. One restore drill uses a Nextcloud artifact (not /tmp/forgejo-drill/).

Restore

See railiance-infra:

  • tools/forgejo-restore-drill.sh
  • docs/forgejo-restore-drill-evidence.md

Download and decrypt a dump from Nextcloud, set BACKUP_LOCAL to the zip path, then run the drill script.

Archive-integrity incident — 2026-09-05

The September 4 encrypted cache sample decrypted but failed ZIP validation. The backup command now waits for the exact dump process to succeed, checks transfer size/hash and validates ZIP CRCs before encryption/upload. A fresh 5.35 GB archive passed these checks. This does not certify the other historical backups or establish a completed offsite application restore. See WP-0029 and its dated evidence; never report a nonempty file as a successful backup.

Activity-core currently receives the validated entry point through the immutable backup-verified-0220ca56520c bundle, applied by scripts/deploy_verified_backup_bundle.py. The wrapper mounts over the existing backup entry point; helpers/libraries live at /opt/railiance-backup-verified. Reapply the source-owned delivery helper after reviewed backup changes and verify its hash/rollout receipt. Roll back through the previous worker ReplicaSet; do not edit another machine's host checkout or remove referenced ConfigMaps.