From 8b5706f6930dc84c35196871930cf940af81f4cb Mon Sep 17 00:00:00 2001 From: tegwick Date: Tue, 7 Jul 2026 19:57:03 +0200 Subject: [PATCH] Complete DISCTL-WP-0003-T02: adopt BackupPolicy.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Codify railiance-backup practices — 3-2-1 model, encryption, retention, Forgejo Option A (2026-07-09), credential separation, and monitoring. Mark T02 done. --- BackupPolicy.md | 286 ++++++++++++++++++ ...CTL-WP-0003-railiance-backup-governance.md | 2 +- 2 files changed, 287 insertions(+), 1 deletion(-) create mode 100644 BackupPolicy.md diff --git a/BackupPolicy.md b/BackupPolicy.md new file mode 100644 index 0000000..80ee1e1 --- /dev/null +++ b/BackupPolicy.md @@ -0,0 +1,286 @@ +# Backup Policy + +**Status:** adopted (initial) +**Updated:** 2026-07-07 +**Owner:** Bernd +**Scope:** Coulomb Social / Helix Forge — Railiance backup lanes +**Register:** `AssetRegister.yaml` (stream inventory) · `DataRegister.yaml` (data classes) + +This policy translates live `railiance-backup` practices into enforceable rules +with verification steps. It governs *what* must be backed up and *how*; backup +scripts run in owner repos (`railiance-cluster`, `railiance-infra`, +`railiance-platform`, `railiance-apps`). + +--- + +## 1. Operating principle + +> Every T1 data class must have a named backup stream, encryption, retention, +> offsite copy (where required), monitoring, and restore evidence. + +Servers are disposable; backups are not. Rebuild + restore is the default +recovery pattern — not server resurrection. + +Honest status reporting is required: streams marked `missing`, `not_wired`, or +`at_risk` in `AssetRegister.yaml` are policy violations until closed. + +--- + +## 2. Architecture — 3-2-1 + immutable/offline + +Target model (per `specs/ResilienceControlSetup.md` §4): + +| Copy | Role | Implementation today | +| --- | --- | --- | +| **Production** | Live data | Production hosts, PVCs, databases | +| **Local fast rollback** | Same-site recovery | `/opt/backup/railiance/{infra,cluster}/`; `~/.cache/railiance/backups/` | +| **Offsite encrypted** | Survives host loss | Nextcloud WebDAV file drop (upload-only) | +| **Immutable/offline** | Survives hostile delete | Phase 1: upload-only drop token (no delete from production); weekly offline copy — **not yet implemented** | + +**3-2-1 compliance by stream:** + +| Stream | 3 copies | 2 media | 1 offsite | Notes | +| --- | --- | --- | --- | --- | +| Dev workstation | ✓ (prod + local cache + Nextcloud) | ✓ (disk + cloud) | ✓ | Operational | +| S1 OS config | ✗ (prod + local only) | ✗ (disk only) | ✗ | **Gap — needs offsite upload** | +| S2 k8s runtime | ✗ (prod + local only) | ✗ (disk only) | ✗ | **Gap — needs offsite upload** | +| S3 platform | — | — | — | Tool missing | +| Forgejo production | ✗ (prod + drill /tmp only) | partial | intended | Automation not wired | +| Gitea canonical | ✗ | ✗ | ✗ | Manual pre-cutover only | + +--- + +## 3. Encryption standard + +All backup artifacts **must** be encrypted with [age](https://age-encryption.org/) +before leaving the host or entering offsite storage. + +### Key custody + +| Key | Operational location | Recovery copy | Used by | +| --- | --- | --- | --- | +| Dev-lane age key | `~/.config/age/railiance-backup.key` | Password manager | `railiance-backup` (dev workstation) | +| S1/S2 age key | `~/.config/sops/age/keys.txt` | Password manager + SOPS recovery | `railiance-backup-s1`, `railiance-backup-s2` | +| Platform/Forgejo lane | Platform backup public key (TBD in OpenBao) | Password manager | Intended S3 + Forgejo dumps | + +**Rules:** + +1. Operational keys: `chmod 700` on `~/.config/age`; `chmod 600` on key files. +2. Password-manager recovery copy is mandatory — it is the only key that survives + hardware failure. +3. Never store private keys in Git, workplans, State Hub, or chat. +4. Offsite credentials live in OpenBao (`platform/workloads/railiance/backup/offsite-lane`); + not on production hosts. + +### Verification + +| Check | Frequency | Pass condition | Owner | +| --- | --- | --- | --- | +| Recovery key in password manager | Monthly | Operator confirms copy exists and decrypts a test artifact | Bernd | +| Operational key permissions | Monthly | `stat` shows 600 on key file, 700 on parent dir | Bernd | +| Cold decrypt drill | Quarterly | Decrypt one artifact from each active lane without using production host | Bernd | + +--- + +## 4. Destination policy + +### Off-node — Nextcloud WebDAV file drop + +Used by: dev workstation lane (operational); Forgejo + platform lane (adopted, not fully wired). + +- **Endpoint:** filesdrop URL — `https:///public.php/dav/filesdrop//` +- **Not valid:** `/public.php/webdav/` (returns HTTP 409 on current Nextcloud) +- **Upload method:** `curl -T` with `-u :` — upload-only; no rclone listing +- **Properties:** upload-only token cannot read or delete existing artifacts from + the production host — supports credential separation (R3) +- **Human link:** `260225-backup-dropoff-link.txt` (gitignored); credentials in OpenBao + +### On-node — local encrypted cache + +| Path | Lane | Retention | +| --- | --- | --- | +| `~/.cache/railiance/backups/` | Dev workstation | 7 per type | +| `/opt/backup/railiance/infra/` | S1 OS config | 7 per type | +| `/opt/backup/railiance/cluster/` | S2 k8s runtime | 7 per type | + +Local caches are **fast rollback only** — not a substitute for offsite copy. + +--- + +## 5. Retention policy + +| Lane | Daily rotations | Weekly rotations | Local prune | Status | +| --- | --- | --- | --- | --- | +| Dev workstation | N/A (continuous daily) | N/A | 7 per type | operational | +| S1 / S2 | N/A (manual) | N/A | 7 per type | operational | +| Forgejo + platform (adopted 2026-07-09) | 14 | 4 (Sunday promote or explicit job) | 7 per type | not wired | +| Gitea pre-cutover | Point-in-time of last manual dump | — | — | at risk | + +Prune logic: `find … \| sort -r \| tail -n +8 \| xargs rm` (keep last 7) in +`railiance-backup`, `railiance-backup-s1`, `railiance-backup-s2`. + +--- + +## 6. Per-stream mechanisms + +Canonical inventory: `AssetRegister.yaml`. Summary: + +| Register ID | Tool | Schedule | Destination | Status | +| --- | --- | --- | --- | --- | +| `backup-dev-workstation` | `railiance-backup` | Daily 02:00 cron | Nextcloud + local cache | operational | +| `backup-s1-os-config` | `railiance-backup-s1` | Manual `make backup` | `/opt/backup/railiance/infra/` | operational | +| `backup-s2-k8s-runtime` | `railiance-backup-s2` | Manual `make backup` | `/opt/backup/railiance/cluster/` | operational | +| `backup-s3-platform` | `railiance-backup` | TBD | Nextcloud (intended) | **missing** | +| `backup-forgejo-production` | `forgejo dump` + `pg_dump` | Daily (intended) | Nextcloud (adopted) | **not_wired** | +| `backup-gitea-canonical` | `gitea dump` / mirror | Pre-cutover manual | None automated | **at_risk** | + +**Related streams** (tracked, not yet full policy coverage): + +- `backup-adjacent-sso` — daily CronJobs (`privacyidea-backup`, `authelia-backup`, `lldap-backup`) on railiance01 +- `backup-apps-pg` — CNPG `apps-pg`; no ScheduledBackup CR + +**Not in git backups:** Git repos are protected by push-to-Gitea remotes. Preflight +verifies clean and pushed state before infrastructure work. + +--- + +## 7. Adopted standard — Forgejo backup (Option A, 2026-07-09) + +**Decision:** Extend the existing Railiance platform backup lane — age-encrypted +artifacts uploaded to Nextcloud WebDAV. + +| Component | Method | Schedule | +| --- | --- | --- | +| Forgejo blob state | `forgejo dump` zip (repos, OCI/npm/generic packages, attachments, LFS, avatars) | Daily | +| PostgreSQL | `pg_dump` from CNPG `forgejo-db` (logical; no WAL/PITR in Phase 1) | Daily | +| Encryption | age (platform backup public key) | Per artifact | +| Destination | Nextcloud WebDAV file drop | Upload after each run | +| Restore proof | `forgejo-restore-drill.sh` from automated backup | Quarterly | + +**RPO / RTO:** 24h / 4h. + +**Implementation owners:** + +- `railiance-platform` — cron, age encrypt, Nextcloud upload, retention prune +- `railiance-infra` — restore runbook + quarterly drill evidence +- `railiance-apps` — no backup secrets in Git; dump runs against live pod + +**Phase 1 exclusions:** CNPG WAL archiving to S3/MinIO; on-node-only backups +without Nextcloud upload. + +**Promotion gate:** No further tier-3 repo cutovers until automated daily backups +succeed for **7 consecutive days** and one restore drill uses a Nextcloud artifact +(not workstation `/tmp`). + +**Current state:** Decision adopted; automation not implemented. Restore path +proven manually (RAIL-HO-WP-0005-T09, 2026-07-04). + +Reference: `the-custodian/docs/forgejo-production-decisions.md` + +--- + +## 8. Credential separation (R3) + +**Rule:** Backup-storage credentials must **not** live on the production host with +permission to delete or overwrite offsite backups. + +### How this is enforced today + +| Control | Mechanism | +| --- | --- | +| Upload-only file drop | Nextcloud public share token permits PUT only — production host cannot list or delete remote artifacts | +| Separate key material | age private keys stored separately from backup upload tokens | +| OpenBao custody | Offsite lane credentials in `platform/workloads/railiance/backup/offsite-lane` — not in Git | +| No shared admin | Production k3s/Gitea admin credentials ≠ Nextcloud admin credentials | + +### Verification steps + +| Step | Frequency | Procedure | Owner | +| --- | --- | --- | --- | +| CS-1 Token scope | Quarterly | Confirm production backup token cannot DELETE or LIST on Nextcloud drop (curl probe) | Bernd | +| CS-2 Credential inventory | Monthly | Verify no backup-storage admin credentials in `/etc`, kube Secrets, or Ansible vars on railiance01 | Bernd | +| CS-3 Separate accounts | Quarterly | Confirm production, backup, registrar, and payment systems use distinct admin identities | Bernd | +| CS-4 Breach runbook | After adoption | Incident playbook requires: never restore into compromised machine; rotate before restore | Bernd | + +**Known exposure:** S1/S2 local backups on railiance01 are deletable by root on +that host. Mitigation: wire offsite upload for S1/S2 (tracked in pickup queue). + +--- + +## 9. Monitoring and evidence (R9) + +**Alerting owner:** Bernd (until a formal on-call rotation is defined). + +### Required checks + +| Check ID | Check | Frequency | Pass condition | Owner | +| --- | --- | --- | --- | --- | +| M-1 | Backup job succeeded | Daily | `.last-backup` stamp < 24h old for each active lane; cron log shows exit 0 | Bernd | +| M-2 | Artifact freshness | Daily | Latest `db-*.sql.age` and `config-*.tar.gz.age` < 24h (dev lane) | Bernd | +| M-3 | Repository integrity | Weekly | age-decrypt test on one random artifact per lane; file size > 0; pg_dump header valid | Bernd | +| M-4 | Preflight gate | Before infra change | `bin/railiance preflight` exits 0 | Bernd | +| M-5 | Restore drill | Quarterly per T1 stream | Dated evidence in `TestEvidence/` | Bernd | +| M-6 | Register review | Monthly | `AssetRegister.yaml` statuses updated; gaps unchanged or closed | Bernd | + +### Preflight gate (mandatory before infrastructure work) + +```bash +bin/railiance preflight +``` + +| Check | Pass condition | +| --- | --- | +| DB backup freshness | Latest `db-*.sql.age` < 24 hours old | +| Config backup freshness | Latest `config-*.tar.gz.age` < 24 hours old | +| Git repos clean | No uncommitted changes in tracked repos | +| Git repos pushed | No unpushed commits | +| age key present | `~/.config/age/railiance-backup.key` exists | + +Exit 0 = proceed. Exit 1 = **do not proceed**. + +### Silent-failure mitigations + +1. Cron jobs log to `~/.cache/railiance/backup.log` — review on M-1 failure. +2. Backup scripts exit non-zero on any step failure (`set -euo pipefail`). +3. Missing tool (`railiance-platform` `make backup`) is a **policy violation** — + tracked as `backup-s3-platform` status `missing`. +4. Forgejo lane not wired — treat as **degraded** until 7-day promotion gate met. + +### Evidence output + +Every restore drill produces a file under `TestEvidence/`: + +```markdown +# Restore Drill: +Date, backup artifact used, target environment, result, RTO measured, +RPO measured, issues, follow-up actions. +``` + +--- + +## 10. Open gaps (honest) + +| Gap | Risk | Remediation | Owner repo | +| --- | --- | --- | --- | +| S3 `railiance-backup` tool missing | Platform DB unprotected | Restore or implement `tools/cmd/railiance-backup` | `railiance-platform` | +| Forgejo automation not wired | RPO = last manual dump | Daily cron + Nextcloud upload per §7 | `railiance-platform` | +| S1/S2 no offsite upload | Lost if railiance01 destroyed | Extend platform lane upload to S1/S2 artifacts | `railiance-platform` | +| Gitea no automated backup | Canonical forge at risk | CronJob + mirror; pre-cutover dump runbook | `railiance-infra` | +| `apps-pg` no ScheduledBackup | App data unprotected | CNPG ScheduledBackup CR | `railiance-platform` | +| No immutable weekly offline copy | R3 residual risk | Add offline/append-only copy (Phase 2) | TBD | +| Cold decrypt not evidenced | R5 residual risk | Quarterly cold decrypt drill (M-3 extended) | `disaster-control` | + +Track closure in `BackupPickupQueue.md` (DISCTL-WP-0003-T04). + +--- + +## 11. References + +- `AssetRegister.yaml` — stream inventory and status +- `DataRegister.yaml` — data class mapping +- `specs/ResilienceControlSetup.md` — architecture and tier model +- `history/2026-07-04-forgejo-backup-strategy-assessment.md` — gap analysis +- `history/2026-07-04-initial-risk-analysis.md` — R3, R9 risk context +- `railiance-cluster/docs/backup-restore.md` — dev lane operational procedures +- `the-custodian/docs/forgejo-production-decisions.md` — Option A adoption +- `railiance-infra/docs/forgejo-restore-drill-evidence.md` — T09 restore proof \ No newline at end of file diff --git a/workplans/DISCTL-WP-0003-railiance-backup-governance.md b/workplans/DISCTL-WP-0003-railiance-backup-governance.md index bdde246..29975c7 100644 --- a/workplans/DISCTL-WP-0003-railiance-backup-governance.md +++ b/workplans/DISCTL-WP-0003-railiance-backup-governance.md @@ -68,7 +68,7 @@ Feeds `DISCTL-WP-0002-T02`. ```task id: DISCTL-WP-0003-T02 -status: todo +status: done priority: high state_hub_task_id: "2e32792d-4c22-4e76-adf5-44d9e81bd09a" ```