disaster-control/BackupPolicy.md

# Backup Policy

**Status:** adopted (initial)  
**Updated:** 2026-07-07  
**Owner:** Bernd  
**Scope:** Coulomb Social / Helix Forge — Railiance backup lanes  
**Register:** `AssetRegister.yaml` (stream inventory) · `DataRegister.yaml` (data classes)

This policy translates live `railiance-backup` practices into enforceable rules
with verification steps. It governs *what* must be backed up and *how*; backup
scripts run in owner repos (`railiance-cluster`, `railiance-infra`,
`railiance-platform`, `railiance-apps`).

---

## 1. Operating principle

> Every T1 data class must have a named backup stream, encryption, retention,
> offsite copy (where required), monitoring, and restore evidence.

Servers are disposable; backups are not. Rebuild + restore is the default
recovery pattern — not server resurrection.

Honest status reporting is required: streams marked `missing`, `not_wired`, or
`at_risk` in `AssetRegister.yaml` are policy violations until closed.

---

## 2. Architecture — 3-2-1 + immutable/offline

Target model (per `specs/ResilienceControlSetup.md` §4):

| Copy | Role | Implementation today |
| --- | --- | --- |
| **Production** | Live data | Production hosts, PVCs, databases |
| **Local fast rollback** | Same-site recovery | `/opt/backup/railiance/{infra,cluster}/`; `~/.cache/railiance/backups/` |
| **Offsite encrypted** | Survives host loss | Nextcloud WebDAV file drop (upload-only) |
| **Immutable/offline** | Survives hostile delete | Phase 1: upload-only drop token (no delete from production); weekly offline copy — **not yet implemented** |

**3-2-1 compliance by stream:**

| Stream | 3 copies | 2 media | 1 offsite | Notes |
| --- | --- | --- | --- | --- |
| Dev workstation | ✓ (prod + local cache + Nextcloud) | ✓ (disk + cloud) | ✓ | Operational |
| S1 OS config | ✗ (prod + local only) | ✗ (disk only) | ✗ | **Gap — needs offsite upload** |
| S2 k8s runtime | ✗ (prod + local only) | ✗ (disk only) | ✗ | **Gap — needs offsite upload** |
| S3 platform | — | — | — | Tool missing |
| Forgejo production | ✗ (prod + drill /tmp only) | partial | intended | Automation not wired |
| Gitea canonical | ✗ | ✗ | ✗ | Manual pre-cutover only |

---

## 3. Encryption standard

All backup artifacts **must** be encrypted with [age](https://age-encryption.org/)
before leaving the host or entering offsite storage.

### Key custody

| Key | Operational location | Recovery copy | Used by |
| --- | --- | --- | --- |
| Dev-lane age key | `~/.config/age/railiance-backup.key` | Password manager | `railiance-backup` (dev workstation) |
| S1/S2 age key | `~/.config/sops/age/keys.txt` | Password manager + SOPS recovery | `railiance-backup-s1`, `railiance-backup-s2` |
| Platform/Forgejo lane | Platform backup public key (TBD in OpenBao) | Password manager | Intended S3 + Forgejo dumps |

**Rules:**

1. Operational keys: `chmod 700` on `~/.config/age`; `chmod 600` on key files.
2. Password-manager recovery copy is mandatory — it is the only key that survives
   hardware failure.
3. Never store private keys in Git, workplans, State Hub, or chat.
4. Offsite credentials live in OpenBao (`platform/workloads/railiance/backup/offsite-lane`);
   not on production hosts.

### Verification

| Check | Frequency | Pass condition | Owner |
| --- | --- | --- | --- |
| Recovery key in password manager | Monthly | Operator confirms copy exists and decrypts a test artifact | Bernd |
| Operational key permissions | Monthly | `stat` shows 600 on key file, 700 on parent dir | Bernd |
| Cold decrypt drill | Quarterly | Decrypt one artifact from each active lane without using production host | Bernd |

---

## 4. Destination policy

### Off-node — Nextcloud WebDAV file drop

Used by: dev workstation lane (operational); Forgejo + platform lane (adopted, not fully wired).

- **Endpoint:** filesdrop URL — `https://<host>/public.php/dav/filesdrop/<token>/<filename>`
- **Not valid:** `/public.php/webdav/` (returns HTTP 409 on current Nextcloud)
- **Upload method:** `curl -T` with `-u <token>:` — upload-only; no rclone listing
- **Properties:** upload-only token cannot read or delete existing artifacts from
  the production host — supports credential separation (R3)
- **Human link:** `260225-backup-dropoff-link.txt` (gitignored); credentials in OpenBao

### On-node — local encrypted cache

| Path | Lane | Retention |
| --- | --- | --- |
| `~/.cache/railiance/backups/` | Dev workstation | 7 per type |
| `/opt/backup/railiance/infra/` | S1 OS config | 7 per type |
| `/opt/backup/railiance/cluster/` | S2 k8s runtime | 7 per type |

Local caches are **fast rollback only** — not a substitute for offsite copy.

---

## 5. Retention policy

| Lane | Daily rotations | Weekly rotations | Local prune | Status |
| --- | --- | --- | --- | --- |
| Dev workstation | N/A (continuous daily) | N/A | 7 per type | operational |
| S1 / S2 | N/A (manual) | N/A | 7 per type | operational |
| Forgejo + platform (adopted 2026-07-09) | 14 | 4 (Sunday promote or explicit job) | 7 per type | not wired |
| Gitea pre-cutover | Point-in-time of last manual dump | — | — | at risk |

Prune logic: `find … \| sort -r \| tail -n +8 \| xargs rm` (keep last 7) in
`railiance-backup`, `railiance-backup-s1`, `railiance-backup-s2`.

---

## 6. Per-stream mechanisms

Canonical inventory: `AssetRegister.yaml`. Summary:

| Register ID | Tool | Schedule | Destination | Status |
| --- | --- | --- | --- | --- |
| `backup-dev-workstation` | `railiance-backup` | Daily 02:00 cron | Nextcloud + local cache | operational |
| `backup-s1-os-config` | `railiance-backup-s1` | Manual `make backup` | `/opt/backup/railiance/infra/` | operational |
| `backup-s2-k8s-runtime` | `railiance-backup-s2` | Manual `make backup` | `/opt/backup/railiance/cluster/` | operational |
| `backup-s3-platform` | `railiance-backup` | TBD | Nextcloud (intended) | **missing** |
| `backup-forgejo-production` | `forgejo dump` + `pg_dump` | Daily (intended) | Nextcloud (adopted) | **not_wired** |
| `backup-gitea-canonical` | `gitea dump` / mirror | Pre-cutover manual | None automated | **at_risk** |

**Related streams** (tracked, not yet full policy coverage):

- `backup-adjacent-sso` — daily CronJobs (`privacyidea-backup`, `authelia-backup`, `lldap-backup`) on railiance01
- `backup-apps-pg` — CNPG `apps-pg`; no ScheduledBackup CR

**Not in git backups:** Git repos are protected by push-to-Gitea remotes. Preflight
verifies clean and pushed state before infrastructure work.

---

## 7. Adopted standard — Forgejo backup (Option A, 2026-07-09)

**Decision:** Extend the existing Railiance platform backup lane — age-encrypted
artifacts uploaded to Nextcloud WebDAV.

| Component | Method | Schedule |
| --- | --- | --- |
| Forgejo blob state | `forgejo dump` zip (repos, OCI/npm/generic packages, attachments, LFS, avatars) | Daily |
| PostgreSQL | `pg_dump` from CNPG `forgejo-db` (logical; no WAL/PITR in Phase 1) | Daily |
| Encryption | age (platform backup public key) | Per artifact |
| Destination | Nextcloud WebDAV file drop | Upload after each run |
| Restore proof | `forgejo-restore-drill.sh` from automated backup | Quarterly |

**RPO / RTO:** 24h / 4h.

**Implementation owners:**

- `railiance-platform` — cron, age encrypt, Nextcloud upload, retention prune
- `railiance-infra` — restore runbook + quarterly drill evidence
- `railiance-apps` — no backup secrets in Git; dump runs against live pod

**Phase 1 exclusions:** CNPG WAL archiving to S3/MinIO; on-node-only backups
without Nextcloud upload.

**Promotion gate:** No further tier-3 repo cutovers until automated daily backups
succeed for **7 consecutive days** and one restore drill uses a Nextcloud artifact
(not workstation `/tmp`).

**Current state:** Decision adopted; automation not implemented. Restore path
proven manually (RAIL-HO-WP-0005-T09, 2026-07-04).

Reference: `the-custodian/docs/forgejo-production-decisions.md`

---

## 8. Credential separation (R3)

**Rule:** Backup-storage credentials must **not** live on the production host with
permission to delete or overwrite offsite backups.

### How this is enforced today

| Control | Mechanism |
| --- | --- |
| Upload-only file drop | Nextcloud public share token permits PUT only — production host cannot list or delete remote artifacts |
| Separate key material | age private keys stored separately from backup upload tokens |
| OpenBao custody | Offsite lane credentials in `platform/workloads/railiance/backup/offsite-lane` — not in Git |
| No shared admin | Production k3s/Gitea admin credentials ≠ Nextcloud admin credentials |

### Verification steps

| Step | Frequency | Procedure | Owner |
| --- | --- | --- | --- |
| CS-1 Token scope | Quarterly | Confirm production backup token cannot DELETE or LIST on Nextcloud drop (curl probe) | Bernd |
| CS-2 Credential inventory | Monthly | Verify no backup-storage admin credentials in `/etc`, kube Secrets, or Ansible vars on railiance01 | Bernd |
| CS-3 Separate accounts | Quarterly | Confirm production, backup, registrar, and payment systems use distinct admin identities | Bernd |
| CS-4 Breach runbook | After adoption | Incident playbook requires: never restore into compromised machine; rotate before restore | Bernd |

**Known exposure:** S1/S2 local backups on railiance01 are deletable by root on
that host. Mitigation: wire offsite upload for S1/S2 (tracked in pickup queue).

---

## 9. Monitoring and evidence (R9)

**Alerting owner:** Bernd (until a formal on-call rotation is defined).

### Required checks

| Check ID | Check | Frequency | Pass condition | Owner |
| --- | --- | --- | --- | --- |
| M-1 | Backup job succeeded | Daily | `.last-backup` stamp < 24h old for each active lane; cron log shows exit 0 | Bernd |
| M-2 | Artifact freshness | Daily | Latest `db-*.sql.age` and `config-*.tar.gz.age` < 24h (dev lane) | Bernd |
| M-3 | Repository integrity | Weekly | age-decrypt test on one random artifact per lane; file size > 0; pg_dump header valid | Bernd |
| M-4 | Preflight gate | Before infra change | `bin/railiance preflight` exits 0 | Bernd |
| M-5 | Restore drill | Quarterly per T1 stream | Dated evidence in `TestEvidence/` | Bernd |
| M-6 | Register review | Monthly | `AssetRegister.yaml` statuses updated; gaps unchanged or closed | Bernd |

### Preflight gate (mandatory before infrastructure work)

```bash
bin/railiance preflight
```

| Check | Pass condition |
| --- | --- |
| DB backup freshness | Latest `db-*.sql.age` < 24 hours old |
| Config backup freshness | Latest `config-*.tar.gz.age` < 24 hours old |
| Git repos clean | No uncommitted changes in tracked repos |
| Git repos pushed | No unpushed commits |
| age key present | `~/.config/age/railiance-backup.key` exists |

Exit 0 = proceed. Exit 1 = **do not proceed**.

### Silent-failure mitigations

1. Cron jobs log to `~/.cache/railiance/backup.log` — review on M-1 failure.
2. Backup scripts exit non-zero on any step failure (`set -euo pipefail`).
3. Missing tool (`railiance-platform` `make backup`) is a **policy violation** —
   tracked as `backup-s3-platform` status `missing`.
4. Forgejo lane not wired — treat as **degraded** until 7-day promotion gate met.

### Evidence output

Every restore drill produces a file under `TestEvidence/`:

```markdown
# Restore Drill: <system>
Date, backup artifact used, target environment, result, RTO measured,
RPO measured, issues, follow-up actions.
```

---

## 10. Open gaps (honest)

| Gap | Risk | Remediation | Owner repo |
| --- | --- | --- | --- |
| S3 `railiance-backup` tool missing | Platform DB unprotected | Restore or implement `tools/cmd/railiance-backup` | `railiance-platform` |
| Forgejo automation not wired | RPO = last manual dump | Daily cron + Nextcloud upload per §7 | `railiance-platform` |
| S1/S2 no offsite upload | Lost if railiance01 destroyed | Extend platform lane upload to S1/S2 artifacts | `railiance-platform` |
| Gitea no automated backup | Canonical forge at risk | CronJob + mirror; pre-cutover dump runbook | `railiance-infra` |
| `apps-pg` no ScheduledBackup | App data unprotected | CNPG ScheduledBackup CR | `railiance-platform` |
| No immutable weekly offline copy | R3 residual risk | Add offline/append-only copy (Phase 2) | TBD |
| Cold decrypt not evidenced | R5 residual risk | Quarterly cold decrypt drill (M-3 extended) | `disaster-control` |

Track closure in `BackupPickupQueue.md` (DISCTL-WP-0003-T04).

---

## 11. References

- `AssetRegister.yaml` — stream inventory and status
- `DataRegister.yaml` — data class mapping
- `specs/ResilienceControlSetup.md` — architecture and tier model
- `history/2026-07-04-forgejo-backup-strategy-assessment.md` — gap analysis
- `history/2026-07-04-initial-risk-analysis.md` — R3, R9 risk context
- `railiance-cluster/docs/backup-restore.md` — dev lane operational procedures
- `the-custodian/docs/forgejo-production-decisions.md` — Option A adoption
- `railiance-infra/docs/forgejo-restore-drill-evidence.md` — T09 restore proof