Complete DISCTL-WP-0003-T02: adopt BackupPolicy.md
Codify railiance-backup practices — 3-2-1 model, encryption, retention, Forgejo Option A (2026-07-09), credential separation, and monitoring. Mark T02 done.
This commit is contained in:
parent
46061c357a
commit
8b5706f693
2 changed files with 287 additions and 1 deletions
286
BackupPolicy.md
Normal file
286
BackupPolicy.md
Normal file
|
|
@ -0,0 +1,286 @@
|
|||
# Backup Policy
|
||||
|
||||
**Status:** adopted (initial)
|
||||
**Updated:** 2026-07-07
|
||||
**Owner:** Bernd
|
||||
**Scope:** Coulomb Social / Helix Forge — Railiance backup lanes
|
||||
**Register:** `AssetRegister.yaml` (stream inventory) · `DataRegister.yaml` (data classes)
|
||||
|
||||
This policy translates live `railiance-backup` practices into enforceable rules
|
||||
with verification steps. It governs *what* must be backed up and *how*; backup
|
||||
scripts run in owner repos (`railiance-cluster`, `railiance-infra`,
|
||||
`railiance-platform`, `railiance-apps`).
|
||||
|
||||
---
|
||||
|
||||
## 1. Operating principle
|
||||
|
||||
> Every T1 data class must have a named backup stream, encryption, retention,
|
||||
> offsite copy (where required), monitoring, and restore evidence.
|
||||
|
||||
Servers are disposable; backups are not. Rebuild + restore is the default
|
||||
recovery pattern — not server resurrection.
|
||||
|
||||
Honest status reporting is required: streams marked `missing`, `not_wired`, or
|
||||
`at_risk` in `AssetRegister.yaml` are policy violations until closed.
|
||||
|
||||
---
|
||||
|
||||
## 2. Architecture — 3-2-1 + immutable/offline
|
||||
|
||||
Target model (per `specs/ResilienceControlSetup.md` §4):
|
||||
|
||||
| Copy | Role | Implementation today |
|
||||
| --- | --- | --- |
|
||||
| **Production** | Live data | Production hosts, PVCs, databases |
|
||||
| **Local fast rollback** | Same-site recovery | `/opt/backup/railiance/{infra,cluster}/`; `~/.cache/railiance/backups/` |
|
||||
| **Offsite encrypted** | Survives host loss | Nextcloud WebDAV file drop (upload-only) |
|
||||
| **Immutable/offline** | Survives hostile delete | Phase 1: upload-only drop token (no delete from production); weekly offline copy — **not yet implemented** |
|
||||
|
||||
**3-2-1 compliance by stream:**
|
||||
|
||||
| Stream | 3 copies | 2 media | 1 offsite | Notes |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Dev workstation | ✓ (prod + local cache + Nextcloud) | ✓ (disk + cloud) | ✓ | Operational |
|
||||
| S1 OS config | ✗ (prod + local only) | ✗ (disk only) | ✗ | **Gap — needs offsite upload** |
|
||||
| S2 k8s runtime | ✗ (prod + local only) | ✗ (disk only) | ✗ | **Gap — needs offsite upload** |
|
||||
| S3 platform | — | — | — | Tool missing |
|
||||
| Forgejo production | ✗ (prod + drill /tmp only) | partial | intended | Automation not wired |
|
||||
| Gitea canonical | ✗ | ✗ | ✗ | Manual pre-cutover only |
|
||||
|
||||
---
|
||||
|
||||
## 3. Encryption standard
|
||||
|
||||
All backup artifacts **must** be encrypted with [age](https://age-encryption.org/)
|
||||
before leaving the host or entering offsite storage.
|
||||
|
||||
### Key custody
|
||||
|
||||
| Key | Operational location | Recovery copy | Used by |
|
||||
| --- | --- | --- | --- |
|
||||
| Dev-lane age key | `~/.config/age/railiance-backup.key` | Password manager | `railiance-backup` (dev workstation) |
|
||||
| S1/S2 age key | `~/.config/sops/age/keys.txt` | Password manager + SOPS recovery | `railiance-backup-s1`, `railiance-backup-s2` |
|
||||
| Platform/Forgejo lane | Platform backup public key (TBD in OpenBao) | Password manager | Intended S3 + Forgejo dumps |
|
||||
|
||||
**Rules:**
|
||||
|
||||
1. Operational keys: `chmod 700` on `~/.config/age`; `chmod 600` on key files.
|
||||
2. Password-manager recovery copy is mandatory — it is the only key that survives
|
||||
hardware failure.
|
||||
3. Never store private keys in Git, workplans, State Hub, or chat.
|
||||
4. Offsite credentials live in OpenBao (`platform/workloads/railiance/backup/offsite-lane`);
|
||||
not on production hosts.
|
||||
|
||||
### Verification
|
||||
|
||||
| Check | Frequency | Pass condition | Owner |
|
||||
| --- | --- | --- | --- |
|
||||
| Recovery key in password manager | Monthly | Operator confirms copy exists and decrypts a test artifact | Bernd |
|
||||
| Operational key permissions | Monthly | `stat` shows 600 on key file, 700 on parent dir | Bernd |
|
||||
| Cold decrypt drill | Quarterly | Decrypt one artifact from each active lane without using production host | Bernd |
|
||||
|
||||
---
|
||||
|
||||
## 4. Destination policy
|
||||
|
||||
### Off-node — Nextcloud WebDAV file drop
|
||||
|
||||
Used by: dev workstation lane (operational); Forgejo + platform lane (adopted, not fully wired).
|
||||
|
||||
- **Endpoint:** filesdrop URL — `https://<host>/public.php/dav/filesdrop/<token>/<filename>`
|
||||
- **Not valid:** `/public.php/webdav/` (returns HTTP 409 on current Nextcloud)
|
||||
- **Upload method:** `curl -T` with `-u <token>:` — upload-only; no rclone listing
|
||||
- **Properties:** upload-only token cannot read or delete existing artifacts from
|
||||
the production host — supports credential separation (R3)
|
||||
- **Human link:** `260225-backup-dropoff-link.txt` (gitignored); credentials in OpenBao
|
||||
|
||||
### On-node — local encrypted cache
|
||||
|
||||
| Path | Lane | Retention |
|
||||
| --- | --- | --- |
|
||||
| `~/.cache/railiance/backups/` | Dev workstation | 7 per type |
|
||||
| `/opt/backup/railiance/infra/` | S1 OS config | 7 per type |
|
||||
| `/opt/backup/railiance/cluster/` | S2 k8s runtime | 7 per type |
|
||||
|
||||
Local caches are **fast rollback only** — not a substitute for offsite copy.
|
||||
|
||||
---
|
||||
|
||||
## 5. Retention policy
|
||||
|
||||
| Lane | Daily rotations | Weekly rotations | Local prune | Status |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Dev workstation | N/A (continuous daily) | N/A | 7 per type | operational |
|
||||
| S1 / S2 | N/A (manual) | N/A | 7 per type | operational |
|
||||
| Forgejo + platform (adopted 2026-07-09) | 14 | 4 (Sunday promote or explicit job) | 7 per type | not wired |
|
||||
| Gitea pre-cutover | Point-in-time of last manual dump | — | — | at risk |
|
||||
|
||||
Prune logic: `find … \| sort -r \| tail -n +8 \| xargs rm` (keep last 7) in
|
||||
`railiance-backup`, `railiance-backup-s1`, `railiance-backup-s2`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Per-stream mechanisms
|
||||
|
||||
Canonical inventory: `AssetRegister.yaml`. Summary:
|
||||
|
||||
| Register ID | Tool | Schedule | Destination | Status |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `backup-dev-workstation` | `railiance-backup` | Daily 02:00 cron | Nextcloud + local cache | operational |
|
||||
| `backup-s1-os-config` | `railiance-backup-s1` | Manual `make backup` | `/opt/backup/railiance/infra/` | operational |
|
||||
| `backup-s2-k8s-runtime` | `railiance-backup-s2` | Manual `make backup` | `/opt/backup/railiance/cluster/` | operational |
|
||||
| `backup-s3-platform` | `railiance-backup` | TBD | Nextcloud (intended) | **missing** |
|
||||
| `backup-forgejo-production` | `forgejo dump` + `pg_dump` | Daily (intended) | Nextcloud (adopted) | **not_wired** |
|
||||
| `backup-gitea-canonical` | `gitea dump` / mirror | Pre-cutover manual | None automated | **at_risk** |
|
||||
|
||||
**Related streams** (tracked, not yet full policy coverage):
|
||||
|
||||
- `backup-adjacent-sso` — daily CronJobs (`privacyidea-backup`, `authelia-backup`, `lldap-backup`) on railiance01
|
||||
- `backup-apps-pg` — CNPG `apps-pg`; no ScheduledBackup CR
|
||||
|
||||
**Not in git backups:** Git repos are protected by push-to-Gitea remotes. Preflight
|
||||
verifies clean and pushed state before infrastructure work.
|
||||
|
||||
---
|
||||
|
||||
## 7. Adopted standard — Forgejo backup (Option A, 2026-07-09)
|
||||
|
||||
**Decision:** Extend the existing Railiance platform backup lane — age-encrypted
|
||||
artifacts uploaded to Nextcloud WebDAV.
|
||||
|
||||
| Component | Method | Schedule |
|
||||
| --- | --- | --- |
|
||||
| Forgejo blob state | `forgejo dump` zip (repos, OCI/npm/generic packages, attachments, LFS, avatars) | Daily |
|
||||
| PostgreSQL | `pg_dump` from CNPG `forgejo-db` (logical; no WAL/PITR in Phase 1) | Daily |
|
||||
| Encryption | age (platform backup public key) | Per artifact |
|
||||
| Destination | Nextcloud WebDAV file drop | Upload after each run |
|
||||
| Restore proof | `forgejo-restore-drill.sh` from automated backup | Quarterly |
|
||||
|
||||
**RPO / RTO:** 24h / 4h.
|
||||
|
||||
**Implementation owners:**
|
||||
|
||||
- `railiance-platform` — cron, age encrypt, Nextcloud upload, retention prune
|
||||
- `railiance-infra` — restore runbook + quarterly drill evidence
|
||||
- `railiance-apps` — no backup secrets in Git; dump runs against live pod
|
||||
|
||||
**Phase 1 exclusions:** CNPG WAL archiving to S3/MinIO; on-node-only backups
|
||||
without Nextcloud upload.
|
||||
|
||||
**Promotion gate:** No further tier-3 repo cutovers until automated daily backups
|
||||
succeed for **7 consecutive days** and one restore drill uses a Nextcloud artifact
|
||||
(not workstation `/tmp`).
|
||||
|
||||
**Current state:** Decision adopted; automation not implemented. Restore path
|
||||
proven manually (RAIL-HO-WP-0005-T09, 2026-07-04).
|
||||
|
||||
Reference: `the-custodian/docs/forgejo-production-decisions.md`
|
||||
|
||||
---
|
||||
|
||||
## 8. Credential separation (R3)
|
||||
|
||||
**Rule:** Backup-storage credentials must **not** live on the production host with
|
||||
permission to delete or overwrite offsite backups.
|
||||
|
||||
### How this is enforced today
|
||||
|
||||
| Control | Mechanism |
|
||||
| --- | --- |
|
||||
| Upload-only file drop | Nextcloud public share token permits PUT only — production host cannot list or delete remote artifacts |
|
||||
| Separate key material | age private keys stored separately from backup upload tokens |
|
||||
| OpenBao custody | Offsite lane credentials in `platform/workloads/railiance/backup/offsite-lane` — not in Git |
|
||||
| No shared admin | Production k3s/Gitea admin credentials ≠ Nextcloud admin credentials |
|
||||
|
||||
### Verification steps
|
||||
|
||||
| Step | Frequency | Procedure | Owner |
|
||||
| --- | --- | --- | --- |
|
||||
| CS-1 Token scope | Quarterly | Confirm production backup token cannot DELETE or LIST on Nextcloud drop (curl probe) | Bernd |
|
||||
| CS-2 Credential inventory | Monthly | Verify no backup-storage admin credentials in `/etc`, kube Secrets, or Ansible vars on railiance01 | Bernd |
|
||||
| CS-3 Separate accounts | Quarterly | Confirm production, backup, registrar, and payment systems use distinct admin identities | Bernd |
|
||||
| CS-4 Breach runbook | After adoption | Incident playbook requires: never restore into compromised machine; rotate before restore | Bernd |
|
||||
|
||||
**Known exposure:** S1/S2 local backups on railiance01 are deletable by root on
|
||||
that host. Mitigation: wire offsite upload for S1/S2 (tracked in pickup queue).
|
||||
|
||||
---
|
||||
|
||||
## 9. Monitoring and evidence (R9)
|
||||
|
||||
**Alerting owner:** Bernd (until a formal on-call rotation is defined).
|
||||
|
||||
### Required checks
|
||||
|
||||
| Check ID | Check | Frequency | Pass condition | Owner |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| M-1 | Backup job succeeded | Daily | `.last-backup` stamp < 24h old for each active lane; cron log shows exit 0 | Bernd |
|
||||
| M-2 | Artifact freshness | Daily | Latest `db-*.sql.age` and `config-*.tar.gz.age` < 24h (dev lane) | Bernd |
|
||||
| M-3 | Repository integrity | Weekly | age-decrypt test on one random artifact per lane; file size > 0; pg_dump header valid | Bernd |
|
||||
| M-4 | Preflight gate | Before infra change | `bin/railiance preflight` exits 0 | Bernd |
|
||||
| M-5 | Restore drill | Quarterly per T1 stream | Dated evidence in `TestEvidence/` | Bernd |
|
||||
| M-6 | Register review | Monthly | `AssetRegister.yaml` statuses updated; gaps unchanged or closed | Bernd |
|
||||
|
||||
### Preflight gate (mandatory before infrastructure work)
|
||||
|
||||
```bash
|
||||
bin/railiance preflight
|
||||
```
|
||||
|
||||
| Check | Pass condition |
|
||||
| --- | --- |
|
||||
| DB backup freshness | Latest `db-*.sql.age` < 24 hours old |
|
||||
| Config backup freshness | Latest `config-*.tar.gz.age` < 24 hours old |
|
||||
| Git repos clean | No uncommitted changes in tracked repos |
|
||||
| Git repos pushed | No unpushed commits |
|
||||
| age key present | `~/.config/age/railiance-backup.key` exists |
|
||||
|
||||
Exit 0 = proceed. Exit 1 = **do not proceed**.
|
||||
|
||||
### Silent-failure mitigations
|
||||
|
||||
1. Cron jobs log to `~/.cache/railiance/backup.log` — review on M-1 failure.
|
||||
2. Backup scripts exit non-zero on any step failure (`set -euo pipefail`).
|
||||
3. Missing tool (`railiance-platform` `make backup`) is a **policy violation** —
|
||||
tracked as `backup-s3-platform` status `missing`.
|
||||
4. Forgejo lane not wired — treat as **degraded** until 7-day promotion gate met.
|
||||
|
||||
### Evidence output
|
||||
|
||||
Every restore drill produces a file under `TestEvidence/`:
|
||||
|
||||
```markdown
|
||||
# Restore Drill: <system>
|
||||
Date, backup artifact used, target environment, result, RTO measured,
|
||||
RPO measured, issues, follow-up actions.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. Open gaps (honest)
|
||||
|
||||
| Gap | Risk | Remediation | Owner repo |
|
||||
| --- | --- | --- | --- |
|
||||
| S3 `railiance-backup` tool missing | Platform DB unprotected | Restore or implement `tools/cmd/railiance-backup` | `railiance-platform` |
|
||||
| Forgejo automation not wired | RPO = last manual dump | Daily cron + Nextcloud upload per §7 | `railiance-platform` |
|
||||
| S1/S2 no offsite upload | Lost if railiance01 destroyed | Extend platform lane upload to S1/S2 artifacts | `railiance-platform` |
|
||||
| Gitea no automated backup | Canonical forge at risk | CronJob + mirror; pre-cutover dump runbook | `railiance-infra` |
|
||||
| `apps-pg` no ScheduledBackup | App data unprotected | CNPG ScheduledBackup CR | `railiance-platform` |
|
||||
| No immutable weekly offline copy | R3 residual risk | Add offline/append-only copy (Phase 2) | TBD |
|
||||
| Cold decrypt not evidenced | R5 residual risk | Quarterly cold decrypt drill (M-3 extended) | `disaster-control` |
|
||||
|
||||
Track closure in `BackupPickupQueue.md` (DISCTL-WP-0003-T04).
|
||||
|
||||
---
|
||||
|
||||
## 11. References
|
||||
|
||||
- `AssetRegister.yaml` — stream inventory and status
|
||||
- `DataRegister.yaml` — data class mapping
|
||||
- `specs/ResilienceControlSetup.md` — architecture and tier model
|
||||
- `history/2026-07-04-forgejo-backup-strategy-assessment.md` — gap analysis
|
||||
- `history/2026-07-04-initial-risk-analysis.md` — R3, R9 risk context
|
||||
- `railiance-cluster/docs/backup-restore.md` — dev lane operational procedures
|
||||
- `the-custodian/docs/forgejo-production-decisions.md` — Option A adoption
|
||||
- `railiance-infra/docs/forgejo-restore-drill-evidence.md` — T09 restore proof
|
||||
|
|
@ -68,7 +68,7 @@ Feeds `DISCTL-WP-0002-T02`.
|
|||
|
||||
```task
|
||||
id: DISCTL-WP-0003-T02
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "2e32792d-4c22-4e76-adf5-44d9e81bd09a"
|
||||
```
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue