disaster-control/BackupPolicy.md

Backup Policy

Status: adopted (initial)
Updated: 2026-10-23
Owner: Bernd
Scope: Coulomb Social / Helix Forge — Railiance backup lanes
Register: AssetRegister.yaml (stream inventory) · DataRegister.yaml (data classes)

This policy translates live railiance-backup practices into enforceable rules with verification steps. It governs what must be backed up and how; backup scripts run in owner repos (railiance-cluster, railiance-infra, railiance-platform, railiance-apps).

---

1. Operating principle

Every T1 data class must have a named backup stream, encryption, retention, offsite copy (where required), monitoring, and restore evidence.

Servers are disposable; backups are not. Rebuild + restore is the default recovery pattern — not server resurrection.

Honest status reporting is required: streams marked missing, not_wired, or at_risk in AssetRegister.yaml are policy violations until closed.

---

2. Architecture — 3-2-1 + immutable/offline

Target model (per specs/ResilienceControlSetup.md §4):

Copy	Role	Implementation today
Production	Live data	Production hosts, PVCs, databases
Local fast rollback	Same-site recovery	/opt/backup/railiance/{infra,cluster}/; ~/.cache/railiance/backups/
Offsite encrypted	Survives host loss	Nextcloud WebDAV file drop (upload-only)
Immutable/offline	Survives hostile delete	Phase 1: upload-only drop token (no delete from production); weekly offline copy — not yet implemented

3-2-1 compliance by stream:

Stream	3 copies	2 media	1 offsite	Notes
Dev workstation	✓ (prod + local cache + Nextcloud)	✓ (disk + cloud)	✓	Operational
S1 OS config	✗ (prod + local only)	✗ (disk only)	✗	Gap — needs offsite upload
S2 k8s runtime	✗ (prod + local only)	✗ (disk only)	✗	Gap — needs offsite upload
S3 platform	—	—	—	Tool missing
Forgejo production	✗ (prod + drill /tmp only)	partial	intended	Automation not wired
Gitea canonical	✗	✗	✗	Manual pre-cutover only
NetKingdom identity	✓ (prod + on-host + Storage Share)	✓ (disk + cloud)	✓ (single provider)	Operational since 2026-10-16 (NK-WP-0049-T14); single off-host provider accepted until 2027-01-31 (G1)

---

3. Encryption standard

All backup artifacts must be encrypted with age before leaving the host or entering offsite storage.

Key custody

Key	Operational location	Recovery copy	Used by
Dev-lane age key	~/.config/age/railiance-backup.key	Password manager	railiance-backup (dev workstation)
S1/S2 age key	~/.config/sops/age/keys.txt	Password manager + SOPS recovery	railiance-backup-s1, railiance-backup-s2
Platform/Forgejo lane	Platform backup public key (TBD in OpenBao)	Password manager	Intended S3 + Forgejo dumps

Rules:

1. Operational keys: chmod 700 on ~/.config/age; chmod 600 on key files.
2. Password-manager recovery copy is mandatory — it is the only key that survives hardware failure.
3. Never store private keys in Git, workplans, State Hub, or chat.
4. Offsite credentials live in OpenBao (platform/workloads/railiance/backup/offsite-lane); not on production hosts.

Verification

Check	Frequency	Pass condition	Owner
Recovery key in password manager	Monthly	Operator confirms copy exists and decrypts a test artifact	Bernd
Operational key permissions	Monthly	stat shows 600 on key file, 700 on parent dir	Bernd
Cold decrypt drill	Quarterly	Decrypt one artifact from each active lane without using production host	Bernd

---

4. Destination policy

Off-node — Nextcloud WebDAV file drop

Used by: dev workstation lane (operational); Forgejo + platform lane (adopted, not fully wired).

• Endpoint: filesdrop URL — https://<host>/public.php/dav/filesdrop/<token>/<filename>
• Not valid: /public.php/webdav/ (returns HTTP 409 on current Nextcloud)
• Upload method: curl -T with -u <token>: — upload-only; no rclone listing
• Properties: upload-only token cannot read or delete existing artifacts from the production host — supports credential separation (R3)
• Human link: 260225-backup-dropoff-link.txt (gitignored); credentials in OpenBao

On-node — local encrypted cache

Path	Lane	Retention
~/.cache/railiance/backups/	Dev workstation	7 per type
/opt/backup/railiance/infra/	S1 OS config	7 per type
/opt/backup/railiance/cluster/	S2 k8s runtime	7 per type

Local caches are fast rollback only — not a substitute for offsite copy.

---

5. Retention policy

Lane	Daily rotations	Weekly rotations	Local prune	Status
Dev workstation	N/A (continuous daily)	N/A	7 per type	operational
S1 / S2	N/A (manual)	N/A	7 per type	operational
Forgejo + platform (adopted 2026-07-09)	14	4 (Sunday promote or explicit job)	7 per type	not wired
Gitea pre-cutover	Point-in-time of last manual dump	—	—	at risk

Prune logic: find … \| sort -r \| tail -n +8 \| xargs rm (keep last 7) in railiance-backup, railiance-backup-s1, railiance-backup-s2.

---

6. Per-stream mechanisms

Canonical inventory: AssetRegister.yaml. Summary:

Register ID	Tool	Schedule	Destination	Status
backup-dev-workstation	railiance-backup	Daily 02:00 cron	Nextcloud + local cache	operational
backup-s1-os-config	railiance-backup-s1	Manual make backup	/opt/backup/railiance/infra/	operational
backup-s2-k8s-runtime	railiance-backup-s2	Manual make backup	/opt/backup/railiance/cluster/	operational
backup-s3-platform	railiance-backup	TBD	Nextcloud (intended)	missing
backup-forgejo-production	forgejo dump + pg_dump	Daily (intended)	Nextcloud (adopted)	not_wired
backup-gitea-canonical	gitea dump / mirror	Pre-cutover manual	None automated	at_risk
backup-netkingdom-identity	net-kingdom identity profile (LLDAP, privacyIDEA DB/token material/audit, KeyCape client registry archive)	Daily 03:45 UTC (+≤5 min jitter)	Storage Share (single off-host provider), age	operational

Related streams (tracked, not yet full policy coverage):

• backup-adjacent-sso — daily CronJobs (privacyidea-backup, authelia-backup, lldap-backup) on railiance01; LLDAP and privacyIDEA are governed as backup-netkingdom-identity since 2026-10-23 (DISCTL-WP-0004)
• backup-apps-pg — CNPG apps-pg; no ScheduledBackup CR

Not in git backups: Git repos are protected by push-to-Gitea remotes. Preflight verifies clean and pushed state before infrastructure work.

---

7. Adopted standard — Forgejo backup (Option A, 2026-07-09)

Decision: Extend the existing Railiance platform backup lane — age-encrypted artifacts uploaded to Nextcloud WebDAV.

Component	Method	Schedule
Forgejo blob state	forgejo dump zip (repos, OCI/npm/generic packages, attachments, LFS, avatars)	Daily
PostgreSQL	pg_dump from CNPG forgejo-db (logical; no WAL/PITR in Phase 1)	Daily
Encryption	age (platform backup public key)	Per artifact
Destination	Nextcloud WebDAV file drop	Upload after each run
Restore proof	forgejo-restore-drill.sh from automated backup	Quarterly

RPO / RTO: 24h / 4h.

Implementation owners:

• railiance-platform — cron, age encrypt, Nextcloud upload, retention prune
• railiance-infra — restore runbook + quarterly drill evidence
• railiance-apps — no backup secrets in Git; dump runs against live pod

Phase 1 exclusions: CNPG WAL archiving to S3/MinIO; on-node-only backups without Nextcloud upload.

Promotion gate: No further tier-3 repo cutovers until automated daily backups succeed for 7 consecutive days and one restore drill uses a Nextcloud artifact (not workstation /tmp).

Current state: Decision adopted; automation not implemented. Restore path proven manually (RAIL-HO-WP-0005-T09, 2026-07-04).

Reference: the-custodian/docs/forgejo-production-decisions.md

---

8. Credential separation (R3)

Rule: Backup-storage credentials must not live on the production host with permission to delete or overwrite offsite backups.

How this is enforced today

Control	Mechanism
Upload-only file drop	Nextcloud public share token permits PUT only — production host cannot list or delete remote artifacts
Separate key material	age private keys stored separately from backup upload tokens
OpenBao custody	Offsite lane credentials in platform/workloads/railiance/backup/offsite-lane — not in Git
No shared admin	Production k3s/Gitea admin credentials ≠ Nextcloud admin credentials

Verification steps

Step	Frequency	Procedure	Owner
CS-1 Token scope	Quarterly	Confirm production backup token cannot DELETE or LIST on Nextcloud drop (curl probe)	Bernd
CS-2 Credential inventory	Monthly	Verify no backup-storage admin credentials in /etc, kube Secrets, or Ansible vars on railiance01	Bernd
CS-3 Separate accounts	Quarterly	Confirm production, backup, registrar, and payment systems use distinct admin identities	Bernd
CS-4 Breach runbook	After adoption	Incident playbook requires: never restore into compromised machine; rotate before restore	Bernd

Known exposure: S1/S2 local backups on railiance01 are deletable by root on that host. Mitigation: wire offsite upload for S1/S2 (tracked in pickup queue).

---

9. Monitoring and evidence (R9)

Alerting owner: Bernd (until a formal on-call rotation is defined).

Required checks

Check ID	Check	Frequency	Pass condition	Owner
M-1	Backup job succeeded	Daily	.last-backup stamp < 24h old for each active lane; cron log shows exit 0	Bernd
M-2	Artifact freshness	Daily	Latest db-*.sql.age and config-*.tar.gz.age < 24h (dev lane)	Bernd
M-3	Repository integrity	Weekly	age-decrypt test on one random artifact per lane; file size > 0; pg_dump header valid	Bernd
M-4	Preflight gate	Before infra change	bin/railiance preflight exits 0	Bernd
M-5	Restore drill	Quarterly per T1 stream	Dated evidence in TestEvidence/	Bernd
M-6	Register review	Monthly	AssetRegister.yaml statuses updated; gaps unchanged or closed	Bernd
M-7	NetKingdom identity lane freshness	Hourly (on-host)	Latest identity backup < 25h old; on-host only until outside-node watchdog (RAPP-TELEMETRY-WP-0001-T04) lands (G2)	net-kingdom
M-8	NetKingdom identity restore drill	Quarterly; first by 2027-01-31	Isolated restore per net-kingdom/docs/identity-restore-drill.md (NK-WP-0053); evidence under net-kingdom/docs/evidence/<date>-identity-restore-drill/, linked from TestEvidence/	net-kingdom

Preflight gate (mandatory before infrastructure work)

bin/railiance preflight

Check	Pass condition
DB backup freshness	Latest db-*.sql.age < 24 hours old
Config backup freshness	Latest config-*.tar.gz.age < 24 hours old
Git repos clean	No uncommitted changes in tracked repos
Git repos pushed	No unpushed commits
age key present	~/.config/age/railiance-backup.key exists

Exit 0 = proceed. Exit 1 = do not proceed.

Silent-failure mitigations

1. Cron jobs log to ~/.cache/railiance/backup.log — review on M-1 failure.
2. Backup scripts exit non-zero on any step failure (set -euo pipefail).
3. Missing tool (railiance-platform make backup) is a policy violation — tracked as backup-s3-platform status missing.
4. Forgejo lane not wired — treat as degraded until 7-day promotion gate met.

Evidence output

Every restore drill produces a file under TestEvidence/:

# Restore Drill: <system>
Date, backup artifact used, target environment, result, RTO measured,
RPO measured, issues, follow-up actions.

---

10. Open gaps (honest)

Gap	Risk	Remediation	Owner repo
S3 railiance-backup tool missing	Platform DB unprotected	Restore or implement tools/cmd/railiance-backup	railiance-platform
Forgejo automation not wired	RPO = last manual dump	Daily cron + Nextcloud upload per §7	railiance-platform
S1/S2 no offsite upload	Lost if railiance01 destroyed	Extend platform lane upload to S1/S2 artifacts	railiance-platform
Gitea no automated backup	Canonical forge at risk	CronJob + mirror; pre-cutover dump runbook	railiance-infra
apps-pg no ScheduledBackup	App data unprotected	CNPG ScheduledBackup CR	railiance-platform
No immutable weekly offline copy	R3 residual risk	Add offline/append-only copy (Phase 2)	TBD
Cold decrypt not evidenced	R5 residual risk	Quarterly cold decrypt drill (M-3 extended)	disaster-control
Identity lane single off-host provider (G1)	Identity backups lost if Storage Share fails or is compromised	Accepted until 2027-01-31 (decision e4c288d9); review in DISCTL-WP-0004-T04	railiance-platform / net-kingdom
Identity lane on-host-only freshness detection (G2)	Silent backup failure if railiance01 itself is down	Accepted until 2027-01-31; outside-node watchdog owed by RAPP-TELEMETRY-WP-0001-T04	rapp-telemetry

Track closure in BackupPickupQueue.md (DISCTL-WP-0003-T04).

---

11. References

• AssetRegister.yaml — stream inventory and status
• DataRegister.yaml — data class mapping
• specs/ResilienceControlSetup.md — architecture and tier model
• history/2026-07-04-forgejo-backup-strategy-assessment.md — gap analysis
• history/2026-07-04-initial-risk-analysis.md — R3, R9 risk context
• railiance-cluster/docs/backup-restore.md — dev lane operational procedures
• the-custodian/docs/forgejo-production-decisions.md — Option A adoption
• railiance-infra/docs/forgejo-restore-drill-evidence.md — T09 restore proof
• net-kingdom/docs/identity-restore-drill.md — quarterly isolated identity restore drill (NK-WP-0053)