railiance-infra/workplans/RAIL-HO-WP-0012-s1-backup-recovery-loop.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	related	state_hub_workstream_id
RAIL-HO-WP-0012	workplan	Close the encrypted S1 backup and recovery loop	financials	railiance-infra	active	codex	railiance	2026-08-23	2026-08-23	RAIL-HO-WP-0011	5ea28f8f-376c-5230-8bb7-ca871c1a75f4

RAIL-HO-WP-0012 — encrypted S1 backup and recovery loop

Goal

Turn the existing local railiance-backup-s1 helper into a fail-closed, scheduled, off-host, and restore-tested S1 recovery capability. Preserve the repository boundary: this covers host operating-system configuration and package-selection evidence only, not Kubernetes, database, Forgejo, or application data.

This workplan does not authorize deployment of a systemd timer, access to an age private key, an off-host upload, a live /etc restore, or deletion of a retained artifact. Each live action retains its normal owner review and exact operator approval.

Current State and Risks

The source-side local loop now has a declaration, atomic encrypted bundle, metadata-only evidence, integrity/freshness status, exact-approval retention, isolated restore, and separately approved timer deployment interface. No timer has been deployed and no retained artifact has been deleted by this workplan. Off-host transfer and an attended drill remain open, so the repository still cannot claim operational recoverability from local source tests alone.

Delivery Order

T01 → T02 → T03 establish a locally testable contract. T04 and T05 may proceed after T02 but need separate deployment/custody review. T06 depends on all prior tasks and an attended isolated drill.

T01 — Define the S1 backup declaration and safety contract

id: RAIL-HO-WP-0012-T01
status: done
priority: high
state_hub_task_id: "0823a432-d43d-526f-be28-765dabc8c1ed"

Create a machine-readable declaration for included paths, optional paths, artifact classes, public encryption recipients, retention, and excluded higher-layer data. Validate that every selected path is absolute and bounded, that no private-key or credential path can be selected, and that the recipient contract cannot drift silently from the repository's approved public metadata.

Done when: the current intended S1 files resolve deterministically, unsafe or relative paths fail closed, optional absence is explicit, and validation requires neither root nor a decryption key.

T02 — Produce atomic encrypted artifacts, manifests, and receipts

id: RAIL-HO-WP-0012-T02
status: done
priority: high
state_hub_task_id: "689b3c34-acd0-58dd-8705-0ed621f4d9ab"

Refactor the backup helper around the T01 declaration. Check privileges before creating the destination, stage output on the same filesystem, fail on an unreadable required input, encrypt before publication, and atomically publish only complete artifacts. Emit a metadata-only manifest and receipt containing host identity, source revision, declaration digest, artifact class, encrypted size/hash, completion time, and retention decision.

Exercise it against disposable fixture roots and output directories. Tests must prove a failed collection or encryption cannot produce a passing receipt or a final-named partial artifact.

Done when: a fixture backup is deterministic in membership, encrypted before publication, fully described by redacted metadata, and failure-injection tests leave no ambiguous success state.

T03 — Add non-destructive inspection and isolated restore validation

id: RAIL-HO-WP-0012-T03
status: done
priority: high
state_hub_task_id: "f1c344a2-6609-5900-87ce-0744d651bead"

Provide a restore command that defaults to inspection or extraction beneath an explicit empty staging directory. It must verify receipt/manifest hashes before decryption, suppress decrypted content from logs, reject path traversal and links escaping the staging root, and compare restored membership and critical file modes with the declaration.

Restoring over live /etc is outside this task and must remain impossible without a separately designed exact-approval interface.

Done when: a fixture artifact restores into disposable storage, tampering and unsafe members fail closed, no plaintext enters repository paths or logs, and no default invocation can overwrite a host file.

T04 — Define scheduling, freshness, and local retention

id: RAIL-HO-WP-0012-T04
status: done
priority: medium
state_hub_task_id: "867c1959-36d0-5446-88cd-4b71727f320a"

Add source-controlled systemd service/timer units and Ansible deployment for a bounded backup cadence. Define maximum acceptable age, failure state, disk budget, atomic pruning, and an operator status interface. Keep timer deployment separate from source validation and require explicit host-change approval.

Done when: unit rendering and calendar behavior test locally, stale/missing backup status fails, pruning never crosses the declared backup directory, and no timer is installed merely by running a verification command.

T05 — Integrate the governed write-only off-host lane

id: RAIL-HO-WP-0012-T05
status: progress
priority: high
state_hub_task_id: "783dff8b-849e-5ed9-a668-af1184be7bdd"

Coordinate with railiance-platform through the existing railiance-backup-offsite-lane route. Upload only completed encrypted artifacts plus metadata manifests, use a host-local or workload-owned credential projection, and record a non-secret remote object identity and encrypted hash. Do not place WebDAV credentials or an age private key in this repository, State Hub, command output, or agent prompts.

Done when: the owner accepts the exact projection and write contract, a dry-run proves object naming and collision behavior, a controlled upload is visible through owner-provided metadata, and local/off-host retention cannot delete the only recoverable copy.

Source preparation (2026-08-23): spec/s1-offsite.yaml and scripts/s1_offsite.py define a pending-owner contract that wraps the complete encrypted bundle and metadata into one deterministic immutable object. The upload uses If-None-Match: *, accepts only a freshly rendered exact envelope approval, suppresses endpoint/response data, and emits a local metadata-only receipt. Production pruning now refuses candidates without a receipt bound to the current manifest. A disposable WebDAV fixture proves successful upload, collision and redirect refusal, pending-owner refusal, and absence of the injected token from retained evidence.

Owner review remains mandatory. The exact review digest is rendered by make s1-offsite-review; spec/s1-offsite-owner-acceptance.yaml remains pending, so no live upload can execute. Review of the owner implementation also found a tracked secret-shaped default in its Forgejo backup script. Its value was not copied or recorded; railiance-platform must confirm it is retired or rotated before accepting this contract. The route's missing authoritative workload identity keeps this interface attended rather than scheduled.

T06 — Perform an attended isolated restore drill

id: RAIL-HO-WP-0012-T06
status: wait
priority: high
state_hub_task_id: "2ae6feab-3aa3-58e8-a4a9-1bd70a5e30ca"

Select a fresh off-host artifact by metadata, retrieve it through the governed lane, and restore it into an isolated disposable filesystem. Prove manifest integrity, expected membership and modes, usable SSH/fail2ban/UFW configuration syntax, package-selection readability, and cleanup of all decrypted temporary material. Do not restore over a live host in this drill.

Done when: a metadata-only drill receipt records artifact and source identity, checks, duration, cleanup, and result; an independent observer can distinguish a successful recovery from a backup-only claim without seeing decrypted configuration.

Acceptance

• S1 backup membership, exclusions, recipients, and retention are declared and validated from source.
• Backup publication is atomic, encrypted, and accompanied by a safe manifest and receipt.
• Restore defaults to isolated inspection and rejects tampering, traversal, and accidental live overwrite.
• Scheduling and freshness checks are source-controlled but deployed only through an explicit host-change approval.
• A governed off-host copy and an attended isolated restore drill prove the selected S1 state is recoverable.
• No cluster, platform, tenant, secret-issuance, or private-key custody responsibility is absorbed into this repository.

Source delivery record — 2026-08-23

• make validate-s1-backup passes the declaration and 21 fixture recovery tests using disposable age identities and storage.
• The full repository suite passes 51 tests. The timer calendar is accepted by systemd-analyze; deployment YAML and separation from bootstrap are checked as source contracts.
• Native ansible-playbook --syntax-check was unavailable on the development workstation. The deployment remains unexecuted and gated by DEPLOY-RAIL-HO-WP-0012-S1-BACKUP-TIMER plus a clean committed revision.
• T05 is source-prepared and pending owner acceptance; T06 remains wait. No off-host write, retained-artifact deletion, private-key access, or live-host restore occurred.