railiance-infra/workplans/RAIL-HO-WP-0012-s1-backup-recovery-loop.md

---
id: RAIL-HO-WP-0012
type: workplan
title: "Close the encrypted S1 backup and recovery loop"
domain: financials
repo: railiance-infra
status: active
owner: codex
topic_slug: railiance
created: "2026-08-23"
updated: "2026-08-23"
related:
  - RAIL-HO-WP-0011
state_hub_workstream_id: "5ea28f8f-376c-5230-8bb7-ca871c1a75f4"
---

# RAIL-HO-WP-0012 — encrypted S1 backup and recovery loop

## Goal

Turn the existing local `railiance-backup-s1` helper into a fail-closed,
scheduled, off-host, and restore-tested S1 recovery capability. Preserve the
repository boundary: this covers host operating-system configuration and
package-selection evidence only, not Kubernetes, database, Forgejo, or
application data.

This workplan does **not** authorize deployment of a systemd timer, access to an
age private key, an off-host upload, a live `/etc` restore, or deletion of a
retained artifact. Each live action retains its normal owner review and exact
operator approval.

## Current State and Risks

The source-side local loop now has a declaration, atomic encrypted bundle,
metadata-only evidence, integrity/freshness status, exact-approval retention,
isolated restore, and separately approved timer deployment interface. No timer
has been deployed and no retained artifact has been deleted by this workplan.
Off-host transfer and an attended drill remain open, so the repository still
cannot claim operational recoverability from local source tests alone.

## Delivery Order

T01 → T02 → T03 establish a locally testable contract. T04 and T05 may proceed
after T02 but need separate deployment/custody review. T06 depends on all prior
tasks and an attended isolated drill.

## T01 — Define the S1 backup declaration and safety contract

```task
id: RAIL-HO-WP-0012-T01
status: done
priority: high
state_hub_task_id: "0823a432-d43d-526f-be28-765dabc8c1ed"
```

Create a machine-readable declaration for included paths, optional paths,
artifact classes, public encryption recipients, retention, and excluded
higher-layer data. Validate that every selected path is absolute and bounded,
that no private-key or credential path can be selected, and that the recipient
contract cannot drift silently from the repository's approved public metadata.

**Done when:** the current intended S1 files resolve deterministically, unsafe
or relative paths fail closed, optional absence is explicit, and validation
requires neither root nor a decryption key.

## T02 — Produce atomic encrypted artifacts, manifests, and receipts

```task
id: RAIL-HO-WP-0012-T02
status: done
priority: high
state_hub_task_id: "689b3c34-acd0-58dd-8705-0ed621f4d9ab"
```

Refactor the backup helper around the T01 declaration. Check privileges before
creating the destination, stage output on the same filesystem, fail on an
unreadable required input, encrypt before publication, and atomically publish
only complete artifacts. Emit a metadata-only manifest and receipt containing
host identity, source revision, declaration digest, artifact class, encrypted
size/hash, completion time, and retention decision.

Exercise it against disposable fixture roots and output directories. Tests
must prove a failed collection or encryption cannot produce a passing receipt
or a final-named partial artifact.

**Done when:** a fixture backup is deterministic in membership, encrypted
before publication, fully described by redacted metadata, and failure-injection
tests leave no ambiguous success state.

## T03 — Add non-destructive inspection and isolated restore validation

```task
id: RAIL-HO-WP-0012-T03
status: done
priority: high
state_hub_task_id: "f1c344a2-6609-5900-87ce-0744d651bead"
```

Provide a restore command that defaults to inspection or extraction beneath an
explicit empty staging directory. It must verify receipt/manifest hashes before
decryption, suppress decrypted content from logs, reject path traversal and
links escaping the staging root, and compare restored membership and critical
file modes with the declaration.

Restoring over live `/etc` is outside this task and must remain impossible
without a separately designed exact-approval interface.

**Done when:** a fixture artifact restores into disposable storage, tampering
and unsafe members fail closed, no plaintext enters repository paths or logs,
and no default invocation can overwrite a host file.

## T04 — Define scheduling, freshness, and local retention

```task
id: RAIL-HO-WP-0012-T04
status: done
priority: medium
state_hub_task_id: "867c1959-36d0-5446-88cd-4b71727f320a"
```

Add source-controlled systemd service/timer units and Ansible deployment for a
bounded backup cadence. Define maximum acceptable age, failure state, disk
budget, atomic pruning, and an operator status interface. Keep timer deployment
separate from source validation and require explicit host-change approval.

**Done when:** unit rendering and calendar behavior test locally, stale/missing
backup status fails, pruning never crosses the declared backup directory, and
no timer is installed merely by running a verification command.

## T05 — Integrate the governed write-only off-host lane

```task
id: RAIL-HO-WP-0012-T05
status: progress
priority: high
state_hub_task_id: "783dff8b-849e-5ed9-a668-af1184be7bdd"
```

Coordinate with `railiance-platform` through the existing
`railiance-backup-offsite-lane` route. Upload only completed encrypted
artifacts plus metadata manifests, use a host-local or workload-owned
credential projection, and record a non-secret remote object identity and
encrypted hash. Do not place WebDAV credentials or an age private key in this
repository, State Hub, command output, or agent prompts.

**Done when:** the owner accepts the exact projection and write contract, a
dry-run proves object naming and collision behavior, a controlled upload is
visible through owner-provided metadata, and local/off-host retention cannot
delete the only recoverable copy.

**Source preparation (2026-08-23):** `spec/s1-offsite.yaml` and
`scripts/s1_offsite.py` define a pending-owner contract that wraps the complete
encrypted bundle and metadata into one deterministic immutable object. The
upload uses `If-None-Match: *`, accepts only a freshly rendered exact envelope
approval, suppresses endpoint/response data, and emits a local metadata-only
receipt. Production pruning now refuses candidates without a receipt bound to
the current manifest. A disposable WebDAV fixture proves successful upload,
collision and redirect refusal, pending-owner refusal, and absence of the
injected token from retained evidence.

Owner review remains mandatory. The exact review digest is rendered by
`make s1-offsite-review`; `spec/s1-offsite-owner-acceptance.yaml` remains
`pending`, so no live upload can execute. Review of the owner implementation
also found a tracked secret-shaped default in its Forgejo backup script. Its
value was not copied or recorded; `railiance-platform` must confirm it is
retired or rotated before accepting this contract. The route's missing
authoritative workload identity keeps this interface attended rather than
scheduled.

## T06 — Perform an attended isolated restore drill

```task
id: RAIL-HO-WP-0012-T06
status: wait
priority: high
state_hub_task_id: "2ae6feab-3aa3-58e8-a4a9-1bd70a5e30ca"
```

Select a fresh off-host artifact by metadata, retrieve it through the governed
lane, and restore it into an isolated disposable filesystem. Prove manifest
integrity, expected membership and modes, usable SSH/fail2ban/UFW configuration
syntax, package-selection readability, and cleanup of all decrypted temporary
material. Do not restore over a live host in this drill.

**Done when:** a metadata-only drill receipt records artifact and source
identity, checks, duration, cleanup, and result; an independent observer can
distinguish a successful recovery from a backup-only claim without seeing
decrypted configuration.

## Acceptance

- [x] S1 backup membership, exclusions, recipients, and retention are declared
      and validated from source.
- [x] Backup publication is atomic, encrypted, and accompanied by a safe
      manifest and receipt.
- [x] Restore defaults to isolated inspection and rejects tampering, traversal,
      and accidental live overwrite.
- [x] Scheduling and freshness checks are source-controlled but deployed only
      through an explicit host-change approval.
- [ ] A governed off-host copy and an attended isolated restore drill prove the
      selected S1 state is recoverable.
- [x] No cluster, platform, tenant, secret-issuance, or private-key custody
      responsibility is absorbed into this repository.

## Source delivery record — 2026-08-23

- `make validate-s1-backup` passes the declaration and 21 fixture recovery
  tests using disposable age identities and storage.
- The full repository suite passes 51 tests. The timer calendar is accepted by
  `systemd-analyze`; deployment YAML and separation from bootstrap are checked
  as source contracts.
- Native `ansible-playbook --syntax-check` was unavailable on the development
  workstation. The deployment remains unexecuted and gated by
  `DEPLOY-RAIL-HO-WP-0012-S1-BACKUP-TIMER` plus a clean committed revision.
- T05 is source-prepared and pending owner acceptance; T06 remains `wait`. No
  off-host write, retained-artifact deletion, private-key access, or live-host
  restore occurred.