# Coordinated `railiance01` reboot and recovery This is the owner-review checklist for `RAILIANCE-WP-0024-T03`. A reboot is a single-node platform outage, not a Kubernetes rolling operation and not an HA claim. This document does not authorize a reboot. ## Responsibility map | Phase | Accountable owner | Required evidence | | --- | --- | --- | | Window, notifications, go/no-go | operator + `railiance-platform` | approval/window id; new scenario id; fresh human decision; named driver | | Independent abort authority (RPF-WP-0017) | `railiance-infra` | named abort owner acknowledgement; may stop the window at any step without a platform vote | | Attended login containment | `railiance-platform` | `reviews/RPF-WP-0017-ops-warden-reacceptance-2026-10-14.json` accepting the exact ops-warden revision | | Host access, provider console, reboot, firewall, time | `railiance-infra` | SSH; same-day `RAIL01-CONSOLE-*` receipt; host returns; UFW/NTP/tunnel checks | | k3s/node/DNS/operators | `railiance-cluster` | node Ready; named core platform workloads Ready; failing-pod diff | | OpenBao snapshot, Shamir unseal, post-unseal checks | `railiance-platform` | current Raft snapshot receipt; 2-of-3 quorum attestation; unsealed verification | | All CNPG clusters (`platform-pg` backup/WAL) | `rapp-postgres` (+ owning app for each cluster) | per-cluster UID, primary PVC UID, systemID, 1/1 primary; fresh `platform-pg` Barman backup | | ESO stores and projections | `railiance-platform` | stores Valid; ExternalSecrets SecretSynced after unseal | | Audit custody and synthetic verification | `audit-core` | health/readiness/integrity and bounded synthetic E2 result | The last observed reboot on 2026-08-16 showed the expected dependency order: the host and k3s returned, audit-core was unready for about 40 seconds while DNS/PostgreSQL recovered, and OpenBao returned sealed. Existing mounted Secrets remained, but new leases could not refresh until unseal. ## Fail-closed preflight Run the value-safe, read-only gate: ```bash python3 scripts/audit-core-recovery-preflight.py node-reboot ``` The automated checks (`automated_checks` in the output) require: - one Ready node, active k3s, and the boot id recorded; - `timedatectl` `NTPSynchronized=yes` (receipt/backup ages are untrusted otherwise); - UFW active with exactly the declared allow set — `OpenSSH` and Nydus `2224/tcp` (v4/v6) — and `6443`/`8472` not open (railiance-infra RAIL-HO-WP-0009 / Goss declaration); - the workstation `k3s-api-railiance01` tunnel answering (`bridge check`); - every named core platform workload Ready, plus every other Deployment and StatefulSet in their namespaces: CoreDNS, local-path-provisioner, traefik, metrics-server, cert-manager, the CNPG operator, external-secrets, knative-serving (activator, controller, net-kourier-controller) and the kourier gateway, argocd, and openbao; - every CNPG cluster, discovered live (today `apps-pg`, `forgejo-db`, `net-kingdom-pg`, `platform-pg`, `platform-pg-2`, `state-hub-db`, `target-revenue-pg`, `user-engine-pg`; a vanished expected cluster fails), 1/1 Ready, current primary on a healthy PVC, `ContinuousArchiving=True` where backup is configured, with cluster UID, primary PVC UID, and PostgreSQL `systemID` (from `pg_controldata`, value-safe) recorded; - a completed `platform-pg` backup no older than 26 hours; - every ClusterSecretStore `Valid` and every ExternalSecret `SecretSynced`; - unsealed OpenBao and the reviewed audit-core digest in both the Deployment spec and the running pod (`imageID`), 1/1 Ready; - before the reboot, every Terminating PVC resolved or explicitly listed with `--accept-terminating-pvc`. The output also records the all-namespace failing-pod baseline (`reboot_evidence.failing_pods`). On 2026-10-14 it held six pre-existing pods: three `coulomb/ihp-railiance-probe-*`, `forgejo/forgejo-gitea-*`, `forgejo/forgejo-runner-*`, and `telemetry/telemetry-smtp-proof-20260928-*`. Keep the pre-reboot JSON; it is the comparison input after the reboot. The following are human gates and make `ready_for_live_execution` remain false until supplied: - approved window id and acknowledgements from all five owners; - a new scenario id (`--scenario-id`) and a fresh human GO decision (`--human-decision-id`). Consumed decisions `449a697a-…` and `85724c0c-…` and scenario `WARDEN-WP-0027-T02-DRILL-20260822-01` are not reusable and are denied by the preflight; - the RPF-WP-0017 re-acceptance receipt `reviews/RPF-WP-0017-ops-warden-reacceptance-2026-10-14.json` recording acceptance of the exact ops-warden revision used for the attended login (`11b2a9d…` was rejected; `6e14300…` accepted on 2026-10-14). The accepted revision must be pushed to ops-warden `origin/main` and installed with `uv tool install --reinstall /home/worsch/ops-warden` from that clean checkout before the window. The preflight compares every installed `warden` module byte-for-byte with the accepted revision and closes this gate on any mismatch; rerun it at the hold point immediately before the attended login and abort if it no longer matches (the 2026-09-01 WARDEN-WP-0036-T02 operation ran on unaccepted code — do not repeat that); - a named attended driver/abort operator and the `railiance-infra` independent abort owner acknowledgement (`--infra-abort-owner-ack`); - independent provider-console access, not SSH alone: a value-safe `RAIL01-CONSOLE-` receipt recorded by railiance-infra under its `docs/evidence/railiance01-console-access/` (procedure: railiance-infra `docs/railiance01-console-access.md`), passed with `--provider-console-receipt`. The preflight accepts it only when it is dated the window day, names the approved window id, and has `console_shows_host` and `independent_of_guest_network` true. No out-of-band console is established today, so T03 cannot pass this gate until railiance-infra RAIL-HO-WP-0011-T09 is done; - a current non-secret OpenBao Raft snapshot receipt that validates against `docs/openbao-reboot-snapshot-receipt.example.json`, the live cluster id and Raft index, and the default 24-hour freshness bound; - explicit availability of two of three Shamir shares through their approved out-of-band custody paths. Receipt validation failures are reported in `receipt_gate_failures` and never abort evidence capture. Do not reuse old restore-drill templates as current snapshot evidence. Do not put a snapshot, unseal share, token, password, or Secret response in Git or State Hub. ## Pre-reboot hold point 1. Announce the outage and stop new deployment, migration, credential-change, backup-restore, and security-engagement mutations on `railiance01`. 2. Confirm no temporary Whitehat or other expiring projection will cross the window. Clean it first if present. 3. Capture the read-only preflight result (keep the JSON as the pre-reboot evidence: boot id, failing-pod baseline, CNPG UIDs/PVC UIDs/systemIDs) and the audit-core chain-head attestation. 4. Confirm the latest `platform-pg` Barman backup is completed, WAL archiving is healthy, and the PVC is Bound. If the daily backup is stale, take and verify an on-demand backup before continuing. `state-hub-db` and `user-engine-pg` have no CNPG backup configured (nor do `net-kingdom-pg` and `target-revenue-pg`); their owners should take and verify a fresh backup through their own lane (for `user-engine-pg`, the rapp-user-engine backup CronJob) or record an explicit waiver before the window. 5. Resolve the stuck Terminating PVC `platform-pg-drill/platform-pg-drill-1` (owner: `rapp-postgres`, separately approved) or list it for the window with `--accept-terminating-pvc platform-pg-drill/platform-pg-drill-1`. 6. Create and escrow a fresh OpenBao Raft snapshot through the approved encrypted backup lane. Record only its evidence receipt and verification. Pass that receipt to the preflight with `--openbao-snapshot-evidence /path/to/receipt.json`; a merely existing, stale, placeholder, wrong-cluster, unencrypted, or on-host receipt is denied. The reviewed lane is `make openbao-reboot-snapshot` (offline tests: `make openbao-reboot-snapshot-test`). It runs `scripts/openbao-reboot-snapshot.py` as the single silent child of the contained `openbao-platform-admin-login` (via `scripts/openbao-attended-exec.py`): ```bash make openbao-reboot-snapshot \ SNAPSHOT_SCENARIO_ID=RPF-WP-0015-T03- \ SNAPSHOT_RECEIPT=/abs/new/receipt.json \ SNAPSHOT_CUSTODIAN_ESCROW_REF= \ SNAPSHOT_CUSTODIAN_RECIPIENT=age1... # only if that offline key differs from the offsite recipient SNAPSHOT_PRIVATE_DIR=/abs/0700/dir # optional; default is the session HOME (prefer tmpfs) ``` Recoverability requires a decryption key held outside OpenBao: the escrow reference is mandatory and the preflight rejects a receipt without one. If the custodian's offline key is the offsite recipient, the reference alone is recorded; otherwise the snapshot is also encrypted to `SNAPSHOT_CUSTODIAN_RECIPIENT` (age: any one recipient key decrypts). The receipt lists all recipients. Reading the offsite `AGE_PRIVATE_KEY` for the decrypt-stream proof is a founder decision (in memory only), recorded in `reviews/RPF-WP-0015-T03-founder-go-20261014.json#escrow_key_read` and cited by the receipt. Snapshots upload to the founder-created `openbao/` folder. In order it reads `bao status` (initialized, unsealed, Raft), saves the snapshot through the tunnel into a fresh 0700 temp dir outside Git, re-reads status (same `cluster_id`, index not behind), age-encrypts to the offsite lane recipient declared in `lib/railiance-backup-common.sh` (plus the custodian recipient, if given), proves the lane's escrow key derives that recipient and decrypt-streams the ciphertext to the plaintext sha256, then uploads ciphertext only to the governed Nextcloud lane with `If-None-Match: *`. The temp dir is always shredded. The receipt (0600, `encrypted_location_ref` `offhost-custody::`) is self-checked with the preflight's own validator before it is written; on any failure the file holds a value-safe `status: failed` record instead, which the preflight rejects. 7. Have both Shamir share custodians present. Prove provider-console access and SSH access (same-day `RAIL01-CONSOLE-*` receipt). Record the exact restart operator, abort operator, and the railiance-infra independent abort owner. 8. Re-run the preflight with the approval and attestation flags. The result must say `ready_for_live_execution: true`. Pause for the final operator go/no-go; this is the last reversible point. ## Reboot and recovery order 1. `railiance-infra` performs one ordinary host reboot and retains the provider console. Do not reinstall the OS or k3s. 2. Recover host network/SSH, then verify `k3s` active and the single node Ready. If SSH does not return, use the provider console; do not change firewall declarations as an ad hoc recovery. `railiance-infra` verifies: the boot id differs from the pre-reboot capture and exactly one boot occurred since it; UFW is active with only `OpenSSH` + `2224/tcp`, `6443`/`8472` not open; the `k3s-api-railiance01` tunnel answers; and `timedatectl` reports `NTPSynchronized=yes`. Do not unseal, run OIDC/ESO checks, or trust any receipt or backup age until time is synchronized. 3. `railiance-cluster` verifies these core platform workloads are Running and reconciled: CoreDNS, local-path-provisioner, traefik, metrics-server, cert-manager (+ cainjector/webhook), CNPG operator (`cnpg-system/cnpg-controller-manager`), external-secrets (+ webhook/cert controller), knative-serving (activator, controller, net-kourier-controller) and `kourier-system/3scale-kourier-gateway`, argocd, and the OpenBao pod (Running but sealed, so not Ready yet). Application readiness may still be false at this point. Images not already cached on the node cannot be pulled until in-cluster Forgejo (`forgejo.coulomb.social`) and `forgejo-db` are up; `ImagePullBackOff` on such pods is an expected lag, not a failure, until Forgejo is serving — escalate only if it persists after Forgejo is Ready. 4. `rapp-postgres` (with each cluster's owner) verifies for every CNPG cluster the same cluster UID, primary PVC UID, and `systemID` as the pre-reboot evidence, primary 1/1 Ready, `ContinuousArchiving=True` where configured, and no re-bootstrap. A changed systemID or PVC UID is a stop condition. 5. Expect OpenBao to be sealed. Two custodians unseal it through the attended, non-recorded share ceremony. Run `make openbao-status` and `make openbao-verify-post-unseal` after it reports initialized and unsealed. 6. Verify every ClusterSecretStore is `Valid` and every ExternalSecret is `SecretSynced` (not only `openbao-audit-core`/`openbao-audit-core-database` and the audit-core projections). If they remain in the known stale `Vault is sealed`/`InvalidProviderConfig` state, the platform owner may restart only Deployment `external-secrets/external-secrets`, then recheck reconciliation. 7. Do not restart audit-core to cure database timing. `/healthz` must remain the liveness boundary; wait for `/readyz` to recover naturally after DNS, PostgreSQL, OpenBao, and ESO are healthy. Record pod identity/restart count. 8. Run audit-core integrity verification against the pre-reboot chain head and one separately approved synthetic E2. The E2 runs through the reviewed sender lane in its own attended login: `make audit-t02-attended-window` with `AUDIT_T02_MODE=e2` and an approval receipt whose `task_id` is `RAILIANCE-WP-0024-T03`. It projects one synthetic sender and sends one event twice through the driver's `expect-recovered` phase: the first answer must be a fresh `accepted` (HTTP 200/202), the second `duplicate` for the same fixture (accepted exactly once); it touches no lease. The harness refuses unless both hold and the pod UID and restart count are unchanged. The driver owns no server-side object; the synthetic event stays permanently in the append-only chain under its trial tenant, and only the sender identity is removed. Confirm no chain break, the prior marker remains readable, and the new event is accepted exactly once. Window `rpf-wp-0015-t03-window-20261014-01` does not run this E2. Founder decision `rpf-wp-0015-t03-founder-go-20261014-01` defers it to its own attended window and keeps tonight to the chain check. Do not mark T03 as proving the synthetic E2 path until that follow-up window runs. 9. Re-run the read-only preflight in post-reboot mode: ```bash python3 scripts/audit-core-recovery-preflight.py node-reboot \ --pre-reboot-evidence /path/to/pre-reboot-preflight.json ``` `reboot_comparison.passed` must be true: boot id changed with exactly one boot since the pre-reboot capture, no CNPG identity change, and no failing pod outside the pre-reboot baseline (`failing_pod_diff.new_failing` empty). Then end the mutation freeze and notify all owners with value-safe evidence. ## Stop, escalation, and recovery boundaries - Before reboot: abort for missing approval, owner, console receipt, unseal quorum, fresh PostgreSQL backup, verified OpenBao snapshot, healthy WAL/PVC, accepted ops-warden revision, new scenario id/fresh decision, NTP sync, or any overlapping live mutation. `railiance-infra` may abort independently. - Attended login: if the login emits any unexpected credential output (token, secret id, password, key, or unrecognised value-shaped text), stop immediately: self-revoke the issued credential, abort the window, and never retry under the same scenario or decision. A new scenario id and a fresh human decision are required before any further attempt. - After reboot: retain the outage and escalate for a missing host, more than one boot, firewall drift, unsynchronized time, node/PVC/systemID identity change, CNPG re-bootstrap, new failing pods outside the baseline (other than the expected pre-Forgejo image-pull lag), OpenBao initialization loss, failed unseal, audit chain break, or any evidence of data loss. - A reboot cannot be rolled back. Never run `k3s-uninstall`, initialize OpenBao, recover PostgreSQL in place, delete a PVC, or weaken network policy as a shortcut. Host recovery belongs to `railiance-infra`; Kubernetes recovery to `railiance-cluster`; OpenBao snapshot restoration to `railiance-platform`; PostgreSQL restoration always creates a new CNPG Cluster under `rapp-postgres`' restore procedure. ## Evidence contract Allowed: approval/window ids, scenario and decision ids, receipt ids/paths, owner acknowledgements, CNPG cluster UIDs and systemIDs, UFW rule targets, NTP sync boolean, failing pod names, timestamps, host boot id or uptime, node/cluster/PVC names and UIDs, image digests, backup name/time/age, OpenBao initialized/sealed booleans and version, resource readiness conditions, Secret resource versions and key names, HTTP status sequence, audit chain-head hash, event ids/hashes for declared synthetic fixtures, pod UIDs/restart counts, and final notifications. Forbidden: unseal/recovery shares, OpenBao tokens or response bodies, database credentials/usernames/DSNs, Secret data, sender bearer values, private keys, provider-console credentials, request payloads, or decrypted snapshots.