railiance-platform/docs/railiance01-coordinated-reboot.md

Coordinated railiance01 reboot and recovery

This is the owner-review checklist for RAILIANCE-WP-0024-T03. A reboot is a single-node platform outage, not a Kubernetes rolling operation and not an HA claim. This document does not authorize a reboot.

Responsibility map

Phase	Accountable owner	Required evidence
Window, notifications, go/no-go	operator + railiance-platform	approval/window id; new scenario id; fresh human decision; named driver
Independent abort authority (RPF-WP-0017)	railiance-infra	named abort owner acknowledgement; may stop the window at any step without a platform vote
Attended login containment	railiance-platform	reviews/RPF-WP-0017-ops-warden-reacceptance-2026-10-14.json accepting the exact ops-warden revision
Host access, provider console, reboot, firewall, time	railiance-infra	SSH; same-day RAIL01-CONSOLE-* receipt; host returns; UFW/NTP/tunnel checks
k3s/node/DNS/operators	railiance-cluster	node Ready; named core platform workloads Ready; failing-pod diff
OpenBao snapshot, Shamir unseal, post-unseal checks	railiance-platform	current Raft snapshot receipt; 2-of-3 quorum attestation; unsealed verification
All CNPG clusters (platform-pg backup/WAL)	rapp-postgres (+ owning app for each cluster)	per-cluster UID, primary PVC UID, systemID, 1/1 primary; fresh platform-pg Barman backup
ESO stores and projections	railiance-platform	stores Valid; ExternalSecrets SecretSynced after unseal
Audit custody and synthetic verification	audit-core	health/readiness/integrity and bounded synthetic E2 result

The last observed reboot on 2026-08-16 showed the expected dependency order: the host and k3s returned, audit-core was unready for about 40 seconds while DNS/PostgreSQL recovered, and OpenBao returned sealed. Existing mounted Secrets remained, but new leases could not refresh until unseal.

Fail-closed preflight

Run the value-safe, read-only gate:

python3 scripts/audit-core-recovery-preflight.py node-reboot

The automated checks (automated_checks in the output) require:

• one Ready node, active k3s, and the boot id recorded;
• timedatectl NTPSynchronized=yes (receipt/backup ages are untrusted otherwise);
• UFW active with exactly the declared allow set — OpenSSH and Nydus 2224/tcp (v4/v6) — and 6443/8472 not open (railiance-infra RAIL-HO-WP-0009 / Goss declaration);
• the workstation k3s-api-railiance01 tunnel answering (bridge check);
• every named core platform workload Ready, plus every other Deployment and StatefulSet in their namespaces: CoreDNS, local-path-provisioner, traefik, metrics-server, cert-manager, the CNPG operator, external-secrets, knative-serving (activator, controller, net-kourier-controller) and the kourier gateway, argocd, and openbao;
• every CNPG cluster, discovered live (today apps-pg, forgejo-db, net-kingdom-pg, platform-pg, platform-pg-2, state-hub-db, target-revenue-pg, user-engine-pg; a vanished expected cluster fails), 1/1 Ready, current primary on a healthy PVC, ContinuousArchiving=True where backup is configured, with cluster UID, primary PVC UID, and PostgreSQL systemID (from pg_controldata, value-safe) recorded;
• a completed platform-pg backup no older than 26 hours;
• every ClusterSecretStore Valid and every ExternalSecret SecretSynced;
• unsealed OpenBao and the reviewed audit-core digest in both the Deployment spec and the running pod (imageID), 1/1 Ready;
• before the reboot, every Terminating PVC resolved or explicitly listed with --accept-terminating-pvc.

The output also records the all-namespace failing-pod baseline (reboot_evidence.failing_pods). On 2026-10-14 it held six pre-existing pods: three coulomb/ihp-railiance-probe-*, forgejo/forgejo-gitea-*, forgejo/forgejo-runner-*, and telemetry/telemetry-smtp-proof-20260928-*. Keep the pre-reboot JSON; it is the comparison input after the reboot.

The following are human gates and make ready_for_live_execution remain false until supplied:

• approved window id and acknowledgements from all five owners;
• a new scenario id (--scenario-id) and a fresh human GO decision (--human-decision-id). Consumed decisions 449a697a-… and 85724c0c-… and scenario WARDEN-WP-0027-T02-DRILL-20260822-01 are not reusable and are denied by the preflight;
• the RPF-WP-0017 re-acceptance receipt reviews/RPF-WP-0017-ops-warden-reacceptance-2026-10-14.json recording acceptance of the exact ops-warden revision used for the attended login (11b2a9d… was rejected; 6e14300… accepted on 2026-10-14). The accepted revision must be pushed to ops-warden origin/main and installed with uv tool install --reinstall /home/worsch/ops-warden from that clean checkout before the window. The preflight compares every installed warden module byte-for-byte with the accepted revision and closes this gate on any mismatch; rerun it at the hold point immediately before the attended login and abort if it no longer matches (the 2026-09-01 WARDEN-WP-0036-T02 operation ran on unaccepted code — do not repeat that);
• a named attended driver/abort operator and the railiance-infra independent abort owner acknowledgement (--infra-abort-owner-ack);
• independent provider-console access, not SSH alone: a value-safe RAIL01-CONSOLE-<YYYYMMDDTHHMMSSZ> receipt recorded by railiance-infra under its docs/evidence/railiance01-console-access/ (procedure: railiance-infra docs/railiance01-console-access.md), passed with --provider-console-receipt. The preflight accepts it only when it is dated the window day, names the approved window id, and has console_shows_host and independent_of_guest_network true. No out-of-band console is established today, so T03 cannot pass this gate until railiance-infra RAIL-HO-WP-0011-T09 is done;
• a current non-secret OpenBao Raft snapshot receipt that validates against docs/openbao-reboot-snapshot-receipt.example.json, the live cluster id and Raft index, and the default 24-hour freshness bound;
• explicit availability of two of three Shamir shares through their approved out-of-band custody paths.

Receipt validation failures are reported in receipt_gate_failures and never abort evidence capture. Do not reuse old restore-drill templates as current snapshot evidence. Do not put a snapshot, unseal share, token, password, or Secret response in Git or State Hub.

Pre-reboot hold point

1. Announce the outage and stop new deployment, migration, credential-change, backup-restore, and security-engagement mutations on railiance01.

2. Confirm no temporary Whitehat or other expiring projection will cross the window. Clean it first if present.

3. Capture the read-only preflight result (keep the JSON as the pre-reboot evidence: boot id, failing-pod baseline, CNPG UIDs/PVC UIDs/systemIDs) and the audit-core chain-head attestation.

4. Confirm the latest platform-pg Barman backup is completed, WAL archiving is healthy, and the PVC is Bound. If the daily backup is stale, take and verify an on-demand backup before continuing. state-hub-db and user-engine-pg have no CNPG backup configured (nor do net-kingdom-pg and target-revenue-pg); their owners should take and verify a fresh backup through their own lane (for user-engine-pg, the rapp-user-engine backup CronJob) or record an explicit waiver before the window.

5. Resolve the stuck Terminating PVC platform-pg-drill/platform-pg-drill-1 (owner: rapp-postgres, separately approved) or list it for the window with --accept-terminating-pvc platform-pg-drill/platform-pg-drill-1.

6. Create and escrow a fresh OpenBao Raft snapshot through the approved encrypted backup lane. Record only its evidence receipt and verification. Pass that receipt to the preflight with --openbao-snapshot-evidence /path/to/receipt.json; a merely existing, stale, placeholder, wrong-cluster, unencrypted, or on-host receipt is denied.

   The reviewed lane is make openbao-reboot-snapshot (offline tests: make openbao-reboot-snapshot-test). It runs scripts/openbao-reboot-snapshot.py as the single silent child of the contained openbao-platform-admin-login (via scripts/openbao-attended-exec.py):

   make openbao-reboot-snapshot \
     SNAPSHOT_SCENARIO_ID=RPF-WP-0015-T03-<scenario> \
     SNAPSHOT_RECEIPT=/abs/new/receipt.json \
     SNAPSHOT_CUSTODIAN_ESCROW_REF=<where a decryption key is held outside OpenBao> \
     SNAPSHOT_CUSTODIAN_RECIPIENT=age1...   # only if that offline key differs from the offsite recipient
     SNAPSHOT_PRIVATE_DIR=/abs/0700/dir   # optional; default is the session HOME (prefer tmpfs)
   

   Recoverability requires a decryption key held outside OpenBao: the escrow reference is mandatory and the preflight rejects a receipt without one. If the custodian's offline key is the offsite recipient, the reference alone is recorded; otherwise the snapshot is also encrypted to SNAPSHOT_CUSTODIAN_RECIPIENT (age: any one recipient key decrypts). The receipt lists all recipients. Reading the offsite AGE_PRIVATE_KEY for the decrypt-stream proof is a founder decision (in memory only), recorded in reviews/RPF-WP-0015-T03-founder-go-20261014.json#escrow_key_read and cited by the receipt. Snapshots upload to the founder-created openbao/ folder.

   In order it reads bao status (initialized, unsealed, Raft), saves the snapshot through the tunnel into a fresh 0700 temp dir outside Git, re-reads status (same cluster_id, index not behind), age-encrypts to the offsite lane recipient declared in lib/railiance-backup-common.sh (plus the custodian recipient, if given), proves the lane's escrow key derives that recipient and decrypt-streams the ciphertext to the plaintext sha256, then uploads ciphertext only to the governed Nextcloud lane with If-None-Match: *. The temp dir is always shredded. The receipt (0600, encrypted_location_ref offhost-custody:<scenario>:<ts>) is self-checked with the preflight's own validator before it is written; on any failure the file holds a value-safe status: failed record instead, which the preflight rejects.

7. Have both Shamir share custodians present. Prove provider-console access and SSH access (same-day RAIL01-CONSOLE-* receipt). Record the exact restart operator, abort operator, and the railiance-infra independent abort owner.

8. Re-run the preflight with the approval and attestation flags. The result must say ready_for_live_execution: true. Pause for the final operator go/no-go; this is the last reversible point.

Reboot and recovery order

1. railiance-infra performs one ordinary host reboot and retains the provider console. Do not reinstall the OS or k3s.

2. Recover host network/SSH, then verify k3s active and the single node Ready. If SSH does not return, use the provider console; do not change firewall declarations as an ad hoc recovery. railiance-infra verifies: the boot id differs from the pre-reboot capture and exactly one boot occurred since it; UFW is active with only OpenSSH + 2224/tcp, 6443/8472 not open; the k3s-api-railiance01 tunnel answers; and timedatectl reports NTPSynchronized=yes. Do not unseal, run OIDC/ESO checks, or trust any receipt or backup age until time is synchronized.

3. railiance-cluster verifies these core platform workloads are Running and reconciled: CoreDNS, local-path-provisioner, traefik, metrics-server, cert-manager (+ cainjector/webhook), CNPG operator (cnpg-system/cnpg-controller-manager), external-secrets (+ webhook/cert controller), knative-serving (activator, controller, net-kourier-controller) and kourier-system/3scale-kourier-gateway, argocd, and the OpenBao pod (Running but sealed, so not Ready yet). Application readiness may still be false at this point. Images not already cached on the node cannot be pulled until in-cluster Forgejo (forgejo.coulomb.social) and forgejo-db are up; ImagePullBackOff on such pods is an expected lag, not a failure, until Forgejo is serving — escalate only if it persists after Forgejo is Ready.

4. rapp-postgres (with each cluster's owner) verifies for every CNPG cluster the same cluster UID, primary PVC UID, and systemID as the pre-reboot evidence, primary 1/1 Ready, ContinuousArchiving=True where configured, and no re-bootstrap. A changed systemID or PVC UID is a stop condition.

5. Expect OpenBao to be sealed. Two custodians unseal it through the attended, non-recorded share ceremony. Run make openbao-status and make openbao-verify-post-unseal after it reports initialized and unsealed.

6. Verify every ClusterSecretStore is Valid and every ExternalSecret is SecretSynced (not only openbao-audit-core/openbao-audit-core-database and the audit-core projections). If they remain in the known stale Vault is sealed/InvalidProviderConfig state, the platform owner may restart only Deployment external-secrets/external-secrets, then recheck reconciliation.

7. Do not restart audit-core to cure database timing. /healthz must remain the liveness boundary; wait for /readyz to recover naturally after DNS, PostgreSQL, OpenBao, and ESO are healthy. Record pod identity/restart count.

8. Run audit-core integrity verification against the pre-reboot chain head and one separately approved synthetic E2. The E2 runs through the reviewed sender lane in its own attended login: make audit-t02-attended-window with AUDIT_T02_MODE=e2 and an approval receipt whose task_id is RAILIANCE-WP-0024-T03. It projects one synthetic sender and sends one event twice through the driver's expect-recovered phase: the first answer must be a fresh accepted (HTTP 200/202), the second duplicate for the same fixture (accepted exactly once); it touches no lease. The harness refuses unless both hold and the pod UID and restart count are unchanged. The driver owns no server-side object; the synthetic event stays permanently in the append-only chain under its trial tenant, and only the sender identity is removed. Confirm no chain break, the prior marker remains readable, and the new event is accepted exactly once. Window rpf-wp-0015-t03-window-20261014-01 does not run this E2. Founder decision rpf-wp-0015-t03-founder-go-20261014-01 defers it to its own attended window and keeps tonight to the chain check. Do not mark T03 as proving the synthetic E2 path until that follow-up window runs.

9. Re-run the read-only preflight in post-reboot mode:

   python3 scripts/audit-core-recovery-preflight.py node-reboot \
     --pre-reboot-evidence /path/to/pre-reboot-preflight.json
   

   reboot_comparison.passed must be true: boot id changed with exactly one boot since the pre-reboot capture, no CNPG identity change, and no failing pod outside the pre-reboot baseline (failing_pod_diff.new_failing empty). Then end the mutation freeze and notify all owners with value-safe evidence.

Stop, escalation, and recovery boundaries

• Before reboot: abort for missing approval, owner, console receipt, unseal quorum, fresh PostgreSQL backup, verified OpenBao snapshot, healthy WAL/PVC, accepted ops-warden revision, new scenario id/fresh decision, NTP sync, or any overlapping live mutation. railiance-infra may abort independently.
• Attended login: if the login emits any unexpected credential output (token, secret id, password, key, or unrecognised value-shaped text), stop immediately: self-revoke the issued credential, abort the window, and never retry under the same scenario or decision. A new scenario id and a fresh human decision are required before any further attempt.
• After reboot: retain the outage and escalate for a missing host, more than one boot, firewall drift, unsynchronized time, node/PVC/systemID identity change, CNPG re-bootstrap, new failing pods outside the baseline (other than the expected pre-Forgejo image-pull lag), OpenBao initialization loss, failed unseal, audit chain break, or any evidence of data loss.
• A reboot cannot be rolled back. Never run k3s-uninstall, initialize OpenBao, recover PostgreSQL in place, delete a PVC, or weaken network policy as a shortcut. Host recovery belongs to railiance-infra; Kubernetes recovery to railiance-cluster; OpenBao snapshot restoration to railiance-platform; PostgreSQL restoration always creates a new CNPG Cluster under rapp-postgres' restore procedure.

Evidence contract

Allowed: approval/window ids, scenario and decision ids, receipt ids/paths, owner acknowledgements, CNPG cluster UIDs and systemIDs, UFW rule targets, NTP sync boolean, failing pod names, timestamps, host boot id or uptime, node/cluster/PVC names and UIDs, image digests, backup name/time/age, OpenBao initialized/sealed booleans and version, resource readiness conditions, Secret resource versions and key names, HTTP status sequence, audit chain-head hash, event ids/hashes for declared synthetic fixtures, pod UIDs/restart counts, and final notifications.

Forbidden: unseal/recovery shares, OpenBao tokens or response bodies, database credentials/usernames/DSNs, Secret data, sender bearer values, private keys, provider-console credentials, request payloads, or decrypted snapshots.