railiance-platform/docs/railiance01-coordinated-reboot.md

Coordinated railiance01 reboot and recovery

This is the owner-review checklist for RAILIANCE-WP-0024-T03. A reboot is a single-node platform outage, not a Kubernetes rolling operation and not an HA claim. This document does not authorize a reboot.

Responsibility map

Phase	Accountable owner	Required evidence
Window, notifications, abort authority	operator + railiance-platform	approval/window id; named driver and abort operator
Host access, provider console, reboot	railiance-infra	SSH and independent provider-console access; host returns
k3s/node/DNS/operators	railiance-cluster	node Ready; core platform pods reconciled
OpenBao snapshot, Shamir unseal, post-unseal checks	railiance-platform	current Raft snapshot receipt; 2-of-3 quorum attestation; unsealed verification
platform-pg backup, WAL, PVC, primary	rapp-postgres	fresh completed Barman backup; continuous archiving; 1/1 primary
ESO stores and projections	railiance-platform	stores Valid; ExternalSecrets SecretSynced after unseal
Audit custody and synthetic verification	audit-core	health/readiness/integrity and bounded synthetic E2 result

The last observed reboot on 2026-08-16 showed the expected dependency order: the host and k3s returned, audit-core was unready for about 40 seconds while DNS/PostgreSQL recovered, and OpenBao returned sealed. Existing mounted Secrets remained, but new leases could not refresh until unseal.

Fail-closed preflight

Run the value-safe, read-only gate:

python3 scripts/audit-core-recovery-preflight.py node-reboot

The automated checks require one Ready node, active k3s, healthy 1/1 platform-pg, continuous archiving, a successful backup no older than 26 hours, unsealed OpenBao, Valid audit-core stores, SecretSynced projections, and the reviewed 1/1 Ready audit-core image. The following are human gates and make ready_for_live_execution remain false until supplied:

• approved window id and acknowledgements from all five owners;
• a named attended driver/abort operator;
• independent provider-console access, not SSH alone;
• a current non-secret OpenBao Raft snapshot receipt that validates against docs/openbao-reboot-snapshot-receipt.example.json, the live cluster id and Raft index, and the default 24-hour freshness bound;
• explicit availability of two of three Shamir shares through their approved out-of-band custody paths.

Do not reuse old restore-drill templates as current snapshot evidence. Do not put a snapshot, unseal share, token, password, or Secret response in Git or State Hub.

Pre-reboot hold point

1. Announce the outage and stop new deployment, migration, credential-change, backup-restore, and security-engagement mutations on railiance01.
2. Confirm no temporary Whitehat or other expiring projection will cross the window. Clean it first if present.
3. Capture the read-only preflight result and audit-core chain-head attestation.
4. Confirm the latest platform-pg Barman backup is completed, WAL archiving is healthy, and the PVC is Bound. If the daily backup is stale, take and verify an on-demand backup before continuing.
5. Create and escrow a fresh OpenBao Raft snapshot through the approved encrypted backup lane. Record only its evidence receipt and verification. Pass that receipt to the preflight with --openbao-snapshot-evidence /path/to/receipt.json; a merely existing, stale, placeholder, wrong-cluster, unencrypted, or on-host receipt is denied.
6. Have both Shamir share custodians present. Prove provider-console access and SSH access. Record the exact restart and abort operators.
7. Re-run the preflight with the approval and attestation flags. The result must say ready_for_live_execution: true. Pause for the final operator go/no-go; this is the last reversible point.

Reboot and recovery order

1. railiance-infra performs one ordinary host reboot and retains the provider console. Do not reinstall the OS or k3s.
2. Recover host network/SSH, then verify k3s active and the single node Ready. If SSH does not return, use the provider console; do not change firewall declarations as an ad hoc recovery.
3. Verify CoreDNS, CNPG operator, External Secrets operator, and the OpenBao pod are running. Application readiness may still be false at this point.
4. rapp-postgres verifies the same platform-pg PVC and cluster identity, primary 1/1 Ready, ContinuousArchiving=True, and no re-bootstrap.
5. Expect OpenBao to be sealed. Two custodians unseal it through the attended, non-recorded share ceremony. Run make openbao-status and make openbao-verify-post-unseal after it reports initialized and unsealed.
6. Verify openbao-audit-core and openbao-audit-core-database are Valid and the audit-core sender/database ExternalSecrets are SecretSynced. If they remain in the known stale Vault is sealed/InvalidProviderConfig state, the platform owner may restart only Deployment external-secrets/external-secrets, then recheck reconciliation.
7. Do not restart audit-core to cure database timing. /healthz must remain the liveness boundary; wait for /readyz to recover naturally after DNS, PostgreSQL, OpenBao, and ESO are healthy. Record pod identity/restart count.
8. Run audit-core integrity verification against the pre-reboot chain head and one separately approved synthetic E2. Confirm no chain break, the prior marker remains readable, and the new event is accepted exactly once.
9. Re-run the read-only preflight, end the mutation freeze, and notify all owners with value-safe evidence.

Stop, escalation, and recovery boundaries

• Before reboot: abort for missing approval, owner, console access, unseal quorum, fresh PostgreSQL backup, verified OpenBao snapshot, healthy WAL/PVC, or any overlapping live mutation.
• After reboot: retain the outage and escalate for a missing host, node/PVC identity change, CNPG re-bootstrap, OpenBao initialization loss, failed unseal, audit chain break, or any evidence of data loss.
• A reboot cannot be rolled back. Never run k3s-uninstall, initialize OpenBao, recover PostgreSQL in place, delete a PVC, or weaken network policy as a shortcut. Host recovery belongs to railiance-infra; Kubernetes recovery to railiance-cluster; OpenBao snapshot restoration to railiance-platform; PostgreSQL restoration always creates a new CNPG Cluster under rapp-postgres' restore procedure.

Evidence contract

Allowed: approval/window ids, owner acknowledgements, timestamps, host boot id or uptime, node/cluster/PVC names and UIDs, image digests, backup name/time/age, OpenBao initialized/sealed booleans and version, resource readiness conditions, Secret resource versions and key names, HTTP status sequence, audit chain-head hash, event ids/hashes for declared synthetic fixtures, pod UIDs/restart counts, and final notifications.

Forbidden: unseal/recovery shares, OpenBao tokens or response bodies, database credentials/usernames/DSNs, Secret data, sender bearer values, private keys, provider-console credentials, request payloads, or decrypted snapshots.