railiance-platform/docs/railiance01-coordinated-reboot.md

# Coordinated `railiance01` reboot and recovery

This is the owner-review checklist for `RAILIANCE-WP-0024-T03`. A reboot is a
single-node platform outage, not a Kubernetes rolling operation and not an HA
claim. This document does not authorize a reboot.

## Responsibility map

| Phase | Accountable owner | Required evidence |
| --- | --- | --- |
| Window, notifications, abort authority | operator + `railiance-platform` | approval/window id; named driver and abort operator |
| Host access, provider console, reboot | `railiance-infra` | SSH and independent provider-console access; host returns |
| k3s/node/DNS/operators | `railiance-cluster` | node Ready; core platform pods reconciled |
| OpenBao snapshot, Shamir unseal, post-unseal checks | `railiance-platform` | current Raft snapshot receipt; 2-of-3 quorum attestation; unsealed verification |
| `platform-pg` backup, WAL, PVC, primary | `rapp-postgres` | fresh completed Barman backup; continuous archiving; 1/1 primary |
| ESO stores and projections | `railiance-platform` | stores Valid; ExternalSecrets SecretSynced after unseal |
| Audit custody and synthetic verification | `audit-core` | health/readiness/integrity and bounded synthetic E2 result |

The last observed reboot on 2026-08-16 showed the expected dependency order:
the host and k3s returned, audit-core was unready for about 40 seconds while
DNS/PostgreSQL recovered, and OpenBao returned sealed. Existing mounted
Secrets remained, but new leases could not refresh until unseal.

## Fail-closed preflight

Run the value-safe, read-only gate:

```bash
python3 scripts/audit-core-recovery-preflight.py node-reboot
```

The automated checks require one Ready node, active k3s, healthy 1/1
`platform-pg`, continuous archiving, a successful backup no older than 26
hours, unsealed OpenBao, Valid audit-core stores, SecretSynced projections, and
the reviewed 1/1 Ready audit-core image. The following are human gates and make
`ready_for_live_execution` remain false until supplied:

- approved window id and acknowledgements from all five owners;
- a named attended driver/abort operator;
- independent provider-console access, not SSH alone;
- a current non-secret OpenBao Raft snapshot receipt that validates against
  `docs/openbao-reboot-snapshot-receipt.example.json`, the live cluster id and
  Raft index, and the default 24-hour freshness bound;
- explicit availability of two of three Shamir shares through their approved
  out-of-band custody paths.

Do not reuse old restore-drill templates as current snapshot evidence. Do not
put a snapshot, unseal share, token, password, or Secret response in Git or
State Hub.

## Pre-reboot hold point

1. Announce the outage and stop new deployment, migration, credential-change,
   backup-restore, and security-engagement mutations on `railiance01`.
2. Confirm no temporary Whitehat or other expiring projection will cross the
   window. Clean it first if present.
3. Capture the read-only preflight result and audit-core chain-head attestation.
4. Confirm the latest `platform-pg` Barman backup is completed, WAL archiving
   is healthy, and the PVC is Bound. If the daily backup is stale, take and
   verify an on-demand backup before continuing.
5. Create and escrow a fresh OpenBao Raft snapshot through the approved
   encrypted backup lane. Record only its evidence receipt and verification.
   Pass that receipt to the preflight with
   `--openbao-snapshot-evidence /path/to/receipt.json`; a merely existing,
   stale, placeholder, wrong-cluster, unencrypted, or on-host receipt is denied.
6. Have both Shamir share custodians present. Prove provider-console access and
   SSH access. Record the exact restart and abort operators.
7. Re-run the preflight with the approval and attestation flags. The result
   must say `ready_for_live_execution: true`. Pause for the final operator
   go/no-go; this is the last reversible point.

## Reboot and recovery order

1. `railiance-infra` performs one ordinary host reboot and retains the provider
   console. Do not reinstall the OS or k3s.
2. Recover host network/SSH, then verify `k3s` active and the single node Ready.
   If SSH does not return, use the provider console; do not change firewall
   declarations as an ad hoc recovery.
3. Verify CoreDNS, CNPG operator, External Secrets operator, and the OpenBao pod
   are running. Application readiness may still be false at this point.
4. `rapp-postgres` verifies the same `platform-pg` PVC and cluster identity,
   primary 1/1 Ready, `ContinuousArchiving=True`, and no re-bootstrap.
5. Expect OpenBao to be sealed. Two custodians unseal it through the attended,
   non-recorded share ceremony. Run `make openbao-status` and
   `make openbao-verify-post-unseal` after it reports initialized and unsealed.
6. Verify `openbao-audit-core` and `openbao-audit-core-database` are Valid and
   the audit-core sender/database ExternalSecrets are SecretSynced. If they
   remain in the known stale `Vault is sealed`/`InvalidProviderConfig` state,
   the platform owner may restart only Deployment
   `external-secrets/external-secrets`, then recheck reconciliation.
7. Do not restart audit-core to cure database timing. `/healthz` must remain the
   liveness boundary; wait for `/readyz` to recover naturally after DNS,
   PostgreSQL, OpenBao, and ESO are healthy. Record pod identity/restart count.
8. Run audit-core integrity verification against the pre-reboot chain head and
   one separately approved synthetic E2. Confirm no chain break, the prior
   marker remains readable, and the new event is accepted exactly once.
9. Re-run the read-only preflight, end the mutation freeze, and notify all
   owners with value-safe evidence.

## Stop, escalation, and recovery boundaries

- Before reboot: abort for missing approval, owner, console access, unseal
  quorum, fresh PostgreSQL backup, verified OpenBao snapshot, healthy WAL/PVC,
  or any overlapping live mutation.
- After reboot: retain the outage and escalate for a missing host, node/PVC
  identity change, CNPG re-bootstrap, OpenBao initialization loss, failed
  unseal, audit chain break, or any evidence of data loss.
- A reboot cannot be rolled back. Never run `k3s-uninstall`, initialize OpenBao,
  recover PostgreSQL in place, delete a PVC, or weaken network policy as a
  shortcut. Host recovery belongs to `railiance-infra`; Kubernetes recovery to
  `railiance-cluster`; OpenBao snapshot restoration to `railiance-platform`;
  PostgreSQL restoration always creates a new CNPG Cluster under
  `rapp-postgres`' restore procedure.

## Evidence contract

Allowed: approval/window ids, owner acknowledgements, timestamps, host boot id
or uptime, node/cluster/PVC names and UIDs, image digests, backup name/time/age,
OpenBao initialized/sealed booleans and version, resource readiness conditions,
Secret resource versions and key names, HTTP status sequence, audit chain-head
hash, event ids/hashes for declared synthetic fixtures, pod UIDs/restart counts,
and final notifications.

Forbidden: unseal/recovery shares, OpenBao tokens or response bodies, database
credentials/usernames/DSNs, Secret data, sender bearer values, private keys,
provider-console credentials, request payloads, or decrypted snapshots.