railiance-platform/docs/railiance01-coordinated-reboot.md

# Coordinated `railiance01` reboot and recovery

This is the owner-review checklist for `RAILIANCE-WP-0024-T03`. A reboot is a
single-node platform outage, not a Kubernetes rolling operation and not an HA
claim. This document does not authorize a reboot.

## Responsibility map

| Phase | Accountable owner | Required evidence |
| --- | --- | --- |
| Window, notifications, go/no-go | operator + `railiance-platform` | approval/window id; new scenario id; fresh human decision; named driver |
| Independent abort authority (RPF-WP-0017) | `railiance-infra` | named abort owner acknowledgement; may stop the window at any step without a platform vote |
| Attended login containment | `railiance-platform` | `reviews/RPF-WP-0017-ops-warden-reacceptance-2026-10-14.json` accepting the exact ops-warden revision |
| Host access, provider console, reboot, firewall, time | `railiance-infra` | SSH; same-day `RAIL01-CONSOLE-*` receipt; host returns; UFW/NTP/tunnel checks |
| k3s/node/DNS/operators | `railiance-cluster` | node Ready; named core platform workloads Ready; failing-pod diff |
| OpenBao snapshot, Shamir unseal, post-unseal checks | `railiance-platform` | current Raft snapshot receipt; 2-of-3 quorum attestation; unsealed verification |
| All CNPG clusters (`platform-pg` backup/WAL) | `rapp-postgres` (+ owning app for each cluster) | per-cluster UID, primary PVC UID, systemID, 1/1 primary; fresh `platform-pg` Barman backup |
| ESO stores and projections | `railiance-platform` | stores Valid; ExternalSecrets SecretSynced after unseal |
| Audit custody and synthetic verification | `audit-core` | health/readiness/integrity and bounded synthetic E2 result |

The last observed reboot on 2026-08-16 showed the expected dependency order:
the host and k3s returned, audit-core was unready for about 40 seconds while
DNS/PostgreSQL recovered, and OpenBao returned sealed. Existing mounted
Secrets remained, but new leases could not refresh until unseal.

## Fail-closed preflight

Run the value-safe, read-only gate:

```bash
python3 scripts/audit-core-recovery-preflight.py node-reboot
```

The automated checks (`automated_checks` in the output) require:

- one Ready node, active k3s, and the boot id recorded;
- `timedatectl` `NTPSynchronized=yes` (receipt/backup ages are untrusted otherwise);
- UFW active with exactly the declared allow set — `OpenSSH` and Nydus
  `2224/tcp` (v4/v6) — and `6443`/`8472` not open (railiance-infra
  RAIL-HO-WP-0009 / Goss declaration);
- the workstation `k3s-api-railiance01` tunnel answering (`bridge check`);
- every named core platform workload Ready, plus every other Deployment and
  StatefulSet in their namespaces: CoreDNS, local-path-provisioner, traefik,
  metrics-server, cert-manager, the CNPG operator, external-secrets,
  knative-serving (activator, controller, net-kourier-controller) and the
  kourier gateway, argocd, and openbao;
- every CNPG cluster, discovered live (today `apps-pg`, `forgejo-db`,
  `net-kingdom-pg`, `platform-pg`, `platform-pg-2`, `state-hub-db`,
  `target-revenue-pg`, `user-engine-pg`; a vanished expected cluster fails),
  1/1 Ready, current primary on a healthy PVC, `ContinuousArchiving=True` where
  backup is configured, with cluster UID, primary PVC UID, and PostgreSQL
  `systemID` (from `pg_controldata`, value-safe) recorded;
- a completed `platform-pg` backup no older than 26 hours;
- every ClusterSecretStore `Valid` and every ExternalSecret `SecretSynced`;
- unsealed OpenBao and the reviewed audit-core digest in both the Deployment
  spec and the running pod (`imageID`), 1/1 Ready;
- before the reboot, every Terminating PVC resolved or explicitly listed with
  `--accept-terminating-pvc`.

The output also records the all-namespace failing-pod baseline
(`reboot_evidence.failing_pods`). On 2026-10-14 it held six pre-existing pods:
three `coulomb/ihp-railiance-probe-*`, `forgejo/forgejo-gitea-*`,
`forgejo/forgejo-runner-*`, and `telemetry/telemetry-smtp-proof-20260928-*`.
Keep the pre-reboot JSON; it is the comparison input after the reboot.

The following are human gates and make `ready_for_live_execution` remain false
until supplied:

- approved window id and acknowledgements from all five owners;
- a new scenario id (`--scenario-id`) and a fresh human GO decision
  (`--human-decision-id`). Consumed decisions `449a697a-…` and `85724c0c-…` and
  scenario `WARDEN-WP-0027-T02-DRILL-20260822-01` are not reusable and are
  denied by the preflight;
- the RPF-WP-0017 re-acceptance receipt
  `reviews/RPF-WP-0017-ops-warden-reacceptance-2026-10-14.json` recording
  acceptance of the exact ops-warden revision used for the attended login
  (`11b2a9d…` was rejected; `6e14300…` accepted on 2026-10-14). The accepted
  revision must be pushed to ops-warden `origin/main` and installed with
  `uv tool install --reinstall /home/worsch/ops-warden` from that clean checkout
  before the window. The preflight compares every installed `warden` module
  byte-for-byte with the accepted revision and closes this gate on any mismatch;
  rerun it at the hold point immediately before the attended login and abort if
  it no longer matches (the 2026-09-01 WARDEN-WP-0036-T02 operation ran on
  unaccepted code — do not repeat that);
- a named attended driver/abort operator and the `railiance-infra`
  independent abort owner acknowledgement (`--infra-abort-owner-ack`);
- independent provider-console access, not SSH alone: a value-safe
  `RAIL01-CONSOLE-<YYYYMMDDTHHMMSSZ>` receipt recorded by railiance-infra under
  its `docs/evidence/railiance01-console-access/` (procedure:
  railiance-infra `docs/railiance01-console-access.md`), passed with
  `--provider-console-receipt`. The preflight accepts it only when it is dated
  the window day, names the approved window id, and has `console_shows_host`
  and `independent_of_guest_network` true. No out-of-band console is
  established today, so T03 cannot pass this gate until railiance-infra
  RAIL-HO-WP-0011-T09 is done;
- a current non-secret OpenBao Raft snapshot receipt that validates against
  `docs/openbao-reboot-snapshot-receipt.example.json`, the live cluster id and
  Raft index, and the default 24-hour freshness bound;
- explicit availability of two of three Shamir shares through their approved
  out-of-band custody paths.

Receipt validation failures are reported in `receipt_gate_failures` and never
abort evidence capture. Do not reuse old restore-drill templates as current
snapshot evidence. Do not
put a snapshot, unseal share, token, password, or Secret response in Git or
State Hub.

## Pre-reboot hold point

1. Announce the outage and stop new deployment, migration, credential-change,
   backup-restore, and security-engagement mutations on `railiance01`.
2. Confirm no temporary Whitehat or other expiring projection will cross the
   window. Clean it first if present.
3. Capture the read-only preflight result (keep the JSON as the pre-reboot
   evidence: boot id, failing-pod baseline, CNPG UIDs/PVC UIDs/systemIDs) and
   the audit-core chain-head attestation.
4. Confirm the latest `platform-pg` Barman backup is completed, WAL archiving
   is healthy, and the PVC is Bound. If the daily backup is stale, take and
   verify an on-demand backup before continuing. `state-hub-db` and
   `user-engine-pg` have no CNPG backup configured (nor do `net-kingdom-pg` and
   `target-revenue-pg`); their owners should take and verify a fresh backup
   through their own lane (for `user-engine-pg`, the rapp-user-engine backup
   CronJob) or record an explicit waiver before the window.
5. Resolve the stuck Terminating PVC `platform-pg-drill/platform-pg-drill-1`
   (owner: `rapp-postgres`, separately approved) or list it for the window with
   `--accept-terminating-pvc platform-pg-drill/platform-pg-drill-1`.
6. Create and escrow a fresh OpenBao Raft snapshot through the approved
   encrypted backup lane. Record only its evidence receipt and verification.
   Pass that receipt to the preflight with
   `--openbao-snapshot-evidence /path/to/receipt.json`; a merely existing,
   stale, placeholder, wrong-cluster, unencrypted, or on-host receipt is denied.

   The reviewed lane is `make openbao-reboot-snapshot` (offline tests:
   `make openbao-reboot-snapshot-test`). It runs
   `scripts/openbao-reboot-snapshot.py` as the single silent child of the
   contained `openbao-platform-admin-login` (via
   `scripts/openbao-attended-exec.py`):

   ```bash
   make openbao-reboot-snapshot \
     SNAPSHOT_SCENARIO_ID=RPF-WP-0015-T03-<scenario> \
     SNAPSHOT_RECEIPT=/abs/new/receipt.json \
     SNAPSHOT_CUSTODIAN_ESCROW_REF=<where a decryption key is held outside OpenBao> \
     SNAPSHOT_CUSTODIAN_RECIPIENT=age1...   # only if that offline key differs from the offsite recipient
     SNAPSHOT_PRIVATE_DIR=/abs/0700/dir   # optional; default is the session HOME (prefer tmpfs)
   ```

   Recoverability requires a decryption key held outside OpenBao: the escrow
   reference is mandatory and the preflight rejects a receipt without one. If the
   custodian's offline key is the offsite recipient, the reference alone is
   recorded; otherwise the snapshot is also encrypted to
   `SNAPSHOT_CUSTODIAN_RECIPIENT` (age: any one recipient key decrypts). The
   receipt lists all recipients. Reading the offsite `AGE_PRIVATE_KEY` for the
   decrypt-stream proof is a founder decision (in memory only), recorded in
   `reviews/RPF-WP-0015-T03-founder-go-20261014.json#escrow_key_read` and cited by
   the receipt. Snapshots upload to the founder-created `openbao/` folder.

   In order it reads `bao status` (initialized, unsealed, Raft), saves the
   snapshot through the tunnel into a fresh 0700 temp dir outside Git, re-reads
   status (same `cluster_id`, index not behind), age-encrypts to the offsite
   lane recipient declared in `lib/railiance-backup-common.sh` (plus the custodian
   recipient, if given), proves the
   lane's escrow key derives that recipient and decrypt-streams the ciphertext
   to the plaintext sha256, then uploads ciphertext only to the governed
   Nextcloud lane with `If-None-Match: *`. The temp dir is always shredded.
   The receipt (0600, `encrypted_location_ref`
   `offhost-custody:<scenario>:<ts>`) is self-checked with the preflight's own
   validator before it is written; on any failure the file holds a value-safe
   `status: failed` record instead, which the preflight rejects.
7. Have both Shamir share custodians present. Prove provider-console access and
   SSH access (same-day `RAIL01-CONSOLE-*` receipt). Record the exact restart
   operator, abort operator, and the railiance-infra independent abort owner.
8. Re-run the preflight with the approval and attestation flags. The result
   must say `ready_for_live_execution: true`. Pause for the final operator
   go/no-go; this is the last reversible point.

## Reboot and recovery order

1. `railiance-infra` performs one ordinary host reboot and retains the provider
   console. Do not reinstall the OS or k3s.
2. Recover host network/SSH, then verify `k3s` active and the single node Ready.
   If SSH does not return, use the provider console; do not change firewall
   declarations as an ad hoc recovery. `railiance-infra` verifies: the boot id
   differs from the pre-reboot capture and exactly one boot occurred since it;
   UFW is active with only `OpenSSH` + `2224/tcp`, `6443`/`8472` not open; the
   `k3s-api-railiance01` tunnel answers; and `timedatectl` reports
   `NTPSynchronized=yes`. Do not unseal, run OIDC/ESO checks, or trust any
   receipt or backup age until time is synchronized.
3. `railiance-cluster` verifies these core platform workloads are Running and
   reconciled: CoreDNS, local-path-provisioner, traefik, metrics-server,
   cert-manager (+ cainjector/webhook), CNPG operator
   (`cnpg-system/cnpg-controller-manager`), external-secrets (+ webhook/cert
   controller), knative-serving (activator, controller, net-kourier-controller)
   and `kourier-system/3scale-kourier-gateway`, argocd, and the OpenBao pod
   (Running but sealed, so not Ready yet). Application readiness may still be
   false at this point. Images not already cached on the node cannot be pulled
   until in-cluster Forgejo (`forgejo.coulomb.social`) and `forgejo-db` are up;
   `ImagePullBackOff` on such pods is an expected lag, not a failure, until
   Forgejo is serving — escalate only if it persists after Forgejo is Ready.
4. `rapp-postgres` (with each cluster's owner) verifies for every CNPG cluster
   the same cluster UID, primary PVC UID, and `systemID` as the pre-reboot
   evidence, primary 1/1 Ready, `ContinuousArchiving=True` where configured,
   and no re-bootstrap. A changed systemID or PVC UID is a stop condition.
5. Expect OpenBao to be sealed. Two custodians unseal it through the attended,
   non-recorded share ceremony. Run `make openbao-status` and
   `make openbao-verify-post-unseal` after it reports initialized and unsealed.
6. Verify every ClusterSecretStore is `Valid` and every ExternalSecret is
   `SecretSynced` (not only `openbao-audit-core`/`openbao-audit-core-database`
   and the audit-core projections). If they remain in the known stale `Vault is sealed`/`InvalidProviderConfig` state,
   the platform owner may restart only Deployment
   `external-secrets/external-secrets`, then recheck reconciliation.
7. Do not restart audit-core to cure database timing. `/healthz` must remain the
   liveness boundary; wait for `/readyz` to recover naturally after DNS,
   PostgreSQL, OpenBao, and ESO are healthy. Record pod identity/restart count.
8. Run audit-core integrity verification against the pre-reboot chain head and
   one separately approved synthetic E2. The E2 runs through the reviewed sender lane in
   its own attended login: `make audit-t02-attended-window` with
   `AUDIT_T02_MODE=e2` and an approval receipt whose `task_id` is
   `RAILIANCE-WP-0024-T03`. It projects one synthetic sender and sends one event
   twice through the driver's `expect-recovered` phase: the first answer must be a
   fresh `accepted` (HTTP 200/202), the second `duplicate` for the same fixture
   (accepted exactly once); it touches no lease. The harness refuses unless both
   hold and the pod UID and restart count are unchanged. The driver owns no
   server-side object; the synthetic event stays permanently in the append-only
   chain under its trial tenant, and only the sender identity is removed. Confirm no chain
   break, the prior marker remains readable, and the new event is accepted
   exactly once.
   Window `rpf-wp-0015-t03-window-20261014-01` does not run this E2. Founder
   decision `rpf-wp-0015-t03-founder-go-20261014-01` defers it to its own
   attended window and keeps tonight to the chain check. Do not mark T03 as
   proving the synthetic E2 path until that follow-up window runs.
9. Re-run the read-only preflight in post-reboot mode:

   ```bash
   python3 scripts/audit-core-recovery-preflight.py node-reboot \
     --pre-reboot-evidence /path/to/pre-reboot-preflight.json
   ```

   `reboot_comparison.passed` must be true: boot id changed with exactly one
   boot since the pre-reboot capture, no CNPG identity change, and no failing
   pod outside the pre-reboot baseline (`failing_pod_diff.new_failing` empty).
   Then end the mutation freeze and notify all owners with value-safe evidence.

## Stop, escalation, and recovery boundaries

- Before reboot: abort for missing approval, owner, console receipt, unseal
  quorum, fresh PostgreSQL backup, verified OpenBao snapshot, healthy WAL/PVC,
  accepted ops-warden revision, new scenario id/fresh decision, NTP sync, or
  any overlapping live mutation. `railiance-infra` may abort independently.
- Attended login: if the login emits any unexpected credential output (token,
  secret id, password, key, or unrecognised value-shaped text), stop
  immediately: self-revoke the issued credential, abort the window, and never
  retry under the same scenario or decision. A new scenario id and a fresh
  human decision are required before any further attempt.
- After reboot: retain the outage and escalate for a missing host, more than
  one boot, firewall drift, unsynchronized time, node/PVC/systemID identity
  change, CNPG re-bootstrap, new failing pods outside the baseline (other than
  the expected pre-Forgejo image-pull lag), OpenBao initialization loss, failed
  unseal, audit chain break, or any evidence of data loss.
- A reboot cannot be rolled back. Never run `k3s-uninstall`, initialize OpenBao,
  recover PostgreSQL in place, delete a PVC, or weaken network policy as a
  shortcut. Host recovery belongs to `railiance-infra`; Kubernetes recovery to
  `railiance-cluster`; OpenBao snapshot restoration to `railiance-platform`;
  PostgreSQL restoration always creates a new CNPG Cluster under
  `rapp-postgres`' restore procedure.

## Evidence contract

Allowed: approval/window ids, scenario and decision ids, receipt ids/paths,
owner acknowledgements, CNPG cluster UIDs and systemIDs, UFW rule targets,
NTP sync boolean, failing pod names, timestamps, host boot id
or uptime, node/cluster/PVC names and UIDs, image digests, backup name/time/age,
OpenBao initialized/sealed booleans and version, resource readiness conditions,
Secret resource versions and key names, HTTP status sequence, audit chain-head
hash, event ids/hashes for declared synthetic fixtures, pod UIDs/restart counts,
and final notifications.

Forbidden: unseal/recovery shares, OpenBao tokens or response bodies, database
credentials/usernames/DSNs, Secret data, sender bearer values, private keys,
provider-console credentials, request payloads, or decrypted snapshots.