railiance-master/docs/openbao-coulombcore-to-railiance01-runbook.md

OpenBao CoulombCore to railiance01 migration runbook

This runbook moves the authoritative OpenBao Raft state without changing the public API name. OpenBao remains a stateful rail-kubernetes workload. It must not run on Knative or scale to zero.

Ratified design

• Restore a source OpenBao 2.5.4 Raft snapshot into a target OpenBao 2.5.4 single-node Raft release. Never initialize a second lasting authority.
• Preserve bao.coulomb.social and its TLS identity. Consumers move first to the private openbao.openbao.svc:8200 endpoint in bounded waves; public DNS changes only after private probes pass.
• Preserve the source Shamir barrier for this migration. Two of the three existing unseal shares are required through an attended, non-logged ceremony. A later auto-unseal migration is separate work and also requires the existing threshold once.
• Encrypt snapshots before offsite transfer with the established Railiance backup recipient. Never store plaintext snapshots, tokens, or shares in Git, State Hub, shell history, or chat.
• The source remains unsealed and authoritative until the target passes two restart/unseal cycles and all private consumer waves pass. During cutover it is write-frozen, then scaled to zero while its PVCs, audit PVC, manifests, TLS material, and encrypted snapshot remain retained for at least 14 days.

Proven preparation

On 2026-08-03 an authoritative snapshot was encrypted and uploaded through the approved offsite backup lane. Its non-secret fingerprints are:

• plaintext size: 106837 bytes
• plaintext SHA-256: 4933da9837f00054fc6017cbe7921e83248456af3b4b003945beed08eda9f22b
• encrypted size: 107053 bytes
• encrypted SHA-256: 662538c3e94486b51de316d2d308ae8c4671825841823616370d7f550165fd30
• recipient: age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz

A second fresh snapshot restored successfully into an isolated OpenBao 2.5.4 pod on railiance01. OpenBao correctly resealed immediately after restore under the source Shamir barrier. On 2026-08-03 the operator supplied the rotated 2-of-3 quorum through hidden prompts. Two restart/unseal cycles passed; the restored cluster became active with the source cluster ID, its inventory matched, a scoped AppRole proved exact-path read and sibling denial, and the declarative file audit device wrote successfully. Evidence is in docs/evidence/openbao-isolated-restore-2026-08-03.json. The disposable instance and plaintext staging were then removed. The empty target Helm release has been aligned from 2.5.3 to 2.5.4.

Preconditions and abort points

1. Confirm the encrypted offsite artifact is retrievable and its encrypted fingerprint matches. Abort if it does not.
2. Confirm two custodians can supply source shares without disclosure to the agent, logs, or terminal history. Abort if quorum is unavailable.
3. Freeze administrative writes and record the final Raft index and snapshot fingerprint. Abort on writes after the final snapshot.
4. Restore only to the empty target PVC, unseal with the source quorum, and probe through a private endpoint. Abort before consumer changes if counts, exact-path capability, sibling denial, audit output, OIDC metadata, or SSH role metadata differ.
5. Restart and unseal twice. Abort before consumer changes if either cycle fails.
6. Move consumers one namespace at a time. Reverse the current wave on any ExternalSecret, rotation, rollout, or negative-access failure.
7. Change DNS only after every railiance01 consumer has no CoulombCore runtime dependency. Restore the prior A record on any public-path regression.

Consumer waves

The six ClusterSecretStores cover nine ExternalSecrets. Move low-blast-radius namespaces first, then application and identity control planes:

1. reuse
2. activity-core
3. rapp-qonto
4. forgejo
5. user-engine
6. sso

Initially reuse the restored AppRoles against the private endpoint. For each wave, construct namespace- and service-account-restricted Kubernetes auth, prove the exact path and sibling denial, switch the store, force refresh, observe workload rollout, and retain the AppRole until reversal is proven. The restored Kubernetes auth configuration must be rewritten for the railiance01 API and reviewer identity before use.

Rollback, retention, RTO, and RPO

• Rollback owner: railiance-platform; DNS owner: the Coulomb DNS operator; workload owners validate their namespace waves.
• Before DNS cutover, reverse a store endpoint/auth change and refresh its ExternalSecrets. The source remains authoritative.
• After DNS cutover, restore the CoulombCore workloads and previous DNS A record, then reverse private store endpoints. Do not restore a newer target snapshot over the retained source without a new reviewed recovery plan.
• Target RTO is 60 minutes from an abort decision. RPO is the bounded final write-freeze window; the intended RPO is zero acknowledged writes.
• Keep the stopped source state and final encrypted snapshot for at least 14 days. PVC, Raft, Secret, audit, or TLS deletion requires fresh explicit approval after a successful railiance01 disaster-recovery drill.

Completion evidence

Completion requires machine-readable source/target inventories, two successful restart/unseal cycles, capability tests with no secret values recorded, all nine ExternalSecrets Ready on local identity, public TLS/OIDC/operator probes, backup retrieval proof, absence of source traffic throughout the observation window, and State Hub/file consistency.