107 lines
5.4 KiB
Markdown
107 lines
5.4 KiB
Markdown
# OpenBao CoulombCore to railiance01 migration runbook
|
|
|
|
This runbook moves the authoritative OpenBao Raft state without changing the
|
|
public API name. OpenBao remains a stateful `rail-kubernetes` workload. It must
|
|
not run on Knative or scale to zero.
|
|
|
|
## Ratified design
|
|
|
|
- Restore a source OpenBao 2.5.4 Raft snapshot into a target OpenBao 2.5.4
|
|
single-node Raft release. Never initialize a second lasting authority.
|
|
- Preserve `bao.coulomb.social` and its TLS identity. Consumers move first to
|
|
the private `openbao.openbao.svc:8200` endpoint in bounded waves; public DNS
|
|
changes only after private probes pass.
|
|
- Preserve the source Shamir barrier for this migration. Two of the three
|
|
existing unseal shares are required through an attended, non-logged
|
|
ceremony. A later auto-unseal migration is separate work and also requires
|
|
the existing threshold once.
|
|
- Encrypt snapshots before offsite transfer with the established Railiance
|
|
backup recipient. Never store plaintext snapshots, tokens, or shares in Git,
|
|
State Hub, shell history, or chat.
|
|
- The source remains unsealed and authoritative until the target passes two
|
|
restart/unseal cycles and all private consumer waves pass. During cutover it
|
|
is write-frozen, then scaled to zero while its PVCs, audit PVC, manifests,
|
|
TLS material, and encrypted snapshot remain retained for at least 14 days.
|
|
|
|
## Proven preparation
|
|
|
|
On 2026-08-03 an authoritative snapshot was encrypted and uploaded through the
|
|
approved offsite backup lane. Its non-secret fingerprints are:
|
|
|
|
- plaintext size: 106837 bytes
|
|
- plaintext SHA-256: `4933da9837f00054fc6017cbe7921e83248456af3b4b003945beed08eda9f22b`
|
|
- encrypted size: 107053 bytes
|
|
- encrypted SHA-256: `662538c3e94486b51de316d2d308ae8c4671825841823616370d7f550165fd30`
|
|
- recipient: `age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz`
|
|
|
|
A second fresh snapshot restored successfully into an isolated OpenBao 2.5.4
|
|
pod on railiance01. OpenBao correctly resealed immediately after restore under
|
|
the source Shamir barrier. On 2026-08-03 the operator supplied the rotated
|
|
2-of-3 quorum through hidden prompts. Two restart/unseal cycles passed; the
|
|
restored cluster became active with the source cluster ID, its inventory
|
|
matched, a scoped AppRole proved exact-path read and sibling denial, and the
|
|
declarative file audit device wrote successfully. Evidence is in
|
|
`docs/evidence/openbao-isolated-restore-2026-08-03.json`. The disposable
|
|
instance and plaintext staging were then removed. The empty target Helm release
|
|
has been aligned from 2.5.3 to 2.5.4.
|
|
|
|
## Preconditions and abort points
|
|
|
|
1. Confirm the encrypted offsite artifact is retrievable and its encrypted
|
|
fingerprint matches. Abort if it does not.
|
|
2. Confirm two custodians can supply source shares without disclosure to the
|
|
agent, logs, or terminal history. Abort if quorum is unavailable.
|
|
3. Freeze administrative writes and record the final Raft index and snapshot
|
|
fingerprint. Abort on writes after the final snapshot.
|
|
4. Restore only to the empty target PVC, unseal with the source quorum, and
|
|
probe through a private endpoint. Abort before consumer changes if counts,
|
|
exact-path capability, sibling denial, audit output, OIDC metadata, or SSH
|
|
role metadata differ.
|
|
5. Restart and unseal twice. Abort before consumer changes if either cycle
|
|
fails.
|
|
6. Move consumers one namespace at a time. Reverse the current wave on any
|
|
ExternalSecret, rotation, rollout, or negative-access failure.
|
|
7. Change DNS only after every railiance01 consumer has no CoulombCore runtime
|
|
dependency. Restore the prior A record on any public-path regression.
|
|
|
|
## Consumer waves
|
|
|
|
The six ClusterSecretStores cover nine ExternalSecrets. Move low-blast-radius
|
|
namespaces first, then application and identity control planes:
|
|
|
|
1. `reuse`
|
|
2. `activity-core`
|
|
3. `rapp-qonto`
|
|
4. `forgejo`
|
|
5. `user-engine`
|
|
6. `sso`
|
|
|
|
Initially reuse the restored AppRoles against the private endpoint. For each
|
|
wave, construct namespace- and service-account-restricted Kubernetes auth,
|
|
prove the exact path and sibling denial, switch the store, force refresh,
|
|
observe workload rollout, and retain the AppRole until reversal is proven.
|
|
The restored Kubernetes auth configuration must be rewritten for the
|
|
railiance01 API and reviewer identity before use.
|
|
|
|
## Rollback, retention, RTO, and RPO
|
|
|
|
- Rollback owner: `railiance-platform`; DNS owner: the Coulomb DNS operator;
|
|
workload owners validate their namespace waves.
|
|
- Before DNS cutover, reverse a store endpoint/auth change and refresh its
|
|
ExternalSecrets. The source remains authoritative.
|
|
- After DNS cutover, restore the CoulombCore workloads and previous DNS A
|
|
record, then reverse private store endpoints. Do not restore a newer target
|
|
snapshot over the retained source without a new reviewed recovery plan.
|
|
- Target RTO is 60 minutes from an abort decision. RPO is the bounded final
|
|
write-freeze window; the intended RPO is zero acknowledged writes.
|
|
- Keep the stopped source state and final encrypted snapshot for at least 14
|
|
days. PVC, Raft, Secret, audit, or TLS deletion requires fresh explicit
|
|
approval after a successful railiance01 disaster-recovery drill.
|
|
|
|
## Completion evidence
|
|
|
|
Completion requires machine-readable source/target inventories, two successful
|
|
restart/unseal cycles, capability tests with no secret values recorded, all
|
|
nine ExternalSecrets Ready on local identity, public TLS/OIDC/operator probes,
|
|
backup retrieval proof, absence of source traffic throughout the observation
|
|
window, and State Hub/file consistency.
|