--- id: RAILIANCE-WP-0020 type: workplan title: "Migrate authoritative OpenBao from CoulombCore to reef-railiance" domain: financials repo: railiance-master status: active owner: codex topic_slug: railiance created: "2026-07-30" updated: "2026-08-03" depends_on: - NK-WP-0022 state_hub_workstream_id: "0616a297-18c5-4c4e-a4fc-69135b3f9a15" --- # RAILIANCE-WP-0020 - OpenBao migration to reef-railiance Move the authoritative OpenBao runtime from CoulombCore to the default `rail-kubernetes` platform path on `reef-railiance`, preserving every secret, policy, auth method, identity, lease-relevant contract, audit record, and recovery capability. This is a stateful security control plane and must never run on scale-to-zero Knative. ## Current-state review - `bao.coulomb.social` resolves to CoulombCore (`92.205.130.254`). - CoulombCore runs initialized, unsealed OpenBao 2.5.4 with integrated Raft, 5 GiB data PVC, 2 GiB audit PVC, and the public UI gateway. - Every railiance01 ClusterSecretStore currently uses `https://bao.coulomb.social`. - railiance01 has an OpenBao 2.5.3 StatefulSet with local Raft PVCs, but it is uninitialized, sealed, has no ingress, and serves no consumers. - Cross-cluster ESO access currently uses transitional AppRoles. Once OpenBao is local, same-cluster Kubernetes authentication is the preferred steady state. - The reef is a single node. Migration improves topology consolidation but does not create infrastructure high availability; backup and rollback are mandatory. ## T01 - Inventory authoritative state and consumer contracts ```task id: RAILIANCE-WP-0020-T01 status: done priority: high state_hub_task_id: "0d40e879-ae4e-45de-aa31-81bf110924dc" ``` Capture non-secret inventories of mounts, auth methods, policies, roles, entities, aliases, audit devices, Raft peers, versions, storage, ingress, ClusterSecretStores, ExternalSecrets, direct clients, automation scripts, and backup jobs. Record counts and fingerprints without secret values. Identify all consumers that bypass ESO or depend on the public address. Done when source, empty target, and every consumer have a machine-readable dependency map plus explicit rollback owners. Completed 2026-08-03. The non-secret source, target, and consumer dependency map is recorded in `docs/evidence/openbao-migration-inventory-2026-08-03.json`. It includes six ClusterSecretStores, nine ExternalSecrets, state/auth counts, storage and version topology, and rollback ownership without secret values. ## T02 - Ratify migration, seal, and custody design ```task id: RAILIANCE-WP-0020-T02 status: done priority: high state_hub_task_id: "c266dce7-7d89-454a-a299-48fc169efea1" ``` Choose and document a supported Raft snapshot migration procedure; align target OpenBao version before restore. Define unseal/recovery-key custody, TLS and DNS continuity, audit retention, target PVC handling, service identity, backup encryption, operator access, and emergency rollback. Use ops-mason for any new credential, policy, AppRole, or Kubernetes-auth construction plan and retain its executive approval gate. Done when the procedure has preconditions, abort points, rollback commands, RTO/RPO, and no secret material in Git or State Hub. Completed 2026-08-03. The ratified procedure, custody boundary, consumer waves, abort points, rollback, retention, RTO, and RPO are documented in `docs/openbao-coulombcore-to-railiance01-runbook.md`. The empty target release was aligned to OpenBao 2.5.4, matching the source. ## T03 - Prove backup and isolated restore before cutover ```task id: RAILIANCE-WP-0020-T03 status: wait priority: high state_hub_task_id: "88671108-cf8c-4de6-b1ce-06e99b1c3d70" ``` Take an encrypted authoritative Raft snapshot and matching configuration/audit evidence through approved custody. Restore it into an isolated non-serving instance at the target version. Verify inventory counts, representative exact-path reads, sibling denial, auth-method metadata, audit continuity, and restart/unseal behavior. Done when restore evidence is machine-readable and rollback does not depend on the live source remaining healthy. 2026-08-03: Created a fresh authoritative snapshot, encrypted it before offsite upload, removed plaintext staging, and recorded only fingerprints. A second fresh snapshot restored successfully into an isolated OpenBao 2.5.4 pod on railiance01. The restored instance correctly resealed under the source Shamir barrier. The remaining semantic and repeated restart/unseal proof needs two of the three existing source shares through the attended custody ceremony; no supported automated custody route exposes those shares. ## T04 - Restore authoritative state on railiance01 ```task id: RAILIANCE-WP-0020-T04 status: wait priority: high state_hub_task_id: "d04d7fdd-bd00-407a-8954-101b680ed327" ``` Freeze administrative writes for a bounded window, take the final snapshot, replace the empty target state through the ratified procedure, restore configuration and audit continuity, and bring the target up privately. Do not change public DNS or consumers yet. Done when target semantic probes match the source and repeated restart/unseal tests pass through a private endpoint. ## T05 - Migrate consumers to local workload identity ```task id: RAILIANCE-WP-0020-T05 status: wait priority: high state_hub_task_id: "f0a82e4e-8074-4a22-b1e9-f01ec0ff76a1" ``` Move ClusterSecretStores and other railiance01 clients in bounded waves. Replace transitional cross-cluster AppRoles with namespace-restricted same-cluster Kubernetes-auth roles where supported. For each wave prove ExternalSecret readiness, exact-path access, sibling denial, rotation, workload rollout, and reversal before continuing. Done when all railiance01 consumers use the local private service and no runtime dependency requires the CoulombCore API. ## T06 - Cut over public OpenBao DNS and operator access ```task id: RAILIANCE-WP-0020-T06 status: wait priority: high state_hub_task_id: "f458d110-5fdd-465a-8a23-4ada1051fb12" ``` After T03-T05, move `bao.coulomb.social` to railiance01 with TLS continuity. Verify status, UI gateway, OIDC/operator login, scoped automation, audit delivery, backup, revocation, rotation, and negative access through normal DNS. Keep the source sealed or write-frozen and immediately recoverable. Done when the observation window passes without source traffic or divergence. ## T07 - Retire CoulombCore OpenBao reversibly ```task id: RAILIANCE-WP-0020-T07 status: wait priority: high state_hub_task_id: "db2443e6-bad7-4a71-b45f-d4b81099df15" ``` Scale the CoulombCore OpenBao and UI gateway to zero after successful cutover. Retain manifests, encrypted final snapshot, PVCs, Secrets, TLS material, and documented rollback for at least 30 days. Confirm no DNS, monitor, client, tunnel, catalog entry, or automation still targets the old runtime. Done when CoulombCore serves no OpenBao traffic and rollback remains tested. ## T08 - Final cleanup and closure ```task id: RAILIANCE-WP-0020-T08 status: wait priority: medium state_hub_task_id: "1dc50d98-64e8-4755-a52e-2b94b765019e" ``` After retention expiry, a successful railiance01 disaster-recovery drill, and new explicit approval for destructive deletion, remove stale CoulombCore OpenBao resources and transitional AppRole/bootstrap material. Update topology, runbooks, routing catalog, State Hub, and the CoulombCore retirement plan. Done when reef-railiance is the sole authoritative OpenBao location and all source-backed records agree. ## Safety constraints - Never initialize or overwrite either OpenBao instance without verified snapshot and rollback evidence. - Never print, export, compare, or log secret values; use counts, names, capabilities, and cryptographic fingerprints where appropriate. - Do not switch DNS before private target restore and consumer waves pass. - Do not revoke transitional access until the replacement path and reversal are proven. - Final PVC, Secret, Raft, recovery-key, or audit deletion requires separate explicit approval after the retention window.