railiance-master/workplans/RMASTER-WP-0020-openbao-migration-to-reef-railiance.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	depends_on	state_hub_workstream_id
RMASTER-WP-0020	workplan	Migrate authoritative OpenBao from CoulombCore to reef-railiance	financials	railiance-master	backlog	codex	railiance	2026-07-30	2026-08-04	NK-WP-0022	0616a297-18c5-4c4e-a4fc-69135b3f9a15

RMASTER-WP-0020 - OpenBao migration to reef-railiance

Move the authoritative OpenBao runtime from CoulombCore to the default rail-kubernetes platform path on reef-railiance, preserving every secret, policy, auth method, identity, lease-relevant contract, audit record, and recovery capability. This is a stateful security control plane and must never run on scale-to-zero Knative.

Current-state review

• bao.coulomb.social resolves to CoulombCore (92.205.130.254).
• CoulombCore runs initialized, unsealed OpenBao 2.5.4 with integrated Raft, 5 GiB data PVC, 2 GiB audit PVC, and the public UI gateway.
• Every railiance01 ClusterSecretStore currently uses https://bao.coulomb.social.
• railiance01 has an OpenBao 2.5.3 StatefulSet with local Raft PVCs, but it is uninitialized, sealed, has no ingress, and serves no consumers.
• Cross-cluster ESO access currently uses transitional AppRoles. Once OpenBao is local, same-cluster Kubernetes authentication is the preferred steady state.
• The reef is a single node. Migration improves topology consolidation but does not create infrastructure high availability; backup and rollback are mandatory.

T01 - Inventory authoritative state and consumer contracts

id: RMASTER-WP-0020-T01
status: done
priority: high
state_hub_task_id: "0d40e879-ae4e-45de-aa31-81bf110924dc"

Capture non-secret inventories of mounts, auth methods, policies, roles, entities, aliases, audit devices, Raft peers, versions, storage, ingress, ClusterSecretStores, ExternalSecrets, direct clients, automation scripts, and backup jobs. Record counts and fingerprints without secret values. Identify all consumers that bypass ESO or depend on the public address.

Done when source, empty target, and every consumer have a machine-readable dependency map plus explicit rollback owners.

Completed 2026-08-03. The non-secret source, target, and consumer dependency map is recorded in docs/evidence/openbao-migration-inventory-2026-08-03.json. It includes six ClusterSecretStores, nine ExternalSecrets, state/auth counts, storage and version topology, and rollback ownership without secret values.

T02 - Ratify migration, seal, and custody design

id: RMASTER-WP-0020-T02
status: done
priority: high
state_hub_task_id: "c266dce7-7d89-454a-a299-48fc169efea1"

Choose and document a supported Raft snapshot migration procedure; align target OpenBao version before restore. Define unseal/recovery-key custody, TLS and DNS continuity, audit retention, target PVC handling, service identity, backup encryption, operator access, and emergency rollback. Use ops-mason for any new credential, policy, AppRole, or Kubernetes-auth construction plan and retain its executive approval gate.

Done when the procedure has preconditions, abort points, rollback commands, RTO/RPO, and no secret material in Git or State Hub.

Completed 2026-08-03. The ratified procedure, custody boundary, consumer waves, abort points, rollback, retention, RTO, and RPO are documented in docs/openbao-coulombcore-to-railiance01-runbook.md. The empty target release was aligned to OpenBao 2.5.4, matching the source.

T03 - Prove backup and isolated restore before cutover

id: RMASTER-WP-0020-T03
status: done
priority: high
state_hub_task_id: "88671108-cf8c-4de6-b1ce-06e99b1c3d70"

Take an encrypted authoritative Raft snapshot and matching configuration/audit evidence through approved custody. Restore it into an isolated non-serving instance at the target version. Verify inventory counts, representative exact-path reads, sibling denial, auth-method metadata, audit continuity, and restart/unseal behavior.

Done when restore evidence is machine-readable and rollback does not depend on the live source remaining healthy.

2026-08-03: Created a fresh authoritative snapshot, encrypted it before offsite upload, removed plaintext staging, and recorded only fingerprints. A second fresh snapshot restored successfully into an isolated OpenBao 2.5.4 pod on railiance01. The restored instance correctly resealed under the source Shamir barrier. The remaining semantic and repeated restart/unseal proof needs two of the three existing source shares through the attended custody ceremony; no supported automated custody route exposes those shares.

Completed 2026-08-03. The operator supplied two rotated shares through hidden prompts. Two restart/unseal cycles passed. The restored cluster became active with the source cluster ID; inventory counts matched; the existing scoped Qonto AppRole proved exact-path read and sibling deny; and declarative file audit output was non-empty. Machine-readable evidence is in docs/evidence/openbao-isolated-restore-2026-08-03.json.

T04 - Restore authoritative state on railiance01

id: RMASTER-WP-0020-T04
status: done
priority: high
state_hub_task_id: "d04d7fdd-bd00-407a-8954-101b680ed327"

Freeze administrative writes for a bounded window, take the final snapshot, replace the empty target state through the ratified procedure, restore configuration and audit continuity, and bring the target up privately. Do not change public DNS or consumers yet.

Done when target semantic probes match the source and repeated restart/unseal tests pass through a private endpoint.

Completed 2026-08-03. A final source snapshot was encrypted and uploaded offsite (plaintext SHA-256 ac5e27536c01c3a019c3dde1069866569112c76d8ab42ec2bdb45724cc9b13f8, encrypted SHA-256 22003d8fcb12ef17d6edafad9358ed2b0e538b21a8daee86445d9b85482f9161). The still-empty railiance01 target was restored privately, temporary init material and plaintext staging were removed, and two attended target restart/unseal cycles passed. The target is active on OpenBao 2.5.4 with the source cluster ID and a non-empty persistent audit file. CoulombCore remains live and no public DNS changed.

T05 - Migrate consumers to local workload identity

id: RMASTER-WP-0020-T05
status: done
priority: high
state_hub_task_id: "f0a82e4e-8074-4a22-b1e9-f01ec0ff76a1"

Move ClusterSecretStores and other railiance01 clients in bounded waves. Replace transitional cross-cluster AppRoles with namespace-restricted same-cluster Kubernetes-auth roles where supported. For each wave prove ExternalSecret readiness, exact-path access, sibling denial, rotation, workload rollout, and reversal before continuing.

Done when all railiance01 consumers use the local private service and no runtime dependency requires the CoulombCore API.

2026-08-03: Wave 1 (reuse) moved to http://openbao.openbao.svc:8200. The ClusterSecretStore remained Valid, a forced ExternalSecret refresh completed as SecretSynced/Ready, and the declarative owner change was committed in railiance-platform@33b36e8.

Completed 2026-08-03. Activity-core, Qonto, Forgejo, user-engine, and SSO were moved in separate waves. All six ClusterSecretStores are Valid on the private service and all nine ExternalSecrets refreshed as SecretSynced/Ready; affected Knative, application, Forgejo, and identity pods remained healthy. Declarative changes are in railiance-platform@741f209 and net-kingdom@0ec6f8c. Transitional tokens/AppRoles remain temporarily available for bounded rollback; same-cluster Kubernetes identity conversion remains post-cutover hardening.

T06 - Cut over public OpenBao DNS and operator access

id: RMASTER-WP-0020-T06
status: done
priority: high
state_hub_task_id: "f458d110-5fdd-465a-8a23-4ada1051fb12"

After T03-T05, move bao.coulomb.social to railiance01 with TLS continuity. Verify status, UI gateway, OIDC/operator login, scoped automation, audit delivery, backup, revocation, rotation, and negative access through normal DNS. Keep the source sealed or write-frozen and immediately recoverable.

Done when the observation window passes without source traffic or divergence.

2026-08-03: Deployed the rapp-openbao UI overlay gateway, Traefik middleware, Ingress, and cert-manager Certificate on railiance01. The gateway is Ready. The ACME HTTP-01 challenge is correctly pending because public DNS still points to CoulombCore. No automated DNS credential route exists; change the A record to 92.205.62.239, then verify certificate, UI/OIDC, API, audit, backup, and negative access before retiring the source.

Completed 2026-08-03. All authoritative and sampled public resolvers converged on 92.205.62.239. The rapp-openbao gateway serves HTTP 200 for UI and health with a valid cert-manager-owned Let's Encrypt certificate. The target remains active/unsealed with the preserved cluster ID; its audit file is growing and all six stores plus nine ExternalSecrets remain Ready. The source OIDC initiation behavior was compared directly and no target-only regression was found.

T07 - Retire CoulombCore OpenBao reversibly

id: RMASTER-WP-0020-T07
status: done
priority: high
state_hub_task_id: "db2443e6-bad7-4a71-b45f-d4b81099df15"

Scale the CoulombCore OpenBao and UI gateway to zero after successful cutover. Retain manifests, encrypted final snapshot, PVCs, Secrets, TLS material, and documented rollback for at least 14 days. Confirm no DNS, monitor, client, tunnel, catalog entry, or automation still targets the old runtime.

Done when CoulombCore serves no OpenBao traffic and rollback remains tested.

Completed 2026-08-03. After more than one DNS TTL and multi-resolver convergence, the old gateway showed only its Kubernetes readiness probes and no external traffic. CoulombCore StatefulSet openbao and Deployment openbao-ui-gateway were scaled to zero and their pods terminated. Data and audit PVCs, Services, Ingress, TLS and Helm Secrets, manifests, rotated unseal custody, and encrypted final snapshot remain retained for rollback. A post-retirement public probe returned HTTP 200 from railiance01 with valid TLS.

T08 - Final cleanup and closure

id: RMASTER-WP-0020-T08
status: todo
priority: medium
state_hub_task_id: "1dc50d98-64e8-4755-a52e-2b94b765019e"

After retention expiry, a successful railiance01 disaster-recovery drill, and new explicit approval for destructive deletion, remove stale CoulombCore OpenBao resources and transitional AppRole/bootstrap material. Update topology, runbooks, routing catalog, State Hub, and the CoulombCore retirement plan.

Done when reef-railiance is the sole authoritative OpenBao location and all source-backed records agree.

Retention gate opened 2026-08-03. Do not delete the retained CoulombCore PVCs, Secrets, Helm records, Ingress, or Services before 2026-08-17. Final deletion also requires a successful railiance01 disaster-recovery drill and fresh explicit destructive-cleanup approval.

2026-08-04: The workplan moved to backlog for the shortened retention window. Activity Core one-shot definition activity-definitions/openbao-retention-closeout.md reactivates T08 at 08:00 Europe/Berlin on 2026-08-17 by emitting a claimable ops run. The schedule never performs destructive cleanup itself.

Safety constraints

• Never initialize or overwrite either OpenBao instance without verified snapshot and rollback evidence.
• Never print, export, compare, or log secret values; use counts, names, capabilities, and cryptographic fingerprints where appropriate.
• Do not switch DNS before private target restore and consumer waves pass.
• Do not revoke transitional access until the replacement path and reversal are proven.
• Final PVC, Secret, Raft, recovery-key, or audit deletion requires separate explicit approval after the retention window.