railiance-master/workplans/RMASTER-WP-0020-openbao-migration-to-reef-railiance.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	depends_on	related	state_hub_workstream_id
RMASTER-WP-0020	workplan	Migrate authoritative OpenBao from CoulombCore to reef-railiance	financials	railiance-master	blocked	codex	railiance	2026-07-30	2026-08-23	NK-WP-0022	ADR-0008 RMASTER-WP-0023	0616a297-18c5-4c4e-a4fc-69135b3f9a15

RMASTER-WP-0020 - OpenBao migration to reef-railiance

Move the authoritative OpenBao runtime from CoulombCore onto rail-kubernetes hosted by reef-railiance (rapp-openbao), preserving every secret, policy, auth method, identity, lease-relevant contract, audit record, and recovery capability. This is a stateful security control plane and must never run on scale-to-zero Knative.

Current-state review

• bao.coulomb.social resolves to CoulombCore (92.205.130.254).
• CoulombCore runs initialized, unsealed OpenBao 2.5.4 with integrated Raft, 5 GiB data PVC, 2 GiB audit PVC, and the public UI gateway.
• Every railiance01 ClusterSecretStore currently uses https://bao.coulomb.social.
• railiance01 has an OpenBao 2.5.3 StatefulSet with local Raft PVCs, but it is uninitialized, sealed, has no ingress, and serves no consumers.
• Cross-cluster ESO access currently uses transitional AppRoles. Once OpenBao is local, same-cluster Kubernetes authentication is the preferred steady state.
• The reef is a single node. Migration improves topology consolidation but does not create infrastructure high availability; backup and rollback are mandatory.

T01 - Inventory authoritative state and consumer contracts

id: RMASTER-WP-0020-T01
status: done
priority: high
state_hub_task_id: "0d40e879-ae4e-45de-aa31-81bf110924dc"

Capture non-secret inventories of mounts, auth methods, policies, roles, entities, aliases, audit devices, Raft peers, versions, storage, ingress, ClusterSecretStores, ExternalSecrets, direct clients, automation scripts, and backup jobs. Record counts and fingerprints without secret values. Identify all consumers that bypass ESO or depend on the public address.

Done when source, empty target, and every consumer have a machine-readable dependency map plus explicit rollback owners.

Completed 2026-08-03. The non-secret source, target, and consumer dependency map is recorded in docs/evidence/openbao-migration-inventory-2026-08-03.json. It includes six ClusterSecretStores, nine ExternalSecrets, state/auth counts, storage and version topology, and rollback ownership without secret values.

T02 - Ratify migration, seal, and custody design

id: RMASTER-WP-0020-T02
status: done
priority: high
state_hub_task_id: "c266dce7-7d89-454a-a299-48fc169efea1"

Choose and document a supported Raft snapshot migration procedure; align target OpenBao version before restore. Define unseal/recovery-key custody, TLS and DNS continuity, audit retention, target PVC handling, service identity, backup encryption, operator access, and emergency rollback. Use ops-mason for any new credential, policy, AppRole, or Kubernetes-auth construction plan and retain its executive approval gate.

Done when the procedure has preconditions, abort points, rollback commands, RTO/RPO, and no secret material in Git or State Hub.

Completed 2026-08-03. The ratified procedure, custody boundary, consumer waves, abort points, rollback, retention, RTO, and RPO are documented in docs/openbao-coulombcore-to-railiance01-runbook.md. The empty target release was aligned to OpenBao 2.5.4, matching the source.

T03 - Prove backup and isolated restore before cutover

id: RMASTER-WP-0020-T03
status: done
priority: high
state_hub_task_id: "88671108-cf8c-4de6-b1ce-06e99b1c3d70"

Take an encrypted authoritative Raft snapshot and matching configuration/audit evidence through approved custody. Restore it into an isolated non-serving instance at the target version. Verify inventory counts, representative exact-path reads, sibling denial, auth-method metadata, audit continuity, and restart/unseal behavior.

Done when restore evidence is machine-readable and rollback does not depend on the live source remaining healthy.

2026-08-03: Created a fresh authoritative snapshot, encrypted it before offsite upload, removed plaintext staging, and recorded only fingerprints. A second fresh snapshot restored successfully into an isolated OpenBao 2.5.4 pod on railiance01. The restored instance correctly resealed under the source Shamir barrier. The remaining semantic and repeated restart/unseal proof needs two of the three existing source shares through the attended custody ceremony; no supported automated custody route exposes those shares.

Completed 2026-08-03. The operator supplied two rotated shares through hidden prompts. Two restart/unseal cycles passed. The restored cluster became active with the source cluster ID; inventory counts matched; the existing scoped Qonto AppRole proved exact-path read and sibling deny; and declarative file audit output was non-empty. Machine-readable evidence is in docs/evidence/openbao-isolated-restore-2026-08-03.json.

T04 - Restore authoritative state on railiance01

id: RMASTER-WP-0020-T04
status: done
priority: high
state_hub_task_id: "d04d7fdd-bd00-407a-8954-101b680ed327"

Freeze administrative writes for a bounded window, take the final snapshot, replace the empty target state through the ratified procedure, restore configuration and audit continuity, and bring the target up privately. Do not change public DNS or consumers yet.

Done when target semantic probes match the source and repeated restart/unseal tests pass through a private endpoint.

Completed 2026-08-03. A final source snapshot was encrypted and uploaded offsite (plaintext SHA-256 ac5e27536c01c3a019c3dde1069866569112c76d8ab42ec2bdb45724cc9b13f8, encrypted SHA-256 22003d8fcb12ef17d6edafad9358ed2b0e538b21a8daee86445d9b85482f9161). The still-empty railiance01 target was restored privately, temporary init material and plaintext staging were removed, and two attended target restart/unseal cycles passed. The target is active on OpenBao 2.5.4 with the source cluster ID and a non-empty persistent audit file. CoulombCore remains live and no public DNS changed.

T05 - Migrate consumers to local workload identity

id: RMASTER-WP-0020-T05
status: done
priority: high
state_hub_task_id: "f0a82e4e-8074-4a22-b1e9-f01ec0ff76a1"

Move ClusterSecretStores and other railiance01 clients in bounded waves. Replace transitional cross-cluster AppRoles with namespace-restricted same-cluster Kubernetes-auth roles where supported. For each wave prove ExternalSecret readiness, exact-path access, sibling denial, rotation, workload rollout, and reversal before continuing.

Done when all railiance01 consumers use the local private service and no runtime dependency requires the CoulombCore API.

2026-08-03: Wave 1 (reuse) moved to http://openbao.openbao.svc:8200. The ClusterSecretStore remained Valid, a forced ExternalSecret refresh completed as SecretSynced/Ready, and the declarative owner change was committed in railiance-platform@33b36e8.

Completed 2026-08-03. Activity-core, Qonto, Forgejo, user-engine, and SSO were moved in separate waves. All six ClusterSecretStores are Valid on the private service and all nine ExternalSecrets refreshed as SecretSynced/Ready; affected Knative, application, Forgejo, and identity pods remained healthy. Declarative changes are in railiance-platform@741f209 and net-kingdom@0ec6f8c. Transitional tokens/AppRoles remain temporarily available for bounded rollback; same-cluster Kubernetes identity conversion remains post-cutover hardening.

T06 - Cut over public OpenBao DNS and operator access

id: RMASTER-WP-0020-T06
status: done
priority: high
state_hub_task_id: "f458d110-5fdd-465a-8a23-4ada1051fb12"

After T03-T05, move bao.coulomb.social to railiance01 with TLS continuity. Verify status, UI gateway, OIDC/operator login, scoped automation, audit delivery, backup, revocation, rotation, and negative access through normal DNS. Keep the source sealed or write-frozen and immediately recoverable.

Done when the observation window passes without source traffic or divergence.

2026-08-03: Deployed the rapp-openbao UI overlay gateway, Traefik middleware, Ingress, and cert-manager Certificate on railiance01. The gateway is Ready. The ACME HTTP-01 challenge is correctly pending because public DNS still points to CoulombCore. No automated DNS credential route exists; change the A record to 92.205.62.239, then verify certificate, UI/OIDC, API, audit, backup, and negative access before retiring the source.

Completed 2026-08-03. All authoritative and sampled public resolvers converged on 92.205.62.239. The rapp-openbao gateway serves HTTP 200 for UI and health with a valid cert-manager-owned Let's Encrypt certificate. The target remains active/unsealed with the preserved cluster ID; its audit file is growing and all six stores plus nine ExternalSecrets remain Ready. The source OIDC initiation behavior was compared directly and no target-only regression was found.

T07 - Retire CoulombCore OpenBao reversibly

id: RMASTER-WP-0020-T07
status: done
priority: high
state_hub_task_id: "db2443e6-bad7-4a71-b45f-d4b81099df15"

Scale the CoulombCore OpenBao and UI gateway to zero after successful cutover. Retain manifests, encrypted final snapshot, PVCs, Secrets, TLS material, and documented rollback for at least 14 days. Confirm no DNS, monitor, client, tunnel, catalog entry, or automation still targets the old runtime.

Done when CoulombCore serves no OpenBao traffic and rollback remains tested.

Completed 2026-08-03. After more than one DNS TTL and multi-resolver convergence, the old gateway showed only its Kubernetes readiness probes and no external traffic. CoulombCore StatefulSet openbao and Deployment openbao-ui-gateway were scaled to zero and their pods terminated. Data and audit PVCs, Services, Ingress, TLS and Helm Secrets, manifests, rotated unseal custody, and encrypted final snapshot remain retained for rollback. A post-retirement public probe returned HTTP 200 from railiance01 with valid TLS.

T08 - Final cleanup and closure

id: RMASTER-WP-0020-T08
status: wait
priority: medium
state_hub_task_id: "1dc50d98-64e8-4755-a52e-2b94b765019e"

After retention expiry, a successful railiance01 disaster-recovery drill, and new explicit approval for destructive deletion, remove stale CoulombCore OpenBao resources and transitional AppRole/bootstrap material. Update topology, runbooks, routing catalog, State Hub, and the CoulombCore retirement plan.

Done when reef-railiance is the sole authoritative OpenBao location and all source-backed records agree.

Retention gate opened 2026-08-03. Do not delete the retained CoulombCore PVCs, Secrets, Helm records, Ingress, or Services before 2026-08-17. Final deletion also requires a successful railiance01 disaster-recovery drill and fresh explicit destructive-cleanup approval.

2026-08-04: The workplan moved to backlog for the shortened retention window. Activity Core one-shot definition activity-definitions/openbao-retention-closeout.md reactivates T08 at 08:00 Europe/Berlin on 2026-08-17 by emitting a claimable ops run. The schedule never performs destructive cleanup itself.

2026-08-14: Workplan moved from backlog to blocked. T08 is wait. The three gates are still closed: retention until 2026-08-17, a post-cutover railiance01 disaster-recovery drill (the 2026-08-03 isolated restore is T03, not this), and fresh explicit approval for destructive CoulombCore deletion. Do not reopen until those three are true. The 2026-08-17 one-shot may move the workplan to active; it still must not delete anything.

2026-08-22 review: the time-based retention gate has expired, but the other two gates have not. rapp-openbao contains the restore-drill validator and an example evidence document, not completed post-cutover disaster-recovery evidence. This session also carries no fresh, explicit authorization to delete the retained CoulombCore PVCs, Secrets, Helm records, Ingress, or Services. T08 therefore remains wait; finishing the migration workplan is not itself destructive-cleanup approval. The package-owner evidence request is State Hub message 02ec17ef-14c2-4c91-9768-7e9ac8714325.

T09 - Retract public OpenBao listener

id: RMASTER-WP-0020-T09
status: progress
priority: high
state_hub_task_id: "287920d3-b038-495f-b521-d16cfc7ba941"

Depends on T08 remaining blocked until its three gates open; this task can be routed sooner.

T06 published bao.coulomb.social on Railiance01 for DNS continuity. ADR-0008 later split admission from exposure. The 2026-08-15 snapshot lists that name as a routed close, not a grant. Target inventory is public_ingress: false. Consumers already use openbao.openbao.svc:8200.

Route to rapp-openbao / railiance-platform: drop public Ingress, keep ClusterIP, set exposure.posture: operator with a named ops-bridge tunnel for the UI. Do not implement NetworkPolicy here.

Done when: bao.coulomb.social is not a public listener we publish, the runbook no longer treats public DNS as the steady state, and the declaration carries operator (or private) rather than an implicit public grant.

2026-08-18: railiance-platform confirmed from live cluster state that bao.coulomb.social resolves to Railiance01 and is published by the openbao/openbao-ui-gateway Traefik Ingress with a trusted certificate, HSTS, and rate limiting. This is the deliberate T06 migration-continuity surface, not accidental drift. The later operator-accepted ADR-0008 decision is nevertheless controlling: the snapshot records this name as a routed close, no exposure grant exists, and the target remains public_ingress: false. T09 is therefore in progress and routed back to the owning repo to retract the public listener while preserving the ClusterIP and establishing the named operator tunnel. No destructive T08 cleanup is authorized by this ruling.

2026-08-22 live review: openbao/openbao-ui-gateway still publishes bao.coulomb.social through a Traefik Ingress at 92.205.62.239; its gateway pod is running, and the declarative overlay still contains that host. No named operator tunnel replacement is recorded. T09 therefore remains progress and the public-listener retraction acceptance criterion is not met. The platform owner follow-up is 800eb865-404f-4b2b-9363-94247f2bbdad.

2026-08-23 implementation: canonical rapp-openbao revision 8e5347b and railiance-platform registrar revision e8e7119 remove the Ingress from ordinary deploy source, declare exposure.posture: operator, document the named openbao-ui-railiance01 tunnel on local port 18200, and add a guarded Ingress-only retraction with an exact rollback manifest. Server dry-runs pass; the ClusterIP gateway is Ready; ops-bridge reports the named tunnel healthy; and the expected overlay is reachable through the loopback URL. The live public Ingress was deliberately not deleted. T09 remains progress pending admission of the exact loopback callback by KeyCape and the OpenBao role, then one value-safe attended MFA login. The KeyCape request is State Hub message 5e56b413-d8ec-4718-b432-2debc40498ca. None of this authorizes or advances T08 destructive cleanup.

2026-08-23 callback progress: Net Kingdom revision 61aeafe and registered NK-WP-0032 add the exact loopback callback to owner source. The Railiance01 KeyCape client was patched and restarted, and value-safe verification proved the exact redirect plus OIDC discovery live. Railiance Platform revision d18649f adds a silent, tested owner command for the governed role update. Its first attended execution failed closed before child handoff and did not change OpenBao; Warden could not confirm self-revocation. Platform revision 7f71dff records that terminal NO-GO, and ops-warden review is requested in State Hub message 755408d8-2749-43a3-9f2b-8df83bda87af. T09 remains progress: the private path is healthy and the public Ingress remains present, but the OpenBao role callback and attended loopback login are not yet proven.

Safety constraints

• Never initialize or overwrite either OpenBao instance without verified snapshot and rollback evidence.
• Never print, export, compare, or log secret values; use counts, names, capabilities, and cryptographic fingerprints where appropriate.
• Do not switch DNS before private target restore and consumer waves pass.
• Do not revoke transitional access until the replacement path and reversal are proven.
• Final PVC, Secret, Raft, recovery-key, or audit deletion requires separate explicit approval after the retention window.