railiance-master/workplans/RMASTER-WP-0020-openbao-migration-to-reef-railiance.md

---
id: RMASTER-WP-0020
type: workplan
title: "Migrate authoritative OpenBao from CoulombCore to reef-railiance"
domain: financials
repo: railiance-master
status: blocked
owner: codex
topic_slug: railiance
created: "2026-07-30"
updated: "2026-08-18"
depends_on:
  - NK-WP-0022
related:
  - ADR-0008
  - RMASTER-WP-0023
state_hub_workstream_id: "0616a297-18c5-4c4e-a4fc-69135b3f9a15"
---

# RMASTER-WP-0020 - OpenBao migration to reef-railiance

Move the authoritative OpenBao runtime from CoulombCore onto
`rail-kubernetes` **hosted by** `reef-railiance` (`rapp-openbao`),
preserving every secret,
policy, auth method, identity, lease-relevant contract, audit record, and
recovery capability. This is a stateful security control plane and must never
run on scale-to-zero Knative.

## Current-state review

- `bao.coulomb.social` resolves to CoulombCore (`92.205.130.254`).
- CoulombCore runs initialized, unsealed OpenBao 2.5.4 with integrated Raft,
  5 GiB data PVC, 2 GiB audit PVC, and the public UI gateway.
- Every railiance01 ClusterSecretStore currently uses
  `https://bao.coulomb.social`.
- railiance01 has an OpenBao 2.5.3 StatefulSet with local Raft PVCs, but it is
  uninitialized, sealed, has no ingress, and serves no consumers.
- Cross-cluster ESO access currently uses transitional AppRoles. Once OpenBao
  is local, same-cluster Kubernetes authentication is the preferred steady
  state.
- The reef is a single node. Migration improves topology consolidation but
  does not create infrastructure high availability; backup and rollback are
  mandatory.

## T01 - Inventory authoritative state and consumer contracts

```task
id: RMASTER-WP-0020-T01
status: done
priority: high
state_hub_task_id: "0d40e879-ae4e-45de-aa31-81bf110924dc"
```

Capture non-secret inventories of mounts, auth methods, policies, roles,
entities, aliases, audit devices, Raft peers, versions, storage, ingress,
ClusterSecretStores, ExternalSecrets, direct clients, automation scripts, and
backup jobs. Record counts and fingerprints without secret values. Identify
all consumers that bypass ESO or depend on the public address.

Done when source, empty target, and every consumer have a machine-readable
dependency map plus explicit rollback owners.

Completed 2026-08-03. The non-secret source, target, and consumer dependency
map is recorded in
`docs/evidence/openbao-migration-inventory-2026-08-03.json`. It includes six
ClusterSecretStores, nine ExternalSecrets, state/auth counts, storage and
version topology, and rollback ownership without secret values.

## T02 - Ratify migration, seal, and custody design

```task
id: RMASTER-WP-0020-T02
status: done
priority: high
state_hub_task_id: "c266dce7-7d89-454a-a299-48fc169efea1"
```

Choose and document a supported Raft snapshot migration procedure; align target
OpenBao version before restore. Define unseal/recovery-key custody, TLS and DNS
continuity, audit retention, target PVC handling, service identity, backup
encryption, operator access, and emergency rollback. Use ops-mason for any new
credential, policy, AppRole, or Kubernetes-auth construction plan and retain
its executive approval gate.

Done when the procedure has preconditions, abort points, rollback commands,
RTO/RPO, and no secret material in Git or State Hub.

Completed 2026-08-03. The ratified procedure, custody boundary, consumer
waves, abort points, rollback, retention, RTO, and RPO are documented in
`docs/openbao-coulombcore-to-railiance01-runbook.md`. The empty target release
was aligned to OpenBao 2.5.4, matching the source.

## T03 - Prove backup and isolated restore before cutover

```task
id: RMASTER-WP-0020-T03
status: done
priority: high
state_hub_task_id: "88671108-cf8c-4de6-b1ce-06e99b1c3d70"
```

Take an encrypted authoritative Raft snapshot and matching configuration/audit
evidence through approved custody. Restore it into an isolated non-serving
instance at the target version. Verify inventory counts, representative
exact-path reads, sibling denial, auth-method metadata, audit continuity, and
restart/unseal behavior.

Done when restore evidence is machine-readable and rollback does not depend on
the live source remaining healthy.

2026-08-03: Created a fresh authoritative snapshot, encrypted it before
offsite upload, removed plaintext staging, and recorded only fingerprints.
A second fresh snapshot restored successfully into an isolated OpenBao 2.5.4
pod on railiance01. The restored instance correctly resealed under the source
Shamir barrier. The remaining semantic and repeated restart/unseal proof needs
two of the three existing source shares through the attended custody ceremony;
no supported automated custody route exposes those shares.

Completed 2026-08-03. The operator supplied two rotated shares through hidden
prompts. Two restart/unseal cycles passed. The restored cluster became active
with the source cluster ID; inventory counts matched; the existing scoped Qonto
AppRole proved exact-path `read` and sibling `deny`; and declarative file audit
output was non-empty. Machine-readable evidence is in
`docs/evidence/openbao-isolated-restore-2026-08-03.json`.

## T04 - Restore authoritative state on railiance01

```task
id: RMASTER-WP-0020-T04
status: done
priority: high
state_hub_task_id: "d04d7fdd-bd00-407a-8954-101b680ed327"
```

Freeze administrative writes for a bounded window, take the final snapshot,
replace the empty target state through the ratified procedure, restore
configuration and audit continuity, and bring the target up privately. Do not
change public DNS or consumers yet.

Done when target semantic probes match the source and repeated restart/unseal
tests pass through a private endpoint.

Completed 2026-08-03. A final source snapshot was encrypted and uploaded
offsite (plaintext SHA-256
`ac5e27536c01c3a019c3dde1069866569112c76d8ab42ec2bdb45724cc9b13f8`,
encrypted SHA-256
`22003d8fcb12ef17d6edafad9358ed2b0e538b21a8daee86445d9b85482f9161`).
The still-empty railiance01 target was restored privately, temporary init
material and plaintext staging were removed, and two attended target
restart/unseal cycles passed. The target is active on OpenBao 2.5.4 with the
source cluster ID and a non-empty persistent audit file. CoulombCore remains
live and no public DNS changed.

## T05 - Migrate consumers to local workload identity

```task
id: RMASTER-WP-0020-T05
status: done
priority: high
state_hub_task_id: "f0a82e4e-8074-4a22-b1e9-f01ec0ff76a1"
```

Move ClusterSecretStores and other railiance01 clients in bounded waves.
Replace transitional cross-cluster AppRoles with namespace-restricted
same-cluster Kubernetes-auth roles where supported. For each wave prove
ExternalSecret readiness, exact-path access, sibling denial, rotation, workload
rollout, and reversal before continuing.

Done when all railiance01 consumers use the local private service and no
runtime dependency requires the CoulombCore API.

2026-08-03: Wave 1 (`reuse`) moved to
`http://openbao.openbao.svc:8200`. The ClusterSecretStore remained Valid, a
forced ExternalSecret refresh completed as SecretSynced/Ready, and the
declarative owner change was committed in `railiance-platform@33b36e8`.

Completed 2026-08-03. Activity-core, Qonto, Forgejo, user-engine, and SSO were
moved in separate waves. All six ClusterSecretStores are Valid on the private
service and all nine ExternalSecrets refreshed as SecretSynced/Ready; affected
Knative, application, Forgejo, and identity pods remained healthy. Declarative
changes are in `railiance-platform@741f209` and `net-kingdom@0ec6f8c`.
Transitional tokens/AppRoles remain temporarily available for bounded rollback;
same-cluster Kubernetes identity conversion remains post-cutover hardening.

## T06 - Cut over public OpenBao DNS and operator access

```task
id: RMASTER-WP-0020-T06
status: done
priority: high
state_hub_task_id: "f458d110-5fdd-465a-8a23-4ada1051fb12"
```

After T03-T05, move `bao.coulomb.social` to railiance01 with TLS continuity.
Verify status, UI gateway, OIDC/operator login, scoped automation, audit
delivery, backup, revocation, rotation, and negative access through normal DNS.
Keep the source sealed or write-frozen and immediately recoverable.

Done when the observation window passes without source traffic or divergence.

2026-08-03: Deployed the rapp-openbao UI overlay gateway, Traefik middleware,
Ingress, and cert-manager Certificate on railiance01. The gateway is Ready.
The ACME HTTP-01 challenge is correctly pending because public DNS still points
to CoulombCore. No automated DNS credential route exists; change the A record
to `92.205.62.239`, then verify certificate, UI/OIDC, API, audit, backup, and
negative access before retiring the source.

Completed 2026-08-03. All authoritative and sampled public resolvers converged
on `92.205.62.239`. The rapp-openbao gateway serves HTTP 200 for UI and health
with a valid cert-manager-owned Let's Encrypt certificate. The target remains
active/unsealed with the preserved cluster ID; its audit file is growing and
all six stores plus nine ExternalSecrets remain Ready. The source OIDC
initiation behavior was compared directly and no target-only regression was
found.

## T07 - Retire CoulombCore OpenBao reversibly

```task
id: RMASTER-WP-0020-T07
status: done
priority: high
state_hub_task_id: "db2443e6-bad7-4a71-b45f-d4b81099df15"
```

Scale the CoulombCore OpenBao and UI gateway to zero after successful cutover.
Retain manifests, encrypted final snapshot, PVCs, Secrets, TLS material, and
documented rollback for at least 14 days. Confirm no DNS, monitor, client,
tunnel, catalog entry, or automation still targets the old runtime.

Done when CoulombCore serves no OpenBao traffic and rollback remains tested.

Completed 2026-08-03. After more than one DNS TTL and multi-resolver
convergence, the old gateway showed only its Kubernetes readiness probes and
no external traffic. CoulombCore StatefulSet `openbao` and Deployment
`openbao-ui-gateway` were scaled to zero and their pods terminated. Data and
audit PVCs, Services, Ingress, TLS and Helm Secrets, manifests, rotated unseal
custody, and encrypted final snapshot remain retained for rollback. A
post-retirement public probe returned HTTP 200 from railiance01 with valid TLS.

## T08 - Final cleanup and closure

```task
id: RMASTER-WP-0020-T08
status: wait
priority: medium
state_hub_task_id: "1dc50d98-64e8-4755-a52e-2b94b765019e"
```

After retention expiry, a successful railiance01 disaster-recovery drill, and
new explicit approval for destructive deletion, remove stale CoulombCore
OpenBao resources and transitional AppRole/bootstrap material. Update topology,
runbooks, routing catalog, State Hub, and the CoulombCore retirement plan.

Done when reef-railiance is the sole authoritative OpenBao location and all
source-backed records agree.

Retention gate opened 2026-08-03. Do not delete the retained CoulombCore PVCs,
Secrets, Helm records, Ingress, or Services before 2026-08-17. Final deletion
also requires a successful railiance01 disaster-recovery drill and fresh
explicit destructive-cleanup approval.

2026-08-04: The workplan moved to backlog for the shortened retention window.
Activity Core one-shot definition
`activity-definitions/openbao-retention-closeout.md` reactivates T08 at 08:00
Europe/Berlin on 2026-08-17 by emitting a claimable ops run. The schedule never
performs destructive cleanup itself.

2026-08-14: Workplan moved from `backlog` to `blocked`. T08 is `wait`.
The three gates are still closed: retention until 2026-08-17, a post-cutover
railiance01 disaster-recovery drill (the 2026-08-03 isolated restore is T03,
not this), and fresh explicit approval for destructive CoulombCore deletion.
Do not reopen until those three are true. The 2026-08-17 one-shot may move
the workplan to `active`; it still must not delete anything.

## T09 - Retract public OpenBao listener

```task
id: RMASTER-WP-0020-T09
status: progress
priority: high
state_hub_task_id: "287920d3-b038-495f-b521-d16cfc7ba941"
```

Depends on T08 remaining blocked until its three gates open; this task
can be routed sooner.

T06 published `bao.coulomb.social` on Railiance01 for DNS continuity.
ADR-0008 later split admission from exposure. The 2026-08-15 snapshot
lists that name as a **routed close, not a grant**. Target inventory is
`public_ingress: false`. Consumers already use
`openbao.openbao.svc:8200`.

Route to `rapp-openbao` / `railiance-platform`: drop public Ingress,
keep ClusterIP, set `exposure.posture: operator` with a named
ops-bridge tunnel for the UI. Do not implement NetworkPolicy here.

**Done when:** `bao.coulomb.social` is not a public listener we publish,
the runbook no longer treats public DNS as the steady state, and the
declaration carries `operator` (or `private`) rather than an implicit
public grant.

2026-08-18: `railiance-platform` confirmed from live cluster state that
`bao.coulomb.social` resolves to Railiance01 and is published by the
`openbao/openbao-ui-gateway` Traefik Ingress with a trusted certificate,
HSTS, and rate limiting. This is the deliberate T06 migration-continuity
surface, not accidental drift. The later operator-accepted ADR-0008 decision
is nevertheless controlling: the snapshot records this name as a routed
close, no exposure grant exists, and the target remains
`public_ingress: false`. T09 is therefore in progress and routed back to the
owning repo to retract the public listener while preserving the ClusterIP and
establishing the named operator tunnel. No destructive T08 cleanup is
authorized by this ruling.

## Safety constraints

- Never initialize or overwrite either OpenBao instance without verified
  snapshot and rollback evidence.
- Never print, export, compare, or log secret values; use counts, names,
  capabilities, and cryptographic fingerprints where appropriate.
- Do not switch DNS before private target restore and consumer waves pass.
- Do not revoke transitional access until the replacement path and reversal
  are proven.
- Final PVC, Secret, Raft, recovery-key, or audit deletion requires separate
  explicit approval after the retention window.