---
id: CUST-WP-0011
type: workplan
title: "Pragmatic State Hub Migration to railiance01"
domain: infotech
repo: state-hub
status: finished
owner: custodian
topic_slug: custodian
created: "2026-03-11"
updated: "2026-07-06"
finished: "2026-07-06"
state_hub_workstream_id: "967baafb-d92d-405a-ba0b-0d00d37c4940"
supersedes_intent_from: "Migrate Custodian State Hub to ThreePhoenix Cluster"
follow_up_workplan: CUST-WP-0038
---

> **Terminology note:** Historical text in this archived workplan may use the legacy term "workstream". The fleet term is **workplan** (`canon/standards/workplan-terminology-fleet_v0.1.md`).

# Pragmatic State Hub Migration to railiance01

## Goal

Move the Custodian State Hub (FastAPI + PostgreSQL) from the WSL2 operator
workstation to the current railiance01 Kubernetes environment, using the
Railiance production-readiness path that exists now:

- CloudNative PG (`cnpg`) for the State Hub database in the `databases`
  namespace.
- State Hub as an S5 workload in `railiance-apps`.
- Platform/database ownership in `railiance-platform`.
- Access through the existing private tunnel/ops-bridge model, not public
  exposure.
- WSL2 retained as a disaster-recovery fallback until the cluster deployment
  has proven stable.

This is a deliberate pragmatic step. It improves durability and multi-machine
access before the full ThreePhoenix target is ready. The ultimate multi-node,
replicated, long-term cluster goal is preserved in `CUST-WP-0038`.

## Context Update

The original 2026-03-11 version of this workplan targeted a future
ThreePhoenix cluster with Railiance01/02/03, Longhorn, and full HA gates before
starting. That was correct as an end-state, but it blocks useful progress now.

The current Railiance architecture has moved on:

- `railiance-infra/workplans/RAIL-HO-WP-0004-production-readiness.md`
  supersedes the older Bitnami PostgreSQL HA platform baseline.
- CloudNative PG is the deployed database operator.
- `RAIL-HO-WP-0004-T09` is the Railiance-side task for deploying State Hub to
  the cluster, and it still requires human decisions before live data
  migration.

This workplan is now the Custodian-side coordination and safety plan for that
T09 effort.

## Safety Contract

State Hub is irreplaceable episodic memory. This migration may prepare, deploy,
test, and compare as much as needed, but it must not make the cluster the only
source of truth until the explicit cutover gate is satisfied.

Rules:

- A fresh WSL2 backup and restore drill is mandatory before data migration.
- The WSL2 State Hub remains available as rollback until stabilisation passes.
- Any task that changes the live writer endpoint requires explicit human
  approval.
- A failed cluster deploy must leave the WSL2 instance untouched and usable.
- Row counts and key API checks must match before cutover.

## Target Architecture After This Workplan

```
Operator workstation / COULOMBCORE / other agent hosts
  -> local MCP server subprocess
     -> http://127.0.0.1:8000 or configured API_BASE
        -> private tunnel / ops-bridge
           -> railiance01 k3s
              -> state-hub Service
                 -> FastAPI Deployment
                 -> state-hub-db CloudNative PG Cluster
```

Key properties:

- Single-node pragmatic deployment on railiance01.
- No public unauthenticated exposure.
- Database managed by cnpg, not an ad-hoc Postgres StatefulSet.
- WSL2 retained as DR fallback during stabilisation.
- Future multi-node HA and storage replication are deferred to `CUST-WP-0038`.

## Open Human Decisions

Resolve these before T04/T05 can become live migration work:

1. Final State Hub hostname or tunnel-only endpoint.
2. Container registry choice: Gitea registry vs external interim registry.
3. Exposure model: ClusterIP plus tunnel, private ingress, or both.
4. Approval window for freezing WSL2 writes and migrating the production DB.
5. Stabilisation duration before WSL2 can be considered non-primary fallback.

## Tasks

### T01 — Drill WSL2 State Hub backup restore

```task
id: CUST-WP-0011-T01
status: done
priority: high
state_hub_task_id: "b0caf112-dc1d-43a8-9f27-d627dd4aa2bf"
completed: "2026-05-02"
```

Take a fresh State Hub backup from the current WSL2 instance and restore it
into an isolated test PostgreSQL instance.

Minimum checks:

- Restore completes without errors.
- Core table row counts match the live WSL2 database.
- `/state/summary` can be served from the restored copy if wired to a test API.
- Drill result is recorded in State Hub progress and, if useful, episodic
  memory.

**Done when:** backup and restore are proven within 24 hours of live migration
work.

Result: completed 2026-05-02. A fresh dump from `infra-postgres-1` restored
into disposable container `state-hub-restore-test` on `127.0.0.1:5433`.
Application health and summary checks against the restored database returned
HTTP 200. Restored row counts matched production exactly, including 117
workstreams, 989 tasks, 1423 progress events, and 208 token events.

---

### T02 — Align with Railiance deployment plan

```task
id: CUST-WP-0011-T02
status: done
priority: high
state_hub_task_id: "24887dd9-7d50-4cc4-add7-bffa1454b80c"
completed: "2026-05-02"
```

Update the cross-repo plan so this Custodian workplan and
`RAIL-HO-WP-0004-T09` point to the same architecture.

Expected outputs:

- `RAIL-HO-WP-0004-T09` remains the Railiance-side execution task.
- This workplan remains the Custodian-side safety/cutover task list.
- Any stale Longhorn/Postgres StatefulSet assumptions are removed from the
  near-term migration plan.
- The future HA goal is referenced through `CUST-WP-0038`.

**Done when:** both workplans describe compatible responsibilities and gates.

Result: completed 2026-05-02. `RAIL-HO-WP-0004-T09` now names the same
pragmatic railiance01 path: cnpg database, S5 State Hub workload, restore drill
precondition, empty deploy before data copy, explicit human approval before
freezing WSL2 writes, and WSL2 retained as fallback. Full ThreePhoenix HA stays
deferred to `CUST-WP-0038`.

---

### T03 — Build and publish State Hub container image

```task
id: CUST-WP-0011-T03
status: done
priority: high
completed: "2026-06-25"
state_hub_task_id: "79908ade-3e38-451b-a403-2361a16a3f3a"
```

Package this repository as a production image.

Requirements:

- Dockerfile builds from the current Python/uv project.
- Alembic and runtime dependencies are available inside the image.
- Image exposes the FastAPI service on port 8000.
- Image tag is pushed to the chosen registry.
- Build provenance is documented in the commit/workplan.

**Done when:** railiance01 can pull the image and a dry-run deployment resolves
it.

Progress 2026-05-03: added `Dockerfile`,
`.dockerignore`, and `docs/container-image.md`. Built
local image `state-hub:local` successfully:
`sha256:e96dbd1e7d2b63e4fb17584c8c2216088a2c9937bfe880c2ad565c7a9f51c0fc`
(~106 MB). Verified container `/state/health` returns HTTP 200 against the
current database when run locally with host networking. Verified Alembic is
available in-image and reports current revision `r5m6n7o8p9q0 (head)`.

Progress 2026-05-03: registry target decision resolved to the self-hosted
Gitea registry. A local SSH tunnel to the NodePort can reach Gitea, but Docker
login/push still receives HTTP 404 from `/v2/`. Runtime inspection shows the
live Gitea `app.ini` has no `[packages]` section, so package registry
enablement/routing must be applied before publishing `state-hub:local`.

Progress 2026-05-15: rebuilt the image from current State Hub sources as
`state-hub:local` with digest
`sha256:039d29654ccb3754c6ecdbe497c6364bbd8452edcdcb7fa937dd9debf5b734ff`
(106004480 bytes). Verified `/state/health` returns
`{"status":"ok","db":"connected"}` from a temporary container on host port
18000 and confirmed in-image Alembic reports `t7o8p9q0r1s2 (head)`. Build
provenance is recorded in `docs/container-image.md`.

Completed 2026-06-25: adapted the Dockerfile for the current `hub-core`
runtime dependency by installing it from the named Docker build context
`hub_core_src=/home/worsch/hub-core`. Built current commit `b536741` as
`state-hub:local`, `state-hub:b536741`, and
`gitea.coulomb.social/coulomb/state-hub:b536741`. Verified in-image imports,
Alembic head `e9f0a1b2c3d4`, and `/state/health` from a temporary container on
port 18082. Pushed the image to the self-hosted Gitea registry with digest
`sha256:3184dfd67f127cf8bd5303d7a210d6dc32e7ab05a5da5d51eab5b9a37dab4d4e`
and verified railiance01 can pull it with `sudo crictl pull`.

---

### T04 — Define State Hub database and app manifests

```task
id: CUST-WP-0011-T04
status: done
priority: high
completed: "2026-06-25"
state_hub_task_id: "a7baf2eb-abd7-4aa3-b2cb-a5370ac09844"
```

Create the cluster-side deployment assets using current Railiance boundaries:

- `railiance-platform`: `state-hub-db` cnpg cluster and database credentials.
- `railiance-apps`: State Hub Deployment, Service, ConfigMap, Secret/External
  Secret reference, and optional private Ingress.
- Health probes use `GET /state/health`.
- Environment includes `DATABASE_URL` and any required API settings.

**Done when:** manifests lint/apply in a non-destructive dry run and ownership
boundaries are documented.

Completed 2026-06-25: added a source-owned Railiance deployment handoff under
`deploy/railiance/` with platform manifests for `state-hub-db` CNPG, database
credentials, database NetworkPolicies, an app Helm chart, production values, and
a `state-hub-env` Secret template. Added Make targets for rendering,
client-side dry-run validation, and namespace-aware server-side dry-run
validation. Verified:

- `make railiance-state-hub-render`
- `make railiance-state-hub-client-dry-run`
- `make railiance-state-hub-server-dry-run`

The server dry-run validates platform resources and the Namespace manifest
against the live cluster API. Because the `state-hub` namespace does not yet
exist, it explicitly falls back to client dry-run for namespaced app manifests;
Kubernetes cannot persist a dry-run Namespace for subsequent namespaced
server-side validation. Ownership boundaries and promotion notes are documented
in `deploy/railiance/README.md`.

---

### T05 — Deploy empty State Hub and run migrations on railiance01

```task
id: CUST-WP-0011-T05
status: done
priority: high
completed: "2026-06-25"
state_hub_task_id: "a307dd46-a8e2-49df-b016-c187759ebcf1"
```

Deploy State Hub against an empty `state-hub-db` cnpg database and run Alembic
migrations in the cluster environment.

Checks:

- Pod reaches Ready.
- `/state/health` returns healthy through the intended private access path.
- Alembic reports head.
- Logs show no repeated crash/restart loop.

**Done when:** an empty but structurally valid State Hub runs on railiance01.

Completed 2026-06-25: deployed an empty State Hub stack to railiance01.
Created the `state-hub` namespace, generated live-only database and app runtime
Secrets, created the dedicated `state-hub-db` CNPG cluster, and applied database
NetworkPolicies. Fixed the State Hub database egress policy to allow the
in-cluster Kubernetes API service on TCP 443 as well as 6443, which CNPG
needed during initdb. Ran Alembic migrations in a one-shot Kubernetes Job
using image `gitea.coulomb.social/coulomb/state-hub:b536741`; migrations
completed through `e9f0a1b2c3d4 (head)`. Installed the Helm release
`state-hub` into the pre-created namespace with `namespace.create=false`.
Verified Deployment rollout, zero pod restarts, service creation, pod logs,
in-pod Alembic current revision, and `/state/health` via temporary port-forward
returning `{"status":"ok","db":"connected"}`.

---

### T06 — Restore WSL2 data copy into cluster and compare

```task
id: CUST-WP-0011-T06
status: done
priority: high
completed: "2026-06-25"
state_hub_task_id: "03753b88-824c-4448-97b2-f7315d145060"
```

Restore a fresh WSL2 dump into the cluster database while WSL2 remains the live
source of truth.

Required comparison:

- Table row counts match.
- Representative workstreams, tasks, decisions, progress events, repos, and
  token events are queryable.
- Dashboard and MCP summary calls return expected data through the cluster API.
- Any mismatch is investigated before proceeding.

**Done when:** cluster data is a verified copy of WSL2, but not yet the only
writer.

Completed 2026-06-25: restored a fresh WSL2 State Hub PostgreSQL dump into
the cluster `state-hub-db` database while WSL2 remained the live source of truth.
Scaled the cluster API to zero during restore, restored the newer dump after a
one-row live `progress_events` drift was detected, then scaled the API back
to one replica. Verified all public table row counts match between WSL2 and
the cluster, including `workplans=569`, `tasks=3673`,
`decisions=71`, `progress_events=6232`, `managed_repos=71`,
and `token_events=1881`. Representative restored-data checks found this
workplan, the T06 task, decisions, progress events, the `state-hub` repo,
and token events queryable. Cluster API `/state/health` returned
`{"status":"ok","db":"connected"}` through a temporary port-forward,
and `/state/summary` returned expected totals. Temporary dump files and
port-forwards were removed after verification.

---

### T07 — Cut over private access to cluster State Hub

```task
id: CUST-WP-0011-T07
status: done
priority: medium
state_hub_task_id: "ff1de25e-c301-4b86-9420-84dfe72e565e"
```

With human approval, freeze WSL2 writes, take a final dump, restore it to the
cluster, compare counts again, and redirect the active private access path to
the cluster API.

Accepted approaches:

- Keep local MCP config pointed at `http://127.0.0.1:8000` and move that port
  to an ops-bridge/SSH tunnel.
- Or set the MCP server `API_BASE` to the chosen private cluster endpoint.

**Done when:** `get_state_summary()` and dashboard live data are served by the
cluster State Hub, and WSL2 is no longer receiving normal writes.

Result: completed 2026-07-03 with explicit operator approval ("go forward with
1 and 2 and 3"). Sequence: cluster image refreshed to `ea1fd23` (adds the
add_progress_event workplan_id alias; schema head `e9f0a1b2c3d4` unchanged);
WSL2 uvicorn stopped (freeze); final `pg_dump` from `infra-postgres-1`
restored into CNPG `state-hub-db`/`state_hub` with `SET ROLE state_hub`
ownership; row counts matched exactly (633 workplans, 3964 tasks, 8192
progress events, 14 topics, 1933 token events); private access rewired via
ops-bridge `state-hub-primary` forward tunnel so `127.0.0.1:8000` serves the
cluster hub. The railiance01 automation chain (`:18000`) verified intact.
First primary-served write: progress event `56aab39b`. WSL2 fallback restart:
`bridge down state-hub-primary && cd ~/state-hub && make api`.

---

### T08 — Stabilise with WSL2 retained as fallback

```task
id: CUST-WP-0011-T08
status: done
priority: medium
state_hub_task_id: "e06a59a0-5310-4c1c-9ba5-7cfaadda62e2"
completed: "2026-07-06"
```

Run the cluster State Hub as primary while keeping the WSL2 instance available
as a fallback.

Monitor:

- State Hub pod restarts.
- cnpg cluster health.
- Backup job success.
- Dashboard and MCP behavior from each operator machine.
- Consistency sync behavior for file-backed workplans.

**Done when:** the agreed stabilisation window passes without data loss or
unresolved operational defects.

Completed 2026-07-06: three days post-cutover (2026-07-03) with no data loss
or blocking operational defects. Verification on 2026-07-06:

- `state-hub-primary` ops-bridge tunnel `connected`; `GET /state/health`
  returns `{"status":"ok","db":"connected"}`.
- Deployment `state-hub` 1/1 Ready (one restart 2d8h ago, currently stable).
- CNPG `state-hub-db` reports `Cluster in healthy state`.
- Hub totals growing normally (635 workstreams, 3975 tasks); recent writes
  from 2026-07-04 confirmed.
- Manual `POST /consistency/sweep/remote-all` completed with `exit_code: 0`.
- WSL2 fallback path documented and retained (`bridge down state-hub-primary &&
  make api`).

Follow-ups outside this workplan: scheduled activity-core sweeps paused after
cutover (service-inventory gap); CNPG scheduled backups not yet configured.

---

### T09 — Document operating model and defer final WSL2 retirement

```task
id: CUST-WP-0011-T09
status: done
priority: low
state_hub_task_id: "d75a2d49-f3b1-4bdd-b9e1-a1c6a9744681"
completed: "2026-07-06"
```

Document the new operating model:

- How agents reach State Hub.
- How backups and restores work.
- How to roll back to WSL2 if needed.
- Which parts remain pragmatic/single-node.
- Which long-term requirements moved to `CUST-WP-0038`.

Do not permanently retire WSL2 in this workplan unless a separate human
decision is recorded. Retirement belongs after proven stability or in the
future HA workplan.

**Done when:** runbooks and project instructions match the deployed reality.

Completed 2026-07-06: added `docs/cluster-operating-model.md` covering cluster
access (`state-hub-primary`), remote tunnel mesh, manual backup/restore paths,
WSL2 rollback procedure, consistency sync, pragmatic single-node limits, and
`CUST-WP-0038` deferrals. Updated `README.md`, `AGENTS.md`,
`docs/onboarding.md`, and `infra/README.md` to reference the new runbook.
WSL2 retirement explicitly deferred.

## References

- `railiance-infra/workplans/RAIL-HO-WP-0004-production-readiness.md`
- `RAIL-HO-WP-0004-T09` — Railiance-side State Hub deployment task
- `CUST-WP-0038` — future full ThreePhoenix HA State Hub migration
- Constitution constraint: production data migration and fallback retirement
  require explicit human approval
