core-hub/workplans/CORE-WP-0011-railiance01-production-relocation.md

---
id: CORE-WP-0011
type: workplan
title: "Relocate Core Hub production from CoulombCore to railiance01"
domain: infotech
repo: core-hub
status: finished
owner: codex
topic_slug: infotech
created: "2026-08-20"
updated: "2026-08-21"
parent_project: prj-state-hub-retirement
parent_workplan: SHR-WP-0002
related:
  - CORE-WP-0005
  - CORE-WP-0007
  - CORE-WP-0010
  - HUB-WP-0004
---

# Relocate Core Hub production from CoulombCore to railiance01

## Goal

Move the existing Core Hub production service at `hub.coulomb.social` from
CoulombCore to railiance01 before CoulombCore is decommissioned. Preserve the
current `/api/v2` contract, data, hostname, credentials boundary, and rollback
evidence. Do not make runtime absorption into hub-core a prerequisite.

## Decision and constraints

- Use the interim-move shape requested by `SHR-WP-0002-T03`: package and move
  Core Hub as-is, then resolve long-term absorption under `CORE-WP-0010` and
  `HUB-WP-0004` on their own architecture schedule.
- Use the governed Repo Manager rapp path. Because the live deployment truth is
  a Helm chart, initialize a managed package instead of asking `rapp wrap` to
  invent or ingest Helm content; retain this repo's manifests as source evidence
  until the package is verified.
- Treat the CoulombCore decommission date as a scheduling input, not as a
  reason to collapse the hub-core library-vs-thin-host decision.
- Any traffic flip, final data freeze/transfer, or old-runtime removal requires
  explicit operator approval and a rehearsed rollback.

## Inventory source and target runtime

```task
id: CORE-WP-0011-T01
status: done
priority: high
```

Capture the live CoulombCore image, Deployment, Service, Ingress, database,
secret references, resource policy, and tunnel/DNS ownership. Compare them to
railiance01 and the checked-in `k8s/railiance-production/` manifests.

Local review on 2026-08-20 confirms that the repo already carries a namespace,
migration Job, API Deployment, Service, Ingress, health probes, and the
historical migration/cutover runbook. The manifests need a live-state diff and
modern platform-policy review before reuse. The retirement project reports no
`core-hub` namespace or Deployment on railiance01. A read-only preflight on
2026-08-20 confirmed the current Kubernetes context reaches the Ready k3s node
`239.62.205.92.host.secureserver.net` (`92.205.62.239`, Kubernetes
`v1.35.1+k3s1`), has no `core-hub` namespace or resources, and retains an
`inter-hub` namespace. Pin the CoulombCore decommission date while this task is
in progress so later gates have explicit deadlines. The retirement deadline is
now pinned to **2026-08-31** (11 days from workplan creation). Source-cluster
live state and target platform-policy comparison remain open.

The registry audit found that production/staging manifests, runbooks, and
Forgejo CI already publish and pull `forgejo.coulomb.social/coulomb/core-hub`,
but the local Makefile default and deployed-smoke evidence still named the
CoulombCore-hosted Gitea registry. Those two active references were changed to
Forgejo on 2026-08-20. The only remaining Gitea image reference is historical
evidence in finished `CORE-WP-0004`; confirm the live source Deployment image
and digest before declaring the dependency closed.

Completed after the documented local API tunnel proved stale: existing direct
SSH access provided a metadata-only `kubectl`/Helm path without retrieving a
new credential. Production is Ready on CoulombCore Kubernetes `v1.30.3+k3s1`,
Helm chart `0.1.0`, using mutable tag `a33191a` whose pulled image ID is
`sha256:5e1334eb25975b5f7792d2ae9289898586224133896c6aa55bd27cdc776fb10c`.
It has one combined environment Secret, empty security contexts, the default
ServiceAccount and no NetworkPolicies. Health/readiness pass publicly. Helm
reports release revision 1 while the Deployment reports revision 2, so source
rollback must retain the actual Deployment/ReplicaSet state rather than rely on
Helm history alone. Secret inspection recorded key names and byte sizes only;
no value was read. Full evidence and the target-package diff are in
`docs/evidence/core-hub-coulombcore-inventory-2026-08-20.md`.

## Build the relocatable application package

```task
id: CORE-WP-0011-T02
status: done
priority: high
```

Wrap the current service through the governed rapp path. Pin the source
revision and image digest, express namespace/runtime/database/secret
dependencies without credential values, and add render/package checks. Reuse
the current container and `/healthz`, `/readyz`, and deployed-smoke surfaces.

Done when the package renders deterministically, policy checks pass, and its
rendered workload is equivalent to the live service except for documented
railiance01 platform adaptations.

Completed locally on 2026-08-20 in `/home/worsch/rapp-core-hub`. Repo Manager
initialized a Helm-managed package because the live deployment truth is the
existing `railiance-apps/charts/core-hub` chart; `rmgr rapp wrap` deliberately
refuses to invent or ingest Helm charts. The package absorbs that chart and
adds an immutable Forgejo digest pin, safe-private Ingress default,
non-root/read-only security contexts, ServiceAccount hardening,
default-deny/DNS/Postgres/Traefik NetworkPolicies, deterministic render,
Alembic migration, live smoke, and Helm rollback surfaces. Evidence:

- package tests: 5 passed;
- Helm lint: passed;
- family validator: 4 declarations ok;
- railiance01 Helm/Kubernetes server dry-run: passed;
- current image pin:
  `sha256:0db6f7c646c6d4c1fbc844dc266fcf883584637e1c1f47802efe1c6a98558d3c`.

Published and registered on 2026-08-20 after the operator created the Forgejo
stub. Local history was attached without discarding either root. Private
Forgejo `main` now includes credential-contract release commit `03effc8`.
Chart `0.3.0` separates the static API-token Secret from runtime and migration
database lease Secrets and supervises the application through mounted-file
rotation; seven tests prove separation, safe URL construction, no credential
output, and restart signaling. State Hub registers active
repo `rapp-core-hub` as `445ea292-70bb-435c-8c69-c4cb57a6985b`, with the
`bnt-lap001:/home/worsch/rapp-core-hub` host path, the canonical stub-root Git
fingerprint, and Core Hub's topic ID. The package uses registered workplan
prefix `RAPPCOREHUB-WP-`; consistency passes with only the expected warning
that this workstation is not the production workplan-UUID registrar.

## Deploy shadow runtime and rehearse data transfer

```task
id: CORE-WP-0011-T03
status: done
priority: high
```

Provision the railiance01 namespace, database role/database, governed secret
delivery, migration Job, Deployment, and Service without moving production
traffic. Rehearse backup/restore or logical transfer from CoulombCore, compare
table and row counts, and run public plus authenticated consumer smokes against
the private target endpoint.

Done when the shadow runtime is healthy, data reconciliation is repeatable,
and ops-hub plus activity-core smokes pass against railiance01.

Preflight completed 2026-08-20 without applying resources. ESO CRDs and the
working audit-core two-store pattern exist on railiance01, but the `core-hub`
namespace, Core Hub ClusterSecretStores, projected Secrets, and accepted
`rapp-postgres` consumer declaration do not. The exact target image digest
pulls and imports Core Hub, Uvicorn, and Alembic under UID/GID 10001 with a
read-only root filesystem. Package tests, family validation and server dry-run
pass. This task is `wait` on `RPF-WP-0021-T02/T04`; do not substitute placeholder
Secrets or a standing database password. The executable gate order, evidence
requirements, private smokes, live lease-rotation proof and rehearsal rollback
are recorded in `docs/deployment/railiance01-shadow-runbook.md` so work can
resume immediately after platform admission.

Completed 2026-08-21. `RPF-WP-0021` admitted Core Hub as the fourth and final
`platform-pg` consumer, established separate static API-token and dynamic
runtime/migration database lanes, and restored a consistent 39-row source
archive. The source `public` tables were explicitly reconciled into the
accepted target `core_hub` schema; all eight tables, Alembic revision
`20260627_0002`, ownership and runtime grants matched before shadow smoke
activity. The migration Job completed, Helm revision 2 deployed package chart
`0.3.1` at `rapp-core-hub` commit `7cfa8bb`, and no Ingress exists.

The hardened private verifier and authenticated smoke
`20260820224419-43dc05` passed all ten checks. A fresh runtime lease then
changed the projected Secret and application child PID while the pod UID and
container restart count remained stable; readiness returned without exposing
a credential. The shadow now contains the expected smoke-created rows, while
the public endpoint and source writer remain on CoulombCore. Repo-local
evidence is in
`docs/evidence/core-hub-railiance01-private-shadow-2026-08-21.md`.

## Cut over production traffic

```task
id: CORE-WP-0011-T04
status: done
priority: high
```

With explicit operator approval, freeze writes for the bounded final-transfer
window, transfer and reconcile the final data, then move
`hub.coulomb.social` and the active ops-bridge tunnel from CoulombCore to
railiance01. Verify TLS, health/readiness, public catalogs, protected routes,
operator console, consumer smokes, and the first post-cutover write.

Rollback must restore the prior traffic/tunnel target and database write path;
the old runtime stays intact until stabilization closes.

Cutover progress 2026-08-21: explicit operator approval entered this gate.
Helm revision 3 enabled the chart `0.3.1` Ingress on railiance01 while public
DNS still targeted CoulombCore. The source application was reconnected with
database-default read-only mode and continues serving health/readiness, so
stale-DNS readers remain available and writers fail closed. A fresh frozen
archive was captured, the backed-up target rehearsal rows were replaced, and
the source `public` tables were mapped into target schema `core_hub`. All eight
table counts and order-independent row-content digests match; Alembic
`20260627_0002`, migration, the hardened verifier, direct target health/
readiness, unauthenticated denial, and authenticated protected read pass.

Completed 2026-08-21 after the operator changed the authoritative IONOS A
record from `92.205.130.254` to `92.205.62.239`. All four authoritative name
servers and Google/Quad9 returned railiance01; Cloudflare retained the old
address under its pre-change TTL, so CoulombCore remains read-only while that
cache drains. A temporary, challenge-path-only bridge on CoulombCore and a
solver-label/port-specific target NetworkPolicy allowed cert-manager to issue
the production certificate without reopening either application or database
access; both temporary resources were removed immediately afterward.

The Let's Encrypt certificate verifies for `hub.coulomb.social`. Forced target
checks passed health/readiness, all four public catalogs, protected-route and
operator-console auth boundaries, authenticated reads, and the hardened live
verifier. The first post-cutover write returned 201 and read back as event
`e56bca14-51f8-4a73-bdab-f660e360fb14`. The ops-hub public gate and the
deployed activity-core stabilization resolver both passed against railiance01.
No active ops-bridge tunnel fronts the public hostname; it is a direct IONOS
DNS route. The obsolete local `core-hub-staging-coulombcore` tunnel is deferred
to T05. Current evidence and exact rollback state are in
`docs/evidence/core-hub-railiance01-cutover-2026-08-21.md`.

## Stabilize and release CoulombCore dependencies

```task
id: CORE-WP-0011-T05
status: done
priority: high
```

Run a time-bounded stabilization window with scheduled health and consumer
checks. After operator sign-off, remove Core Hub from the CoulombCore shutdown
critical path, retire `core-hub-staging` and any obsolete tunnel there, and
record retained backup/rollback evidence and residual owners. Do not archive
this repo unless the separate absorption workplan has also completed.

Stabilization started 2026-08-21. Cloudflare, Google, Quad9 and the local
resolver now all return `92.205.62.239`; normal public HTTPS reaches
railiance01 with valid TLS. Helm revision 4 runs package chart `0.3.2`, which
permanently carries the solver-label/kube-system/TCP-8089 NetworkPolicy needed
for future certificate renewals, and the post-upgrade verifier passes. The
CoulombCore Deployment remains Ready for rollback, but database-level
`default_transaction_read_only=on` was reverified. Keep the source frozen until
the bounded observation window and operator sign-off close this task.

Completed 2026-08-21 after multiple original DNS TTLs and two independent
consumer snapshots. The target recorded 989 successful requests, four expected
401s and no 5xx/error match; its pod stayed Ready with zero restarts, all
ExternalSecrets and TLS remained Ready, and the post-retirement package
verifier passed. Completed target and encrypted source backups were identified
before both CoulombCore Core Hub Deployments were scaled to zero. The obsolete
staging tunnel was removed from configuration and its exact supervisor/SSH
processes were stopped; port 18010 is closed. Source Services, Ingresses,
Secrets and the read-only database remain only as reversible retention
artifacts and no longer block CoulombCore shutdown. Closeout evidence is in
`docs/evidence/core-hub-railiance01-stabilization-2026-08-21.md`.

## Acceptance

- [x] Decommission deadline and live source/target inventory are recorded
- [x] Governed rapp package renders and passes policy checks
- [x] Shadow restore/reconciliation and consumer smokes pass on railiance01
- [x] Production traffic and tunnel move with explicit approval and rollback
- [x] Stabilization passes and CoulombCore no longer hosts Core Hub dependencies
- [x] `CORE-WP-0010` remains independently sequenced behind `HUB-WP-0004`