--- id: CORE-WP-0011 type: workplan title: "Relocate Core Hub production from CoulombCore to railiance01" domain: infotech repo: core-hub status: finished owner: codex topic_slug: infotech created: "2026-08-20" updated: "2026-08-21" parent_project: prj-state-hub-retirement parent_workplan: SHR-WP-0002 related: - CORE-WP-0005 - CORE-WP-0007 - CORE-WP-0010 - HUB-WP-0004 --- # Relocate Core Hub production from CoulombCore to railiance01 ## Goal Move the existing Core Hub production service at `hub.coulomb.social` from CoulombCore to railiance01 before CoulombCore is decommissioned. Preserve the current `/api/v2` contract, data, hostname, credentials boundary, and rollback evidence. Do not make runtime absorption into hub-core a prerequisite. ## Decision and constraints - Use the interim-move shape requested by `SHR-WP-0002-T03`: package and move Core Hub as-is, then resolve long-term absorption under `CORE-WP-0010` and `HUB-WP-0004` on their own architecture schedule. - Use the governed Repo Manager rapp path. Because the live deployment truth is a Helm chart, initialize a managed package instead of asking `rapp wrap` to invent or ingest Helm content; retain this repo's manifests as source evidence until the package is verified. - Treat the CoulombCore decommission date as a scheduling input, not as a reason to collapse the hub-core library-vs-thin-host decision. - Any traffic flip, final data freeze/transfer, or old-runtime removal requires explicit operator approval and a rehearsed rollback. ## Inventory source and target runtime ```task id: CORE-WP-0011-T01 status: done priority: high ``` Capture the live CoulombCore image, Deployment, Service, Ingress, database, secret references, resource policy, and tunnel/DNS ownership. Compare them to railiance01 and the checked-in `k8s/railiance-production/` manifests. Local review on 2026-08-20 confirms that the repo already carries a namespace, migration Job, API Deployment, Service, Ingress, health probes, and the historical migration/cutover runbook. The manifests need a live-state diff and modern platform-policy review before reuse. The retirement project reports no `core-hub` namespace or Deployment on railiance01. A read-only preflight on 2026-08-20 confirmed the current Kubernetes context reaches the Ready k3s node `239.62.205.92.host.secureserver.net` (`92.205.62.239`, Kubernetes `v1.35.1+k3s1`), has no `core-hub` namespace or resources, and retains an `inter-hub` namespace. Pin the CoulombCore decommission date while this task is in progress so later gates have explicit deadlines. The retirement deadline is now pinned to **2026-08-31** (11 days from workplan creation). Source-cluster live state and target platform-policy comparison remain open. The registry audit found that production/staging manifests, runbooks, and Forgejo CI already publish and pull `forgejo.coulomb.social/coulomb/core-hub`, but the local Makefile default and deployed-smoke evidence still named the CoulombCore-hosted Gitea registry. Those two active references were changed to Forgejo on 2026-08-20. The only remaining Gitea image reference is historical evidence in finished `CORE-WP-0004`; confirm the live source Deployment image and digest before declaring the dependency closed. Completed after the documented local API tunnel proved stale: existing direct SSH access provided a metadata-only `kubectl`/Helm path without retrieving a new credential. Production is Ready on CoulombCore Kubernetes `v1.30.3+k3s1`, Helm chart `0.1.0`, using mutable tag `a33191a` whose pulled image ID is `sha256:5e1334eb25975b5f7792d2ae9289898586224133896c6aa55bd27cdc776fb10c`. It has one combined environment Secret, empty security contexts, the default ServiceAccount and no NetworkPolicies. Health/readiness pass publicly. Helm reports release revision 1 while the Deployment reports revision 2, so source rollback must retain the actual Deployment/ReplicaSet state rather than rely on Helm history alone. Secret inspection recorded key names and byte sizes only; no value was read. Full evidence and the target-package diff are in `docs/evidence/core-hub-coulombcore-inventory-2026-08-20.md`. ## Build the relocatable application package ```task id: CORE-WP-0011-T02 status: done priority: high ``` Wrap the current service through the governed rapp path. Pin the source revision and image digest, express namespace/runtime/database/secret dependencies without credential values, and add render/package checks. Reuse the current container and `/healthz`, `/readyz`, and deployed-smoke surfaces. Done when the package renders deterministically, policy checks pass, and its rendered workload is equivalent to the live service except for documented railiance01 platform adaptations. Completed locally on 2026-08-20 in `/home/worsch/rapp-core-hub`. Repo Manager initialized a Helm-managed package because the live deployment truth is the existing `railiance-apps/charts/core-hub` chart; `rmgr rapp wrap` deliberately refuses to invent or ingest Helm charts. The package absorbs that chart and adds an immutable Forgejo digest pin, safe-private Ingress default, non-root/read-only security contexts, ServiceAccount hardening, default-deny/DNS/Postgres/Traefik NetworkPolicies, deterministic render, Alembic migration, live smoke, and Helm rollback surfaces. Evidence: - package tests: 5 passed; - Helm lint: passed; - family validator: 4 declarations ok; - railiance01 Helm/Kubernetes server dry-run: passed; - current image pin: `sha256:0db6f7c646c6d4c1fbc844dc266fcf883584637e1c1f47802efe1c6a98558d3c`. Published and registered on 2026-08-20 after the operator created the Forgejo stub. Local history was attached without discarding either root. Private Forgejo `main` now includes credential-contract release commit `03effc8`. Chart `0.3.0` separates the static API-token Secret from runtime and migration database lease Secrets and supervises the application through mounted-file rotation; seven tests prove separation, safe URL construction, no credential output, and restart signaling. State Hub registers active repo `rapp-core-hub` as `445ea292-70bb-435c-8c69-c4cb57a6985b`, with the `bnt-lap001:/home/worsch/rapp-core-hub` host path, the canonical stub-root Git fingerprint, and Core Hub's topic ID. The package uses registered workplan prefix `RAPPCOREHUB-WP-`; consistency passes with only the expected warning that this workstation is not the production workplan-UUID registrar. ## Deploy shadow runtime and rehearse data transfer ```task id: CORE-WP-0011-T03 status: done priority: high ``` Provision the railiance01 namespace, database role/database, governed secret delivery, migration Job, Deployment, and Service without moving production traffic. Rehearse backup/restore or logical transfer from CoulombCore, compare table and row counts, and run public plus authenticated consumer smokes against the private target endpoint. Done when the shadow runtime is healthy, data reconciliation is repeatable, and ops-hub plus activity-core smokes pass against railiance01. Preflight completed 2026-08-20 without applying resources. ESO CRDs and the working audit-core two-store pattern exist on railiance01, but the `core-hub` namespace, Core Hub ClusterSecretStores, projected Secrets, and accepted `rapp-postgres` consumer declaration do not. The exact target image digest pulls and imports Core Hub, Uvicorn, and Alembic under UID/GID 10001 with a read-only root filesystem. Package tests, family validation and server dry-run pass. This task is `wait` on `RPF-WP-0021-T02/T04`; do not substitute placeholder Secrets or a standing database password. The executable gate order, evidence requirements, private smokes, live lease-rotation proof and rehearsal rollback are recorded in `docs/deployment/railiance01-shadow-runbook.md` so work can resume immediately after platform admission. Completed 2026-08-21. `RPF-WP-0021` admitted Core Hub as the fourth and final `platform-pg` consumer, established separate static API-token and dynamic runtime/migration database lanes, and restored a consistent 39-row source archive. The source `public` tables were explicitly reconciled into the accepted target `core_hub` schema; all eight tables, Alembic revision `20260627_0002`, ownership and runtime grants matched before shadow smoke activity. The migration Job completed, Helm revision 2 deployed package chart `0.3.1` at `rapp-core-hub` commit `7cfa8bb`, and no Ingress exists. The hardened private verifier and authenticated smoke `20260820224419-43dc05` passed all ten checks. A fresh runtime lease then changed the projected Secret and application child PID while the pod UID and container restart count remained stable; readiness returned without exposing a credential. The shadow now contains the expected smoke-created rows, while the public endpoint and source writer remain on CoulombCore. Repo-local evidence is in `docs/evidence/core-hub-railiance01-private-shadow-2026-08-21.md`. ## Cut over production traffic ```task id: CORE-WP-0011-T04 status: done priority: high ``` With explicit operator approval, freeze writes for the bounded final-transfer window, transfer and reconcile the final data, then move `hub.coulomb.social` and the active ops-bridge tunnel from CoulombCore to railiance01. Verify TLS, health/readiness, public catalogs, protected routes, operator console, consumer smokes, and the first post-cutover write. Rollback must restore the prior traffic/tunnel target and database write path; the old runtime stays intact until stabilization closes. Cutover progress 2026-08-21: explicit operator approval entered this gate. Helm revision 3 enabled the chart `0.3.1` Ingress on railiance01 while public DNS still targeted CoulombCore. The source application was reconnected with database-default read-only mode and continues serving health/readiness, so stale-DNS readers remain available and writers fail closed. A fresh frozen archive was captured, the backed-up target rehearsal rows were replaced, and the source `public` tables were mapped into target schema `core_hub`. All eight table counts and order-independent row-content digests match; Alembic `20260627_0002`, migration, the hardened verifier, direct target health/ readiness, unauthenticated denial, and authenticated protected read pass. Completed 2026-08-21 after the operator changed the authoritative IONOS A record from `92.205.130.254` to `92.205.62.239`. All four authoritative name servers and Google/Quad9 returned railiance01; Cloudflare retained the old address under its pre-change TTL, so CoulombCore remains read-only while that cache drains. A temporary, challenge-path-only bridge on CoulombCore and a solver-label/port-specific target NetworkPolicy allowed cert-manager to issue the production certificate without reopening either application or database access; both temporary resources were removed immediately afterward. The Let's Encrypt certificate verifies for `hub.coulomb.social`. Forced target checks passed health/readiness, all four public catalogs, protected-route and operator-console auth boundaries, authenticated reads, and the hardened live verifier. The first post-cutover write returned 201 and read back as event `e56bca14-51f8-4a73-bdab-f660e360fb14`. The ops-hub public gate and the deployed activity-core stabilization resolver both passed against railiance01. No active ops-bridge tunnel fronts the public hostname; it is a direct IONOS DNS route. The obsolete local `core-hub-staging-coulombcore` tunnel is deferred to T05. Current evidence and exact rollback state are in `docs/evidence/core-hub-railiance01-cutover-2026-08-21.md`. ## Stabilize and release CoulombCore dependencies ```task id: CORE-WP-0011-T05 status: done priority: high ``` Run a time-bounded stabilization window with scheduled health and consumer checks. After operator sign-off, remove Core Hub from the CoulombCore shutdown critical path, retire `core-hub-staging` and any obsolete tunnel there, and record retained backup/rollback evidence and residual owners. Do not archive this repo unless the separate absorption workplan has also completed. Stabilization started 2026-08-21. Cloudflare, Google, Quad9 and the local resolver now all return `92.205.62.239`; normal public HTTPS reaches railiance01 with valid TLS. Helm revision 4 runs package chart `0.3.2`, which permanently carries the solver-label/kube-system/TCP-8089 NetworkPolicy needed for future certificate renewals, and the post-upgrade verifier passes. The CoulombCore Deployment remains Ready for rollback, but database-level `default_transaction_read_only=on` was reverified. Keep the source frozen until the bounded observation window and operator sign-off close this task. Completed 2026-08-21 after multiple original DNS TTLs and two independent consumer snapshots. The target recorded 989 successful requests, four expected 401s and no 5xx/error match; its pod stayed Ready with zero restarts, all ExternalSecrets and TLS remained Ready, and the post-retirement package verifier passed. Completed target and encrypted source backups were identified before both CoulombCore Core Hub Deployments were scaled to zero. The obsolete staging tunnel was removed from configuration and its exact supervisor/SSH processes were stopped; port 18010 is closed. Source Services, Ingresses, Secrets and the read-only database remain only as reversible retention artifacts and no longer block CoulombCore shutdown. Closeout evidence is in `docs/evidence/core-hub-railiance01-stabilization-2026-08-21.md`. ## Acceptance - [x] Decommission deadline and live source/target inventory are recorded - [x] Governed rapp package renders and passes policy checks - [x] Shadow restore/reconciliation and consumer smokes pass on railiance01 - [x] Production traffic and tunnel move with explicit approval and rollback - [x] Stabilization passes and CoulombCore no longer hosts Core Hub dependencies - [x] `CORE-WP-0010` remains independently sequenced behind `HUB-WP-0004`