core-hub/workplans/CORE-WP-0011-railiance01-production-relocation.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	parent_project	parent_workplan	related	state_hub_workstream_id
CORE-WP-0011	workplan	Relocate Core Hub production from CoulombCore to railiance01	infotech	core-hub	finished	codex	infotech	2026-08-20	2026-08-21	prj-state-hub-retirement	SHR-WP-0002	CORE-WP-0005 CORE-WP-0007 CORE-WP-0010 HUB-WP-0004	e46d44a1-405f-5aeb-ad09-2eb03102c3f0

Relocate Core Hub production from CoulombCore to railiance01

Goal

Move the existing Core Hub production service at hub.coulomb.social from CoulombCore to railiance01 before CoulombCore is decommissioned. Preserve the current /api/v2 contract, data, hostname, credentials boundary, and rollback evidence. Do not make runtime absorption into hub-core a prerequisite.

Decision and constraints

• Use the interim-move shape requested by SHR-WP-0002-T03: package and move Core Hub as-is, then resolve long-term absorption under CORE-WP-0010 and HUB-WP-0004 on their own architecture schedule.
• Use the governed Repo Manager rapp path. Because the live deployment truth is a Helm chart, initialize a managed package instead of asking rapp wrap to invent or ingest Helm content; retain this repo's manifests as source evidence until the package is verified.
• Treat the CoulombCore decommission date as a scheduling input, not as a reason to collapse the hub-core library-vs-thin-host decision.
• Any traffic flip, final data freeze/transfer, or old-runtime removal requires explicit operator approval and a rehearsed rollback.

Inventory source and target runtime

id: CORE-WP-0011-T01
status: done
priority: high
state_hub_task_id: "e6138fe2-e54f-512b-860e-5af9efb4aa5b"

Capture the live CoulombCore image, Deployment, Service, Ingress, database, secret references, resource policy, and tunnel/DNS ownership. Compare them to railiance01 and the checked-in k8s/railiance-production/ manifests.

Local review on 2026-08-20 confirms that the repo already carries a namespace, migration Job, API Deployment, Service, Ingress, health probes, and the historical migration/cutover runbook. The manifests need a live-state diff and modern platform-policy review before reuse. The retirement project reports no core-hub namespace or Deployment on railiance01. A read-only preflight on 2026-08-20 confirmed the current Kubernetes context reaches the Ready k3s node 239.62.205.92.host.secureserver.net (92.205.62.239, Kubernetes v1.35.1+k3s1), has no core-hub namespace or resources, and retains an inter-hub namespace. Pin the CoulombCore decommission date while this task is in progress so later gates have explicit deadlines. The retirement deadline is now pinned to 2026-08-31 (11 days from workplan creation). Source-cluster live state and target platform-policy comparison remain open.

The registry audit found that production/staging manifests, runbooks, and Forgejo CI already publish and pull forgejo.coulomb.social/coulomb/core-hub, but the local Makefile default and deployed-smoke evidence still named the CoulombCore-hosted Gitea registry. Those two active references were changed to Forgejo on 2026-08-20. The only remaining Gitea image reference is historical evidence in finished CORE-WP-0004; confirm the live source Deployment image and digest before declaring the dependency closed.

Completed after the documented local API tunnel proved stale: existing direct SSH access provided a metadata-only kubectl/Helm path without retrieving a new credential. Production is Ready on CoulombCore Kubernetes v1.30.3+k3s1, Helm chart 0.1.0, using mutable tag a33191a whose pulled image ID is sha256:5e1334eb25975b5f7792d2ae9289898586224133896c6aa55bd27cdc776fb10c. It has one combined environment Secret, empty security contexts, the default ServiceAccount and no NetworkPolicies. Health/readiness pass publicly. Helm reports release revision 1 while the Deployment reports revision 2, so source rollback must retain the actual Deployment/ReplicaSet state rather than rely on Helm history alone. Secret inspection recorded key names and byte sizes only; no value was read. Full evidence and the target-package diff are in docs/evidence/core-hub-coulombcore-inventory-2026-08-20.md.

Build the relocatable application package

id: CORE-WP-0011-T02
status: done
priority: high
state_hub_task_id: "2fbbc5b8-0261-5e11-ac95-3d34a0afb43d"

Wrap the current service through the governed rapp path. Pin the source revision and image digest, express namespace/runtime/database/secret dependencies without credential values, and add render/package checks. Reuse the current container and /healthz, /readyz, and deployed-smoke surfaces.

Done when the package renders deterministically, policy checks pass, and its rendered workload is equivalent to the live service except for documented railiance01 platform adaptations.

Completed locally on 2026-08-20 in /home/worsch/rapp-core-hub. Repo Manager initialized a Helm-managed package because the live deployment truth is the existing railiance-apps/charts/core-hub chart; rmgr rapp wrap deliberately refuses to invent or ingest Helm charts. The package absorbs that chart and adds an immutable Forgejo digest pin, safe-private Ingress default, non-root/read-only security contexts, ServiceAccount hardening, default-deny/DNS/Postgres/Traefik NetworkPolicies, deterministic render, Alembic migration, live smoke, and Helm rollback surfaces. Evidence:

• package tests: 5 passed;
• Helm lint: passed;
• family validator: 4 declarations ok;
• railiance01 Helm/Kubernetes server dry-run: passed;
• current image pin: sha256:0db6f7c646c6d4c1fbc844dc266fcf883584637e1c1f47802efe1c6a98558d3c.

Published and registered on 2026-08-20 after the operator created the Forgejo stub. Local history was attached without discarding either root. Private Forgejo main now includes credential-contract release commit 03effc8. Chart 0.3.0 separates the static API-token Secret from runtime and migration database lease Secrets and supervises the application through mounted-file rotation; seven tests prove separation, safe URL construction, no credential output, and restart signaling. State Hub registers active repo rapp-core-hub as 445ea292-70bb-435c-8c69-c4cb57a6985b, with the bnt-lap001:/home/worsch/rapp-core-hub host path, the canonical stub-root Git fingerprint, and Core Hub's topic ID. The package uses registered workplan prefix RAPPCOREHUB-WP-; consistency passes with only the expected warning that this workstation is not the production workplan-UUID registrar.

Deploy shadow runtime and rehearse data transfer

id: CORE-WP-0011-T03
status: done
priority: high
state_hub_task_id: "991e614b-652a-5c81-a936-36a72ebe49ec"

Provision the railiance01 namespace, database role/database, governed secret delivery, migration Job, Deployment, and Service without moving production traffic. Rehearse backup/restore or logical transfer from CoulombCore, compare table and row counts, and run public plus authenticated consumer smokes against the private target endpoint.

Done when the shadow runtime is healthy, data reconciliation is repeatable, and ops-hub plus activity-core smokes pass against railiance01.

Preflight completed 2026-08-20 without applying resources. ESO CRDs and the working audit-core two-store pattern exist on railiance01, but the core-hub namespace, Core Hub ClusterSecretStores, projected Secrets, and accepted rapp-postgres consumer declaration do not. The exact target image digest pulls and imports Core Hub, Uvicorn, and Alembic under UID/GID 10001 with a read-only root filesystem. Package tests, family validation and server dry-run pass. This task is wait on RPF-WP-0021-T02/T04; do not substitute placeholder Secrets or a standing database password. The executable gate order, evidence requirements, private smokes, live lease-rotation proof and rehearsal rollback are recorded in docs/deployment/railiance01-shadow-runbook.md so work can resume immediately after platform admission.

Completed 2026-08-21. RPF-WP-0021 admitted Core Hub as the fourth and final platform-pg consumer, established separate static API-token and dynamic runtime/migration database lanes, and restored a consistent 39-row source archive. The source public tables were explicitly reconciled into the accepted target core_hub schema; all eight tables, Alembic revision 20260627_0002, ownership and runtime grants matched before shadow smoke activity. The migration Job completed, Helm revision 2 deployed package chart 0.3.1 at rapp-core-hub commit 7cfa8bb, and no Ingress exists.

The hardened private verifier and authenticated smoke 20260820224419-43dc05 passed all ten checks. A fresh runtime lease then changed the projected Secret and application child PID while the pod UID and container restart count remained stable; readiness returned without exposing a credential. The shadow now contains the expected smoke-created rows, while the public endpoint and source writer remain on CoulombCore. Repo-local evidence is in docs/evidence/core-hub-railiance01-private-shadow-2026-08-21.md.

Cut over production traffic

id: CORE-WP-0011-T04
status: done
priority: high
state_hub_task_id: "491528bd-8d14-50eb-b1b6-01f1097a857a"

With explicit operator approval, freeze writes for the bounded final-transfer window, transfer and reconcile the final data, then move hub.coulomb.social and the active ops-bridge tunnel from CoulombCore to railiance01. Verify TLS, health/readiness, public catalogs, protected routes, operator console, consumer smokes, and the first post-cutover write.

Rollback must restore the prior traffic/tunnel target and database write path; the old runtime stays intact until stabilization closes.

Cutover progress 2026-08-21: explicit operator approval entered this gate. Helm revision 3 enabled the chart 0.3.1 Ingress on railiance01 while public DNS still targeted CoulombCore. The source application was reconnected with database-default read-only mode and continues serving health/readiness, so stale-DNS readers remain available and writers fail closed. A fresh frozen archive was captured, the backed-up target rehearsal rows were replaced, and the source public tables were mapped into target schema core_hub. All eight table counts and order-independent row-content digests match; Alembic 20260627_0002, migration, the hardened verifier, direct target health/ readiness, unauthenticated denial, and authenticated protected read pass.

Completed 2026-08-21 after the operator changed the authoritative IONOS A record from 92.205.130.254 to 92.205.62.239. All four authoritative name servers and Google/Quad9 returned railiance01; Cloudflare retained the old address under its pre-change TTL, so CoulombCore remains read-only while that cache drains. A temporary, challenge-path-only bridge on CoulombCore and a solver-label/port-specific target NetworkPolicy allowed cert-manager to issue the production certificate without reopening either application or database access; both temporary resources were removed immediately afterward.

The Let's Encrypt certificate verifies for hub.coulomb.social. Forced target checks passed health/readiness, all four public catalogs, protected-route and operator-console auth boundaries, authenticated reads, and the hardened live verifier. The first post-cutover write returned 201 and read back as event e56bca14-51f8-4a73-bdab-f660e360fb14. The ops-hub public gate and the deployed activity-core stabilization resolver both passed against railiance01. No active ops-bridge tunnel fronts the public hostname; it is a direct IONOS DNS route. The obsolete local core-hub-staging-coulombcore tunnel is deferred to T05. Current evidence and exact rollback state are in docs/evidence/core-hub-railiance01-cutover-2026-08-21.md.

Stabilize and release CoulombCore dependencies

id: CORE-WP-0011-T05
status: done
priority: high
state_hub_task_id: "7a23b3e0-c0ab-5061-b1e8-3cdb28b8acce"

Run a time-bounded stabilization window with scheduled health and consumer checks. After operator sign-off, remove Core Hub from the CoulombCore shutdown critical path, retire core-hub-staging and any obsolete tunnel there, and record retained backup/rollback evidence and residual owners. Do not archive this repo unless the separate absorption workplan has also completed.

Stabilization started 2026-08-21. Cloudflare, Google, Quad9 and the local resolver now all return 92.205.62.239; normal public HTTPS reaches railiance01 with valid TLS. Helm revision 4 runs package chart 0.3.2, which permanently carries the solver-label/kube-system/TCP-8089 NetworkPolicy needed for future certificate renewals, and the post-upgrade verifier passes. The CoulombCore Deployment remains Ready for rollback, but database-level default_transaction_read_only=on was reverified. Keep the source frozen until the bounded observation window and operator sign-off close this task.

Completed 2026-08-21 after multiple original DNS TTLs and two independent consumer snapshots. The target recorded 989 successful requests, four expected 401s and no 5xx/error match; its pod stayed Ready with zero restarts, all ExternalSecrets and TLS remained Ready, and the post-retirement package verifier passed. Completed target and encrypted source backups were identified before both CoulombCore Core Hub Deployments were scaled to zero. The obsolete staging tunnel was removed from configuration and its exact supervisor/SSH processes were stopped; port 18010 is closed. Source Services, Ingresses, Secrets and the read-only database remain only as reversible retention artifacts and no longer block CoulombCore shutdown. Closeout evidence is in docs/evidence/core-hub-railiance01-stabilization-2026-08-21.md.

Acceptance

• Decommission deadline and live source/target inventory are recorded
• Governed rapp package renders and passes policy checks
• Shadow restore/reconciliation and consumer smokes pass on railiance01
• Production traffic and tunnel move with explicit approval and rollback
• Stabilization passes and CoulombCore no longer hosts Core Hub dependencies
• CORE-WP-0010 remains independently sequenced behind HUB-WP-0004