railiance-platform/workplans/RPF-WP-0021-core-hub-platform-onboarding.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	related	origin	origin_ref
RPF-WP-0021	workplan	Unblock rapp-core-hub publication and platform onboarding	financials	railiance-platform	finished	codex	railiance	2026-08-20	2026-08-21	CORE-WP-0011 RAPPCOREHUB-WP-0001 RAPP-POSTGRES-WP-0003	request	CORE-WP-0011

RPF-WP-0021 — Core Hub platform onboarding

Goal

Provide the platform-owned dependencies that let the validated rapp-core-hub package publish and deploy as a private shadow runtime on railiance01 before the 2026-08-31 CoulombCore retirement deadline.

This work stops before public traffic cutover, the final data freeze, or retirement of the old runtime. Those remain explicit operator gates in CORE-WP-0011.

Current evidence

• /home/worsch/rapp-core-hub is clean at commit 18e4e70; its full package check, family validation, and railiance01 server-side dry-run passed.
• coulomb/rapp-core-hub is private and authoritative in Forgejo. Its history preserves validated package commit 4403c9b, and unauthenticated API access returns 404.
• State Hub registered the repository and canonical workplan prefix RAPPCOREHUB-WP-. The production registrar still needs to assign the workplan UUID.
• agent-high-risk-boundary deliberately denies agent data-read to the Forgejo site-admin PAT. That control is not removed to make publication convenient; an attended operator identity must authorize the child-process transport.
• The package requests a rapp-postgres.v1 consumer and a Kubernetes Secret containing CORE_HUB_DATABASE_URL plus CORE_HUB_API_TOKEN.
• A live metadata-only railiance01 check on 2026-08-20 found OpenBao Ready and the Kubernetes-auth-backed openbao ClusterSecretStore Valid. The existing audit-core KV and database stores are also Valid and their ExternalSecrets are synced, so the two-store shape has a live reference on the target rail.

Boundaries

• No secret value enters Git, State Hub, logs, command output, or agent context.
• A repository is created only through the owned Forgejo admin lane.
• The static Core Hub API token and dynamic PostgreSQL lease remain distinct custody sources even if the workload consumes both as environment data.
• A short-lived database lease is not copied into a standing OpenBao KV path.
• A private shadow deployment is allowed only after the database credential rotation behavior is explicit and testable.
• Public Ingress, final transfer, DNS/tunnel changes, and old-runtime removal remain outside this workplan.

T01 — Publish and register rapp-core-hub

id: RPF-WP-0021-T01
status: done
priority: high

Under an attended net-kingdom-admins OIDC login to forgejo-admin-workload-kv-read, verify capabilities on the Forgejo admin KV data and metadata paths without reading the value. Then use warden access forgejo-admin-api-token --exec to create the private coulomb/rapp-core-hub repository, push commit 4403c9b, configure the remote, and run statehub fix-consistency from the published repo.

Current blocker: both the original request and a 2026-08-20 retry reached OpenBao but received HTTP 403 on sys/internal/ui/mounts/platform/workloads/forgejo/forgejo-admin. The retry also found the local warden audit path read-only. Do not weaken agent-high-risk-boundary; repair or establish the operator session and use the non-disclosing exec transport.

Concurrent publication evidence, discovered at 07:55Z: Forgejo repository coulomb/rapp-core-hub was created at 07:42:41Z, but it is public rather than private and contains only Forgejo's Initial commit (eea94d9, README only). The local validated commit 4403c9b is not the remote main, although the local remote is configured. Do not push the package into the public repo. First change visibility to private through the owned admin lane, correct the repository description, then push and verify refs/heads/main equals the validated commit.

Completed 2026-08-20 after operator authorization. Unauthenticated Forgejo API access now returns 404 while the configured Git remote is accessible. Remote main is 18e4e70: its history preserves validated package commit 4403c9b, the Forgejo README stub through merge 7dd057e, publication evidence, the project goal, and the registered-prefix alignment. Local and remote main match. make check, the railiance01 server-side dry-run, and family validation (4 declaration(s) ok) passed after publication. The repo is registered in State Hub; workplan UUID assignment remains with the production registrar.

T02 — Accept the Core Hub PostgreSQL consumer

id: RPF-WP-0021-T02
status: done
priority: high

Review /home/worsch/rapp-core-hub/handoffs/postgres-consumer.yaml in rapp-postgres, add the accepted declaration, render it, and run package tests. The requested database, schema, role split, connection limits, timeout, service class, and tenant-keying posture match the existing consumer contract.

Capacity is an admission condition. With audit-core, tenant-engine, and the counted isolation probe, Core Hub is the fourth declaration on platform-pg: it reaches but does not exceed the ADR-0004 ceiling. Before admitting this third workload consumer, record the required measured memory baseline. The untracked consumers/issue-core.yaml is not a real consumer — ISSUE-WP-0007 explicitly removed Postgres from that package — and must not silently consume a slot.

OpenBao database role configuration remains an attended operation under T04.

Read-only preflight 2026-08-20: platform-pg-1 used 9m CPU and 362Mi memory against a 1Gi limit, with only audit_core and operator sessions present; tenant_engine was not yet a database. This establishes current headroom but does not satisfy ADR-0004's representative multi-consumer peak measurement. Evidence: docs/evidence/core-hub-postgres-capacity-preflight-2026-08-20.md.

Capacity admission completed 2026-08-20. A bounded live probe opened 56 sessions in four declaration groups and repeated the occupancy with one active bounded sort per group. platform-pg peaked at 174Mi/1Gi; the 85Mi increase is about 1.52Mi per opened backend. Even adding that delta to the earlier 362Mi sample yields 447Mi, under 44% of the limit. Core Hub is accepted as the fourth and final declaration without raising the pod limit. Evidence: docs/evidence/core-hub-postgres-capacity-admission-2026-08-20.md.

Completed 2026-08-20. rapp-postgres commit 4f316cc accepts and publishes the Core Hub declaration as the fourth platform-pg consumer. All 31 package tests and consumer rendering passed; the railiance01 server-side dry-run accepted the generated resource. The idempotent package provisioner then created core_hub, its core_hub schema, distinct NOLOGIN owner/migration/ runtime roles and bounded role settings. CNPG Database/core-hub reports status.applied=true, targets platform-pg, retains the database on resource deletion, and names core_hub_owner as owner.

T03 — Correct the runtime credential delivery contract

id: RPF-WP-0021-T03
status: done
priority: high

Separate the two credential sources before shadow deployment:

1. CORE_HUB_API_TOKEN is a standing workload secret in OpenBao KV, governed by a proposed CCR and projected by a namespace-limited ESO store.
2. Runtime and migration PostgreSQL credentials come from distinct database/creds/core-hub-* roles and the database-engine ESO store. One extract must provide each username/password pair so leases are not mixed.

The current package consumes one core-hub-runtime Secret through envFrom for both the Deployment and migration Job. That is not yet a safe dynamic lease contract: environment variables do not update when ESO refreshes a Secret, and no workload reloader is declared. Update the package to consume separate runtime/migration database Secrets and the static API-token Secret, then either:

• restart the Deployment automatically and demonstrably before each lease expires; or
• change Core Hub to re-read mounted credential files when opening a new database connection.

Prefer the mounted-file approach for steady state. A time-bounded automatic restart contract may be used for the relocation only if rollback and lease overlap are proven and the temporary limitation is recorded.

Completed 2026-08-21. Static API, runtime database, and migration database credentials are mounted from three disjoint Secrets; the database URL exists only in the supervised child environment, and the migration Job mounts only its role. Live proof exposed that chart 0.3.0 watcher exit 75 could accumulate planned Kubelet restarts into CrashLoopBackOff. Chart 0.3.1 (rapp-core-hub commit 7cfa8bb) instead keeps PID 1 alive and restarts only the application child after reading the refreshed files. Nine package tests and live proof showed a changed Secret resourceVersion and child PID with stable pod UID, container restart count 0 -> 0, restored readiness, and the five-minute refresh interval restored.

T04 — Declare and apply the OpenBao/ESO lanes

id: RPF-WP-0021-T04
status: done
priority: high

After T03 fixes the consuming contract:

• draft and approve a workload-KV CCR for the static API token;
• add a least-privilege ESO policy covering only the Core Hub static-token KV path and database/creds/core-hub-runtime / core-hub-migration;
• configure the database roles from the accepted Postgres declaration;
• create namespace-limited KV and database ClusterSecretStore objects;
• project runtime and migration Secrets without printing values; and
• prove positive sync, cross-namespace denial, lease rotation, and migration role separation.

The working audit-core implementation is the reference for the two-store shape. Do not copy its earlier, rejected idea of freezing a dynamic lease in KV.

Progress 2026-08-20: drafted CCR-2026-0013 for the static API token, added its exact-path read policy, added a not-yet-applied namespace-limited openbao-core-hub-runtime ClusterSecretStore, and extended the agent high-risk boundary to deny direct data reads on the new path. Database-engine policy/store manifests wait for the accepted Postgres declaration and the T03 consumer correction.

Progress 2026-08-20 after T02 admission: added an exact two-path database ESO policy, a namespace-limited openbao-core-hub-database store, and three disjoint ExternalSecrets. Runtime and migration each use one dataFrom.extract so a username/password pair comes from one lease; both refresh every five minutes. The database store and all projections passed railiance01 server-side dry-run. The package namespace is live with the reviewed platform-pg client label. Live role/store activation correctly stopped because the current OpenBao session receives 403 and CCR-2026-0013 is still proposed; no authority or placeholder Secret was improvised. State Hub decision 4f70311b-a4c4-4959-b1a1-cdd1be97d523 now carries the non-secret review coordinates and awaits the required platform-operator/Core-Hub-owner approval. rapp-postgres commit cfde57a now also owns the Core Hub database-engine roles, 30-minute runtime/10-minute migration TTLs, exact two-path ESO policy, and non-disclosing renewable-token delivery script. Its 33 tests pass. Live configuration still waits for the attended OpenBao authority; concurrent tenant-engine workplan edits in that repository were preserved and excluded from this commit.

Completed 2026-08-21 after the operator approved State Hub decision 4f70311b-a4c4-4959-b1a1-cdd1be97d523 and completed attended OpenBao OIDC. The policy, auth role, database roles, two namespace-limited stores, and three disjoint ExternalSecrets are live and Ready. The existing API token moved directly from source Kubernetes custody into OpenBao without appearing in output. Outside-namespace store denial, exact-path Kubernetes-auth denial, runtime DDL denial, migration DDL success, ESO sync, and live lease rotation all passed. CCR-2026-0013 is verified.

T05 — Support the private shadow restore and verification

id: RPF-WP-0021-T05
status: done
priority: high

After T01-T04, support CORE-WP-0011-T03: label the namespace for platform-pg, restore or logically transfer the source database, reconcile table and row counts, run the migration Job with the migration lease, deploy with Ingress disabled, and verify health plus authenticated consumers against the private endpoint.

This task ends with repeatable shadow evidence. It does not flip hub.coulomb.social, freeze production writes, or remove CoulombCore.

Preflight 2026-08-20: the source has Alembic revision 20260627_0002, eight public tables and 39 total rows. The admitted target intentionally has no application tables before restore. A ready disposable pod in the labeled core-hub namespace resolved the platform-pg service and reached its PostgreSQL port; it was deleted after the check. No dump or restore was attempted ahead of the T04 credential gate. Evidence: docs/evidence/core-hub-shadow-preflight-2026-08-20.md.

Completed 2026-08-21. A consistent logical archive restored all eight source tables and 39 rows at Alembic revision 20260627_0002; the restored tables were reconciled into the accepted core_hub schema with owner and runtime grants verified. Helm release revision 2 deployed chart 0.3.1 with Ingress disabled, and the migration Job completed. The hardened private verifier and authenticated smoke 20260820224419-43dc05 passed all health, readiness, catalog, denial, and protected-operation checks. Detailed non-secret evidence: docs/evidence/core-hub-private-shadow-2026-08-21.md.

Acceptance

• Private Forgejo repo exists, validated commit 4403c9b is preserved, and the registered workplan prefix is consistent.
• Core Hub is an accepted, capacity-reviewed rapp-postgres consumer.
• Static API-token and dynamic database credentials have distinct, least-privilege delivery paths.
• Database credential rotation cannot leave a running pod using an expired environment value.
• Private shadow restore, reconciliation, migration, and consumer smokes pass without enabling public Ingress.