--- id: RPF-WP-0021 type: workplan title: "Unblock rapp-core-hub publication and platform onboarding" domain: financials repo: railiance-platform status: finished owner: codex topic_slug: railiance created: "2026-08-20" updated: "2026-08-21" related: - CORE-WP-0011 - RAPPCOREHUB-WP-0001 - RAPP-POSTGRES-WP-0003 origin: request origin_ref: CORE-WP-0011 state_hub_workstream_id: "aa7fe3ff-a76c-5f26-a1e0-a35e48b28d60" --- # RPF-WP-0021 — Core Hub platform onboarding ## Goal Provide the platform-owned dependencies that let the validated `rapp-core-hub` package publish and deploy as a private shadow runtime on railiance01 before the 2026-08-31 CoulombCore retirement deadline. This work stops before public traffic cutover, the final data freeze, or retirement of the old runtime. Those remain explicit operator gates in `CORE-WP-0011`. ## Current evidence - `/home/worsch/rapp-core-hub` is clean at commit `18e4e70`; its full package check, family validation, and railiance01 server-side dry-run passed. - `coulomb/rapp-core-hub` is private and authoritative in Forgejo. Its history preserves validated package commit `4403c9b`, and unauthenticated API access returns 404. - State Hub registered the repository and canonical workplan prefix `RAPPCOREHUB-WP-`. The production registrar still needs to assign the workplan UUID. - `agent-high-risk-boundary` deliberately denies agent data-read to the Forgejo site-admin PAT. That control is not removed to make publication convenient; an attended operator identity must authorize the child-process transport. - The package requests a `rapp-postgres.v1` consumer and a Kubernetes Secret containing `CORE_HUB_DATABASE_URL` plus `CORE_HUB_API_TOKEN`. - A live metadata-only railiance01 check on 2026-08-20 found OpenBao Ready and the Kubernetes-auth-backed `openbao` ClusterSecretStore Valid. The existing audit-core KV and database stores are also Valid and their ExternalSecrets are synced, so the two-store shape has a live reference on the target rail. ## Boundaries - No secret value enters Git, State Hub, logs, command output, or agent context. - A repository is created only through the owned Forgejo admin lane. - The static Core Hub API token and dynamic PostgreSQL lease remain distinct custody sources even if the workload consumes both as environment data. - A short-lived database lease is not copied into a standing OpenBao KV path. - A private shadow deployment is allowed only after the database credential rotation behavior is explicit and testable. - Public Ingress, final transfer, DNS/tunnel changes, and old-runtime removal remain outside this workplan. ## T01 — Publish and register `rapp-core-hub` ```task id: RPF-WP-0021-T01 status: done priority: high state_hub_task_id: "a7cd3fc0-4469-55a6-aed0-a8cacb34c033" ``` Under an attended `net-kingdom-admins` OIDC login to `forgejo-admin-workload-kv-read`, verify capabilities on the Forgejo admin KV data and metadata paths without reading the value. Then use `warden access forgejo-admin-api-token --exec` to create the private `coulomb/rapp-core-hub` repository, push commit `4403c9b`, configure the remote, and run `statehub fix-consistency` from the published repo. Current blocker: both the original request and a 2026-08-20 retry reached OpenBao but received HTTP 403 on `sys/internal/ui/mounts/platform/workloads/forgejo/forgejo-admin`. The retry also found the local warden audit path read-only. Do not weaken `agent-high-risk-boundary`; repair or establish the operator session and use the non-disclosing exec transport. Concurrent publication evidence, discovered at 07:55Z: Forgejo repository `coulomb/rapp-core-hub` was created at 07:42:41Z, but it is public rather than private and contains only Forgejo's `Initial commit` (`eea94d9`, README only). The local validated commit `4403c9b` is not the remote `main`, although the local remote is configured. Do not push the package into the public repo. First change visibility to private through the owned admin lane, correct the repository description, then push and verify `refs/heads/main` equals the validated commit. Completed 2026-08-20 after operator authorization. Unauthenticated Forgejo API access now returns 404 while the configured Git remote is accessible. Remote `main` is `18e4e70`: its history preserves validated package commit `4403c9b`, the Forgejo README stub through merge `7dd057e`, publication evidence, the project goal, and the registered-prefix alignment. Local and remote `main` match. `make check`, the railiance01 server-side dry-run, and family validation (`4 declaration(s) ok`) passed after publication. The repo is registered in State Hub; workplan UUID assignment remains with the production registrar. ## T02 — Accept the Core Hub PostgreSQL consumer ```task id: RPF-WP-0021-T02 status: done priority: high state_hub_task_id: "7cac453b-2e85-56d9-abae-d983e382246b" ``` Review `/home/worsch/rapp-core-hub/handoffs/postgres-consumer.yaml` in `rapp-postgres`, add the accepted declaration, render it, and run package tests. The requested database, schema, role split, connection limits, timeout, service class, and tenant-keying posture match the existing consumer contract. Capacity is an admission condition. With `audit-core`, `tenant-engine`, and the counted isolation probe, Core Hub is the fourth declaration on `platform-pg`: it reaches but does not exceed the ADR-0004 ceiling. Before admitting this third workload consumer, record the required measured memory baseline. The untracked `consumers/issue-core.yaml` is not a real consumer — `ISSUE-WP-0007` explicitly removed Postgres from that package — and must not silently consume a slot. OpenBao database role configuration remains an attended operation under T04. Read-only preflight 2026-08-20: `platform-pg-1` used 9m CPU and 362Mi memory against a 1Gi limit, with only `audit_core` and operator sessions present; `tenant_engine` was not yet a database. This establishes current headroom but does not satisfy ADR-0004's representative multi-consumer peak measurement. Evidence: `docs/evidence/core-hub-postgres-capacity-preflight-2026-08-20.md`. Capacity admission completed 2026-08-20. A bounded live probe opened 56 sessions in four declaration groups and repeated the occupancy with one active bounded sort per group. `platform-pg` peaked at 174Mi/1Gi; the 85Mi increase is about 1.52Mi per opened backend. Even adding that delta to the earlier 362Mi sample yields 447Mi, under 44% of the limit. Core Hub is accepted as the fourth and final declaration without raising the pod limit. Evidence: `docs/evidence/core-hub-postgres-capacity-admission-2026-08-20.md`. Completed 2026-08-20. `rapp-postgres` commit `4f316cc` accepts and publishes the Core Hub declaration as the fourth `platform-pg` consumer. All 31 package tests and consumer rendering passed; the railiance01 server-side dry-run accepted the generated resource. The idempotent package provisioner then created `core_hub`, its `core_hub` schema, distinct NOLOGIN owner/migration/ runtime roles and bounded role settings. CNPG `Database/core-hub` reports `status.applied=true`, targets `platform-pg`, retains the database on resource deletion, and names `core_hub_owner` as owner. ## T03 — Correct the runtime credential delivery contract ```task id: RPF-WP-0021-T03 status: done priority: high state_hub_task_id: "223d5a77-b7c3-5d1a-a93c-4cb9628b457a" ``` Separate the two credential sources before shadow deployment: 1. `CORE_HUB_API_TOKEN` is a standing workload secret in OpenBao KV, governed by a proposed CCR and projected by a namespace-limited ESO store. 2. Runtime and migration PostgreSQL credentials come from distinct `database/creds/core-hub-*` roles and the database-engine ESO store. One extract must provide each username/password pair so leases are not mixed. The current package consumes one `core-hub-runtime` Secret through `envFrom` for both the Deployment and migration Job. That is not yet a safe dynamic lease contract: environment variables do not update when ESO refreshes a Secret, and no workload reloader is declared. Update the package to consume separate runtime/migration database Secrets and the static API-token Secret, then either: - restart the Deployment automatically and demonstrably before each lease expires; or - change Core Hub to re-read mounted credential files when opening a new database connection. Prefer the mounted-file approach for steady state. A time-bounded automatic restart contract may be used for the relocation only if rollback and lease overlap are proven and the temporary limitation is recorded. Completed 2026-08-21. Static API, runtime database, and migration database credentials are mounted from three disjoint Secrets; the database URL exists only in the supervised child environment, and the migration Job mounts only its role. Live proof exposed that chart `0.3.0` watcher exit 75 could accumulate planned Kubelet restarts into CrashLoopBackOff. Chart `0.3.1` (`rapp-core-hub` commit `7cfa8bb`) instead keeps PID 1 alive and restarts only the application child after reading the refreshed files. Nine package tests and live proof showed a changed Secret resourceVersion and child PID with stable pod UID, container restart count `0 -> 0`, restored readiness, and the five-minute refresh interval restored. ## T04 — Declare and apply the OpenBao/ESO lanes ```task id: RPF-WP-0021-T04 status: done priority: high state_hub_task_id: "50713b54-38a8-543d-8c74-0e3fa186fbb8" ``` After T03 fixes the consuming contract: - draft and approve a workload-KV CCR for the static API token; - add a least-privilege ESO policy covering only the Core Hub static-token KV path and `database/creds/core-hub-runtime` / `core-hub-migration`; - configure the database roles from the accepted Postgres declaration; - create namespace-limited KV and database `ClusterSecretStore` objects; - project runtime and migration Secrets without printing values; and - prove positive sync, cross-namespace denial, lease rotation, and migration role separation. The working audit-core implementation is the reference for the two-store shape. Do not copy its earlier, rejected idea of freezing a dynamic lease in KV. Progress 2026-08-20: drafted `CCR-2026-0013` for the static API token, added its exact-path read policy, added a not-yet-applied namespace-limited `openbao-core-hub-runtime` ClusterSecretStore, and extended the agent high-risk boundary to deny direct data reads on the new path. Database-engine policy/store manifests wait for the accepted Postgres declaration and the T03 consumer correction. Progress 2026-08-20 after T02 admission: added an exact two-path database ESO policy, a namespace-limited `openbao-core-hub-database` store, and three disjoint ExternalSecrets. Runtime and migration each use one `dataFrom.extract` so a username/password pair comes from one lease; both refresh every five minutes. The database store and all projections passed railiance01 server-side dry-run. The package namespace is live with the reviewed `platform-pg` client label. Live role/store activation correctly stopped because the current OpenBao session receives 403 and CCR-2026-0013 is still proposed; no authority or placeholder Secret was improvised. State Hub decision `4f70311b-a4c4-4959-b1a1-cdd1be97d523` now carries the non-secret review coordinates and awaits the required platform-operator/Core-Hub-owner approval. `rapp-postgres` commit `cfde57a` now also owns the Core Hub database-engine roles, 30-minute runtime/10-minute migration TTLs, exact two-path ESO policy, and non-disclosing renewable-token delivery script. Its 33 tests pass. Live configuration still waits for the attended OpenBao authority; concurrent tenant-engine workplan edits in that repository were preserved and excluded from this commit. Completed 2026-08-21 after the operator approved State Hub decision `4f70311b-a4c4-4959-b1a1-cdd1be97d523` and completed attended OpenBao OIDC. The policy, auth role, database roles, two namespace-limited stores, and three disjoint ExternalSecrets are live and Ready. The existing API token moved directly from source Kubernetes custody into OpenBao without appearing in output. Outside-namespace store denial, exact-path Kubernetes-auth denial, runtime DDL denial, migration DDL success, ESO sync, and live lease rotation all passed. `CCR-2026-0013` is verified. ## T05 — Support the private shadow restore and verification ```task id: RPF-WP-0021-T05 status: done priority: high state_hub_task_id: "ba4984f9-c6c5-5eb2-b4c4-83cbec59298c" ``` After T01-T04, support `CORE-WP-0011-T03`: label the namespace for `platform-pg`, restore or logically transfer the source database, reconcile table and row counts, run the migration Job with the migration lease, deploy with Ingress disabled, and verify health plus authenticated consumers against the private endpoint. This task ends with repeatable shadow evidence. It does not flip `hub.coulomb.social`, freeze production writes, or remove CoulombCore. Preflight 2026-08-20: the source has Alembic revision `20260627_0002`, eight public tables and 39 total rows. The admitted target intentionally has no application tables before restore. A ready disposable pod in the labeled `core-hub` namespace resolved the `platform-pg` service and reached its PostgreSQL port; it was deleted after the check. No dump or restore was attempted ahead of the T04 credential gate. Evidence: `docs/evidence/core-hub-shadow-preflight-2026-08-20.md`. Completed 2026-08-21. A consistent logical archive restored all eight source tables and 39 rows at Alembic revision `20260627_0002`; the restored tables were reconciled into the accepted `core_hub` schema with owner and runtime grants verified. Helm release revision 2 deployed chart `0.3.1` with Ingress disabled, and the migration Job completed. The hardened private verifier and authenticated smoke `20260820224419-43dc05` passed all health, readiness, catalog, denial, and protected-operation checks. Detailed non-secret evidence: `docs/evidence/core-hub-private-shadow-2026-08-21.md`. ## Acceptance - [x] Private Forgejo repo exists, validated commit `4403c9b` is preserved, and the registered workplan prefix is consistent. - [x] Core Hub is an accepted, capacity-reviewed `rapp-postgres` consumer. - [x] Static API-token and dynamic database credentials have distinct, least-privilege delivery paths. - [x] Database credential rotation cannot leave a running pod using an expired environment value. - [x] Private shadow restore, reconciliation, migration, and consumer smokes pass without enabling public Ingress.