railiance-platform/workplans/RPF-WP-0021-core-hub-platform-onboarding.md

---
id: RPF-WP-0021
type: workplan
title: "Unblock rapp-core-hub publication and platform onboarding"
domain: financials
repo: railiance-platform
status: finished
owner: codex
topic_slug: railiance
created: "2026-08-20"
updated: "2026-08-21"
related:
  - CORE-WP-0011
  - RAPPCOREHUB-WP-0001
  - RAPP-POSTGRES-WP-0003
origin: request
origin_ref: CORE-WP-0011
state_hub_workstream_id: "aa7fe3ff-a76c-5f26-a1e0-a35e48b28d60"
---

# RPF-WP-0021 — Core Hub platform onboarding

## Goal

Provide the platform-owned dependencies that let the validated
`rapp-core-hub` package publish and deploy as a private shadow runtime on
railiance01 before the 2026-08-31 CoulombCore retirement deadline.

This work stops before public traffic cutover, the final data freeze, or
retirement of the old runtime. Those remain explicit operator gates in
`CORE-WP-0011`.

## Current evidence

- `/home/worsch/rapp-core-hub` is clean at commit `18e4e70`; its full package
  check, family validation, and railiance01 server-side dry-run passed.
- `coulomb/rapp-core-hub` is private and authoritative in Forgejo. Its history
  preserves validated package commit `4403c9b`, and unauthenticated API access
  returns 404.
- State Hub registered the repository and canonical workplan prefix
  `RAPPCOREHUB-WP-`. The production registrar still needs to assign the
  workplan UUID.
- `agent-high-risk-boundary` deliberately denies agent data-read to the
  Forgejo site-admin PAT. That control is not removed to make publication
  convenient; an attended operator identity must authorize the child-process
  transport.
- The package requests a `rapp-postgres.v1` consumer and a Kubernetes Secret
  containing `CORE_HUB_DATABASE_URL` plus `CORE_HUB_API_TOKEN`.
- A live metadata-only railiance01 check on 2026-08-20 found OpenBao Ready and
  the Kubernetes-auth-backed `openbao` ClusterSecretStore Valid. The existing
  audit-core KV and database stores are also Valid and their ExternalSecrets
  are synced, so the two-store shape has a live reference on the target rail.

## Boundaries

- No secret value enters Git, State Hub, logs, command output, or agent
  context.
- A repository is created only through the owned Forgejo admin lane.
- The static Core Hub API token and dynamic PostgreSQL lease remain distinct
  custody sources even if the workload consumes both as environment data.
- A short-lived database lease is not copied into a standing OpenBao KV path.
- A private shadow deployment is allowed only after the database credential
  rotation behavior is explicit and testable.
- Public Ingress, final transfer, DNS/tunnel changes, and old-runtime removal
  remain outside this workplan.

## T01 — Publish and register `rapp-core-hub`

```task
id: RPF-WP-0021-T01
status: done
priority: high
state_hub_task_id: "a7cd3fc0-4469-55a6-aed0-a8cacb34c033"
```

Under an attended `net-kingdom-admins` OIDC login to
`forgejo-admin-workload-kv-read`, verify capabilities on the Forgejo admin KV
data and metadata paths without reading the value. Then use
`warden access forgejo-admin-api-token --exec` to create the private
`coulomb/rapp-core-hub` repository, push commit `4403c9b`, configure the remote,
and run `statehub fix-consistency` from the published repo.

Current blocker: both the original request and a 2026-08-20 retry reached
OpenBao but received HTTP 403 on
`sys/internal/ui/mounts/platform/workloads/forgejo/forgejo-admin`. The retry
also found the local warden audit path read-only. Do not weaken
`agent-high-risk-boundary`; repair or establish the operator session and use
the non-disclosing exec transport.

Concurrent publication evidence, discovered at 07:55Z: Forgejo repository
`coulomb/rapp-core-hub` was created at 07:42:41Z, but it is public rather than
private and contains only Forgejo's `Initial commit` (`eea94d9`, README only).
The local validated commit `4403c9b` is not the remote `main`, although the
local remote is configured. Do not push the package into the public repo.
First change visibility to private through the owned admin lane, correct the
repository description, then push and verify `refs/heads/main` equals the
validated commit.

Completed 2026-08-20 after operator authorization. Unauthenticated Forgejo API
access now returns 404 while the configured Git remote is accessible. Remote
`main` is `18e4e70`: its history preserves validated package commit `4403c9b`,
the Forgejo README stub through merge `7dd057e`, publication evidence, the
project goal, and the registered-prefix alignment. Local and remote `main`
match.
`make check`, the railiance01 server-side dry-run, and family validation
(`4 declaration(s) ok`) passed after publication. The repo is registered in
State Hub; workplan UUID assignment remains with the production registrar.

## T02 — Accept the Core Hub PostgreSQL consumer

```task
id: RPF-WP-0021-T02
status: done
priority: high
state_hub_task_id: "7cac453b-2e85-56d9-abae-d983e382246b"
```

Review `/home/worsch/rapp-core-hub/handoffs/postgres-consumer.yaml` in
`rapp-postgres`, add the accepted declaration, render it, and run package
tests. The requested database, schema, role split, connection limits, timeout,
service class, and tenant-keying posture match the existing consumer contract.

Capacity is an admission condition. With `audit-core`, `tenant-engine`, and
the counted isolation probe, Core Hub is the fourth declaration on
`platform-pg`: it reaches but does not exceed the ADR-0004 ceiling. Before
admitting this third workload consumer, record the required measured memory
baseline. The untracked `consumers/issue-core.yaml` is not a real consumer —
`ISSUE-WP-0007` explicitly removed Postgres from that package — and must not
silently consume a slot.

OpenBao database role configuration remains an attended operation under T04.

Read-only preflight 2026-08-20: `platform-pg-1` used 9m CPU and 362Mi memory
against a 1Gi limit, with only `audit_core` and operator sessions present;
`tenant_engine` was not yet a database. This establishes current headroom but
does not satisfy ADR-0004's representative multi-consumer peak measurement.
Evidence: `docs/evidence/core-hub-postgres-capacity-preflight-2026-08-20.md`.

Capacity admission completed 2026-08-20. A bounded live probe opened 56
sessions in four declaration groups and repeated the occupancy with one active
bounded sort per group. `platform-pg` peaked at 174Mi/1Gi; the 85Mi increase is
about 1.52Mi per opened backend. Even adding that delta to the earlier 362Mi
sample yields 447Mi, under 44% of the limit. Core Hub is accepted as the fourth
and final declaration without raising the pod limit. Evidence:
`docs/evidence/core-hub-postgres-capacity-admission-2026-08-20.md`.

Completed 2026-08-20. `rapp-postgres` commit `4f316cc` accepts and publishes
the Core Hub declaration as the fourth `platform-pg` consumer. All 31 package
tests and consumer rendering passed; the railiance01 server-side dry-run
accepted the generated resource. The idempotent package provisioner then
created `core_hub`, its `core_hub` schema, distinct NOLOGIN owner/migration/
runtime roles and bounded role settings. CNPG `Database/core-hub` reports
`status.applied=true`, targets `platform-pg`, retains the database on resource
deletion, and names `core_hub_owner` as owner.

## T03 — Correct the runtime credential delivery contract

```task
id: RPF-WP-0021-T03
status: done
priority: high
state_hub_task_id: "223d5a77-b7c3-5d1a-a93c-4cb9628b457a"
```

Separate the two credential sources before shadow deployment:

1. `CORE_HUB_API_TOKEN` is a standing workload secret in OpenBao KV, governed
   by a proposed CCR and projected by a namespace-limited ESO store.
2. Runtime and migration PostgreSQL credentials come from distinct
   `database/creds/core-hub-*` roles and the database-engine ESO store. One
   extract must provide each username/password pair so leases are not mixed.

The current package consumes one `core-hub-runtime` Secret through `envFrom`
for both the Deployment and migration Job. That is not yet a safe dynamic
lease contract: environment variables do not update when ESO refreshes a
Secret, and no workload reloader is declared. Update the package to consume
separate runtime/migration database Secrets and the static API-token Secret,
then either:

- restart the Deployment automatically and demonstrably before each lease
  expires; or
- change Core Hub to re-read mounted credential files when opening a new
  database connection.

Prefer the mounted-file approach for steady state. A time-bounded automatic
restart contract may be used for the relocation only if rollback and lease
overlap are proven and the temporary limitation is recorded.

Completed 2026-08-21. Static API, runtime database, and migration database
credentials are mounted from three disjoint Secrets; the database URL exists
only in the supervised child environment, and the migration Job mounts only
its role. Live proof exposed that chart `0.3.0` watcher exit 75 could accumulate
planned Kubelet restarts into CrashLoopBackOff. Chart `0.3.1` (`rapp-core-hub`
commit `7cfa8bb`) instead keeps PID 1 alive and restarts only the application
child after reading the refreshed files. Nine package tests and live proof
showed a changed Secret resourceVersion and child PID with stable pod UID,
container restart count `0 -> 0`, restored readiness, and the five-minute
refresh interval restored.

## T04 — Declare and apply the OpenBao/ESO lanes

```task
id: RPF-WP-0021-T04
status: done
priority: high
state_hub_task_id: "50713b54-38a8-543d-8c74-0e3fa186fbb8"
```

After T03 fixes the consuming contract:

- draft and approve a workload-KV CCR for the static API token;
- add a least-privilege ESO policy covering only the Core Hub static-token KV
  path and `database/creds/core-hub-runtime` / `core-hub-migration`;
- configure the database roles from the accepted Postgres declaration;
- create namespace-limited KV and database `ClusterSecretStore` objects;
- project runtime and migration Secrets without printing values; and
- prove positive sync, cross-namespace denial, lease rotation, and migration
  role separation.

The working audit-core implementation is the reference for the two-store
shape. Do not copy its earlier, rejected idea of freezing a dynamic lease in
KV.

Progress 2026-08-20: drafted `CCR-2026-0013` for the static API token, added
its exact-path read policy, added a not-yet-applied namespace-limited
`openbao-core-hub-runtime` ClusterSecretStore, and extended the agent
high-risk boundary to deny direct data reads on the new path. Database-engine
policy/store manifests wait for the accepted Postgres declaration and the T03
consumer correction.

Progress 2026-08-20 after T02 admission: added an exact two-path database ESO
policy, a namespace-limited `openbao-core-hub-database` store, and three
disjoint ExternalSecrets. Runtime and migration each use one `dataFrom.extract`
so a username/password pair comes from one lease; both refresh every five
minutes. The database store and all projections passed railiance01 server-side
dry-run. The package namespace is live with the reviewed `platform-pg` client
label. Live role/store activation correctly stopped because the current
OpenBao session receives 403 and CCR-2026-0013 is still proposed; no authority
or placeholder Secret was improvised. State Hub decision
`4f70311b-a4c4-4959-b1a1-cdd1be97d523` now carries the non-secret review
coordinates and awaits the required platform-operator/Core-Hub-owner approval.
`rapp-postgres` commit `cfde57a` now also owns the Core Hub database-engine
roles, 30-minute runtime/10-minute migration TTLs, exact two-path ESO policy,
and non-disclosing renewable-token delivery script. Its 33 tests pass. Live
configuration still waits for the attended OpenBao authority; concurrent
tenant-engine workplan edits in that repository were preserved and excluded
from this commit.

Completed 2026-08-21 after the operator approved State Hub decision
`4f70311b-a4c4-4959-b1a1-cdd1be97d523` and completed attended OpenBao OIDC.
The policy, auth role, database roles, two namespace-limited stores, and three
disjoint ExternalSecrets are live and Ready. The existing API token moved
directly from source Kubernetes custody into OpenBao without appearing in
output. Outside-namespace store denial, exact-path Kubernetes-auth denial,
runtime DDL denial, migration DDL success, ESO sync, and live lease rotation
all passed. `CCR-2026-0013` is verified.

## T05 — Support the private shadow restore and verification

```task
id: RPF-WP-0021-T05
status: done
priority: high
state_hub_task_id: "ba4984f9-c6c5-5eb2-b4c4-83cbec59298c"
```

After T01-T04, support `CORE-WP-0011-T03`: label the namespace for
`platform-pg`, restore or logically transfer the source database, reconcile
table and row counts, run the migration Job with the migration lease, deploy
with Ingress disabled, and verify health plus authenticated consumers against
the private endpoint.

This task ends with repeatable shadow evidence. It does not flip
`hub.coulomb.social`, freeze production writes, or remove CoulombCore.

Preflight 2026-08-20: the source has Alembic revision `20260627_0002`, eight
public tables and 39 total rows. The admitted target intentionally has no
application tables before restore. A ready disposable pod in the labeled
`core-hub` namespace resolved the `platform-pg` service and reached its
PostgreSQL port; it was deleted after the check. No dump or restore was
attempted ahead of the T04 credential gate. Evidence:
`docs/evidence/core-hub-shadow-preflight-2026-08-20.md`.

Completed 2026-08-21. A consistent logical archive restored all eight source
tables and 39 rows at Alembic revision `20260627_0002`; the restored tables
were reconciled into the accepted `core_hub` schema with owner and runtime
grants verified. Helm release revision 2 deployed chart `0.3.1` with Ingress
disabled, and the migration Job completed. The hardened private verifier and
authenticated smoke `20260820224419-43dc05` passed all health, readiness,
catalog, denial, and protected-operation checks. Detailed non-secret evidence:
`docs/evidence/core-hub-private-shadow-2026-08-21.md`.

## Acceptance

- [x] Private Forgejo repo exists, validated commit `4403c9b` is preserved,
      and the registered workplan prefix is consistent.
- [x] Core Hub is an accepted, capacity-reviewed `rapp-postgres` consumer.
- [x] Static API-token and dynamic database credentials have distinct,
      least-privilege delivery paths.
- [x] Database credential rotation cannot leave a running pod using an expired
      environment value.
- [x] Private shadow restore, reconciliation, migration, and consumer smokes
      pass without enabling public Ingress.