railiance-platform/workplans/RPF-WP-0021-core-hub-platform-onboarding.md
codex ceed3c051d
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s
docs: record provider incident and core hub rotation
2026-08-20 22:47:38 +02:00

10 KiB

id type title domain repo status owner topic_slug created updated related origin origin_ref
RPF-WP-0021 workplan Unblock rapp-core-hub publication and platform onboarding financials railiance-platform active codex railiance 2026-08-20 2026-08-20
CORE-WP-0011
RAPPCOREHUB-WP-0001
RAPP-POSTGRES-WP-0003
request CORE-WP-0011

RPF-WP-0021 — Core Hub platform onboarding

Goal

Provide the platform-owned dependencies that let the validated rapp-core-hub package publish and deploy as a private shadow runtime on railiance01 before the 2026-08-31 CoulombCore retirement deadline.

This work stops before public traffic cutover, the final data freeze, or retirement of the old runtime. Those remain explicit operator gates in CORE-WP-0011.

Current evidence

  • /home/worsch/rapp-core-hub is clean at commit 18e4e70; its full package check, family validation, and railiance01 server-side dry-run passed.
  • coulomb/rapp-core-hub is private and authoritative in Forgejo. Its history preserves validated package commit 4403c9b, and unauthenticated API access returns 404.
  • State Hub registered the repository and canonical workplan prefix RAPPCOREHUB-WP-. The production registrar still needs to assign the workplan UUID.
  • agent-high-risk-boundary deliberately denies agent data-read to the Forgejo site-admin PAT. That control is not removed to make publication convenient; an attended operator identity must authorize the child-process transport.
  • The package requests a rapp-postgres.v1 consumer and a Kubernetes Secret containing CORE_HUB_DATABASE_URL plus CORE_HUB_API_TOKEN.
  • A live metadata-only railiance01 check on 2026-08-20 found OpenBao Ready and the Kubernetes-auth-backed openbao ClusterSecretStore Valid. The existing audit-core KV and database stores are also Valid and their ExternalSecrets are synced, so the two-store shape has a live reference on the target rail.

Boundaries

  • No secret value enters Git, State Hub, logs, command output, or agent context.
  • A repository is created only through the owned Forgejo admin lane.
  • The static Core Hub API token and dynamic PostgreSQL lease remain distinct custody sources even if the workload consumes both as environment data.
  • A short-lived database lease is not copied into a standing OpenBao KV path.
  • A private shadow deployment is allowed only after the database credential rotation behavior is explicit and testable.
  • Public Ingress, final transfer, DNS/tunnel changes, and old-runtime removal remain outside this workplan.

T01 — Publish and register rapp-core-hub

id: RPF-WP-0021-T01
status: done
priority: high

Under an attended net-kingdom-admins OIDC login to forgejo-admin-workload-kv-read, verify capabilities on the Forgejo admin KV data and metadata paths without reading the value. Then use warden access forgejo-admin-api-token --exec to create the private coulomb/rapp-core-hub repository, push commit 4403c9b, configure the remote, and run statehub fix-consistency from the published repo.

Current blocker: both the original request and a 2026-08-20 retry reached OpenBao but received HTTP 403 on sys/internal/ui/mounts/platform/workloads/forgejo/forgejo-admin. The retry also found the local warden audit path read-only. Do not weaken agent-high-risk-boundary; repair or establish the operator session and use the non-disclosing exec transport.

Concurrent publication evidence, discovered at 07:55Z: Forgejo repository coulomb/rapp-core-hub was created at 07:42:41Z, but it is public rather than private and contains only Forgejo's Initial commit (eea94d9, README only). The local validated commit 4403c9b is not the remote main, although the local remote is configured. Do not push the package into the public repo. First change visibility to private through the owned admin lane, correct the repository description, then push and verify refs/heads/main equals the validated commit.

Completed 2026-08-20 after operator authorization. Unauthenticated Forgejo API access now returns 404 while the configured Git remote is accessible. Remote main is 18e4e70: its history preserves validated package commit 4403c9b, the Forgejo README stub through merge 7dd057e, publication evidence, the project goal, and the registered-prefix alignment. Local and remote main match. make check, the railiance01 server-side dry-run, and family validation (4 declaration(s) ok) passed after publication. The repo is registered in State Hub; workplan UUID assignment remains with the production registrar.

T02 — Accept the Core Hub PostgreSQL consumer

id: RPF-WP-0021-T02
status: progress
priority: high

Review /home/worsch/rapp-core-hub/handoffs/postgres-consumer.yaml in rapp-postgres, add the accepted declaration, render it, and run package tests. The requested database, schema, role split, connection limits, timeout, service class, and tenant-keying posture match the existing consumer contract.

Capacity is an admission condition. With audit-core, tenant-engine, and the counted isolation probe, Core Hub is the fourth declaration on platform-pg: it reaches but does not exceed the ADR-0004 ceiling. Before admitting this third workload consumer, record the required measured memory baseline. The untracked consumers/issue-core.yaml is not a real consumer — ISSUE-WP-0007 explicitly removed Postgres from that package — and must not silently consume a slot.

Live apply, database role creation, namespace labeling, and OpenBao database role configuration remain attended operations after declaration review.

Read-only preflight 2026-08-20: platform-pg-1 used 9m CPU and 362Mi memory against a 1Gi limit, with only audit_core and operator sessions present; tenant_engine was not yet a database. This establishes current headroom but does not satisfy ADR-0004's representative multi-consumer peak measurement. Evidence: docs/evidence/core-hub-postgres-capacity-preflight-2026-08-20.md.

T03 — Correct the runtime credential delivery contract

id: RPF-WP-0021-T03
status: progress
priority: high

Separate the two credential sources before shadow deployment:

  1. CORE_HUB_API_TOKEN is a standing workload secret in OpenBao KV, governed by a proposed CCR and projected by a namespace-limited ESO store.
  2. Runtime and migration PostgreSQL credentials come from distinct database/creds/core-hub-* roles and the database-engine ESO store. One extract must provide each username/password pair so leases are not mixed.

The current package consumes one core-hub-runtime Secret through envFrom for both the Deployment and migration Job. That is not yet a safe dynamic lease contract: environment variables do not update when ESO refreshes a Secret, and no workload reloader is declared. Update the package to consume separate runtime/migration database Secrets and the static API-token Secret, then either:

  • restart the Deployment automatically and demonstrably before each lease expires; or
  • change Core Hub to re-read mounted credential files when opening a new database connection.

Prefer the mounted-file approach for steady state. A time-bounded automatic restart contract may be used for the relocation only if rollback and lease overlap are proven and the temporary limitation is recorded.

Progress 2026-08-20: package chart 0.3.0 implements the bounded relocation contract. Static API, runtime database, and migration database credentials are mounted from three disjoint Secrets; the database URL is constructed only in the supervised child environment. A PID 1 watcher fingerprints mounted files without logging values and exits 75 on change so Kubernetes restarts the API on the refreshed lease. The migration Job mounts only its role. Seven package tests prove separation, URL encoding, no credential output, and rotation exit; Helm lint, family validation, normal server dry-run, and migration-enabled server dry-run pass. T03 remains progress until T04 proves a live ESO refresh, container replacement, and readiness before the old runtime lease expires.

T04 — Declare and apply the OpenBao/ESO lanes

id: RPF-WP-0021-T04
status: progress
priority: high

After T03 fixes the consuming contract:

  • draft and approve a workload-KV CCR for the static API token;
  • add a least-privilege ESO policy covering only the Core Hub static-token KV path and database/creds/core-hub-runtime / core-hub-migration;
  • configure the database roles from the accepted Postgres declaration;
  • create namespace-limited KV and database ClusterSecretStore objects;
  • project runtime and migration Secrets without printing values; and
  • prove positive sync, cross-namespace denial, lease rotation, and migration role separation.

The working audit-core implementation is the reference for the two-store shape. Do not copy its earlier, rejected idea of freezing a dynamic lease in KV.

Progress 2026-08-20: drafted CCR-2026-0013 for the static API token, added its exact-path read policy, added a not-yet-applied namespace-limited openbao-core-hub-runtime ClusterSecretStore, and extended the agent high-risk boundary to deny direct data reads on the new path. Database-engine policy/store manifests wait for the accepted Postgres declaration and the T03 consumer correction.

T05 — Support the private shadow restore and verification

id: RPF-WP-0021-T05
status: wait
priority: high

After T01-T04, support CORE-WP-0011-T03: label the namespace for platform-pg, restore or logically transfer the source database, reconcile table and row counts, run the migration Job with the migration lease, deploy with Ingress disabled, and verify health plus authenticated consumers against the private endpoint.

This task ends with repeatable shadow evidence. It does not flip hub.coulomb.social, freeze production writes, or remove CoulombCore.

Acceptance

  • Private Forgejo repo exists, validated commit 4403c9b is preserved, and the registered workplan prefix is consistent.
  • Core Hub is an accepted, capacity-reviewed rapp-postgres consumer.
  • Static API-token and dynamic database credentials have distinct, least-privilege delivery paths.
  • Database credential rotation cannot leave a running pod using an expired environment value.
  • Private shadow restore, reconciliation, migration, and consumer smokes pass without enabling public Ingress.