feat: prepare core hub platform onboarding

2026-08-20 11:18:30 +02:00 · 2026-08-20 11:18:30 +02:00 · ac69f90300
commit ac69f90300
parent f19f89ae58
8 changed files with 473 additions and 0 deletions
--- a/WORK-RECORDS.md
+++ b/WORK-RECORDS.md
@ -23,6 +23,7 @@
 | workplan | RPF-WP-0018 | finished | — | workplans/RPF-WP-0018-policy-surface-alignment.md |
 | workplan | RPF-WP-0019 | active | — | workplans/RPF-WP-0019-apps-pg-recoverability-and-controls.md |
 | workplan | RPF-WP-0020 | proposed | — | workplans/RPF-WP-0020-ccr-schema-drift.md |
+| workplan | RPF-WP-0021 | active | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
 | task | RAILIANCE-WP-0005-T01 | done | — | workplans/RAILIANCE-WP-0005-credential-request-and-lease-broker.md |
 | task | RAILIANCE-WP-0005-T02 | done | — | workplans/RAILIANCE-WP-0005-credential-request-and-lease-broker.md |
 | task | RAILIANCE-WP-0005-T03 | done | — | workplans/RAILIANCE-WP-0005-credential-request-and-lease-broker.md |
@ -98,3 +99,8 @@
 | task | RPF-WP-0020-T02 | todo | — | workplans/RPF-WP-0020-ccr-schema-drift.md |
 | task | RPF-WP-0020-T03 | todo | — | workplans/RPF-WP-0020-ccr-schema-drift.md |
 | task | RPF-WP-0020-T04 | todo | — | workplans/RPF-WP-0020-ccr-schema-drift.md |
+| task | RPF-WP-0021-T01 | done | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
+| task | RPF-WP-0021-T02 | progress | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
+| task | RPF-WP-0021-T03 | progress | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
+| task | RPF-WP-0021-T04 | progress | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
+| task | RPF-WP-0021-T05 | wait | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
--- a/argocd/platform-addons/openbao-secretstore/openbao-core-hub-runtime.clustersecretstore.yaml
+++ b/argocd/platform-addons/openbao-secretstore/openbao-core-hub-runtime.clustersecretstore.yaml
@ -0,0 +1,30 @@
+# CCR-2026-0013. Railiance01-local OpenBao Kubernetes-auth store for the
+# standing Core Hub protected-API token. Dynamic PostgreSQL leases use a
+# separate database-mount store owned with the rapp-core-hub consumer package.
+#
+# Do not apply until CCR approval and confirmation of the
+# external-secrets-core-hub-runtime Kubernetes auth role.
+apiVersion: external-secrets.io/v1
+kind: ClusterSecretStore
+metadata:
+  name: openbao-core-hub-runtime
+  labels:
+    app.kubernetes.io/part-of: railiance-gitops
+    railiance-platform/component: external-secrets
+    app.kubernetes.io/name: core-hub
+spec:
+  provider:
+    vault:
+      server: http://openbao.openbao.svc:8200
+      path: platform
+      version: v2
+      auth:
+        kubernetes:
+          mountPath: kubernetes
+          role: external-secrets-core-hub-runtime
+          serviceAccountRef:
+            name: external-secrets
+            namespace: external-secrets
+  conditions:
+    - namespaces:
+        - core-hub
--- a/credential-change-requests/CCR-2026-0013-core-hub-runtime-api-token.yaml
+++ b/credential-change-requests/CCR-2026-0013-core-hub-runtime-api-token.yaml
@ -0,0 +1,115 @@
+id: CCR-2026-0013
+kind: credential-change-request
+schema_version: 1
+request_type: workload-kv-read
+title: Core Hub runtime API token lane
+status: proposed
+created: '2026-08-20'
+updated: '2026-08-20'
+requester:
+  agent: codex
+  reason: >-
+    CORE-WP-0011 and RAPP-CORE-HUB-WP-0001 relocate Core Hub production to
+    railiance01. The validated package requires CORE_HUB_API_TOKEN, but the
+    value must move through platform custody rather than a copied Kubernetes
+    Secret or Git. This request covers only the standing application token;
+    PostgreSQL credentials remain dynamic database-engine leases.
+review:
+  required: true
+  required_approvers:
+  - platform-operator
+  - core-hub-owner
+  comments: []
+target:
+  domain: infotech
+  tenant: core-hub
+  workload: core-hub
+  rapp: rapp-core-hub
+  environment: production
+  purpose: >-
+    Project the existing Core Hub protected-API token into the private
+    railiance01 runtime through a namespace-limited ExternalSecret while
+    preserving the token across the relocation window.
+openbao:
+  mount: platform
+  kv_path: platform/workloads/core-hub/runtime
+  fields:
+  - CORE_HUB_API_TOKEN
+  policy_name: workload-kv-read-core-hub-runtime
+  policy_file: openbao/policies/workload-kv-read-core-hub-runtime.hcl
+  auth:
+    method: kubernetes
+    mount: kubernetes
+    role: external-secrets-core-hub-runtime
+    bound_claims:
+      service_account_names:
+      - external-secrets
+      service_account_namespaces:
+      - external-secrets
+    bound_claims_confirmed: false
+    policies:
+    - workload-kv-read-core-hub-runtime
+    ttl: 15m
+access_frontdoor:
+  type: ops-warden
+  catalog_id: core-hub-runtime-api-token
+  selector: Core Hub runtime protected API token
+  command: warden access core-hub-runtime-api-token --fetch CORE_HUB_API_TOKEN
+  resolvable: false
+  readiness: pending-review
+delivery:
+  surface: external-secrets
+  target: >-
+    ClusterSecretStore openbao-core-hub-runtime, limited to namespace
+    core-hub, to ExternalSecret core-hub/core-hub-api-token and Secret
+    core-hub-api-token. The rapp package consumes this Secret separately from
+    runtime and migration PostgreSQL lease Secrets.
+risk:
+  classification: high
+  notes:
+  - The token authorizes protected Core Hub API operations and must not be
+    printed, placed on argv, committed, or recorded in State Hub.
+  - This CCR does not authorize a standing database password or a database URL
+    in KV. Those come from the OpenBao database engine.
+  - Preserve the current token during the relocation so existing protected API
+    consumers do not fail at the same time as the infrastructure move.
+  - Later rotation is coordinated with every protected API consumer because
+    Core Hub currently accepts one token.
+verification:
+  positive:
+  - An approved core-hub ExternalSecret syncs CORE_HUB_API_TOKEN into Secret
+    core-hub-api-token without printing the value.
+  - Protected API smoke succeeds against the private shadow endpoint using an
+    injected token after the source database has been reconciled.
+  negative:
+  - A namespace outside core-hub cannot use openbao-core-hub-runtime.
+  - A token without workload-kv-read-core-hub-runtime cannot read the KV data
+    path.
+  - The Core Hub workload identity cannot write or list sibling workload paths.
+  activation_conditions:
+  - Platform operator and Core Hub owner approve this CCR.
+  - The source token is transferred through attended operator custody without
+    entering logs, chat, Git, or State Hub.
+  - The Kubernetes auth role and policy are applied to railiance01-local
+    OpenBao and the bound service-account claims are verified.
+  - The package consumes the API-token Secret separately from dynamic database
+    lease Secrets.
+  - Positive and negative evidence is recorded with non-secret timestamps or
+    OpenBao audit references.
+  evidence: []
+lifecycle:
+  deactivate: >-
+    Disable the front door and Kubernetes auth role, remove the ExternalSecret,
+    and delete the materialized Secret only after Core Hub is retired or a
+    reviewed fallback is active.
+  rotate: >-
+    Coordinate a new random token with all protected API consumers, update the
+    OpenBao field through attended custody, wait for ESO sync, roll Core Hub and
+    consumers, verify protected calls, then invalidate the old token.
+  compromised: >-
+    Disable protected writes, replace the token in OpenBao and every consumer,
+    roll the workload, review Core Hub protected-operation logs, and open an
+    incident follow-up.
+state_hub:
+  workplan_id: RPF-WP-0021
+  task_id: RPF-WP-0021-T04
--- a/docs/evidence/core-hub-postgres-capacity-preflight-2026-08-20.md
+++ b/docs/evidence/core-hub-postgres-capacity-preflight-2026-08-20.md
@ -0,0 +1,58 @@
+# Core Hub PostgreSQL capacity preflight — 2026-08-20
+
+Workplan: `RPF-WP-0021-T02`
+
+Target: railiance01 `databases/platform-pg`
+
+Mode: live, read-only, non-secret
+
+## Purpose
+
+Record the current `platform-pg` resource posture before reviewing Core Hub as
+the fourth declaration on the cell. This is an admission preflight, not the
+controlled multi-consumer measurement required by `rapp-postgres` ADR-0004.
+
+## Observed baseline
+
+Collected through `ssh railiance01` using `kubectl top`, pod resource metadata,
+and read-only `pg_stat_activity` / database-size queries.
+
+| Signal | Observation |
+| --- | --- |
+| PostgreSQL pod CPU | 9m |
+| PostgreSQL pod memory | 362Mi |
+| CPU request / limit | 100m / 1 core |
+| Memory request / limit | 256Mi / 1Gi |
+| `max_connections` | 100 |
+| `shared_buffers` | 128MB |
+| `work_mem` | 4MB |
+| `maintenance_work_mem` | 64MB |
+
+Connection snapshot:
+
+| Database | Total | Active | Idle |
+| --- | ---: | ---: | ---: |
+| `audit_core` | 1 | 0 | 1 |
+| `postgres` | 2 | 1 | 1 |
+| no database | 6 | 0 | 0 |
+
+Database sizes were approximately 8MiB each for `audit_core`,
+`isolation_probe`, `platform_meta`, `postgres`, and `template1`.
+
+## Interpretation
+
+- The pod was using about 35% of its 1Gi memory limit during this quiet
+  snapshot. That is useful headroom evidence and not a peak or per-backend
+  measurement.
+- `tenant_engine` was not present in the database list, so this sample does
+  not exercise the declared second workload consumer.
+- Core Hub would reach the four-declaration ceiling after `audit-core`,
+  `tenant-engine`, and the counted isolation probe. It does not exceed the
+  ceiling if the stale, untracked Issue Core draft is removed as required by
+  `ISSUE-WP-0007`.
+- This snapshot does **not** close ADR-0004 section 5. Admission still needs a
+  coordinated measurement with the declared consumers active, including
+  connection/backend memory under representative load and a recorded peak.
+
+No PostgreSQL credential, query payload, tenant identifier, or secret value
+was read or recorded.
--- a/openbao/policies/agent-high-risk-boundary.hcl
+++ b/openbao/policies/agent-high-risk-boundary.hcl
@ -22,6 +22,12 @@ path "platform/data/workloads/activity-core/llm-connect/llm-connect-provider-sec
 path "platform/metadata/workloads/activity-core/llm-connect/llm-connect-provider-secrets" {
  capabilities = ["read"]
 }
+path "platform/data/workloads/core-hub/runtime" {
+  capabilities = ["deny"]
+}
+path "platform/metadata/workloads/core-hub/runtime" {
+  capabilities = ["read"]
+}

 # --- tenant high-risk (WARDEN-WP-0028) ---
 path "tenants/data/binky/company-email/imap" {
--- a/openbao/policies/workload-kv-read-core-hub-runtime.hcl
+++ b/openbao/policies/workload-kv-read-core-hub-runtime.hcl
@ -0,0 +1,7 @@
+path "platform/data/workloads/core-hub/runtime" {
+  capabilities = ["read"]
+}
+
+path "platform/metadata/workloads/core-hub/runtime" {
+  capabilities = ["read"]
+}
--- a/tests/test_credential_change.py
+++ b/tests/test_credential_change.py
@ -76,6 +76,34 @@ class CredentialChangeTests(unittest.TestCase):
        self.assertEqual(errors, [])
        self.assertEqual(ccr["target"]["rapp"], "rapp-qonto")

+    def test_core_hub_runtime_lane_is_split_and_agent_denied(self) -> None:
+        path = (
+            REPO_DIR
+            / "credential-change-requests/CCR-2026-0013-core-hub-runtime-api-token.yaml"
+        )
+        ccr, errors, warnings = credential_change.validate_ccr(path)
+        self.assertEqual(errors, [])
+        self.assertEqual(ccr["target"]["rapp"], "rapp-core-hub")
+        self.assertEqual(ccr["openbao"]["fields"], ["CORE_HUB_API_TOKEN"])
+        self.assertNotIn("CORE_HUB_DATABASE_URL", ccr["openbao"]["fields"])
+        self.assertTrue(any("bound claim is not confirmed" in item for item in warnings))
+
+        boundary = (
+            REPO_DIR / "openbao/policies/agent-high-risk-boundary.hcl"
+        ).read_text()
+        self.assertIn('path "platform/data/workloads/core-hub/runtime"', boundary)
+        self.assertRegex(
+            boundary,
+            r'path "platform/data/workloads/core-hub/runtime" \{\s*capabilities = \["deny"\]',
+        )
+
+        store = credential_change.load_yaml(
+            REPO_DIR
+            / "argocd/platform-addons/openbao-secretstore/openbao-core-hub-runtime.clustersecretstore.yaml"
+        )
+        self.assertEqual(store["spec"]["provider"]["vault"]["path"], "platform")
+        self.assertEqual(store["spec"]["conditions"][0]["namespaces"], ["core-hub"])
+
    def test_target_rapp_rejects_non_slug(self) -> None:
        path = self.unapproved_ccr()
        data = credential_change.load_yaml(path)
--- a/workplans/RPF-WP-0021-core-hub-platform-onboarding.md
+++ b/workplans/RPF-WP-0021-core-hub-platform-onboarding.md
@ -0,0 +1,223 @@
+---
+id: RPF-WP-0021
+type: workplan
+title: "Unblock rapp-core-hub publication and platform onboarding"
+domain: financials
+repo: railiance-platform
+status: active
+owner: codex
+topic_slug: railiance
+created: "2026-08-20"
+updated: "2026-08-20"
+related:
+  - CORE-WP-0011
+  - RAPP-CORE-HUB-WP-0001
+  - RAPP-POSTGRES-WP-0003
+origin: request
+origin_ref: CORE-WP-0011
+---
+
+# RPF-WP-0021 — Core Hub platform onboarding
+
+## Goal
+
+Provide the platform-owned dependencies that let the validated
+`rapp-core-hub` package publish and deploy as a private shadow runtime on
+railiance01 before the 2026-08-31 CoulombCore retirement deadline.
+
+This work stops before public traffic cutover, the final data freeze, or
+retirement of the old runtime. Those remain explicit operator gates in
+`CORE-WP-0011`.
+
+## Current evidence
+
+- `/home/worsch/rapp-core-hub` is clean at commit `4403c9b`; its recorded
+  package checks and railiance01 server-side dry-run passed.
+- `coulomb/rapp-core-hub` does not exist in Forgejo (HTTP 404 on 2026-08-20).
+- The routed `forgejo-admin-api-token` lane is active, but the present caller
+  receives HTTP 403 on the OpenBao mount lookup before `warden --exec` can
+  start the Forgejo request.
+- `agent-high-risk-boundary` deliberately denies agent data-read to the
+  Forgejo site-admin PAT. That control is not removed to make publication
+  convenient; an attended operator identity must authorize the child-process
+  transport.
+- The package requests a `rapp-postgres.v1` consumer and a Kubernetes Secret
+  containing `CORE_HUB_DATABASE_URL` plus `CORE_HUB_API_TOKEN`.
+- A live metadata-only railiance01 check on 2026-08-20 found OpenBao Ready and
+  the Kubernetes-auth-backed `openbao` ClusterSecretStore Valid. The existing
+  audit-core KV and database stores are also Valid and their ExternalSecrets
+  are synced, so the two-store shape has a live reference on the target rail.
+
+## Boundaries
+
+- No secret value enters Git, State Hub, logs, command output, or agent
+  context.
+- A repository is created only through the owned Forgejo admin lane.
+- The static Core Hub API token and dynamic PostgreSQL lease remain distinct
+  custody sources even if the workload consumes both as environment data.
+- A short-lived database lease is not copied into a standing OpenBao KV path.
+- A private shadow deployment is allowed only after the database credential
+  rotation behavior is explicit and testable.
+- Public Ingress, final transfer, DNS/tunnel changes, and old-runtime removal
+  remain outside this workplan.
+
+## T01 — Publish and register `rapp-core-hub`
+
+```task
+id: RPF-WP-0021-T01
+status: done
+priority: high
+```
+
+Under an attended `net-kingdom-admins` OIDC login to
+`forgejo-admin-workload-kv-read`, verify capabilities on the Forgejo admin KV
+data and metadata paths without reading the value. Then use
+`warden access forgejo-admin-api-token --exec` to create the private
+`coulomb/rapp-core-hub` repository, push commit `4403c9b`, configure the remote,
+and run `statehub fix-consistency` from the published repo.
+
+Current blocker: both the original request and a 2026-08-20 retry reached
+OpenBao but received HTTP 403 on
+`sys/internal/ui/mounts/platform/workloads/forgejo/forgejo-admin`. The retry
+also found the local warden audit path read-only. Do not weaken
+`agent-high-risk-boundary`; repair or establish the operator session and use
+the non-disclosing exec transport.
+
+Concurrent publication evidence, discovered at 07:55Z: Forgejo repository
+`coulomb/rapp-core-hub` was created at 07:42:41Z, but it is public rather than
+private and contains only Forgejo's `Initial commit` (`eea94d9`, README only).
+The local validated commit `4403c9b` is not the remote `main`, although the
+local remote is configured. Do not push the package into the public repo.
+First change visibility to private through the owned admin lane, correct the
+repository description, then push and verify `refs/heads/main` equals the
+validated commit.
+
+Completed 2026-08-20 after operator authorization. Unauthenticated Forgejo API
+access now returns 404 while the configured Git remote is accessible. Remote
+`main` is `cd43c9f`: its history preserves validated package commit `4403c9b`,
+the Forgejo README stub through merge `7dd057e`, the State Hub consistency
+commit, and the publication evidence update. Local and remote `main` match.
+`make check`, the railiance01 server-side dry-run, and family validation
+(`4 declaration(s) ok`) passed after publication. The repo is registered in
+State Hub; workplan UUID assignment remains with the production registrar.
+
+## T02 — Accept the Core Hub PostgreSQL consumer
+
+```task
+id: RPF-WP-0021-T02
+status: progress
+priority: high
+```
+
+Review `/home/worsch/rapp-core-hub/handoffs/postgres-consumer.yaml` in
+`rapp-postgres`, add the accepted declaration, render it, and run package
+tests. The requested database, schema, role split, connection limits, timeout,
+service class, and tenant-keying posture match the existing consumer contract.
+
+Capacity is an admission condition. With `audit-core`, `tenant-engine`, and
+the counted isolation probe, Core Hub is the fourth declaration on
+`platform-pg`: it reaches but does not exceed the ADR-0004 ceiling. Before
+admitting this third workload consumer, record the required measured memory
+baseline. The untracked `consumers/issue-core.yaml` is not a real consumer —
+`ISSUE-WP-0007` explicitly removed Postgres from that package — and must not
+silently consume a slot.
+
+Live apply, database role creation, namespace labeling, and OpenBao database
+role configuration remain attended operations after declaration review.
+
+Read-only preflight 2026-08-20: `platform-pg-1` used 9m CPU and 362Mi memory
+against a 1Gi limit, with only `audit_core` and operator sessions present;
+`tenant_engine` was not yet a database. This establishes current headroom but
+does not satisfy ADR-0004's representative multi-consumer peak measurement.
+Evidence: `docs/evidence/core-hub-postgres-capacity-preflight-2026-08-20.md`.
+
+## T03 — Correct the runtime credential delivery contract
+
+```task
+id: RPF-WP-0021-T03
+status: progress
+priority: high
+```
+
+Separate the two credential sources before shadow deployment:
+
+1. `CORE_HUB_API_TOKEN` is a standing workload secret in OpenBao KV, governed
+   by a proposed CCR and projected by a namespace-limited ESO store.
+2. Runtime and migration PostgreSQL credentials come from distinct
+   `database/creds/core-hub-*` roles and the database-engine ESO store. One
+   extract must provide each username/password pair so leases are not mixed.
+
+The current package consumes one `core-hub-runtime` Secret through `envFrom`
+for both the Deployment and migration Job. That is not yet a safe dynamic
+lease contract: environment variables do not update when ESO refreshes a
+Secret, and no workload reloader is declared. Update the package to consume
+separate runtime/migration database Secrets and the static API-token Secret,
+then either:
+
+- restart the Deployment automatically and demonstrably before each lease
+  expires; or
+- change Core Hub to re-read mounted credential files when opening a new
+  database connection.
+
+Prefer the mounted-file approach for steady state. A time-bounded automatic
+restart contract may be used for the relocation only if rollback and lease
+overlap are proven and the temporary limitation is recorded.
+
+## T04 — Declare and apply the OpenBao/ESO lanes
+
+```task
+id: RPF-WP-0021-T04
+status: progress
+priority: high
+```
+
+After T03 fixes the consuming contract:
+
+- draft and approve a workload-KV CCR for the static API token;
+- add a least-privilege ESO policy covering only the Core Hub static-token KV
+  path and `database/creds/core-hub-runtime` / `core-hub-migration`;
+- configure the database roles from the accepted Postgres declaration;
+- create namespace-limited KV and database `ClusterSecretStore` objects;
+- project runtime and migration Secrets without printing values; and
+- prove positive sync, cross-namespace denial, lease rotation, and migration
+  role separation.
+
+The working audit-core implementation is the reference for the two-store
+shape. Do not copy its earlier, rejected idea of freezing a dynamic lease in
+KV.
+
+Progress 2026-08-20: drafted `CCR-2026-0013` for the static API token, added
+its exact-path read policy, added a not-yet-applied namespace-limited
+`openbao-core-hub-runtime` ClusterSecretStore, and extended the agent
+high-risk boundary to deny direct data reads on the new path. Database-engine
+policy/store manifests wait for the accepted Postgres declaration and the T03
+consumer correction.
+
+## T05 — Support the private shadow restore and verification
+
+```task
+id: RPF-WP-0021-T05
+status: wait
+priority: high
+```
+
+After T01-T04, support `CORE-WP-0011-T03`: label the namespace for
+`platform-pg`, restore or logically transfer the source database, reconcile
+table and row counts, run the migration Job with the migration lease, deploy
+with Ingress disabled, and verify health plus authenticated consumers against
+the private endpoint.
+
+This task ends with repeatable shadow evidence. It does not flip
+`hub.coulomb.social`, freeze production writes, or remove CoulombCore.
+
+## Acceptance
+
+- [ ] Private Forgejo repo exists, commit `4403c9b` is pushed, and State Hub
+      registration is consistent.
+- [ ] Core Hub is an accepted, capacity-reviewed `rapp-postgres` consumer.
+- [ ] Static API-token and dynamic database credentials have distinct,
+      least-privilege delivery paths.
+- [ ] Database credential rotation cannot leave a running pod using an expired
+      environment value.
+- [ ] Private shadow restore, reconciliation, migration, and consumer smokes
+      pass without enabling public Ingress.