diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index f313af4..44d070a 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -23,6 +23,7 @@ | workplan | RPF-WP-0018 | finished | — | workplans/RPF-WP-0018-policy-surface-alignment.md | | workplan | RPF-WP-0019 | active | — | workplans/RPF-WP-0019-apps-pg-recoverability-and-controls.md | | workplan | RPF-WP-0020 | proposed | — | workplans/RPF-WP-0020-ccr-schema-drift.md | +| workplan | RPF-WP-0021 | active | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md | | task | RAILIANCE-WP-0005-T01 | done | — | workplans/RAILIANCE-WP-0005-credential-request-and-lease-broker.md | | task | RAILIANCE-WP-0005-T02 | done | — | workplans/RAILIANCE-WP-0005-credential-request-and-lease-broker.md | | task | RAILIANCE-WP-0005-T03 | done | — | workplans/RAILIANCE-WP-0005-credential-request-and-lease-broker.md | @@ -98,3 +99,8 @@ | task | RPF-WP-0020-T02 | todo | — | workplans/RPF-WP-0020-ccr-schema-drift.md | | task | RPF-WP-0020-T03 | todo | — | workplans/RPF-WP-0020-ccr-schema-drift.md | | task | RPF-WP-0020-T04 | todo | — | workplans/RPF-WP-0020-ccr-schema-drift.md | +| task | RPF-WP-0021-T01 | done | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md | +| task | RPF-WP-0021-T02 | progress | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md | +| task | RPF-WP-0021-T03 | progress | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md | +| task | RPF-WP-0021-T04 | progress | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md | +| task | RPF-WP-0021-T05 | wait | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md | diff --git a/argocd/platform-addons/openbao-secretstore/openbao-core-hub-runtime.clustersecretstore.yaml b/argocd/platform-addons/openbao-secretstore/openbao-core-hub-runtime.clustersecretstore.yaml new file mode 100644 index 0000000..35aea25 --- /dev/null +++ b/argocd/platform-addons/openbao-secretstore/openbao-core-hub-runtime.clustersecretstore.yaml @@ -0,0 +1,30 @@ +# CCR-2026-0013. Railiance01-local OpenBao Kubernetes-auth store for the +# standing Core Hub protected-API token. Dynamic PostgreSQL leases use a +# separate database-mount store owned with the rapp-core-hub consumer package. +# +# Do not apply until CCR approval and confirmation of the +# external-secrets-core-hub-runtime Kubernetes auth role. +apiVersion: external-secrets.io/v1 +kind: ClusterSecretStore +metadata: + name: openbao-core-hub-runtime + labels: + app.kubernetes.io/part-of: railiance-gitops + railiance-platform/component: external-secrets + app.kubernetes.io/name: core-hub +spec: + provider: + vault: + server: http://openbao.openbao.svc:8200 + path: platform + version: v2 + auth: + kubernetes: + mountPath: kubernetes + role: external-secrets-core-hub-runtime + serviceAccountRef: + name: external-secrets + namespace: external-secrets + conditions: + - namespaces: + - core-hub diff --git a/credential-change-requests/CCR-2026-0013-core-hub-runtime-api-token.yaml b/credential-change-requests/CCR-2026-0013-core-hub-runtime-api-token.yaml new file mode 100644 index 0000000..e05c5d6 --- /dev/null +++ b/credential-change-requests/CCR-2026-0013-core-hub-runtime-api-token.yaml @@ -0,0 +1,115 @@ +id: CCR-2026-0013 +kind: credential-change-request +schema_version: 1 +request_type: workload-kv-read +title: Core Hub runtime API token lane +status: proposed +created: '2026-08-20' +updated: '2026-08-20' +requester: + agent: codex + reason: >- + CORE-WP-0011 and RAPP-CORE-HUB-WP-0001 relocate Core Hub production to + railiance01. The validated package requires CORE_HUB_API_TOKEN, but the + value must move through platform custody rather than a copied Kubernetes + Secret or Git. This request covers only the standing application token; + PostgreSQL credentials remain dynamic database-engine leases. +review: + required: true + required_approvers: + - platform-operator + - core-hub-owner + comments: [] +target: + domain: infotech + tenant: core-hub + workload: core-hub + rapp: rapp-core-hub + environment: production + purpose: >- + Project the existing Core Hub protected-API token into the private + railiance01 runtime through a namespace-limited ExternalSecret while + preserving the token across the relocation window. +openbao: + mount: platform + kv_path: platform/workloads/core-hub/runtime + fields: + - CORE_HUB_API_TOKEN + policy_name: workload-kv-read-core-hub-runtime + policy_file: openbao/policies/workload-kv-read-core-hub-runtime.hcl + auth: + method: kubernetes + mount: kubernetes + role: external-secrets-core-hub-runtime + bound_claims: + service_account_names: + - external-secrets + service_account_namespaces: + - external-secrets + bound_claims_confirmed: false + policies: + - workload-kv-read-core-hub-runtime + ttl: 15m +access_frontdoor: + type: ops-warden + catalog_id: core-hub-runtime-api-token + selector: Core Hub runtime protected API token + command: warden access core-hub-runtime-api-token --fetch CORE_HUB_API_TOKEN + resolvable: false + readiness: pending-review +delivery: + surface: external-secrets + target: >- + ClusterSecretStore openbao-core-hub-runtime, limited to namespace + core-hub, to ExternalSecret core-hub/core-hub-api-token and Secret + core-hub-api-token. The rapp package consumes this Secret separately from + runtime and migration PostgreSQL lease Secrets. +risk: + classification: high + notes: + - The token authorizes protected Core Hub API operations and must not be + printed, placed on argv, committed, or recorded in State Hub. + - This CCR does not authorize a standing database password or a database URL + in KV. Those come from the OpenBao database engine. + - Preserve the current token during the relocation so existing protected API + consumers do not fail at the same time as the infrastructure move. + - Later rotation is coordinated with every protected API consumer because + Core Hub currently accepts one token. +verification: + positive: + - An approved core-hub ExternalSecret syncs CORE_HUB_API_TOKEN into Secret + core-hub-api-token without printing the value. + - Protected API smoke succeeds against the private shadow endpoint using an + injected token after the source database has been reconciled. + negative: + - A namespace outside core-hub cannot use openbao-core-hub-runtime. + - A token without workload-kv-read-core-hub-runtime cannot read the KV data + path. + - The Core Hub workload identity cannot write or list sibling workload paths. + activation_conditions: + - Platform operator and Core Hub owner approve this CCR. + - The source token is transferred through attended operator custody without + entering logs, chat, Git, or State Hub. + - The Kubernetes auth role and policy are applied to railiance01-local + OpenBao and the bound service-account claims are verified. + - The package consumes the API-token Secret separately from dynamic database + lease Secrets. + - Positive and negative evidence is recorded with non-secret timestamps or + OpenBao audit references. + evidence: [] +lifecycle: + deactivate: >- + Disable the front door and Kubernetes auth role, remove the ExternalSecret, + and delete the materialized Secret only after Core Hub is retired or a + reviewed fallback is active. + rotate: >- + Coordinate a new random token with all protected API consumers, update the + OpenBao field through attended custody, wait for ESO sync, roll Core Hub and + consumers, verify protected calls, then invalidate the old token. + compromised: >- + Disable protected writes, replace the token in OpenBao and every consumer, + roll the workload, review Core Hub protected-operation logs, and open an + incident follow-up. +state_hub: + workplan_id: RPF-WP-0021 + task_id: RPF-WP-0021-T04 diff --git a/docs/evidence/core-hub-postgres-capacity-preflight-2026-08-20.md b/docs/evidence/core-hub-postgres-capacity-preflight-2026-08-20.md new file mode 100644 index 0000000..ef827b6 --- /dev/null +++ b/docs/evidence/core-hub-postgres-capacity-preflight-2026-08-20.md @@ -0,0 +1,58 @@ +# Core Hub PostgreSQL capacity preflight — 2026-08-20 + +Workplan: `RPF-WP-0021-T02` + +Target: railiance01 `databases/platform-pg` + +Mode: live, read-only, non-secret + +## Purpose + +Record the current `platform-pg` resource posture before reviewing Core Hub as +the fourth declaration on the cell. This is an admission preflight, not the +controlled multi-consumer measurement required by `rapp-postgres` ADR-0004. + +## Observed baseline + +Collected through `ssh railiance01` using `kubectl top`, pod resource metadata, +and read-only `pg_stat_activity` / database-size queries. + +| Signal | Observation | +| --- | --- | +| PostgreSQL pod CPU | 9m | +| PostgreSQL pod memory | 362Mi | +| CPU request / limit | 100m / 1 core | +| Memory request / limit | 256Mi / 1Gi | +| `max_connections` | 100 | +| `shared_buffers` | 128MB | +| `work_mem` | 4MB | +| `maintenance_work_mem` | 64MB | + +Connection snapshot: + +| Database | Total | Active | Idle | +| --- | ---: | ---: | ---: | +| `audit_core` | 1 | 0 | 1 | +| `postgres` | 2 | 1 | 1 | +| no database | 6 | 0 | 0 | + +Database sizes were approximately 8MiB each for `audit_core`, +`isolation_probe`, `platform_meta`, `postgres`, and `template1`. + +## Interpretation + +- The pod was using about 35% of its 1Gi memory limit during this quiet + snapshot. That is useful headroom evidence and not a peak or per-backend + measurement. +- `tenant_engine` was not present in the database list, so this sample does + not exercise the declared second workload consumer. +- Core Hub would reach the four-declaration ceiling after `audit-core`, + `tenant-engine`, and the counted isolation probe. It does not exceed the + ceiling if the stale, untracked Issue Core draft is removed as required by + `ISSUE-WP-0007`. +- This snapshot does **not** close ADR-0004 section 5. Admission still needs a + coordinated measurement with the declared consumers active, including + connection/backend memory under representative load and a recorded peak. + +No PostgreSQL credential, query payload, tenant identifier, or secret value +was read or recorded. diff --git a/openbao/policies/agent-high-risk-boundary.hcl b/openbao/policies/agent-high-risk-boundary.hcl index 1cec247..c9df4c1 100644 --- a/openbao/policies/agent-high-risk-boundary.hcl +++ b/openbao/policies/agent-high-risk-boundary.hcl @@ -22,6 +22,12 @@ path "platform/data/workloads/activity-core/llm-connect/llm-connect-provider-sec path "platform/metadata/workloads/activity-core/llm-connect/llm-connect-provider-secrets" { capabilities = ["read"] } +path "platform/data/workloads/core-hub/runtime" { + capabilities = ["deny"] +} +path "platform/metadata/workloads/core-hub/runtime" { + capabilities = ["read"] +} # --- tenant high-risk (WARDEN-WP-0028) --- path "tenants/data/binky/company-email/imap" { diff --git a/openbao/policies/workload-kv-read-core-hub-runtime.hcl b/openbao/policies/workload-kv-read-core-hub-runtime.hcl new file mode 100644 index 0000000..8a679e6 --- /dev/null +++ b/openbao/policies/workload-kv-read-core-hub-runtime.hcl @@ -0,0 +1,7 @@ +path "platform/data/workloads/core-hub/runtime" { + capabilities = ["read"] +} + +path "platform/metadata/workloads/core-hub/runtime" { + capabilities = ["read"] +} diff --git a/tests/test_credential_change.py b/tests/test_credential_change.py index 50682f7..eef1c62 100644 --- a/tests/test_credential_change.py +++ b/tests/test_credential_change.py @@ -76,6 +76,34 @@ class CredentialChangeTests(unittest.TestCase): self.assertEqual(errors, []) self.assertEqual(ccr["target"]["rapp"], "rapp-qonto") + def test_core_hub_runtime_lane_is_split_and_agent_denied(self) -> None: + path = ( + REPO_DIR + / "credential-change-requests/CCR-2026-0013-core-hub-runtime-api-token.yaml" + ) + ccr, errors, warnings = credential_change.validate_ccr(path) + self.assertEqual(errors, []) + self.assertEqual(ccr["target"]["rapp"], "rapp-core-hub") + self.assertEqual(ccr["openbao"]["fields"], ["CORE_HUB_API_TOKEN"]) + self.assertNotIn("CORE_HUB_DATABASE_URL", ccr["openbao"]["fields"]) + self.assertTrue(any("bound claim is not confirmed" in item for item in warnings)) + + boundary = ( + REPO_DIR / "openbao/policies/agent-high-risk-boundary.hcl" + ).read_text() + self.assertIn('path "platform/data/workloads/core-hub/runtime"', boundary) + self.assertRegex( + boundary, + r'path "platform/data/workloads/core-hub/runtime" \{\s*capabilities = \["deny"\]', + ) + + store = credential_change.load_yaml( + REPO_DIR + / "argocd/platform-addons/openbao-secretstore/openbao-core-hub-runtime.clustersecretstore.yaml" + ) + self.assertEqual(store["spec"]["provider"]["vault"]["path"], "platform") + self.assertEqual(store["spec"]["conditions"][0]["namespaces"], ["core-hub"]) + def test_target_rapp_rejects_non_slug(self) -> None: path = self.unapproved_ccr() data = credential_change.load_yaml(path) diff --git a/workplans/RPF-WP-0021-core-hub-platform-onboarding.md b/workplans/RPF-WP-0021-core-hub-platform-onboarding.md new file mode 100644 index 0000000..f319053 --- /dev/null +++ b/workplans/RPF-WP-0021-core-hub-platform-onboarding.md @@ -0,0 +1,223 @@ +--- +id: RPF-WP-0021 +type: workplan +title: "Unblock rapp-core-hub publication and platform onboarding" +domain: financials +repo: railiance-platform +status: active +owner: codex +topic_slug: railiance +created: "2026-08-20" +updated: "2026-08-20" +related: + - CORE-WP-0011 + - RAPP-CORE-HUB-WP-0001 + - RAPP-POSTGRES-WP-0003 +origin: request +origin_ref: CORE-WP-0011 +--- + +# RPF-WP-0021 — Core Hub platform onboarding + +## Goal + +Provide the platform-owned dependencies that let the validated +`rapp-core-hub` package publish and deploy as a private shadow runtime on +railiance01 before the 2026-08-31 CoulombCore retirement deadline. + +This work stops before public traffic cutover, the final data freeze, or +retirement of the old runtime. Those remain explicit operator gates in +`CORE-WP-0011`. + +## Current evidence + +- `/home/worsch/rapp-core-hub` is clean at commit `4403c9b`; its recorded + package checks and railiance01 server-side dry-run passed. +- `coulomb/rapp-core-hub` does not exist in Forgejo (HTTP 404 on 2026-08-20). +- The routed `forgejo-admin-api-token` lane is active, but the present caller + receives HTTP 403 on the OpenBao mount lookup before `warden --exec` can + start the Forgejo request. +- `agent-high-risk-boundary` deliberately denies agent data-read to the + Forgejo site-admin PAT. That control is not removed to make publication + convenient; an attended operator identity must authorize the child-process + transport. +- The package requests a `rapp-postgres.v1` consumer and a Kubernetes Secret + containing `CORE_HUB_DATABASE_URL` plus `CORE_HUB_API_TOKEN`. +- A live metadata-only railiance01 check on 2026-08-20 found OpenBao Ready and + the Kubernetes-auth-backed `openbao` ClusterSecretStore Valid. The existing + audit-core KV and database stores are also Valid and their ExternalSecrets + are synced, so the two-store shape has a live reference on the target rail. + +## Boundaries + +- No secret value enters Git, State Hub, logs, command output, or agent + context. +- A repository is created only through the owned Forgejo admin lane. +- The static Core Hub API token and dynamic PostgreSQL lease remain distinct + custody sources even if the workload consumes both as environment data. +- A short-lived database lease is not copied into a standing OpenBao KV path. +- A private shadow deployment is allowed only after the database credential + rotation behavior is explicit and testable. +- Public Ingress, final transfer, DNS/tunnel changes, and old-runtime removal + remain outside this workplan. + +## T01 — Publish and register `rapp-core-hub` + +```task +id: RPF-WP-0021-T01 +status: done +priority: high +``` + +Under an attended `net-kingdom-admins` OIDC login to +`forgejo-admin-workload-kv-read`, verify capabilities on the Forgejo admin KV +data and metadata paths without reading the value. Then use +`warden access forgejo-admin-api-token --exec` to create the private +`coulomb/rapp-core-hub` repository, push commit `4403c9b`, configure the remote, +and run `statehub fix-consistency` from the published repo. + +Current blocker: both the original request and a 2026-08-20 retry reached +OpenBao but received HTTP 403 on +`sys/internal/ui/mounts/platform/workloads/forgejo/forgejo-admin`. The retry +also found the local warden audit path read-only. Do not weaken +`agent-high-risk-boundary`; repair or establish the operator session and use +the non-disclosing exec transport. + +Concurrent publication evidence, discovered at 07:55Z: Forgejo repository +`coulomb/rapp-core-hub` was created at 07:42:41Z, but it is public rather than +private and contains only Forgejo's `Initial commit` (`eea94d9`, README only). +The local validated commit `4403c9b` is not the remote `main`, although the +local remote is configured. Do not push the package into the public repo. +First change visibility to private through the owned admin lane, correct the +repository description, then push and verify `refs/heads/main` equals the +validated commit. + +Completed 2026-08-20 after operator authorization. Unauthenticated Forgejo API +access now returns 404 while the configured Git remote is accessible. Remote +`main` is `cd43c9f`: its history preserves validated package commit `4403c9b`, +the Forgejo README stub through merge `7dd057e`, the State Hub consistency +commit, and the publication evidence update. Local and remote `main` match. +`make check`, the railiance01 server-side dry-run, and family validation +(`4 declaration(s) ok`) passed after publication. The repo is registered in +State Hub; workplan UUID assignment remains with the production registrar. + +## T02 — Accept the Core Hub PostgreSQL consumer + +```task +id: RPF-WP-0021-T02 +status: progress +priority: high +``` + +Review `/home/worsch/rapp-core-hub/handoffs/postgres-consumer.yaml` in +`rapp-postgres`, add the accepted declaration, render it, and run package +tests. The requested database, schema, role split, connection limits, timeout, +service class, and tenant-keying posture match the existing consumer contract. + +Capacity is an admission condition. With `audit-core`, `tenant-engine`, and +the counted isolation probe, Core Hub is the fourth declaration on +`platform-pg`: it reaches but does not exceed the ADR-0004 ceiling. Before +admitting this third workload consumer, record the required measured memory +baseline. The untracked `consumers/issue-core.yaml` is not a real consumer — +`ISSUE-WP-0007` explicitly removed Postgres from that package — and must not +silently consume a slot. + +Live apply, database role creation, namespace labeling, and OpenBao database +role configuration remain attended operations after declaration review. + +Read-only preflight 2026-08-20: `platform-pg-1` used 9m CPU and 362Mi memory +against a 1Gi limit, with only `audit_core` and operator sessions present; +`tenant_engine` was not yet a database. This establishes current headroom but +does not satisfy ADR-0004's representative multi-consumer peak measurement. +Evidence: `docs/evidence/core-hub-postgres-capacity-preflight-2026-08-20.md`. + +## T03 — Correct the runtime credential delivery contract + +```task +id: RPF-WP-0021-T03 +status: progress +priority: high +``` + +Separate the two credential sources before shadow deployment: + +1. `CORE_HUB_API_TOKEN` is a standing workload secret in OpenBao KV, governed + by a proposed CCR and projected by a namespace-limited ESO store. +2. Runtime and migration PostgreSQL credentials come from distinct + `database/creds/core-hub-*` roles and the database-engine ESO store. One + extract must provide each username/password pair so leases are not mixed. + +The current package consumes one `core-hub-runtime` Secret through `envFrom` +for both the Deployment and migration Job. That is not yet a safe dynamic +lease contract: environment variables do not update when ESO refreshes a +Secret, and no workload reloader is declared. Update the package to consume +separate runtime/migration database Secrets and the static API-token Secret, +then either: + +- restart the Deployment automatically and demonstrably before each lease + expires; or +- change Core Hub to re-read mounted credential files when opening a new + database connection. + +Prefer the mounted-file approach for steady state. A time-bounded automatic +restart contract may be used for the relocation only if rollback and lease +overlap are proven and the temporary limitation is recorded. + +## T04 — Declare and apply the OpenBao/ESO lanes + +```task +id: RPF-WP-0021-T04 +status: progress +priority: high +``` + +After T03 fixes the consuming contract: + +- draft and approve a workload-KV CCR for the static API token; +- add a least-privilege ESO policy covering only the Core Hub static-token KV + path and `database/creds/core-hub-runtime` / `core-hub-migration`; +- configure the database roles from the accepted Postgres declaration; +- create namespace-limited KV and database `ClusterSecretStore` objects; +- project runtime and migration Secrets without printing values; and +- prove positive sync, cross-namespace denial, lease rotation, and migration + role separation. + +The working audit-core implementation is the reference for the two-store +shape. Do not copy its earlier, rejected idea of freezing a dynamic lease in +KV. + +Progress 2026-08-20: drafted `CCR-2026-0013` for the static API token, added +its exact-path read policy, added a not-yet-applied namespace-limited +`openbao-core-hub-runtime` ClusterSecretStore, and extended the agent +high-risk boundary to deny direct data reads on the new path. Database-engine +policy/store manifests wait for the accepted Postgres declaration and the T03 +consumer correction. + +## T05 — Support the private shadow restore and verification + +```task +id: RPF-WP-0021-T05 +status: wait +priority: high +``` + +After T01-T04, support `CORE-WP-0011-T03`: label the namespace for +`platform-pg`, restore or logically transfer the source database, reconcile +table and row counts, run the migration Job with the migration lease, deploy +with Ingress disabled, and verify health plus authenticated consumers against +the private endpoint. + +This task ends with repeatable shadow evidence. It does not flip +`hub.coulomb.social`, freeze production writes, or remove CoulombCore. + +## Acceptance + +- [ ] Private Forgejo repo exists, commit `4403c9b` is pushed, and State Hub + registration is consistent. +- [ ] Core Hub is an accepted, capacity-reviewed `rapp-postgres` consumer. +- [ ] Static API-token and dynamic database credentials have distinct, + least-privilege delivery paths. +- [ ] Database credential rotation cannot leave a running pod using an expired + environment value. +- [ ] Private shadow restore, reconciliation, migration, and consumer smokes + pass without enabling public Ingress.