diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index b0b3dff..65df375 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -23,7 +23,7 @@ | workplan | RPF-WP-0018 | finished | — | workplans/RPF-WP-0018-policy-surface-alignment.md | | workplan | RPF-WP-0019 | finished | — | workplans/RPF-WP-0019-apps-pg-recoverability-and-controls.md | | workplan | RPF-WP-0020 | proposed | — | workplans/RPF-WP-0020-ccr-schema-drift.md | -| workplan | RPF-WP-0021 | active | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md | +| workplan | RPF-WP-0021 | finished | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md | | task | RAILIANCE-WP-0005-T01 | done | — | workplans/RAILIANCE-WP-0005-credential-request-and-lease-broker.md | | task | RAILIANCE-WP-0005-T02 | done | — | workplans/RAILIANCE-WP-0005-credential-request-and-lease-broker.md | | task | RAILIANCE-WP-0005-T03 | done | — | workplans/RAILIANCE-WP-0005-credential-request-and-lease-broker.md | @@ -102,6 +102,6 @@ | task | RPF-WP-0020-T04 | todo | — | workplans/RPF-WP-0020-ccr-schema-drift.md | | task | RPF-WP-0021-T01 | done | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md | | task | RPF-WP-0021-T02 | done | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md | -| task | RPF-WP-0021-T03 | progress | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md | -| task | RPF-WP-0021-T04 | progress | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md | -| task | RPF-WP-0021-T05 | wait | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md | +| task | RPF-WP-0021-T03 | done | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md | +| task | RPF-WP-0021-T04 | done | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md | +| task | RPF-WP-0021-T05 | done | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md | diff --git a/credential-change-requests/CCR-2026-0013-core-hub-runtime-api-token.yaml b/credential-change-requests/CCR-2026-0013-core-hub-runtime-api-token.yaml index 36bb20e..c9626b3 100644 --- a/credential-change-requests/CCR-2026-0013-core-hub-runtime-api-token.yaml +++ b/credential-change-requests/CCR-2026-0013-core-hub-runtime-api-token.yaml @@ -3,33 +3,44 @@ kind: credential-change-request schema_version: 1 request_type: workload-kv-read title: Core Hub runtime API token lane -status: proposed +status: verified created: '2026-08-20' updated: '2026-08-20' requester: agent: codex - reason: >- - CORE-WP-0011 and RAPP-CORE-HUB-WP-0001 relocate Core Hub production to - railiance01. The validated package requires CORE_HUB_API_TOKEN, but the - value must move through platform custody rather than a copied Kubernetes - Secret or Git. This request covers only the standing application token; - PostgreSQL credentials remain dynamic database-engine leases. + reason: CORE-WP-0011 and RAPP-CORE-HUB-WP-0001 relocate Core Hub production to railiance01. + The validated package requires CORE_HUB_API_TOKEN, but the value must move through + platform custody rather than a copied Kubernetes Secret or Git. This request covers + only the standing application token; PostgreSQL credentials remain dynamic database-engine + leases. review: required: true required_approvers: - platform-operator - core-hub-owner - comments: [] + comments: + - at: '2026-08-20T22:02:19+00:00' + reviewer: platform operator / Core Hub owner (chat approval) + decision: approved + comment: 'State Hub decision 4f70311b-a4c4-4959-b1a1-cdd1be97d523: approved for + the governed private Core Hub relocation; static API-token custody remains separate + from dynamic PostgreSQL leases.' + - at: '2026-08-20T22:03:00+00:00' + reviewer: codex + decision: binding_confirmed + comment: Railiance01 has ServiceAccount external-secrets/external-secrets and + the External Secrets deployment uses that exact identity. The proposed Kubernetes-auth + bound claims match the live subject; store use remains independently limited + to namespace core-hub. target: domain: infotech tenant: core-hub workload: core-hub rapp: rapp-core-hub environment: production - purpose: >- - Project the existing Core Hub protected-API token into the private - railiance01 runtime through a namespace-limited ExternalSecret while - preserving the token across the relocation window. + purpose: Project the existing Core Hub protected-API token into the private railiance01 + runtime through a namespace-limited ExternalSecret while preserving the token + across the relocation window. openbao: mount: platform kv_path: platform/workloads/core-hub/runtime @@ -46,7 +57,7 @@ openbao: - external-secrets service_account_namespaces: - external-secrets - bound_claims_confirmed: false + bound_claims_confirmed: true policies: - workload-kv-read-core-hub-runtime ttl: 15m @@ -59,60 +70,102 @@ access_frontdoor: readiness: pending-review delivery: surface: external-secrets - target: >- - ClusterSecretStore openbao-core-hub-runtime, limited to namespace - core-hub, to ExternalSecret core-hub/core-hub-api-token and Secret - core-hub-api-token. The rapp package consumes this Secret separately from - runtime and migration PostgreSQL lease Secrets. + target: ClusterSecretStore openbao-core-hub-runtime, limited to namespace core-hub, + to ExternalSecret core-hub/core-hub-api-token and Secret core-hub-api-token. The + rapp package consumes this Secret separately from runtime and migration PostgreSQL + lease Secrets. risk: classification: high notes: - - The token authorizes protected Core Hub API operations and must not be - printed, placed on argv, committed, or recorded in State Hub. - - This CCR does not authorize a standing database password or a database URL - in KV. Those come from the OpenBao database engine. - - Preserve the current token during the relocation so existing protected API - consumers do not fail at the same time as the infrastructure move. - - Later rotation is coordinated with every protected API consumer because - Core Hub currently accepts one token. + - The token authorizes protected Core Hub API operations and must not be printed, + placed on argv, committed, or recorded in State Hub. + - This CCR does not authorize a standing database password or a database URL in + KV. Those come from the OpenBao database engine. + - Preserve the current token during the relocation so existing protected API consumers + do not fail at the same time as the infrastructure move. + - Later rotation is coordinated with every protected API consumer because Core Hub + currently accepts one token. verification: positive: - - An approved core-hub ExternalSecret syncs CORE_HUB_API_TOKEN into Secret - core-hub-api-token without printing the value. - - Protected API smoke succeeds against the private shadow endpoint using an - injected token after the source database has been reconciled. + - An approved core-hub ExternalSecret syncs CORE_HUB_API_TOKEN into Secret core-hub-api-token + without printing the value. + - Protected API smoke succeeds against the private shadow endpoint using an injected + token after the source database has been reconciled. negative: - A namespace outside core-hub cannot use openbao-core-hub-runtime. - - A token without workload-kv-read-core-hub-runtime cannot read the KV data - path. + - A token without workload-kv-read-core-hub-runtime cannot read the KV data path. - The Core Hub workload identity cannot write or list sibling workload paths. activation_conditions: - Platform operator and Core Hub owner approve this CCR. - - The source token is transferred through attended operator custody without - entering logs, chat, Git, or State Hub. - - The Kubernetes auth role and policy are applied to railiance01-local - OpenBao and the bound service-account claims are verified. - - The package consumes the API-token Secret separately from dynamic database - lease Secrets. - - Positive and negative evidence is recorded with non-secret timestamps or - OpenBao audit references. - evidence: [] + - The source token is transferred through attended operator custody without entering + logs, chat, Git, or State Hub. + - The Kubernetes auth role and policy are applied to railiance01-local OpenBao and + the bound service-account claims are verified. + - The package consumes the API-token Secret separately from dynamic database lease + Secrets. + - Positive and negative evidence is recorded with non-secret timestamps or OpenBao + audit references. + evidence: + - at: '2026-08-20T22:07:02+00:00' + actor: platform operator / Core Hub owner (chat approval) + kind: metadata_apply + result: passed + details: + - OpenBao policy and auth-role metadata apply completed without secret values + - at: '2026-08-20T22:11:45+00:00' + actor: codex + kind: secret_provisioned + result: passed + details: + - Source API token transferred directly from CoulombCore Kubernetes custody to + OpenBao KV version 1; value not printed or staged + - at: '2026-08-20T22:11:45+00:00' + actor: codex + kind: positive_verification + result: passed + details: + - Railiance01 stores Valid; API/runtime/migration ExternalSecrets SecretSynced + at 2026-08-20T22:07:42Z with expected key names only + - at: '2026-08-20T22:11:45+00:00' + actor: codex + kind: negative_verification + result: passed + details: + - Both stores rejected core-hub-denial-probe with SecretSyncedError and created + no target Secret; Kubernetes-auth token had read on exact data/metadata paths + and deny on sibling/parent paths + - at: '2026-08-20T22:11:46+00:00' + actor: codex + kind: database_role_separation + result: passed + details: + - Runtime lease connected but CREATE was denied and pg_has_role(core_hub_migrate)=false; + migration lease created and dropped bounded probe table; probe pods deleted + - at: '2026-08-20T22:45:58+00:00' + actor: codex + kind: private_shadow_verification + result: passed + details: + - 'Private railiance01 shadow: OpenBao KV and database stores Valid; least-privilege + denial and role-separation probes passed; chart 0.3.1 rotated the runtime lease + in place with Secret resourceVersion change, child PID change, stable pod UID, + container restartCount 0->0, readiness true, and 5m refresh restored; post-rotation + authenticated smoke 20260820224419-43dc05 passed all 10 checks; public Ingress + remained absent.' lifecycle: - deactivate: >- - Disable the front door and Kubernetes auth role, remove the ExternalSecret, - and delete the materialized Secret only after Core Hub is retired or a - reviewed fallback is active. - rotate: >- - Coordinate a new random token with all protected API consumers, update the - OpenBao field through attended custody, wait for ESO sync, roll Core Hub and - consumers, verify protected calls, then invalidate the old token. - compromised: >- - Disable protected writes, replace the token in OpenBao and every consumer, - roll the workload, review Core Hub protected-operation logs, and open an - incident follow-up. + deactivate: Disable the front door and Kubernetes auth role, remove the ExternalSecret, + and delete the materialized Secret only after Core Hub is retired or a reviewed + fallback is active. + rotate: Coordinate a new random token with all protected API consumers, update the + OpenBao field through attended custody, wait for ESO sync, roll Core Hub and consumers, + verify protected calls, then invalidate the old token. + compromised: Disable protected writes, replace the token in OpenBao and every consumer, + roll the workload, review Core Hub protected-operation logs, and open an incident + follow-up. state_hub: workplan_id: RPF-WP-0021 task_id: RPF-WP-0021-T04 decision_id: 4f70311b-a4c4-4959-b1a1-cdd1be97d523 decision_api_url: http://127.0.0.1:8000/decisions/4f70311b-a4c4-4959-b1a1-cdd1be97d523 decision_dashboard_url: http://127.0.0.1:3000/decisions + decision_resolved_at: '2026-08-20T22:02:19Z' diff --git a/docs/evidence/core-hub-private-shadow-2026-08-21.md b/docs/evidence/core-hub-private-shadow-2026-08-21.md new file mode 100644 index 0000000..c0c8b36 --- /dev/null +++ b/docs/evidence/core-hub-private-shadow-2026-08-21.md @@ -0,0 +1,79 @@ +# Core Hub private shadow evidence — 2026-08-21 + +## Scope and safety boundary + +This record covers the private Core Hub relocation shadow on railiance01 for +`RPF-WP-0021`. Public Ingress was absent throughout. The CoulombCore writer, +public routing, DNS, and source runtime were not changed or retired. No secret +value is included in this record. + +## Credential custody and isolation + +- State Hub decision `4f70311b-a4c4-4959-b1a1-cdd1be97d523` was approved by + the platform operator / Core Hub owner at `2026-08-20T22:02:19Z`. +- An attended OpenBao OIDC session carried `platform-admin`; the policy and + Kubernetes-auth role for `CCR-2026-0013` were applied without displaying a + protected value. +- The existing API token moved directly from source Kubernetes custody into + `platform/workloads/core-hub/runtime`; it was neither copied through a + staging file nor printed. +- `openbao-core-hub-runtime` and `openbao-core-hub-database` reported + `Ready=True`, reason `Valid`. The API-token, runtime-database, and + migration-database ExternalSecrets each reported `Ready=True` and used + disjoint target Secrets. +- A disposable outside namespace could use neither ClusterSecretStore and + created no target Secret. A Kubernetes-auth probe could read only the exact + Core Hub KV data/metadata paths and was denied on sibling and parent paths. +- A runtime database lease connected and read but could not create objects or + inherit `core_hub_migrate`. A migration lease created and dropped a bounded + probe table. Probe tokens, pods, Secrets, and namespace were removed. + +## Logical shadow restore and reconciliation + +- The consistent source archive was captured at `2026-08-20 22:12:26 UTC` + while the source writer remained live. +- Archive size: `22175` bytes. +- SHA-256: + `3770946a1032c4109fea32fc00f4411c2f4d9311a28fb17bfc3d0d2de888db8b`. +- Source and restored target matched at Alembic revision `20260627_0002`, + eight tables, and 39 rows before shadow smoke activity. Per-table counts + were `1, 3, 10, 2, 2, 3, 2, 16`. +- The source archive used `public`; all eight restored tables were moved into + the accepted `core_hub` schema under the migration lease. Reconciliation + confirmed `core_hub_owner` ownership, runtime SELECT grants on every table, + and owner membership for the migration role. + +## Package, deployment, and migration + +- `rapp-postgres` published the Core Hub database-engine roles and ESO policy + in `cfde57a`; the live provisioner completed successfully. +- `rapp-core-hub` added migration-only database/DNS egress in `1cedab2` and a + fail-closed owned-port verifier in `8d729fd`. +- Live rotation exposed that watcher exit 75 caused scheduled pod + CrashLoopBackOff. The shadow runtime was temporarily stabilized at a + 20-minute refresh interval while the source package was corrected. +- `rapp-core-hub` chart `0.3.1`, commit `7cfa8bb`, keeps its PID 1 watcher alive + and restarts only the child with a freshly constructed environment. All nine + package tests, Helm lint, deterministic rendering, and package checks passed. +- Helm release `core-hub` revision 2 reported `deployed`; the private API + Deployment was Ready, the migration Job completed, all three package + NetworkPolicies were present, and no Ingress existed. + +## Credential rotation and private smoke + +- A forced database lease refresh changed the runtime Secret resourceVersion. + The application child PID changed while the pod name and UID stayed fixed, + container restart count stayed `0 -> 0`, and readiness returned `true`. +- The intended ExternalSecret interval was restored to five minutes. The next + scheduled reconcile at `2026-08-20T22:45:01Z` left the pod Ready with the + same UID and zero container restarts; no CrashLoop behavior recurred. +- The hardened package verifier passed on an owned, unused private port. +- Authenticated deployed-smoke run `20260820224419-43dc05` completed at + `2026-08-20T22:44:20Z`. All ten checks passed: health, readiness, OpenAPI, + four public catalogs, unauthenticated 401 enforcement on protected routes, + and operator-authenticated protected reads/writes. Its temporary token file + was mode 0600 and shredded on exit. + +`CCR-2026-0013` is verified. Public promotion, final write freeze/transfer, +DNS changes, and CoulombCore retirement remain separate operator gates under +`CORE-WP-0011`. diff --git a/tests/test_credential_change.py b/tests/test_credential_change.py index 7d80006..c068c05 100644 --- a/tests/test_credential_change.py +++ b/tests/test_credential_change.py @@ -86,7 +86,8 @@ class CredentialChangeTests(unittest.TestCase): self.assertEqual(ccr["target"]["rapp"], "rapp-core-hub") self.assertEqual(ccr["openbao"]["fields"], ["CORE_HUB_API_TOKEN"]) self.assertNotIn("CORE_HUB_DATABASE_URL", ccr["openbao"]["fields"]) - self.assertTrue(any("bound claim is not confirmed" in item for item in warnings)) + self.assertEqual(warnings, []) + self.assertTrue(ccr["openbao"]["auth"]["bound_claims_confirmed"]) boundary = ( REPO_DIR / "openbao/policies/agent-high-risk-boundary.hcl" diff --git a/workplans/RPF-WP-0021-core-hub-platform-onboarding.md b/workplans/RPF-WP-0021-core-hub-platform-onboarding.md index ce4c2ec..4dbcc25 100644 --- a/workplans/RPF-WP-0021-core-hub-platform-onboarding.md +++ b/workplans/RPF-WP-0021-core-hub-platform-onboarding.md @@ -4,11 +4,11 @@ type: workplan title: "Unblock rapp-core-hub publication and platform onboarding" domain: financials repo: railiance-platform -status: active +status: finished owner: codex topic_slug: railiance created: "2026-08-20" -updated: "2026-08-20" +updated: "2026-08-21" related: - CORE-WP-0011 - RAPPCOREHUB-WP-0001 @@ -154,7 +154,7 @@ deletion, and names `core_hub_owner` as owner. ```task id: RPF-WP-0021-T03 -status: progress +status: done priority: high ``` @@ -182,22 +182,22 @@ Prefer the mounted-file approach for steady state. A time-bounded automatic restart contract may be used for the relocation only if rollback and lease overlap are proven and the temporary limitation is recorded. -Progress 2026-08-20: package chart `0.3.0` implements the bounded relocation -contract. Static API, runtime database, and migration database credentials are -mounted from three disjoint Secrets; the database URL is constructed only in -the supervised child environment. A PID 1 watcher fingerprints mounted files -without logging values and exits 75 on change so Kubernetes restarts the API -on the refreshed lease. The migration Job mounts only its role. Seven package -tests prove separation, URL encoding, no credential output, and rotation exit; -Helm lint, family validation, normal server dry-run, and migration-enabled -server dry-run pass. T03 remains `progress` until T04 proves a live ESO refresh, -container replacement, and readiness before the old runtime lease expires. +Completed 2026-08-21. Static API, runtime database, and migration database +credentials are mounted from three disjoint Secrets; the database URL exists +only in the supervised child environment, and the migration Job mounts only +its role. Live proof exposed that chart `0.3.0` watcher exit 75 could accumulate +planned Kubelet restarts into CrashLoopBackOff. Chart `0.3.1` (`rapp-core-hub` +commit `7cfa8bb`) instead keeps PID 1 alive and restarts only the application +child after reading the refreshed files. Nine package tests and live proof +showed a changed Secret resourceVersion and child PID with stable pod UID, +container restart count `0 -> 0`, restored readiness, and the five-minute +refresh interval restored. ## T04 — Declare and apply the OpenBao/ESO lanes ```task id: RPF-WP-0021-T04 -status: progress +status: done priority: high ``` @@ -241,11 +241,20 @@ configuration still waits for the attended OpenBao authority; concurrent tenant-engine workplan edits in that repository were preserved and excluded from this commit. +Completed 2026-08-21 after the operator approved State Hub decision +`4f70311b-a4c4-4959-b1a1-cdd1be97d523` and completed attended OpenBao OIDC. +The policy, auth role, database roles, two namespace-limited stores, and three +disjoint ExternalSecrets are live and Ready. The existing API token moved +directly from source Kubernetes custody into OpenBao without appearing in +output. Outside-namespace store denial, exact-path Kubernetes-auth denial, +runtime DDL denial, migration DDL success, ESO sync, and live lease rotation +all passed. `CCR-2026-0013` is verified. + ## T05 — Support the private shadow restore and verification ```task id: RPF-WP-0021-T05 -status: wait +status: done priority: high ``` @@ -266,14 +275,23 @@ PostgreSQL port; it was deleted after the check. No dump or restore was attempted ahead of the T04 credential gate. Evidence: `docs/evidence/core-hub-shadow-preflight-2026-08-20.md`. +Completed 2026-08-21. A consistent logical archive restored all eight source +tables and 39 rows at Alembic revision `20260627_0002`; the restored tables +were reconciled into the accepted `core_hub` schema with owner and runtime +grants verified. Helm release revision 2 deployed chart `0.3.1` with Ingress +disabled, and the migration Job completed. The hardened private verifier and +authenticated smoke `20260820224419-43dc05` passed all health, readiness, +catalog, denial, and protected-operation checks. Detailed non-secret evidence: +`docs/evidence/core-hub-private-shadow-2026-08-21.md`. + ## Acceptance - [x] Private Forgejo repo exists, validated commit `4403c9b` is preserved, and the registered workplan prefix is consistent. - [x] Core Hub is an accepted, capacity-reviewed `rapp-postgres` consumer. -- [ ] Static API-token and dynamic database credentials have distinct, +- [x] Static API-token and dynamic database credentials have distinct, least-privilege delivery paths. -- [ ] Database credential rotation cannot leave a running pod using an expired +- [x] Database credential rotation cannot leave a running pod using an expired environment value. -- [ ] Private shadow restore, reconciliation, migration, and consumer smokes +- [x] Private shadow restore, reconciliation, migration, and consumer smokes pass without enabling public Ingress.