Finish Core Hub private shadow onboarding

WORK-RECORDS.md

@@ -23,7 +23,7 @@
 | workplan | RPF-WP-0018 | finished | — | workplans/RPF-WP-0018-policy-surface-alignment.md |
 | workplan | RPF-WP-0019 | finished | — | workplans/RPF-WP-0019-apps-pg-recoverability-and-controls.md |
 | workplan | RPF-WP-0020 | proposed | — | workplans/RPF-WP-0020-ccr-schema-drift.md |
-| workplan | RPF-WP-0021 | active | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
+| workplan | RPF-WP-0021 | finished | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
 | task | RAILIANCE-WP-0005-T01 | done | — | workplans/RAILIANCE-WP-0005-credential-request-and-lease-broker.md |
 | task | RAILIANCE-WP-0005-T02 | done | — | workplans/RAILIANCE-WP-0005-credential-request-and-lease-broker.md |
 | task | RAILIANCE-WP-0005-T03 | done | — | workplans/RAILIANCE-WP-0005-credential-request-and-lease-broker.md |
@@ -102,6 +102,6 @@
 | task | RPF-WP-0020-T04 | todo | — | workplans/RPF-WP-0020-ccr-schema-drift.md |
 | task | RPF-WP-0021-T01 | done | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
 | task | RPF-WP-0021-T02 | done | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
-| task | RPF-WP-0021-T03 | progress | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
+| task | RPF-WP-0021-T03 | done | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
-| task | RPF-WP-0021-T04 | progress | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
+| task | RPF-WP-0021-T04 | done | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
-| task | RPF-WP-0021-T05 | wait | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
+| task | RPF-WP-0021-T05 | done | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |

credential-change-requests/CCR-2026-0013-core-hub-runtime-api-token.yaml

@@ -3,33 +3,44 @@ kind: credential-change-request
 schema_version: 1
 request_type: workload-kv-read
 title: Core Hub runtime API token lane
-status: proposed
+status: verified
 created: '2026-08-20'
 updated: '2026-08-20'
 requester:
   agent: codex
-  reason: >-
+  reason: CORE-WP-0011 and RAPP-CORE-HUB-WP-0001 relocate Core Hub production to railiance01.
-    CORE-WP-0011 and RAPP-CORE-HUB-WP-0001 relocate Core Hub production to
+    The validated package requires CORE_HUB_API_TOKEN, but the value must move through
-    railiance01. The validated package requires CORE_HUB_API_TOKEN, but the
+    platform custody rather than a copied Kubernetes Secret or Git. This request covers
-    value must move through platform custody rather than a copied Kubernetes
+    only the standing application token; PostgreSQL credentials remain dynamic database-engine
-    Secret or Git. This request covers only the standing application token;
+    leases.
-    PostgreSQL credentials remain dynamic database-engine leases.
 review:
   required: true
   required_approvers:
   - platform-operator
   - core-hub-owner
-  comments: []
+  comments:
+  - at: '2026-08-20T22:02:19+00:00'
+    reviewer: platform operator / Core Hub owner (chat approval)
+    decision: approved
+    comment: 'State Hub decision 4f70311b-a4c4-4959-b1a1-cdd1be97d523: approved for
+      the governed private Core Hub relocation; static API-token custody remains separate
+      from dynamic PostgreSQL leases.'
+  - at: '2026-08-20T22:03:00+00:00'
+    reviewer: codex
+    decision: binding_confirmed
+    comment: Railiance01 has ServiceAccount external-secrets/external-secrets and
+      the External Secrets deployment uses that exact identity. The proposed Kubernetes-auth
+      bound claims match the live subject; store use remains independently limited
+      to namespace core-hub.
 target:
   domain: infotech
   tenant: core-hub
   workload: core-hub
   rapp: rapp-core-hub
   environment: production
-  purpose: >-
+  purpose: Project the existing Core Hub protected-API token into the private railiance01
-    Project the existing Core Hub protected-API token into the private
+    runtime through a namespace-limited ExternalSecret while preserving the token
-    railiance01 runtime through a namespace-limited ExternalSecret while
+    across the relocation window.
-    preserving the token across the relocation window.
 openbao:
   mount: platform
   kv_path: platform/workloads/core-hub/runtime
@@ -46,7 +57,7 @@ openbao:
       - external-secrets
       service_account_namespaces:
       - external-secrets
-    bound_claims_confirmed: false
+    bound_claims_confirmed: true
     policies:
     - workload-kv-read-core-hub-runtime
     ttl: 15m
@@ -59,60 +70,102 @@ access_frontdoor:
   readiness: pending-review
 delivery:
   surface: external-secrets
-  target: >-
+  target: ClusterSecretStore openbao-core-hub-runtime, limited to namespace core-hub,
-    ClusterSecretStore openbao-core-hub-runtime, limited to namespace
+    to ExternalSecret core-hub/core-hub-api-token and Secret core-hub-api-token. The
-    core-hub, to ExternalSecret core-hub/core-hub-api-token and Secret
+    rapp package consumes this Secret separately from runtime and migration PostgreSQL
-    core-hub-api-token. The rapp package consumes this Secret separately from
+    lease Secrets.
-    runtime and migration PostgreSQL lease Secrets.
 risk:
   classification: high
   notes:
-  - The token authorizes protected Core Hub API operations and must not be
+  - The token authorizes protected Core Hub API operations and must not be printed,
-    printed, placed on argv, committed, or recorded in State Hub.
+    placed on argv, committed, or recorded in State Hub.
-  - This CCR does not authorize a standing database password or a database URL
+  - This CCR does not authorize a standing database password or a database URL in
-    in KV. Those come from the OpenBao database engine.
+    KV. Those come from the OpenBao database engine.
-  - Preserve the current token during the relocation so existing protected API
+  - Preserve the current token during the relocation so existing protected API consumers
-    consumers do not fail at the same time as the infrastructure move.
+    do not fail at the same time as the infrastructure move.
-  - Later rotation is coordinated with every protected API consumer because
+  - Later rotation is coordinated with every protected API consumer because Core Hub
-    Core Hub currently accepts one token.
+    currently accepts one token.
 verification:
   positive:
-  - An approved core-hub ExternalSecret syncs CORE_HUB_API_TOKEN into Secret
+  - An approved core-hub ExternalSecret syncs CORE_HUB_API_TOKEN into Secret core-hub-api-token
-    core-hub-api-token without printing the value.
+    without printing the value.
-  - Protected API smoke succeeds against the private shadow endpoint using an
+  - Protected API smoke succeeds against the private shadow endpoint using an injected
-    injected token after the source database has been reconciled.
+    token after the source database has been reconciled.
   negative:
   - A namespace outside core-hub cannot use openbao-core-hub-runtime.
-  - A token without workload-kv-read-core-hub-runtime cannot read the KV data
+  - A token without workload-kv-read-core-hub-runtime cannot read the KV data path.
-    path.
   - The Core Hub workload identity cannot write or list sibling workload paths.
   activation_conditions:
   - Platform operator and Core Hub owner approve this CCR.
-  - The source token is transferred through attended operator custody without
+  - The source token is transferred through attended operator custody without entering
-    entering logs, chat, Git, or State Hub.
+    logs, chat, Git, or State Hub.
-  - The Kubernetes auth role and policy are applied to railiance01-local
+  - The Kubernetes auth role and policy are applied to railiance01-local OpenBao and
-    OpenBao and the bound service-account claims are verified.
+    the bound service-account claims are verified.
-  - The package consumes the API-token Secret separately from dynamic database
+  - The package consumes the API-token Secret separately from dynamic database lease
-    lease Secrets.
+    Secrets.
-  - Positive and negative evidence is recorded with non-secret timestamps or
+  - Positive and negative evidence is recorded with non-secret timestamps or OpenBao
-    OpenBao audit references.
+    audit references.
-  evidence: []
+  evidence:
+  - at: '2026-08-20T22:07:02+00:00'
+    actor: platform operator / Core Hub owner (chat approval)
+    kind: metadata_apply
+    result: passed
+    details:
+    - OpenBao policy and auth-role metadata apply completed without secret values
+  - at: '2026-08-20T22:11:45+00:00'
+    actor: codex
+    kind: secret_provisioned
+    result: passed
+    details:
+    - Source API token transferred directly from CoulombCore Kubernetes custody to
+      OpenBao KV version 1; value not printed or staged
+  - at: '2026-08-20T22:11:45+00:00'
+    actor: codex
+    kind: positive_verification
+    result: passed
+    details:
+    - Railiance01 stores Valid; API/runtime/migration ExternalSecrets SecretSynced
+      at 2026-08-20T22:07:42Z with expected key names only
+  - at: '2026-08-20T22:11:45+00:00'
+    actor: codex
+    kind: negative_verification
+    result: passed
+    details:
+    - Both stores rejected core-hub-denial-probe with SecretSyncedError and created
+      no target Secret; Kubernetes-auth token had read on exact data/metadata paths
+      and deny on sibling/parent paths
+  - at: '2026-08-20T22:11:46+00:00'
+    actor: codex
+    kind: database_role_separation
+    result: passed
+    details:
+    - Runtime lease connected but CREATE was denied and pg_has_role(core_hub_migrate)=false;
+      migration lease created and dropped bounded probe table; probe pods deleted
+  - at: '2026-08-20T22:45:58+00:00'
+    actor: codex
+    kind: private_shadow_verification
+    result: passed
+    details:
+    - 'Private railiance01 shadow: OpenBao KV and database stores Valid; least-privilege
+      denial and role-separation probes passed; chart 0.3.1 rotated the runtime lease
+      in place with Secret resourceVersion change, child PID change, stable pod UID,
+      container restartCount 0->0, readiness true, and 5m refresh restored; post-rotation
+      authenticated smoke 20260820224419-43dc05 passed all 10 checks; public Ingress
+      remained absent.'
 lifecycle:
-  deactivate: >-
+  deactivate: Disable the front door and Kubernetes auth role, remove the ExternalSecret,
-    Disable the front door and Kubernetes auth role, remove the ExternalSecret,
+    and delete the materialized Secret only after Core Hub is retired or a reviewed
-    and delete the materialized Secret only after Core Hub is retired or a
+    fallback is active.
-    reviewed fallback is active.
+  rotate: Coordinate a new random token with all protected API consumers, update the
-  rotate: >-
+    OpenBao field through attended custody, wait for ESO sync, roll Core Hub and consumers,
-    Coordinate a new random token with all protected API consumers, update the
+    verify protected calls, then invalidate the old token.
-    OpenBao field through attended custody, wait for ESO sync, roll Core Hub and
+  compromised: Disable protected writes, replace the token in OpenBao and every consumer,
-    consumers, verify protected calls, then invalidate the old token.
+    roll the workload, review Core Hub protected-operation logs, and open an incident
-  compromised: >-
+    follow-up.
-    Disable protected writes, replace the token in OpenBao and every consumer,
-    roll the workload, review Core Hub protected-operation logs, and open an
-    incident follow-up.
 state_hub:
   workplan_id: RPF-WP-0021
   task_id: RPF-WP-0021-T04
   decision_id: 4f70311b-a4c4-4959-b1a1-cdd1be97d523
   decision_api_url: http://127.0.0.1:8000/decisions/4f70311b-a4c4-4959-b1a1-cdd1be97d523
   decision_dashboard_url: http://127.0.0.1:3000/decisions
+  decision_resolved_at: '2026-08-20T22:02:19Z'

docs/evidence/core-hub-private-shadow-2026-08-21.md

@@ -0,0 +1,79 @@
+# Core Hub private shadow evidence — 2026-08-21
+
+## Scope and safety boundary
+
+This record covers the private Core Hub relocation shadow on railiance01 for
+`RPF-WP-0021`. Public Ingress was absent throughout. The CoulombCore writer,
+public routing, DNS, and source runtime were not changed or retired. No secret
+value is included in this record.
+
+## Credential custody and isolation
+
+- State Hub decision `4f70311b-a4c4-4959-b1a1-cdd1be97d523` was approved by
+  the platform operator / Core Hub owner at `2026-08-20T22:02:19Z`.
+- An attended OpenBao OIDC session carried `platform-admin`; the policy and
+  Kubernetes-auth role for `CCR-2026-0013` were applied without displaying a
+  protected value.
+- The existing API token moved directly from source Kubernetes custody into
+  `platform/workloads/core-hub/runtime`; it was neither copied through a
+  staging file nor printed.
+- `openbao-core-hub-runtime` and `openbao-core-hub-database` reported
+  `Ready=True`, reason `Valid`. The API-token, runtime-database, and
+  migration-database ExternalSecrets each reported `Ready=True` and used
+  disjoint target Secrets.
+- A disposable outside namespace could use neither ClusterSecretStore and
+  created no target Secret. A Kubernetes-auth probe could read only the exact
+  Core Hub KV data/metadata paths and was denied on sibling and parent paths.
+- A runtime database lease connected and read but could not create objects or
+  inherit `core_hub_migrate`. A migration lease created and dropped a bounded
+  probe table. Probe tokens, pods, Secrets, and namespace were removed.
+
+## Logical shadow restore and reconciliation
+
+- The consistent source archive was captured at `2026-08-20 22:12:26 UTC`
+  while the source writer remained live.
+- Archive size: `22175` bytes.
+- SHA-256:
+  `3770946a1032c4109fea32fc00f4411c2f4d9311a28fb17bfc3d0d2de888db8b`.
+- Source and restored target matched at Alembic revision `20260627_0002`,
+  eight tables, and 39 rows before shadow smoke activity. Per-table counts
+  were `1, 3, 10, 2, 2, 3, 2, 16`.
+- The source archive used `public`; all eight restored tables were moved into
+  the accepted `core_hub` schema under the migration lease. Reconciliation
+  confirmed `core_hub_owner` ownership, runtime SELECT grants on every table,
+  and owner membership for the migration role.
+
+## Package, deployment, and migration
+
+- `rapp-postgres` published the Core Hub database-engine roles and ESO policy
+  in `cfde57a`; the live provisioner completed successfully.
+- `rapp-core-hub` added migration-only database/DNS egress in `1cedab2` and a
+  fail-closed owned-port verifier in `8d729fd`.
+- Live rotation exposed that watcher exit 75 caused scheduled pod
+  CrashLoopBackOff. The shadow runtime was temporarily stabilized at a
+  20-minute refresh interval while the source package was corrected.
+- `rapp-core-hub` chart `0.3.1`, commit `7cfa8bb`, keeps its PID 1 watcher alive
+  and restarts only the child with a freshly constructed environment. All nine
+  package tests, Helm lint, deterministic rendering, and package checks passed.
+- Helm release `core-hub` revision 2 reported `deployed`; the private API
+  Deployment was Ready, the migration Job completed, all three package
+  NetworkPolicies were present, and no Ingress existed.
+
+## Credential rotation and private smoke
+
+- A forced database lease refresh changed the runtime Secret resourceVersion.
+  The application child PID changed while the pod name and UID stayed fixed,
+  container restart count stayed `0 -> 0`, and readiness returned `true`.
+- The intended ExternalSecret interval was restored to five minutes. The next
+  scheduled reconcile at `2026-08-20T22:45:01Z` left the pod Ready with the
+  same UID and zero container restarts; no CrashLoop behavior recurred.
+- The hardened package verifier passed on an owned, unused private port.
+- Authenticated deployed-smoke run `20260820224419-43dc05` completed at
+  `2026-08-20T22:44:20Z`. All ten checks passed: health, readiness, OpenAPI,
+  four public catalogs, unauthenticated 401 enforcement on protected routes,
+  and operator-authenticated protected reads/writes. Its temporary token file
+  was mode 0600 and shredded on exit.
+
+`CCR-2026-0013` is verified. Public promotion, final write freeze/transfer,
+DNS changes, and CoulombCore retirement remain separate operator gates under
+`CORE-WP-0011`.

tests/test_credential_change.py

@@ -86,7 +86,8 @@ class CredentialChangeTests(unittest.TestCase):
         self.assertEqual(ccr["target"]["rapp"], "rapp-core-hub")
         self.assertEqual(ccr["openbao"]["fields"], ["CORE_HUB_API_TOKEN"])
         self.assertNotIn("CORE_HUB_DATABASE_URL", ccr["openbao"]["fields"])
-        self.assertTrue(any("bound claim is not confirmed" in item for item in warnings))
+        self.assertEqual(warnings, [])
+        self.assertTrue(ccr["openbao"]["auth"]["bound_claims_confirmed"])
 
         boundary = (
             REPO_DIR / "openbao/policies/agent-high-risk-boundary.hcl"

workplans/RPF-WP-0021-core-hub-platform-onboarding.md

@@ -4,11 +4,11 @@ type: workplan
 title: "Unblock rapp-core-hub publication and platform onboarding"
 domain: financials
 repo: railiance-platform
-status: active
+status: finished
 owner: codex
 topic_slug: railiance
 created: "2026-08-20"
-updated: "2026-08-20"
+updated: "2026-08-21"
 related:
   - CORE-WP-0011
   - RAPPCOREHUB-WP-0001
@@ -154,7 +154,7 @@ deletion, and names `core_hub_owner` as owner.
 
 ```task
 id: RPF-WP-0021-T03
-status: progress
+status: done
 priority: high
 ```
 
@@ -182,22 +182,22 @@ Prefer the mounted-file approach for steady state. A time-bounded automatic
 restart contract may be used for the relocation only if rollback and lease
 overlap are proven and the temporary limitation is recorded.
 
-Progress 2026-08-20: package chart `0.3.0` implements the bounded relocation
+Completed 2026-08-21. Static API, runtime database, and migration database
-contract. Static API, runtime database, and migration database credentials are
+credentials are mounted from three disjoint Secrets; the database URL exists
-mounted from three disjoint Secrets; the database URL is constructed only in
+only in the supervised child environment, and the migration Job mounts only
-the supervised child environment. A PID 1 watcher fingerprints mounted files
+its role. Live proof exposed that chart `0.3.0` watcher exit 75 could accumulate
-without logging values and exits 75 on change so Kubernetes restarts the API
+planned Kubelet restarts into CrashLoopBackOff. Chart `0.3.1` (`rapp-core-hub`
-on the refreshed lease. The migration Job mounts only its role. Seven package
+commit `7cfa8bb`) instead keeps PID 1 alive and restarts only the application
-tests prove separation, URL encoding, no credential output, and rotation exit;
+child after reading the refreshed files. Nine package tests and live proof
-Helm lint, family validation, normal server dry-run, and migration-enabled
+showed a changed Secret resourceVersion and child PID with stable pod UID,
-server dry-run pass. T03 remains `progress` until T04 proves a live ESO refresh,
+container restart count `0 -> 0`, restored readiness, and the five-minute
-container replacement, and readiness before the old runtime lease expires.
+refresh interval restored.
 
 ## T04 — Declare and apply the OpenBao/ESO lanes
 
 ```task
 id: RPF-WP-0021-T04
-status: progress
+status: done
 priority: high
 ```
 
@@ -241,11 +241,20 @@ configuration still waits for the attended OpenBao authority; concurrent
 tenant-engine workplan edits in that repository were preserved and excluded
 from this commit.
 
+Completed 2026-08-21 after the operator approved State Hub decision
+`4f70311b-a4c4-4959-b1a1-cdd1be97d523` and completed attended OpenBao OIDC.
+The policy, auth role, database roles, two namespace-limited stores, and three
+disjoint ExternalSecrets are live and Ready. The existing API token moved
+directly from source Kubernetes custody into OpenBao without appearing in
+output. Outside-namespace store denial, exact-path Kubernetes-auth denial,
+runtime DDL denial, migration DDL success, ESO sync, and live lease rotation
+all passed. `CCR-2026-0013` is verified.
+
 ## T05 — Support the private shadow restore and verification
 
 ```task
 id: RPF-WP-0021-T05
-status: wait
+status: done
 priority: high
 ```
 
@@ -266,14 +275,23 @@ PostgreSQL port; it was deleted after the check. No dump or restore was
 attempted ahead of the T04 credential gate. Evidence:
 `docs/evidence/core-hub-shadow-preflight-2026-08-20.md`.
 
+Completed 2026-08-21. A consistent logical archive restored all eight source
+tables and 39 rows at Alembic revision `20260627_0002`; the restored tables
+were reconciled into the accepted `core_hub` schema with owner and runtime
+grants verified. Helm release revision 2 deployed chart `0.3.1` with Ingress
+disabled, and the migration Job completed. The hardened private verifier and
+authenticated smoke `20260820224419-43dc05` passed all health, readiness,
+catalog, denial, and protected-operation checks. Detailed non-secret evidence:
+`docs/evidence/core-hub-private-shadow-2026-08-21.md`.
+
 ## Acceptance
 
 - [x] Private Forgejo repo exists, validated commit `4403c9b` is preserved,
       and the registered workplan prefix is consistent.
 - [x] Core Hub is an accepted, capacity-reviewed `rapp-postgres` consumer.
-- [ ] Static API-token and dynamic database credentials have distinct,
+- [x] Static API-token and dynamic database credentials have distinct,
       least-privilege delivery paths.
-- [ ] Database credential rotation cannot leave a running pod using an expired
+- [x] Database credential rotation cannot leave a running pod using an expired
       environment value.
-- [ ] Private shadow restore, reconciliation, migration, and consumer smokes
+- [x] Private shadow restore, reconciliation, migration, and consumer smokes
       pass without enabling public Ingress.