Finish Core Hub private shadow onboarding
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s

This commit is contained in:
codex 2026-08-21 00:51:06 +02:00
parent a18d3dea33
commit 1206ed55da
5 changed files with 227 additions and 76 deletions

View file

@ -23,7 +23,7 @@
| workplan | RPF-WP-0018 | finished | — | workplans/RPF-WP-0018-policy-surface-alignment.md |
| workplan | RPF-WP-0019 | finished | — | workplans/RPF-WP-0019-apps-pg-recoverability-and-controls.md |
| workplan | RPF-WP-0020 | proposed | — | workplans/RPF-WP-0020-ccr-schema-drift.md |
| workplan | RPF-WP-0021 | active | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
| workplan | RPF-WP-0021 | finished | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
| task | RAILIANCE-WP-0005-T01 | done | — | workplans/RAILIANCE-WP-0005-credential-request-and-lease-broker.md |
| task | RAILIANCE-WP-0005-T02 | done | — | workplans/RAILIANCE-WP-0005-credential-request-and-lease-broker.md |
| task | RAILIANCE-WP-0005-T03 | done | — | workplans/RAILIANCE-WP-0005-credential-request-and-lease-broker.md |
@ -102,6 +102,6 @@
| task | RPF-WP-0020-T04 | todo | — | workplans/RPF-WP-0020-ccr-schema-drift.md |
| task | RPF-WP-0021-T01 | done | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
| task | RPF-WP-0021-T02 | done | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
| task | RPF-WP-0021-T03 | progress | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
| task | RPF-WP-0021-T04 | progress | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
| task | RPF-WP-0021-T05 | wait | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
| task | RPF-WP-0021-T03 | done | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
| task | RPF-WP-0021-T04 | done | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
| task | RPF-WP-0021-T05 | done | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |

View file

@ -3,33 +3,44 @@ kind: credential-change-request
schema_version: 1
request_type: workload-kv-read
title: Core Hub runtime API token lane
status: proposed
status: verified
created: '2026-08-20'
updated: '2026-08-20'
requester:
agent: codex
reason: >-
CORE-WP-0011 and RAPP-CORE-HUB-WP-0001 relocate Core Hub production to
railiance01. The validated package requires CORE_HUB_API_TOKEN, but the
value must move through platform custody rather than a copied Kubernetes
Secret or Git. This request covers only the standing application token;
PostgreSQL credentials remain dynamic database-engine leases.
reason: CORE-WP-0011 and RAPP-CORE-HUB-WP-0001 relocate Core Hub production to railiance01.
The validated package requires CORE_HUB_API_TOKEN, but the value must move through
platform custody rather than a copied Kubernetes Secret or Git. This request covers
only the standing application token; PostgreSQL credentials remain dynamic database-engine
leases.
review:
required: true
required_approvers:
- platform-operator
- core-hub-owner
comments: []
comments:
- at: '2026-08-20T22:02:19+00:00'
reviewer: platform operator / Core Hub owner (chat approval)
decision: approved
comment: 'State Hub decision 4f70311b-a4c4-4959-b1a1-cdd1be97d523: approved for
the governed private Core Hub relocation; static API-token custody remains separate
from dynamic PostgreSQL leases.'
- at: '2026-08-20T22:03:00+00:00'
reviewer: codex
decision: binding_confirmed
comment: Railiance01 has ServiceAccount external-secrets/external-secrets and
the External Secrets deployment uses that exact identity. The proposed Kubernetes-auth
bound claims match the live subject; store use remains independently limited
to namespace core-hub.
target:
domain: infotech
tenant: core-hub
workload: core-hub
rapp: rapp-core-hub
environment: production
purpose: >-
Project the existing Core Hub protected-API token into the private
railiance01 runtime through a namespace-limited ExternalSecret while
preserving the token across the relocation window.
purpose: Project the existing Core Hub protected-API token into the private railiance01
runtime through a namespace-limited ExternalSecret while preserving the token
across the relocation window.
openbao:
mount: platform
kv_path: platform/workloads/core-hub/runtime
@ -46,7 +57,7 @@ openbao:
- external-secrets
service_account_namespaces:
- external-secrets
bound_claims_confirmed: false
bound_claims_confirmed: true
policies:
- workload-kv-read-core-hub-runtime
ttl: 15m
@ -59,60 +70,102 @@ access_frontdoor:
readiness: pending-review
delivery:
surface: external-secrets
target: >-
ClusterSecretStore openbao-core-hub-runtime, limited to namespace
core-hub, to ExternalSecret core-hub/core-hub-api-token and Secret
core-hub-api-token. The rapp package consumes this Secret separately from
runtime and migration PostgreSQL lease Secrets.
target: ClusterSecretStore openbao-core-hub-runtime, limited to namespace core-hub,
to ExternalSecret core-hub/core-hub-api-token and Secret core-hub-api-token. The
rapp package consumes this Secret separately from runtime and migration PostgreSQL
lease Secrets.
risk:
classification: high
notes:
- The token authorizes protected Core Hub API operations and must not be
printed, placed on argv, committed, or recorded in State Hub.
- This CCR does not authorize a standing database password or a database URL
in KV. Those come from the OpenBao database engine.
- Preserve the current token during the relocation so existing protected API
consumers do not fail at the same time as the infrastructure move.
- Later rotation is coordinated with every protected API consumer because
Core Hub currently accepts one token.
- The token authorizes protected Core Hub API operations and must not be printed,
placed on argv, committed, or recorded in State Hub.
- This CCR does not authorize a standing database password or a database URL in
KV. Those come from the OpenBao database engine.
- Preserve the current token during the relocation so existing protected API consumers
do not fail at the same time as the infrastructure move.
- Later rotation is coordinated with every protected API consumer because Core Hub
currently accepts one token.
verification:
positive:
- An approved core-hub ExternalSecret syncs CORE_HUB_API_TOKEN into Secret
core-hub-api-token without printing the value.
- Protected API smoke succeeds against the private shadow endpoint using an
injected token after the source database has been reconciled.
- An approved core-hub ExternalSecret syncs CORE_HUB_API_TOKEN into Secret core-hub-api-token
without printing the value.
- Protected API smoke succeeds against the private shadow endpoint using an injected
token after the source database has been reconciled.
negative:
- A namespace outside core-hub cannot use openbao-core-hub-runtime.
- A token without workload-kv-read-core-hub-runtime cannot read the KV data
path.
- A token without workload-kv-read-core-hub-runtime cannot read the KV data path.
- The Core Hub workload identity cannot write or list sibling workload paths.
activation_conditions:
- Platform operator and Core Hub owner approve this CCR.
- The source token is transferred through attended operator custody without
entering logs, chat, Git, or State Hub.
- The Kubernetes auth role and policy are applied to railiance01-local
OpenBao and the bound service-account claims are verified.
- The package consumes the API-token Secret separately from dynamic database
lease Secrets.
- Positive and negative evidence is recorded with non-secret timestamps or
OpenBao audit references.
evidence: []
- The source token is transferred through attended operator custody without entering
logs, chat, Git, or State Hub.
- The Kubernetes auth role and policy are applied to railiance01-local OpenBao and
the bound service-account claims are verified.
- The package consumes the API-token Secret separately from dynamic database lease
Secrets.
- Positive and negative evidence is recorded with non-secret timestamps or OpenBao
audit references.
evidence:
- at: '2026-08-20T22:07:02+00:00'
actor: platform operator / Core Hub owner (chat approval)
kind: metadata_apply
result: passed
details:
- OpenBao policy and auth-role metadata apply completed without secret values
- at: '2026-08-20T22:11:45+00:00'
actor: codex
kind: secret_provisioned
result: passed
details:
- Source API token transferred directly from CoulombCore Kubernetes custody to
OpenBao KV version 1; value not printed or staged
- at: '2026-08-20T22:11:45+00:00'
actor: codex
kind: positive_verification
result: passed
details:
- Railiance01 stores Valid; API/runtime/migration ExternalSecrets SecretSynced
at 2026-08-20T22:07:42Z with expected key names only
- at: '2026-08-20T22:11:45+00:00'
actor: codex
kind: negative_verification
result: passed
details:
- Both stores rejected core-hub-denial-probe with SecretSyncedError and created
no target Secret; Kubernetes-auth token had read on exact data/metadata paths
and deny on sibling/parent paths
- at: '2026-08-20T22:11:46+00:00'
actor: codex
kind: database_role_separation
result: passed
details:
- Runtime lease connected but CREATE was denied and pg_has_role(core_hub_migrate)=false;
migration lease created and dropped bounded probe table; probe pods deleted
- at: '2026-08-20T22:45:58+00:00'
actor: codex
kind: private_shadow_verification
result: passed
details:
- 'Private railiance01 shadow: OpenBao KV and database stores Valid; least-privilege
denial and role-separation probes passed; chart 0.3.1 rotated the runtime lease
in place with Secret resourceVersion change, child PID change, stable pod UID,
container restartCount 0->0, readiness true, and 5m refresh restored; post-rotation
authenticated smoke 20260820224419-43dc05 passed all 10 checks; public Ingress
remained absent.'
lifecycle:
deactivate: >-
Disable the front door and Kubernetes auth role, remove the ExternalSecret,
and delete the materialized Secret only after Core Hub is retired or a
reviewed fallback is active.
rotate: >-
Coordinate a new random token with all protected API consumers, update the
OpenBao field through attended custody, wait for ESO sync, roll Core Hub and
consumers, verify protected calls, then invalidate the old token.
compromised: >-
Disable protected writes, replace the token in OpenBao and every consumer,
roll the workload, review Core Hub protected-operation logs, and open an
incident follow-up.
deactivate: Disable the front door and Kubernetes auth role, remove the ExternalSecret,
and delete the materialized Secret only after Core Hub is retired or a reviewed
fallback is active.
rotate: Coordinate a new random token with all protected API consumers, update the
OpenBao field through attended custody, wait for ESO sync, roll Core Hub and consumers,
verify protected calls, then invalidate the old token.
compromised: Disable protected writes, replace the token in OpenBao and every consumer,
roll the workload, review Core Hub protected-operation logs, and open an incident
follow-up.
state_hub:
workplan_id: RPF-WP-0021
task_id: RPF-WP-0021-T04
decision_id: 4f70311b-a4c4-4959-b1a1-cdd1be97d523
decision_api_url: http://127.0.0.1:8000/decisions/4f70311b-a4c4-4959-b1a1-cdd1be97d523
decision_dashboard_url: http://127.0.0.1:3000/decisions
decision_resolved_at: '2026-08-20T22:02:19Z'

View file

@ -0,0 +1,79 @@
# Core Hub private shadow evidence — 2026-08-21
## Scope and safety boundary
This record covers the private Core Hub relocation shadow on railiance01 for
`RPF-WP-0021`. Public Ingress was absent throughout. The CoulombCore writer,
public routing, DNS, and source runtime were not changed or retired. No secret
value is included in this record.
## Credential custody and isolation
- State Hub decision `4f70311b-a4c4-4959-b1a1-cdd1be97d523` was approved by
the platform operator / Core Hub owner at `2026-08-20T22:02:19Z`.
- An attended OpenBao OIDC session carried `platform-admin`; the policy and
Kubernetes-auth role for `CCR-2026-0013` were applied without displaying a
protected value.
- The existing API token moved directly from source Kubernetes custody into
`platform/workloads/core-hub/runtime`; it was neither copied through a
staging file nor printed.
- `openbao-core-hub-runtime` and `openbao-core-hub-database` reported
`Ready=True`, reason `Valid`. The API-token, runtime-database, and
migration-database ExternalSecrets each reported `Ready=True` and used
disjoint target Secrets.
- A disposable outside namespace could use neither ClusterSecretStore and
created no target Secret. A Kubernetes-auth probe could read only the exact
Core Hub KV data/metadata paths and was denied on sibling and parent paths.
- A runtime database lease connected and read but could not create objects or
inherit `core_hub_migrate`. A migration lease created and dropped a bounded
probe table. Probe tokens, pods, Secrets, and namespace were removed.
## Logical shadow restore and reconciliation
- The consistent source archive was captured at `2026-08-20 22:12:26 UTC`
while the source writer remained live.
- Archive size: `22175` bytes.
- SHA-256:
`3770946a1032c4109fea32fc00f4411c2f4d9311a28fb17bfc3d0d2de888db8b`.
- Source and restored target matched at Alembic revision `20260627_0002`,
eight tables, and 39 rows before shadow smoke activity. Per-table counts
were `1, 3, 10, 2, 2, 3, 2, 16`.
- The source archive used `public`; all eight restored tables were moved into
the accepted `core_hub` schema under the migration lease. Reconciliation
confirmed `core_hub_owner` ownership, runtime SELECT grants on every table,
and owner membership for the migration role.
## Package, deployment, and migration
- `rapp-postgres` published the Core Hub database-engine roles and ESO policy
in `cfde57a`; the live provisioner completed successfully.
- `rapp-core-hub` added migration-only database/DNS egress in `1cedab2` and a
fail-closed owned-port verifier in `8d729fd`.
- Live rotation exposed that watcher exit 75 caused scheduled pod
CrashLoopBackOff. The shadow runtime was temporarily stabilized at a
20-minute refresh interval while the source package was corrected.
- `rapp-core-hub` chart `0.3.1`, commit `7cfa8bb`, keeps its PID 1 watcher alive
and restarts only the child with a freshly constructed environment. All nine
package tests, Helm lint, deterministic rendering, and package checks passed.
- Helm release `core-hub` revision 2 reported `deployed`; the private API
Deployment was Ready, the migration Job completed, all three package
NetworkPolicies were present, and no Ingress existed.
## Credential rotation and private smoke
- A forced database lease refresh changed the runtime Secret resourceVersion.
The application child PID changed while the pod name and UID stayed fixed,
container restart count stayed `0 -> 0`, and readiness returned `true`.
- The intended ExternalSecret interval was restored to five minutes. The next
scheduled reconcile at `2026-08-20T22:45:01Z` left the pod Ready with the
same UID and zero container restarts; no CrashLoop behavior recurred.
- The hardened package verifier passed on an owned, unused private port.
- Authenticated deployed-smoke run `20260820224419-43dc05` completed at
`2026-08-20T22:44:20Z`. All ten checks passed: health, readiness, OpenAPI,
four public catalogs, unauthenticated 401 enforcement on protected routes,
and operator-authenticated protected reads/writes. Its temporary token file
was mode 0600 and shredded on exit.
`CCR-2026-0013` is verified. Public promotion, final write freeze/transfer,
DNS changes, and CoulombCore retirement remain separate operator gates under
`CORE-WP-0011`.

View file

@ -86,7 +86,8 @@ class CredentialChangeTests(unittest.TestCase):
self.assertEqual(ccr["target"]["rapp"], "rapp-core-hub")
self.assertEqual(ccr["openbao"]["fields"], ["CORE_HUB_API_TOKEN"])
self.assertNotIn("CORE_HUB_DATABASE_URL", ccr["openbao"]["fields"])
self.assertTrue(any("bound claim is not confirmed" in item for item in warnings))
self.assertEqual(warnings, [])
self.assertTrue(ccr["openbao"]["auth"]["bound_claims_confirmed"])
boundary = (
REPO_DIR / "openbao/policies/agent-high-risk-boundary.hcl"

View file

@ -4,11 +4,11 @@ type: workplan
title: "Unblock rapp-core-hub publication and platform onboarding"
domain: financials
repo: railiance-platform
status: active
status: finished
owner: codex
topic_slug: railiance
created: "2026-08-20"
updated: "2026-08-20"
updated: "2026-08-21"
related:
- CORE-WP-0011
- RAPPCOREHUB-WP-0001
@ -154,7 +154,7 @@ deletion, and names `core_hub_owner` as owner.
```task
id: RPF-WP-0021-T03
status: progress
status: done
priority: high
```
@ -182,22 +182,22 @@ Prefer the mounted-file approach for steady state. A time-bounded automatic
restart contract may be used for the relocation only if rollback and lease
overlap are proven and the temporary limitation is recorded.
Progress 2026-08-20: package chart `0.3.0` implements the bounded relocation
contract. Static API, runtime database, and migration database credentials are
mounted from three disjoint Secrets; the database URL is constructed only in
the supervised child environment. A PID 1 watcher fingerprints mounted files
without logging values and exits 75 on change so Kubernetes restarts the API
on the refreshed lease. The migration Job mounts only its role. Seven package
tests prove separation, URL encoding, no credential output, and rotation exit;
Helm lint, family validation, normal server dry-run, and migration-enabled
server dry-run pass. T03 remains `progress` until T04 proves a live ESO refresh,
container replacement, and readiness before the old runtime lease expires.
Completed 2026-08-21. Static API, runtime database, and migration database
credentials are mounted from three disjoint Secrets; the database URL exists
only in the supervised child environment, and the migration Job mounts only
its role. Live proof exposed that chart `0.3.0` watcher exit 75 could accumulate
planned Kubelet restarts into CrashLoopBackOff. Chart `0.3.1` (`rapp-core-hub`
commit `7cfa8bb`) instead keeps PID 1 alive and restarts only the application
child after reading the refreshed files. Nine package tests and live proof
showed a changed Secret resourceVersion and child PID with stable pod UID,
container restart count `0 -> 0`, restored readiness, and the five-minute
refresh interval restored.
## T04 — Declare and apply the OpenBao/ESO lanes
```task
id: RPF-WP-0021-T04
status: progress
status: done
priority: high
```
@ -241,11 +241,20 @@ configuration still waits for the attended OpenBao authority; concurrent
tenant-engine workplan edits in that repository were preserved and excluded
from this commit.
Completed 2026-08-21 after the operator approved State Hub decision
`4f70311b-a4c4-4959-b1a1-cdd1be97d523` and completed attended OpenBao OIDC.
The policy, auth role, database roles, two namespace-limited stores, and three
disjoint ExternalSecrets are live and Ready. The existing API token moved
directly from source Kubernetes custody into OpenBao without appearing in
output. Outside-namespace store denial, exact-path Kubernetes-auth denial,
runtime DDL denial, migration DDL success, ESO sync, and live lease rotation
all passed. `CCR-2026-0013` is verified.
## T05 — Support the private shadow restore and verification
```task
id: RPF-WP-0021-T05
status: wait
status: done
priority: high
```
@ -266,14 +275,23 @@ PostgreSQL port; it was deleted after the check. No dump or restore was
attempted ahead of the T04 credential gate. Evidence:
`docs/evidence/core-hub-shadow-preflight-2026-08-20.md`.
Completed 2026-08-21. A consistent logical archive restored all eight source
tables and 39 rows at Alembic revision `20260627_0002`; the restored tables
were reconciled into the accepted `core_hub` schema with owner and runtime
grants verified. Helm release revision 2 deployed chart `0.3.1` with Ingress
disabled, and the migration Job completed. The hardened private verifier and
authenticated smoke `20260820224419-43dc05` passed all health, readiness,
catalog, denial, and protected-operation checks. Detailed non-secret evidence:
`docs/evidence/core-hub-private-shadow-2026-08-21.md`.
## Acceptance
- [x] Private Forgejo repo exists, validated commit `4403c9b` is preserved,
and the registered workplan prefix is consistent.
- [x] Core Hub is an accepted, capacity-reviewed `rapp-postgres` consumer.
- [ ] Static API-token and dynamic database credentials have distinct,
- [x] Static API-token and dynamic database credentials have distinct,
least-privilege delivery paths.
- [ ] Database credential rotation cannot leave a running pod using an expired
- [x] Database credential rotation cannot leave a running pod using an expired
environment value.
- [ ] Private shadow restore, reconciliation, migration, and consumer smokes
- [x] Private shadow restore, reconciliation, migration, and consumer smokes
pass without enabling public Ingress.