# Core Hub private shadow evidence — 2026-08-21

## Scope and safety boundary

This record covers the private Core Hub relocation shadow on railiance01 for
`RPF-WP-0021`. Public Ingress was absent throughout. The CoulombCore writer,
public routing, DNS, and source runtime were not changed or retired. No secret
value is included in this record.

## Credential custody and isolation

- State Hub decision `4f70311b-a4c4-4959-b1a1-cdd1be97d523` was approved by
  the platform operator / Core Hub owner at `2026-08-20T22:02:19Z`.
- An attended OpenBao OIDC session carried `platform-admin`; the policy and
  Kubernetes-auth role for `CCR-2026-0013` were applied without displaying a
  protected value.
- The existing API token moved directly from source Kubernetes custody into
  `platform/workloads/core-hub/runtime`; it was neither copied through a
  staging file nor printed.
- `openbao-core-hub-runtime` and `openbao-core-hub-database` reported
  `Ready=True`, reason `Valid`. The API-token, runtime-database, and
  migration-database ExternalSecrets each reported `Ready=True` and used
  disjoint target Secrets.
- A disposable outside namespace could use neither ClusterSecretStore and
  created no target Secret. A Kubernetes-auth probe could read only the exact
  Core Hub KV data/metadata paths and was denied on sibling and parent paths.
- A runtime database lease connected and read but could not create objects or
  inherit `core_hub_migrate`. A migration lease created and dropped a bounded
  probe table. Probe tokens, pods, Secrets, and namespace were removed.

## Logical shadow restore and reconciliation

- The consistent source archive was captured at `2026-08-20 22:12:26 UTC`
  while the source writer remained live.
- Archive size: `22175` bytes.
- SHA-256:
  `3770946a1032c4109fea32fc00f4411c2f4d9311a28fb17bfc3d0d2de888db8b`.
- Source and restored target matched at Alembic revision `20260627_0002`,
  eight tables, and 39 rows before shadow smoke activity. Per-table counts
  were `1, 3, 10, 2, 2, 3, 2, 16`.
- The source archive used `public`; all eight restored tables were moved into
  the accepted `core_hub` schema under the migration lease. Reconciliation
  confirmed `core_hub_owner` ownership, runtime SELECT grants on every table,
  and owner membership for the migration role.

## Package, deployment, and migration

- `rapp-postgres` published the Core Hub database-engine roles and ESO policy
  in `cfde57a`; the live provisioner completed successfully.
- `rapp-core-hub` added migration-only database/DNS egress in `1cedab2` and a
  fail-closed owned-port verifier in `8d729fd`.
- Live rotation exposed that watcher exit 75 caused scheduled pod
  CrashLoopBackOff. The shadow runtime was temporarily stabilized at a
  20-minute refresh interval while the source package was corrected.
- `rapp-core-hub` chart `0.3.1`, commit `7cfa8bb`, keeps its PID 1 watcher alive
  and restarts only the child with a freshly constructed environment. All nine
  package tests, Helm lint, deterministic rendering, and package checks passed.
- Helm release `core-hub` revision 2 reported `deployed`; the private API
  Deployment was Ready, the migration Job completed, all three package
  NetworkPolicies were present, and no Ingress existed.

## Credential rotation and private smoke

- A forced database lease refresh changed the runtime Secret resourceVersion.
  The application child PID changed while the pod name and UID stayed fixed,
  container restart count stayed `0 -> 0`, and readiness returned `true`.
- The intended ExternalSecret interval was restored to five minutes. The next
  scheduled reconcile at `2026-08-20T22:45:01Z` left the pod Ready with the
  same UID and zero container restarts; no CrashLoop behavior recurred.
- The hardened package verifier passed on an owned, unused private port.
- Authenticated deployed-smoke run `20260820224419-43dc05` completed at
  `2026-08-20T22:44:20Z`. All ten checks passed: health, readiness, OpenAPI,
  four public catalogs, unauthenticated 401 enforcement on protected routes,
  and operator-authenticated protected reads/writes. Its temporary token file
  was mode 0600 and shredded on exit.

## Scheduled rotation stabilization observation

- At `2026-08-21T22:23:39Z`, after more than six hours of live stabilization,
  both five-minute hub-core database ExternalSecrets completed another
  scheduled reconciliation with `Ready=True` / `SecretSynced`.
- The runtime target Secret resourceVersion advanced from `50635316` to
  `50636878`; the migration target advanced from `50635319` to `50636875`.
  Only metadata was observed; neither Secret data nor a credential value was
  read or logged.
- After projected-volume propagation, the candidate application child PID
  changed from `224` to `278`. The pod UID remained
  `64bfd630-4f9c-4df8-9939-c737bd5c5a8a`, readiness remained true, and its
  container restart count remained zero. The
  `openbao-core-hub-database` ClusterSecretStore remained `Ready=True` /
  `Valid`.
- This closes the deliberate scheduled-rotation stabilization gate in
  `RAILIANCE-WP-0023-T02` without forcing a refresh or disrupting the pod.

`CCR-2026-0013` is verified. Public promotion, final write freeze/transfer,
DNS changes, and CoulombCore retirement remain separate operator gates under
`CORE-WP-0011`.