railiance-platform/docs/evidence/core-hub-private-shadow-2026-08-21.md
codex 1206ed55da
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s
Finish Core Hub private shadow onboarding
2026-08-21 00:51:06 +02:00

4.2 KiB

Core Hub private shadow evidence — 2026-08-21

Scope and safety boundary

This record covers the private Core Hub relocation shadow on railiance01 for RPF-WP-0021. Public Ingress was absent throughout. The CoulombCore writer, public routing, DNS, and source runtime were not changed or retired. No secret value is included in this record.

Credential custody and isolation

  • State Hub decision 4f70311b-a4c4-4959-b1a1-cdd1be97d523 was approved by the platform operator / Core Hub owner at 2026-08-20T22:02:19Z.
  • An attended OpenBao OIDC session carried platform-admin; the policy and Kubernetes-auth role for CCR-2026-0013 were applied without displaying a protected value.
  • The existing API token moved directly from source Kubernetes custody into platform/workloads/core-hub/runtime; it was neither copied through a staging file nor printed.
  • openbao-core-hub-runtime and openbao-core-hub-database reported Ready=True, reason Valid. The API-token, runtime-database, and migration-database ExternalSecrets each reported Ready=True and used disjoint target Secrets.
  • A disposable outside namespace could use neither ClusterSecretStore and created no target Secret. A Kubernetes-auth probe could read only the exact Core Hub KV data/metadata paths and was denied on sibling and parent paths.
  • A runtime database lease connected and read but could not create objects or inherit core_hub_migrate. A migration lease created and dropped a bounded probe table. Probe tokens, pods, Secrets, and namespace were removed.

Logical shadow restore and reconciliation

  • The consistent source archive was captured at 2026-08-20 22:12:26 UTC while the source writer remained live.
  • Archive size: 22175 bytes.
  • SHA-256: 3770946a1032c4109fea32fc00f4411c2f4d9311a28fb17bfc3d0d2de888db8b.
  • Source and restored target matched at Alembic revision 20260627_0002, eight tables, and 39 rows before shadow smoke activity. Per-table counts were 1, 3, 10, 2, 2, 3, 2, 16.
  • The source archive used public; all eight restored tables were moved into the accepted core_hub schema under the migration lease. Reconciliation confirmed core_hub_owner ownership, runtime SELECT grants on every table, and owner membership for the migration role.

Package, deployment, and migration

  • rapp-postgres published the Core Hub database-engine roles and ESO policy in cfde57a; the live provisioner completed successfully.
  • rapp-core-hub added migration-only database/DNS egress in 1cedab2 and a fail-closed owned-port verifier in 8d729fd.
  • Live rotation exposed that watcher exit 75 caused scheduled pod CrashLoopBackOff. The shadow runtime was temporarily stabilized at a 20-minute refresh interval while the source package was corrected.
  • rapp-core-hub chart 0.3.1, commit 7cfa8bb, keeps its PID 1 watcher alive and restarts only the child with a freshly constructed environment. All nine package tests, Helm lint, deterministic rendering, and package checks passed.
  • Helm release core-hub revision 2 reported deployed; the private API Deployment was Ready, the migration Job completed, all three package NetworkPolicies were present, and no Ingress existed.

Credential rotation and private smoke

  • A forced database lease refresh changed the runtime Secret resourceVersion. The application child PID changed while the pod name and UID stayed fixed, container restart count stayed 0 -> 0, and readiness returned true.
  • The intended ExternalSecret interval was restored to five minutes. The next scheduled reconcile at 2026-08-20T22:45:01Z left the pod Ready with the same UID and zero container restarts; no CrashLoop behavior recurred.
  • The hardened package verifier passed on an owned, unused private port.
  • Authenticated deployed-smoke run 20260820224419-43dc05 completed at 2026-08-20T22:44:20Z. All ten checks passed: health, readiness, OpenAPI, four public catalogs, unauthenticated 401 enforcement on protected routes, and operator-authenticated protected reads/writes. Its temporary token file was mode 0600 and shredded on exit.

CCR-2026-0013 is verified. Public promotion, final write freeze/transfer, DNS changes, and CoulombCore retirement remain separate operator gates under CORE-WP-0011.