From a619e995def1a637e2ec4af757072472d4a36ba4 Mon Sep 17 00:00:00 2001 From: tegwick Date: Thu, 20 Aug 2026 23:06:20 +0200 Subject: [PATCH] docs: define railiance01 shadow rehearsal --- docs/deployment/railiance01-shadow-runbook.md | 121 ++++++++++++++++++ ...-0011-railiance01-production-relocation.md | 5 +- 2 files changed, 125 insertions(+), 1 deletion(-) create mode 100644 docs/deployment/railiance01-shadow-runbook.md diff --git a/docs/deployment/railiance01-shadow-runbook.md b/docs/deployment/railiance01-shadow-runbook.md new file mode 100644 index 0000000..12630d4 --- /dev/null +++ b/docs/deployment/railiance01-shadow-runbook.md @@ -0,0 +1,121 @@ +# Core Hub railiance01 shadow runbook + +This runbook starts only after `RPF-WP-0021-T02/T04` have accepted the +PostgreSQL consumer and materialized the three governed Secret objects. It +stops before public Ingress, DNS, tunnel changes, a final write freeze, or any +CoulombCore teardown. + +Authoritative deploy input: `/home/worsch/rapp-core-hub` at or after package +commit `03effc8`, chart `0.3.0`. + +## 1. Admission gates + +Confirm metadata only: + +- the accepted `rapp-postgres` declaration names distinct runtime and migration + roles for database `core_hub`; +- namespace `core-hub` carries the reviewed PostgreSQL-client labels; +- `core-hub-api-token` exposes key `CORE_HUB_API_TOKEN`; +- `core-hub-runtime-database` and `core-hub-migration-database` each expose + `username` and `password` keys from separate database-engine leases; +- the Core Hub KV and database ClusterSecretStores are Ready and limited to the + namespace; and +- the old runtime lease will remain valid for the proven ESO refresh, watcher + poll, graceful shutdown, restart and readiness interval. + +Use `kubectl get` and `kubectl describe`; never print `.data`, decode a Secret, +or place a token/DSN on an argument vector. Do not create placeholder Secrets +to get past admission. + +## 2. Rehearsal copy + +The platform/database owner creates a consistent logical backup of the +CoulombCore `core_hub` database and restores it into the accepted railiance01 +consumer. Keep the source writer live for the rehearsal; record its snapshot +timestamp and acknowledge that later writes will make this copy stale. + +Record non-secret source and target evidence: + +- Alembic revision; +- public-schema table set and per-table row counts; +- database/schema ownership and runtime-versus-migration grants; +- dump checksum, restore start/end timestamps and restore exit status; and +- any sequence values that must remain monotonic after cutover. + +Do not copy the source `core-hub-prod-env` Secret. The target database roles +come from OpenBao's database engine, and the standing API token moves only +through the approved static-token custody lane. + +## 3. Schema and private deployment + +From `rapp-core-hub`, with the reviewed railiance01 kubeconfig: + +```sh +make check +make server-dry-run +make migrate +make deploy +``` + +`make deploy` keeps Ingress disabled by default. Verify that no +`hub.coulomb.social` Ingress was created and that the public endpoint still +resolves to CoulombCore. + +The migration Job must mount only `core-hub-migration-database`. The API pod +must mount only `core-hub-runtime-database` and `core-hub-api-token`. + +## 4. Shadow verification + +Run package metadata, rollout and unauthenticated checks: + +```sh +make verify-live +``` + +Then port-forward the private Service and run Core Hub's authenticated smoke +with an approved token file: + +```sh +kubectl --kubeconfig "$KUBECONFIG" -n core-hub port-forward \ + service/core-hub-api 18010:8010 + +CORE_HUB_BASE_URL=http://127.0.0.1:18010 \ +CORE_HUB_OPERATOR_TOKEN_FILE=/approved/path/to/operator-token \ +CORE_HUB_SMOKE_OUTPUT=.local/smoke/railiance01-shadow.json \ +make deployed-smoke +``` + +The token file is mode 0600, outside Git, and never copied into evidence. +Retain only the non-secret JSON smoke report. Run the ops-hub gate and the +activity-core sink against the same private endpoint through their governed +credential paths. + +Repeat table sets and row counts after migrations and smokes. Explain every +difference; do not call the rehearsal reconciled merely because the API is +Ready. + +## 5. Lease-rotation proof + +With the platform operator, observe a real runtime database lease refresh: + +1. record the old pod UID/restart count and lease metadata without values; +2. let ESO project a renewed/reissued runtime lease normally; +3. confirm the watcher reports rotation without printing a credential; +4. confirm the old child terminates and the pod/container is replaced or + restarted before the old lease expires; +5. wait for readiness and rerun health plus authenticated smoke; and +6. prove the migration role is absent from the API pod. + +Only this live evidence closes `RAPPCOREHUB-WP-0001-T07` and the consumer side +of `RPF-WP-0021-T03`. + +## 6. Rehearsal rollback + +If any migration, reconciliation, smoke or rotation check fails, stop the +railiance01 Deployment and preserve logs plus database evidence. The public +Ingress remains absent, so production continues on CoulombCore. Reset the +target from a fresh approved backup before retrying; do not repair copied data +ad hoc and do not modify the source runtime. + +Production cutover uses a separate operator-approved freeze/final-transfer +procedure under `CORE-WP-0011-T04`. diff --git a/workplans/CORE-WP-0011-railiance01-production-relocation.md b/workplans/CORE-WP-0011-railiance01-production-relocation.md index 20da43a..283308a 100644 --- a/workplans/CORE-WP-0011-railiance01-production-relocation.md +++ b/workplans/CORE-WP-0011-railiance01-production-relocation.md @@ -157,7 +157,10 @@ namespace, Core Hub ClusterSecretStores, projected Secrets, and accepted pulls and imports Core Hub, Uvicorn, and Alembic under UID/GID 10001 with a read-only root filesystem. Package tests, family validation and server dry-run pass. This task is `wait` on `RPF-WP-0021-T02/T04`; do not substitute placeholder -Secrets or a standing database password. +Secrets or a standing database password. The executable gate order, evidence +requirements, private smokes, live lease-rotation proof and rehearsal rollback +are recorded in `docs/deployment/railiance01-shadow-runbook.md` so work can +resume immediately after platform admission. ## Cut over production traffic