docs: define railiance01 shadow rehearsal

docs/deployment/railiance01-shadow-runbook.md

@@ -0,0 +1,121 @@
+# Core Hub railiance01 shadow runbook
+
+This runbook starts only after `RPF-WP-0021-T02/T04` have accepted the
+PostgreSQL consumer and materialized the three governed Secret objects. It
+stops before public Ingress, DNS, tunnel changes, a final write freeze, or any
+CoulombCore teardown.
+
+Authoritative deploy input: `/home/worsch/rapp-core-hub` at or after package
+commit `03effc8`, chart `0.3.0`.
+
+## 1. Admission gates
+
+Confirm metadata only:
+
+- the accepted `rapp-postgres` declaration names distinct runtime and migration
+  roles for database `core_hub`;
+- namespace `core-hub` carries the reviewed PostgreSQL-client labels;
+- `core-hub-api-token` exposes key `CORE_HUB_API_TOKEN`;
+- `core-hub-runtime-database` and `core-hub-migration-database` each expose
+  `username` and `password` keys from separate database-engine leases;
+- the Core Hub KV and database ClusterSecretStores are Ready and limited to the
+  namespace; and
+- the old runtime lease will remain valid for the proven ESO refresh, watcher
+  poll, graceful shutdown, restart and readiness interval.
+
+Use `kubectl get` and `kubectl describe`; never print `.data`, decode a Secret,
+or place a token/DSN on an argument vector. Do not create placeholder Secrets
+to get past admission.
+
+## 2. Rehearsal copy
+
+The platform/database owner creates a consistent logical backup of the
+CoulombCore `core_hub` database and restores it into the accepted railiance01
+consumer. Keep the source writer live for the rehearsal; record its snapshot
+timestamp and acknowledge that later writes will make this copy stale.
+
+Record non-secret source and target evidence:
+
+- Alembic revision;
+- public-schema table set and per-table row counts;
+- database/schema ownership and runtime-versus-migration grants;
+- dump checksum, restore start/end timestamps and restore exit status; and
+- any sequence values that must remain monotonic after cutover.
+
+Do not copy the source `core-hub-prod-env` Secret. The target database roles
+come from OpenBao's database engine, and the standing API token moves only
+through the approved static-token custody lane.
+
+## 3. Schema and private deployment
+
+From `rapp-core-hub`, with the reviewed railiance01 kubeconfig:
+
+```sh
+make check
+make server-dry-run
+make migrate
+make deploy
+```
+
+`make deploy` keeps Ingress disabled by default. Verify that no
+`hub.coulomb.social` Ingress was created and that the public endpoint still
+resolves to CoulombCore.
+
+The migration Job must mount only `core-hub-migration-database`. The API pod
+must mount only `core-hub-runtime-database` and `core-hub-api-token`.
+
+## 4. Shadow verification
+
+Run package metadata, rollout and unauthenticated checks:
+
+```sh
+make verify-live
+```
+
+Then port-forward the private Service and run Core Hub's authenticated smoke
+with an approved token file:
+
+```sh
+kubectl --kubeconfig "$KUBECONFIG" -n core-hub port-forward \
+  service/core-hub-api 18010:8010
+
+CORE_HUB_BASE_URL=http://127.0.0.1:18010 \
+CORE_HUB_OPERATOR_TOKEN_FILE=/approved/path/to/operator-token \
+CORE_HUB_SMOKE_OUTPUT=.local/smoke/railiance01-shadow.json \
+make deployed-smoke
+```
+
+The token file is mode 0600, outside Git, and never copied into evidence.
+Retain only the non-secret JSON smoke report. Run the ops-hub gate and the
+activity-core sink against the same private endpoint through their governed
+credential paths.
+
+Repeat table sets and row counts after migrations and smokes. Explain every
+difference; do not call the rehearsal reconciled merely because the API is
+Ready.
+
+## 5. Lease-rotation proof
+
+With the platform operator, observe a real runtime database lease refresh:
+
+1. record the old pod UID/restart count and lease metadata without values;
+2. let ESO project a renewed/reissued runtime lease normally;
+3. confirm the watcher reports rotation without printing a credential;
+4. confirm the old child terminates and the pod/container is replaced or
+   restarted before the old lease expires;
+5. wait for readiness and rerun health plus authenticated smoke; and
+6. prove the migration role is absent from the API pod.
+
+Only this live evidence closes `RAPPCOREHUB-WP-0001-T07` and the consumer side
+of `RPF-WP-0021-T03`.
+
+## 6. Rehearsal rollback
+
+If any migration, reconciliation, smoke or rotation check fails, stop the
+railiance01 Deployment and preserve logs plus database evidence. The public
+Ingress remains absent, so production continues on CoulombCore. Reset the
+target from a fresh approved backup before retrying; do not repair copied data
+ad hoc and do not modify the source runtime.
+
+Production cutover uses a separate operator-approved freeze/final-transfer
+procedure under `CORE-WP-0011-T04`.