Core Hub railiance01 shadow runbook

This runbook starts only after RPF-WP-0021-T02/T04 have accepted the PostgreSQL consumer and materialized the three governed Secret objects. It stops before public Ingress, DNS, tunnel changes, a final write freeze, or any CoulombCore teardown.

Authoritative deploy input: /home/worsch/rapp-core-hub at or after package commit 7cfa8bb, chart 0.3.1.

The private-shadow rehearsal completed on 2026-08-21. The procedure below is retained for repeatability and rollback; completion evidence is in docs/evidence/core-hub-railiance01-private-shadow-2026-08-21.md. It still does not authorize public cutover.

1. Admission gates

Confirm metadata only:

the accepted rapp-postgres declaration names distinct runtime and migration roles for database core_hub;
namespace core-hub carries the reviewed PostgreSQL-client labels;
core-hub-api-token exposes key CORE_HUB_API_TOKEN;
core-hub-runtime-database and core-hub-migration-database each expose username and password keys from separate database-engine leases;
the Core Hub KV and database ClusterSecretStores are Ready and limited to the namespace; and
the old runtime lease will remain valid for the proven ESO refresh, watcher poll, graceful shutdown, restart and readiness interval.

Use kubectl get and kubectl describe; never print .data, decode a Secret, or place a token/DSN on an argument vector. Do not create placeholder Secrets to get past admission.

2. Rehearsal copy

The platform/database owner creates a consistent logical backup of the CoulombCore core_hub database and restores it into the accepted railiance01 consumer. Keep the source writer live for the rehearsal; record its snapshot timestamp and acknowledge that later writes will make this copy stale.

Record non-secret source and target evidence:

Alembic revision;
public-schema table set and per-table row counts;
database/schema ownership and runtime-versus-migration grants;
dump checksum, restore start/end timestamps and restore exit status; and
any sequence values that must remain monotonic after cutover.

Do not copy the source core-hub-prod-env Secret. The target database roles come from OpenBao's database engine, and the standing API token moves only through the approved static-token custody lane.

3. Schema and private deployment

From rapp-core-hub, with the reviewed railiance01 kubeconfig:

make check
make server-dry-run
make migrate
make deploy

make deploy keeps Ingress disabled by default. Verify that no hub.coulomb.social Ingress was created and that the public endpoint still resolves to CoulombCore.

The migration Job must mount only core-hub-migration-database. The API pod must mount only core-hub-runtime-database and core-hub-api-token.

4. Shadow verification

Run package metadata, rollout and unauthenticated checks:

make verify-live

Then port-forward the private Service and run Core Hub's authenticated smoke with an approved token file:

kubectl --kubeconfig "$KUBECONFIG" -n core-hub port-forward \
  service/core-hub-api 18010:8010

CORE_HUB_BASE_URL=http://127.0.0.1:18010 \
CORE_HUB_OPERATOR_TOKEN_FILE=/approved/path/to/operator-token \
CORE_HUB_SMOKE_OUTPUT=.local/smoke/railiance01-shadow.json \
make deployed-smoke

The token file is mode 0600, outside Git, and never copied into evidence. Retain only the non-secret JSON smoke report. Run the ops-hub gate and the activity-core sink against the same private endpoint through their governed credential paths.

Repeat table sets and row counts after migrations and smokes. Explain every difference; do not call the rehearsal reconciled merely because the API is Ready.

5. Lease-rotation proof

With the platform operator, observe a real runtime database lease refresh:

record the old pod UID/restart count and lease metadata without values;
let ESO project a renewed/reissued runtime lease normally;
confirm the watcher reports rotation without printing a credential;
confirm the old child terminates and the pod/container is replaced or restarted before the old lease expires;
wait for readiness and rerun health plus authenticated smoke; and
prove the migration role is absent from the API pod.

Only this live evidence closes RAPPCOREHUB-WP-0001-T07 and the consumer side of RPF-WP-0021-T03.

6. Rehearsal rollback

If any migration, reconciliation, smoke or rotation check fails, stop the railiance01 Deployment and preserve logs plus database evidence. The public Ingress remains absent, so production continues on CoulombCore. Reset the target from a fresh approved backup before retrying; do not repair copied data ad hoc and do not modify the source runtime.

Production cutover uses a separate operator-approved freeze/final-transfer procedure under CORE-WP-0011-T04.

4.8 KiB Raw Blame History