# Core Hub railiance01 shadow runbook

This runbook starts only after `RPF-WP-0021-T02/T04` have accepted the
PostgreSQL consumer and materialized the three governed Secret objects. It
stops before public Ingress, DNS, tunnel changes, a final write freeze, or any
CoulombCore teardown.

Authoritative deploy input: `/home/worsch/rapp-core-hub` at or after package
commit `7cfa8bb`, chart `0.3.1`.

The private-shadow rehearsal completed on 2026-08-21. The procedure below is
retained for repeatability and rollback; completion evidence is in
`docs/evidence/core-hub-railiance01-private-shadow-2026-08-21.md`. It still
does not authorize public cutover.

## 1. Admission gates

Confirm metadata only:

- the accepted `rapp-postgres` declaration names distinct runtime and migration
  roles for database `core_hub`;
- namespace `core-hub` carries the reviewed PostgreSQL-client labels;
- `core-hub-api-token` exposes key `CORE_HUB_API_TOKEN`;
- `core-hub-runtime-database` and `core-hub-migration-database` each expose
  `username` and `password` keys from separate database-engine leases;
- the Core Hub KV and database ClusterSecretStores are Ready and limited to the
  namespace; and
- the old runtime lease will remain valid for the proven ESO refresh, watcher
  poll, graceful shutdown, restart and readiness interval.

Use `kubectl get` and `kubectl describe`; never print `.data`, decode a Secret,
or place a token/DSN on an argument vector. Do not create placeholder Secrets
to get past admission.

## 2. Rehearsal copy

The platform/database owner creates a consistent logical backup of the
CoulombCore `core_hub` database and restores it into the accepted railiance01
consumer. Keep the source writer live for the rehearsal; record its snapshot
timestamp and acknowledge that later writes will make this copy stale.

Record non-secret source and target evidence:

- Alembic revision;
- public-schema table set and per-table row counts;
- database/schema ownership and runtime-versus-migration grants;
- dump checksum, restore start/end timestamps and restore exit status; and
- any sequence values that must remain monotonic after cutover.

Do not copy the source `core-hub-prod-env` Secret. The target database roles
come from OpenBao's database engine, and the standing API token moves only
through the approved static-token custody lane.

## 3. Schema and private deployment

From `rapp-core-hub`, with the reviewed railiance01 kubeconfig:

```sh
make check
make server-dry-run
make migrate
make deploy
```

`make deploy` keeps Ingress disabled by default. Verify that no
`hub.coulomb.social` Ingress was created and that the public endpoint still
resolves to CoulombCore.

The migration Job must mount only `core-hub-migration-database`. The API pod
must mount only `core-hub-runtime-database` and `core-hub-api-token`.

## 4. Shadow verification

Run package metadata, rollout and unauthenticated checks:

```sh
make verify-live
```

Then port-forward the private Service and run Core Hub's authenticated smoke
with an approved token file:

```sh
kubectl --kubeconfig "$KUBECONFIG" -n core-hub port-forward \
  service/core-hub-api 18010:8010

CORE_HUB_BASE_URL=http://127.0.0.1:18010 \
CORE_HUB_OPERATOR_TOKEN_FILE=/approved/path/to/operator-token \
CORE_HUB_SMOKE_OUTPUT=.local/smoke/railiance01-shadow.json \
make deployed-smoke
```

The token file is mode 0600, outside Git, and never copied into evidence.
Retain only the non-secret JSON smoke report. Run the ops-hub gate and the
activity-core sink against the same private endpoint through their governed
credential paths.

Repeat table sets and row counts after migrations and smokes. Explain every
difference; do not call the rehearsal reconciled merely because the API is
Ready.

## 5. Lease-rotation proof

With the platform operator, observe a real runtime database lease refresh:

1. record the old pod UID/restart count and lease metadata without values;
2. let ESO project a renewed/reissued runtime lease normally;
3. confirm the watcher reports rotation without printing a credential;
4. confirm the old child terminates and the pod/container is replaced or
   restarted before the old lease expires;
5. wait for readiness and rerun health plus authenticated smoke; and
6. prove the migration role is absent from the API pod.

Only this live evidence closes `RAPPCOREHUB-WP-0001-T07` and the consumer side
of `RPF-WP-0021-T03`.

## 6. Rehearsal rollback

If any migration, reconciliation, smoke or rotation check fails, stop the
railiance01 Deployment and preserve logs plus database evidence. The public
Ingress remains absent, so production continues on CoulombCore. Reset the
target from a fresh approved backup before retrying; do not repair copied data
ad hoc and do not modify the source runtime.

Production cutover uses a separate operator-approved freeze/final-transfer
procedure under `CORE-WP-0011-T04`.