126 lines
4.8 KiB
Markdown
126 lines
4.8 KiB
Markdown
# Core Hub railiance01 shadow runbook
|
|
|
|
This runbook starts only after `RPF-WP-0021-T02/T04` have accepted the
|
|
PostgreSQL consumer and materialized the three governed Secret objects. It
|
|
stops before public Ingress, DNS, tunnel changes, a final write freeze, or any
|
|
CoulombCore teardown.
|
|
|
|
Authoritative deploy input: `/home/worsch/rapp-core-hub` at or after package
|
|
commit `7cfa8bb`, chart `0.3.1`.
|
|
|
|
The private-shadow rehearsal completed on 2026-08-21. The procedure below is
|
|
retained for repeatability and rollback; completion evidence is in
|
|
`docs/evidence/core-hub-railiance01-private-shadow-2026-08-21.md`. It still
|
|
does not authorize public cutover.
|
|
|
|
## 1. Admission gates
|
|
|
|
Confirm metadata only:
|
|
|
|
- the accepted `rapp-postgres` declaration names distinct runtime and migration
|
|
roles for database `core_hub`;
|
|
- namespace `core-hub` carries the reviewed PostgreSQL-client labels;
|
|
- `core-hub-api-token` exposes key `CORE_HUB_API_TOKEN`;
|
|
- `core-hub-runtime-database` and `core-hub-migration-database` each expose
|
|
`username` and `password` keys from separate database-engine leases;
|
|
- the Core Hub KV and database ClusterSecretStores are Ready and limited to the
|
|
namespace; and
|
|
- the old runtime lease will remain valid for the proven ESO refresh, watcher
|
|
poll, graceful shutdown, restart and readiness interval.
|
|
|
|
Use `kubectl get` and `kubectl describe`; never print `.data`, decode a Secret,
|
|
or place a token/DSN on an argument vector. Do not create placeholder Secrets
|
|
to get past admission.
|
|
|
|
## 2. Rehearsal copy
|
|
|
|
The platform/database owner creates a consistent logical backup of the
|
|
CoulombCore `core_hub` database and restores it into the accepted railiance01
|
|
consumer. Keep the source writer live for the rehearsal; record its snapshot
|
|
timestamp and acknowledge that later writes will make this copy stale.
|
|
|
|
Record non-secret source and target evidence:
|
|
|
|
- Alembic revision;
|
|
- public-schema table set and per-table row counts;
|
|
- database/schema ownership and runtime-versus-migration grants;
|
|
- dump checksum, restore start/end timestamps and restore exit status; and
|
|
- any sequence values that must remain monotonic after cutover.
|
|
|
|
Do not copy the source `core-hub-prod-env` Secret. The target database roles
|
|
come from OpenBao's database engine, and the standing API token moves only
|
|
through the approved static-token custody lane.
|
|
|
|
## 3. Schema and private deployment
|
|
|
|
From `rapp-core-hub`, with the reviewed railiance01 kubeconfig:
|
|
|
|
```sh
|
|
make check
|
|
make server-dry-run
|
|
make migrate
|
|
make deploy
|
|
```
|
|
|
|
`make deploy` keeps Ingress disabled by default. Verify that no
|
|
`hub.coulomb.social` Ingress was created and that the public endpoint still
|
|
resolves to CoulombCore.
|
|
|
|
The migration Job must mount only `core-hub-migration-database`. The API pod
|
|
must mount only `core-hub-runtime-database` and `core-hub-api-token`.
|
|
|
|
## 4. Shadow verification
|
|
|
|
Run package metadata, rollout and unauthenticated checks:
|
|
|
|
```sh
|
|
make verify-live
|
|
```
|
|
|
|
Then port-forward the private Service and run Core Hub's authenticated smoke
|
|
with an approved token file:
|
|
|
|
```sh
|
|
kubectl --kubeconfig "$KUBECONFIG" -n core-hub port-forward \
|
|
service/core-hub-api 18010:8010
|
|
|
|
CORE_HUB_BASE_URL=http://127.0.0.1:18010 \
|
|
CORE_HUB_OPERATOR_TOKEN_FILE=/approved/path/to/operator-token \
|
|
CORE_HUB_SMOKE_OUTPUT=.local/smoke/railiance01-shadow.json \
|
|
make deployed-smoke
|
|
```
|
|
|
|
The token file is mode 0600, outside Git, and never copied into evidence.
|
|
Retain only the non-secret JSON smoke report. Run the ops-hub gate and the
|
|
activity-core sink against the same private endpoint through their governed
|
|
credential paths.
|
|
|
|
Repeat table sets and row counts after migrations and smokes. Explain every
|
|
difference; do not call the rehearsal reconciled merely because the API is
|
|
Ready.
|
|
|
|
## 5. Lease-rotation proof
|
|
|
|
With the platform operator, observe a real runtime database lease refresh:
|
|
|
|
1. record the old pod UID/restart count and lease metadata without values;
|
|
2. let ESO project a renewed/reissued runtime lease normally;
|
|
3. confirm the watcher reports rotation without printing a credential;
|
|
4. confirm the old child terminates and the pod/container is replaced or
|
|
restarted before the old lease expires;
|
|
5. wait for readiness and rerun health plus authenticated smoke; and
|
|
6. prove the migration role is absent from the API pod.
|
|
|
|
Only this live evidence closes `RAPPCOREHUB-WP-0001-T07` and the consumer side
|
|
of `RPF-WP-0021-T03`.
|
|
|
|
## 6. Rehearsal rollback
|
|
|
|
If any migration, reconciliation, smoke or rotation check fails, stop the
|
|
railiance01 Deployment and preserve logs plus database evidence. The public
|
|
Ingress remains absent, so production continues on CoulombCore. Reset the
|
|
target from a fresh approved backup before retrying; do not repair copied data
|
|
ad hoc and do not modify the source runtime.
|
|
|
|
Production cutover uses a separate operator-approved freeze/final-transfer
|
|
procedure under `CORE-WP-0011-T04`.
|