ops-hub/docs/bootstrap-runbook.md

Ops Hub Bootstrap Runbook

Date: 2026-08-31

Purpose

This runbook is the operator-ready path for activating ops-hub in production Hub Core's /api/v2 compatibility surface. It covers the preferred API bootstrap, key custody expectations, the ops-warden remote execution lane, and the explicit SQL fallback.

Use this when finishing CUST-WP-0047-T05 or any later ops-hub Inter-Hub activation task.

Inputs

• Hub-extension package: registry/hub-extension/v0.1.0/ops-hub.extension.json
• Manifest draft: seeds/ops-hub-manifest.draft.json
• Widget seed: seeds/ops-hub-widgets.seed.json
• API helper: scripts/ops-hub-bootstrap-api.py
• Migration fallback: seeds/ops-hub-bootstrap.sql
• Production Inter-Hub base URL: https://hub.coulomb.social

Secret Custody Rules

• Do not paste operator keys into Codex-visible chat, workplans, commits, or shell transcripts.
• Prefer IHUB_OPERATOR_KEY_FILE over IHUB_OPERATOR_KEY.
• Operator key temp files must be mode 0600 and removed after the run.
• The generated ops-hub runtime key is display-once material. Store it in OpenBao or the approved operator secret store immediately.
• The helper prints only non-secret ids, key prefixes, and file paths.

Preferred API Bootstrap

First confirm the API surface:

make interhub-gate

Then materialize the Inter-Hub operator key into a temporary file. Example local pattern:

umask 077
operator_key_file="$(mktemp)"
# Write the operator key into "$operator_key_file" from the approved secret
# source. Do not echo it into shared logs.

Plan the bootstrap without mutations:

make interhub-bootstrap-dry-run \
  IHUB_BASE=https://hub.coulomb.social \
  IHUB_OPERATOR_KEY_FILE="$operator_key_file"

Run the attended bootstrap:

make interhub-bootstrap \
  IHUB_BASE=https://hub.coulomb.social \
  IHUB_OPERATOR_KEY_FILE="$operator_key_file"

The helper creates or reuses:

• the ops-hub hub row
• the active ops-hub capability manifest
• the ops-hub API consumer
• an ops-hub runtime API key, when one was not supplied
• the seed widgets from seeds/ops-hub-widgets.seed.json
• the initial Gitea readiness event, unless --skip-event is used directly

If the helper creates a runtime key, it writes the full value to a 0600 file and prints the path plus key prefix. Store that key immediately, then remove the file:

runtime_key_file="/path/printed/as/runtimeKey.keyFile"
# Example path only; use the approved OpenBao mount/path for the environment.
bao kv put secret/platform/operators/ops-hub/runtime \
  OPS_HUB_KEY="$(cat "$runtime_key_file")"

rm -f "$operator_key_file" "$runtime_key_file"

If an ops-hub runtime key already exists in the secret store, materialize it as OPS_HUB_KEY_FILE before the run. The helper will reuse it instead of creating a new display-once key.

Remote Execution With Ops-Warden

When the operator key must stay on a trusted host, use the ops-warden access lane instead of moving the key into the workstation session:

1. Add or reuse an agt actor such as agt-codex-interhub-bootstrap with a narrow principal such as agt-interhub-bootstrap and a short TTL.
2. Use ops-ssh-wrapper to acquire a short-lived SSH certificate.
3. On the target host, railiance-infra should map the principal to a narrow force-command or wrapper that runs only this bootstrap path.
4. The host wrapper reads the operator key from OpenBao or an attended temp file, runs make interhub-bootstrap, stores the generated ops-hub runtime key back into OpenBao, and emits non-secret JSON evidence.

See ops-warden/wiki/InterHubBootstrapAccessLane.md for the certificate envelope and host-boundary rules.

SQL Fallback Path

Prefer the API bootstrap whenever possible. Use SQL only when the operator explicitly approves deployment-side migration/bootstrap execution.

From a shell with Railiance01 Kubernetes access:

kubectl exec -i -n databases net-kingdom-pg-1 -- \
  psql -U postgres -d interhub \
  < /home/worsch/ops-hub/seeds/ops-hub-bootstrap.sql

If using the HostEurope kubeconfig from the workstation:

KUBECONFIG=~/.kube/config-hosteurope \
kubectl exec -i -n databases net-kingdom-pg-1 -- \
  psql -U postgres -d interhub \
  < /home/worsch/ops-hub/seeds/ops-hub-bootstrap.sql

The SQL fallback creates the hub, active manifest, registry entries, API consumer row, and seed widgets. It does not create the one-time visible static API key. Generate that through the authenticated Inter-Hub UI or the supported API helper and store it outside Git.

Validation

After manifest activation:

curl -s https://hub.coulomb.social/api/v2/widget-types
curl -s https://hub.coulomb.social/api/v2/event-types
curl -s https://hub.coulomb.social/api/v2/annotation-categories

Expected: ops-owned vocabulary appears in the relevant registries.

After API key creation or reuse:

curl -s -X POST https://hub.coulomb.social/api/v2/token \
  -H "Content-Type: application/x-www-form-urlencoded" \
  --data-urlencode "grant_type=client_credentials" \
  --data-urlencode "client_id=<api-consumer-id>" \
  --data-urlencode "client_secret=<static-api-key>" \
  --data-urlencode "scope=framework:read hub:ops-hub:read hub:ops-hub:write"

Expected: a short-lived access token is returned.

After widget seeding:

curl -s https://hub.coulomb.social/api/v2/hub-registry

Expected: ops-hub is visible, and the operator can see the seeded widgets in the authenticated UI.

Authenticated rerun boundary

Ops Hub bootstrap and consumer gates passed during the hub-core candidate cutover (RAPPCOREHUB-WP-0002-T04), and the 2026-08-31 post-cutover public gate passes against https://hub.coulomb.social. There is no pending bootstrap mutation merely to finish OPS-WP-0003.

An authenticated rerun remains a high-risk attended operation because it may create a consumer key or production records. Use it only for a real registration change, start with warden route show inter-hub-bootstrap-ssh --json, and record non-secret evidence. For routine validation use:

make test
make hub-extension-check
make hub-core-conformance
make interhub-gate