railiance-platform/docs/reuse-surface-runtime-secrets-rotation-runbook.md

reuse-surface Runtime Secrets — Rotation Runbook

Status: active (RAILIANCE-WP-0011-T04 / CCR-2026-0005)
Date: 2026-07-07

Operator procedure for rotating REUSE_SURFACE_TOKEN and/or REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET in OpenBao custody. Never paste secret values into Git, State Hub, chat, prompts, or shell history.

Canonical lane: platform/workloads/reuse/reuse-surface/runtime-secrets
Delivery: ExternalSecret reuse/reuse-surface-runtime → reuse-surface-env (Railiance01)
Lifecycle generator: scripts/credential-change.py lifecycle-plan CCR-2026-0005 --action rotate

---

Prerequisites

• OpenBao operator authority on https://bao.coulomb.social
• Railiance01 kubeconfig: ~/.kube/config-hosteurope
• Forgejo admin API token: /tmp/forgejo-tegwick-api-token (when rotating webhook HMAC)
• railiance-apps checkout with make reuse-runtime-es-status, make reuse-forgejo-webhook, make reuse-webhook-smoke

---

Rotate REUSE_SURFACE_TOKEN only

Hub write clients and operators must pick up the new bearer token. Forgejo webhook HMAC is unchanged.

1. Write new value in OpenBao (generate locally; never log):

   NEW_TOKEN="$(openssl rand -hex 32)"
   bao kv patch platform/workloads/reuse/reuse-surface/runtime-secrets \
     REUSE_SURFACE_TOKEN="${NEW_TOKEN}"
   unset NEW_TOKEN
   

2. Force ESO refresh (default interval is 1h):

   kubectl --kubeconfig ~/.kube/config-hosteurope annotate externalsecret \
     reuse-surface-runtime -n reuse \
     force-sync="$(date +%s)" --overwrite
   make -C ~/railiance-apps reuse-runtime-es-status
   

3. Roll the hub Deployment (envFrom does not hot-reload):

   kubectl --kubeconfig ~/.kube/config-hosteurope rollout restart \
     deployment/reuse-surface -n reuse
   kubectl --kubeconfig ~/.kube/config-hosteurope rollout status \
     deployment/reuse-surface -n reuse
   

4. Smoke:

   make -C ~/railiance-apps reuse-webhook-smoke
   export REUSE_SURFACE_URL=https://reuse.coulomb.social
   export REUSE_SURFACE_TOKEN=$(
     bao kv get -field=REUSE_SURFACE_TOKEN \
       platform/workloads/reuse/reuse-surface/runtime-secrets
   )
   reuse-surface hub status
   unset REUSE_SURFACE_TOKEN
   

5. Record non-secret evidence:

   cd ~/railiance-platform
   scripts/credential-change.py lifecycle-event CCR-2026-0005 --action rotate \
     --actor <operator> --reason "REUSE_SURFACE_TOKEN rotation" \
     --detail "ESO SecretSynced + hub rollout + smoke passed" --record-state-hub
   

---

Rotate REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET only

The hub pod and the Forgejo org webhook (coulomb, hook id 1) must share the same HMAC. Order matters: OpenBao → ESO → hub rollout → Forgejo webhook.

1. Write new HMAC in OpenBao:

   NEW_HMAC="$(openssl rand -hex 32)"
   bao kv patch platform/workloads/reuse/reuse-surface/runtime-secrets \
     REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET="${NEW_HMAC}"
   unset NEW_HMAC
   

2. Force ESO refresh and roll hub (steps 2–3 from token rotation above).

3. Reconcile Forgejo org webhook (reads HMAC from materialized cluster Secret):

   make -C ~/railiance-apps reuse-forgejo-webhook
   

4. Smoke (signed POST must return 200; unsigned must return 401):

   make -C ~/railiance-apps reuse-webhook-smoke
   

5. Record lifecycle event (same as token rotation; note webhook field in reason).

---

Rotate both fields

Perform token rotation steps 1–4, then run make reuse-forgejo-webhook and make reuse-webhook-smoke before recording evidence. Alternatively patch both fields in a single bao kv patch and follow the webhook rotation order (ESO → rollout → Forgejo webhook → smoke).

---

Verification checklist

Check	Expected
ExternalSecret reuse/reuse-surface-runtime	Ready=True, SecretSynced
Unsigned POST /v1/webhooks/forgejo	401
Signed no-op push payload	200 with "accepted": false
GET /v1/federated	200
reuse-surface hub status with new token	200 on /health

Automated: make -C ~/railiance-apps reuse-webhook-smoke

---

Compromise or emergency

Use the CCR generator first — disable the ops-warden front door before metadata changes:

scripts/credential-change.py lifecycle-plan CCR-2026-0005 --action compromise

Consumer-specific steps (Forgejo webhook alignment, materialized Secret deletion) are in docs/credential-lane-lifecycle-runbook.md under CCR-2026-0005.