139 lines
4.5 KiB
Markdown
139 lines
4.5 KiB
Markdown
|
# reuse-surface Runtime Secrets — Rotation Runbook
|
|||
|
|
|
|||
|
|
Status: active (RAILIANCE-WP-0011-T04 / CCR-2026-0005)
|
|||
|
|
Date: 2026-07-07
|
|||
|
|
|
|||
|
|
Operator procedure for rotating `REUSE_SURFACE_TOKEN` and/or
|
|||
|
|
`REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET` in OpenBao custody. Never paste secret
|
|||
|
|
values into Git, State Hub, chat, prompts, or shell history.
|
|||
|
|
|
|||
|
|
**Canonical lane:** `platform/workloads/reuse/reuse-surface/runtime-secrets`
|
|||
|
|
**Delivery:** `ExternalSecret reuse/reuse-surface-runtime` → `reuse-surface-env` (Railiance01)
|
|||
|
|
**Lifecycle generator:** `scripts/credential-change.py lifecycle-plan CCR-2026-0005 --action rotate`
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Prerequisites
|
|||
|
|
|
|||
|
|
- OpenBao operator authority on `https://bao.coulomb.social`
|
|||
|
|
- Railiance01 kubeconfig: `~/.kube/config-hosteurope`
|
|||
|
|
- Forgejo admin API token: `/tmp/forgejo-tegwick-api-token` (when rotating webhook HMAC)
|
|||
|
|
- `railiance-apps` checkout with `make reuse-runtime-es-status`, `make reuse-forgejo-webhook`,
|
|||
|
|
`make reuse-webhook-smoke`
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Rotate `REUSE_SURFACE_TOKEN` only
|
|||
|
|
|
|||
|
|
Hub write clients and operators must pick up the new bearer token. Forgejo webhook
|
|||
|
|
HMAC is unchanged.
|
|||
|
|
|
|||
|
|
1. **Write new value in OpenBao** (generate locally; never log):
|
|||
|
|
```bash
|
|||
|
|
NEW_TOKEN="$(openssl rand -hex 32)"
|
|||
|
|
bao kv patch platform/workloads/reuse/reuse-surface/runtime-secrets \
|
|||
|
|
REUSE_SURFACE_TOKEN="${NEW_TOKEN}"
|
|||
|
|
unset NEW_TOKEN
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
2. **Force ESO refresh** (default interval is 1h):
|
|||
|
|
```bash
|
|||
|
|
kubectl --kubeconfig ~/.kube/config-hosteurope annotate externalsecret \
|
|||
|
|
reuse-surface-runtime -n reuse \
|
|||
|
|
force-sync="$(date +%s)" --overwrite
|
|||
|
|
make -C ~/railiance-apps reuse-runtime-es-status
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
3. **Roll the hub Deployment** (`envFrom` does not hot-reload):
|
|||
|
|
```bash
|
|||
|
|
kubectl --kubeconfig ~/.kube/config-hosteurope rollout restart \
|
|||
|
|
deployment/reuse-surface -n reuse
|
|||
|
|
kubectl --kubeconfig ~/.kube/config-hosteurope rollout status \
|
|||
|
|
deployment/reuse-surface -n reuse
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
4. **Smoke**:
|
|||
|
|
```bash
|
|||
|
|
make -C ~/railiance-apps reuse-webhook-smoke
|
|||
|
|
export REUSE_SURFACE_URL=https://reuse.coulomb.social
|
|||
|
|
export REUSE_SURFACE_TOKEN=$(
|
|||
|
|
bao kv get -field=REUSE_SURFACE_TOKEN \
|
|||
|
|
platform/workloads/reuse/reuse-surface/runtime-secrets
|
|||
|
|
)
|
|||
|
|
reuse-surface hub status
|
|||
|
|
unset REUSE_SURFACE_TOKEN
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
5. **Record** non-secret evidence:
|
|||
|
|
```bash
|
|||
|
|
cd ~/railiance-platform
|
|||
|
|
scripts/credential-change.py lifecycle-event CCR-2026-0005 --action rotate \
|
|||
|
|
--actor <operator> --reason "REUSE_SURFACE_TOKEN rotation" \
|
|||
|
|
--detail "ESO SecretSynced + hub rollout + smoke passed" --record-state-hub
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Rotate `REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET` only
|
|||
|
|
|
|||
|
|
The hub pod **and** the Forgejo org webhook (`coulomb`, hook id `1`) must share
|
|||
|
|
the same HMAC. Order matters: OpenBao → ESO → hub rollout → Forgejo webhook.
|
|||
|
|
|
|||
|
|
1. **Write new HMAC in OpenBao**:
|
|||
|
|
```bash
|
|||
|
|
NEW_HMAC="$(openssl rand -hex 32)"
|
|||
|
|
bao kv patch platform/workloads/reuse/reuse-surface/runtime-secrets \
|
|||
|
|
REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET="${NEW_HMAC}"
|
|||
|
|
unset NEW_HMAC
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
2. **Force ESO refresh and roll hub** (steps 2–3 from token rotation above).
|
|||
|
|
|
|||
|
|
3. **Reconcile Forgejo org webhook** (reads HMAC from materialized cluster Secret):
|
|||
|
|
```bash
|
|||
|
|
make -C ~/railiance-apps reuse-forgejo-webhook
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
4. **Smoke** (signed POST must return 200; unsigned must return 401):
|
|||
|
|
```bash
|
|||
|
|
make -C ~/railiance-apps reuse-webhook-smoke
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
5. **Record** lifecycle event (same as token rotation; note webhook field in reason).
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Rotate both fields
|
|||
|
|
|
|||
|
|
Perform token rotation steps 1–4, then run `make reuse-forgejo-webhook` and
|
|||
|
|
`make reuse-webhook-smoke` before recording evidence. Alternatively patch both
|
|||
|
|
fields in a single `bao kv patch` and follow the webhook rotation order (ESO →
|
|||
|
|
rollout → Forgejo webhook → smoke).
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Verification checklist
|
|||
|
|
|
|||
|
|
| Check | Expected |
|
|||
|
|
| --- | --- |
|
|||
|
|
| `ExternalSecret reuse/reuse-surface-runtime` | `Ready=True`, `SecretSynced` |
|
|||
|
|
| Unsigned `POST /v1/webhooks/forgejo` | `401` |
|
|||
|
|
| Signed no-op push payload | `200` with `"accepted": false` |
|
|||
|
|
| `GET /v1/federated` | `200` |
|
|||
|
|
| `reuse-surface hub status` with new token | `200` on `/health` |
|
|||
|
|
|
|||
|
|
Automated: `make -C ~/railiance-apps reuse-webhook-smoke`
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Compromise or emergency
|
|||
|
|
|
|||
|
|
Use the CCR generator first — disable the ops-warden front door before metadata
|
|||
|
|
changes:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
scripts/credential-change.py lifecycle-plan CCR-2026-0005 --action compromise
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Consumer-specific steps (Forgejo webhook alignment, materialized Secret deletion)
|
|||
|
|
are in `docs/credential-lane-lifecycle-runbook.md` under CCR-2026-0005.
|