tegwick
839a4418a9
Document OpenBao patch, ESO force-sync, hub rollout, and Forgejo webhook reconcile; extend credential-lane lifecycle for CCR-2026-0005; finish workplan.
139 lines
No EOL
4.5 KiB
Markdown
139 lines
No EOL
4.5 KiB
Markdown
# reuse-surface Runtime Secrets — Rotation Runbook
|
||
|
||
Status: active (RAILIANCE-WP-0011-T04 / CCR-2026-0005)
|
||
Date: 2026-07-07
|
||
|
||
Operator procedure for rotating `REUSE_SURFACE_TOKEN` and/or
|
||
`REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET` in OpenBao custody. Never paste secret
|
||
values into Git, State Hub, chat, prompts, or shell history.
|
||
|
||
**Canonical lane:** `platform/workloads/reuse/reuse-surface/runtime-secrets`
|
||
**Delivery:** `ExternalSecret reuse/reuse-surface-runtime` → `reuse-surface-env` (Railiance01)
|
||
**Lifecycle generator:** `scripts/credential-change.py lifecycle-plan CCR-2026-0005 --action rotate`
|
||
|
||
---
|
||
|
||
## Prerequisites
|
||
|
||
- OpenBao operator authority on `https://bao.coulomb.social`
|
||
- Railiance01 kubeconfig: `~/.kube/config-hosteurope`
|
||
- Forgejo admin API token: `/tmp/forgejo-tegwick-api-token` (when rotating webhook HMAC)
|
||
- `railiance-apps` checkout with `make reuse-runtime-es-status`, `make reuse-forgejo-webhook`,
|
||
`make reuse-webhook-smoke`
|
||
|
||
---
|
||
|
||
## Rotate `REUSE_SURFACE_TOKEN` only
|
||
|
||
Hub write clients and operators must pick up the new bearer token. Forgejo webhook
|
||
HMAC is unchanged.
|
||
|
||
1. **Write new value in OpenBao** (generate locally; never log):
|
||
```bash
|
||
NEW_TOKEN="$(openssl rand -hex 32)"
|
||
bao kv patch platform/workloads/reuse/reuse-surface/runtime-secrets \
|
||
REUSE_SURFACE_TOKEN="${NEW_TOKEN}"
|
||
unset NEW_TOKEN
|
||
```
|
||
|
||
2. **Force ESO refresh** (default interval is 1h):
|
||
```bash
|
||
kubectl --kubeconfig ~/.kube/config-hosteurope annotate externalsecret \
|
||
reuse-surface-runtime -n reuse \
|
||
force-sync="$(date +%s)" --overwrite
|
||
make -C ~/railiance-apps reuse-runtime-es-status
|
||
```
|
||
|
||
3. **Roll the hub Deployment** (`envFrom` does not hot-reload):
|
||
```bash
|
||
kubectl --kubeconfig ~/.kube/config-hosteurope rollout restart \
|
||
deployment/reuse-surface -n reuse
|
||
kubectl --kubeconfig ~/.kube/config-hosteurope rollout status \
|
||
deployment/reuse-surface -n reuse
|
||
```
|
||
|
||
4. **Smoke**:
|
||
```bash
|
||
make -C ~/railiance-apps reuse-webhook-smoke
|
||
export REUSE_SURFACE_URL=https://reuse.coulomb.social
|
||
export REUSE_SURFACE_TOKEN=$(
|
||
bao kv get -field=REUSE_SURFACE_TOKEN \
|
||
platform/workloads/reuse/reuse-surface/runtime-secrets
|
||
)
|
||
reuse-surface hub status
|
||
unset REUSE_SURFACE_TOKEN
|
||
```
|
||
|
||
5. **Record** non-secret evidence:
|
||
```bash
|
||
cd ~/railiance-platform
|
||
scripts/credential-change.py lifecycle-event CCR-2026-0005 --action rotate \
|
||
--actor <operator> --reason "REUSE_SURFACE_TOKEN rotation" \
|
||
--detail "ESO SecretSynced + hub rollout + smoke passed" --record-state-hub
|
||
```
|
||
|
||
---
|
||
|
||
## Rotate `REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET` only
|
||
|
||
The hub pod **and** the Forgejo org webhook (`coulomb`, hook id `1`) must share
|
||
the same HMAC. Order matters: OpenBao → ESO → hub rollout → Forgejo webhook.
|
||
|
||
1. **Write new HMAC in OpenBao**:
|
||
```bash
|
||
NEW_HMAC="$(openssl rand -hex 32)"
|
||
bao kv patch platform/workloads/reuse/reuse-surface/runtime-secrets \
|
||
REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET="${NEW_HMAC}"
|
||
unset NEW_HMAC
|
||
```
|
||
|
||
2. **Force ESO refresh and roll hub** (steps 2–3 from token rotation above).
|
||
|
||
3. **Reconcile Forgejo org webhook** (reads HMAC from materialized cluster Secret):
|
||
```bash
|
||
make -C ~/railiance-apps reuse-forgejo-webhook
|
||
```
|
||
|
||
4. **Smoke** (signed POST must return 200; unsigned must return 401):
|
||
```bash
|
||
make -C ~/railiance-apps reuse-webhook-smoke
|
||
```
|
||
|
||
5. **Record** lifecycle event (same as token rotation; note webhook field in reason).
|
||
|
||
---
|
||
|
||
## Rotate both fields
|
||
|
||
Perform token rotation steps 1–4, then run `make reuse-forgejo-webhook` and
|
||
`make reuse-webhook-smoke` before recording evidence. Alternatively patch both
|
||
fields in a single `bao kv patch` and follow the webhook rotation order (ESO →
|
||
rollout → Forgejo webhook → smoke).
|
||
|
||
---
|
||
|
||
## Verification checklist
|
||
|
||
| Check | Expected |
|
||
| --- | --- |
|
||
| `ExternalSecret reuse/reuse-surface-runtime` | `Ready=True`, `SecretSynced` |
|
||
| Unsigned `POST /v1/webhooks/forgejo` | `401` |
|
||
| Signed no-op push payload | `200` with `"accepted": false` |
|
||
| `GET /v1/federated` | `200` |
|
||
| `reuse-surface hub status` with new token | `200` on `/health` |
|
||
|
||
Automated: `make -C ~/railiance-apps reuse-webhook-smoke`
|
||
|
||
---
|
||
|
||
## Compromise or emergency
|
||
|
||
Use the CCR generator first — disable the ops-warden front door before metadata
|
||
changes:
|
||
|
||
```bash
|
||
scripts/credential-change.py lifecycle-plan CCR-2026-0005 --action compromise
|
||
```
|
||
|
||
Consumer-specific steps (Forgejo webhook alignment, materialized Secret deletion)
|
||
are in `docs/credential-lane-lifecycle-runbook.md` under CCR-2026-0005. |