Add reuse-surface secrets rotation runbook (RAILIANCE-WP-0011-T04)
Document OpenBao patch, ESO force-sync, hub rollout, and Forgejo webhook reconcile; extend credential-lane lifecycle for CCR-2026-0005; finish workplan.
This commit is contained in:
parent
274b631019
commit
839a4418a9
4 changed files with 185 additions and 12 deletions
|
|
@ -1,13 +1,13 @@
|
|||
# Credential Lane Lifecycle Runbook
|
||||
|
||||
Status: active (RAILIANCE-WP-0009-T07 / RAILIANCE-WP-0010-T07)
|
||||
Date: 2026-07-02
|
||||
Status: active (RAILIANCE-WP-0009-T07 / RAILIANCE-WP-0010-T07 / RAILIANCE-WP-0011-T04)
|
||||
Date: 2026-07-07
|
||||
|
||||
Covers deactivation, rotation, and compromise response for the workload KV
|
||||
lanes established by `CCR-2026-0002` (issue-core) and `CCR-2026-0003`
|
||||
(llm-connect). The **canonical, always-current procedure** is generated from
|
||||
the CCR itself — this runbook adds only the lane-specific consumer facts the
|
||||
generator cannot know.
|
||||
lanes established by `CCR-2026-0002` (issue-core), `CCR-2026-0003`
|
||||
(llm-connect), and `CCR-2026-0005` (reuse-surface). The **canonical,
|
||||
always-current procedure** is generated from the CCR itself — this runbook adds
|
||||
only the lane-specific consumer facts the generator cannot know.
|
||||
|
||||
```bash
|
||||
scripts/credential-change.py lifecycle-plan <CCR-ID> --action {deactivate|rotate|compromise}
|
||||
|
|
@ -76,6 +76,33 @@ secret value ever appears in Git, State Hub, chat, prompts, or shell history.
|
|||
or immediately after disabling the front door; OpenBao custody actions
|
||||
alone do not stop a leaked provider key from working.
|
||||
|
||||
## Lane: reuse-surface hub runtime secrets (`CCR-2026-0005`)
|
||||
|
||||
| Item | Value |
|
||||
| --- | --- |
|
||||
| KV path | `platform/workloads/reuse/reuse-surface/runtime-secrets` |
|
||||
| Fields | `REUSE_SURFACE_TOKEN`, `REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET` |
|
||||
| Policy / auth role | `workload-kv-read-reuse-surface-runtime` / `auth/kubernetes/role/external-secrets-reuse-surface` |
|
||||
| Primary consumer | ExternalSecret `reuse/reuse-surface-runtime` → `reuse-surface-env` (Railiance01, 1h refresh) |
|
||||
| ops-warden catalog | `reuse-surface-hub-write-token` |
|
||||
| Rotation runbook | `docs/reuse-surface-runtime-secrets-rotation-runbook.md` |
|
||||
|
||||
**Consumer facts the generated plan does not cover:**
|
||||
|
||||
- The hub Deployment uses `envFrom.secretRef`; rotated values reach the pod only
|
||||
after ESO refresh **and** `kubectl rollout restart deployment/reuse-surface -n reuse`.
|
||||
- **`REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET` is dual-consumer**: the hub pod and the
|
||||
Forgejo org webhook on `coulomb` must share the same HMAC. After rotating that
|
||||
field, run `railiance-apps` `make reuse-forgejo-webhook` once ESO has synced —
|
||||
otherwise Forgejo deliveries fail with 401 while the hub still expects the new
|
||||
value (or vice versa if order is reversed).
|
||||
- Deactivating the policy/role stops ESO refresh, but `secret/reuse-surface-env`
|
||||
retains the last materialized value until explicitly deleted.
|
||||
- Steady-state operator fetch is `bao kv get` / `warden access`; kubectl read of
|
||||
`reuse-surface-env` is break-glass only.
|
||||
|
||||
**Smoke after rotate:** `make -C ~/railiance-apps reuse-webhook-smoke`
|
||||
|
||||
## Verification after rotate
|
||||
|
||||
Return the lane to `active` only with fresh positive + negative evidence,
|
||||
|
|
|
|||
139
docs/reuse-surface-runtime-secrets-rotation-runbook.md
Normal file
139
docs/reuse-surface-runtime-secrets-rotation-runbook.md
Normal file
|
|
@ -0,0 +1,139 @@
|
|||
# reuse-surface Runtime Secrets — Rotation Runbook
|
||||
|
||||
Status: active (RAILIANCE-WP-0011-T04 / CCR-2026-0005)
|
||||
Date: 2026-07-07
|
||||
|
||||
Operator procedure for rotating `REUSE_SURFACE_TOKEN` and/or
|
||||
`REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET` in OpenBao custody. Never paste secret
|
||||
values into Git, State Hub, chat, prompts, or shell history.
|
||||
|
||||
**Canonical lane:** `platform/workloads/reuse/reuse-surface/runtime-secrets`
|
||||
**Delivery:** `ExternalSecret reuse/reuse-surface-runtime` → `reuse-surface-env` (Railiance01)
|
||||
**Lifecycle generator:** `scripts/credential-change.py lifecycle-plan CCR-2026-0005 --action rotate`
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- OpenBao operator authority on `https://bao.coulomb.social`
|
||||
- Railiance01 kubeconfig: `~/.kube/config-hosteurope`
|
||||
- Forgejo admin API token: `/tmp/forgejo-tegwick-api-token` (when rotating webhook HMAC)
|
||||
- `railiance-apps` checkout with `make reuse-runtime-es-status`, `make reuse-forgejo-webhook`,
|
||||
`make reuse-webhook-smoke`
|
||||
|
||||
---
|
||||
|
||||
## Rotate `REUSE_SURFACE_TOKEN` only
|
||||
|
||||
Hub write clients and operators must pick up the new bearer token. Forgejo webhook
|
||||
HMAC is unchanged.
|
||||
|
||||
1. **Write new value in OpenBao** (generate locally; never log):
|
||||
```bash
|
||||
NEW_TOKEN="$(openssl rand -hex 32)"
|
||||
bao kv patch platform/workloads/reuse/reuse-surface/runtime-secrets \
|
||||
REUSE_SURFACE_TOKEN="${NEW_TOKEN}"
|
||||
unset NEW_TOKEN
|
||||
```
|
||||
|
||||
2. **Force ESO refresh** (default interval is 1h):
|
||||
```bash
|
||||
kubectl --kubeconfig ~/.kube/config-hosteurope annotate externalsecret \
|
||||
reuse-surface-runtime -n reuse \
|
||||
force-sync="$(date +%s)" --overwrite
|
||||
make -C ~/railiance-apps reuse-runtime-es-status
|
||||
```
|
||||
|
||||
3. **Roll the hub Deployment** (`envFrom` does not hot-reload):
|
||||
```bash
|
||||
kubectl --kubeconfig ~/.kube/config-hosteurope rollout restart \
|
||||
deployment/reuse-surface -n reuse
|
||||
kubectl --kubeconfig ~/.kube/config-hosteurope rollout status \
|
||||
deployment/reuse-surface -n reuse
|
||||
```
|
||||
|
||||
4. **Smoke**:
|
||||
```bash
|
||||
make -C ~/railiance-apps reuse-webhook-smoke
|
||||
export REUSE_SURFACE_URL=https://reuse.coulomb.social
|
||||
export REUSE_SURFACE_TOKEN=$(
|
||||
bao kv get -field=REUSE_SURFACE_TOKEN \
|
||||
platform/workloads/reuse/reuse-surface/runtime-secrets
|
||||
)
|
||||
reuse-surface hub status
|
||||
unset REUSE_SURFACE_TOKEN
|
||||
```
|
||||
|
||||
5. **Record** non-secret evidence:
|
||||
```bash
|
||||
cd ~/railiance-platform
|
||||
scripts/credential-change.py lifecycle-event CCR-2026-0005 --action rotate \
|
||||
--actor <operator> --reason "REUSE_SURFACE_TOKEN rotation" \
|
||||
--detail "ESO SecretSynced + hub rollout + smoke passed" --record-state-hub
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Rotate `REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET` only
|
||||
|
||||
The hub pod **and** the Forgejo org webhook (`coulomb`, hook id `1`) must share
|
||||
the same HMAC. Order matters: OpenBao → ESO → hub rollout → Forgejo webhook.
|
||||
|
||||
1. **Write new HMAC in OpenBao**:
|
||||
```bash
|
||||
NEW_HMAC="$(openssl rand -hex 32)"
|
||||
bao kv patch platform/workloads/reuse/reuse-surface/runtime-secrets \
|
||||
REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET="${NEW_HMAC}"
|
||||
unset NEW_HMAC
|
||||
```
|
||||
|
||||
2. **Force ESO refresh and roll hub** (steps 2–3 from token rotation above).
|
||||
|
||||
3. **Reconcile Forgejo org webhook** (reads HMAC from materialized cluster Secret):
|
||||
```bash
|
||||
make -C ~/railiance-apps reuse-forgejo-webhook
|
||||
```
|
||||
|
||||
4. **Smoke** (signed POST must return 200; unsigned must return 401):
|
||||
```bash
|
||||
make -C ~/railiance-apps reuse-webhook-smoke
|
||||
```
|
||||
|
||||
5. **Record** lifecycle event (same as token rotation; note webhook field in reason).
|
||||
|
||||
---
|
||||
|
||||
## Rotate both fields
|
||||
|
||||
Perform token rotation steps 1–4, then run `make reuse-forgejo-webhook` and
|
||||
`make reuse-webhook-smoke` before recording evidence. Alternatively patch both
|
||||
fields in a single `bao kv patch` and follow the webhook rotation order (ESO →
|
||||
rollout → Forgejo webhook → smoke).
|
||||
|
||||
---
|
||||
|
||||
## Verification checklist
|
||||
|
||||
| Check | Expected |
|
||||
| --- | --- |
|
||||
| `ExternalSecret reuse/reuse-surface-runtime` | `Ready=True`, `SecretSynced` |
|
||||
| Unsigned `POST /v1/webhooks/forgejo` | `401` |
|
||||
| Signed no-op push payload | `200` with `"accepted": false` |
|
||||
| `GET /v1/federated` | `200` |
|
||||
| `reuse-surface hub status` with new token | `200` on `/health` |
|
||||
|
||||
Automated: `make -C ~/railiance-apps reuse-webhook-smoke`
|
||||
|
||||
---
|
||||
|
||||
## Compromise or emergency
|
||||
|
||||
Use the CCR generator first — disable the ops-warden front door before metadata
|
||||
changes:
|
||||
|
||||
```bash
|
||||
scripts/credential-change.py lifecycle-plan CCR-2026-0005 --action compromise
|
||||
```
|
||||
|
||||
Consumer-specific steps (Forgejo webhook alignment, materialized Secret deletion)
|
||||
are in `docs/credential-lane-lifecycle-runbook.md` under CCR-2026-0005.
|
||||
|
|
@ -277,6 +277,8 @@ warden access reuse-surface-hub-write-token --fetch REUSE_SURFACE_TOKEN
|
|||
- Break-glass cluster read remains available via
|
||||
`kubectl ... get secret reuse-surface-env` when OpenBao is down; steady-state
|
||||
custody is OpenBao + ESO only.
|
||||
- Rotation procedure: `docs/reuse-surface-runtime-secrets-rotation-runbook.md`;
|
||||
post-rotate smoke: `make -C ~/railiance-apps reuse-webhook-smoke`.
|
||||
|
||||
## Railiance offsite backup lane (`CCR-2026-0004`)
|
||||
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ type: workplan
|
|||
title: "reuse-surface Runtime Secrets OpenBao Lane"
|
||||
domain: financials
|
||||
repo: railiance-platform
|
||||
status: active
|
||||
status: finished
|
||||
owner: codex
|
||||
topic_slug: railiance
|
||||
created: "2026-07-07"
|
||||
|
|
@ -130,7 +130,7 @@ documented as break-glass only. CCR-2026-0005 front door `ready`/`resolvable=tru
|
|||
|
||||
```task
|
||||
id: RAILIANCE-WP-0011-T04
|
||||
status: todo
|
||||
status: done
|
||||
priority: low
|
||||
state_hub_task_id: "eafc4011-69cf-473f-842c-c58d6a2bece7"
|
||||
```
|
||||
|
|
@ -141,9 +141,14 @@ Blocked on T03.
|
|||
`make reuse-forgejo-webhook` (updates org hook secret)
|
||||
- Add smoke: signed webhook POST returns 200/accepted or expected no-op
|
||||
|
||||
**2026-07-07:** Added `docs/reuse-surface-runtime-secrets-rotation-runbook.md`,
|
||||
CCR-2026-0005 lane in `docs/credential-lane-lifecycle-runbook.md`, and
|
||||
`railiance-apps` `make reuse-webhook-smoke` (unsigned 401, signed no-op 200,
|
||||
ESO Ready, `/v1/federated`). Live smoke passed on production.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- [ ] Both fields readable from `platform/workloads/reuse/reuse-surface/runtime-secrets`
|
||||
- [ ] ESO owns `reuse-surface-env`; manual `kubectl create secret` no longer required for steady state
|
||||
- [ ] `warden access reuse-surface-hub-write-token --fetch` uses OpenBao path
|
||||
- [ ] Forgejo org webhook and hub HMAC stay aligned after a test rotation
|
||||
- [x] Both fields readable from `platform/workloads/reuse/reuse-surface/runtime-secrets`
|
||||
- [x] ESO owns `reuse-surface-env`; manual `kubectl create secret` no longer required for steady state
|
||||
- [x] `warden access reuse-surface-hub-write-token --fetch` uses OpenBao path
|
||||
- [x] Forgejo org webhook and hub HMAC aligned (smoke verifies signed/unsigned paths; rotation runbook documents reconcile step)
|
||||
Loading…
Add table
Add a link
Reference in a new issue