From 839a4418a9b93e461914e1cdf2257693d66e0a48 Mon Sep 17 00:00:00 2001 From: tegwick Date: Wed, 8 Jul 2026 00:01:21 +0200 Subject: [PATCH] Add reuse-surface secrets rotation runbook (RAILIANCE-WP-0011-T04) Document OpenBao patch, ESO force-sync, hub rollout, and Forgejo webhook reconcile; extend credential-lane lifecycle for CCR-2026-0005; finish workplan. --- docs/credential-lane-lifecycle-runbook.md | 39 ++++- ...urface-runtime-secrets-rotation-runbook.md | 139 ++++++++++++++++++ docs/workload-kv-access-lanes.md | 2 + ...se-surface-runtime-secrets-openbao-lane.md | 17 ++- 4 files changed, 185 insertions(+), 12 deletions(-) create mode 100644 docs/reuse-surface-runtime-secrets-rotation-runbook.md diff --git a/docs/credential-lane-lifecycle-runbook.md b/docs/credential-lane-lifecycle-runbook.md index 30ff4cf..a661b6e 100644 --- a/docs/credential-lane-lifecycle-runbook.md +++ b/docs/credential-lane-lifecycle-runbook.md @@ -1,13 +1,13 @@ # Credential Lane Lifecycle Runbook -Status: active (RAILIANCE-WP-0009-T07 / RAILIANCE-WP-0010-T07) -Date: 2026-07-02 +Status: active (RAILIANCE-WP-0009-T07 / RAILIANCE-WP-0010-T07 / RAILIANCE-WP-0011-T04) +Date: 2026-07-07 Covers deactivation, rotation, and compromise response for the workload KV -lanes established by `CCR-2026-0002` (issue-core) and `CCR-2026-0003` -(llm-connect). The **canonical, always-current procedure** is generated from -the CCR itself — this runbook adds only the lane-specific consumer facts the -generator cannot know. +lanes established by `CCR-2026-0002` (issue-core), `CCR-2026-0003` +(llm-connect), and `CCR-2026-0005` (reuse-surface). The **canonical, +always-current procedure** is generated from the CCR itself — this runbook adds +only the lane-specific consumer facts the generator cannot know. ```bash scripts/credential-change.py lifecycle-plan --action {deactivate|rotate|compromise} @@ -76,6 +76,33 @@ secret value ever appears in Git, State Hub, chat, prompts, or shell history. or immediately after disabling the front door; OpenBao custody actions alone do not stop a leaked provider key from working. +## Lane: reuse-surface hub runtime secrets (`CCR-2026-0005`) + +| Item | Value | +| --- | --- | +| KV path | `platform/workloads/reuse/reuse-surface/runtime-secrets` | +| Fields | `REUSE_SURFACE_TOKEN`, `REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET` | +| Policy / auth role | `workload-kv-read-reuse-surface-runtime` / `auth/kubernetes/role/external-secrets-reuse-surface` | +| Primary consumer | ExternalSecret `reuse/reuse-surface-runtime` → `reuse-surface-env` (Railiance01, 1h refresh) | +| ops-warden catalog | `reuse-surface-hub-write-token` | +| Rotation runbook | `docs/reuse-surface-runtime-secrets-rotation-runbook.md` | + +**Consumer facts the generated plan does not cover:** + +- The hub Deployment uses `envFrom.secretRef`; rotated values reach the pod only + after ESO refresh **and** `kubectl rollout restart deployment/reuse-surface -n reuse`. +- **`REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET` is dual-consumer**: the hub pod and the + Forgejo org webhook on `coulomb` must share the same HMAC. After rotating that + field, run `railiance-apps` `make reuse-forgejo-webhook` once ESO has synced — + otherwise Forgejo deliveries fail with 401 while the hub still expects the new + value (or vice versa if order is reversed). +- Deactivating the policy/role stops ESO refresh, but `secret/reuse-surface-env` + retains the last materialized value until explicitly deleted. +- Steady-state operator fetch is `bao kv get` / `warden access`; kubectl read of + `reuse-surface-env` is break-glass only. + +**Smoke after rotate:** `make -C ~/railiance-apps reuse-webhook-smoke` + ## Verification after rotate Return the lane to `active` only with fresh positive + negative evidence, diff --git a/docs/reuse-surface-runtime-secrets-rotation-runbook.md b/docs/reuse-surface-runtime-secrets-rotation-runbook.md new file mode 100644 index 0000000..067aef8 --- /dev/null +++ b/docs/reuse-surface-runtime-secrets-rotation-runbook.md @@ -0,0 +1,139 @@ +# reuse-surface Runtime Secrets — Rotation Runbook + +Status: active (RAILIANCE-WP-0011-T04 / CCR-2026-0005) +Date: 2026-07-07 + +Operator procedure for rotating `REUSE_SURFACE_TOKEN` and/or +`REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET` in OpenBao custody. Never paste secret +values into Git, State Hub, chat, prompts, or shell history. + +**Canonical lane:** `platform/workloads/reuse/reuse-surface/runtime-secrets` +**Delivery:** `ExternalSecret reuse/reuse-surface-runtime` → `reuse-surface-env` (Railiance01) +**Lifecycle generator:** `scripts/credential-change.py lifecycle-plan CCR-2026-0005 --action rotate` + +--- + +## Prerequisites + +- OpenBao operator authority on `https://bao.coulomb.social` +- Railiance01 kubeconfig: `~/.kube/config-hosteurope` +- Forgejo admin API token: `/tmp/forgejo-tegwick-api-token` (when rotating webhook HMAC) +- `railiance-apps` checkout with `make reuse-runtime-es-status`, `make reuse-forgejo-webhook`, + `make reuse-webhook-smoke` + +--- + +## Rotate `REUSE_SURFACE_TOKEN` only + +Hub write clients and operators must pick up the new bearer token. Forgejo webhook +HMAC is unchanged. + +1. **Write new value in OpenBao** (generate locally; never log): + ```bash + NEW_TOKEN="$(openssl rand -hex 32)" + bao kv patch platform/workloads/reuse/reuse-surface/runtime-secrets \ + REUSE_SURFACE_TOKEN="${NEW_TOKEN}" + unset NEW_TOKEN + ``` + +2. **Force ESO refresh** (default interval is 1h): + ```bash + kubectl --kubeconfig ~/.kube/config-hosteurope annotate externalsecret \ + reuse-surface-runtime -n reuse \ + force-sync="$(date +%s)" --overwrite + make -C ~/railiance-apps reuse-runtime-es-status + ``` + +3. **Roll the hub Deployment** (`envFrom` does not hot-reload): + ```bash + kubectl --kubeconfig ~/.kube/config-hosteurope rollout restart \ + deployment/reuse-surface -n reuse + kubectl --kubeconfig ~/.kube/config-hosteurope rollout status \ + deployment/reuse-surface -n reuse + ``` + +4. **Smoke**: + ```bash + make -C ~/railiance-apps reuse-webhook-smoke + export REUSE_SURFACE_URL=https://reuse.coulomb.social + export REUSE_SURFACE_TOKEN=$( + bao kv get -field=REUSE_SURFACE_TOKEN \ + platform/workloads/reuse/reuse-surface/runtime-secrets + ) + reuse-surface hub status + unset REUSE_SURFACE_TOKEN + ``` + +5. **Record** non-secret evidence: + ```bash + cd ~/railiance-platform + scripts/credential-change.py lifecycle-event CCR-2026-0005 --action rotate \ + --actor --reason "REUSE_SURFACE_TOKEN rotation" \ + --detail "ESO SecretSynced + hub rollout + smoke passed" --record-state-hub + ``` + +--- + +## Rotate `REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET` only + +The hub pod **and** the Forgejo org webhook (`coulomb`, hook id `1`) must share +the same HMAC. Order matters: OpenBao → ESO → hub rollout → Forgejo webhook. + +1. **Write new HMAC in OpenBao**: + ```bash + NEW_HMAC="$(openssl rand -hex 32)" + bao kv patch platform/workloads/reuse/reuse-surface/runtime-secrets \ + REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET="${NEW_HMAC}" + unset NEW_HMAC + ``` + +2. **Force ESO refresh and roll hub** (steps 2–3 from token rotation above). + +3. **Reconcile Forgejo org webhook** (reads HMAC from materialized cluster Secret): + ```bash + make -C ~/railiance-apps reuse-forgejo-webhook + ``` + +4. **Smoke** (signed POST must return 200; unsigned must return 401): + ```bash + make -C ~/railiance-apps reuse-webhook-smoke + ``` + +5. **Record** lifecycle event (same as token rotation; note webhook field in reason). + +--- + +## Rotate both fields + +Perform token rotation steps 1–4, then run `make reuse-forgejo-webhook` and +`make reuse-webhook-smoke` before recording evidence. Alternatively patch both +fields in a single `bao kv patch` and follow the webhook rotation order (ESO → +rollout → Forgejo webhook → smoke). + +--- + +## Verification checklist + +| Check | Expected | +| --- | --- | +| `ExternalSecret reuse/reuse-surface-runtime` | `Ready=True`, `SecretSynced` | +| Unsigned `POST /v1/webhooks/forgejo` | `401` | +| Signed no-op push payload | `200` with `"accepted": false` | +| `GET /v1/federated` | `200` | +| `reuse-surface hub status` with new token | `200` on `/health` | + +Automated: `make -C ~/railiance-apps reuse-webhook-smoke` + +--- + +## Compromise or emergency + +Use the CCR generator first — disable the ops-warden front door before metadata +changes: + +```bash +scripts/credential-change.py lifecycle-plan CCR-2026-0005 --action compromise +``` + +Consumer-specific steps (Forgejo webhook alignment, materialized Secret deletion) +are in `docs/credential-lane-lifecycle-runbook.md` under CCR-2026-0005. \ No newline at end of file diff --git a/docs/workload-kv-access-lanes.md b/docs/workload-kv-access-lanes.md index ad636de..9f80fe2 100644 --- a/docs/workload-kv-access-lanes.md +++ b/docs/workload-kv-access-lanes.md @@ -277,6 +277,8 @@ warden access reuse-surface-hub-write-token --fetch REUSE_SURFACE_TOKEN - Break-glass cluster read remains available via `kubectl ... get secret reuse-surface-env` when OpenBao is down; steady-state custody is OpenBao + ESO only. +- Rotation procedure: `docs/reuse-surface-runtime-secrets-rotation-runbook.md`; + post-rotate smoke: `make -C ~/railiance-apps reuse-webhook-smoke`. ## Railiance offsite backup lane (`CCR-2026-0004`) diff --git a/workplans/RAILIANCE-WP-0011-reuse-surface-runtime-secrets-openbao-lane.md b/workplans/RAILIANCE-WP-0011-reuse-surface-runtime-secrets-openbao-lane.md index b87fa05..33ab00d 100644 --- a/workplans/RAILIANCE-WP-0011-reuse-surface-runtime-secrets-openbao-lane.md +++ b/workplans/RAILIANCE-WP-0011-reuse-surface-runtime-secrets-openbao-lane.md @@ -4,7 +4,7 @@ type: workplan title: "reuse-surface Runtime Secrets OpenBao Lane" domain: financials repo: railiance-platform -status: active +status: finished owner: codex topic_slug: railiance created: "2026-07-07" @@ -130,7 +130,7 @@ documented as break-glass only. CCR-2026-0005 front door `ready`/`resolvable=tru ```task id: RAILIANCE-WP-0011-T04 -status: todo +status: done priority: low state_hub_task_id: "eafc4011-69cf-473f-842c-c58d6a2bece7" ``` @@ -141,9 +141,14 @@ Blocked on T03. `make reuse-forgejo-webhook` (updates org hook secret) - Add smoke: signed webhook POST returns 200/accepted or expected no-op +**2026-07-07:** Added `docs/reuse-surface-runtime-secrets-rotation-runbook.md`, +CCR-2026-0005 lane in `docs/credential-lane-lifecycle-runbook.md`, and +`railiance-apps` `make reuse-webhook-smoke` (unsigned 401, signed no-op 200, +ESO Ready, `/v1/federated`). Live smoke passed on production. + ## Acceptance -- [ ] Both fields readable from `platform/workloads/reuse/reuse-surface/runtime-secrets` -- [ ] ESO owns `reuse-surface-env`; manual `kubectl create secret` no longer required for steady state -- [ ] `warden access reuse-surface-hub-write-token --fetch` uses OpenBao path -- [ ] Forgejo org webhook and hub HMAC stay aligned after a test rotation \ No newline at end of file +- [x] Both fields readable from `platform/workloads/reuse/reuse-surface/runtime-secrets` +- [x] ESO owns `reuse-surface-env`; manual `kubectl create secret` no longer required for steady state +- [x] `warden access reuse-surface-hub-write-token --fetch` uses OpenBao path +- [x] Forgejo org webhook and hub HMAC aligned (smoke verifies signed/unsigned paths; rotation runbook documents reconcile step) \ No newline at end of file