# Credential Lane Lifecycle Runbook

Status: active (RAILIANCE-WP-0009-T07 / RAILIANCE-WP-0010-T07 / RAILIANCE-WP-0011-T04)
Date: 2026-07-07

Covers deactivation, rotation, and compromise response for the workload KV
lanes established by `CCR-2026-0002` (issue-core), `CCR-2026-0003`
(llm-connect), and `CCR-2026-0005` (reuse-surface). The **canonical,
always-current procedure** is generated from the CCR itself — this runbook adds
only the lane-specific consumer facts the generator cannot know.

```bash
scripts/credential-change.py lifecycle-plan <CCR-ID> --action {deactivate|rotate|compromise}
# then execute the rendered steps and record:
scripts/credential-change.py lifecycle-event <CCR-ID> --action <action> \
  --actor <operator> --reason "<non-secret>" --detail "<non-secret>" --record-state-hub
```

All three actions share the same invariants: the front door goes
non-resolvable *first*, OpenBao metadata changes use approved operator or
delegated-applier authority (never `platform-admin` handoffs), audit
evidence is preserved (never delete the audit device or its entries), and no
secret value ever appears in Git, State Hub, chat, prompts, or shell history.

## Lane: issue-core runtime ingestion (`CCR-2026-0002`)

| Item | Value |
| --- | --- |
| KV path | `platform/workloads/issue-core/issue-core/issue-core-runtime` |
| Fields | `ISSUE_CORE_API_KEY`, `GITEA_BACKEND_TOKEN` |
| Policy / auth role | `workload-kv-read-issue-core-runtime` / `auth/kubernetes/role/external-secrets-issue-core` |
| Primary consumer | ExternalSecret `issue-core/issue-core-runtime` (CoulombCore cluster, 1h refresh) |
| ops-warden catalog | `issue-core-ingestion-api-key` |

**Consumer facts the generated plan does not cover:**

- Deactivating the policy/role stops the ExternalSecret from *refreshing*,
  but the materialized Kubernetes Secret **persists** with the last value —
  a real deactivation or compromise response must also delete
  `secret/issue-core-runtime` in the `issue-core` namespace (ESO will not
  recreate it while the lane is down) and restart the issue-core Deployment.
- **`ISSUE_CORE_API_KEY` has a second consumer**: railiance01's
  `activity-core/ExternalSecret actcore-issue-core-runtime` merges the key into
  `actcore-runtime-secret` (1h refresh). Rotation and compromise response MUST
  update OpenBao first, wait for ExternalSecret refresh (or annotate
  `force-sync`), then restart `deploy/actcore-worker` and
  `deploy/actcore-event-router`, or activity-core emission silently starts
  failing with 401s on the next run.
- `GITEA_BACKEND_TOKEN` is a scoped Forgejo token for service user
  `issue-core-svc`; rotating it means minting a new token in Forgejo first,
  then updating OpenBao — order matters, or ingestion breaks between steps.

## Lane: llm-connect OpenRouter provider key (`CCR-2026-0003`)

| Item | Value |
| --- | --- |
| KV path | `platform/workloads/activity-core/llm-connect/llm-connect-provider-secrets` |
| Field | `OPENROUTER_API_KEY` |
| Policy / auth role | `workload-kv-read-llm-connect-provider-secrets` / `auth/kubernetes/role/external-secrets-activity-core` |
| Primary consumer | ExternalSecret `activity-core/llm-connect-provider-secrets` (CoulombCore cluster, 1h refresh) |
| ops-warden catalog | `openrouter-llm-connect` |

**Consumer facts the generated plan does not cover:**

- llm-connect consumes the Secret via `envFrom`, so a rotated value reaches
  the runtime only after `kubectl -n activity-core rollout restart
  deploy/llm-connect` (CoulombCore). Wait for the ExternalSecret refresh (or
  `force-sync` annotate) *before* restarting.
- **The railiance01 llm-connect instance is out of scope of this lane**: it
  uses a bootstrap-provisioned Secret from
  `activity-core/k8s/railiance/bootstrap-secrets.sh`. Rotating the OpenRouter
  key upstream (at OpenRouter) invalidates *both* copies — a provider-side
  rotation therefore always requires the railiance01 manual update too, or
  the daily triage runs start failing with provider auth errors.
- Compromise response for a provider key has an extra step the plan cannot
  render: **revoke the key at OpenRouter itself** (provider console) before
  or immediately after disabling the front door; OpenBao custody actions
  alone do not stop a leaked provider key from working.

## Lane: reuse-surface hub runtime secrets (`CCR-2026-0005`)

| Item | Value |
| --- | --- |
| KV path | `platform/workloads/reuse/reuse-surface/runtime-secrets` |
| Fields | `REUSE_SURFACE_TOKEN`, `REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET` |
| Policy / auth role | `workload-kv-read-reuse-surface-runtime` / `auth/kubernetes/role/external-secrets-reuse-surface` |
| Primary consumer | ExternalSecret `reuse/reuse-surface-runtime` → `reuse-surface-env` (Railiance01, 1h refresh) |
| ops-warden catalog | `reuse-surface-hub-write-token` |
| Rotation runbook | `docs/reuse-surface-runtime-secrets-rotation-runbook.md` |

**Consumer facts the generated plan does not cover:**

- The hub Deployment uses `envFrom.secretRef`; rotated values reach the pod only
  after ESO refresh **and** `kubectl rollout restart deployment/reuse-surface -n reuse`.
- **`REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET` is dual-consumer**: the hub pod and the
  Forgejo org webhook on `coulomb` must share the same HMAC. After rotating that
  field, run `railiance-apps` `make reuse-forgejo-webhook` once ESO has synced —
  otherwise Forgejo deliveries fail with 401 while the hub still expects the new
  value (or vice versa if order is reversed).
- Deactivating the policy/role stops ESO refresh, but `secret/reuse-surface-env`
  retains the last materialized value until explicitly deleted.
- Steady-state operator fetch is `bao kv get` / `warden access`; kubectl read of
  `reuse-surface-env` is break-glass only.

**Smoke after rotate:** `make -C ~/railiance-apps reuse-webhook-smoke`

## Verification after rotate

Return the lane to `active` only with fresh positive + negative evidence,
same shape as activation (2026-07-02 precedent):

- positive: ExternalSecret `SecretSynced=True` with a new refresh timestamp,
  consumer pod healthy after restart;
- negative: a `default`-policy token denied on the KV data path, matched in
  the file audit device by path and timestamp;
- record via `lifecycle-event ... --record-state-hub` and notify ops-warden
  to flip the catalog entry back to active.
