railiance-platform/docs/credential-lane-lifecycle-runbook.md

Credential Lane Lifecycle Runbook

Status: active (RAILIANCE-WP-0009-T07 / RAILIANCE-WP-0010-T07 / RAILIANCE-WP-0011-T04) Date: 2026-07-07

Covers deactivation, rotation, and compromise response for the workload KV lanes established by CCR-2026-0002 (issue-core), CCR-2026-0003 (llm-connect), and CCR-2026-0005 (reuse-surface). The canonical, always-current procedure is generated from the CCR itself — this runbook adds only the lane-specific consumer facts the generator cannot know.

scripts/credential-change.py lifecycle-plan <CCR-ID> --action {deactivate|rotate|compromise}
# then execute the rendered steps and record:
scripts/credential-change.py lifecycle-event <CCR-ID> --action <action> \
  --actor <operator> --reason "<non-secret>" --detail "<non-secret>" --record-state-hub

All three actions share the same invariants: the front door goes non-resolvable first, OpenBao metadata changes use approved operator or delegated-applier authority (never platform-admin handoffs), audit evidence is preserved (never delete the audit device or its entries), and no secret value ever appears in Git, State Hub, chat, prompts, or shell history.

Lane: issue-core runtime ingestion (CCR-2026-0002)

Item	Value
KV path	platform/workloads/issue-core/issue-core/issue-core-runtime
Fields	ISSUE_CORE_API_KEY, GITEA_BACKEND_TOKEN
Policy / auth role	workload-kv-read-issue-core-runtime / auth/kubernetes/role/external-secrets-issue-core
Primary consumer	ExternalSecret issue-core/issue-core-runtime (CoulombCore cluster, 1h refresh)
ops-warden catalog	issue-core-ingestion-api-key

Consumer facts the generated plan does not cover:

• Deactivating the policy/role stops the ExternalSecret from refreshing, but the materialized Kubernetes Secret persists with the last value — a real deactivation or compromise response must also delete secret/issue-core-runtime in the issue-core namespace (ESO will not recreate it while the lane is down) and restart the issue-core Deployment.
• ISSUE_CORE_API_KEY has a second consumer: railiance01's activity-core/ExternalSecret actcore-issue-core-runtime merges the key into actcore-runtime-secret (1h refresh). Rotation and compromise response MUST update OpenBao first, wait for ExternalSecret refresh (or annotate force-sync), then restart deploy/actcore-worker and deploy/actcore-event-router, or activity-core emission silently starts failing with 401s on the next run.
• GITEA_BACKEND_TOKEN is a scoped Forgejo token for service user issue-core-svc; rotating it means minting a new token in Forgejo first, then updating OpenBao — order matters, or ingestion breaks between steps.

Lane: llm-connect OpenRouter provider key (CCR-2026-0003)

Item	Value
KV path	platform/workloads/activity-core/llm-connect/llm-connect-provider-secrets
Field	OPENROUTER_API_KEY
Policy / auth role	workload-kv-read-llm-connect-provider-secrets / auth/kubernetes/role/external-secrets-activity-core
Primary consumer	ExternalSecret activity-core/llm-connect-provider-secrets (CoulombCore cluster, 1h refresh)
ops-warden catalog	openrouter-llm-connect

Consumer facts the generated plan does not cover:

• llm-connect consumes the Secret via envFrom, so a rotated value reaches the runtime only after kubectl -n activity-core rollout restart deploy/llm-connect (CoulombCore). Wait for the ExternalSecret refresh (or force-sync annotate) before restarting.
• The railiance01 llm-connect instance is out of scope of this lane: it uses a bootstrap-provisioned Secret from activity-core/k8s/railiance/bootstrap-secrets.sh. Rotating the OpenRouter key upstream (at OpenRouter) invalidates both copies — a provider-side rotation therefore always requires the railiance01 manual update too, or the daily triage runs start failing with provider auth errors.
• Compromise response for a provider key has an extra step the plan cannot render: revoke the key at OpenRouter itself (provider console) before or immediately after disabling the front door; OpenBao custody actions alone do not stop a leaked provider key from working.

Lane: reuse-surface hub runtime secrets (CCR-2026-0005)

Item	Value
KV path	platform/workloads/reuse/reuse-surface/runtime-secrets
Fields	REUSE_SURFACE_TOKEN, REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET
Policy / auth role	workload-kv-read-reuse-surface-runtime / auth/kubernetes/role/external-secrets-reuse-surface
Primary consumer	ExternalSecret reuse/reuse-surface-runtime → reuse-surface-env (Railiance01, 1h refresh)
ops-warden catalog	reuse-surface-hub-write-token
Rotation runbook	docs/reuse-surface-runtime-secrets-rotation-runbook.md

Consumer facts the generated plan does not cover:

• The hub Deployment uses envFrom.secretRef; rotated values reach the pod only after ESO refresh and kubectl rollout restart deployment/reuse-surface -n reuse.
• REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET is dual-consumer: the hub pod and the Forgejo org webhook on coulomb must share the same HMAC. After rotating that field, run railiance-apps make reuse-forgejo-webhook once ESO has synced — otherwise Forgejo deliveries fail with 401 while the hub still expects the new value (or vice versa if order is reversed).
• Deactivating the policy/role stops ESO refresh, but secret/reuse-surface-env retains the last materialized value until explicitly deleted.
• Steady-state operator fetch is bao kv get / warden access; kubectl read of reuse-surface-env is break-glass only.

Smoke after rotate: make -C ~/railiance-apps reuse-webhook-smoke

Verification after rotate

Return the lane to active only with fresh positive + negative evidence, same shape as activation (2026-07-02 precedent):

• positive: ExternalSecret SecretSynced=True with a new refresh timestamp, consumer pod healthy after restart;
• negative: a default-policy token denied on the KV data path, matched in the file audit device by path and timestamp;
• record via lifecycle-event ... --record-state-hub and notify ops-warden to flip the catalog entry back to active.