0ae7cdab92
Gitea has been replaced by Forgejo. Updates prose mentions in docs/ while deliberately preserving live names that a blind sweep would have broken: GITEA_BACKEND_TOKEN is a provisioned OpenBao field in CCR-2026-0002, GITEA_URL is a Makefile variable, and gitea-db is a running cnpg cluster. Capitalisation discriminates prose from identifiers. Archived workplans are left as historical record. Blueprint 5.2: forge placement decided - workload to rapp-forgejo, layer responsibility to S4 railiance-enablement, which already declares the handoff contract and gains its first concrete owned responsibility. Blueprint 5.6: whether Railiance should use ArgoCD at all is recorded as an open decision rather than acted on by removing mentions. S3 runs four live ArgoCD Applications plus AppProjects, and the two most recent commits here add more, so deleting the documentation would describe a system that does not exist. The question is real; it needs a decision and a migration, not a docs edit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
117 lines
6.2 KiB
Markdown
117 lines
6.2 KiB
Markdown
# Credential Lane Lifecycle Runbook
|
|
|
|
Status: active (RAILIANCE-WP-0009-T07 / RAILIANCE-WP-0010-T07 / RAILIANCE-WP-0011-T04)
|
|
Date: 2026-07-07
|
|
|
|
Covers deactivation, rotation, and compromise response for the workload KV
|
|
lanes established by `CCR-2026-0002` (issue-core), `CCR-2026-0003`
|
|
(llm-connect), and `CCR-2026-0005` (reuse-surface). The **canonical,
|
|
always-current procedure** is generated from the CCR itself — this runbook adds
|
|
only the lane-specific consumer facts the generator cannot know.
|
|
|
|
```bash
|
|
scripts/credential-change.py lifecycle-plan <CCR-ID> --action {deactivate|rotate|compromise}
|
|
# then execute the rendered steps and record:
|
|
scripts/credential-change.py lifecycle-event <CCR-ID> --action <action> \
|
|
--actor <operator> --reason "<non-secret>" --detail "<non-secret>" --record-state-hub
|
|
```
|
|
|
|
All three actions share the same invariants: the front door goes
|
|
non-resolvable *first*, OpenBao metadata changes use approved operator or
|
|
delegated-applier authority (never `platform-admin` handoffs), audit
|
|
evidence is preserved (never delete the audit device or its entries), and no
|
|
secret value ever appears in Git, State Hub, chat, prompts, or shell history.
|
|
|
|
## Lane: issue-core runtime ingestion (`CCR-2026-0002`)
|
|
|
|
| Item | Value |
|
|
| --- | --- |
|
|
| KV path | `platform/workloads/issue-core/issue-core/issue-core-runtime` |
|
|
| Fields | `ISSUE_CORE_API_KEY`, `GITEA_BACKEND_TOKEN` |
|
|
| Policy / auth role | `workload-kv-read-issue-core-runtime` / `auth/kubernetes/role/external-secrets-issue-core` |
|
|
| Primary consumer | ExternalSecret `issue-core/issue-core-runtime` (CoulombCore cluster, 1h refresh) |
|
|
| ops-warden catalog | `issue-core-ingestion-api-key` |
|
|
|
|
**Consumer facts the generated plan does not cover:**
|
|
|
|
- Deactivating the policy/role stops the ExternalSecret from *refreshing*,
|
|
but the materialized Kubernetes Secret **persists** with the last value —
|
|
a real deactivation or compromise response must also delete
|
|
`secret/issue-core-runtime` in the `issue-core` namespace (ESO will not
|
|
recreate it while the lane is down) and restart the issue-core Deployment.
|
|
- **`ISSUE_CORE_API_KEY` has a second consumer**: railiance01's
|
|
`activity-core/ExternalSecret actcore-issue-core-runtime` merges the key into
|
|
`actcore-runtime-secret` (1h refresh). Rotation and compromise response MUST
|
|
update OpenBao first, wait for ExternalSecret refresh (or annotate
|
|
`force-sync`), then restart `deploy/actcore-worker` and
|
|
`deploy/actcore-event-router`, or activity-core emission silently starts
|
|
failing with 401s on the next run.
|
|
- `GITEA_BACKEND_TOKEN` is a scoped Forgejo token for service user
|
|
`issue-core-svc`; rotating it means minting a new token in Forgejo first,
|
|
then updating OpenBao — order matters, or ingestion breaks between steps.
|
|
|
|
## Lane: llm-connect OpenRouter provider key (`CCR-2026-0003`)
|
|
|
|
| Item | Value |
|
|
| --- | --- |
|
|
| KV path | `platform/workloads/activity-core/llm-connect/llm-connect-provider-secrets` |
|
|
| Field | `OPENROUTER_API_KEY` |
|
|
| Policy / auth role | `workload-kv-read-llm-connect-provider-secrets` / `auth/kubernetes/role/external-secrets-activity-core` |
|
|
| Primary consumer | ExternalSecret `activity-core/llm-connect-provider-secrets` (CoulombCore cluster, 1h refresh) |
|
|
| ops-warden catalog | `openrouter-llm-connect` |
|
|
|
|
**Consumer facts the generated plan does not cover:**
|
|
|
|
- llm-connect consumes the Secret via `envFrom`, so a rotated value reaches
|
|
the runtime only after `kubectl -n activity-core rollout restart
|
|
deploy/llm-connect` (CoulombCore). Wait for the ExternalSecret refresh (or
|
|
`force-sync` annotate) *before* restarting.
|
|
- **The railiance01 llm-connect instance is out of scope of this lane**: it
|
|
uses a bootstrap-provisioned Secret from
|
|
`activity-core/k8s/railiance/bootstrap-secrets.sh`. Rotating the OpenRouter
|
|
key upstream (at OpenRouter) invalidates *both* copies — a provider-side
|
|
rotation therefore always requires the railiance01 manual update too, or
|
|
the daily triage runs start failing with provider auth errors.
|
|
- Compromise response for a provider key has an extra step the plan cannot
|
|
render: **revoke the key at OpenRouter itself** (provider console) before
|
|
or immediately after disabling the front door; OpenBao custody actions
|
|
alone do not stop a leaked provider key from working.
|
|
|
|
## Lane: reuse-surface hub runtime secrets (`CCR-2026-0005`)
|
|
|
|
| Item | Value |
|
|
| --- | --- |
|
|
| KV path | `platform/workloads/reuse/reuse-surface/runtime-secrets` |
|
|
| Fields | `REUSE_SURFACE_TOKEN`, `REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET` |
|
|
| Policy / auth role | `workload-kv-read-reuse-surface-runtime` / `auth/kubernetes/role/external-secrets-reuse-surface` |
|
|
| Primary consumer | ExternalSecret `reuse/reuse-surface-runtime` → `reuse-surface-env` (Railiance01, 1h refresh) |
|
|
| ops-warden catalog | `reuse-surface-hub-write-token` |
|
|
| Rotation runbook | `docs/reuse-surface-runtime-secrets-rotation-runbook.md` |
|
|
|
|
**Consumer facts the generated plan does not cover:**
|
|
|
|
- The hub Deployment uses `envFrom.secretRef`; rotated values reach the pod only
|
|
after ESO refresh **and** `kubectl rollout restart deployment/reuse-surface -n reuse`.
|
|
- **`REUSE_SURFACE_FORGEJO_WEBHOOK_SECRET` is dual-consumer**: the hub pod and the
|
|
Forgejo org webhook on `coulomb` must share the same HMAC. After rotating that
|
|
field, run `railiance-apps` `make reuse-forgejo-webhook` once ESO has synced —
|
|
otherwise Forgejo deliveries fail with 401 while the hub still expects the new
|
|
value (or vice versa if order is reversed).
|
|
- Deactivating the policy/role stops ESO refresh, but `secret/reuse-surface-env`
|
|
retains the last materialized value until explicitly deleted.
|
|
- Steady-state operator fetch is `bao kv get` / `warden access`; kubectl read of
|
|
`reuse-surface-env` is break-glass only.
|
|
|
|
**Smoke after rotate:** `make -C ~/railiance-apps reuse-webhook-smoke`
|
|
|
|
## Verification after rotate
|
|
|
|
Return the lane to `active` only with fresh positive + negative evidence,
|
|
same shape as activation (2026-07-02 precedent):
|
|
|
|
- positive: ExternalSecret `SecretSynced=True` with a new refresh timestamp,
|
|
consumer pod healthy after restart;
|
|
- negative: a `default`-policy token denied on the KV data path, matched in
|
|
the file audit device by path and timestamp;
|
|
- record via `lifecycle-event ... --record-state-hub` and notify ops-warden
|
|
to flip the catalog entry back to active.
|