--- id: RAILIANCE-WP-0010 type: workplan title: "llm-connect OpenRouter Provider Key Lane" domain: financials repo: railiance-platform status: finished owner: codex topic_slug: railiance planning_priority: high planning_order: 10 created: "2026-06-29" updated: "2026-07-02" depends_on_workplans: - RAIL-PL-WP-0002 - RAILIANCE-WP-0004 - RAILIANCE-WP-0007 - RAILIANCE-WP-0008 related_state_hub_messages: - "f76d3a9e-a98f-4081-885d-b79d94312699" related_ccrs: - CCR-2026-0003 state_hub_workstream_id: "f364d405-a85d-4b89-b600-1964ab436cad" --- # RAILIANCE-WP-0010 - llm-connect OpenRouter Provider Key Lane ## Goal Promote the draft llm-connect OpenRouter provider-key access lane from a proposed CCR to a reviewed, least-privilege, verified OpenBao workload KV lane that the live llm-connect runtime can consume through External Secrets and that `ops-warden` can reference without holding the provider key. This keeps provider credential custody in the shared platform layer while leaving llm-connect behavior, model routing, and provider-specific runtime logic with the owning application/service. No task in this workplan may paste, commit, log, or send secret values through Git, State Hub, chat, prompts, shell history, or workplan text. ## Suggestion Reviewed Ops-warden confirmed the whynot-design lane in State Hub message `f76d3a9e-a98f-4081-885d-b79d94312699` and noted that the OpenRouter/llm-connect sibling lane remains draft on its side. The repo already has the proposed non-secret CCR: - `credential-change-requests/CCR-2026-0003-llm-connect-openrouter-api-key.yaml` Ops-warden already uses `openrouter-llm-connect` as the canonical draft catalog id in its routing catalog and playbook. This workplan and `CCR-2026-0003` now align to that selector so automated callers have one stable route name before activation. ## INTENT Fit This work belongs in railiance-platform because it provides the dependable secret custody and delivery substrate for a shared runtime service. It does not decide which model or provider llm-connect should use. The platform-owned output is the OpenBao path, read policy, Kubernetes auth role, External Secrets target, verification evidence, and ops-warden handoff. The plan supports these `INTENT.md` principles: - secure custody: the provider key stays in OpenBao/operator custody; - stable interfaces: llm-connect consumes a documented path, field, role, and External Secrets target; - operable and observable: activation requires positive and negative checks plus non-secret audit evidence; - independently evolvable: the provider key storage and front-door routing can change without forcing runtime consumers to know internal topology. ## Proposed Contract | Item | Proposed value | | --- | --- | | CCR | `CCR-2026-0003` | | ops-warden catalog id | `openrouter-llm-connect` | | Tenant/org | `activity-core` | | Workload/project | `llm-connect` | | KV mount | `platform` | | OpenBao CLI path | `platform/workloads/activity-core/llm-connect/llm-connect-provider-secrets` | | Secret field | `OPENROUTER_API_KEY` | | Read policy | `workload-kv-read-llm-connect-provider-secrets` | | Policy file | `openbao/policies/workload-kv-read-llm-connect-provider-secrets.hcl` | | Auth method | Kubernetes auth | | Auth role | `external-secrets-activity-core` | | OpenBao auth service account | `external-secrets` | | OpenBao auth namespace | `external-secrets` | | Delivery surface | Future activity-core ExternalSecret to Secret `llm-connect-provider-secrets` | | ops-warden command | `warden access openrouter-llm-connect --fetch OPENROUTER_API_KEY` | ## Tasks ## T01 - Review CCR scope and selector naming ```task id: RAILIANCE-WP-0010-T01 status: done priority: high state_hub_task_id: "307b75a6-a3a8-473b-b171-7379d2848698" ``` Review `CCR-2026-0003` with the platform operator and activity-core owner before any live OpenBao apply. Acceptance: - The activity-core owner confirms that llm-connect should receive `OPENROUTER_API_KEY` through this platform lane. - ops-warden and railiance-platform agree on one stable catalog id/selector: `openrouter-llm-connect`. - Review comments and approval state are recorded in the CCR without secret values. - The lane remains clearly platform-owned secret custody, not llm-connect model routing or provider selection logic. **2026-06-30:** Confirmed `activity-core` namespace exists and Kubernetes Secret `activity-core/llm-connect-provider-secrets` exists, but no activity-core `ExternalSecret` exists yet. Kept canonical CCR catalog id `llm-connect-openrouter-api-key`; ops-warden previously mentioned `openrouter-llm-connect`, so selector agreement remains open and this task stays `progress`. OpenBao public seal status now reports `sealed=false`; the prior sealed message is no longer the active blocker. **2026-07-01:** Resolved the selector naming ambiguity in favor of ops-warden canon. The local ops-warden routing catalog and playbook define `openrouter-llm-connect` as the draft OpenRouter/llm-connect route, so `CCR-2026-0003` and this workplan now use that catalog id and command shape. T01 remains `progress` until activity-core/platform approval moves the CCR out of `proposed`. ## T02 - Confirm Kubernetes auth and External Secrets binding ```task id: RAILIANCE-WP-0010-T02 status: done priority: high state_hub_task_id: "829192f5-4502-44e0-8020-656d74d5282a" ``` Confirm the exact Kubernetes service account, namespace, and External Secrets target that should consume this lane. Acceptance: - The service account and namespace are confirmed as `llm-connect`/`activity-core` or the CCR is updated with the approved alternative. - The auth role binds only the approved service account and namespace. - The External Secrets target is confirmed as `llm-connect-provider-secrets` or updated with the approved alternative. - No direct human or agent read path is activated unless separately approved. **2026-06-30:** Confirmed the proposed `llm-connect` service account does not exist and the current `llm-connect` Deployment uses the namespace `default` service account. Updated `CCR-2026-0003` to the approved platform ESO pattern: OpenBao Kubernetes auth role `external-secrets-activity-core` bound to `external-secrets/external-secrets`. Added `argocd/platform-addons/openbao-secretstore/openbao-activity-core.clustersecretstore.yaml`, limited to the `activity-core` namespace, and Make target `openbao-configure-external-secrets-activity-core` for the matching OpenBao role/policy apply. `kubectl kustomize argocd/platform-addons/openbao-secretstore` renders both the existing issue-core store and the new activity-core store. `credential-change.py applier-dry-run CCR-2026-0003` now blocks only because the CCR is still `proposed`. ## T03 - Apply or confirm least-privilege OpenBao metadata ```task id: RAILIANCE-WP-0010-T03 status: done priority: high state_hub_task_id: "42796ef5-c4a0-45a7-ae41-0ebdeccdb01d" ``` Apply the read policy and Kubernetes auth role only after review and binding confirmation. Acceptance: - `openbao/policies/workload-kv-read-llm-connect-provider-secrets.hcl` grants only read access to the exact KV-v2 data and metadata paths. - The Kubernetes auth role attaches only `workload-kv-read-llm-connect-provider-secrets`. - Live apply uses an approved operator path or the delegated applier from `RAILIANCE-WP-0008`; broad `platform-admin` handoffs are avoided where possible. - Apply evidence records only policy name, role name, actor, timestamp, and non-secret OpenBao request ids. ## T04 - Provision the provider key through approved custody ```task id: RAILIANCE-WP-0010-T04 status: done priority: high state_hub_task_id: "651f6ec8-b7d6-45e6-9fef-08646ff737c2" ``` Have an approved operator create or confirm the OpenBao KV entry and `OPENROUTER_API_KEY` field. Acceptance: - The path exists at `platform/workloads/activity-core/llm-connect/llm-connect-provider-secrets`. - Field `OPENROUTER_API_KEY` is present. - The value is entered directly through OpenBao/operator custody, never through Git, State Hub, chat, prompts, workplans, or shell history. - Non-secret evidence records only path, field name, actor, timestamp, and verification result. ## T05 - Verify positive and negative access ```task id: RAILIANCE-WP-0010-T05 status: done priority: high state_hub_task_id: "d538cfc0-bf68-4889-a5b3-ed94c1679856" ``` Prove that the approved llm-connect identity can consume the lane and that other identities cannot. Acceptance: - Positive verification shows the approved llm-connect service account can read `OPENROUTER_API_KEY` through OpenBao or External Secrets without printing the value. - Negative verification shows an unapproved service account cannot read the path. - OpenBao audit evidence exists for allowed and denied attempts, recorded only as non-secret request ids or timestamps. - Verification includes the External Secrets delivery path because that is the intended production consumer interface. ## T06 - Activate ops-warden catalog front door ```task id: RAILIANCE-WP-0010-T06 status: done priority: medium state_hub_task_id: "376de3fe-ef9c-4b57-b238-1ba21ac8bb1c" ``` Send ops-warden the non-secret pointers needed to promote the agreed OpenRouter/llm-connect selector from draft to active. Acceptance: - The handoff includes only catalog id, mount, path, field name, auth role, policy name/path, External Secrets target, optional flex-auth ref, and runbook/workplan links. - ops-warden confirms the catalog entry has no unresolved placeholders. - ops-warden confirms it proxies access as the caller and holds no provider key value. - The CCR front-door readiness becomes active/resolvable only after positive and negative verification. **2026-07-02:** T06 done. ops-warden promoted catalog id `openrouter-llm-connect` from draft to active (ops-warden commit `364eb7d`) following its own promotion checklist: concrete zero-placeholder handoff (`warden route show openrouter-llm-connect --json` reports `status: active`, `resolvable: true`), playbook gate marked met, draft tables updated, routing tests passing (45/45). The entry carries pointers only — ops-warden proxies reads as the caller and holds no provider key value. `CCR-2026-0003` recorded the `frontdoor_activation` evidence and moved to `status: active` with `readiness: ready`. Promotion happened only after the 2026-07-02 positive/negative verification. ## T07 - Record lifecycle operations ```task id: RAILIANCE-WP-0010-T07 status: done priority: medium state_hub_task_id: "130155a5-e0f9-49f8-ba27-b48098746f02" ``` Document how to deactivate, rotate, and respond to compromise for this lane. Acceptance: - Deactivation disables the ops-warden front door and removes or detaches the auth role policy without deleting required audit evidence. - Rotation keeps the provider key inside OpenBao/operator custody and records only non-secret evidence. - Compromise response names the immediate front-door disable, provider-key rotation, and follow-up incident workplan path. ## Exit Criteria - `CCR-2026-0003` is reviewed, approved, applied, verified, and active. - llm-connect can consume `OPENROUTER_API_KEY` through the approved platform interface. - Unauthorized access is denied and recorded. - ops-warden can resolve the agreed OpenRouter/llm-connect selector without storing the value. - No secret values appear in Git, State Hub, chat, prompts, logs, or workplans. ## Progress 2026-07-02 — approval and metadata apply `CCR-2026-0003` approved by bernd.worsch (platform-operator + activity-core-owner); T01 closes on that approval with the `openrouter-llm-connect` selector already aligned. - T03 done: policy `workload-kv-read-llm-connect-provider-secrets` and kubernetes auth role applied via the constrained prod-applier child token; State Hub apply evidence `04c70285`. - T04 remains the live gate: the KV entry at `platform/workloads/activity-core/llm-connect/llm-connect-provider-secrets` does not exist yet — the operator must enter `OPENROUTER_API_KEY` through OpenBao custody. The activity-core namespace also has no ExternalSecret object for this lane yet. ops-warden checkpoint message: `6b058584`. ## Progress 2026-07-02 — value provisioned, lane live end-to-end - T04 done: the operator entered `OPENROUTER_API_KEY` directly through OpenBao custody (KV metadata: version 1, created 2026-07-02T10:18Z). The value never passed through Git, State Hub, chat, or agent hands. - T05 done: positive — new `ExternalSecret activity-core/llm-connect-provider-secrets` (ClusterSecretStore `openbao-activity-core`, creationPolicy Owner) reached `SecretSynced=True` at 10:54Z, took ownership of the previously manual Secret, and the llm-connect deployment rolled out cleanly on the OpenBao-delivered value (pod ready, 0 restarts, /health probes passing). Negative — default-policy token denied on the KV path (10:08Z probe, audit-logged). Manifest committed in llm-connect `dfd2ce7` (`deploy/k8s/activity-core-llm-connect/externalsecret.yaml`). - T06 progress: activation update sent to ops-warden; `openrouter-llm-connect` can now leave draft once the catalog confirmation lands. - Scope note: this closes the CoulombCore lane the CCR describes. The separate llm-connect instance on the railiance01 k3s cluster still consumes its bootstrap-provisioned Secret; migrating it is railiance01-cluster work, not part of CCR-2026-0003. ## T07 completed 2026-07-02 Lifecycle operations documented in `docs/credential-lane-lifecycle-runbook.md`: the canonical per-action procedure is generated by `scripts/credential-change.py lifecycle-plan --action {deactivate|rotate|compromise}`, and the runbook adds the lane-specific consumer facts (materialized-Secret persistence, second consumers, restart requirements, provider-side revocation for the OpenRouter key) plus the post-rotate verification contract. Front-door disable comes first in every action; audit evidence is never deleted; values stay in OpenBao/operator custody.