railiance-platform/workplans/RPF-WP-0004-llm-connect-openrouter-provider-key-lane.md

id	type	title	domain	repo	status	owner	topic_slug	planning_priority	planning_order	created	updated	depends_on_workplans	related_state_hub_messages	related_ccrs	state_hub_workstream_id
RPF-WP-0004	workplan	llm-connect OpenRouter Provider Key Lane	financials	railiance-platform	finished	codex	railiance	high	10	2026-06-29	2026-08-21	RAIL-PL-WP-0002 RAILIANCE-WP-0004 RAILIANCE-WP-0007 RAILIANCE-WP-0008	f76d3a9e-a98f-4081-885d-b79d94312699	CCR-2026-0003	afc66064-7ba4-5f11-b61f-d76890244b7e

RPF-WP-0004 - llm-connect OpenRouter Provider Key Lane

Goal

Promote the draft llm-connect OpenRouter provider-key access lane from a proposed CCR to a reviewed, least-privilege, verified OpenBao workload KV lane that the live llm-connect runtime can consume through External Secrets and that ops-warden can reference without holding the provider key.

This keeps provider credential custody in the shared platform layer while leaving llm-connect behavior, model routing, and provider-specific runtime logic with the owning application/service.

No task in this workplan may paste, commit, log, or send secret values through Git, State Hub, chat, prompts, shell history, or workplan text.

Suggestion Reviewed

Ops-warden confirmed the whynot-design lane in State Hub message f76d3a9e-a98f-4081-885d-b79d94312699 and noted that the OpenRouter/llm-connect sibling lane remains draft on its side. The repo already has the proposed non-secret CCR:

• credential-change-requests/CCR-2026-0003-llm-connect-openrouter-api-key.yaml

Ops-warden already uses openrouter-llm-connect as the canonical draft catalog id in its routing catalog and playbook. This workplan and CCR-2026-0003 now align to that selector so automated callers have one stable route name before activation.

INTENT Fit

This work belongs in railiance-platform because it provides the dependable secret custody and delivery substrate for a shared runtime service. It does not decide which model or provider llm-connect should use. The platform-owned output is the OpenBao path, read policy, Kubernetes auth role, External Secrets target, verification evidence, and ops-warden handoff.

The plan supports these INTENT.md principles:

• secure custody: the provider key stays in OpenBao/operator custody;
• stable interfaces: llm-connect consumes a documented path, field, role, and External Secrets target;
• operable and observable: activation requires positive and negative checks plus non-secret audit evidence;
• independently evolvable: the provider key storage and front-door routing can change without forcing runtime consumers to know internal topology.

Proposed Contract

Item	Proposed value
CCR	CCR-2026-0003
ops-warden catalog id	openrouter-llm-connect
Tenant/org	activity-core
Workload/project	llm-connect
KV mount	platform
OpenBao CLI path	platform/workloads/activity-core/llm-connect/llm-connect-provider-secrets
Secret field	OPENROUTER_API_KEY
Read policy	workload-kv-read-llm-connect-provider-secrets
Policy file	openbao/policies/workload-kv-read-llm-connect-provider-secrets.hcl
Auth method	Kubernetes auth
Auth role	external-secrets-activity-core
OpenBao auth service account	external-secrets
OpenBao auth namespace	external-secrets
Delivery surface	Future activity-core ExternalSecret to Secret llm-connect-provider-secrets
ops-warden command	warden access openrouter-llm-connect --fetch OPENROUTER_API_KEY

Tasks

T01 - Review CCR scope and selector naming

id: RPF-WP-0004-T01
status: done
priority: high
state_hub_task_id: "b635b254-e24d-5f34-b5d9-06ca167765e8"

Review CCR-2026-0003 with the platform operator and activity-core owner before any live OpenBao apply.

Acceptance:

• The activity-core owner confirms that llm-connect should receive OPENROUTER_API_KEY through this platform lane.
• ops-warden and railiance-platform agree on one stable catalog id/selector: openrouter-llm-connect.
• Review comments and approval state are recorded in the CCR without secret values.
• The lane remains clearly platform-owned secret custody, not llm-connect model routing or provider selection logic.

2026-06-30: Confirmed activity-core namespace exists and Kubernetes Secret activity-core/llm-connect-provider-secrets exists, but no activity-core ExternalSecret exists yet. Kept canonical CCR catalog id llm-connect-openrouter-api-key; ops-warden previously mentioned openrouter-llm-connect, so selector agreement remains open and this task stays progress. OpenBao public seal status now reports sealed=false; the prior sealed message is no longer the active blocker.

2026-07-01: Resolved the selector naming ambiguity in favor of ops-warden canon. The local ops-warden routing catalog and playbook define openrouter-llm-connect as the draft OpenRouter/llm-connect route, so CCR-2026-0003 and this workplan now use that catalog id and command shape. T01 remains progress until activity-core/platform approval moves the CCR out of proposed.

T02 - Confirm Kubernetes auth and External Secrets binding

id: RPF-WP-0004-T02
status: done
priority: high
state_hub_task_id: "c692c020-6705-55c4-9a80-358f83fc67f8"

Confirm the exact Kubernetes service account, namespace, and External Secrets target that should consume this lane.

Acceptance:

• The service account and namespace are confirmed as llm-connect/activity-core or the CCR is updated with the approved alternative.
• The auth role binds only the approved service account and namespace.
• The External Secrets target is confirmed as llm-connect-provider-secrets or updated with the approved alternative.
• No direct human or agent read path is activated unless separately approved.

2026-06-30: Confirmed the proposed llm-connect service account does not exist and the current llm-connect Deployment uses the namespace default service account. Updated CCR-2026-0003 to the approved platform ESO pattern: OpenBao Kubernetes auth role external-secrets-activity-core bound to external-secrets/external-secrets. Added argocd/platform-addons/openbao-secretstore/openbao-activity-core.clustersecretstore.yaml, limited to the activity-core namespace, and Make target openbao-configure-external-secrets-activity-core for the matching OpenBao role/policy apply. kubectl kustomize argocd/platform-addons/openbao-secretstore renders both the existing issue-core store and the new activity-core store. credential-change.py applier-dry-run CCR-2026-0003 now blocks only because the CCR is still proposed.

T03 - Apply or confirm least-privilege OpenBao metadata

id: RPF-WP-0004-T03
status: done
priority: high
state_hub_task_id: "55abf986-bc9c-5025-9513-36ca0f2861d3"

Apply the read policy and Kubernetes auth role only after review and binding confirmation.

Acceptance:

• openbao/policies/workload-kv-read-llm-connect-provider-secrets.hcl grants only read access to the exact KV-v2 data and metadata paths.
• The Kubernetes auth role attaches only workload-kv-read-llm-connect-provider-secrets.
• Live apply uses an approved operator path or the delegated applier from RAILIANCE-WP-0008; broad platform-admin handoffs are avoided where possible.
• Apply evidence records only policy name, role name, actor, timestamp, and non-secret OpenBao request ids.

T04 - Provision the provider key through approved custody

id: RPF-WP-0004-T04
status: done
priority: high
state_hub_task_id: "cb1272ca-9cbc-5ca6-bf3c-af0c2c18c887"

Have an approved operator create or confirm the OpenBao KV entry and OPENROUTER_API_KEY field.

Acceptance:

• The path exists at platform/workloads/activity-core/llm-connect/llm-connect-provider-secrets.
• Field OPENROUTER_API_KEY is present.
• The value is entered directly through OpenBao/operator custody, never through Git, State Hub, chat, prompts, workplans, or shell history.
• Non-secret evidence records only path, field name, actor, timestamp, and verification result.

T05 - Verify positive and negative access

id: RPF-WP-0004-T05
status: done
priority: high
state_hub_task_id: "cdc65efd-a733-5a13-b439-85bc24712143"

Prove that the approved llm-connect identity can consume the lane and that other identities cannot.

Acceptance:

• Positive verification shows the approved llm-connect service account can read OPENROUTER_API_KEY through OpenBao or External Secrets without printing the value.
• Negative verification shows an unapproved service account cannot read the path.
• OpenBao audit evidence exists for allowed and denied attempts, recorded only as non-secret request ids or timestamps.
• Verification includes the External Secrets delivery path because that is the intended production consumer interface.

T06 - Activate ops-warden catalog front door

id: RPF-WP-0004-T06
status: done
priority: medium
state_hub_task_id: "83f6801c-1355-5b0f-95ba-c35f45175fff"

Send ops-warden the non-secret pointers needed to promote the agreed OpenRouter/llm-connect selector from draft to active.

Acceptance:

• The handoff includes only catalog id, mount, path, field name, auth role, policy name/path, External Secrets target, optional flex-auth ref, and runbook/workplan links.
• ops-warden confirms the catalog entry has no unresolved placeholders.
• ops-warden confirms it proxies access as the caller and holds no provider key value.
• The CCR front-door readiness becomes active/resolvable only after positive and negative verification.

2026-07-02: T06 done. ops-warden promoted catalog id openrouter-llm-connect from draft to active (ops-warden commit 364eb7d) following its own promotion checklist: concrete zero-placeholder handoff (warden route show openrouter-llm-connect --json reports status: active, resolvable: true), playbook gate marked met, draft tables updated, routing tests passing (45/45). The entry carries pointers only — ops-warden proxies reads as the caller and holds no provider key value. CCR-2026-0003 recorded the frontdoor_activation evidence and moved to status: active with readiness: ready. Promotion happened only after the 2026-07-02 positive/negative verification.

T07 - Record lifecycle operations

id: RPF-WP-0004-T07
status: done
priority: medium
state_hub_task_id: "d9386de3-5b68-5ed2-a44b-3b21705b720f"

Document how to deactivate, rotate, and respond to compromise for this lane.

Acceptance:

• Deactivation disables the ops-warden front door and removes or detaches the auth role policy without deleting required audit evidence.
• Rotation keeps the provider key inside OpenBao/operator custody and records only non-secret evidence.
• Compromise response names the immediate front-door disable, provider-key rotation, and follow-up incident workplan path.

T08 - Reconcile the railiance01 consumer after provider 401

id: RPF-WP-0004-T08
status: cancel
priority: high
state_hub_task_id: "934f7424-890a-57a7-93cd-56719f02c935"

Restore the approved OpenBao-to-ESO delivery path for the railiance01 llm-connect instance after its bootstrap key began returning OpenRouter 401.

Read-only diagnosis on 2026-08-20 found a healthy, 61-day-old Deployment using an unmanaged Secret created on 2026-06-19. No ExternalSecret exists on railiance01. ClusterSecretStore/openbao-activity-core is Valid, but its ESO token reports deny for both the canonical llm-connect data and metadata paths because the activity-core token bootstrap default omits workload-kv-read-llm-connect-provider-secrets.

Repair the durable bootstrap policy set, mint a replacement limited ESO token through the existing attended platform-admin handoff, apply the already reviewed llm-connect ExternalSecret, and wait for SecretSynced=True. Restart the Deployment only after sync, then run the in-cluster fixture smoke. If the OpenBao-delivered key still returns 401, stop and require provider-side rotation; do not copy or inspect either key value.

Done when railiance01 consumes the canonical lane, the fixture smoke passes, and rollback/non-secret evidence is recorded.

Progress 2026-08-20: repaired the durable delivery path. The activity-core ESO bootstrap now includes the approved exact-path llm-connect policy; a new periodic token was minted through the existing mode-0600 platform-admin handoff, and capabilities report read on only the requested data and metadata paths. The reviewed llm-connect ExternalSecret was applied on railiance01, reported Ready=True / SecretSynced, and took ownership of the one-key Secret. deployment/llm-connect rolled out successfully, and the real actcore-worker reaches its Service /health with status=ok.

The full fixture still returns sanitized OpenRouter HTTP 401 when run inside the new llm-connect pod. Therefore the canonical provider key itself is no longer accepted. T08 waits on an attended OpenRouter account owner to mint a replacement key. Keep the old key active until the new value is written from a mode-0600 file to the existing KV path, ESO reports a new successful refresh, the Deployment is restarted, and the fixture passes; then revoke the old key. No provider-key value was read or printed during diagnosis or repair.

Closure 2026-08-21: the railiance-platform portion is complete: the durable OpenBao-to-ESO delivery path is repaired, the ExternalSecret is synced, and the workload consumes the canonical lane. The remaining action is an attended replacement of a provider key rejected by OpenRouter, followed by a consumer fixture smoke. That operational incident is already tracked by the owning consumer as ACTIVITY-WP-0031-T01, so this duplicate task is cancelled rather than keeping the completed platform-lane workplan active.

Exit Criteria

• CCR-2026-0003 is reviewed, approved, applied, verified, and active.
• llm-connect can consume OPENROUTER_API_KEY through the approved platform interface.
• Unauthorized access is denied and recorded.
• ops-warden can resolve the agreed OpenRouter/llm-connect selector without storing the value.
• No secret values appear in Git, State Hub, chat, prompts, logs, or workplans.

Progress 2026-07-02 — approval and metadata apply

CCR-2026-0003 approved by bernd.worsch (platform-operator + activity-core-owner); T01 closes on that approval with the openrouter-llm-connect selector already aligned.

• T03 done: policy workload-kv-read-llm-connect-provider-secrets and kubernetes auth role applied via the constrained prod-applier child token; State Hub apply evidence 04c70285.
• T04 remains the live gate: the KV entry at platform/workloads/activity-core/llm-connect/llm-connect-provider-secrets does not exist yet — the operator must enter OPENROUTER_API_KEY through OpenBao custody. The activity-core namespace also has no ExternalSecret object for this lane yet. ops-warden checkpoint message: 6b058584.

Progress 2026-07-02 — value provisioned, lane live end-to-end

• T04 done: the operator entered OPENROUTER_API_KEY directly through OpenBao custody (KV metadata: version 1, created 2026-07-02T10:18Z). The value never passed through Git, State Hub, chat, or agent hands.
• T05 done: positive — new ExternalSecret activity-core/llm-connect-provider-secrets (ClusterSecretStore openbao-activity-core, creationPolicy Owner) reached SecretSynced=True at 10:54Z, took ownership of the previously manual Secret, and the llm-connect deployment rolled out cleanly on the OpenBao-delivered value (pod ready, 0 restarts, /health probes passing). Negative — default-policy token denied on the KV path (10:08Z probe, audit-logged). Manifest committed in llm-connect dfd2ce7 (deploy/k8s/activity-core-llm-connect/externalsecret.yaml).
• T06 progress: activation update sent to ops-warden; openrouter-llm-connect can now leave draft once the catalog confirmation lands.
• Scope note: this closes the CoulombCore lane the CCR describes. The separate llm-connect instance on the railiance01 k3s cluster still consumes its bootstrap-provisioned Secret; migrating it is railiance01-cluster work, not part of CCR-2026-0003.

T07 completed 2026-07-02

Lifecycle operations documented in docs/credential-lane-lifecycle-runbook.md: the canonical per-action procedure is generated by scripts/credential-change.py lifecycle-plan <CCR> --action {deactivate|rotate|compromise}, and the runbook adds the lane-specific consumer facts (materialized-Secret persistence, second consumers, restart requirements, provider-side revocation for the OpenRouter key) plus the post-rotate verification contract. Front-door disable comes first in every action; audit evidence is never deleted; values stay in OpenBao/operator custody.