railiance-platform/workplans/RPF-WP-0004-llm-connect-openrouter-provider-key-lane.md

---
id: RPF-WP-0004
type: workplan
title: "llm-connect OpenRouter Provider Key Lane"
domain: financials
repo: railiance-platform
status: finished
owner: codex
topic_slug: railiance
planning_priority: high
planning_order: 10
created: "2026-06-29"
updated: "2026-08-21"
depends_on_workplans:
  - RAIL-PL-WP-0002
  - RAILIANCE-WP-0004
  - RAILIANCE-WP-0007
  - RAILIANCE-WP-0008
related_state_hub_messages:
  - "f76d3a9e-a98f-4081-885d-b79d94312699"
related_ccrs:
  - CCR-2026-0003
state_hub_workstream_id: "afc66064-7ba4-5f11-b61f-d76890244b7e"
---

# RPF-WP-0004 - llm-connect OpenRouter Provider Key Lane

## Goal

Promote the draft llm-connect OpenRouter provider-key access lane from a
proposed CCR to a reviewed, least-privilege, verified OpenBao workload KV lane
that the live llm-connect runtime can consume through External Secrets and that
`ops-warden` can reference without holding the provider key.

This keeps provider credential custody in the shared platform layer while
leaving llm-connect behavior, model routing, and provider-specific runtime
logic with the owning application/service.

No task in this workplan may paste, commit, log, or send secret values through
Git, State Hub, chat, prompts, shell history, or workplan text.

## Suggestion Reviewed

Ops-warden confirmed the whynot-design lane in State Hub message
`f76d3a9e-a98f-4081-885d-b79d94312699` and noted that the OpenRouter/llm-connect
sibling lane remains draft on its side. The repo already has the proposed
non-secret CCR:

- `credential-change-requests/CCR-2026-0003-llm-connect-openrouter-api-key.yaml`

Ops-warden already uses `openrouter-llm-connect` as the canonical draft
catalog id in its routing catalog and playbook. This workplan and
`CCR-2026-0003` now align to that selector so automated callers have
one stable route name before activation.

## INTENT Fit

This work belongs in railiance-platform because it provides the dependable
secret custody and delivery substrate for a shared runtime service. It does not
decide which model or provider llm-connect should use. The platform-owned
output is the OpenBao path, read policy, Kubernetes auth role, External Secrets
target, verification evidence, and ops-warden handoff.

The plan supports these `INTENT.md` principles:

- secure custody: the provider key stays in OpenBao/operator custody;
- stable interfaces: llm-connect consumes a documented path, field, role, and
  External Secrets target;
- operable and observable: activation requires positive and negative checks
  plus non-secret audit evidence;
- independently evolvable: the provider key storage and front-door routing can
  change without forcing runtime consumers to know internal topology.

## Proposed Contract

| Item | Proposed value |
| --- | --- |
| CCR | `CCR-2026-0003` |
| ops-warden catalog id | `openrouter-llm-connect` |
| Tenant/org | `activity-core` |
| Workload/project | `llm-connect` |
| KV mount | `platform` |
| OpenBao CLI path | `platform/workloads/activity-core/llm-connect/llm-connect-provider-secrets` |
| Secret field | `OPENROUTER_API_KEY` |
| Read policy | `workload-kv-read-llm-connect-provider-secrets` |
| Policy file | `openbao/policies/workload-kv-read-llm-connect-provider-secrets.hcl` |
| Auth method | Kubernetes auth |
| Auth role | `external-secrets-activity-core` |
| OpenBao auth service account | `external-secrets` |
| OpenBao auth namespace | `external-secrets` |
| Delivery surface | Future activity-core ExternalSecret to Secret `llm-connect-provider-secrets` |
| ops-warden command | `warden access openrouter-llm-connect --fetch OPENROUTER_API_KEY` |

## Tasks

## T01 - Review CCR scope and selector naming

```task
id: RPF-WP-0004-T01
status: done
priority: high
state_hub_task_id: "b635b254-e24d-5f34-b5d9-06ca167765e8"
```

Review `CCR-2026-0003` with the platform operator and activity-core owner
before any live OpenBao apply.

Acceptance:

- The activity-core owner confirms that llm-connect should receive
  `OPENROUTER_API_KEY` through this platform lane.
- ops-warden and railiance-platform agree on one stable catalog id/selector:
  `openrouter-llm-connect`.
- Review comments and approval state are recorded in the CCR without secret
  values.
- The lane remains clearly platform-owned secret custody, not llm-connect model
  routing or provider selection logic.

**2026-06-30:** Confirmed `activity-core` namespace exists and Kubernetes Secret
`activity-core/llm-connect-provider-secrets` exists, but no activity-core
`ExternalSecret` exists yet. Kept canonical CCR catalog id
`llm-connect-openrouter-api-key`; ops-warden previously mentioned
`openrouter-llm-connect`, so selector agreement remains open and this task stays
`progress`. OpenBao public seal status now reports `sealed=false`; the prior
sealed message is no longer the active blocker.

**2026-07-01:** Resolved the selector naming ambiguity in favor of ops-warden
canon. The local ops-warden routing catalog and playbook define
`openrouter-llm-connect` as the draft OpenRouter/llm-connect route, so
`CCR-2026-0003` and this workplan now use that catalog id and command
shape. T01 remains `progress` until activity-core/platform approval moves
the CCR out of `proposed`.

## T02 - Confirm Kubernetes auth and External Secrets binding

```task
id: RPF-WP-0004-T02
status: done
priority: high
state_hub_task_id: "c692c020-6705-55c4-9a80-358f83fc67f8"
```

Confirm the exact Kubernetes service account, namespace, and External Secrets
target that should consume this lane.

Acceptance:

- The service account and namespace are confirmed as
  `llm-connect`/`activity-core` or the CCR is updated with the approved
  alternative.
- The auth role binds only the approved service account and namespace.
- The External Secrets target is confirmed as
  `llm-connect-provider-secrets` or updated with the approved alternative.
- No direct human or agent read path is activated unless separately approved.

**2026-06-30:** Confirmed the proposed `llm-connect` service account does not
exist and the current `llm-connect` Deployment uses the namespace `default`
service account. Updated `CCR-2026-0003` to the approved platform ESO pattern:
OpenBao Kubernetes auth role `external-secrets-activity-core` bound to
`external-secrets/external-secrets`. Added
`argocd/platform-addons/openbao-secretstore/openbao-activity-core.clustersecretstore.yaml`,
limited to the `activity-core` namespace, and Make target
`openbao-configure-external-secrets-activity-core` for the matching OpenBao
role/policy apply. `kubectl kustomize argocd/platform-addons/openbao-secretstore`
renders both the existing issue-core store and the new activity-core store.
`credential-change.py applier-dry-run CCR-2026-0003` now blocks only because the
CCR is still `proposed`.

## T03 - Apply or confirm least-privilege OpenBao metadata

```task
id: RPF-WP-0004-T03
status: done
priority: high
state_hub_task_id: "55abf986-bc9c-5025-9513-36ca0f2861d3"
```

Apply the read policy and Kubernetes auth role only after review and binding
confirmation.

Acceptance:

- `openbao/policies/workload-kv-read-llm-connect-provider-secrets.hcl` grants
  only read access to the exact KV-v2 data and metadata paths.
- The Kubernetes auth role attaches only
  `workload-kv-read-llm-connect-provider-secrets`.
- Live apply uses an approved operator path or the delegated applier from
  `RAILIANCE-WP-0008`; broad `platform-admin` handoffs are avoided where
  possible.
- Apply evidence records only policy name, role name, actor, timestamp, and
  non-secret OpenBao request ids.

## T04 - Provision the provider key through approved custody

```task
id: RPF-WP-0004-T04
status: done
priority: high
state_hub_task_id: "cb1272ca-9cbc-5ca6-bf3c-af0c2c18c887"
```

Have an approved operator create or confirm the OpenBao KV entry and
`OPENROUTER_API_KEY` field.

Acceptance:

- The path exists at
  `platform/workloads/activity-core/llm-connect/llm-connect-provider-secrets`.
- Field `OPENROUTER_API_KEY` is present.
- The value is entered directly through OpenBao/operator custody, never through
  Git, State Hub, chat, prompts, workplans, or shell history.
- Non-secret evidence records only path, field name, actor, timestamp, and
  verification result.

## T05 - Verify positive and negative access

```task
id: RPF-WP-0004-T05
status: done
priority: high
state_hub_task_id: "cdc65efd-a733-5a13-b439-85bc24712143"
```

Prove that the approved llm-connect identity can consume the lane and that
other identities cannot.

Acceptance:

- Positive verification shows the approved llm-connect service account can read
  `OPENROUTER_API_KEY` through OpenBao or External Secrets without printing the
  value.
- Negative verification shows an unapproved service account cannot read the
  path.
- OpenBao audit evidence exists for allowed and denied attempts, recorded only
  as non-secret request ids or timestamps.
- Verification includes the External Secrets delivery path because that is the
  intended production consumer interface.

## T06 - Activate ops-warden catalog front door

```task
id: RPF-WP-0004-T06
status: done
priority: medium
state_hub_task_id: "83f6801c-1355-5b0f-95ba-c35f45175fff"
```

Send ops-warden the non-secret pointers needed to promote the agreed
OpenRouter/llm-connect selector from draft to active.

Acceptance:

- The handoff includes only catalog id, mount, path, field name, auth role,
  policy name/path, External Secrets target, optional flex-auth ref, and
  runbook/workplan links.
- ops-warden confirms the catalog entry has no unresolved placeholders.
- ops-warden confirms it proxies access as the caller and holds no provider key
  value.
- The CCR front-door readiness becomes active/resolvable only after positive
  and negative verification.

**2026-07-02:** T06 done. ops-warden promoted catalog id
`openrouter-llm-connect` from draft to active (ops-warden commit `364eb7d`)
following its own promotion checklist: concrete zero-placeholder handoff
(`warden route show openrouter-llm-connect --json` reports `status: active`,
`resolvable: true`), playbook gate marked met, draft tables updated, routing
tests passing (45/45). The entry carries pointers only — ops-warden proxies
reads as the caller and holds no provider key value. `CCR-2026-0003` recorded
the `frontdoor_activation` evidence and moved to `status: active` with
`readiness: ready`. Promotion happened only after the 2026-07-02
positive/negative verification.

## T07 - Record lifecycle operations

```task
id: RPF-WP-0004-T07
status: done
priority: medium
state_hub_task_id: "d9386de3-5b68-5ed2-a44b-3b21705b720f"
```

Document how to deactivate, rotate, and respond to compromise for this lane.

Acceptance:

- Deactivation disables the ops-warden front door and removes or detaches the
  auth role policy without deleting required audit evidence.
- Rotation keeps the provider key inside OpenBao/operator custody and records
  only non-secret evidence.
- Compromise response names the immediate front-door disable, provider-key
  rotation, and follow-up incident workplan path.

## T08 - Reconcile the railiance01 consumer after provider 401

```task
id: RPF-WP-0004-T08
status: cancel
priority: high
```

Restore the approved OpenBao-to-ESO delivery path for the railiance01
llm-connect instance after its bootstrap key began returning OpenRouter 401.

Read-only diagnosis on 2026-08-20 found a healthy, 61-day-old Deployment using
an unmanaged Secret created on 2026-06-19. No `ExternalSecret` exists on
railiance01. `ClusterSecretStore/openbao-activity-core` is Valid, but its ESO
token reports `deny` for both the canonical llm-connect data and metadata
paths because the activity-core token bootstrap default omits
`workload-kv-read-llm-connect-provider-secrets`.

Repair the durable bootstrap policy set, mint a replacement limited ESO token
through the existing attended platform-admin handoff, apply the already
reviewed llm-connect `ExternalSecret`, and wait for `SecretSynced=True`. Restart
the Deployment only after sync, then run the in-cluster fixture smoke. If the
OpenBao-delivered key still returns 401, stop and require provider-side
rotation; do not copy or inspect either key value.

Done when railiance01 consumes the canonical lane, the fixture smoke passes,
and rollback/non-secret evidence is recorded.

Progress 2026-08-20: repaired the durable delivery path. The activity-core ESO
bootstrap now includes the approved exact-path llm-connect policy; a new
periodic token was minted through the existing mode-0600 platform-admin
handoff, and capabilities report `read` on only the requested data and metadata
paths. The reviewed llm-connect `ExternalSecret` was applied on railiance01,
reported `Ready=True` / `SecretSynced`, and took ownership of the one-key
Secret. `deployment/llm-connect` rolled out successfully, and the real
`actcore-worker` reaches its Service `/health` with `status=ok`.

The full fixture still returns sanitized OpenRouter HTTP 401 when run inside
the new llm-connect pod. Therefore the canonical provider key itself is no
longer accepted. T08 waits on an attended OpenRouter account owner to mint a
replacement key. Keep the old key active until the new value is written from a
mode-0600 file to the existing KV path, ESO reports a new successful refresh,
the Deployment is restarted, and the fixture passes; then revoke the old key.
No provider-key value was read or printed during diagnosis or repair.

Closure 2026-08-21: the railiance-platform portion is complete: the durable
OpenBao-to-ESO delivery path is repaired, the `ExternalSecret` is synced, and
the workload consumes the canonical lane. The remaining action is an attended
replacement of a provider key rejected by OpenRouter, followed by a consumer
fixture smoke. That operational incident is already tracked by the owning
consumer as `ACTIVITY-WP-0031-T01`, so this duplicate task is cancelled rather
than keeping the completed platform-lane workplan active.

## Exit Criteria

- `CCR-2026-0003` is reviewed, approved, applied, verified, and active.
- llm-connect can consume `OPENROUTER_API_KEY` through the approved platform
  interface.
- Unauthorized access is denied and recorded.
- ops-warden can resolve the agreed OpenRouter/llm-connect selector without
  storing the value.
- No secret values appear in Git, State Hub, chat, prompts, logs, or workplans.


## Progress 2026-07-02 — approval and metadata apply

`CCR-2026-0003` approved by bernd.worsch (platform-operator +
activity-core-owner); T01 closes on that approval with the
`openrouter-llm-connect` selector already aligned.

- T03 done: policy `workload-kv-read-llm-connect-provider-secrets` and
  kubernetes auth role applied via the constrained prod-applier child token;
  State Hub apply evidence `04c70285`.
- T04 remains the live gate: the KV entry at
  `platform/workloads/activity-core/llm-connect/llm-connect-provider-secrets`
  does not exist yet — the operator must enter `OPENROUTER_API_KEY` through
  OpenBao custody. The activity-core namespace also has no ExternalSecret
  object for this lane yet. ops-warden checkpoint message: `6b058584`.


## Progress 2026-07-02 — value provisioned, lane live end-to-end

- T04 done: the operator entered `OPENROUTER_API_KEY` directly through OpenBao
  custody (KV metadata: version 1, created 2026-07-02T10:18Z). The value never
  passed through Git, State Hub, chat, or agent hands.
- T05 done: positive — new `ExternalSecret
  activity-core/llm-connect-provider-secrets` (ClusterSecretStore
  `openbao-activity-core`, creationPolicy Owner) reached `SecretSynced=True`
  at 10:54Z, took ownership of the previously manual Secret, and the
  llm-connect deployment rolled out cleanly on the OpenBao-delivered value
  (pod ready, 0 restarts, /health probes passing). Negative — default-policy
  token denied on the KV path (10:08Z probe, audit-logged). Manifest committed
  in llm-connect `dfd2ce7`
  (`deploy/k8s/activity-core-llm-connect/externalsecret.yaml`).
- T06 progress: activation update sent to ops-warden; `openrouter-llm-connect`
  can now leave draft once the catalog confirmation lands.
- Scope note: this closes the CoulombCore lane the CCR describes. The separate
  llm-connect instance on the railiance01 k3s cluster still consumes its
  bootstrap-provisioned Secret; migrating it is railiance01-cluster work, not
  part of CCR-2026-0003.


## T07 completed 2026-07-02

Lifecycle operations documented in
`docs/credential-lane-lifecycle-runbook.md`: the canonical per-action
procedure is generated by `scripts/credential-change.py lifecycle-plan
<CCR> --action {deactivate|rotate|compromise}`, and the runbook adds the
lane-specific consumer facts (materialized-Secret persistence, second
consumers, restart requirements, provider-side revocation for the OpenRouter
key) plus the post-rotate verification contract. Front-door disable comes
first in every action; audit evidence is never deleted; values stay in
OpenBao/operator custody.