---
id: ACTIVITY-WP-0039
type: workplan
title: "Multiple queue worker identities with OpenBao-custodied tokens"
domain: infotech
repo: activity-core
status: active
flavor: implementation
owner: claude-code
topic_slug: activity-core
priority: high
created: "2026-09-23"
updated: "2026-09-23"
related:
  - ACT-ADR-005
  - ACTIVITY-WP-0036
  - SECRETS-WP-0009-T03
  - SECRETS-WP-0011-T04
  - RPF-WP-0045
---

# Multiple Queue Worker Identities with OpenBao-Custodied Tokens

## Origin

Two requests from secrets-engine, both approved on their side on 2026-09-23:

- Message `914853d9` (SECRETS-WP-0009-T03) asks for more than one
  authenticated queue worker. The spend-admitted `rein-aharness metered-once`
  owner should claim as `rein-aharness-metered@railiance01`, separate from the
  running `rein-aharness@railiance01` claim loop. This keeps paid-run
  attribution distinct at the API.
- Message `6e694682` (SECRETS-WP-0011-T04) asks for the worker token to be held
  in OpenBao and synced by the existing `openbao-activity-core` store, instead
  of the hand-generated `openssl rand` value in
  `docs/deploy-ops-run-queue-railiance.md`. secrets-engine will catalog a
  read-only lane for the metered worker's path only. It will never read, copy
  or mint the token.

WP-0036 bound one token to one configured identity. This workplan keeps that
guarantee and extends it to a set of identities.

## Boundary rules

- Each token is bound to exactly one worker identity. The body `worker_id`
  must still match it. No token is shared, and nothing falls back implicitly.
- Worker tokens never equal the operator token, and operator/SSO credentials
  stay out of worker mutations (WP-0036).
- One OpenBao path per worker identity, so a consumer lane can be granted
  exactly one worker's token.
- Minting and writing a token is founder-attended OpenBao work. The store's
  role policy belongs to railiance-platform. No token value passes through
  Git, State Hub, chat, or an agent shell.

## Token-to-identity map in the worker API

```task
id: ACTIVITY-WP-0039-T01
status: done
priority: high
```

Add `ACTIVITY_CORE_WORKERS`, a non-secret comma-separated list of
`worker_id=ENV_NAME` entries. Each ENV_NAME must be `ACTIVITY_CORE_WORKER_TOKEN`
or `ACTIVITY_CORE_WORKER_TOKEN_<SUFFIX>`. When the list is unset, the legacy
`ACTIVITY_CORE_WORKER_ID` + `ACTIVITY_CORE_WORKER_TOKEN` pair behaves exactly as
before. Worker mutations fail closed with 503 when the configuration is
ambiguous. That covers a duplicate identity, a missing or shared token, a token
equal to the operator token, an env name outside the pattern, and a legacy
identity missing from the list. A broken worker configuration does not lock out
operator or SSO reads.

Implemented in `src/activity_core/ops_runs_api.py` (`worker_credentials`).
Tests in `tests/test_ops_runs_api.py` cover per-identity binding, a
cross-identity body rejected with 403, an unknown token rejected with 401, each
fail-closed case, and operator access under broken worker configuration.

## Declare OpenBao paths and the ExternalSecret

```task
id: ACTIVITY-WP-0039-T02
status: wait
priority: high
```

Paths, one per identity, each with field `token`:

- `platform/workloads/activity-core/ops-run-workers/rein-aharness-railiance01`
- `platform/workloads/activity-core/ops-run-workers/rein-aharness-metered-railiance01`

`k8s/railiance/15-externalsecret-worker-tokens.yaml` merges them into
`actcore-runtime-secret` as `ACTIVITY_CORE_WORKER_TOKEN` and
`ACTIVITY_CORE_WORKER_TOKEN_METERED`. The manifest is in the repo but not
applied.

Waiting on railiance-platform to add both exact paths to the
`activity-core-eso` role policy (RPF-WP-0045 pattern), and on secrets-engine to
confirm the metered path for its catalog.

## Seed the tokens in OpenBao

```task
id: ACTIVITY-WP-0039-T03
status: wait
priority: high
```

Founder-attended, through `warden access openbao-platform-admin-login --exec`
(orientation section 5):

- **Metered worker:** mint a fresh random value directly into its path. No
  value exists today.
- **Claim-loop worker:** the founder chooses between two options.
  - (a) Move the current value into OpenBao. The running claim loop needs no
    change, but the value was hand-generated outside custody.
  - (b) Mint a fresh value and update the rein-aharness claim-loop
    configuration on railiance01 (user tegwick) in the same window. This means
    one coordinated restart.

Done when both paths hold a value and no value has been printed or logged.

## Roll out and prove both identities

```task
id: ACTIVITY-WP-0039-T04
status: wait
priority: high
```

Needs the founder's go-ahead (`ADMINISTER @ realm:kubernetes/railiance01`,
`activation=APPROVED`) and a CPU headroom check first (orientation section 4).
Apply the ExternalSecret, add `ACTIVITY_CORE_WORKERS` to
`actcore-runtime-config`, deploy the new image, and restart only the API.
Prove four things:

- The claim loop still polls with HTTP 200.
- The metered token authenticates as `rein-aharness-metered@railiance01`.
- A metered token paired with the loop identity is rejected with HTTP 403.
- The previous hand-set Secret key is now owned by ESO.

Report the revision to secrets-engine on threads `914853d9` and `6e694682`.