activity-core/workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md

---
id: ACTIVITY-WP-0039
type: workplan
title: "Multiple queue worker identities with OpenBao-custodied tokens"
domain: infotech
repo: activity-core
status: active
flavor: implementation
owner: claude-code
topic_slug: activity-core
priority: high
created: "2026-09-23"
updated: "2026-09-23"
related:
  - ACT-ADR-005
  - ACTIVITY-WP-0036
  - SECRETS-WP-0009-T03
  - SECRETS-WP-0011-T04
  - RPF-WP-0045
state_hub_workstream_id: "053ef7c6-c2cf-5f50-aabf-09034c66d1ce"
---

# Multiple Queue Worker Identities with OpenBao-Custodied Tokens

## Origin

Two requests from secrets-engine, both approved on their side on 2026-09-23:

- Message `914853d9` (SECRETS-WP-0009-T03) asks for more than one
  authenticated queue worker. The spend-admitted `rein-aharness metered-once`
  owner should claim as `rein-aharness-metered@railiance01`, separate from the
  running `rein-aharness@railiance01` claim loop. This keeps paid-run
  attribution distinct at the API.
- Message `6e694682` (SECRETS-WP-0011-T04) asks for the worker token to be held
  in OpenBao and synced by the existing `openbao-activity-core` store, instead
  of the hand-generated `openssl rand` value in
  `docs/deploy-ops-run-queue-railiance.md`. secrets-engine will catalog a
  read-only lane for the metered worker's path only. It will never read, copy
  or mint the token.

WP-0036 bound one token to one configured identity. This workplan keeps that
guarantee and extends it to a set of identities.

## Boundary rules

- Each token is bound to exactly one worker identity. The body `worker_id`
  must still match it. No token is shared, and nothing falls back implicitly.
- Worker tokens never equal the operator token, and operator/SSO credentials
  stay out of worker mutations (WP-0036).
- One OpenBao path per worker identity, so a consumer lane can be granted
  exactly one worker's token.
- Minting and writing a token is founder-attended OpenBao work. The store's
  role policy belongs to railiance-platform. No token value passes through
  Git, State Hub, chat, or an agent shell.

## Token-to-identity map in the worker API

```task
id: ACTIVITY-WP-0039-T01
status: done
priority: high
state_hub_task_id: "a82a498a-0673-5a29-94ae-0055672f8e4e"
```

Add `ACTIVITY_CORE_WORKERS`, a non-secret comma-separated list of
`worker_id=ENV_NAME` entries. Each ENV_NAME must be `ACTIVITY_CORE_WORKER_TOKEN`
or `ACTIVITY_CORE_WORKER_TOKEN_<SUFFIX>`. When the list is unset, the legacy
`ACTIVITY_CORE_WORKER_ID` + `ACTIVITY_CORE_WORKER_TOKEN` pair behaves exactly as
before. Worker mutations fail closed with 503 when the configuration is
ambiguous. That covers a duplicate identity, a missing or shared token, a token
equal to the operator token, an env name outside the pattern, and a legacy
identity missing from the list. A broken worker configuration does not lock out
operator or SSO reads.

Implemented in `src/activity_core/ops_runs_api.py` (`worker_credentials`).
Tests in `tests/test_ops_runs_api.py` cover per-identity binding, a
cross-identity body rejected with 403, an unknown token rejected with 401, each
fail-closed case, and operator access under broken worker configuration.

## Declare OpenBao paths and the ExternalSecret

```task
id: ACTIVITY-WP-0039-T02
status: wait
priority: high
state_hub_task_id: "1400bf41-e00a-56cc-9ccf-a1cf023b220a"
```

Paths, one per identity, each with field `token`:

- `platform/workloads/activity-core/ops-run-workers/rein-aharness-railiance01`
- `platform/workloads/activity-core/ops-run-workers/rein-aharness-metered-railiance01`

`k8s/railiance/15-externalsecret-worker-tokens.yaml` merges them into
`actcore-runtime-secret` as `ACTIVITY_CORE_WORKER_TOKEN` and
`ACTIVITY_CORE_WORKER_TOKEN_METERED`. The manifest is in the repo but not
applied.

Waiting on railiance-platform to add both exact paths to the
`activity-core-eso` role policy (RPF-WP-0045 pattern), and on secrets-engine to
confirm the metered path for its catalog.

## Seed the tokens in OpenBao

```task
id: ACTIVITY-WP-0039-T03
status: wait
priority: high
state_hub_task_id: "73d7f032-5ec3-550e-83ac-b9852c09ace4"
```

Founder-attended, through `warden access openbao-platform-admin-login --exec`
(orientation section 5):

- **Metered worker:** mint a fresh random value directly into its path. No
  value exists today.
- **Claim-loop worker:** the founder chooses between two options.
  - (a) Move the current value into OpenBao. The running claim loop needs no
    change, but the value was hand-generated outside custody.
  - (b) Mint a fresh value and update the rein-aharness claim-loop
    configuration on railiance01 (user tegwick) in the same window. This means
    one coordinated restart.

Done when both paths hold a value and no value has been printed or logged.

**Decision 2026-09-23 (founder): option (b).** Both tokens are minted fresh, so
the hand-generated claim-loop value is retired at cutover. The reviewed script
`scripts/wp0039-seed-worker-tokens.sh` does the minting. It runs silently, never
overwrites an existing path, and exits 0 when both paths are verified, 3 on an
unusable existing path, 4 on a write failure, and 5 on a verification failure.
It generates each value inside a pipe and sends it to `bao kv put ... token=-`
on stdin. The founder runs it in their own terminal:

```bash
BAO_ADDR=http://127.0.0.1:18200 VAULT_ADDR=http://127.0.0.1:18200 \
  warden access openbao-platform-admin-login --exec -- \
  sh scripts/wp0039-seed-worker-tokens.sh
```

Read warden's printed line rather than its exit code (orientation section 5).
Minting does not depend on T02, but ESO cannot read the paths until the policy
lands.

## Roll out and prove both identities

```task
id: ACTIVITY-WP-0039-T04
status: wait
priority: high
state_hub_task_id: "13e90456-02ff-52e7-9296-1fd80761fab4"
```

Needs the founder's go-ahead (`ADMINISTER @ realm:kubernetes/railiance01`,
`activation=APPROVED`) and a CPU headroom check first (orientation section 4).
Apply the ExternalSecret, add `ACTIVITY_CORE_WORKERS` to
`actcore-runtime-config`, deploy the new image, and restart only the API.
Prove four things:

- The claim loop still polls with HTTP 200.
- The metered token authenticates as `rein-aharness-metered@railiance01`.
- A metered token paired with the loop identity is rejected with HTTP 403.
- The previous hand-set Secret key is now owned by ESO.

Cutover order for option (b), which keeps the claim loop's gap to seconds.
The pods read `actcore-runtime-secret` only at start.

1. Apply `15-externalsecret-worker-tokens.yaml`, then force a refresh. The
   Secret now holds the new tokens, and the running API still uses the old one.
2. On railiance01, write the new `ACTIVITY_CORE_WORKER_TOKEN` from the Secret
   into `~/.config/rein-aharness/claim-loop.env`. Use a go-template read piped
   into the file, and never print the value.
3. Apply `ACTIVITY_CORE_WORKERS` to `actcore-runtime-config`, then restart
   `deployment/actcore-api`.
4. Restart `rein-aharness-claim-loop.service`, and confirm its first poll
   returns HTTP 200.

Report the revision to secrets-engine on threads `914853d9` and `6e694682`.