activity-core/workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md

---
id: ACTIVITY-WP-0039
type: workplan
title: "Multiple queue worker identities with OpenBao-custodied tokens"
domain: infotech
repo: activity-core
status: finished
flavor: implementation
owner: claude-code
topic_slug: activity-core
priority: high
created: "2026-09-23"
updated: "2026-09-23"
related:
  - ACT-ADR-005
  - ACTIVITY-WP-0036
  - SECRETS-WP-0009-T03
  - SECRETS-WP-0011-T04
  - RPF-WP-0045
state_hub_workstream_id: "053ef7c6-c2cf-5f50-aabf-09034c66d1ce"
---

# Multiple Queue Worker Identities with OpenBao-Custodied Tokens

## Origin

Two requests from secrets-engine, both approved on their side on 2026-09-23:

- Message `914853d9` (SECRETS-WP-0009-T03) asks for more than one
  authenticated queue worker. The spend-admitted `rein-aharness metered-once`
  owner should claim as `rein-aharness-metered@railiance01`, separate from the
  running `rein-aharness@railiance01` claim loop. This keeps paid-run
  attribution distinct at the API.
- Message `6e694682` (SECRETS-WP-0011-T04) asks for the worker token to be held
  in OpenBao and synced by the existing `openbao-activity-core` store, instead
  of the hand-generated `openssl rand` value in
  `docs/deploy-ops-run-queue-railiance.md`. secrets-engine will catalog a
  read-only lane for the metered worker's path only. It will never read, copy
  or mint the token.

WP-0036 bound one token to one configured identity. This workplan keeps that
guarantee and extends it to a set of identities.

## Boundary rules

- Each token is bound to exactly one worker identity. The body `worker_id`
  must still match it. No token is shared, and nothing falls back implicitly.
- Worker tokens never equal the operator token, and operator/SSO credentials
  stay out of worker mutations (WP-0036).
- One OpenBao path per worker identity, so a consumer lane can be granted
  exactly one worker's token.
- Minting and writing a token is founder-attended OpenBao work. The store's
  role policy belongs to railiance-platform. No token value passes through
  Git, State Hub, chat, or an agent shell.

## Token-to-identity map in the worker API

```task
id: ACTIVITY-WP-0039-T01
status: done
priority: high
state_hub_task_id: "a82a498a-0673-5a29-94ae-0055672f8e4e"
```

Add `ACTIVITY_CORE_WORKERS`, a non-secret comma-separated list of
`worker_id=ENV_NAME` entries. Each ENV_NAME must be `ACTIVITY_CORE_WORKER_TOKEN`
or `ACTIVITY_CORE_WORKER_TOKEN_<SUFFIX>`. When the list is unset, the legacy
`ACTIVITY_CORE_WORKER_ID` + `ACTIVITY_CORE_WORKER_TOKEN` pair behaves exactly as
before. Worker mutations fail closed with 503 when the configuration is
ambiguous. That covers a duplicate identity, a missing or shared token, a token
equal to the operator token, an env name outside the pattern, and a legacy
identity missing from the list. A broken worker configuration does not lock out
operator or SSO reads.

Implemented in `src/activity_core/ops_runs_api.py` (`worker_credentials`).
Tests in `tests/test_ops_runs_api.py` cover per-identity binding, a
cross-identity body rejected with 403, an unknown token rejected with 401, each
fail-closed case, and operator access under broken worker configuration.

## Declare OpenBao paths and the ExternalSecret

```task
id: ACTIVITY-WP-0039-T02
status: done
priority: high
state_hub_task_id: "1400bf41-e00a-56cc-9ccf-a1cf023b220a"
```

Paths, one per identity, each with field `token`:

- `platform/workloads/activity-core/ops-run-workers/rein-aharness-railiance01`
- `platform/workloads/activity-core/ops-run-workers/rein-aharness-metered-railiance01`

`k8s/railiance/15-externalsecret-worker-tokens.yaml` merges them into
`actcore-runtime-secret` as `ACTIVITY_CORE_WORKER_TOKEN` and
`ACTIVITY_CORE_WORKER_TOKEN_METERED`. The manifest is in the repo but not
applied.

Waiting on railiance-platform to add both exact paths to the
`activity-core-eso` role policy (RPF-WP-0045 pattern), and on secrets-engine to
confirm the metered path for its catalog.

2026-09-23: secrets-engine confirmed the metered path (message `a0e9afaf`). It
catalogs only `.../rein-aharness-metered-railiance01` as lane
`activity-core-metered-worker-token`, read through its own exact-path AppRole.
Only the railiance-platform policy change remains. The paths are requested in
activity-core's message to railiance-platform. The HCL is two `read` stanzas
on `platform/data/workloads/activity-core/ops-run-workers/<slug>`.

## Seed the tokens in OpenBao

```task
id: ACTIVITY-WP-0039-T03
status: done
priority: high
state_hub_task_id: "73d7f032-5ec3-550e-83ac-b9852c09ace4"
```

Founder-attended, through `warden access openbao-platform-admin-login --exec`
(orientation section 5):

- **Metered worker:** mint a fresh random value directly into its path. No
  value exists today.
- **Claim-loop worker:** the founder chooses between two options.
  - (a) Move the current value into OpenBao. The running claim loop needs no
    change, but the value was hand-generated outside custody.
  - (b) Mint a fresh value and update the rein-aharness claim-loop
    configuration on railiance01 (user tegwick) in the same window. This means
    one coordinated restart.

Done when both paths hold a value and no value has been printed or logged.

**Decision 2026-09-23 (founder): option (b).** Both tokens are minted fresh, so
the hand-generated claim-loop value is retired at cutover. The reviewed script
`scripts/wp0039-seed-worker-tokens.sh` does the minting. It runs silently, never
overwrites an existing path, and exits 0 when both paths are verified, 3 on an
unusable existing path, 4 on a write failure, and 5 on a verification failure.
It generates each value inside a pipe and sends it to `bao kv put ... token=-`
on stdin. The founder runs it in their own terminal:

```bash
BAO_ADDR=http://127.0.0.1:18200 VAULT_ADDR=http://127.0.0.1:18200 \
  WP0039_STATUS=$HOME/.local/state/wp0039-seed.status \
  warden access openbao-platform-admin-login --exec -- \
  sh /home/worsch/activity-core/scripts/wp0039-seed-worker-tokens.sh   # absolute path
```

The first founder run on 2026-09-23 failed closed ("returned a failure or
unexpected output"). warden discards the child's output, so the script now
writes a non-secret step trace to `WP0039_STATUS`. The trace holds the failing
step, the exit code, and bao's error text, never the value. The script was
tested against a stub `bao`: silent, and a no-op on rerun.

The failed runs never started the script. The founder's terminal was outside the
repository, so the relative script path did not resolve, and no status file
was written. With the absolute path, the founder's run on 2026-09-23 succeeded.
warden reported no failure and no revocation warning. The status trace shows
both paths absent beforehand, then written, then verified with a 64-character
token field, then exit 0. No value was printed or logged.

Read warden's printed line rather than its exit code (orientation section 5).
Minting does not depend on T02, but ESO cannot read the paths until the policy
lands.

## Roll out and prove both identities

```task
id: ACTIVITY-WP-0039-T04
status: done
priority: high
state_hub_task_id: "13e90456-02ff-52e7-9296-1fd80761fab4"
```

Needs the founder's go-ahead (`ADMINISTER @ realm:kubernetes/railiance01`,
`activation=APPROVED`) and a CPU headroom check first (orientation section 4).
Apply the ExternalSecret, add `ACTIVITY_CORE_WORKERS` to
`actcore-runtime-config`, deploy the new image, and restart only the API.
Prove four things:

- The claim loop still polls with HTTP 200.
- The metered token authenticates as `rein-aharness-metered@railiance01`.
- A metered token paired with the loop identity is rejected with HTTP 403.
- The previous hand-set Secret key is now owned by ESO.

Cutover order for option (b), which keeps the claim loop's gap to seconds.
The pods read `actcore-runtime-secret` only at start.

1. Apply `15-externalsecret-worker-tokens.yaml`, then force a refresh. The
   Secret now holds the new tokens, and the running API still uses the old one.
2. On railiance01, write the new `ACTIVITY_CORE_WORKER_TOKEN` from the Secret
   into `~/.config/rein-aharness/claim-loop.env`. Use a go-template read piped
   into the file, and never print the value.
3. Apply `ACTIVITY_CORE_WORKERS` to `actcore-runtime-config`, then restart
   `deployment/actcore-api`.
4. Restart `rein-aharness-claim-loop.service`, and confirm its first poll
   returns HTTP 200.

Report the revision to secrets-engine on threads `914853d9` and `6e694682`.

**Cutover attempt, 2026-09-23 (founder go-ahead, build mode).**
T02 was done by railiance-platform at 18:06:29Z (CCR-2026-0029/0030, commit
`fe1665d`).

1. The ExternalSecret `actcore-ops-run-worker-tokens` is applied. Both keys
   synced at 20:35:35Z, and ESO now owns `ACTIVITY_CORE_WORKER_TOKEN`.
2. No `ops_run` was claimed. The claim-loop env file was backed up
   (`claim-loop.env.bak-wp0039-20260923T203559`) and given the new loop
   token without printing it.
3. The image `activity-core:wp0039-20260923` (built from `main` at `1a20c85`)
   was imported. The ConfigMap with `ACTIVITY_CORE_WORKERS` and the API
   Deployment were applied, and the claim loop was restarted.
4. **The new API returned HTTP 500** (`column ops_runs.repository_grant does
   not exist`). Production is at alembic `0009`, and `main` needs `0010` (the
   WP-0038 columns). The running `fi-publication-20260914` image contains no
   WP-0038 code, so WP-0038 was never deployed. The API was rolled back to
   `fi-publication-20260914`, and the loop claimed again with HTTP 200 at
   20:39:23Z. The outage lasted about 1 minute, and no run was affected.

Live state now: the option-(b) loop token rotation is complete. The old code
reads the ESO-synced `ACTIVITY_CORE_WORKER_TOKEN`, which is the new value.
`ACTIVITY_CORE_WORKERS` is in the ConfigMap and is ignored by the old image.
The metered identity is not live yet.

Remaining: run migration `0010`, which is additive (two nullable columns).
That also takes WP-0038's close reconciliation live for the first time. Then
redeploy the API on `wp0039-20260923`, and prove the metered identity and the
403 cross-identity rejection. Both need the founder's decision because they go
beyond the worker-identity change.

**Closed 2026-09-24.** On 2026-09-24 the founder ran migration `0010`, and the
API was redeployed on `wp0039-20260923`. The loop claims with 200 as
`rein-aharness@railiance01`. The metered token authenticates as
`rein-aharness-metered@railiance01` (200, nothing claimed), and paired with the
loop identity it is rejected with 403. This also took WP-0038 live for the
first time. Evidence:
`docs/evidence/ACTIVITY-WP-0039-worker-identity-cutover-2026-09-23.md`.