--- id: ACTIVITY-WP-0039 type: workplan title: "Multiple queue worker identities with OpenBao-custodied tokens" domain: infotech repo: activity-core status: active flavor: implementation owner: claude-code topic_slug: activity-core priority: high created: "2026-09-23" updated: "2026-09-23" related: - ACT-ADR-005 - ACTIVITY-WP-0036 - SECRETS-WP-0009-T03 - SECRETS-WP-0011-T04 - RPF-WP-0045 state_hub_workstream_id: "053ef7c6-c2cf-5f50-aabf-09034c66d1ce" --- # Multiple Queue Worker Identities with OpenBao-Custodied Tokens ## Origin Two requests from secrets-engine, both approved on their side on 2026-09-23: - Message `914853d9` (SECRETS-WP-0009-T03) asks for more than one authenticated queue worker. The spend-admitted `rein-aharness metered-once` owner should claim as `rein-aharness-metered@railiance01`, separate from the running `rein-aharness@railiance01` claim loop. This keeps paid-run attribution distinct at the API. - Message `6e694682` (SECRETS-WP-0011-T04) asks for the worker token to be held in OpenBao and synced by the existing `openbao-activity-core` store, instead of the hand-generated `openssl rand` value in `docs/deploy-ops-run-queue-railiance.md`. secrets-engine will catalog a read-only lane for the metered worker's path only. It will never read, copy or mint the token. WP-0036 bound one token to one configured identity. This workplan keeps that guarantee and extends it to a set of identities. ## Boundary rules - Each token is bound to exactly one worker identity. The body `worker_id` must still match it. No token is shared, and nothing falls back implicitly. - Worker tokens never equal the operator token, and operator/SSO credentials stay out of worker mutations (WP-0036). - One OpenBao path per worker identity, so a consumer lane can be granted exactly one worker's token. - Minting and writing a token is founder-attended OpenBao work. The store's role policy belongs to railiance-platform. No token value passes through Git, State Hub, chat, or an agent shell. ## Token-to-identity map in the worker API ```task id: ACTIVITY-WP-0039-T01 status: done priority: high state_hub_task_id: "a82a498a-0673-5a29-94ae-0055672f8e4e" ``` Add `ACTIVITY_CORE_WORKERS`, a non-secret comma-separated list of `worker_id=ENV_NAME` entries. Each ENV_NAME must be `ACTIVITY_CORE_WORKER_TOKEN` or `ACTIVITY_CORE_WORKER_TOKEN_`. When the list is unset, the legacy `ACTIVITY_CORE_WORKER_ID` + `ACTIVITY_CORE_WORKER_TOKEN` pair behaves exactly as before. Worker mutations fail closed with 503 when the configuration is ambiguous. That covers a duplicate identity, a missing or shared token, a token equal to the operator token, an env name outside the pattern, and a legacy identity missing from the list. A broken worker configuration does not lock out operator or SSO reads. Implemented in `src/activity_core/ops_runs_api.py` (`worker_credentials`). Tests in `tests/test_ops_runs_api.py` cover per-identity binding, a cross-identity body rejected with 403, an unknown token rejected with 401, each fail-closed case, and operator access under broken worker configuration. ## Declare OpenBao paths and the ExternalSecret ```task id: ACTIVITY-WP-0039-T02 status: wait priority: high state_hub_task_id: "1400bf41-e00a-56cc-9ccf-a1cf023b220a" ``` Paths, one per identity, each with field `token`: - `platform/workloads/activity-core/ops-run-workers/rein-aharness-railiance01` - `platform/workloads/activity-core/ops-run-workers/rein-aharness-metered-railiance01` `k8s/railiance/15-externalsecret-worker-tokens.yaml` merges them into `actcore-runtime-secret` as `ACTIVITY_CORE_WORKER_TOKEN` and `ACTIVITY_CORE_WORKER_TOKEN_METERED`. The manifest is in the repo but not applied. Waiting on railiance-platform to add both exact paths to the `activity-core-eso` role policy (RPF-WP-0045 pattern), and on secrets-engine to confirm the metered path for its catalog. ## Seed the tokens in OpenBao ```task id: ACTIVITY-WP-0039-T03 status: wait priority: high state_hub_task_id: "73d7f032-5ec3-550e-83ac-b9852c09ace4" ``` Founder-attended, through `warden access openbao-platform-admin-login --exec` (orientation section 5): - **Metered worker:** mint a fresh random value directly into its path. No value exists today. - **Claim-loop worker:** the founder chooses between two options. - (a) Move the current value into OpenBao. The running claim loop needs no change, but the value was hand-generated outside custody. - (b) Mint a fresh value and update the rein-aharness claim-loop configuration on railiance01 (user tegwick) in the same window. This means one coordinated restart. Done when both paths hold a value and no value has been printed or logged. **Decision 2026-09-23 (founder): option (b).** Both tokens are minted fresh, so the hand-generated claim-loop value is retired at cutover. The reviewed script `scripts/wp0039-seed-worker-tokens.sh` does the minting. It runs silently, never overwrites an existing path, and exits 0 when both paths are verified, 3 on an unusable existing path, 4 on a write failure, and 5 on a verification failure. It generates each value inside a pipe and sends it to `bao kv put ... token=-` on stdin. The founder runs it in their own terminal: ```bash BAO_ADDR=http://127.0.0.1:18200 VAULT_ADDR=http://127.0.0.1:18200 \ WP0039_STATUS=$HOME/.local/state/wp0039-seed.status \ warden access openbao-platform-admin-login --exec -- \ sh scripts/wp0039-seed-worker-tokens.sh ``` The first founder run on 2026-09-23 failed closed ("returned a failure or unexpected output"). warden discards the child's output, so the script now writes a non-secret step trace to `WP0039_STATUS`. The trace holds the failing step, the exit code, and bao's error text, never the value. The script was tested against a stub `bao`: silent, and a no-op on rerun. Read warden's printed line rather than its exit code (orientation section 5). Minting does not depend on T02, but ESO cannot read the paths until the policy lands. ## Roll out and prove both identities ```task id: ACTIVITY-WP-0039-T04 status: wait priority: high state_hub_task_id: "13e90456-02ff-52e7-9296-1fd80761fab4" ``` Needs the founder's go-ahead (`ADMINISTER @ realm:kubernetes/railiance01`, `activation=APPROVED`) and a CPU headroom check first (orientation section 4). Apply the ExternalSecret, add `ACTIVITY_CORE_WORKERS` to `actcore-runtime-config`, deploy the new image, and restart only the API. Prove four things: - The claim loop still polls with HTTP 200. - The metered token authenticates as `rein-aharness-metered@railiance01`. - A metered token paired with the loop identity is rejected with HTTP 403. - The previous hand-set Secret key is now owned by ESO. Cutover order for option (b), which keeps the claim loop's gap to seconds. The pods read `actcore-runtime-secret` only at start. 1. Apply `15-externalsecret-worker-tokens.yaml`, then force a refresh. The Secret now holds the new tokens, and the running API still uses the old one. 2. On railiance01, write the new `ACTIVITY_CORE_WORKER_TOKEN` from the Secret into `~/.config/rein-aharness/claim-loop.env`. Use a go-template read piped into the file, and never print the value. 3. Apply `ACTIVITY_CORE_WORKERS` to `actcore-runtime-config`, then restart `deployment/actcore-api`. 4. Restart `rein-aharness-claim-loop.service`, and confirm its first poll returns HTTP 200. Report the revision to secrets-engine on threads `914853d9` and `6e694682`.