--- id: ACTIVITY-WP-0039 type: workplan title: "Multiple queue worker identities with OpenBao-custodied tokens" domain: infotech repo: activity-core status: active flavor: implementation owner: claude-code topic_slug: activity-core priority: high created: "2026-09-23" updated: "2026-09-23" related: - ACT-ADR-005 - ACTIVITY-WP-0036 - SECRETS-WP-0009-T03 - SECRETS-WP-0011-T04 - RPF-WP-0045 state_hub_workstream_id: "053ef7c6-c2cf-5f50-aabf-09034c66d1ce" --- # Multiple Queue Worker Identities with OpenBao-Custodied Tokens ## Origin Two requests from secrets-engine, both approved on their side on 2026-09-23: - Message `914853d9` (SECRETS-WP-0009-T03) asks for more than one authenticated queue worker. The spend-admitted `rein-aharness metered-once` owner should claim as `rein-aharness-metered@railiance01`, separate from the running `rein-aharness@railiance01` claim loop. This keeps paid-run attribution distinct at the API. - Message `6e694682` (SECRETS-WP-0011-T04) asks for the worker token to be held in OpenBao and synced by the existing `openbao-activity-core` store, instead of the hand-generated `openssl rand` value in `docs/deploy-ops-run-queue-railiance.md`. secrets-engine will catalog a read-only lane for the metered worker's path only. It will never read, copy or mint the token. WP-0036 bound one token to one configured identity. This workplan keeps that guarantee and extends it to a set of identities. ## Boundary rules - Each token is bound to exactly one worker identity. The body `worker_id` must still match it. No token is shared, and nothing falls back implicitly. - Worker tokens never equal the operator token, and operator/SSO credentials stay out of worker mutations (WP-0036). - One OpenBao path per worker identity, so a consumer lane can be granted exactly one worker's token. - Minting and writing a token is founder-attended OpenBao work. The store's role policy belongs to railiance-platform. No token value passes through Git, State Hub, chat, or an agent shell. ## Token-to-identity map in the worker API ```task id: ACTIVITY-WP-0039-T01 status: done priority: high state_hub_task_id: "a82a498a-0673-5a29-94ae-0055672f8e4e" ``` Add `ACTIVITY_CORE_WORKERS`, a non-secret comma-separated list of `worker_id=ENV_NAME` entries. Each ENV_NAME must be `ACTIVITY_CORE_WORKER_TOKEN` or `ACTIVITY_CORE_WORKER_TOKEN_`. When the list is unset, the legacy `ACTIVITY_CORE_WORKER_ID` + `ACTIVITY_CORE_WORKER_TOKEN` pair behaves exactly as before. Worker mutations fail closed with 503 when the configuration is ambiguous. That covers a duplicate identity, a missing or shared token, a token equal to the operator token, an env name outside the pattern, and a legacy identity missing from the list. A broken worker configuration does not lock out operator or SSO reads. Implemented in `src/activity_core/ops_runs_api.py` (`worker_credentials`). Tests in `tests/test_ops_runs_api.py` cover per-identity binding, a cross-identity body rejected with 403, an unknown token rejected with 401, each fail-closed case, and operator access under broken worker configuration. ## Declare OpenBao paths and the ExternalSecret ```task id: ACTIVITY-WP-0039-T02 status: done priority: high state_hub_task_id: "1400bf41-e00a-56cc-9ccf-a1cf023b220a" ``` Paths, one per identity, each with field `token`: - `platform/workloads/activity-core/ops-run-workers/rein-aharness-railiance01` - `platform/workloads/activity-core/ops-run-workers/rein-aharness-metered-railiance01` `k8s/railiance/15-externalsecret-worker-tokens.yaml` merges them into `actcore-runtime-secret` as `ACTIVITY_CORE_WORKER_TOKEN` and `ACTIVITY_CORE_WORKER_TOKEN_METERED`. The manifest is in the repo but not applied. Waiting on railiance-platform to add both exact paths to the `activity-core-eso` role policy (RPF-WP-0045 pattern), and on secrets-engine to confirm the metered path for its catalog. 2026-09-23: secrets-engine confirmed the metered path (message `a0e9afaf`). It catalogs only `.../rein-aharness-metered-railiance01` as lane `activity-core-metered-worker-token`, read through its own exact-path AppRole. Only the railiance-platform policy change remains. The paths are requested in activity-core's message to railiance-platform. The HCL is two `read` stanzas on `platform/data/workloads/activity-core/ops-run-workers/`. ## Seed the tokens in OpenBao ```task id: ACTIVITY-WP-0039-T03 status: done priority: high state_hub_task_id: "73d7f032-5ec3-550e-83ac-b9852c09ace4" ``` Founder-attended, through `warden access openbao-platform-admin-login --exec` (orientation section 5): - **Metered worker:** mint a fresh random value directly into its path. No value exists today. - **Claim-loop worker:** the founder chooses between two options. - (a) Move the current value into OpenBao. The running claim loop needs no change, but the value was hand-generated outside custody. - (b) Mint a fresh value and update the rein-aharness claim-loop configuration on railiance01 (user tegwick) in the same window. This means one coordinated restart. Done when both paths hold a value and no value has been printed or logged. **Decision 2026-09-23 (founder): option (b).** Both tokens are minted fresh, so the hand-generated claim-loop value is retired at cutover. The reviewed script `scripts/wp0039-seed-worker-tokens.sh` does the minting. It runs silently, never overwrites an existing path, and exits 0 when both paths are verified, 3 on an unusable existing path, 4 on a write failure, and 5 on a verification failure. It generates each value inside a pipe and sends it to `bao kv put ... token=-` on stdin. The founder runs it in their own terminal: ```bash BAO_ADDR=http://127.0.0.1:18200 VAULT_ADDR=http://127.0.0.1:18200 \ WP0039_STATUS=$HOME/.local/state/wp0039-seed.status \ warden access openbao-platform-admin-login --exec -- \ sh /home/worsch/activity-core/scripts/wp0039-seed-worker-tokens.sh # absolute path ``` The first founder run on 2026-09-23 failed closed ("returned a failure or unexpected output"). warden discards the child's output, so the script now writes a non-secret step trace to `WP0039_STATUS`. The trace holds the failing step, the exit code, and bao's error text, never the value. The script was tested against a stub `bao`: silent, and a no-op on rerun. The failed runs never started the script. The founder's terminal was outside the repository, so the relative script path did not resolve, and no status file was written. With the absolute path, the founder's run on 2026-09-23 succeeded. warden reported no failure and no revocation warning. The status trace shows both paths absent beforehand, then written, then verified with a 64-character token field, then exit 0. No value was printed or logged. Read warden's printed line rather than its exit code (orientation section 5). Minting does not depend on T02, but ESO cannot read the paths until the policy lands. ## Roll out and prove both identities ```task id: ACTIVITY-WP-0039-T04 status: progress priority: high state_hub_task_id: "13e90456-02ff-52e7-9296-1fd80761fab4" ``` Needs the founder's go-ahead (`ADMINISTER @ realm:kubernetes/railiance01`, `activation=APPROVED`) and a CPU headroom check first (orientation section 4). Apply the ExternalSecret, add `ACTIVITY_CORE_WORKERS` to `actcore-runtime-config`, deploy the new image, and restart only the API. Prove four things: - The claim loop still polls with HTTP 200. - The metered token authenticates as `rein-aharness-metered@railiance01`. - A metered token paired with the loop identity is rejected with HTTP 403. - The previous hand-set Secret key is now owned by ESO. Cutover order for option (b), which keeps the claim loop's gap to seconds. The pods read `actcore-runtime-secret` only at start. 1. Apply `15-externalsecret-worker-tokens.yaml`, then force a refresh. The Secret now holds the new tokens, and the running API still uses the old one. 2. On railiance01, write the new `ACTIVITY_CORE_WORKER_TOKEN` from the Secret into `~/.config/rein-aharness/claim-loop.env`. Use a go-template read piped into the file, and never print the value. 3. Apply `ACTIVITY_CORE_WORKERS` to `actcore-runtime-config`, then restart `deployment/actcore-api`. 4. Restart `rein-aharness-claim-loop.service`, and confirm its first poll returns HTTP 200. Report the revision to secrets-engine on threads `914853d9` and `6e694682`. **Cutover attempt, 2026-09-23 (founder go-ahead, build mode).** T02 was done by railiance-platform at 18:06:29Z (CCR-2026-0029/0030, commit `fe1665d`). 1. The ExternalSecret `actcore-ops-run-worker-tokens` is applied. Both keys synced at 20:35:35Z, and ESO now owns `ACTIVITY_CORE_WORKER_TOKEN`. 2. No `ops_run` was claimed. The claim-loop env file was backed up (`claim-loop.env.bak-wp0039-20260923T203559`) and given the new loop token without printing it. 3. The image `activity-core:wp0039-20260923` (built from `main` at `1a20c85`) was imported. The ConfigMap with `ACTIVITY_CORE_WORKERS` and the API Deployment were applied, and the claim loop was restarted. 4. **The new API returned HTTP 500** (`column ops_runs.repository_grant does not exist`). Production is at alembic `0009`, and `main` needs `0010` (the WP-0038 columns). The running `fi-publication-20260914` image contains no WP-0038 code, so WP-0038 was never deployed. The API was rolled back to `fi-publication-20260914`, and the loop claimed again with HTTP 200 at 20:39:23Z. The outage lasted about 1 minute, and no run was affected. Live state now: the option-(b) loop token rotation is complete. The old code reads the ESO-synced `ACTIVITY_CORE_WORKER_TOKEN`, which is the new value. `ACTIVITY_CORE_WORKERS` is in the ConfigMap and is ignored by the old image. The metered identity is not live yet. Remaining: run migration `0010`, which is additive (two nullable columns). That also takes WP-0038's close reconciliation live for the first time. Then redeploy the API on `wp0039-20260923`, and prove the metered identity and the 403 cross-identity rejection. Both need the founder's decision because they go beyond the worker-identity change.