Bind multiple queue worker identities, one token each (WP-0039-T01)

ACTIVITY_CORE_WORKERS maps worker_id=ENV_NAME, where each token env must be
ACTIVITY_CORE_WORKER_TOKEN[_SUFFIX]. Without the map, the legacy single pair
behaves exactly as before. Duplicate identities, missing or shared tokens, a
token equal to the operator token, and an unlisted legacy identity all fail
worker mutations closed with 503. Operator/SSO reads keep working.

Declare per-identity OpenBao paths and an ExternalSecret, not yet applied.
The policy, seeding and rollout are waiting tasks T02-T04, answering
secrets-engine SECRETS-WP-0009-T03 and SECRETS-WP-0011-T04.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 151606@bnt-lap001
Assistant-Session: 3c0a4ad5-bb8b-4bf7-b9f0-fa5f29204e48

workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md

@@ -0,0 +1,141 @@
+---
+id: ACTIVITY-WP-0039
+type: workplan
+title: "Multiple queue worker identities with OpenBao-custodied tokens"
+domain: infotech
+repo: activity-core
+status: active
+flavor: implementation
+owner: claude-code
+topic_slug: activity-core
+priority: high
+created: "2026-09-23"
+updated: "2026-09-23"
+related:
+  - ACT-ADR-005
+  - ACTIVITY-WP-0036
+  - SECRETS-WP-0009-T03
+  - SECRETS-WP-0011-T04
+  - RPF-WP-0045
+---
+
+# Multiple Queue Worker Identities with OpenBao-Custodied Tokens
+
+## Origin
+
+Two requests from secrets-engine, both approved on their side on 2026-09-23:
+
+- Message `914853d9` (SECRETS-WP-0009-T03) asks for more than one
+  authenticated queue worker. The spend-admitted `rein-aharness metered-once`
+  owner should claim as `rein-aharness-metered@railiance01`, separate from the
+  running `rein-aharness@railiance01` claim loop. This keeps paid-run
+  attribution distinct at the API.
+- Message `6e694682` (SECRETS-WP-0011-T04) asks for the worker token to be held
+  in OpenBao and synced by the existing `openbao-activity-core` store, instead
+  of the hand-generated `openssl rand` value in
+  `docs/deploy-ops-run-queue-railiance.md`. secrets-engine will catalog a
+  read-only lane for the metered worker's path only. It will never read, copy
+  or mint the token.
+
+WP-0036 bound one token to one configured identity. This workplan keeps that
+guarantee and extends it to a set of identities.
+
+## Boundary rules
+
+- Each token is bound to exactly one worker identity. The body `worker_id`
+  must still match it. No token is shared, and nothing falls back implicitly.
+- Worker tokens never equal the operator token, and operator/SSO credentials
+  stay out of worker mutations (WP-0036).
+- One OpenBao path per worker identity, so a consumer lane can be granted
+  exactly one worker's token.
+- Minting and writing a token is founder-attended OpenBao work. The store's
+  role policy belongs to railiance-platform. No token value passes through
+  Git, State Hub, chat, or an agent shell.
+
+## Token-to-identity map in the worker API
+
+```task
+id: ACTIVITY-WP-0039-T01
+status: done
+priority: high
+```
+
+Add `ACTIVITY_CORE_WORKERS`, a non-secret comma-separated list of
+`worker_id=ENV_NAME` entries. Each ENV_NAME must be `ACTIVITY_CORE_WORKER_TOKEN`
+or `ACTIVITY_CORE_WORKER_TOKEN_<SUFFIX>`. When the list is unset, the legacy
+`ACTIVITY_CORE_WORKER_ID` + `ACTIVITY_CORE_WORKER_TOKEN` pair behaves exactly as
+before. Worker mutations fail closed with 503 when the configuration is
+ambiguous. That covers a duplicate identity, a missing or shared token, a token
+equal to the operator token, an env name outside the pattern, and a legacy
+identity missing from the list. A broken worker configuration does not lock out
+operator or SSO reads.
+
+Implemented in `src/activity_core/ops_runs_api.py` (`worker_credentials`).
+Tests in `tests/test_ops_runs_api.py` cover per-identity binding, a
+cross-identity body rejected with 403, an unknown token rejected with 401, each
+fail-closed case, and operator access under broken worker configuration.
+
+## Declare OpenBao paths and the ExternalSecret
+
+```task
+id: ACTIVITY-WP-0039-T02
+status: wait
+priority: high
+```
+
+Paths, one per identity, each with field `token`:
+
+- `platform/workloads/activity-core/ops-run-workers/rein-aharness-railiance01`
+- `platform/workloads/activity-core/ops-run-workers/rein-aharness-metered-railiance01`
+
+`k8s/railiance/15-externalsecret-worker-tokens.yaml` merges them into
+`actcore-runtime-secret` as `ACTIVITY_CORE_WORKER_TOKEN` and
+`ACTIVITY_CORE_WORKER_TOKEN_METERED`. The manifest is in the repo but not
+applied.
+
+Waiting on railiance-platform to add both exact paths to the
+`activity-core-eso` role policy (RPF-WP-0045 pattern), and on secrets-engine to
+confirm the metered path for its catalog.
+
+## Seed the tokens in OpenBao
+
+```task
+id: ACTIVITY-WP-0039-T03
+status: wait
+priority: high
+```
+
+Founder-attended, through `warden access openbao-platform-admin-login --exec`
+(orientation section 5):
+
+- **Metered worker:** mint a fresh random value directly into its path. No
+  value exists today.
+- **Claim-loop worker:** the founder chooses between two options.
+  - (a) Move the current value into OpenBao. The running claim loop needs no
+    change, but the value was hand-generated outside custody.
+  - (b) Mint a fresh value and update the rein-aharness claim-loop
+    configuration on railiance01 (user tegwick) in the same window. This means
+    one coordinated restart.
+
+Done when both paths hold a value and no value has been printed or logged.
+
+## Roll out and prove both identities
+
+```task
+id: ACTIVITY-WP-0039-T04
+status: wait
+priority: high
+```
+
+Needs the founder's go-ahead (`ADMINISTER @ realm:kubernetes/railiance01`,
+`activation=APPROVED`) and a CPU headroom check first (orientation section 4).
+Apply the ExternalSecret, add `ACTIVITY_CORE_WORKERS` to
+`actcore-runtime-config`, deploy the new image, and restart only the API.
+Prove four things:
+
+- The claim loop still polls with HTTP 200.
+- The metered token authenticates as `rein-aharness-metered@railiance01`.
+- A metered token paired with the loop identity is rejected with HTTP 403.
+- The previous hand-set Secret key is now owned by ESO.
+
+Report the revision to secrets-engine on threads `914853d9` and `6e694682`.