activity-core/workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md

id	type	title	domain	repo	status	flavor	owner	topic_slug	priority	created	updated	related	state_hub_workstream_id
ACTIVITY-WP-0039	workplan	Multiple queue worker identities with OpenBao-custodied tokens	infotech	activity-core	active	implementation	claude-code	activity-core	high	2026-09-23	2026-09-23	ACT-ADR-005 ACTIVITY-WP-0036 SECRETS-WP-0009-T03 SECRETS-WP-0011-T04 RPF-WP-0045	053ef7c6-c2cf-5f50-aabf-09034c66d1ce

Multiple Queue Worker Identities with OpenBao-Custodied Tokens

Origin

Two requests from secrets-engine, both approved on their side on 2026-09-23:

• Message 914853d9 (SECRETS-WP-0009-T03) asks for more than one authenticated queue worker. The spend-admitted rein-aharness metered-once owner should claim as rein-aharness-metered@railiance01, separate from the running rein-aharness@railiance01 claim loop. This keeps paid-run attribution distinct at the API.
• Message 6e694682 (SECRETS-WP-0011-T04) asks for the worker token to be held in OpenBao and synced by the existing openbao-activity-core store, instead of the hand-generated openssl rand value in docs/deploy-ops-run-queue-railiance.md. secrets-engine will catalog a read-only lane for the metered worker's path only. It will never read, copy or mint the token.

WP-0036 bound one token to one configured identity. This workplan keeps that guarantee and extends it to a set of identities.

Boundary rules

• Each token is bound to exactly one worker identity. The body worker_id must still match it. No token is shared, and nothing falls back implicitly.
• Worker tokens never equal the operator token, and operator/SSO credentials stay out of worker mutations (WP-0036).
• One OpenBao path per worker identity, so a consumer lane can be granted exactly one worker's token.
• Minting and writing a token is founder-attended OpenBao work. The store's role policy belongs to railiance-platform. No token value passes through Git, State Hub, chat, or an agent shell.

Token-to-identity map in the worker API

id: ACTIVITY-WP-0039-T01
status: done
priority: high
state_hub_task_id: "a82a498a-0673-5a29-94ae-0055672f8e4e"

Add ACTIVITY_CORE_WORKERS, a non-secret comma-separated list of worker_id=ENV_NAME entries. Each ENV_NAME must be ACTIVITY_CORE_WORKER_TOKEN or ACTIVITY_CORE_WORKER_TOKEN_<SUFFIX>. When the list is unset, the legacy ACTIVITY_CORE_WORKER_ID + ACTIVITY_CORE_WORKER_TOKEN pair behaves exactly as before. Worker mutations fail closed with 503 when the configuration is ambiguous. That covers a duplicate identity, a missing or shared token, a token equal to the operator token, an env name outside the pattern, and a legacy identity missing from the list. A broken worker configuration does not lock out operator or SSO reads.

Implemented in src/activity_core/ops_runs_api.py (worker_credentials). Tests in tests/test_ops_runs_api.py cover per-identity binding, a cross-identity body rejected with 403, an unknown token rejected with 401, each fail-closed case, and operator access under broken worker configuration.

Declare OpenBao paths and the ExternalSecret

id: ACTIVITY-WP-0039-T02
status: wait
priority: high
state_hub_task_id: "1400bf41-e00a-56cc-9ccf-a1cf023b220a"

Paths, one per identity, each with field token:

• platform/workloads/activity-core/ops-run-workers/rein-aharness-railiance01
• platform/workloads/activity-core/ops-run-workers/rein-aharness-metered-railiance01

k8s/railiance/15-externalsecret-worker-tokens.yaml merges them into actcore-runtime-secret as ACTIVITY_CORE_WORKER_TOKEN and ACTIVITY_CORE_WORKER_TOKEN_METERED. The manifest is in the repo but not applied.

Waiting on railiance-platform to add both exact paths to the activity-core-eso role policy (RPF-WP-0045 pattern), and on secrets-engine to confirm the metered path for its catalog.

Seed the tokens in OpenBao

id: ACTIVITY-WP-0039-T03
status: wait
priority: high
state_hub_task_id: "73d7f032-5ec3-550e-83ac-b9852c09ace4"

Founder-attended, through warden access openbao-platform-admin-login --exec (orientation section 5):

• Metered worker: mint a fresh random value directly into its path. No value exists today.
• Claim-loop worker: the founder chooses between two options.
  • (a) Move the current value into OpenBao. The running claim loop needs no change, but the value was hand-generated outside custody.
  • (b) Mint a fresh value and update the rein-aharness claim-loop configuration on railiance01 (user tegwick) in the same window. This means one coordinated restart.

Done when both paths hold a value and no value has been printed or logged.

Decision 2026-09-23 (founder): option (b). Both tokens are minted fresh, so the hand-generated claim-loop value is retired at cutover. The reviewed script scripts/wp0039-seed-worker-tokens.sh does the minting. It runs silently, never overwrites an existing path, and exits 0 when both paths are verified, 3 on an unusable existing path, 4 on a write failure, and 5 on a verification failure. It generates each value inside a pipe and sends it to bao kv put ... token=- on stdin. The founder runs it in their own terminal:

BAO_ADDR=http://127.0.0.1:18200 VAULT_ADDR=http://127.0.0.1:18200 \
  WP0039_STATUS=$HOME/.local/state/wp0039-seed.status \
  warden access openbao-platform-admin-login --exec -- \
  sh scripts/wp0039-seed-worker-tokens.sh

The first founder run on 2026-09-23 failed closed ("returned a failure or unexpected output"). warden discards the child's output, so the script now writes a non-secret step trace to WP0039_STATUS. The trace holds the failing step, the exit code, and bao's error text, never the value. The script was tested against a stub bao: silent, and a no-op on rerun.

Read warden's printed line rather than its exit code (orientation section 5). Minting does not depend on T02, but ESO cannot read the paths until the policy lands.

Roll out and prove both identities

id: ACTIVITY-WP-0039-T04
status: wait
priority: high
state_hub_task_id: "13e90456-02ff-52e7-9296-1fd80761fab4"

Needs the founder's go-ahead (ADMINISTER @ realm:kubernetes/railiance01, activation=APPROVED) and a CPU headroom check first (orientation section 4). Apply the ExternalSecret, add ACTIVITY_CORE_WORKERS to actcore-runtime-config, deploy the new image, and restart only the API. Prove four things:

• The claim loop still polls with HTTP 200.
• The metered token authenticates as rein-aharness-metered@railiance01.
• A metered token paired with the loop identity is rejected with HTTP 403.
• The previous hand-set Secret key is now owned by ESO.

Cutover order for option (b), which keeps the claim loop's gap to seconds. The pods read actcore-runtime-secret only at start.

1. Apply 15-externalsecret-worker-tokens.yaml, then force a refresh. The Secret now holds the new tokens, and the running API still uses the old one.
2. On railiance01, write the new ACTIVITY_CORE_WORKER_TOKEN from the Secret into ~/.config/rein-aharness/claim-loop.env. Use a go-template read piped into the file, and never print the value.
3. Apply ACTIVITY_CORE_WORKERS to actcore-runtime-config, then restart deployment/actcore-api.
4. Restart rein-aharness-claim-loop.service, and confirm its first poll returns HTTP 200.

Report the revision to secrets-engine on threads 914853d9 and 6e694682.