activity-core/workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md

id	type	title	domain	repo	status	flavor	owner	topic_slug	priority	created	updated	related	state_hub_workstream_id
ACTIVITY-WP-0039	workplan	Multiple queue worker identities with OpenBao-custodied tokens	infotech	activity-core	active	implementation	claude-code	activity-core	high	2026-09-23	2026-09-23	ACT-ADR-005 ACTIVITY-WP-0036 SECRETS-WP-0009-T03 SECRETS-WP-0011-T04 RPF-WP-0045	053ef7c6-c2cf-5f50-aabf-09034c66d1ce

Multiple Queue Worker Identities with OpenBao-Custodied Tokens

Origin

Two requests from secrets-engine, both approved on their side on 2026-09-23:

• Message 914853d9 (SECRETS-WP-0009-T03) asks for more than one authenticated queue worker. The spend-admitted rein-aharness metered-once owner should claim as rein-aharness-metered@railiance01, separate from the running rein-aharness@railiance01 claim loop. This keeps paid-run attribution distinct at the API.
• Message 6e694682 (SECRETS-WP-0011-T04) asks for the worker token to be held in OpenBao and synced by the existing openbao-activity-core store, instead of the hand-generated openssl rand value in docs/deploy-ops-run-queue-railiance.md. secrets-engine will catalog a read-only lane for the metered worker's path only. It will never read, copy or mint the token.

WP-0036 bound one token to one configured identity. This workplan keeps that guarantee and extends it to a set of identities.

Boundary rules

• Each token is bound to exactly one worker identity. The body worker_id must still match it. No token is shared, and nothing falls back implicitly.
• Worker tokens never equal the operator token, and operator/SSO credentials stay out of worker mutations (WP-0036).
• One OpenBao path per worker identity, so a consumer lane can be granted exactly one worker's token.
• Minting and writing a token is founder-attended OpenBao work. The store's role policy belongs to railiance-platform. No token value passes through Git, State Hub, chat, or an agent shell.

Token-to-identity map in the worker API

id: ACTIVITY-WP-0039-T01
status: done
priority: high
state_hub_task_id: "a82a498a-0673-5a29-94ae-0055672f8e4e"

Add ACTIVITY_CORE_WORKERS, a non-secret comma-separated list of worker_id=ENV_NAME entries. Each ENV_NAME must be ACTIVITY_CORE_WORKER_TOKEN or ACTIVITY_CORE_WORKER_TOKEN_<SUFFIX>. When the list is unset, the legacy ACTIVITY_CORE_WORKER_ID + ACTIVITY_CORE_WORKER_TOKEN pair behaves exactly as before. Worker mutations fail closed with 503 when the configuration is ambiguous. That covers a duplicate identity, a missing or shared token, a token equal to the operator token, an env name outside the pattern, and a legacy identity missing from the list. A broken worker configuration does not lock out operator or SSO reads.

Implemented in src/activity_core/ops_runs_api.py (worker_credentials). Tests in tests/test_ops_runs_api.py cover per-identity binding, a cross-identity body rejected with 403, an unknown token rejected with 401, each fail-closed case, and operator access under broken worker configuration.

Declare OpenBao paths and the ExternalSecret

id: ACTIVITY-WP-0039-T02
status: wait
priority: high
state_hub_task_id: "1400bf41-e00a-56cc-9ccf-a1cf023b220a"

Paths, one per identity, each with field token:

• platform/workloads/activity-core/ops-run-workers/rein-aharness-railiance01
• platform/workloads/activity-core/ops-run-workers/rein-aharness-metered-railiance01

k8s/railiance/15-externalsecret-worker-tokens.yaml merges them into actcore-runtime-secret as ACTIVITY_CORE_WORKER_TOKEN and ACTIVITY_CORE_WORKER_TOKEN_METERED. The manifest is in the repo but not applied.

Waiting on railiance-platform to add both exact paths to the activity-core-eso role policy (RPF-WP-0045 pattern), and on secrets-engine to confirm the metered path for its catalog.

Seed the tokens in OpenBao

id: ACTIVITY-WP-0039-T03
status: wait
priority: high
state_hub_task_id: "73d7f032-5ec3-550e-83ac-b9852c09ace4"

Founder-attended, through warden access openbao-platform-admin-login --exec (orientation section 5):

• Metered worker: mint a fresh random value directly into its path. No value exists today.
• Claim-loop worker: the founder chooses between two options.
  • (a) Move the current value into OpenBao. The running claim loop needs no change, but the value was hand-generated outside custody.
  • (b) Mint a fresh value and update the rein-aharness claim-loop configuration on railiance01 (user tegwick) in the same window. This means one coordinated restart.

Done when both paths hold a value and no value has been printed or logged.

Roll out and prove both identities

id: ACTIVITY-WP-0039-T04
status: wait
priority: high
state_hub_task_id: "13e90456-02ff-52e7-9296-1fd80761fab4"

Needs the founder's go-ahead (ADMINISTER @ realm:kubernetes/railiance01, activation=APPROVED) and a CPU headroom check first (orientation section 4). Apply the ExternalSecret, add ACTIVITY_CORE_WORKERS to actcore-runtime-config, deploy the new image, and restart only the API. Prove four things:

• The claim loop still polls with HTTP 200.
• The metered token authenticates as rein-aharness-metered@railiance01.
• A metered token paired with the loop identity is rejected with HTTP 403.
• The previous hand-set Secret key is now owned by ESO.

Report the revision to secrets-engine on threads 914853d9 and 6e694682.