activity-core/workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md

id	type	title	domain	repo	status	flavor	owner	topic_slug	priority	created	updated	related	state_hub_workstream_id
ACTIVITY-WP-0039	workplan	Multiple queue worker identities with OpenBao-custodied tokens	infotech	activity-core	finished	implementation	claude-code	activity-core	high	2026-09-23	2026-09-23	ACT-ADR-005 ACTIVITY-WP-0036 SECRETS-WP-0009-T03 SECRETS-WP-0011-T04 RPF-WP-0045	053ef7c6-c2cf-5f50-aabf-09034c66d1ce

Multiple Queue Worker Identities with OpenBao-Custodied Tokens

Origin

Two requests from secrets-engine, both approved on their side on 2026-09-23:

• Message 914853d9 (SECRETS-WP-0009-T03) asks for more than one authenticated queue worker. The spend-admitted rein-aharness metered-once owner should claim as rein-aharness-metered@railiance01, separate from the running rein-aharness@railiance01 claim loop. This keeps paid-run attribution distinct at the API.
• Message 6e694682 (SECRETS-WP-0011-T04) asks for the worker token to be held in OpenBao and synced by the existing openbao-activity-core store, instead of the hand-generated openssl rand value in docs/deploy-ops-run-queue-railiance.md. secrets-engine will catalog a read-only lane for the metered worker's path only. It will never read, copy or mint the token.

WP-0036 bound one token to one configured identity. This workplan keeps that guarantee and extends it to a set of identities.

Boundary rules

• Each token is bound to exactly one worker identity. The body worker_id must still match it. No token is shared, and nothing falls back implicitly.
• Worker tokens never equal the operator token, and operator/SSO credentials stay out of worker mutations (WP-0036).
• One OpenBao path per worker identity, so a consumer lane can be granted exactly one worker's token.
• Minting and writing a token is founder-attended OpenBao work. The store's role policy belongs to railiance-platform. No token value passes through Git, State Hub, chat, or an agent shell.

Token-to-identity map in the worker API

id: ACTIVITY-WP-0039-T01
status: done
priority: high
state_hub_task_id: "a82a498a-0673-5a29-94ae-0055672f8e4e"

Add ACTIVITY_CORE_WORKERS, a non-secret comma-separated list of worker_id=ENV_NAME entries. Each ENV_NAME must be ACTIVITY_CORE_WORKER_TOKEN or ACTIVITY_CORE_WORKER_TOKEN_<SUFFIX>. When the list is unset, the legacy ACTIVITY_CORE_WORKER_ID + ACTIVITY_CORE_WORKER_TOKEN pair behaves exactly as before. Worker mutations fail closed with 503 when the configuration is ambiguous. That covers a duplicate identity, a missing or shared token, a token equal to the operator token, an env name outside the pattern, and a legacy identity missing from the list. A broken worker configuration does not lock out operator or SSO reads.

Implemented in src/activity_core/ops_runs_api.py (worker_credentials). Tests in tests/test_ops_runs_api.py cover per-identity binding, a cross-identity body rejected with 403, an unknown token rejected with 401, each fail-closed case, and operator access under broken worker configuration.

Declare OpenBao paths and the ExternalSecret

id: ACTIVITY-WP-0039-T02
status: done
priority: high
state_hub_task_id: "1400bf41-e00a-56cc-9ccf-a1cf023b220a"

Paths, one per identity, each with field token:

• platform/workloads/activity-core/ops-run-workers/rein-aharness-railiance01
• platform/workloads/activity-core/ops-run-workers/rein-aharness-metered-railiance01

k8s/railiance/15-externalsecret-worker-tokens.yaml merges them into actcore-runtime-secret as ACTIVITY_CORE_WORKER_TOKEN and ACTIVITY_CORE_WORKER_TOKEN_METERED. The manifest is in the repo but not applied.

Waiting on railiance-platform to add both exact paths to the activity-core-eso role policy (RPF-WP-0045 pattern), and on secrets-engine to confirm the metered path for its catalog.

2026-09-23: secrets-engine confirmed the metered path (message a0e9afaf). It catalogs only .../rein-aharness-metered-railiance01 as lane activity-core-metered-worker-token, read through its own exact-path AppRole. Only the railiance-platform policy change remains. The paths are requested in activity-core's message to railiance-platform. The HCL is two read stanzas on platform/data/workloads/activity-core/ops-run-workers/<slug>.

Seed the tokens in OpenBao

id: ACTIVITY-WP-0039-T03
status: done
priority: high
state_hub_task_id: "73d7f032-5ec3-550e-83ac-b9852c09ace4"

Founder-attended, through warden access openbao-platform-admin-login --exec (orientation section 5):

• Metered worker: mint a fresh random value directly into its path. No value exists today.
• Claim-loop worker: the founder chooses between two options.
  • (a) Move the current value into OpenBao. The running claim loop needs no change, but the value was hand-generated outside custody.
  • (b) Mint a fresh value and update the rein-aharness claim-loop configuration on railiance01 (user tegwick) in the same window. This means one coordinated restart.

Done when both paths hold a value and no value has been printed or logged.

Decision 2026-09-23 (founder): option (b). Both tokens are minted fresh, so the hand-generated claim-loop value is retired at cutover. The reviewed script scripts/wp0039-seed-worker-tokens.sh does the minting. It runs silently, never overwrites an existing path, and exits 0 when both paths are verified, 3 on an unusable existing path, 4 on a write failure, and 5 on a verification failure. It generates each value inside a pipe and sends it to bao kv put ... token=- on stdin. The founder runs it in their own terminal:

BAO_ADDR=http://127.0.0.1:18200 VAULT_ADDR=http://127.0.0.1:18200 \
  WP0039_STATUS=$HOME/.local/state/wp0039-seed.status \
  warden access openbao-platform-admin-login --exec -- \
  sh /home/worsch/activity-core/scripts/wp0039-seed-worker-tokens.sh   # absolute path

The first founder run on 2026-09-23 failed closed ("returned a failure or unexpected output"). warden discards the child's output, so the script now writes a non-secret step trace to WP0039_STATUS. The trace holds the failing step, the exit code, and bao's error text, never the value. The script was tested against a stub bao: silent, and a no-op on rerun.

The failed runs never started the script. The founder's terminal was outside the repository, so the relative script path did not resolve, and no status file was written. With the absolute path, the founder's run on 2026-09-23 succeeded. warden reported no failure and no revocation warning. The status trace shows both paths absent beforehand, then written, then verified with a 64-character token field, then exit 0. No value was printed or logged.

Read warden's printed line rather than its exit code (orientation section 5). Minting does not depend on T02, but ESO cannot read the paths until the policy lands.

Roll out and prove both identities

id: ACTIVITY-WP-0039-T04
status: done
priority: high
state_hub_task_id: "13e90456-02ff-52e7-9296-1fd80761fab4"

Needs the founder's go-ahead (ADMINISTER @ realm:kubernetes/railiance01, activation=APPROVED) and a CPU headroom check first (orientation section 4). Apply the ExternalSecret, add ACTIVITY_CORE_WORKERS to actcore-runtime-config, deploy the new image, and restart only the API. Prove four things:

• The claim loop still polls with HTTP 200.
• The metered token authenticates as rein-aharness-metered@railiance01.
• A metered token paired with the loop identity is rejected with HTTP 403.
• The previous hand-set Secret key is now owned by ESO.

Cutover order for option (b), which keeps the claim loop's gap to seconds. The pods read actcore-runtime-secret only at start.

1. Apply 15-externalsecret-worker-tokens.yaml, then force a refresh. The Secret now holds the new tokens, and the running API still uses the old one.
2. On railiance01, write the new ACTIVITY_CORE_WORKER_TOKEN from the Secret into ~/.config/rein-aharness/claim-loop.env. Use a go-template read piped into the file, and never print the value.
3. Apply ACTIVITY_CORE_WORKERS to actcore-runtime-config, then restart deployment/actcore-api.
4. Restart rein-aharness-claim-loop.service, and confirm its first poll returns HTTP 200.

Report the revision to secrets-engine on threads 914853d9 and 6e694682.

Cutover attempt, 2026-09-23 (founder go-ahead, build mode). T02 was done by railiance-platform at 18:06:29Z (CCR-2026-0029/0030, commit fe1665d).

1. The ExternalSecret actcore-ops-run-worker-tokens is applied. Both keys synced at 20:35:35Z, and ESO now owns ACTIVITY_CORE_WORKER_TOKEN.
2. No ops_run was claimed. The claim-loop env file was backed up (claim-loop.env.bak-wp0039-20260923T203559) and given the new loop token without printing it.
3. The image activity-core:wp0039-20260923 (built from main at 1a20c85) was imported. The ConfigMap with ACTIVITY_CORE_WORKERS and the API Deployment were applied, and the claim loop was restarted.
4. The new API returned HTTP 500 (column ops_runs.repository_grant does not exist). Production is at alembic 0009, and main needs 0010 (the WP-0038 columns). The running fi-publication-20260914 image contains no WP-0038 code, so WP-0038 was never deployed. The API was rolled back to fi-publication-20260914, and the loop claimed again with HTTP 200 at 20:39:23Z. The outage lasted about 1 minute, and no run was affected.

Live state now: the option-(b) loop token rotation is complete. The old code reads the ESO-synced ACTIVITY_CORE_WORKER_TOKEN, which is the new value. ACTIVITY_CORE_WORKERS is in the ConfigMap and is ignored by the old image. The metered identity is not live yet.

Remaining: run migration 0010, which is additive (two nullable columns). That also takes WP-0038's close reconciliation live for the first time. Then redeploy the API on wp0039-20260923, and prove the metered identity and the 403 cross-identity rejection. Both need the founder's decision because they go beyond the worker-identity change.

Closed 2026-09-24. On 2026-09-24 the founder ran migration 0010, and the API was redeployed on wp0039-20260923. The loop claims with 200 as rein-aharness@railiance01. The metered token authenticates as rein-aharness-metered@railiance01 (200, nothing claimed), and paired with the loop identity it is rejected with 403. This also took WP-0038 live for the first time. Evidence: docs/evidence/ACTIVITY-WP-0039-worker-identity-cutover-2026-09-23.md.