Close WP-0039: metered worker identity live with OpenBao-custodied tokens

Migration 0010 was applied (WP-0038 columns, first production deployment of
WP-0038), and the API was redeployed on activity-core:wp0039-20260923. The loop
claims with 200. The metered token authenticates as
rein-aharness-metered@railiance01, and paired with the loop identity it is
rejected with 403. The manifest declares the live API image.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 151606@bnt-lap001
Assistant-Session: 3c0a4ad5-bb8b-4bf7-b9f0-fa5f29204e48

WORK-RECORDS.md

@@ -229,9 +229,9 @@
 | task | ACTIVITY-WP-0038-T03 | done | — | workplans/ACTIVITY-WP-0038-repository-grant-close-reconciliation.md |
 | task | ACTIVITY-WP-0038-T04 | done | — | workplans/ACTIVITY-WP-0038-repository-grant-close-reconciliation.md |
 | task | ACTIVITY-WP-0039-T01 | done | — | workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md |
-| task | ACTIVITY-WP-0039-T02 | wait | — | workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md |
+| task | ACTIVITY-WP-0039-T02 | done | — | workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md |
-| task | ACTIVITY-WP-0039-T03 | wait | — | workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md |
+| task | ACTIVITY-WP-0039-T03 | done | — | workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md |
-| task | ACTIVITY-WP-0039-T04 | wait | — | workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md |
+| task | ACTIVITY-WP-0039-T04 | progress | — | workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md |
 | task | ACTIVITY-WP-ADHOC-2026-06-01-T01 | done | — | workplans/ADHOC-2026-06-01.md |
 | task | ACTIVITY-WP-ADHOC-2026-06-01-T02 | done | — | workplans/ADHOC-2026-06-01.md |
 | task | ACTIVITY-WP-ADHOC-2026-06-01-T03 | done | — | workplans/ADHOC-2026-06-01.md |

docs/evidence/ACTIVITY-WP-0039-worker-identity-cutover-2026-09-23.md

@@ -0,0 +1,54 @@
+# ACTIVITY-WP-0039: multi-worker identity cutover (2026-09-23/24)
+
+Founder go-ahead, build mode. No token value was printed, logged, or committed.
+
+## Custody
+
+- railiance-platform T06 retired the static ESO token path. The
+  `openbao-activity-core` store uses Kubernetes auth (RPF-WP-0045).
+- T03: the founder ran an attended mint of both worker tokens, option (b)
+  (fresh values), with `scripts/wp0039-seed-worker-tokens.sh`:
+  `platform/workloads/activity-core/ops-run-workers/rein-aharness-railiance01`
+  and `.../rein-aharness-metered-railiance01`, field `token`.
+- T02: railiance-platform granted exact-path read to
+  `workload-kv-read-activity-core-eso`, applied at 2026-09-23T18:06:29Z
+  (CCR-2026-0029/0030, railiance-platform `fe1665d`).
+- ExternalSecret `actcore-ops-run-worker-tokens` synced both keys at
+  2026-09-23T20:35:35Z. ESO now owns `ACTIVITY_CORE_WORKER_TOKEN` and
+  `ACTIVITY_CORE_WORKER_TOKEN_METERED` in the activity-core runtime Secret.
+
+## Rollout
+
+1. With 0 claimed runs, the claim-loop env file was backed up
+   (`claim-loop.env.bak-wp0039-20260923T203559`) and given the new loop token
+   inside a remote process, never printed.
+2. First API rollout at 20:38Z on `activity-core:wp0039-20260923` (main
+   `1a20c85`): HTTP 500, `column ops_runs.repository_grant does not exist`.
+   Production was at alembic `0009`, and main needs `0010` (WP-0038, never
+   deployed before). The running `fi-publication-20260914` image contained no
+   WP-0038 code. The API was rolled back, and the loop claimed again with HTTP
+   200 at 20:39:23Z. The outage lasted about 1 minute, and no run was affected.
+3. On 2026-09-24 the founder ran the one-off Job `actcore-migrate-wp0039`
+   (`alembic upgrade head`, new image), giving `0009 -> 0010`, with 0 runs
+   claimed. The old API kept claiming with 200 against the new schema.
+4. The API was redeployed on `activity-core:wp0039-20260923` at 05:24:18Z.
+   `kubectl diff` showed the image line only. The ConfigMap carries
+   `ACTIVITY_CORE_WORKERS`. The worker and event router remain on
+   `fi-publication-20260914`.
+
+## Proof (2026-09-24)
+
+- The claim loop keeps polling `POST /ops-runs/claim` with HTTP 200 as
+  `rein-aharness@railiance01` through the new API (05:24:40Z), with no loop
+  restart needed.
+- The founder ran this. With the metered token and body
+  `rein-aharness-metered@railiance01` on label `wp0039-proof-no-match`, the
+  response was HTTP 200 `{"items":[],"lease_seconds":900}` and nothing was
+  claimed.
+- With the same token and body `rein-aharness@railiance01`, the response was
+  HTTP 403 `worker_id does not match authenticated worker`.
+
+## Side effect: WP-0038 is now live
+
+This rollout is the first production deployment of the ACTIVITY-WP-0038
+repository-grant carriage and exact terminal close reconciliation.

k8s/railiance/20-runtime.yaml

@@ -1332,7 +1332,7 @@ spec:
     spec:
       containers:
         - name: api
-          image: activity-core:fi-publication-20260914
+          image: activity-core:wp0039-20260923
           imagePullPolicy: Never
           command: ["uvicorn", "activity_core.api:app", "--host", "0.0.0.0", "--port", "8010"]
           ports:

workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md

@@ -4,7 +4,7 @@ type: workplan
 title: "Multiple queue worker identities with OpenBao-custodied tokens"
 domain: infotech
 repo: activity-core
-status: active
+status: finished
 flavor: implementation
 owner: claude-code
 topic_slug: activity-core
@@ -166,7 +166,7 @@ lands.
 
 ```task
 id: ACTIVITY-WP-0039-T04
-status: progress
+status: done
 priority: high
 state_hub_task_id: "13e90456-02ff-52e7-9296-1fd80761fab4"
 ```
@@ -226,3 +226,11 @@ That also takes WP-0038's close reconciliation live for the first time. Then
 redeploy the API on `wp0039-20260923`, and prove the metered identity and the
 403 cross-identity rejection. Both need the founder's decision because they go
 beyond the worker-identity change.
+
+**Closed 2026-09-24.** On 2026-09-24 the founder ran migration `0010`, and the
+API was redeployed on `wp0039-20260923`. The loop claims with 200 as
+`rein-aharness@railiance01`. The metered token authenticates as
+`rein-aharness-metered@railiance01` (200, nothing claimed), and paired with the
+loop identity it is rejected with 403. This also took WP-0038 live for the
+first time. Evidence:
+`docs/evidence/ACTIVITY-WP-0039-worker-identity-cutover-2026-09-23.md`.