diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 8a3ca9d..7a09cdd 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -229,9 +229,9 @@ | task | ACTIVITY-WP-0038-T03 | done | — | workplans/ACTIVITY-WP-0038-repository-grant-close-reconciliation.md | | task | ACTIVITY-WP-0038-T04 | done | — | workplans/ACTIVITY-WP-0038-repository-grant-close-reconciliation.md | | task | ACTIVITY-WP-0039-T01 | done | — | workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md | -| task | ACTIVITY-WP-0039-T02 | wait | — | workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md | -| task | ACTIVITY-WP-0039-T03 | wait | — | workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md | -| task | ACTIVITY-WP-0039-T04 | wait | — | workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md | +| task | ACTIVITY-WP-0039-T02 | done | — | workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md | +| task | ACTIVITY-WP-0039-T03 | done | — | workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md | +| task | ACTIVITY-WP-0039-T04 | progress | — | workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md | | task | ACTIVITY-WP-ADHOC-2026-06-01-T01 | done | — | workplans/ADHOC-2026-06-01.md | | task | ACTIVITY-WP-ADHOC-2026-06-01-T02 | done | — | workplans/ADHOC-2026-06-01.md | | task | ACTIVITY-WP-ADHOC-2026-06-01-T03 | done | — | workplans/ADHOC-2026-06-01.md | diff --git a/docs/evidence/ACTIVITY-WP-0039-worker-identity-cutover-2026-09-23.md b/docs/evidence/ACTIVITY-WP-0039-worker-identity-cutover-2026-09-23.md new file mode 100644 index 0000000..7cd2fab --- /dev/null +++ b/docs/evidence/ACTIVITY-WP-0039-worker-identity-cutover-2026-09-23.md @@ -0,0 +1,54 @@ +# ACTIVITY-WP-0039: multi-worker identity cutover (2026-09-23/24) + +Founder go-ahead, build mode. No token value was printed, logged, or committed. + +## Custody + +- railiance-platform T06 retired the static ESO token path. The + `openbao-activity-core` store uses Kubernetes auth (RPF-WP-0045). +- T03: the founder ran an attended mint of both worker tokens, option (b) + (fresh values), with `scripts/wp0039-seed-worker-tokens.sh`: + `platform/workloads/activity-core/ops-run-workers/rein-aharness-railiance01` + and `.../rein-aharness-metered-railiance01`, field `token`. +- T02: railiance-platform granted exact-path read to + `workload-kv-read-activity-core-eso`, applied at 2026-09-23T18:06:29Z + (CCR-2026-0029/0030, railiance-platform `fe1665d`). +- ExternalSecret `actcore-ops-run-worker-tokens` synced both keys at + 2026-09-23T20:35:35Z. ESO now owns `ACTIVITY_CORE_WORKER_TOKEN` and + `ACTIVITY_CORE_WORKER_TOKEN_METERED` in the activity-core runtime Secret. + +## Rollout + +1. With 0 claimed runs, the claim-loop env file was backed up + (`claim-loop.env.bak-wp0039-20260923T203559`) and given the new loop token + inside a remote process, never printed. +2. First API rollout at 20:38Z on `activity-core:wp0039-20260923` (main + `1a20c85`): HTTP 500, `column ops_runs.repository_grant does not exist`. + Production was at alembic `0009`, and main needs `0010` (WP-0038, never + deployed before). The running `fi-publication-20260914` image contained no + WP-0038 code. The API was rolled back, and the loop claimed again with HTTP + 200 at 20:39:23Z. The outage lasted about 1 minute, and no run was affected. +3. On 2026-09-24 the founder ran the one-off Job `actcore-migrate-wp0039` + (`alembic upgrade head`, new image), giving `0009 -> 0010`, with 0 runs + claimed. The old API kept claiming with 200 against the new schema. +4. The API was redeployed on `activity-core:wp0039-20260923` at 05:24:18Z. + `kubectl diff` showed the image line only. The ConfigMap carries + `ACTIVITY_CORE_WORKERS`. The worker and event router remain on + `fi-publication-20260914`. + +## Proof (2026-09-24) + +- The claim loop keeps polling `POST /ops-runs/claim` with HTTP 200 as + `rein-aharness@railiance01` through the new API (05:24:40Z), with no loop + restart needed. +- The founder ran this. With the metered token and body + `rein-aharness-metered@railiance01` on label `wp0039-proof-no-match`, the + response was HTTP 200 `{"items":[],"lease_seconds":900}` and nothing was + claimed. +- With the same token and body `rein-aharness@railiance01`, the response was + HTTP 403 `worker_id does not match authenticated worker`. + +## Side effect: WP-0038 is now live + +This rollout is the first production deployment of the ACTIVITY-WP-0038 +repository-grant carriage and exact terminal close reconciliation. diff --git a/k8s/railiance/20-runtime.yaml b/k8s/railiance/20-runtime.yaml index 898601e..f3432f3 100644 --- a/k8s/railiance/20-runtime.yaml +++ b/k8s/railiance/20-runtime.yaml @@ -1332,7 +1332,7 @@ spec: spec: containers: - name: api - image: activity-core:fi-publication-20260914 + image: activity-core:wp0039-20260923 imagePullPolicy: Never command: ["uvicorn", "activity_core.api:app", "--host", "0.0.0.0", "--port", "8010"] ports: diff --git a/workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md b/workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md index 568ef14..8c3503d 100644 --- a/workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md +++ b/workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md @@ -4,7 +4,7 @@ type: workplan title: "Multiple queue worker identities with OpenBao-custodied tokens" domain: infotech repo: activity-core -status: active +status: finished flavor: implementation owner: claude-code topic_slug: activity-core @@ -166,7 +166,7 @@ lands. ```task id: ACTIVITY-WP-0039-T04 -status: progress +status: done priority: high state_hub_task_id: "13e90456-02ff-52e7-9296-1fd80761fab4" ``` @@ -226,3 +226,11 @@ That also takes WP-0038's close reconciliation live for the first time. Then redeploy the API on `wp0039-20260923`, and prove the metered identity and the 403 cross-identity rejection. Both need the founder's decision because they go beyond the worker-identity change. + +**Closed 2026-09-24.** On 2026-09-24 the founder ran migration `0010`, and the +API was redeployed on `wp0039-20260923`. The loop claims with 200 as +`rein-aharness@railiance01`. The metered token authenticates as +`rein-aharness-metered@railiance01` (200, nothing claimed), and paired with the +loop identity it is rejected with 403. This also took WP-0038 live for the +first time. Evidence: +`docs/evidence/ACTIVITY-WP-0039-worker-identity-cutover-2026-09-23.md`.