Bind multiple queue worker identities, one token each (WP-0039-T01)

ACTIVITY_CORE_WORKERS maps worker_id=ENV_NAME, where each token env must be
ACTIVITY_CORE_WORKER_TOKEN[_SUFFIX]. Without the map, the legacy single pair
behaves exactly as before. Duplicate identities, missing or shared tokens, a
token equal to the operator token, and an unlisted legacy identity all fail
worker mutations closed with 503. Operator/SSO reads keep working.

Declare per-identity OpenBao paths and an ExternalSecret, not yet applied.
The policy, seeding and rollout are waiting tasks T02-T04, answering
secrets-engine SECRETS-WP-0009-T03 and SECRETS-WP-0011-T04.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 151606@bnt-lap001
Assistant-Session: 3c0a4ad5-bb8b-4bf7-b9f0-fa5f29204e48

docs/deploy-ops-run-queue-railiance.md

@@ -91,16 +91,21 @@ explicit is better for ops):
 | `OPS_RUN_MAX_ATTEMPTS` | `3` | Optional |
 | `OPS_RUN_SLA_HOURS` | `1` | Status stuck threshold |
 | `ACTIVITY_CORE_WORKER_ID` | `rein-aharness@railiance01` | Exact non-secret identity bound to the worker token |
+| `ACTIVITY_CORE_WORKERS` | `rein-aharness@railiance01=ACTIVITY_CORE_WORKER_TOKEN,rein-aharness-metered@railiance01=ACTIVITY_CORE_WORKER_TOKEN_METERED` | Optional token-to-identity map (ACTIVITY-WP-0039); when set it is authoritative |
 
 Worker auth (required before external claim):
 
-```bash
-# Generate once; store in secret — do not commit
-WORKER_TOKEN="$(openssl rand -hex 24)"
-kubectl -n activity-core patch secret actcore-runtime-secret --type merge \
-  -p "{\"stringData\":{\"ACTIVITY_CORE_WORKER_TOKEN\":\"${WORKER_TOKEN}\"}}"
-# Record token in operator secret store (OpenBao / password manager), not chat.
-```
+Worker tokens are held in OpenBao, one path per identity, field `token`:
+`platform/workloads/activity-core/ops-run-workers/<identity-slug>`. The
+ExternalSecret `k8s/railiance/15-externalsecret-worker-tokens.yaml` merges them
+into `actcore-runtime-secret`. Do not generate a token by hand or patch it into
+the Secret. Minting is founder-attended OpenBao work (ACTIVITY-WP-0039-T03).
+Until that rollout, production still carries the earlier hand-set
+`ACTIVITY_CORE_WORKER_TOKEN`.
+
+Each token binds exactly one identity. Shared tokens, a token equal to the
+operator token, and identities missing from `ACTIVITY_CORE_WORKERS` fail closed
+with 503.
 
 Apply the ConfigMap identity and Secret token in the same rollout. If the token
 is present without `ACTIVITY_CORE_WORKER_ID`, worker mutations fail with 503;

k8s/railiance/15-externalsecret-worker-tokens.yaml

@@ -0,0 +1,44 @@
+# Sync ops_run queue worker tokens from OpenBao into actcore-runtime-secret.
+#
+# NOT YET APPLIED (ACTIVITY-WP-0039). Apply only after T02 (railiance-platform
+# adds both exact paths to the activity-core-eso role policy) and T03 (the
+# founder seeds both paths). Applying earlier makes the store fail to read,
+# or overwrites the hand-set ACTIVITY_CORE_WORKER_TOKEN with a missing value.
+#
+# Prereqs on railiance01:
+#   - ClusterSecretStore openbao-activity-core (railiance-platform addon; OpenBao
+#     Kubernetes auth via ServiceAccount activity-core/activity-core-eso since
+#     RPF-WP-0045)
+#
+# One path per worker identity (field: token), so a consumer lane can be
+# granted exactly one worker's token:
+#   rein-aharness@railiance01          -> ACTIVITY_CORE_WORKER_TOKEN
+#   rein-aharness-metered@railiance01  -> ACTIVITY_CORE_WORKER_TOKEN_METERED
+# The identity map is the non-secret ACTIVITY_CORE_WORKERS in
+# actcore-runtime-config.
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: actcore-ops-run-worker-tokens
+  namespace: activity-core
+  labels:
+    app.kubernetes.io/name: activity-core
+    app.kubernetes.io/part-of: activity-core
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao-activity-core
+  target:
+    name: actcore-runtime-secret
+    creationPolicy: Merge
+    deletionPolicy: Retain
+  data:
+    - secretKey: ACTIVITY_CORE_WORKER_TOKEN
+      remoteRef:
+        key: platform/workloads/activity-core/ops-run-workers/rein-aharness-railiance01
+        property: token
+    - secretKey: ACTIVITY_CORE_WORKER_TOKEN_METERED
+      remoteRef:
+        key: platform/workloads/activity-core/ops-run-workers/rein-aharness-metered-railiance01
+        property: token

k8s/railiance/README.md

@@ -45,6 +45,10 @@ workers insert claimable `ops_runs` on emit. Full railiance checklist:
 harness claim client. The non-secret `ACTIVITY_CORE_WORKER_ID` in the runtime
 ConfigMap binds that credential to `rein-aharness@railiance01`; deploy both
 settings together.
+Additional worker identities use the non-secret `ACTIVITY_CORE_WORKERS` map,
+with one OpenBao-held token per identity synced by
+`15-externalsecret-worker-tokens.yaml`. That file is not applied yet; see
+ACTIVITY-WP-0039.
 
 | ExternalSecret | OpenBao path | Secret key |
 | --- | --- | --- |

src/activity_core/ops_runs_api.py

@@ -4,6 +4,7 @@ from __future__ import annotations
 
 import hmac
 import os
+import re
 import uuid
 from collections.abc import Callable
 from datetime import datetime
@@ -38,6 +39,8 @@ _get_db: Callable[[], async_sessionmaker[AsyncSession]] | None = None
 
 WORKER_TOKEN_ENV = "ACTIVITY_CORE_WORKER_TOKEN"
 WORKER_ID_ENV = "ACTIVITY_CORE_WORKER_ID"
+WORKERS_ENV = "ACTIVITY_CORE_WORKERS"
+_WORKER_TOKEN_ENV_RE = re.compile(r"ACTIVITY_CORE_WORKER_TOKEN(?:_[A-Z0-9]+)*")
 
 
 def bind_ops_runs_deps(
@@ -54,6 +57,8 @@ def _db() -> async_sessionmaker[AsyncSession]:
 
 
 def _worker_token_configured() -> bool:
+    if (os.environ.get(WORKERS_ENV) or "").strip():
+        return True
     return bool((os.environ.get(WORKER_TOKEN_ENV) or "").strip())
 
 
@@ -62,6 +67,77 @@ def configured_worker_id() -> str | None:
     return value or None
 
 
+class WorkerConfigError(Exception):
+    """Worker credential configuration is incomplete or ambiguous."""
+
+
+def worker_credentials() -> list[tuple[str, str]]:
+    """Return ``(worker_id, token)`` pairs, each token bound to one identity.
+
+    ``ACTIVITY_CORE_WORKERS`` is a non-secret, comma-separated list of
+    ``worker_id=ENV_NAME`` entries. Each ENV_NAME must be
+    ``ACTIVITY_CORE_WORKER_TOKEN`` or ``ACTIVITY_CORE_WORKER_TOKEN_<SUFFIX>`` and
+    hold that worker's token. Without it, the legacy single pair
+    ``ACTIVITY_CORE_WORKER_ID`` + ``ACTIVITY_CORE_WORKER_TOKEN`` applies.
+    Duplicate identities, missing or shared tokens, and a token equal to the
+    operator token raise ``WorkerConfigError`` so callers fail closed.
+    """
+    raw = (os.environ.get(WORKERS_ENV) or "").strip()
+    if not raw:
+        token = (os.environ.get(WORKER_TOKEN_ENV) or "").strip()
+        if not token:
+            return []
+        worker_id = configured_worker_id()
+        if not worker_id:
+            raise WorkerConfigError(f"worker identity not configured ({WORKER_ID_ENV})")
+        pairs = [(worker_id, token)]
+    else:
+        pairs = []
+        for entry in raw.split(","):
+            entry = entry.strip()
+            if not entry:
+                continue
+            worker_id, sep, env_name = (part.strip() for part in entry.partition("="))
+            if not sep or not worker_id or _WORKER_TOKEN_ENV_RE.fullmatch(env_name) is None:
+                raise WorkerConfigError(f"invalid {WORKERS_ENV} entry")
+            token = (os.environ.get(env_name) or "").strip()
+            if not token:
+                raise WorkerConfigError(f"worker token not configured ({env_name})")
+            pairs.append((worker_id, token))
+        if not pairs:
+            raise WorkerConfigError(f"{WORKERS_ENV} lists no workers")
+        legacy_id = configured_worker_id()
+        if legacy_id and legacy_id not in {worker_id for worker_id, _ in pairs}:
+            raise WorkerConfigError(f"{WORKER_ID_ENV} is not listed in {WORKERS_ENV}")
+
+    ids = [worker_id for worker_id, _ in pairs]
+    tokens = [token for _, token in pairs]
+    if len(set(ids)) != len(ids):
+        raise WorkerConfigError("duplicate worker identity")
+    if len(set(tokens)) != len(tokens):
+        raise WorkerConfigError("worker tokens must not be shared")
+    operator = (os.environ.get("ACTIVITY_CORE_OPERATOR_TOKEN") or "").strip()
+    if operator and operator in tokens:
+        raise WorkerConfigError("worker token must differ from the operator token")
+    return pairs
+
+
+def _authenticate_worker_token(token: str | None) -> str | None:
+    """Return the identity bound to ``token``; raise 503 on bad configuration."""
+    try:
+        pairs = worker_credentials()
+    except WorkerConfigError as exc:
+        raise HTTPException(status_code=503, detail=str(exc)) from exc
+    if not token:
+        return None
+    matched = None
+    for worker_id, expected in pairs:
+        # Compare against every entry so timing does not reveal the position.
+        if hmac.compare_digest(token, expected):
+            matched = worker_id
+    return matched
+
+
 def _extract_worker_token(
     *,
     x_worker_token: str | None,
@@ -85,18 +161,15 @@ def require_worker_or_operator(
     worker_tok = _extract_worker_token(
         x_worker_token=x_worker_token, authorization=authorization
     )
-    expected_worker = (os.environ.get(WORKER_TOKEN_ENV) or "").strip()
-    if (
-        expected_worker
-        and worker_tok
-        and hmac.compare_digest(worker_tok, expected_worker)
-    ):
-        worker_id = configured_worker_id()
-        if not worker_id:
-            raise HTTPException(
-                status_code=503,
-                detail=f"worker identity not configured ({WORKER_ID_ENV})",
-            )
+    worker_id = None
+    if worker_tok:
+        try:
+            worker_id = _authenticate_worker_token(worker_tok)
+        except HTTPException:
+            # Broken worker config must not lock out operator/SSO reads;
+            # worker-only mutations still fail closed in require_worker.
+            worker_id = None
+    if worker_id:
         return f"worker:{worker_id}"
 
     sso = extract_sso_principal(request)
@@ -125,22 +198,16 @@ def require_worker(
     x_worker_token: str | None = None,
     authorization: str | None = None,
 ) -> str:
-    """Return the configured worker identity for a valid worker credential."""
+    """Return the identity bound to a valid worker credential."""
     del request  # reserved for future mTLS/proxy-bound worker principals
-    expected_worker = (os.environ.get(WORKER_TOKEN_ENV) or "").strip()
     worker_tok = _extract_worker_token(
         x_worker_token=x_worker_token, authorization=authorization
     )
 
-    if expected_worker:
-        if not worker_tok or not hmac.compare_digest(worker_tok, expected_worker):
-            raise HTTPException(status_code=401, detail="worker auth required")
-        worker_id = configured_worker_id()
+    if _worker_token_configured():
+        worker_id = _authenticate_worker_token(worker_tok)
         if not worker_id:
-            raise HTTPException(
-                status_code=503,
-                detail=f"worker identity not configured ({WORKER_ID_ENV})",
-            )
+            raise HTTPException(status_code=401, detail="worker auth required")
         return worker_id
 
     if allow_unauth_mutations() and not operator_token_configured():

tests/test_ops_runs_api.py

@@ -165,3 +165,113 @@ def test_close_response_distinguishes_reconciled_and_refusal_codes(
         _close_response(CloseOpsRunOutcome("expired_lease", row))
     assert exc.value.status_code == 409
     assert exc.value.detail["code"] == "expired_lease"
+
+
+def _clear_worker_env(monkeypatch: pytest.MonkeyPatch) -> None:
+    for name in (
+        "ACTIVITY_CORE_WORKERS",
+        "ACTIVITY_CORE_WORKER_ID",
+        "ACTIVITY_CORE_WORKER_TOKEN",
+        "ACTIVITY_CORE_WORKER_TOKEN_METERED",
+        "ACTIVITY_CORE_OPERATOR_TOKEN",
+        "ACTIVITY_CORE_OPS_ALLOW_UNAUTH_MUTATIONS",
+    ):
+        monkeypatch.delenv(name, raising=False)
+
+
+def _two_workers(monkeypatch: pytest.MonkeyPatch) -> None:
+    _clear_worker_env(monkeypatch)
+    monkeypatch.setenv(
+        "ACTIVITY_CORE_WORKERS",
+        "rein-aharness@railiance01=ACTIVITY_CORE_WORKER_TOKEN,"
+        "rein-aharness-metered@railiance01=ACTIVITY_CORE_WORKER_TOKEN_METERED",
+    )
+    monkeypatch.setenv("ACTIVITY_CORE_WORKER_TOKEN", "loop-secret")
+    monkeypatch.setenv("ACTIVITY_CORE_WORKER_TOKEN_METERED", "metered-secret")
+
+
+def test_each_worker_token_binds_exactly_its_own_identity(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    _two_workers(monkeypatch)
+
+    loop = require_worker(_request(), x_worker_token="loop-secret")
+    metered = require_worker(_request(), authorization="Bearer metered-secret")
+
+    assert loop == "rein-aharness@railiance01"
+    assert metered == "rein-aharness-metered@railiance01"
+    with pytest.raises(HTTPException) as exc:
+        bind_worker_id("rein-aharness@railiance01", metered)
+    assert exc.value.status_code == 403
+    with pytest.raises(HTTPException) as exc:
+        require_worker(_request(), x_worker_token="unknown")
+    assert exc.value.status_code == 401
+
+
+def test_claim_endpoint_rejects_cross_identity_body(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    _two_workers(monkeypatch)
+
+    response = _client().post(
+        "/ops-runs/claim",
+        json={"worker_id": "rein-aharness@railiance01", "limit": 1},
+        headers={"X-Worker-Token": "metered-secret"},
+    )
+
+    assert response.status_code == 403
+
+
+@pytest.mark.parametrize(
+    ("workers", "env", "fragment"),
+    [
+        ("a=ACTIVITY_CORE_WORKER_TOKEN,b=ACTIVITY_CORE_WORKER_TOKEN_METERED",
+         {"ACTIVITY_CORE_WORKER_TOKEN": "same", "ACTIVITY_CORE_WORKER_TOKEN_METERED": "same"},
+         "shared"),
+        ("a=ACTIVITY_CORE_WORKER_TOKEN,a=ACTIVITY_CORE_WORKER_TOKEN_METERED",
+         {"ACTIVITY_CORE_WORKER_TOKEN": "x", "ACTIVITY_CORE_WORKER_TOKEN_METERED": "y"},
+         "duplicate"),
+        ("a=ACTIVITY_CORE_WORKER_TOKEN,b=ACTIVITY_CORE_WORKER_TOKEN_METERED",
+         {"ACTIVITY_CORE_WORKER_TOKEN": "x"},
+         "ACTIVITY_CORE_WORKER_TOKEN_METERED"),
+        ("a=ACTIVITY_CORE_OPERATOR_TOKEN",
+         {"ACTIVITY_CORE_OPERATOR_TOKEN": "op"},
+         "invalid"),
+        ("a=ACTIVITY_CORE_WORKER_TOKEN",
+         {"ACTIVITY_CORE_WORKER_TOKEN": "op", "ACTIVITY_CORE_OPERATOR_TOKEN": "op"},
+         "operator"),
+        ("a=ACTIVITY_CORE_WORKER_TOKEN",
+         {"ACTIVITY_CORE_WORKER_TOKEN": "x", "ACTIVITY_CORE_WORKER_ID": "b"},
+         "not listed"),
+    ],
+)
+def test_ambiguous_worker_config_fails_closed(
+    monkeypatch: pytest.MonkeyPatch,
+    workers: str,
+    env: dict[str, str],
+    fragment: str,
+) -> None:
+    _clear_worker_env(monkeypatch)
+    monkeypatch.setenv("ACTIVITY_CORE_WORKERS", workers)
+    for name, value in env.items():
+        monkeypatch.setenv(name, value)
+
+    with pytest.raises(HTTPException) as exc:
+        require_worker(_request(), x_worker_token="x")
+
+    assert exc.value.status_code == 503
+    assert fragment in exc.value.detail
+
+
+def test_broken_worker_config_does_not_lock_out_operator(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    _clear_worker_env(monkeypatch)
+    monkeypatch.setenv("ACTIVITY_CORE_WORKERS", "a=ACTIVITY_CORE_WORKER_TOKEN_MISSING")
+    monkeypatch.setenv("ACTIVITY_CORE_OPERATOR_TOKEN", "operator-secret")
+
+    principal = require_worker_or_operator(
+        _request(), authorization="Bearer operator-secret"
+    )
+
+    assert principal == "operator:token"

workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md

@@ -0,0 +1,141 @@
+---
+id: ACTIVITY-WP-0039
+type: workplan
+title: "Multiple queue worker identities with OpenBao-custodied tokens"
+domain: infotech
+repo: activity-core
+status: active
+flavor: implementation
+owner: claude-code
+topic_slug: activity-core
+priority: high
+created: "2026-09-23"
+updated: "2026-09-23"
+related:
+  - ACT-ADR-005
+  - ACTIVITY-WP-0036
+  - SECRETS-WP-0009-T03
+  - SECRETS-WP-0011-T04
+  - RPF-WP-0045
+---
+
+# Multiple Queue Worker Identities with OpenBao-Custodied Tokens
+
+## Origin
+
+Two requests from secrets-engine, both approved on their side on 2026-09-23:
+
+- Message `914853d9` (SECRETS-WP-0009-T03) asks for more than one
+  authenticated queue worker. The spend-admitted `rein-aharness metered-once`
+  owner should claim as `rein-aharness-metered@railiance01`, separate from the
+  running `rein-aharness@railiance01` claim loop. This keeps paid-run
+  attribution distinct at the API.
+- Message `6e694682` (SECRETS-WP-0011-T04) asks for the worker token to be held
+  in OpenBao and synced by the existing `openbao-activity-core` store, instead
+  of the hand-generated `openssl rand` value in
+  `docs/deploy-ops-run-queue-railiance.md`. secrets-engine will catalog a
+  read-only lane for the metered worker's path only. It will never read, copy
+  or mint the token.
+
+WP-0036 bound one token to one configured identity. This workplan keeps that
+guarantee and extends it to a set of identities.
+
+## Boundary rules
+
+- Each token is bound to exactly one worker identity. The body `worker_id`
+  must still match it. No token is shared, and nothing falls back implicitly.
+- Worker tokens never equal the operator token, and operator/SSO credentials
+  stay out of worker mutations (WP-0036).
+- One OpenBao path per worker identity, so a consumer lane can be granted
+  exactly one worker's token.
+- Minting and writing a token is founder-attended OpenBao work. The store's
+  role policy belongs to railiance-platform. No token value passes through
+  Git, State Hub, chat, or an agent shell.
+
+## Token-to-identity map in the worker API
+
+```task
+id: ACTIVITY-WP-0039-T01
+status: done
+priority: high
+```
+
+Add `ACTIVITY_CORE_WORKERS`, a non-secret comma-separated list of
+`worker_id=ENV_NAME` entries. Each ENV_NAME must be `ACTIVITY_CORE_WORKER_TOKEN`
+or `ACTIVITY_CORE_WORKER_TOKEN_<SUFFIX>`. When the list is unset, the legacy
+`ACTIVITY_CORE_WORKER_ID` + `ACTIVITY_CORE_WORKER_TOKEN` pair behaves exactly as
+before. Worker mutations fail closed with 503 when the configuration is
+ambiguous. That covers a duplicate identity, a missing or shared token, a token
+equal to the operator token, an env name outside the pattern, and a legacy
+identity missing from the list. A broken worker configuration does not lock out
+operator or SSO reads.
+
+Implemented in `src/activity_core/ops_runs_api.py` (`worker_credentials`).
+Tests in `tests/test_ops_runs_api.py` cover per-identity binding, a
+cross-identity body rejected with 403, an unknown token rejected with 401, each
+fail-closed case, and operator access under broken worker configuration.
+
+## Declare OpenBao paths and the ExternalSecret
+
+```task
+id: ACTIVITY-WP-0039-T02
+status: wait
+priority: high
+```
+
+Paths, one per identity, each with field `token`:
+
+- `platform/workloads/activity-core/ops-run-workers/rein-aharness-railiance01`
+- `platform/workloads/activity-core/ops-run-workers/rein-aharness-metered-railiance01`
+
+`k8s/railiance/15-externalsecret-worker-tokens.yaml` merges them into
+`actcore-runtime-secret` as `ACTIVITY_CORE_WORKER_TOKEN` and
+`ACTIVITY_CORE_WORKER_TOKEN_METERED`. The manifest is in the repo but not
+applied.
+
+Waiting on railiance-platform to add both exact paths to the
+`activity-core-eso` role policy (RPF-WP-0045 pattern), and on secrets-engine to
+confirm the metered path for its catalog.
+
+## Seed the tokens in OpenBao
+
+```task
+id: ACTIVITY-WP-0039-T03
+status: wait
+priority: high
+```
+
+Founder-attended, through `warden access openbao-platform-admin-login --exec`
+(orientation section 5):
+
+- **Metered worker:** mint a fresh random value directly into its path. No
+  value exists today.
+- **Claim-loop worker:** the founder chooses between two options.
+  - (a) Move the current value into OpenBao. The running claim loop needs no
+    change, but the value was hand-generated outside custody.
+  - (b) Mint a fresh value and update the rein-aharness claim-loop
+    configuration on railiance01 (user tegwick) in the same window. This means
+    one coordinated restart.
+
+Done when both paths hold a value and no value has been printed or logged.
+
+## Roll out and prove both identities
+
+```task
+id: ACTIVITY-WP-0039-T04
+status: wait
+priority: high
+```
+
+Needs the founder's go-ahead (`ADMINISTER @ realm:kubernetes/railiance01`,
+`activation=APPROVED`) and a CPU headroom check first (orientation section 4).
+Apply the ExternalSecret, add `ACTIVITY_CORE_WORKERS` to
+`actcore-runtime-config`, deploy the new image, and restart only the API.
+Prove four things:
+
+- The claim loop still polls with HTTP 200.
+- The metered token authenticates as `rein-aharness-metered@railiance01`.
+- A metered token paired with the loop identity is rejected with HTTP 403.
+- The previous hand-set Secret key is now owned by ESO.
+
+Report the revision to secrets-engine on threads `914853d9` and `6e694682`.