diff --git a/docs/deploy-ops-run-queue-railiance.md b/docs/deploy-ops-run-queue-railiance.md index 0768258..86d2194 100644 --- a/docs/deploy-ops-run-queue-railiance.md +++ b/docs/deploy-ops-run-queue-railiance.md @@ -91,16 +91,21 @@ explicit is better for ops): | `OPS_RUN_MAX_ATTEMPTS` | `3` | Optional | | `OPS_RUN_SLA_HOURS` | `1` | Status stuck threshold | | `ACTIVITY_CORE_WORKER_ID` | `rein-aharness@railiance01` | Exact non-secret identity bound to the worker token | +| `ACTIVITY_CORE_WORKERS` | `rein-aharness@railiance01=ACTIVITY_CORE_WORKER_TOKEN,rein-aharness-metered@railiance01=ACTIVITY_CORE_WORKER_TOKEN_METERED` | Optional token-to-identity map (ACTIVITY-WP-0039); when set it is authoritative | Worker auth (required before external claim): -```bash -# Generate once; store in secret — do not commit -WORKER_TOKEN="$(openssl rand -hex 24)" -kubectl -n activity-core patch secret actcore-runtime-secret --type merge \ - -p "{\"stringData\":{\"ACTIVITY_CORE_WORKER_TOKEN\":\"${WORKER_TOKEN}\"}}" -# Record token in operator secret store (OpenBao / password manager), not chat. -``` +Worker tokens are held in OpenBao, one path per identity, field `token`: +`platform/workloads/activity-core/ops-run-workers/`. The +ExternalSecret `k8s/railiance/15-externalsecret-worker-tokens.yaml` merges them +into `actcore-runtime-secret`. Do not generate a token by hand or patch it into +the Secret. Minting is founder-attended OpenBao work (ACTIVITY-WP-0039-T03). +Until that rollout, production still carries the earlier hand-set +`ACTIVITY_CORE_WORKER_TOKEN`. + +Each token binds exactly one identity. Shared tokens, a token equal to the +operator token, and identities missing from `ACTIVITY_CORE_WORKERS` fail closed +with 503. Apply the ConfigMap identity and Secret token in the same rollout. If the token is present without `ACTIVITY_CORE_WORKER_ID`, worker mutations fail with 503; diff --git a/k8s/railiance/15-externalsecret-worker-tokens.yaml b/k8s/railiance/15-externalsecret-worker-tokens.yaml new file mode 100644 index 0000000..47d3643 --- /dev/null +++ b/k8s/railiance/15-externalsecret-worker-tokens.yaml @@ -0,0 +1,44 @@ +# Sync ops_run queue worker tokens from OpenBao into actcore-runtime-secret. +# +# NOT YET APPLIED (ACTIVITY-WP-0039). Apply only after T02 (railiance-platform +# adds both exact paths to the activity-core-eso role policy) and T03 (the +# founder seeds both paths). Applying earlier makes the store fail to read, +# or overwrites the hand-set ACTIVITY_CORE_WORKER_TOKEN with a missing value. +# +# Prereqs on railiance01: +# - ClusterSecretStore openbao-activity-core (railiance-platform addon; OpenBao +# Kubernetes auth via ServiceAccount activity-core/activity-core-eso since +# RPF-WP-0045) +# +# One path per worker identity (field: token), so a consumer lane can be +# granted exactly one worker's token: +# rein-aharness@railiance01 -> ACTIVITY_CORE_WORKER_TOKEN +# rein-aharness-metered@railiance01 -> ACTIVITY_CORE_WORKER_TOKEN_METERED +# The identity map is the non-secret ACTIVITY_CORE_WORKERS in +# actcore-runtime-config. +apiVersion: external-secrets.io/v1beta1 +kind: ExternalSecret +metadata: + name: actcore-ops-run-worker-tokens + namespace: activity-core + labels: + app.kubernetes.io/name: activity-core + app.kubernetes.io/part-of: activity-core +spec: + refreshInterval: 1h + secretStoreRef: + kind: ClusterSecretStore + name: openbao-activity-core + target: + name: actcore-runtime-secret + creationPolicy: Merge + deletionPolicy: Retain + data: + - secretKey: ACTIVITY_CORE_WORKER_TOKEN + remoteRef: + key: platform/workloads/activity-core/ops-run-workers/rein-aharness-railiance01 + property: token + - secretKey: ACTIVITY_CORE_WORKER_TOKEN_METERED + remoteRef: + key: platform/workloads/activity-core/ops-run-workers/rein-aharness-metered-railiance01 + property: token diff --git a/k8s/railiance/README.md b/k8s/railiance/README.md index 1f7294b..aa1938a 100644 --- a/k8s/railiance/README.md +++ b/k8s/railiance/README.md @@ -45,6 +45,10 @@ workers insert claimable `ops_runs` on emit. Full railiance checklist: harness claim client. The non-secret `ACTIVITY_CORE_WORKER_ID` in the runtime ConfigMap binds that credential to `rein-aharness@railiance01`; deploy both settings together. +Additional worker identities use the non-secret `ACTIVITY_CORE_WORKERS` map, +with one OpenBao-held token per identity synced by +`15-externalsecret-worker-tokens.yaml`. That file is not applied yet; see +ACTIVITY-WP-0039. | ExternalSecret | OpenBao path | Secret key | | --- | --- | --- | diff --git a/src/activity_core/ops_runs_api.py b/src/activity_core/ops_runs_api.py index 649a20f..31797ce 100644 --- a/src/activity_core/ops_runs_api.py +++ b/src/activity_core/ops_runs_api.py @@ -4,6 +4,7 @@ from __future__ import annotations import hmac import os +import re import uuid from collections.abc import Callable from datetime import datetime @@ -38,6 +39,8 @@ _get_db: Callable[[], async_sessionmaker[AsyncSession]] | None = None WORKER_TOKEN_ENV = "ACTIVITY_CORE_WORKER_TOKEN" WORKER_ID_ENV = "ACTIVITY_CORE_WORKER_ID" +WORKERS_ENV = "ACTIVITY_CORE_WORKERS" +_WORKER_TOKEN_ENV_RE = re.compile(r"ACTIVITY_CORE_WORKER_TOKEN(?:_[A-Z0-9]+)*") def bind_ops_runs_deps( @@ -54,6 +57,8 @@ def _db() -> async_sessionmaker[AsyncSession]: def _worker_token_configured() -> bool: + if (os.environ.get(WORKERS_ENV) or "").strip(): + return True return bool((os.environ.get(WORKER_TOKEN_ENV) or "").strip()) @@ -62,6 +67,77 @@ def configured_worker_id() -> str | None: return value or None +class WorkerConfigError(Exception): + """Worker credential configuration is incomplete or ambiguous.""" + + +def worker_credentials() -> list[tuple[str, str]]: + """Return ``(worker_id, token)`` pairs, each token bound to one identity. + + ``ACTIVITY_CORE_WORKERS`` is a non-secret, comma-separated list of + ``worker_id=ENV_NAME`` entries. Each ENV_NAME must be + ``ACTIVITY_CORE_WORKER_TOKEN`` or ``ACTIVITY_CORE_WORKER_TOKEN_`` and + hold that worker's token. Without it, the legacy single pair + ``ACTIVITY_CORE_WORKER_ID`` + ``ACTIVITY_CORE_WORKER_TOKEN`` applies. + Duplicate identities, missing or shared tokens, and a token equal to the + operator token raise ``WorkerConfigError`` so callers fail closed. + """ + raw = (os.environ.get(WORKERS_ENV) or "").strip() + if not raw: + token = (os.environ.get(WORKER_TOKEN_ENV) or "").strip() + if not token: + return [] + worker_id = configured_worker_id() + if not worker_id: + raise WorkerConfigError(f"worker identity not configured ({WORKER_ID_ENV})") + pairs = [(worker_id, token)] + else: + pairs = [] + for entry in raw.split(","): + entry = entry.strip() + if not entry: + continue + worker_id, sep, env_name = (part.strip() for part in entry.partition("=")) + if not sep or not worker_id or _WORKER_TOKEN_ENV_RE.fullmatch(env_name) is None: + raise WorkerConfigError(f"invalid {WORKERS_ENV} entry") + token = (os.environ.get(env_name) or "").strip() + if not token: + raise WorkerConfigError(f"worker token not configured ({env_name})") + pairs.append((worker_id, token)) + if not pairs: + raise WorkerConfigError(f"{WORKERS_ENV} lists no workers") + legacy_id = configured_worker_id() + if legacy_id and legacy_id not in {worker_id for worker_id, _ in pairs}: + raise WorkerConfigError(f"{WORKER_ID_ENV} is not listed in {WORKERS_ENV}") + + ids = [worker_id for worker_id, _ in pairs] + tokens = [token for _, token in pairs] + if len(set(ids)) != len(ids): + raise WorkerConfigError("duplicate worker identity") + if len(set(tokens)) != len(tokens): + raise WorkerConfigError("worker tokens must not be shared") + operator = (os.environ.get("ACTIVITY_CORE_OPERATOR_TOKEN") or "").strip() + if operator and operator in tokens: + raise WorkerConfigError("worker token must differ from the operator token") + return pairs + + +def _authenticate_worker_token(token: str | None) -> str | None: + """Return the identity bound to ``token``; raise 503 on bad configuration.""" + try: + pairs = worker_credentials() + except WorkerConfigError as exc: + raise HTTPException(status_code=503, detail=str(exc)) from exc + if not token: + return None + matched = None + for worker_id, expected in pairs: + # Compare against every entry so timing does not reveal the position. + if hmac.compare_digest(token, expected): + matched = worker_id + return matched + + def _extract_worker_token( *, x_worker_token: str | None, @@ -85,18 +161,15 @@ def require_worker_or_operator( worker_tok = _extract_worker_token( x_worker_token=x_worker_token, authorization=authorization ) - expected_worker = (os.environ.get(WORKER_TOKEN_ENV) or "").strip() - if ( - expected_worker - and worker_tok - and hmac.compare_digest(worker_tok, expected_worker) - ): - worker_id = configured_worker_id() - if not worker_id: - raise HTTPException( - status_code=503, - detail=f"worker identity not configured ({WORKER_ID_ENV})", - ) + worker_id = None + if worker_tok: + try: + worker_id = _authenticate_worker_token(worker_tok) + except HTTPException: + # Broken worker config must not lock out operator/SSO reads; + # worker-only mutations still fail closed in require_worker. + worker_id = None + if worker_id: return f"worker:{worker_id}" sso = extract_sso_principal(request) @@ -125,22 +198,16 @@ def require_worker( x_worker_token: str | None = None, authorization: str | None = None, ) -> str: - """Return the configured worker identity for a valid worker credential.""" + """Return the identity bound to a valid worker credential.""" del request # reserved for future mTLS/proxy-bound worker principals - expected_worker = (os.environ.get(WORKER_TOKEN_ENV) or "").strip() worker_tok = _extract_worker_token( x_worker_token=x_worker_token, authorization=authorization ) - if expected_worker: - if not worker_tok or not hmac.compare_digest(worker_tok, expected_worker): - raise HTTPException(status_code=401, detail="worker auth required") - worker_id = configured_worker_id() + if _worker_token_configured(): + worker_id = _authenticate_worker_token(worker_tok) if not worker_id: - raise HTTPException( - status_code=503, - detail=f"worker identity not configured ({WORKER_ID_ENV})", - ) + raise HTTPException(status_code=401, detail="worker auth required") return worker_id if allow_unauth_mutations() and not operator_token_configured(): diff --git a/tests/test_ops_runs_api.py b/tests/test_ops_runs_api.py index 4d4290f..31bac6b 100644 --- a/tests/test_ops_runs_api.py +++ b/tests/test_ops_runs_api.py @@ -165,3 +165,113 @@ def test_close_response_distinguishes_reconciled_and_refusal_codes( _close_response(CloseOpsRunOutcome("expired_lease", row)) assert exc.value.status_code == 409 assert exc.value.detail["code"] == "expired_lease" + + +def _clear_worker_env(monkeypatch: pytest.MonkeyPatch) -> None: + for name in ( + "ACTIVITY_CORE_WORKERS", + "ACTIVITY_CORE_WORKER_ID", + "ACTIVITY_CORE_WORKER_TOKEN", + "ACTIVITY_CORE_WORKER_TOKEN_METERED", + "ACTIVITY_CORE_OPERATOR_TOKEN", + "ACTIVITY_CORE_OPS_ALLOW_UNAUTH_MUTATIONS", + ): + monkeypatch.delenv(name, raising=False) + + +def _two_workers(monkeypatch: pytest.MonkeyPatch) -> None: + _clear_worker_env(monkeypatch) + monkeypatch.setenv( + "ACTIVITY_CORE_WORKERS", + "rein-aharness@railiance01=ACTIVITY_CORE_WORKER_TOKEN," + "rein-aharness-metered@railiance01=ACTIVITY_CORE_WORKER_TOKEN_METERED", + ) + monkeypatch.setenv("ACTIVITY_CORE_WORKER_TOKEN", "loop-secret") + monkeypatch.setenv("ACTIVITY_CORE_WORKER_TOKEN_METERED", "metered-secret") + + +def test_each_worker_token_binds_exactly_its_own_identity( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _two_workers(monkeypatch) + + loop = require_worker(_request(), x_worker_token="loop-secret") + metered = require_worker(_request(), authorization="Bearer metered-secret") + + assert loop == "rein-aharness@railiance01" + assert metered == "rein-aharness-metered@railiance01" + with pytest.raises(HTTPException) as exc: + bind_worker_id("rein-aharness@railiance01", metered) + assert exc.value.status_code == 403 + with pytest.raises(HTTPException) as exc: + require_worker(_request(), x_worker_token="unknown") + assert exc.value.status_code == 401 + + +def test_claim_endpoint_rejects_cross_identity_body( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _two_workers(monkeypatch) + + response = _client().post( + "/ops-runs/claim", + json={"worker_id": "rein-aharness@railiance01", "limit": 1}, + headers={"X-Worker-Token": "metered-secret"}, + ) + + assert response.status_code == 403 + + +@pytest.mark.parametrize( + ("workers", "env", "fragment"), + [ + ("a=ACTIVITY_CORE_WORKER_TOKEN,b=ACTIVITY_CORE_WORKER_TOKEN_METERED", + {"ACTIVITY_CORE_WORKER_TOKEN": "same", "ACTIVITY_CORE_WORKER_TOKEN_METERED": "same"}, + "shared"), + ("a=ACTIVITY_CORE_WORKER_TOKEN,a=ACTIVITY_CORE_WORKER_TOKEN_METERED", + {"ACTIVITY_CORE_WORKER_TOKEN": "x", "ACTIVITY_CORE_WORKER_TOKEN_METERED": "y"}, + "duplicate"), + ("a=ACTIVITY_CORE_WORKER_TOKEN,b=ACTIVITY_CORE_WORKER_TOKEN_METERED", + {"ACTIVITY_CORE_WORKER_TOKEN": "x"}, + "ACTIVITY_CORE_WORKER_TOKEN_METERED"), + ("a=ACTIVITY_CORE_OPERATOR_TOKEN", + {"ACTIVITY_CORE_OPERATOR_TOKEN": "op"}, + "invalid"), + ("a=ACTIVITY_CORE_WORKER_TOKEN", + {"ACTIVITY_CORE_WORKER_TOKEN": "op", "ACTIVITY_CORE_OPERATOR_TOKEN": "op"}, + "operator"), + ("a=ACTIVITY_CORE_WORKER_TOKEN", + {"ACTIVITY_CORE_WORKER_TOKEN": "x", "ACTIVITY_CORE_WORKER_ID": "b"}, + "not listed"), + ], +) +def test_ambiguous_worker_config_fails_closed( + monkeypatch: pytest.MonkeyPatch, + workers: str, + env: dict[str, str], + fragment: str, +) -> None: + _clear_worker_env(monkeypatch) + monkeypatch.setenv("ACTIVITY_CORE_WORKERS", workers) + for name, value in env.items(): + monkeypatch.setenv(name, value) + + with pytest.raises(HTTPException) as exc: + require_worker(_request(), x_worker_token="x") + + assert exc.value.status_code == 503 + assert fragment in exc.value.detail + + +def test_broken_worker_config_does_not_lock_out_operator( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _clear_worker_env(monkeypatch) + monkeypatch.setenv("ACTIVITY_CORE_WORKERS", "a=ACTIVITY_CORE_WORKER_TOKEN_MISSING") + monkeypatch.setenv("ACTIVITY_CORE_OPERATOR_TOKEN", "operator-secret") + + principal = require_worker_or_operator( + _request(), authorization="Bearer operator-secret" + ) + + assert principal == "operator:token" diff --git a/workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md b/workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md new file mode 100644 index 0000000..e56092f --- /dev/null +++ b/workplans/ACTIVITY-WP-0039-multi-worker-identity-and-token-custody.md @@ -0,0 +1,141 @@ +--- +id: ACTIVITY-WP-0039 +type: workplan +title: "Multiple queue worker identities with OpenBao-custodied tokens" +domain: infotech +repo: activity-core +status: active +flavor: implementation +owner: claude-code +topic_slug: activity-core +priority: high +created: "2026-09-23" +updated: "2026-09-23" +related: + - ACT-ADR-005 + - ACTIVITY-WP-0036 + - SECRETS-WP-0009-T03 + - SECRETS-WP-0011-T04 + - RPF-WP-0045 +--- + +# Multiple Queue Worker Identities with OpenBao-Custodied Tokens + +## Origin + +Two requests from secrets-engine, both approved on their side on 2026-09-23: + +- Message `914853d9` (SECRETS-WP-0009-T03) asks for more than one + authenticated queue worker. The spend-admitted `rein-aharness metered-once` + owner should claim as `rein-aharness-metered@railiance01`, separate from the + running `rein-aharness@railiance01` claim loop. This keeps paid-run + attribution distinct at the API. +- Message `6e694682` (SECRETS-WP-0011-T04) asks for the worker token to be held + in OpenBao and synced by the existing `openbao-activity-core` store, instead + of the hand-generated `openssl rand` value in + `docs/deploy-ops-run-queue-railiance.md`. secrets-engine will catalog a + read-only lane for the metered worker's path only. It will never read, copy + or mint the token. + +WP-0036 bound one token to one configured identity. This workplan keeps that +guarantee and extends it to a set of identities. + +## Boundary rules + +- Each token is bound to exactly one worker identity. The body `worker_id` + must still match it. No token is shared, and nothing falls back implicitly. +- Worker tokens never equal the operator token, and operator/SSO credentials + stay out of worker mutations (WP-0036). +- One OpenBao path per worker identity, so a consumer lane can be granted + exactly one worker's token. +- Minting and writing a token is founder-attended OpenBao work. The store's + role policy belongs to railiance-platform. No token value passes through + Git, State Hub, chat, or an agent shell. + +## Token-to-identity map in the worker API + +```task +id: ACTIVITY-WP-0039-T01 +status: done +priority: high +``` + +Add `ACTIVITY_CORE_WORKERS`, a non-secret comma-separated list of +`worker_id=ENV_NAME` entries. Each ENV_NAME must be `ACTIVITY_CORE_WORKER_TOKEN` +or `ACTIVITY_CORE_WORKER_TOKEN_`. When the list is unset, the legacy +`ACTIVITY_CORE_WORKER_ID` + `ACTIVITY_CORE_WORKER_TOKEN` pair behaves exactly as +before. Worker mutations fail closed with 503 when the configuration is +ambiguous. That covers a duplicate identity, a missing or shared token, a token +equal to the operator token, an env name outside the pattern, and a legacy +identity missing from the list. A broken worker configuration does not lock out +operator or SSO reads. + +Implemented in `src/activity_core/ops_runs_api.py` (`worker_credentials`). +Tests in `tests/test_ops_runs_api.py` cover per-identity binding, a +cross-identity body rejected with 403, an unknown token rejected with 401, each +fail-closed case, and operator access under broken worker configuration. + +## Declare OpenBao paths and the ExternalSecret + +```task +id: ACTIVITY-WP-0039-T02 +status: wait +priority: high +``` + +Paths, one per identity, each with field `token`: + +- `platform/workloads/activity-core/ops-run-workers/rein-aharness-railiance01` +- `platform/workloads/activity-core/ops-run-workers/rein-aharness-metered-railiance01` + +`k8s/railiance/15-externalsecret-worker-tokens.yaml` merges them into +`actcore-runtime-secret` as `ACTIVITY_CORE_WORKER_TOKEN` and +`ACTIVITY_CORE_WORKER_TOKEN_METERED`. The manifest is in the repo but not +applied. + +Waiting on railiance-platform to add both exact paths to the +`activity-core-eso` role policy (RPF-WP-0045 pattern), and on secrets-engine to +confirm the metered path for its catalog. + +## Seed the tokens in OpenBao + +```task +id: ACTIVITY-WP-0039-T03 +status: wait +priority: high +``` + +Founder-attended, through `warden access openbao-platform-admin-login --exec` +(orientation section 5): + +- **Metered worker:** mint a fresh random value directly into its path. No + value exists today. +- **Claim-loop worker:** the founder chooses between two options. + - (a) Move the current value into OpenBao. The running claim loop needs no + change, but the value was hand-generated outside custody. + - (b) Mint a fresh value and update the rein-aharness claim-loop + configuration on railiance01 (user tegwick) in the same window. This means + one coordinated restart. + +Done when both paths hold a value and no value has been printed or logged. + +## Roll out and prove both identities + +```task +id: ACTIVITY-WP-0039-T04 +status: wait +priority: high +``` + +Needs the founder's go-ahead (`ADMINISTER @ realm:kubernetes/railiance01`, +`activation=APPROVED`) and a CPU headroom check first (orientation section 4). +Apply the ExternalSecret, add `ACTIVITY_CORE_WORKERS` to +`actcore-runtime-config`, deploy the new image, and restart only the API. +Prove four things: + +- The claim loop still polls with HTTP 200. +- The metered token authenticates as `rein-aharness-metered@railiance01`. +- A metered token paired with the loop identity is rejected with HTTP 403. +- The previous hand-set Secret key is now owned by ESO. + +Report the revision to secrets-engine on threads `914853d9` and `6e694682`.