Harden ops run identity and leases
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s
Build and Publish Container Image / build-and-push (push) Successful in 33s

Assistant: codex
Assistant-Model: gpt-5.6-sol
Assistant-Session: 01a028de-e2c8-7732-8521-46a7fc5db82f
This commit is contained in:
tegwick 2026-08-23 13:01:46 +02:00
parent 36161d346f
commit f0a897e088
13 changed files with 522 additions and 50 deletions

View file

@ -96,13 +96,20 @@ When a definition emits a TaskSpec for internal fleet work:
3. Do **not** open a Forgejo issue.
4. Do **not** create a workplan task file for that days fire.
Claim API (sketch; implement in ACTIVITY-WP-0026):
Claim API (implemented in ACTIVITY-WP-0026 and hardened in
ACTIVITY-WP-0036):
- `POST /ops-runs/claim` — lease next open run matching labels / worker id
- `POST /ops-runs/{id}/complete` — succeeded + completion metadata
- `POST /ops-runs/{id}/fail` — failed + retry policy
- `GET /ops-runs?state=open` — operator visibility
Worker mutation credentials are bound to one configured queue `worker_id`;
the request body cannot assert a different claim owner. Heartbeat, completion,
and failure lock the row and require its lease deadline to remain strictly in
the future. Operator/SSO credentials provide visibility and explicit
administration, but do not act as a normal worker identity.
activity-core remains **when / what / where** only: it does **not** run
domain LLM sessions or hold tenant git credentials.

View file

@ -90,8 +90,9 @@ explicit is better for ops):
| `OPS_RUN_LEASE_SECONDS` | `900` | Optional |
| `OPS_RUN_MAX_ATTEMPTS` | `3` | Optional |
| `OPS_RUN_SLA_HOURS` | `1` | Status stuck threshold |
| `ACTIVITY_CORE_WORKER_ID` | `rein-aharness@railiance01` | Exact non-secret identity bound to the worker token |
Optional worker auth (recommended before external claim):
Worker auth (required before external claim):
```bash
# Generate once; store in secret — do not commit
@ -101,6 +102,10 @@ kubectl -n activity-core patch secret actcore-runtime-secret --type merge \
# Record token in operator secret store (OpenBao / password manager), not chat.
```
Apply the ConfigMap identity and Secret token in the same rollout. If the token
is present without `ACTIVITY_CORE_WORKER_ID`, worker mutations fail with 503;
if a request body names another identity, they fail with 403.
Apply:
```bash
@ -223,11 +228,11 @@ curl -sS "http://127.0.0.1:8010/ops/automations/status?since=today" \
### 6. Claim path smoke (manual, no harness yet)
```bash
WORKER_TOKEN=… # from secret if set; else local-dev open auth
WORKER_TOKEN=… # from secret
curl -sS -X POST "http://127.0.0.1:8010/ops-runs/claim" \
-H "Content-Type: application/json" \
-H "X-Worker-Token: ${WORKER_TOKEN}" \
-d '{"worker_id":"smoke@railiance01","labels":["automated"],"limit":1,"lease_seconds":120}' \
-d '{"worker_id":"rein-aharness@railiance01","labels":["automated"],"limit":1,"lease_seconds":120}' \
| python3 -m json.tool
```
@ -239,7 +244,7 @@ RUN_ID=… # from claim response
curl -sS -X POST "http://127.0.0.1:8010/ops-runs/${RUN_ID}/fail" \
-H "Content-Type: application/json" \
-H "X-Worker-Token: ${WORKER_TOKEN}" \
-d '{"worker_id":"smoke@railiance01","error":"T07 smoke only","reopen":true}'
-d '{"worker_id":"rein-aharness@railiance01","error":"T07 smoke only","reopen":true}'
```
- [ ] Claim returns the open run

View file

@ -59,7 +59,9 @@ workplan task file. Not an issue-core or Forgejo ticket.
- `labels_mode`: `any` (default) — run must contain at least one listed label;
`all` — run must contain every listed label; omit `labels` to claim any open run.
- Claim uses `FOR UPDATE SKIP LOCKED` for concurrency safety.
- Stale claims (`state=claimed` and `lease_until < now()`) are reopened before select.
- Stale claims (`state=claimed` and `lease_until <= now()`) are reopened before select.
- Heartbeat, complete, and fail lock the row and require an active lease
(`lease_until > now()`). An expired worker cannot revive or close its claim.
### Complete / fail body
@ -150,9 +152,14 @@ model output are persisted or returned.
## Auth
- **Worker:** `ACTIVITY_CORE_WORKER_TOKEN` via `X-Worker-Token` or
`Authorization: Bearer` (same value accepted on claim/complete/fail/heartbeat).
- **Operator:** existing ops SSO / `ACTIVITY_CORE_OPERATOR_TOKEN` for list/status.
- **Local dev:** `ACTIVITY_CORE_OPS_ALLOW_UNAUTH_MUTATIONS=1` when no tokens set.
`Authorization: Bearer`, bound to the exact non-secret
`ACTIVITY_CORE_WORKER_ID`. The body `worker_id` is a compatibility field and
must match that authenticated identity on claim/complete/fail/heartbeat.
- **Operator:** existing ops SSO / `ACTIVITY_CORE_OPERATOR_TOKEN` for
list/status and explicit lease expiry; operator credentials are not accepted
as worker mutation identities.
- **Local dev:** `ACTIVITY_CORE_OPS_ALLOW_UNAUTH_MUTATIONS=1` is required when
no tokens are set. There is no implicit open mode.
## Env
@ -162,6 +169,7 @@ model output are persisted or returned.
| `OPS_RUN_LEASE_SECONDS` | `900` | Default claim lease |
| `OPS_RUN_MAX_ATTEMPTS` | `3` | Fail permanently after N claims |
| `ACTIVITY_CORE_WORKER_TOKEN` | unset | Harness claim credential |
| `ACTIVITY_CORE_WORKER_ID` | unset | Exact queue identity bound to the worker credential; required when the token is set |
## Consumer (rein-aharness)

View file

@ -456,7 +456,7 @@ curl -sS -X POST "http://localhost:8010/ops-runs/claim" \
# Reopen stale leases
curl -sS -X POST "http://localhost:8010/ops-runs/expire-leases" \
-H "X-Worker-Token: $ACTIVITY_CORE_WORKER_TOKEN"
-H "X-Operator-Token: $ACTIVITY_CORE_OPERATOR_TOKEN"
```
| Env | Default | Meaning |
@ -466,6 +466,12 @@ curl -sS -X POST "http://localhost:8010/ops-runs/expire-leases" \
| `OPS_RUN_MAX_ATTEMPTS` | `3` | Fail permanently after N claims |
| `OPS_RUN_SLA_HOURS` | `1` | Stuck threshold in status |
| `ACTIVITY_CORE_WORKER_TOKEN` | unset | Harness claim auth |
| `ACTIVITY_CORE_WORKER_ID` | unset | Exact worker identity bound to claim auth |
Worker mutations reject a body `worker_id` that differs from
`ACTIVITY_CORE_WORKER_ID`, and heartbeat/complete/fail reject a missing or
expired lease. Set the token and identity together before starting the claim
consumer.
Glas-backed workers may return the Glas 1.0 `GatewayResult` on complete or
fail. Activity Core persists only `result.execution_evidence` plus the existing

View file

@ -22,6 +22,11 @@ issue-core and not a Forgejo ticket.
| Complete | `POST /ops-runs/{id}/complete` | `{ worker_id, result }` |
| Fail | `POST /ops-runs/{id}/fail` | `{ worker_id, error, reopen? }` |
The worker token is bound by Activity Core to one configured `worker_id`.
Every worker mutation must name that exact identity and must still hold a lease
whose deadline is strictly in the future. Operator/SSO authentication does not
substitute for worker authentication on these calls.
Full field list, auth, and env: **`docs/ops-run-queue.md`**.
Consumer implementation (rein-aharness): **REIN-A-0002**.