Harden ops run identity and leases
Assistant: codex Assistant-Model: gpt-5.6-sol Assistant-Session: 01a028de-e2c8-7732-8521-46a7fc5db82f
This commit is contained in:
parent
36161d346f
commit
f0a897e088
13 changed files with 522 additions and 50 deletions
|
|
@ -96,13 +96,20 @@ When a definition emits a TaskSpec for internal fleet work:
|
|||
3. Do **not** open a Forgejo issue.
|
||||
4. Do **not** create a workplan task file for that day’s fire.
|
||||
|
||||
Claim API (sketch; implement in ACTIVITY-WP-0026):
|
||||
Claim API (implemented in ACTIVITY-WP-0026 and hardened in
|
||||
ACTIVITY-WP-0036):
|
||||
|
||||
- `POST /ops-runs/claim` — lease next open run matching labels / worker id
|
||||
- `POST /ops-runs/{id}/complete` — succeeded + completion metadata
|
||||
- `POST /ops-runs/{id}/fail` — failed + retry policy
|
||||
- `GET /ops-runs?state=open` — operator visibility
|
||||
|
||||
Worker mutation credentials are bound to one configured queue `worker_id`;
|
||||
the request body cannot assert a different claim owner. Heartbeat, completion,
|
||||
and failure lock the row and require its lease deadline to remain strictly in
|
||||
the future. Operator/SSO credentials provide visibility and explicit
|
||||
administration, but do not act as a normal worker identity.
|
||||
|
||||
activity-core remains **when / what / where** only: it does **not** run
|
||||
domain LLM sessions or hold tenant git credentials.
|
||||
|
||||
|
|
|
|||
|
|
@ -90,8 +90,9 @@ explicit is better for ops):
|
|||
| `OPS_RUN_LEASE_SECONDS` | `900` | Optional |
|
||||
| `OPS_RUN_MAX_ATTEMPTS` | `3` | Optional |
|
||||
| `OPS_RUN_SLA_HOURS` | `1` | Status stuck threshold |
|
||||
| `ACTIVITY_CORE_WORKER_ID` | `rein-aharness@railiance01` | Exact non-secret identity bound to the worker token |
|
||||
|
||||
Optional worker auth (recommended before external claim):
|
||||
Worker auth (required before external claim):
|
||||
|
||||
```bash
|
||||
# Generate once; store in secret — do not commit
|
||||
|
|
@ -101,6 +102,10 @@ kubectl -n activity-core patch secret actcore-runtime-secret --type merge \
|
|||
# Record token in operator secret store (OpenBao / password manager), not chat.
|
||||
```
|
||||
|
||||
Apply the ConfigMap identity and Secret token in the same rollout. If the token
|
||||
is present without `ACTIVITY_CORE_WORKER_ID`, worker mutations fail with 503;
|
||||
if a request body names another identity, they fail with 403.
|
||||
|
||||
Apply:
|
||||
|
||||
```bash
|
||||
|
|
@ -223,11 +228,11 @@ curl -sS "http://127.0.0.1:8010/ops/automations/status?since=today" \
|
|||
### 6. Claim path smoke (manual, no harness yet)
|
||||
|
||||
```bash
|
||||
WORKER_TOKEN=… # from secret if set; else local-dev open auth
|
||||
WORKER_TOKEN=… # from secret
|
||||
curl -sS -X POST "http://127.0.0.1:8010/ops-runs/claim" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "X-Worker-Token: ${WORKER_TOKEN}" \
|
||||
-d '{"worker_id":"smoke@railiance01","labels":["automated"],"limit":1,"lease_seconds":120}' \
|
||||
-d '{"worker_id":"rein-aharness@railiance01","labels":["automated"],"limit":1,"lease_seconds":120}' \
|
||||
| python3 -m json.tool
|
||||
```
|
||||
|
||||
|
|
@ -239,7 +244,7 @@ RUN_ID=… # from claim response
|
|||
curl -sS -X POST "http://127.0.0.1:8010/ops-runs/${RUN_ID}/fail" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "X-Worker-Token: ${WORKER_TOKEN}" \
|
||||
-d '{"worker_id":"smoke@railiance01","error":"T07 smoke only","reopen":true}'
|
||||
-d '{"worker_id":"rein-aharness@railiance01","error":"T07 smoke only","reopen":true}'
|
||||
```
|
||||
|
||||
- [ ] Claim returns the open run
|
||||
|
|
|
|||
|
|
@ -59,7 +59,9 @@ workplan task file. Not an issue-core or Forgejo ticket.
|
|||
- `labels_mode`: `any` (default) — run must contain at least one listed label;
|
||||
`all` — run must contain every listed label; omit `labels` to claim any open run.
|
||||
- Claim uses `FOR UPDATE SKIP LOCKED` for concurrency safety.
|
||||
- Stale claims (`state=claimed` and `lease_until < now()`) are reopened before select.
|
||||
- Stale claims (`state=claimed` and `lease_until <= now()`) are reopened before select.
|
||||
- Heartbeat, complete, and fail lock the row and require an active lease
|
||||
(`lease_until > now()`). An expired worker cannot revive or close its claim.
|
||||
|
||||
### Complete / fail body
|
||||
|
||||
|
|
@ -150,9 +152,14 @@ model output are persisted or returned.
|
|||
## Auth
|
||||
|
||||
- **Worker:** `ACTIVITY_CORE_WORKER_TOKEN` via `X-Worker-Token` or
|
||||
`Authorization: Bearer` (same value accepted on claim/complete/fail/heartbeat).
|
||||
- **Operator:** existing ops SSO / `ACTIVITY_CORE_OPERATOR_TOKEN` for list/status.
|
||||
- **Local dev:** `ACTIVITY_CORE_OPS_ALLOW_UNAUTH_MUTATIONS=1` when no tokens set.
|
||||
`Authorization: Bearer`, bound to the exact non-secret
|
||||
`ACTIVITY_CORE_WORKER_ID`. The body `worker_id` is a compatibility field and
|
||||
must match that authenticated identity on claim/complete/fail/heartbeat.
|
||||
- **Operator:** existing ops SSO / `ACTIVITY_CORE_OPERATOR_TOKEN` for
|
||||
list/status and explicit lease expiry; operator credentials are not accepted
|
||||
as worker mutation identities.
|
||||
- **Local dev:** `ACTIVITY_CORE_OPS_ALLOW_UNAUTH_MUTATIONS=1` is required when
|
||||
no tokens are set. There is no implicit open mode.
|
||||
|
||||
## Env
|
||||
|
||||
|
|
@ -162,6 +169,7 @@ model output are persisted or returned.
|
|||
| `OPS_RUN_LEASE_SECONDS` | `900` | Default claim lease |
|
||||
| `OPS_RUN_MAX_ATTEMPTS` | `3` | Fail permanently after N claims |
|
||||
| `ACTIVITY_CORE_WORKER_TOKEN` | unset | Harness claim credential |
|
||||
| `ACTIVITY_CORE_WORKER_ID` | unset | Exact queue identity bound to the worker credential; required when the token is set |
|
||||
|
||||
## Consumer (rein-aharness)
|
||||
|
||||
|
|
|
|||
|
|
@ -456,7 +456,7 @@ curl -sS -X POST "http://localhost:8010/ops-runs/claim" \
|
|||
|
||||
# Reopen stale leases
|
||||
curl -sS -X POST "http://localhost:8010/ops-runs/expire-leases" \
|
||||
-H "X-Worker-Token: $ACTIVITY_CORE_WORKER_TOKEN"
|
||||
-H "X-Operator-Token: $ACTIVITY_CORE_OPERATOR_TOKEN"
|
||||
```
|
||||
|
||||
| Env | Default | Meaning |
|
||||
|
|
@ -466,6 +466,12 @@ curl -sS -X POST "http://localhost:8010/ops-runs/expire-leases" \
|
|||
| `OPS_RUN_MAX_ATTEMPTS` | `3` | Fail permanently after N claims |
|
||||
| `OPS_RUN_SLA_HOURS` | `1` | Stuck threshold in status |
|
||||
| `ACTIVITY_CORE_WORKER_TOKEN` | unset | Harness claim auth |
|
||||
| `ACTIVITY_CORE_WORKER_ID` | unset | Exact worker identity bound to claim auth |
|
||||
|
||||
Worker mutations reject a body `worker_id` that differs from
|
||||
`ACTIVITY_CORE_WORKER_ID`, and heartbeat/complete/fail reject a missing or
|
||||
expired lease. Set the token and identity together before starting the claim
|
||||
consumer.
|
||||
|
||||
Glas-backed workers may return the Glas 1.0 `GatewayResult` on complete or
|
||||
fail. Activity Core persists only `result.execution_evidence` plus the existing
|
||||
|
|
|
|||
|
|
@ -22,6 +22,11 @@ issue-core and not a Forgejo ticket.
|
|||
| Complete | `POST /ops-runs/{id}/complete` | `{ worker_id, result }` |
|
||||
| Fail | `POST /ops-runs/{id}/fail` | `{ worker_id, error, reopen? }` |
|
||||
|
||||
The worker token is bound by Activity Core to one configured `worker_id`.
|
||||
Every worker mutation must name that exact identity and must still hold a lease
|
||||
whose deadline is strictly in the future. Operator/SSO authentication does not
|
||||
substitute for worker authentication on these calls.
|
||||
|
||||
Full field list, auth, and env: **`docs/ops-run-queue.md`**.
|
||||
Consumer implementation (rein-aharness): **REIN-A-0002**.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue