Implement ACTIVITY-WP-0026 ops_run claim queue (T01–T06).

Add durable claimable ops_runs table, emit dual-write on TaskSpec, REST claim/lease/complete/fail API, ops status visibility, and consumer docs aligned with ACT-ADR-005. T07 railiance rollout remains deploy-side.
2026-08-03 19:22:50 +02:00 · 2026-08-03 19:22:50 +02:00 · 15eb3a2066
commit 15eb3a2066
parent a145cc4027
15 changed files with 1294 additions and 27 deletions
--- a/docs/issue-core-emission-boundary.md
+++ b/docs/issue-core-emission-boundary.md
@ -3,6 +3,20 @@
 activity-core owns the decision to spawn a task and the audit trail that says
 why it spawned. It does not own downstream task lifecycle state after emission.

+## Ops claim vs issue-core (ACT-ADR-005 / ACTIVITY-WP-0026)
+
+| Concern | Home |
+| ------- | ---- |
+| **Claimable automation run** | `ops_runs` table + `POST /ops-runs/claim` in activity-core |
+| **Fleet visibility / progress** | IssueSink (`state-hub` default) |
+| **External tracker ticket** | issue-core → **Forgejo** (opt-in `ISSUE_SINK_TYPE=rest` only) |
+
+**`ops_run` is not issue-core.** Harnesses must claim ops runs; they must not
+poll issue-core or Forgejo for scheduled FI/Binky-style work. Full contract:
+`docs/ops-run-queue.md` and `docs/task-emission-consumer-contract.md`.
+
+Self-hosted forge product name: **Forgejo** only (no Gitea support path).
+
 ## Sink matrix (ACTIVITY-WP-0022)

 | `ISSUE_SINK_TYPE` | Destination | Default? |
@ -72,6 +86,9 @@ activity-core.
 Failed to connect to backend 'forgejo-inbox': Failed to connect to Gitea API
 ```

+(Historical error string from the Forgejo-compatible client library — the
+deployed forge is **Forgejo**, not Gitea.)
+
 **Disposition (ACTIVITY-WP-0023-T06):** activity-core keeps global default
 `state-hub` and does **not** flip production to `rest`. Path A is owned by
 **issue-core**: rotate/fix `GITEA_BACKEND_TOKEN` (Forgejo backend PAT for the
@ -79,6 +96,9 @@ forgejo-inbox connector — not the activity-core `ISSUE_CORE_API_KEY`
 ingestion key). After issue-core proves `POST /issues/` → **201**, operators may
 opt in per definition / overlay only (WP-0022).

+Internal scheduled automation should use **ops_run claim** (WP-0026), not wait
+on this rest path.
+
 Smoke from worker (does not change sink env):

 ```bash
--- a/docs/ops-run-queue.md
+++ b/docs/ops-run-queue.md
@ -0,0 +1,111 @@
+# Ops run claim queue
+
+**Status:** implemented (ACTIVITY-WP-0026 code path; railiance rollout = T07)  
+**Architecture:** [ACT-ADR-005](adr/adr-005-ops-runs-vs-dev-work-records.md)
+
+Durable, claimable work instances for **scheduled / automation** runs. Not a
+workplan task file. Not an issue-core or Forgejo ticket.
+
+## Model
+
+| Field | Type | Notes |
+| ----- | ---- | ----- |
+| `id` | UUID | Primary key (uuid4; UUIDv7 optional later) |
+| `activity_definition_id` | UUID | FK → activity_definitions |
+| `idempotency_key` | text | **Unique**; default `{activity_id}:{source_id}:{triggering_event_id}` |
+| `target_repo` | text | From TaskSpec |
+| `title` | text | |
+| `description` | text | |
+| `labels` | JSONB array | e.g. `["automated","research-brief"]` |
+| `priority` | text | low \| medium \| high |
+| `state` | text | `open` \| `claimed` \| `succeeded` \| `failed` \| `expired` |
+| `claim_owner` | text nullable | Worker identity |
+| `lease_until` | timestamptz nullable | Claim lease deadline |
+| `attempt` | int | Starts at 0; incremented on each claim |
+| `source_type` | text | rule \| instruction |
+| `source_id` | text | Rule id |
+| `triggering_event_id` | text | Event or workflow key |
+| `approach_hint` | text nullable | Optional from rule |
+| `result` | JSONB | Completion metadata |
+| `created_at` / `updated_at` | timestamptz | |
+
+## API (actcore-api)
+
+| Method | Path | Role |
+| ------ | ---- | ---- |
+| `GET` | `/ops-runs` | List/filter |
+| `GET` | `/ops-runs/{id}` | One row |
+| `POST` | `/ops-runs/claim` | Claim open runs (lease) |
+| `POST` | `/ops-runs/{id}/heartbeat` | Extend lease |
+| `POST` | `/ops-runs/{id}/complete` | succeeded |
+| `POST` | `/ops-runs/{id}/fail` | failed (+ optional reopen) |
+| `POST` | `/ops-runs/expire-leases` | Reopen or expire stale claims |
+
+### Claim body
+
+```json
+{
+  "worker_id": "rein-aharness@railiance01",
+  "labels": ["automated"],
+  "labels_mode": "any",
+  "limit": 1,
+  "lease_seconds": 900
+}
+```
+
+- `labels_mode`: `any` (default) — run must contain at least one listed label;
+  `all` — run must contain every listed label; omit `labels` to claim any open run.
+- Claim uses `FOR UPDATE SKIP LOCKED` for concurrency safety.
+- Stale claims (`state=claimed` and `lease_until < now()`) are reopened before select.
+
+### Complete / fail body
+
+```json
+{ "result": { "path": "briefs/…", "ok": true }, "worker_id": "…" }
+```
+
+```json
+{ "error": "llm timeout", "worker_id": "…", "reopen": false }
+```
+
+If `reopen: true` and `attempt < max_attempts` (env `OPS_RUN_MAX_ATTEMPTS`, default 3),
+state returns to `open`; else `failed`.
+
+## Emit path
+
+On `emit_tasks` (when `OPS_RUN_QUEUE_ENABLED` is truthy, **default true**):
+
+1. Insert `ops_run` with `state=open` (idempotent on unique key).  
+2. Dual-write existing IssueSink (`state-hub` progress by default).  
+3. Write `task_spawn_log` audit as today.
+
+Never requires Forgejo or issue-core for the claim path.
+
+## Auth
+
+- **Worker:** `ACTIVITY_CORE_WORKER_TOKEN` via `X-Worker-Token` or
+  `Authorization: Bearer` (same value accepted on claim/complete/fail/heartbeat).  
+- **Operator:** existing ops SSO / `ACTIVITY_CORE_OPERATOR_TOKEN` for list/status.  
+- **Local dev:** `ACTIVITY_CORE_OPS_ALLOW_UNAUTH_MUTATIONS=1` when no tokens set.
+
+## Env
+
+| Variable | Default | Meaning |
+| -------- | ------- | ------- |
+| `OPS_RUN_QUEUE_ENABLED` | `true` | Create ops_run on emit |
+| `OPS_RUN_LEASE_SECONDS` | `900` | Default claim lease |
+| `OPS_RUN_MAX_ATTEMPTS` | `3` | Fail permanently after N claims |
+| `ACTIVITY_CORE_WORKER_TOKEN` | unset | Harness claim credential |
+
+## Consumer (rein-aharness)
+
+See REIN-A-0002. Claim loop → approach table → execute → complete + domain
+completion event (`fi_daily_brief`, etc.).
+
+## Not this queue
+
+| Concern | Home |
+| ------- | ---- |
+| Multi-day engineering tasks | Workplan files + State Hub |
+| External tracker tickets | issue-core → **Forgejo** (optional projection) |
+| Schedule truth | Temporal + activity definitions |
--- a/docs/recurring-automations-playbook.md
+++ b/docs/recurring-automations-playbook.md
@ -20,14 +20,15 @@ issue-core projection — never as the default ops claim queue.
 ┌─────────────────────────────────────────────────────────────┐
 │ 1. activity-core  — WHEN / WHAT / WHERE                     │
 │    Temporal schedule · context resolvers · rules            │
-│    → emits activity_task_spawn (state-hub sink by default)  │
+│    → INSERT ops_run (open) + dual-write activity_task_spawn │
+│    Claim API: POST /ops-runs/claim  (docs/ops-run-queue.md) │
 └────────────────────────────┬────────────────────────────────┘
                             │
 ┌────────────────────────────▼────────────────────────────────┐
 │ 2. rein-aharness (or domain executor) — DOES THE WORK       │
-│    llm-connect · repo checkout · commit · completion event  │
-│    Host timers on railiance are *interim* until poll/claim  │
-│    of spawns is fully wired (state-hub sink is not claimable)│
+│    claim ops_run · llm-connect · checkout · complete/fail   │
+│    Host timers on railiance are *interim* until REIN-A-0002 │
+│    claim loop is live (state-hub spawn alone is not claimable)│
 └────────────────────────────┬────────────────────────────────┘
                             │
 ┌────────────────────────────▼────────────────────────────────┐
@ -151,17 +152,22 @@ Do not flip production to `rest` globally.

 ## Operator daily habit

-1. Ops console: did automations run? (`/ops/automations/status`)  
-2. State Hub: `activity_task_spawn` + domain completion events  
-3. Domain repo: brief/artifact files + git history  
-4. If something is missing, **fix activity-core first** (schedule paused?
-   definition disabled? resolver due stuck? sink errors?) — do not invent a new cron  
+1. Ops console: did automations run? (`/ops/automations/status`) — includes
+   `ops_runs.counts` and `stuck_open_or_claimed` (SLA, default 1h)  
+2. Claim queue: `GET /ops-runs?state=open` (or failed) — harness backlog  
+3. State Hub: `activity_task_spawn` + domain completion events  
+4. Domain repo: brief/artifact files + git history  
+5. If something is missing, **fix activity-core first** (schedule paused?
+   definition disabled? resolver due stuck? open ops_run with no claim?) —
+   do not invent a new cron  

 ---

 ## Related

 - `INTENT.md` — when/what/where boundary  
+- `docs/adr/adr-005-ops-runs-vs-dev-work-records.md` — ops vs dev work  
+- `docs/ops-run-queue.md` — claim API  
 - `docs/task-emission-consumer-contract.md` — spawn payload + consumer duties  
 - `docs/runbook.md` — sync, trigger, ops UI  
 - `docs/adr/adr-002-definition-format.md` — definition files  
--- a/docs/runbook.md
+++ b/docs/runbook.md
@ -396,6 +396,43 @@ Default: **`ISSUE_SINK_TYPE=state-hub`** (ACTIVITY-WP-0022). See
 `review_required` on instructions is **metadata only** until a downstream
 review queue exists (issue-core / work-record lane) — see ACTIVITY-WP-0023-T09.

+## Ops run claim queue (ACTIVITY-WP-0026)
+
+Durable claimable instances for scheduled automation. Spec:
+`docs/ops-run-queue.md`. Architecture: ACT-ADR-005.
+
+```bash
+# Visibility (counts + stuck SLA in status)
+curl -sS "http://localhost:8010/ops/automations/status?since=today" \
+  | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('ops_runs'))"
+
+# Open runs
+curl -sS "http://localhost:8010/ops-runs?state=open" | python3 -m json.tool
+
+# Claim (worker)
+curl -sS -X POST "http://localhost:8010/ops-runs/claim" \
+  -H "Content-Type: application/json" \
+  -H "X-Worker-Token: $ACTIVITY_CORE_WORKER_TOKEN" \
+  -d '{"worker_id":"rein-aharness@railiance01","labels":["automated"],"limit":1}'
+
+# Reopen stale leases
+curl -sS -X POST "http://localhost:8010/ops-runs/expire-leases" \
+  -H "X-Worker-Token: $ACTIVITY_CORE_WORKER_TOKEN"
+```
+
+| Env | Default | Meaning |
+| --- | --- | --- |
+| `OPS_RUN_QUEUE_ENABLED` | `true` | Insert ops_run on emit |
+| `OPS_RUN_LEASE_SECONDS` | `900` | Claim lease |
+| `OPS_RUN_MAX_ATTEMPTS` | `3` | Fail permanently after N claims |
+| `OPS_RUN_SLA_HOURS` | `1` | Stuck threshold in status |
+| `ACTIVITY_CORE_WORKER_TOKEN` | unset | Harness claim auth |
+
+**Railiance rollout (T07):** apply migration `0007`, set
+`OPS_RUN_QUEUE_ENABLED=true` on worker/api, smoke-trigger FI definition, confirm
+open row via `GET /ops-runs?state=open`. Keep host timers until REIN-A-0002 claim
+loop is proven.
+
 Example distinction from the June 2026 daily triage evidence:

 ```text
--- a/docs/task-emission-consumer-contract.md
+++ b/docs/task-emission-consumer-contract.md
@ -2,12 +2,34 @@

 **Audience:** agent-harness, per-repo Temporal workers, operators.  
 **Owners:** activity-core (producer), consumers (executors).  
-**Related:** ACTIVITY-WP-0022 (sink policy), ACTIVITY-WP-0023-T02 (executor gap).
+**Related:** ACTIVITY-WP-0022 (sink policy), ACTIVITY-WP-0023-T02 (executor gap),
+**ACT-ADR-005 / ACTIVITY-WP-0026** (ops_run claim queue).

 activity-core answers **when / what / where**. It does **not** execute work.
 Consumers must pick up emitted tasks and produce domain evidence.

-## Sink matrix (ACTIVITY-WP-0022)
+## Primary claim path: `ops_run` (ACTIVITY-WP-0026)
+
+For **scheduled / automation** work (FI daily brief, Binky rhythm, etc.), the
+authoritative claimable instance is an **`ops_run`** row in activity-core — not
+issue-core and not a Forgejo ticket.
+
+| Step | Call | Notes |
+| ---- | ---- | ----- |
+| List / poll | `GET /ops-runs?state=open` | Filter by labels via claim body |
+| Claim | `POST /ops-runs/claim` | `{ worker_id, labels?, limit?, lease_seconds? }` |
+| Heartbeat | `POST /ops-runs/{id}/heartbeat` | Extend lease during long runs |
+| Complete | `POST /ops-runs/{id}/complete` | `{ worker_id, result }` |
+| Fail | `POST /ops-runs/{id}/fail` | `{ worker_id, error, reopen? }` |
+
+Full field list, auth, and env: **`docs/ops-run-queue.md`**.  
+Consumer implementation (rein-aharness): **REIN-A-0002**.
+
+Emit dual-writes: `ops_run` (claim) + existing IssueSink progress (`state-hub`
+by default) + `task_spawn_log` audit. Do **not** treat `activity_task_spawn`
+progress as a claim queue — it is visibility only.
+
+## Sink matrix (ACTIVITY-WP-0022) — dual-write / projection

 | `ISSUE_SINK_TYPE` | Behaviour | When to use |
 | --- | --- | --- |
@ -15,12 +37,13 @@ Consumers must pick up emitted tasks and produce domain evidence.
 | **`null`** | Synthetic `null-*` refs in `task_spawn_log` only | Dry-run / contract review |
 | **`rest`** | POST issue-core `/issues/` (may project to Forgejo) | **Explicit opt-in** only when external tracker issues are intended and backend is healthy |

-Unset or unknown values fall back to **`state-hub`** (safe default).
+Unset or unknown values fall back to **`state-hub`** (safe default).  
+Ops claim does **not** require `rest` or issue-core.

 ## Payload: `activity_task_spawn` (State Hub)

 Produced by `StateHubProgressSink`. Consumers should treat `detail` as the
-authoritative task spec.
+**visibility** task spec; claim authority is `ops_run` when the queue is enabled.

 ```json
 {
@ -75,4 +98,7 @@ and still post the progress event — spawn without completion leaves `due=true`
 - Using `TaskExecutorWorkflow` in activity-core for real work (disabled by
  default; ACTIVITY-WP-0023-T08).
 - Global `ISSUE_SINK_TYPE=rest` for all definitions (reintroduces Forgejo spam).
- Treating `task_spawn_log` as task status authority.
+- Treating `task_spawn_log` or State Hub `activity_task_spawn` as claim authority
+  (use `POST /ops-runs/claim` — ACT-ADR-005).
+- Using issue-core or Forgejo as the ops automation claim queue.
+- Product references to **Gitea** — self-hosted forge is **Forgejo** only.