--- id: ACTIVITY-WP-0026 type: workplan title: "Ops run claim queue — durable claimable work for scheduled automation" domain: infotech repo: activity-core status: finished owner: grok topic_slug: activity-core priority: high created: "2026-08-03" updated: "2026-08-18" depends_on: [] related: - ACT-ADR-005 - ACTIVITY-WP-0022 - ACTIVITY-WP-0023 - REIN-A-0002 - ISSUE-WP-0006 - STATE-WP-0078 state_hub_workstream_id: "d05e49dd-dbc9-59f4-bbc8-8bfced64d4a8" --- # ACTIVITY-WP-0026 — Ops run claim queue ## Origin ACT-ADR-005 (ops runs vs development work records). Closes **G2 executor gap** without reopening default **Forgejo** issue spam (ACTIVITY-WP-0022). ## Goal Make internal scheduled automation emit a **claimable `ops_run`** in activity-core so rein-aharness (and peers) can claim, execute, complete, or fail with lease semantics — while activity-core stays when/what/where only. **Non-goals:** execute domain work in Temporal activities; use issue-core or Forgejo as the claim queue; invent Gitea support. ## Architecture (target) ```text Temporal schedule / trigger → resolve context (e.g. fi_brief_status) → rules produce TaskSpec(s) → INSERT ops_run (open) + optional activity_task_spawn progress → rein-aharness claim loop → complete/fail + domain completion event (fi_daily_brief, …) ``` ## Tasks ### T01 — Spec ops_run model and API ```task id: ACTIVITY-WP-0026-T01 status: done priority: high state_hub_task_id: "7e284d2e-09d4-53c3-88ab-88ac1ded033b" ``` Document in `docs/ops-run-queue.md`: | Field | Notes | | ----- | ----- | | `id` | UUIDv7 | | `activity_definition_id` | FK | | `idempotency_key` | unique; e.g. `{def}:{date}` or hash of spawn key | | `target_repo` | string | | `title`, `description` | from TaskSpec | | `labels` | JSON array | | `priority` | | | `state` | `open \| claimed \| succeeded \| failed \| expired` | | `claim_owner` | worker identity | | `lease_until` | timestamptz | | `attempt` | int | | `source_type`, `source_id`, `triggering_event_id` | audit | | `approach_hint` | optional from definition/rule | | `result` | JSON completion metadata | | `created_at`, `updated_at` | | API (REST on actcore-api): - `POST /ops-runs/claim` body: `{ worker_id, labels?, limit? }` → runs - `POST /ops-runs/{id}/heartbeat` extend lease - `POST /ops-runs/{id}/complete` / `fail` - `GET /ops-runs` filter by state, definition, since Auth: same operator/worker posture as existing admin APIs (document; prefer service credential for harness, SSO for humans later). **Done when:** spec reviewed against ACT-ADR-005; OpenAPI-ready field list. --- ### T02 — Schema migration + ORM ```task id: ACTIVITY-WP-0026-T02 status: done priority: high state_hub_task_id: "fd0444bd-1062-54d5-b250-1aa275b82c44" ``` Alembic migration for `ops_runs` (+ indexes on `state`, `lease_until`, `idempotency_key` unique). SQLAlchemy model. Unit tests for constraints. **Done when:** migrate up/down clean on dev; model tests green. --- ### T03 — Emit path: create ops_run on TaskSpec emit ```task id: ACTIVITY-WP-0026-T03 status: done priority: high state_hub_task_id: "50d4c22f-1674-5459-95f4-99e0a7592d18" ``` On successful rule emit (internal fleet path): 1. Upsert/insert `ops_run` with idempotency (duplicate key → no second open row). 2. Keep **dual-write** `activity_task_spawn` progress for transition period. 3. Still write `task_spawn_log` audit. 4. **Never** require Forgejo / issue-core rest for this path. Default sink remains `state-hub` for progress; ops_run is **additional** primary for claim. **Done when:** schedule fire or `/trigger` produces open `ops_run` for FI and Binky-style definitions; tests with mocked sink. --- ### T04 — Claim / lease / expire worker helpers ```task id: ACTIVITY-WP-0026-T04 status: done priority: high state_hub_task_id: "dce47143-c67b-5698-8c2e-c3e2697870c3" ``` Implement claim with `SELECT … FOR UPDATE SKIP LOCKED` (or equivalent) and lease TTL (config: e.g. 15m). Expire job or on-claim sweep: `claimed` past `lease_until` → `open` (retry) or `expired` after max attempts. **Done when:** concurrent claim tests; lease expiry documented. --- ### T05 — Ops visibility: list + SLA signals ```task id: ACTIVITY-WP-0026-T05 status: done priority: medium state_hub_task_id: "ef00ea27-e0f6-5451-abe1-655c1e224312" ``` - Extend `/ops/automations/status` (or sibling) with open/claimed/failed counts. - Signal: “spawned/open without complete by SLA” (config, default 1h for daily). - Document in runbook. **Done when:** operator can see stuck ops_runs without SQL. --- ### T06 — Contract + consumer guide ```task id: ACTIVITY-WP-0026-T06 status: done priority: medium state_hub_task_id: "9705d46d-4088-5d56-be27-2c0cc23c6806" ``` Update: - `docs/task-emission-consumer-contract.md` — claim ops_run, not issue-core - `docs/recurring-automations-playbook.md` — reference ACT-ADR-005 - `docs/issue-core-emission-boundary.md` — ops_run is not issue-core - Use **Forgejo** naming only (no Gitea product references) **Done when:** REIN-A-0002 can implement against the contract without guesswork. --- ### T07 — Railiance rollout + dual-path flag ```task id: ACTIVITY-WP-0026-T07 status: done priority: medium state_hub_task_id: "962a22bf-4f18-58f8-bc0a-873f59916792" ``` **Deploy notes:** [`docs/deploy-ops-run-queue-railiance.md`](../docs/deploy-ops-run-queue-railiance.md) Railiance smoke **2026-08-03** (T07 complete): - [x] Image `activity-core:railiance01-prod` imported; migrate **0006→0007**; `ops_runs` live - [x] `OPS_RUN_QUEUE_ENABLED=true` on ConfigMap; API/worker rolled - [x] FI trigger → open ops_run `8d04877e-…` (labels research-brief/automated) - [x] Manual claim/fail+reopen + claim-loop dry-run + real complete (`skipped_existing` → **succeeded**) - [x] `rein-aharness-claim-loop.service` user unit **active** on railiance01 - [x] Host FI timer still break-glass (REIN-A-0002-T05 proof window) **Done when:** prod smoke checklist green on railiance; flag live. ✅ ## Acceptance - [x] ACT-ADR-005 referenced in SCOPE gaps (G2 disposition → this WP) - [x] ops_run durable + claimable without Forgejo - [x] Emit dual-write progress for transition - [x] Docs use Forgejo-only language for self-hosted forge - [x] REIN-A-0002 unblocked (contract in docs/ops-run-queue.md + consumer contract) **T07:** railiance rollout complete 2026-08-03 (see task notes). ## Out of scope - rein-aharness claim loop (REIN-A-0002) - State Hub full write path for claim (read projection = STATE-WP-0078) - issue-core internal SQLite “fake queue” - Gitea compatibility