activity-core/docs/deploy-ops-run-queue-railiance.md

# Deploy notes — ops_run claim queue on railiance (ACTIVITY-WP-0026-T07)

**Audience:** operator deploying activity-core after WP-0026 code lands  
**Spec:** [ops-run-queue.md](ops-run-queue.md) · **ADR:** [ACT-ADR-005](adr/adr-005-ops-runs-vs-dev-work-records.md)  
**Consumer next:** rein-aharness [REIN-A-0002](https://forgejo.coulomb.social/coulomb/rein-aharness)  
**Image tag:** `activity-core:railiance01-prod`  
**Namespace:** `activity-core`

This is the **prod rollout checklist**, not the implementation. Code path
defaults `OPS_RUN_QUEUE_ENABLED=true`; railiance still needs image + migration
+ ConfigMap apply before open rows appear.

---

## Goals

1. Ship migration **`0007_create_ops_runs`** to railiance Postgres.  
2. Expose claim API on **actcore-api** (image that includes WP-0026).  
3. Worker **emit_tasks** dual-writes open `ops_run` rows.  
4. Smoke: FI (or Binky) one-shot → `GET /ops-runs?state=open` shows a row.  
5. Host systemd timers are break-glass only once claim loop is proven
   (FI dual-clock timer disabled 2026-08-05 after per-fire ops_run keys).

## Non-goals

- Implementing rein-aharness claim (REIN-A-0002).  
- Flipping `ISSUE_SINK_TYPE` to `rest` / Forgejo.  
- Workstation cron.

---

## Pre-flight

| Check | Command / note |
| ----- | -------------- |
| Code on `main` | Includes `migrations/versions/0007_create_ops_runs.py`, `ops_run_queue.py`, `ops_runs_api.py` (commit ≥ `15eb3a2`) |
| Cluster access | `export KUBECONFIG=…` or `ssh railiance01` with k3s |
| API healthy today | `kubectl -n activity-core exec deploy/actcore-api -- … /health` |
| FI definition present | Name contains `fi-daily` / Freedom Intelligence; id from inventory |
| Host timers still run | Keep FI/Binky user-systemd timers until claim proven |

Lookup definition id (use for trigger smoke):

```bash
kubectl -n activity-core exec deploy/actcore-api -- \
  /app/.venv/bin/python3 -c '
import urllib.request, json
d=json.loads(urllib.request.urlopen("http://127.0.0.1:8010/ops/automations").read())
for a in d.get("items") or d.get("automations") or []:
  name=(a.get("name") or "")
  if "reedom" in name or "fi-daily" in name.lower() or "FI " in name:
    print(a.get("id"), name, a.get("enabled"))
'
```

If the inventory shape differs, use:

```bash
curl -sS "http://127.0.0.1:8010/ops/automations" | python3 -m json.tool | less
# via port-forward or SSO host activity.coulomb.social
```

---

## Checklist (execute in order)

### 1. Build and load image

From activity-core checkout (workstation or build host):

```bash
cd ~/activity-core
git pull --ff-only
docker build -t activity-core:railiance01-prod .
docker save -o /tmp/activity-core-railiance01-prod.tar activity-core:railiance01-prod
scp /tmp/activity-core-railiance01-prod.tar railiance01:/tmp/
ssh railiance01 sudo k3s ctr images import /tmp/activity-core-railiance01-prod.tar
```

- [ ] Image import succeeded (`k3s ctr images ls | grep activity-core`)

### 2. Sync manifests + ConfigMap flags

`actcore-runtime-config` must include (defaults match code if omitted, but
explicit is better for ops):

| Key | Value | Notes |
| --- | ----- | ----- |
| `OPS_RUN_QUEUE_ENABLED` | `true` | Create ops_run on emit |
| `OPS_RUN_LEASE_SECONDS` | `900` | Optional |
| `OPS_RUN_MAX_ATTEMPTS` | `3` | Optional |
| `OPS_RUN_SLA_HOURS` | `1` | Status stuck threshold |
| `ACTIVITY_CORE_WORKER_ID` | `rein-aharness@railiance01` | Exact non-secret identity bound to the worker token |

Worker auth (required before external claim):

```bash
# Generate once; store in secret — do not commit
WORKER_TOKEN="$(openssl rand -hex 24)"
kubectl -n activity-core patch secret actcore-runtime-secret --type merge \
  -p "{\"stringData\":{\"ACTIVITY_CORE_WORKER_TOKEN\":\"${WORKER_TOKEN}\"}}"
# Record token in operator secret store (OpenBao / password manager), not chat.
```

Apply the ConfigMap identity and Secret token in the same rollout. If the token
is present without `ACTIVITY_CORE_WORKER_ID`, worker mutations fail with 503;
if a request body names another identity, they fail with 403.

Apply:

```bash
rsync -a k8s/railiance/ railiance01:activity-core/k8s/railiance/
ssh railiance01 'cd ~/activity-core && kubectl apply -f k8s/railiance/20-runtime.yaml'
```

- [ ] ConfigMap shows `OPS_RUN_QUEUE_ENABLED=true`  
      `kubectl -n activity-core get cm actcore-runtime-config -o yaml | grep OPS_RUN`

### 3. Migration job (0007)

Jobs are immutable; delete and re-apply so the new image runs:

```bash
kubectl -n activity-core delete job actcore-migrate --ignore-not-found
kubectl apply -f k8s/railiance/20-runtime.yaml   # or only the Job if split later
kubectl -n activity-core wait --for=condition=complete job/actcore-migrate --timeout=180s
kubectl -n activity-core logs job/actcore-migrate
```

Expect alembic to reach revision **`0007`** (creates table `ops_runs`).

Verify table:

```bash
kubectl -n activity-core exec deploy/actcore-api -- \
  /app/.venv/bin/python3 -c '
import asyncio, os
from sqlalchemy.ext.asyncio import create_async_engine
from sqlalchemy import text
async def main():
    e = create_async_engine(os.environ["ACTCORE_DB_URL"])
    async with e.connect() as c:
        r = await c.execute(text(
            "select to_regclass('\''public.ops_runs'\'') is not null"
        ))
        print("ops_runs_exists", r.scalar())
    await e.dispose()
asyncio.run(main())
'
```

- [ ] Job complete  
- [ ] `ops_runs_exists True`

### 4. Roll API + worker (+ event-router)

```bash
kubectl -n activity-core rollout restart deploy/actcore-api deploy/actcore-worker deploy/actcore-event-router
kubectl -n activity-core rollout status deploy/actcore-api --timeout=180s
kubectl -n activity-core rollout status deploy/actcore-worker --timeout=180s
kubectl -n activity-core rollout status deploy/actcore-event-router --timeout=180s
```

Confirm OpenAPI / routes:

```bash
kubectl -n activity-core exec deploy/actcore-api -- \
  /app/.venv/bin/python3 -c '
import urllib.request, json
# paths that must exist after WP-0026
for path in ["/ops-runs", "/ops/automations/status"]:
  try:
    urllib.request.urlopen(f"http://127.0.0.1:8010{path}")
    print(path, "ok")
  except Exception as e:
    # 401 still proves route exists
    print(path, type(e).__name__, getattr(e, "code", e))
'
```

- [ ] Rollouts healthy  
- [ ] `/ops-runs` reachable (200 or 401, not 404)

### 5. Smoke — emit open ops_run

Prefer **operator trigger** (does not wait for next cron):

```bash
# Port-forward if needed
kubectl -n activity-core port-forward svc/actcore-api 8010:8010 &
# Operator token from secret (break-glass) or SSO via activity.coulomb.social
OP_TOKEN=$(kubectl -n activity-core get secret actcore-runtime-secret \
  -o jsonpath='{.data.ACTIVITY_CORE_OPERATOR_TOKEN}' | base64 -d)

DEF_ID="<fi-activity-definition-uuid>"   # from pre-flight lookup
curl -sS -X POST "http://127.0.0.1:8010/ops/automations/${DEF_ID}/trigger" \
  -H "Content-Type: application/json" \
  -H "X-Operator-Token: ${OP_TOKEN}" \
  -d '{}' | python3 -m json.tool
```

If FI context has `due=false` (brief already posted today), expect **no** new
task/ops_run. Options:

- Trigger a definition known to emit, or  
- Temporarily clear today's completion only in a **test** domain, or  
- Wait for next weekday 07:30 Europe/Berlin fire when due.

List open runs:

```bash
curl -sS "http://127.0.0.1:8010/ops-runs?state=open" \
  -H "X-Operator-Token: ${OP_TOKEN}" | python3 -m json.tool
```

Status block:

```bash
curl -sS "http://127.0.0.1:8010/ops/automations/status?since=today" \
  | python3 -c "import sys,json; print(json.load(sys.stdin).get('ops_runs'))"
```

- [ ] After a successful emit: at least one `state=open` row with labels including
      `research-brief` / `automated` (or Binky equivalents)  
- [ ] `counts.open >= 1` (or expected)  
- [ ] Dual-write still visible: State Hub `activity_task_spawn` progress (unchanged path)

### 6. Claim path smoke (manual, no harness yet)

```bash
WORKER_TOKEN=…   # from secret
curl -sS -X POST "http://127.0.0.1:8010/ops-runs/claim" \
  -H "Content-Type: application/json" \
  -H "X-Worker-Token: ${WORKER_TOKEN}" \
  -d '{"worker_id":"rein-aharness@railiance01","labels":["automated"],"limit":1,"lease_seconds":120}' \
  | python3 -m json.tool
```

Then **complete or fail with reopen** so the run does not sit claimed forever:

```bash
RUN_ID=…   # from claim response
# Prefer fail+reopen if you did not execute real work:
curl -sS -X POST "http://127.0.0.1:8010/ops-runs/${RUN_ID}/fail" \
  -H "Content-Type: application/json" \
  -H "X-Worker-Token: ${WORKER_TOKEN}" \
  -d '{"worker_id":"rein-aharness@railiance01","error":"T07 smoke only","reopen":true}'
```

- [ ] Claim returns the open run  
- [ ] Fail+reopen returns `state=open` (or complete if intentional)

### 7. Dual-path residual (keep until REIN-A-0002)

| Path | Status after T07 |
| ---- | ---------------- |
| Temporal schedule → emit → **ops_run** | **On** (this checklist) |
| State Hub `activity_task_spawn` dual-write | **On** (visibility) |
| railiance host timers → rein-aharness | **Keep** until claim loop proven |
| issue-core / Forgejo as claim queue | **Off** (never for FI/Binky ops) |

- [ ] Confirmed timers not deleted  
- [ ] REIN-A-0002 unblocked (API contract live)

---

## Rollback

| Situation | Action |
| --------- | ------ |
| Bad API/worker image | Roll back image tag / previous tar; restart deploys |
| Migration issues | Table is additive; app old image ignores `ops_runs`. To disable only: set `OPS_RUN_QUEUE_ENABLED=false` on ConfigMap + restart worker |
| Claim spam / bad consumers | Set worker token; revoke old token; fail open runs manually |

Disable without redeploying code:

```bash
kubectl -n activity-core set env deploy/actcore-worker OPS_RUN_QUEUE_ENABLED=false
kubectl -n activity-core set env deploy/actcore-api OPS_RUN_QUEUE_ENABLED=false
# Prefer ConfigMap edit + rollout so Job/migrate stay consistent
```

Dropping `ops_runs` is **not** required for rollback; leave the table.

---

## Railiance smoke log (2026-08-03)

- Migrated to alembic **0007**; `ops_runs` table present  
- FI def `3169ab1f-…` manual trigger → open ops_run → claim-loop selected `fi-research-brief`  
- Completed as **succeeded** (`brief already exists for today`)  
- User systemd: `rein-aharness-claim-loop.service` enabled+active; API via host `kubectl port-forward` on :8010  
- Worker token in `actcore-runtime-secret` + `~/.config/rein-aharness/claim-loop.env`

## Acceptance (T07 done when)

- [x] Migration 0007 applied on railiance  
- [x] `OPS_RUN_QUEUE_ENABLED=true` on runtime ConfigMap  
- [x] API serves `/ops-runs/*`  
- [x] At least one real emit produced `state=open` (then succeeded) (or documented due=false skip + schedule observation)  
- [x] Manual claim smoke succeeded  
- [x] Host timers still in place; REIN-A-0002 claim-loop running; T05 cutover next  

Mark **ACTIVITY-WP-0026-T07** `done` and WP status `done` only after the boxes above.

---

## Related

- `k8s/railiance/README.md` — full stack deploy  
- `docs/runbook.md` — ops_run curl cheatsheet  
- `docs/task-emission-consumer-contract.md` — harness contract  
- `docs/recurring-automations-playbook.md` — organizer layers  
- REIN-A-0002 — claim loop + approach table + timer cutover