activity-core/docs/deploy-ops-run-queue-railiance.md

Deploy notes — ops_run claim queue on railiance (ACTIVITY-WP-0026-T07)

Audience: operator deploying activity-core after WP-0026 code lands
Spec: ops-run-queue.md · ADR: ACT-ADR-005
Consumer next: rein-aharness REIN-A-0002
Image tag: activity-core:railiance01-prod
Namespace: activity-core

This is the prod rollout checklist, not the implementation. Code path defaults OPS_RUN_QUEUE_ENABLED=true; railiance still needs image + migration

• ConfigMap apply before open rows appear.

---

Goals

1. Ship migration 0007_create_ops_runs to railiance Postgres.
2. Expose claim API on actcore-api (image that includes WP-0026).
3. Worker emit_tasks dual-writes open ops_run rows.
4. Smoke: FI (or Binky) one-shot → GET /ops-runs?state=open shows a row.
5. Host systemd timers are break-glass only once claim loop is proven (FI dual-clock timer disabled 2026-08-05 after per-fire ops_run keys).

Non-goals

• Implementing rein-aharness claim (REIN-A-0002).
• Flipping ISSUE_SINK_TYPE to rest / Forgejo.
• Workstation cron.

---

Pre-flight

Check	Command / note
Code on main	Includes migrations/versions/0007_create_ops_runs.py, ops_run_queue.py, ops_runs_api.py (commit ≥ 15eb3a2)
Cluster access	export KUBECONFIG=… or ssh railiance01 with k3s
API healthy today	kubectl -n activity-core exec deploy/actcore-api -- … /health
FI definition present	Name contains fi-daily / Freedom Intelligence; id from inventory
Host timers still run	Keep FI/Binky user-systemd timers until claim proven

Lookup definition id (use for trigger smoke):

kubectl -n activity-core exec deploy/actcore-api -- \
  /app/.venv/bin/python3 -c '
import urllib.request, json
d=json.loads(urllib.request.urlopen("http://127.0.0.1:8010/ops/automations").read())
for a in d.get("items") or d.get("automations") or []:
  name=(a.get("name") or "")
  if "reedom" in name or "fi-daily" in name.lower() or "FI " in name:
    print(a.get("id"), name, a.get("enabled"))
'

If the inventory shape differs, use:

curl -sS "http://127.0.0.1:8010/ops/automations" | python3 -m json.tool | less
# via port-forward or SSO host activity.coulomb.social

---

Checklist (execute in order)

1. Build and load image

From activity-core checkout (workstation or build host):

cd ~/activity-core
git pull --ff-only
docker build -t activity-core:railiance01-prod .
docker save -o /tmp/activity-core-railiance01-prod.tar activity-core:railiance01-prod
scp /tmp/activity-core-railiance01-prod.tar railiance01:/tmp/
ssh railiance01 sudo k3s ctr images import /tmp/activity-core-railiance01-prod.tar

• Image import succeeded (k3s ctr images ls | grep activity-core)

2. Sync manifests + ConfigMap flags

actcore-runtime-config must include (defaults match code if omitted, but explicit is better for ops):

Key	Value	Notes
OPS_RUN_QUEUE_ENABLED	true	Create ops_run on emit
OPS_RUN_LEASE_SECONDS	900	Optional
OPS_RUN_MAX_ATTEMPTS	3	Optional
OPS_RUN_SLA_HOURS	1	Status stuck threshold
ACTIVITY_CORE_WORKER_ID	rein-aharness@railiance01	Exact non-secret identity bound to the worker token

Worker auth (required before external claim):

# Generate once; store in secret — do not commit
WORKER_TOKEN="$(openssl rand -hex 24)"
kubectl -n activity-core patch secret actcore-runtime-secret --type merge \
  -p "{\"stringData\":{\"ACTIVITY_CORE_WORKER_TOKEN\":\"${WORKER_TOKEN}\"}}"
# Record token in operator secret store (OpenBao / password manager), not chat.

Apply the ConfigMap identity and Secret token in the same rollout. If the token is present without ACTIVITY_CORE_WORKER_ID, worker mutations fail with 503; if a request body names another identity, they fail with 403.

Apply:

rsync -a k8s/railiance/ railiance01:activity-core/k8s/railiance/
ssh railiance01 'cd ~/activity-core && kubectl apply -f k8s/railiance/20-runtime.yaml'

• ConfigMap shows OPS_RUN_QUEUE_ENABLED=true
kubectl -n activity-core get cm actcore-runtime-config -o yaml | grep OPS_RUN

3. Migration job (0007)

Jobs are immutable; delete and re-apply so the new image runs:

kubectl -n activity-core delete job actcore-migrate --ignore-not-found
kubectl apply -f k8s/railiance/20-runtime.yaml   # or only the Job if split later
kubectl -n activity-core wait --for=condition=complete job/actcore-migrate --timeout=180s
kubectl -n activity-core logs job/actcore-migrate

Expect alembic to reach revision 0007 (creates table ops_runs).

Verify table:

kubectl -n activity-core exec deploy/actcore-api -- \
  /app/.venv/bin/python3 -c '
import asyncio, os
from sqlalchemy.ext.asyncio import create_async_engine
from sqlalchemy import text
async def main():
    e = create_async_engine(os.environ["ACTCORE_DB_URL"])
    async with e.connect() as c:
        r = await c.execute(text(
            "select to_regclass('\''public.ops_runs'\'') is not null"
        ))
        print("ops_runs_exists", r.scalar())
    await e.dispose()
asyncio.run(main())
'

• Job complete
• ops_runs_exists True

4. Roll API + worker (+ event-router)

kubectl -n activity-core rollout restart deploy/actcore-api deploy/actcore-worker deploy/actcore-event-router
kubectl -n activity-core rollout status deploy/actcore-api --timeout=180s
kubectl -n activity-core rollout status deploy/actcore-worker --timeout=180s
kubectl -n activity-core rollout status deploy/actcore-event-router --timeout=180s

Confirm OpenAPI / routes:

kubectl -n activity-core exec deploy/actcore-api -- \
  /app/.venv/bin/python3 -c '
import urllib.request, json
# paths that must exist after WP-0026
for path in ["/ops-runs", "/ops/automations/status"]:
  try:
    urllib.request.urlopen(f"http://127.0.0.1:8010{path}")
    print(path, "ok")
  except Exception as e:
    # 401 still proves route exists
    print(path, type(e).__name__, getattr(e, "code", e))
'

• Rollouts healthy
• /ops-runs reachable (200 or 401, not 404)

5. Smoke — emit open ops_run

Prefer operator trigger (does not wait for next cron):

# Port-forward if needed
kubectl -n activity-core port-forward svc/actcore-api 8010:8010 &
# Operator token from secret (break-glass) or SSO via activity.coulomb.social
OP_TOKEN=$(kubectl -n activity-core get secret actcore-runtime-secret \
  -o jsonpath='{.data.ACTIVITY_CORE_OPERATOR_TOKEN}' | base64 -d)

DEF_ID="<fi-activity-definition-uuid>"   # from pre-flight lookup
curl -sS -X POST "http://127.0.0.1:8010/ops/automations/${DEF_ID}/trigger" \
  -H "Content-Type: application/json" \
  -H "X-Operator-Token: ${OP_TOKEN}" \
  -d '{}' | python3 -m json.tool

If FI context has due=false (brief already posted today), expect no new task/ops_run. Options:

• Trigger a definition known to emit, or
• Temporarily clear today's completion only in a test domain, or
• Wait for next weekday 07:30 Europe/Berlin fire when due.

List open runs:

curl -sS "http://127.0.0.1:8010/ops-runs?state=open" \
  -H "X-Operator-Token: ${OP_TOKEN}" | python3 -m json.tool

Status block:

curl -sS "http://127.0.0.1:8010/ops/automations/status?since=today" \
  | python3 -c "import sys,json; print(json.load(sys.stdin).get('ops_runs'))"

• After a successful emit: at least one state=open row with labels including research-brief / automated (or Binky equivalents)
• counts.open >= 1 (or expected)
• Dual-write still visible: State Hub activity_task_spawn progress (unchanged path)

6. Claim path smoke (manual, no harness yet)

WORKER_TOKEN=…   # from secret
curl -sS -X POST "http://127.0.0.1:8010/ops-runs/claim" \
  -H "Content-Type: application/json" \
  -H "X-Worker-Token: ${WORKER_TOKEN}" \
  -d '{"worker_id":"rein-aharness@railiance01","labels":["automated"],"limit":1,"lease_seconds":120}' \
  | python3 -m json.tool

Then complete or fail with reopen so the run does not sit claimed forever:

RUN_ID=…   # from claim response
# Prefer fail+reopen if you did not execute real work:
curl -sS -X POST "http://127.0.0.1:8010/ops-runs/${RUN_ID}/fail" \
  -H "Content-Type: application/json" \
  -H "X-Worker-Token: ${WORKER_TOKEN}" \
  -d '{"worker_id":"rein-aharness@railiance01","error":"T07 smoke only","reopen":true}'

• Claim returns the open run
• Fail+reopen returns state=open (or complete if intentional)

7. Dual-path residual (keep until REIN-A-0002)

Path	Status after T07
Temporal schedule → emit → ops_run	On (this checklist)
State Hub activity_task_spawn dual-write	On (visibility)
railiance host timers → rein-aharness	Keep until claim loop proven
issue-core / Forgejo as claim queue	Off (never for FI/Binky ops)

• Confirmed timers not deleted
• REIN-A-0002 unblocked (API contract live)

---

Rollback

Situation	Action
Bad API/worker image	Roll back image tag / previous tar; restart deploys
Migration issues	Table is additive; app old image ignores ops_runs. To disable only: set OPS_RUN_QUEUE_ENABLED=false on ConfigMap + restart worker
Claim spam / bad consumers	Set worker token; revoke old token; fail open runs manually

Disable without redeploying code:

kubectl -n activity-core set env deploy/actcore-worker OPS_RUN_QUEUE_ENABLED=false
kubectl -n activity-core set env deploy/actcore-api OPS_RUN_QUEUE_ENABLED=false
# Prefer ConfigMap edit + rollout so Job/migrate stay consistent

Dropping ops_runs is not required for rollback; leave the table.

---

Railiance smoke log (2026-08-03)

• Migrated to alembic 0007; ops_runs table present
• FI def 3169ab1f-… manual trigger → open ops_run → claim-loop selected fi-research-brief
• Completed as succeeded (brief already exists for today)
• User systemd: rein-aharness-claim-loop.service enabled+active; API via host kubectl port-forward on :8010
• Worker token in actcore-runtime-secret + ~/.config/rein-aharness/claim-loop.env

Acceptance (T07 done when)

• Migration 0007 applied on railiance
• OPS_RUN_QUEUE_ENABLED=true on runtime ConfigMap
• API serves /ops-runs/*
• At least one real emit produced state=open (then succeeded) (or documented due=false skip + schedule observation)
• Manual claim smoke succeeded
• Host timers still in place; REIN-A-0002 claim-loop running; T05 cutover next

Mark ACTIVITY-WP-0026-T07 done and WP status done only after the boxes above.

---

Related

• k8s/railiance/README.md — full stack deploy
• docs/runbook.md — ops_run curl cheatsheet
• docs/task-emission-consumer-contract.md — harness contract
• docs/recurring-automations-playbook.md — organizer layers
• REIN-A-0002 — claim loop + approach table + timer cutover