Document ACTIVITY-WP-0026-T07 railiance deploy checklist.

Add deploy-ops-run-queue-railiance.md (image, migrate 0007, smoke, dual-path residual), wire OPS_RUN_* into runtime ConfigMap, and cross-link runbook/README. T07 stays open until prod smoke is executed.
2026-08-03 19:30:00 +02:00 · 2026-08-03 19:30:00 +02:00 · 6ce77a72bb
commit 6ce77a72bb
parent 32f62ac1ac
8 changed files with 339 additions and 12 deletions
--- a/docs/deploy-ops-run-queue-railiance.md
+++ b/docs/deploy-ops-run-queue-railiance.md
@ -0,0 +1,300 @@
+# Deploy notes — ops_run claim queue on railiance (ACTIVITY-WP-0026-T07)
+
+**Audience:** operator deploying activity-core after WP-0026 code lands  
+**Spec:** [ops-run-queue.md](ops-run-queue.md) · **ADR:** [ACT-ADR-005](adr/adr-005-ops-runs-vs-dev-work-records.md)  
+**Consumer next:** rein-aharness [REIN-A-0002](https://forgejo.coulomb.social/coulomb/rein-aharness)  
+**Image tag:** `activity-core:railiance01-prod`  
+**Namespace:** `activity-core`
+
+This is the **prod rollout checklist**, not the implementation. Code path
+defaults `OPS_RUN_QUEUE_ENABLED=true`; railiance still needs image + migration
+ ConfigMap apply before open rows appear.
+
+---
+
+## Goals
+
+1. Ship migration **`0007_create_ops_runs`** to railiance Postgres.  
+2. Expose claim API on **actcore-api** (image that includes WP-0026).  
+3. Worker **emit_tasks** dual-writes open `ops_run` rows.  
+4. Smoke: FI (or Binky) one-shot → `GET /ops-runs?state=open` shows a row.  
+5. **Do not** remove host systemd timers until REIN-A-0002 claim loop is proven.
+
+## Non-goals
+
+- Implementing rein-aharness claim (REIN-A-0002).  
+- Flipping `ISSUE_SINK_TYPE` to `rest` / Forgejo.  
+- Workstation cron.
+
+---
+
+## Pre-flight
+
+| Check | Command / note |
+| ----- | -------------- |
+| Code on `main` | Includes `migrations/versions/0007_create_ops_runs.py`, `ops_run_queue.py`, `ops_runs_api.py` (commit ≥ `15eb3a2`) |
+| Cluster access | `export KUBECONFIG=…` or `ssh railiance01` with k3s |
+| API healthy today | `kubectl -n activity-core exec deploy/actcore-api -- … /health` |
+| FI definition present | Name contains `fi-daily` / Freedom Intelligence; id from inventory |
+| Host timers still run | Keep FI/Binky user-systemd timers until claim proven |
+
+Lookup definition id (use for trigger smoke):
+
+```bash
+kubectl -n activity-core exec deploy/actcore-api -- \
+  /app/.venv/bin/python3 -c '
+import urllib.request, json
+d=json.loads(urllib.request.urlopen("http://127.0.0.1:8010/ops/automations").read())
+for a in d.get("items") or d.get("automations") or []:
+  name=(a.get("name") or "")
+  if "reedom" in name or "fi-daily" in name.lower() or "FI " in name:
+    print(a.get("id"), name, a.get("enabled"))
+'
+```
+
+If the inventory shape differs, use:
+
+```bash
+curl -sS "http://127.0.0.1:8010/ops/automations" | python3 -m json.tool | less
+# via port-forward or SSO host activity.coulomb.social
+```
+
+---
+
+## Checklist (execute in order)
+
+### 1. Build and load image
+
+From activity-core checkout (workstation or build host):
+
+```bash
+cd ~/activity-core
+git pull --ff-only
+docker build -t activity-core:railiance01-prod .
+docker save -o /tmp/activity-core-railiance01-prod.tar activity-core:railiance01-prod
+scp /tmp/activity-core-railiance01-prod.tar railiance01:/tmp/
+ssh railiance01 sudo k3s ctr images import /tmp/activity-core-railiance01-prod.tar
+```
+
+- [ ] Image import succeeded (`k3s ctr images ls | grep activity-core`)
+
+### 2. Sync manifests + ConfigMap flags
+
+`actcore-runtime-config` must include (defaults match code if omitted, but
+explicit is better for ops):
+
+| Key | Value | Notes |
+| --- | ----- | ----- |
+| `OPS_RUN_QUEUE_ENABLED` | `true` | Create ops_run on emit |
+| `OPS_RUN_LEASE_SECONDS` | `900` | Optional |
+| `OPS_RUN_MAX_ATTEMPTS` | `3` | Optional |
+| `OPS_RUN_SLA_HOURS` | `1` | Status stuck threshold |
+
+Optional worker auth (recommended before external claim):
+
+```bash
+# Generate once; store in secret — do not commit
+WORKER_TOKEN="$(openssl rand -hex 24)"
+kubectl -n activity-core patch secret actcore-runtime-secret --type merge \
+  -p "{\"stringData\":{\"ACTIVITY_CORE_WORKER_TOKEN\":\"${WORKER_TOKEN}\"}}"
+# Record token in operator secret store (OpenBao / password manager), not chat.
+```
+
+Apply:
+
+```bash
+rsync -a k8s/railiance/ railiance01:activity-core/k8s/railiance/
+ssh railiance01 'cd ~/activity-core && kubectl apply -f k8s/railiance/20-runtime.yaml'
+```
+
+- [ ] ConfigMap shows `OPS_RUN_QUEUE_ENABLED=true`  
+      `kubectl -n activity-core get cm actcore-runtime-config -o yaml | grep OPS_RUN`
+
+### 3. Migration job (0007)
+
+Jobs are immutable; delete and re-apply so the new image runs:
+
+```bash
+kubectl -n activity-core delete job actcore-migrate --ignore-not-found
+kubectl apply -f k8s/railiance/20-runtime.yaml   # or only the Job if split later
+kubectl -n activity-core wait --for=condition=complete job/actcore-migrate --timeout=180s
+kubectl -n activity-core logs job/actcore-migrate
+```
+
+Expect alembic to reach revision **`0007`** (creates table `ops_runs`).
+
+Verify table:
+
+```bash
+kubectl -n activity-core exec deploy/actcore-api -- \
+  /app/.venv/bin/python3 -c '
+import asyncio, os
+from sqlalchemy.ext.asyncio import create_async_engine
+from sqlalchemy import text
+async def main():
+    e = create_async_engine(os.environ["ACTCORE_DB_URL"])
+    async with e.connect() as c:
+        r = await c.execute(text(
+            "select to_regclass('\''public.ops_runs'\'') is not null"
+        ))
+        print("ops_runs_exists", r.scalar())
+    await e.dispose()
+asyncio.run(main())
+'
+```
+
+- [ ] Job complete  
+- [ ] `ops_runs_exists True`
+
+### 4. Roll API + worker (+ event-router)
+
+```bash
+kubectl -n activity-core rollout restart deploy/actcore-api deploy/actcore-worker deploy/actcore-event-router
+kubectl -n activity-core rollout status deploy/actcore-api --timeout=180s
+kubectl -n activity-core rollout status deploy/actcore-worker --timeout=180s
+kubectl -n activity-core rollout status deploy/actcore-event-router --timeout=180s
+```
+
+Confirm OpenAPI / routes:
+
+```bash
+kubectl -n activity-core exec deploy/actcore-api -- \
+  /app/.venv/bin/python3 -c '
+import urllib.request, json
+# paths that must exist after WP-0026
+for path in ["/ops-runs", "/ops/automations/status"]:
+  try:
+    urllib.request.urlopen(f"http://127.0.0.1:8010{path}")
+    print(path, "ok")
+  except Exception as e:
+    # 401 still proves route exists
+    print(path, type(e).__name__, getattr(e, "code", e))
+'
+```
+
+- [ ] Rollouts healthy  
+- [ ] `/ops-runs` reachable (200 or 401, not 404)
+
+### 5. Smoke — emit open ops_run
+
+Prefer **operator trigger** (does not wait for next cron):
+
+```bash
+# Port-forward if needed
+kubectl -n activity-core port-forward svc/actcore-api 8010:8010 &
+# Operator token from secret (break-glass) or SSO via activity.coulomb.social
+OP_TOKEN=$(kubectl -n activity-core get secret actcore-runtime-secret \
+  -o jsonpath='{.data.ACTIVITY_CORE_OPERATOR_TOKEN}' | base64 -d)
+
+DEF_ID="<fi-activity-definition-uuid>"   # from pre-flight lookup
+curl -sS -X POST "http://127.0.0.1:8010/ops/automations/${DEF_ID}/trigger" \
+  -H "Content-Type: application/json" \
+  -H "X-Operator-Token: ${OP_TOKEN}" \
+  -d '{}' | python3 -m json.tool
+```
+
+If FI context has `due=false` (brief already posted today), expect **no** new
+task/ops_run. Options:
+
+- Trigger a definition known to emit, or  
+- Temporarily clear today's completion only in a **test** domain, or  
+- Wait for next weekday 07:30 Europe/Berlin fire when due.
+
+List open runs:
+
+```bash
+curl -sS "http://127.0.0.1:8010/ops-runs?state=open" \
+  -H "X-Operator-Token: ${OP_TOKEN}" | python3 -m json.tool
+```
+
+Status block:
+
+```bash
+curl -sS "http://127.0.0.1:8010/ops/automations/status?since=today" \
+  | python3 -c "import sys,json; print(json.load(sys.stdin).get('ops_runs'))"
+```
+
+- [ ] After a successful emit: at least one `state=open` row with labels including
+      `research-brief` / `automated` (or Binky equivalents)  
+- [ ] `counts.open >= 1` (or expected)  
+- [ ] Dual-write still visible: State Hub `activity_task_spawn` progress (unchanged path)
+
+### 6. Claim path smoke (manual, no harness yet)
+
+```bash
+WORKER_TOKEN=…   # from secret if set; else local-dev open auth
+curl -sS -X POST "http://127.0.0.1:8010/ops-runs/claim" \
+  -H "Content-Type: application/json" \
+  -H "X-Worker-Token: ${WORKER_TOKEN}" \
+  -d '{"worker_id":"smoke@railiance01","labels":["automated"],"limit":1,"lease_seconds":120}' \
+  | python3 -m json.tool
+```
+
+Then **complete or fail with reopen** so the run does not sit claimed forever:
+
+```bash
+RUN_ID=…   # from claim response
+# Prefer fail+reopen if you did not execute real work:
+curl -sS -X POST "http://127.0.0.1:8010/ops-runs/${RUN_ID}/fail" \
+  -H "Content-Type: application/json" \
+  -H "X-Worker-Token: ${WORKER_TOKEN}" \
+  -d '{"worker_id":"smoke@railiance01","error":"T07 smoke only","reopen":true}'
+```
+
+- [ ] Claim returns the open run  
+- [ ] Fail+reopen returns `state=open` (or complete if intentional)
+
+### 7. Dual-path residual (keep until REIN-A-0002)
+
+| Path | Status after T07 |
+| ---- | ---------------- |
+| Temporal schedule → emit → **ops_run** | **On** (this checklist) |
+| State Hub `activity_task_spawn` dual-write | **On** (visibility) |
+| railiance host timers → rein-aharness | **Keep** until claim loop proven |
+| issue-core / Forgejo as claim queue | **Off** (never for FI/Binky ops) |
+
+- [ ] Confirmed timers not deleted  
+- [ ] REIN-A-0002 unblocked (API contract live)
+
+---
+
+## Rollback
+
+| Situation | Action |
+| --------- | ------ |
+| Bad API/worker image | Roll back image tag / previous tar; restart deploys |
+| Migration issues | Table is additive; app old image ignores `ops_runs`. To disable only: set `OPS_RUN_QUEUE_ENABLED=false` on ConfigMap + restart worker |
+| Claim spam / bad consumers | Set worker token; revoke old token; fail open runs manually |
+
+Disable without redeploying code:
+
+```bash
+kubectl -n activity-core set env deploy/actcore-worker OPS_RUN_QUEUE_ENABLED=false
+kubectl -n activity-core set env deploy/actcore-api OPS_RUN_QUEUE_ENABLED=false
+# Prefer ConfigMap edit + rollout so Job/migrate stay consistent
+```
+
+Dropping `ops_runs` is **not** required for rollback; leave the table.
+
+---
+
+## Acceptance (T07 done when)
+
+- [ ] Migration 0007 applied on railiance  
+- [ ] `OPS_RUN_QUEUE_ENABLED=true` on runtime ConfigMap  
+- [ ] API serves `/ops-runs/*`  
+- [ ] At least one real emit produced `state=open` (or documented due=false skip + schedule observation)  
+- [ ] Manual claim smoke succeeded  
+- [ ] Host timers still in place; REIN-A-0002 noted as next  
+
+Mark **ACTIVITY-WP-0026-T07** `done` and WP status `done` only after the boxes above.
+
+---
+
+## Related
+
+- `k8s/railiance/README.md` — full stack deploy  
+- `docs/runbook.md` — ops_run curl cheatsheet  
+- `docs/task-emission-consumer-contract.md` — harness contract  
+- `docs/recurring-automations-playbook.md` — organizer layers  
+- REIN-A-0002 — claim loop + approach table + timer cutover  
--- a/docs/ops-run-queue.md
+++ b/docs/ops-run-queue.md
@ -1,7 +1,8 @@
 # Ops run claim queue

 **Status:** implemented (ACTIVITY-WP-0026 code path; railiance rollout = T07)  
-**Architecture:** [ACT-ADR-005](adr/adr-005-ops-runs-vs-dev-work-records.md)
+**Architecture:** [ACT-ADR-005](adr/adr-005-ops-runs-vs-dev-work-records.md)  
+**Deploy checklist:** [deploy-ops-run-queue-railiance.md](deploy-ops-run-queue-railiance.md)

 Durable, claimable work instances for **scheduled / automation** runs. Not a
 workplan task file. Not an issue-core or Forgejo ticket.
--- a/docs/recurring-automations-playbook.md
+++ b/docs/recurring-automations-playbook.md
@ -168,6 +168,7 @@ Do not flip production to `rest` globally.
 - `INTENT.md` — when/what/where boundary  
 - `docs/adr/adr-005-ops-runs-vs-dev-work-records.md` — ops vs dev work  
 - `docs/ops-run-queue.md` — claim API  
+- `docs/deploy-ops-run-queue-railiance.md` — T07 railiance rollout checklist  
 - `docs/task-emission-consumer-contract.md` — spawn payload + consumer duties  
 - `docs/runbook.md` — sync, trigger, ops UI  
 - `docs/adr/adr-002-definition-format.md` — definition files  
--- a/docs/runbook.md
+++ b/docs/runbook.md
@ -428,10 +428,10 @@ curl -sS -X POST "http://localhost:8010/ops-runs/expire-leases" \
 | `OPS_RUN_SLA_HOURS` | `1` | Stuck threshold in status |
 | `ACTIVITY_CORE_WORKER_TOKEN` | unset | Harness claim auth |

-**Railiance rollout (T07):** apply migration `0007`, set
-`OPS_RUN_QUEUE_ENABLED=true` on worker/api, smoke-trigger FI definition, confirm
-open row via `GET /ops-runs?state=open`. Keep host timers until REIN-A-0002 claim
-loop is proven.
+**Railiance rollout (T07):** full checklist with image import, migrate job,
+smoke trigger, claim test, and dual-path residual:
+
+→ **`docs/deploy-ops-run-queue-railiance.md`**

 Example distinction from the June 2026 daily triage evidence:

--- a/k8s/railiance/20-runtime.yaml
+++ b/k8s/railiance/20-runtime.yaml
@ -18,6 +18,12 @@ data:
  # ACTIVITY-WP-0021: state-hub until issue-core Forgejo PAT (GITEA_BACKEND_TOKEN)
  # is rotated; switch back to "rest" after path A smoke returns 201.
  ISSUE_SINK_TYPE: "state-hub"
+  # ACTIVITY-WP-0026 / ACT-ADR-005 — claimable ops_run on emit (not Forgejo)
+  OPS_RUN_QUEUE_ENABLED: "true"
+  OPS_RUN_LEASE_SECONDS: "900"
+  OPS_RUN_MAX_ATTEMPTS: "3"
+  OPS_RUN_SLA_HOURS: "1"
+  # ACTIVITY_CORE_WORKER_TOKEN lives in actcore-runtime-secret (optional until REIN-A-0002)
  ACTIVITY_DEFINITION_DIRS: /etc/activity-core/external-definitions
  CUSTODIAN_REPO_ROOT: /var/custodian
  ACTIVITY_CORE_ROOT: /etc/activity-core
--- a/k8s/railiance/README.md
+++ b/k8s/railiance/README.md
@ -31,7 +31,14 @@ provisions the Inter-Hub ops-hub key.
 `ISSUE_SINK_TYPE` defaults to **`state-hub`** (ACTIVITY-WP-0022; no silent Forgejo
 issues). Set `rest` only for intentional issue-core projection when the backend
 is healthy. `ISSUE_CORE_API_KEY` and `FORGEJO_TOKEN` are synced from OpenBao into
-`actcore-runtime-secret` by ExternalSecrets:
+`actcore-runtime-secret` by ExternalSecrets.
+
+**Ops run claim queue (ACTIVITY-WP-0026):** ConfigMap sets
+`OPS_RUN_QUEUE_ENABLED=true`. After image + migrate job (alembic **0007**),
+workers insert claimable `ops_runs` on emit. Full railiance checklist:
+`docs/deploy-ops-run-queue-railiance.md`. Keep host timers until REIN-A-0002.
+Optional `ACTIVITY_CORE_WORKER_TOKEN` in `actcore-runtime-secret` for harness
+claim auth.

 | ExternalSecret | OpenBao path | Secret key |
 | --- | --- | --- |
--- a/k8s/railiance/bootstrap-secrets.sh
+++ b/k8s/railiance/bootstrap-secrets.sh
@ -42,3 +42,7 @@ fi
 # ISSUE_CORE_API_KEY is merged into actcore-runtime-secret by ExternalSecret
 # actcore-issue-core-runtime (k8s/railiance/15-externalsecret-issue-core.yaml).
 # Apply that manifest after ClusterSecretStore openbao-activity-core is Ready.
+#
+# Optional (ACTIVITY-WP-0026 / REIN-A-0002): patch ACTIVITY_CORE_WORKER_TOKEN and
+# ACTIVITY_CORE_OPERATOR_TOKEN into actcore-runtime-secret — see
+# docs/deploy-ops-run-queue-railiance.md. Not auto-generated here.
--- a/workplans/ACTIVITY-WP-0026-ops-run-claim-queue.md
+++ b/workplans/ACTIVITY-WP-0026-ops-run-claim-queue.md
@ -196,12 +196,19 @@ priority: medium
 state_hub_task_id: "b1ce221b-edb2-42ed-91a1-e9f144a58d1e"
 ```

- Feature flag `OPS_RUN_QUEUE_ENABLED=true` on railiance.  
- Deploy migration + API.  
- Smoke: trigger FI definition → open ops_run visible.  
- Coordinate REIN-A-0002 cutover; keep host timers until harness claim proven.
+**Deploy notes + executable checklist (docs only until smoke green):**

-**Done when:** prod smoke checklist green; flag documented.
+→ [`docs/deploy-ops-run-queue-railiance.md`](../docs/deploy-ops-run-queue-railiance.md)
+
+Manifest prep (landed with this notes commit):
+
+- [x] `OPS_RUN_*` keys in `k8s/railiance/20-runtime.yaml` ConfigMap  
+- [x] Checklist: image → migrate 0007 → rollout → FI trigger smoke → manual claim  
+- [x] Dual-path residual documented (keep host timers; REIN-A-0002 next)  
+- [ ] **Operator executes checklist on railiance** (still open)  
+- [ ] Prod smoke boxes ticked in deploy doc → mark this task `done`
+
+**Done when:** prod smoke checklist green on railiance; flag live.

 ## Acceptance

@ -211,7 +218,8 @@ state_hub_task_id: "b1ce221b-edb2-42ed-91a1-e9f144a58d1e"
 - [x] Docs use Forgejo-only language for self-hosted forge  
 - [x] REIN-A-0002 unblocked (contract in docs/ops-run-queue.md + consumer contract)  

-**T07 remaining:** railiance migrate + flag + smoke (deploy, not code).
+**T07 remaining:** execute `docs/deploy-ops-run-queue-railiance.md` on railiance
+(image + migrate + smoke). Notes/checklist + ConfigMap flags are in-repo.

 ## Out of scope