diff --git a/docs/deploy-ops-run-queue-railiance.md b/docs/deploy-ops-run-queue-railiance.md new file mode 100644 index 0000000..d2ea1b3 --- /dev/null +++ b/docs/deploy-ops-run-queue-railiance.md @@ -0,0 +1,300 @@ +# Deploy notes — ops_run claim queue on railiance (ACTIVITY-WP-0026-T07) + +**Audience:** operator deploying activity-core after WP-0026 code lands +**Spec:** [ops-run-queue.md](ops-run-queue.md) · **ADR:** [ACT-ADR-005](adr/adr-005-ops-runs-vs-dev-work-records.md) +**Consumer next:** rein-aharness [REIN-A-0002](https://forgejo.coulomb.social/coulomb/rein-aharness) +**Image tag:** `activity-core:railiance01-prod` +**Namespace:** `activity-core` + +This is the **prod rollout checklist**, not the implementation. Code path +defaults `OPS_RUN_QUEUE_ENABLED=true`; railiance still needs image + migration ++ ConfigMap apply before open rows appear. + +--- + +## Goals + +1. Ship migration **`0007_create_ops_runs`** to railiance Postgres. +2. Expose claim API on **actcore-api** (image that includes WP-0026). +3. Worker **emit_tasks** dual-writes open `ops_run` rows. +4. Smoke: FI (or Binky) one-shot → `GET /ops-runs?state=open` shows a row. +5. **Do not** remove host systemd timers until REIN-A-0002 claim loop is proven. + +## Non-goals + +- Implementing rein-aharness claim (REIN-A-0002). +- Flipping `ISSUE_SINK_TYPE` to `rest` / Forgejo. +- Workstation cron. + +--- + +## Pre-flight + +| Check | Command / note | +| ----- | -------------- | +| Code on `main` | Includes `migrations/versions/0007_create_ops_runs.py`, `ops_run_queue.py`, `ops_runs_api.py` (commit ≥ `15eb3a2`) | +| Cluster access | `export KUBECONFIG=…` or `ssh railiance01` with k3s | +| API healthy today | `kubectl -n activity-core exec deploy/actcore-api -- … /health` | +| FI definition present | Name contains `fi-daily` / Freedom Intelligence; id from inventory | +| Host timers still run | Keep FI/Binky user-systemd timers until claim proven | + +Lookup definition id (use for trigger smoke): + +```bash +kubectl -n activity-core exec deploy/actcore-api -- \ + /app/.venv/bin/python3 -c ' +import urllib.request, json +d=json.loads(urllib.request.urlopen("http://127.0.0.1:8010/ops/automations").read()) +for a in d.get("items") or d.get("automations") or []: + name=(a.get("name") or "") + if "reedom" in name or "fi-daily" in name.lower() or "FI " in name: + print(a.get("id"), name, a.get("enabled")) +' +``` + +If the inventory shape differs, use: + +```bash +curl -sS "http://127.0.0.1:8010/ops/automations" | python3 -m json.tool | less +# via port-forward or SSO host activity.coulomb.social +``` + +--- + +## Checklist (execute in order) + +### 1. Build and load image + +From activity-core checkout (workstation or build host): + +```bash +cd ~/activity-core +git pull --ff-only +docker build -t activity-core:railiance01-prod . +docker save -o /tmp/activity-core-railiance01-prod.tar activity-core:railiance01-prod +scp /tmp/activity-core-railiance01-prod.tar railiance01:/tmp/ +ssh railiance01 sudo k3s ctr images import /tmp/activity-core-railiance01-prod.tar +``` + +- [ ] Image import succeeded (`k3s ctr images ls | grep activity-core`) + +### 2. Sync manifests + ConfigMap flags + +`actcore-runtime-config` must include (defaults match code if omitted, but +explicit is better for ops): + +| Key | Value | Notes | +| --- | ----- | ----- | +| `OPS_RUN_QUEUE_ENABLED` | `true` | Create ops_run on emit | +| `OPS_RUN_LEASE_SECONDS` | `900` | Optional | +| `OPS_RUN_MAX_ATTEMPTS` | `3` | Optional | +| `OPS_RUN_SLA_HOURS` | `1` | Status stuck threshold | + +Optional worker auth (recommended before external claim): + +```bash +# Generate once; store in secret — do not commit +WORKER_TOKEN="$(openssl rand -hex 24)" +kubectl -n activity-core patch secret actcore-runtime-secret --type merge \ + -p "{\"stringData\":{\"ACTIVITY_CORE_WORKER_TOKEN\":\"${WORKER_TOKEN}\"}}" +# Record token in operator secret store (OpenBao / password manager), not chat. +``` + +Apply: + +```bash +rsync -a k8s/railiance/ railiance01:activity-core/k8s/railiance/ +ssh railiance01 'cd ~/activity-core && kubectl apply -f k8s/railiance/20-runtime.yaml' +``` + +- [ ] ConfigMap shows `OPS_RUN_QUEUE_ENABLED=true` + `kubectl -n activity-core get cm actcore-runtime-config -o yaml | grep OPS_RUN` + +### 3. Migration job (0007) + +Jobs are immutable; delete and re-apply so the new image runs: + +```bash +kubectl -n activity-core delete job actcore-migrate --ignore-not-found +kubectl apply -f k8s/railiance/20-runtime.yaml # or only the Job if split later +kubectl -n activity-core wait --for=condition=complete job/actcore-migrate --timeout=180s +kubectl -n activity-core logs job/actcore-migrate +``` + +Expect alembic to reach revision **`0007`** (creates table `ops_runs`). + +Verify table: + +```bash +kubectl -n activity-core exec deploy/actcore-api -- \ + /app/.venv/bin/python3 -c ' +import asyncio, os +from sqlalchemy.ext.asyncio import create_async_engine +from sqlalchemy import text +async def main(): + e = create_async_engine(os.environ["ACTCORE_DB_URL"]) + async with e.connect() as c: + r = await c.execute(text( + "select to_regclass('\''public.ops_runs'\'') is not null" + )) + print("ops_runs_exists", r.scalar()) + await e.dispose() +asyncio.run(main()) +' +``` + +- [ ] Job complete +- [ ] `ops_runs_exists True` + +### 4. Roll API + worker (+ event-router) + +```bash +kubectl -n activity-core rollout restart deploy/actcore-api deploy/actcore-worker deploy/actcore-event-router +kubectl -n activity-core rollout status deploy/actcore-api --timeout=180s +kubectl -n activity-core rollout status deploy/actcore-worker --timeout=180s +kubectl -n activity-core rollout status deploy/actcore-event-router --timeout=180s +``` + +Confirm OpenAPI / routes: + +```bash +kubectl -n activity-core exec deploy/actcore-api -- \ + /app/.venv/bin/python3 -c ' +import urllib.request, json +# paths that must exist after WP-0026 +for path in ["/ops-runs", "/ops/automations/status"]: + try: + urllib.request.urlopen(f"http://127.0.0.1:8010{path}") + print(path, "ok") + except Exception as e: + # 401 still proves route exists + print(path, type(e).__name__, getattr(e, "code", e)) +' +``` + +- [ ] Rollouts healthy +- [ ] `/ops-runs` reachable (200 or 401, not 404) + +### 5. Smoke — emit open ops_run + +Prefer **operator trigger** (does not wait for next cron): + +```bash +# Port-forward if needed +kubectl -n activity-core port-forward svc/actcore-api 8010:8010 & +# Operator token from secret (break-glass) or SSO via activity.coulomb.social +OP_TOKEN=$(kubectl -n activity-core get secret actcore-runtime-secret \ + -o jsonpath='{.data.ACTIVITY_CORE_OPERATOR_TOKEN}' | base64 -d) + +DEF_ID="" # from pre-flight lookup +curl -sS -X POST "http://127.0.0.1:8010/ops/automations/${DEF_ID}/trigger" \ + -H "Content-Type: application/json" \ + -H "X-Operator-Token: ${OP_TOKEN}" \ + -d '{}' | python3 -m json.tool +``` + +If FI context has `due=false` (brief already posted today), expect **no** new +task/ops_run. Options: + +- Trigger a definition known to emit, or +- Temporarily clear today's completion only in a **test** domain, or +- Wait for next weekday 07:30 Europe/Berlin fire when due. + +List open runs: + +```bash +curl -sS "http://127.0.0.1:8010/ops-runs?state=open" \ + -H "X-Operator-Token: ${OP_TOKEN}" | python3 -m json.tool +``` + +Status block: + +```bash +curl -sS "http://127.0.0.1:8010/ops/automations/status?since=today" \ + | python3 -c "import sys,json; print(json.load(sys.stdin).get('ops_runs'))" +``` + +- [ ] After a successful emit: at least one `state=open` row with labels including + `research-brief` / `automated` (or Binky equivalents) +- [ ] `counts.open >= 1` (or expected) +- [ ] Dual-write still visible: State Hub `activity_task_spawn` progress (unchanged path) + +### 6. Claim path smoke (manual, no harness yet) + +```bash +WORKER_TOKEN=… # from secret if set; else local-dev open auth +curl -sS -X POST "http://127.0.0.1:8010/ops-runs/claim" \ + -H "Content-Type: application/json" \ + -H "X-Worker-Token: ${WORKER_TOKEN}" \ + -d '{"worker_id":"smoke@railiance01","labels":["automated"],"limit":1,"lease_seconds":120}' \ + | python3 -m json.tool +``` + +Then **complete or fail with reopen** so the run does not sit claimed forever: + +```bash +RUN_ID=… # from claim response +# Prefer fail+reopen if you did not execute real work: +curl -sS -X POST "http://127.0.0.1:8010/ops-runs/${RUN_ID}/fail" \ + -H "Content-Type: application/json" \ + -H "X-Worker-Token: ${WORKER_TOKEN}" \ + -d '{"worker_id":"smoke@railiance01","error":"T07 smoke only","reopen":true}' +``` + +- [ ] Claim returns the open run +- [ ] Fail+reopen returns `state=open` (or complete if intentional) + +### 7. Dual-path residual (keep until REIN-A-0002) + +| Path | Status after T07 | +| ---- | ---------------- | +| Temporal schedule → emit → **ops_run** | **On** (this checklist) | +| State Hub `activity_task_spawn` dual-write | **On** (visibility) | +| railiance host timers → rein-aharness | **Keep** until claim loop proven | +| issue-core / Forgejo as claim queue | **Off** (never for FI/Binky ops) | + +- [ ] Confirmed timers not deleted +- [ ] REIN-A-0002 unblocked (API contract live) + +--- + +## Rollback + +| Situation | Action | +| --------- | ------ | +| Bad API/worker image | Roll back image tag / previous tar; restart deploys | +| Migration issues | Table is additive; app old image ignores `ops_runs`. To disable only: set `OPS_RUN_QUEUE_ENABLED=false` on ConfigMap + restart worker | +| Claim spam / bad consumers | Set worker token; revoke old token; fail open runs manually | + +Disable without redeploying code: + +```bash +kubectl -n activity-core set env deploy/actcore-worker OPS_RUN_QUEUE_ENABLED=false +kubectl -n activity-core set env deploy/actcore-api OPS_RUN_QUEUE_ENABLED=false +# Prefer ConfigMap edit + rollout so Job/migrate stay consistent +``` + +Dropping `ops_runs` is **not** required for rollback; leave the table. + +--- + +## Acceptance (T07 done when) + +- [ ] Migration 0007 applied on railiance +- [ ] `OPS_RUN_QUEUE_ENABLED=true` on runtime ConfigMap +- [ ] API serves `/ops-runs/*` +- [ ] At least one real emit produced `state=open` (or documented due=false skip + schedule observation) +- [ ] Manual claim smoke succeeded +- [ ] Host timers still in place; REIN-A-0002 noted as next + +Mark **ACTIVITY-WP-0026-T07** `done` and WP status `done` only after the boxes above. + +--- + +## Related + +- `k8s/railiance/README.md` — full stack deploy +- `docs/runbook.md` — ops_run curl cheatsheet +- `docs/task-emission-consumer-contract.md` — harness contract +- `docs/recurring-automations-playbook.md` — organizer layers +- REIN-A-0002 — claim loop + approach table + timer cutover diff --git a/docs/ops-run-queue.md b/docs/ops-run-queue.md index aaf77d2..dccf918 100644 --- a/docs/ops-run-queue.md +++ b/docs/ops-run-queue.md @@ -1,7 +1,8 @@ # Ops run claim queue **Status:** implemented (ACTIVITY-WP-0026 code path; railiance rollout = T07) -**Architecture:** [ACT-ADR-005](adr/adr-005-ops-runs-vs-dev-work-records.md) +**Architecture:** [ACT-ADR-005](adr/adr-005-ops-runs-vs-dev-work-records.md) +**Deploy checklist:** [deploy-ops-run-queue-railiance.md](deploy-ops-run-queue-railiance.md) Durable, claimable work instances for **scheduled / automation** runs. Not a workplan task file. Not an issue-core or Forgejo ticket. diff --git a/docs/recurring-automations-playbook.md b/docs/recurring-automations-playbook.md index 5b25125..bcebcce 100644 --- a/docs/recurring-automations-playbook.md +++ b/docs/recurring-automations-playbook.md @@ -168,6 +168,7 @@ Do not flip production to `rest` globally. - `INTENT.md` — when/what/where boundary - `docs/adr/adr-005-ops-runs-vs-dev-work-records.md` — ops vs dev work - `docs/ops-run-queue.md` — claim API +- `docs/deploy-ops-run-queue-railiance.md` — T07 railiance rollout checklist - `docs/task-emission-consumer-contract.md` — spawn payload + consumer duties - `docs/runbook.md` — sync, trigger, ops UI - `docs/adr/adr-002-definition-format.md` — definition files diff --git a/docs/runbook.md b/docs/runbook.md index 77722e8..ff0f187 100644 --- a/docs/runbook.md +++ b/docs/runbook.md @@ -428,10 +428,10 @@ curl -sS -X POST "http://localhost:8010/ops-runs/expire-leases" \ | `OPS_RUN_SLA_HOURS` | `1` | Stuck threshold in status | | `ACTIVITY_CORE_WORKER_TOKEN` | unset | Harness claim auth | -**Railiance rollout (T07):** apply migration `0007`, set -`OPS_RUN_QUEUE_ENABLED=true` on worker/api, smoke-trigger FI definition, confirm -open row via `GET /ops-runs?state=open`. Keep host timers until REIN-A-0002 claim -loop is proven. +**Railiance rollout (T07):** full checklist with image import, migrate job, +smoke trigger, claim test, and dual-path residual: + +→ **`docs/deploy-ops-run-queue-railiance.md`** Example distinction from the June 2026 daily triage evidence: diff --git a/k8s/railiance/20-runtime.yaml b/k8s/railiance/20-runtime.yaml index 797c4e1..9fdcf68 100644 --- a/k8s/railiance/20-runtime.yaml +++ b/k8s/railiance/20-runtime.yaml @@ -18,6 +18,12 @@ data: # ACTIVITY-WP-0021: state-hub until issue-core Forgejo PAT (GITEA_BACKEND_TOKEN) # is rotated; switch back to "rest" after path A smoke returns 201. ISSUE_SINK_TYPE: "state-hub" + # ACTIVITY-WP-0026 / ACT-ADR-005 — claimable ops_run on emit (not Forgejo) + OPS_RUN_QUEUE_ENABLED: "true" + OPS_RUN_LEASE_SECONDS: "900" + OPS_RUN_MAX_ATTEMPTS: "3" + OPS_RUN_SLA_HOURS: "1" + # ACTIVITY_CORE_WORKER_TOKEN lives in actcore-runtime-secret (optional until REIN-A-0002) ACTIVITY_DEFINITION_DIRS: /etc/activity-core/external-definitions CUSTODIAN_REPO_ROOT: /var/custodian ACTIVITY_CORE_ROOT: /etc/activity-core diff --git a/k8s/railiance/README.md b/k8s/railiance/README.md index 67c934c..ccc1d43 100644 --- a/k8s/railiance/README.md +++ b/k8s/railiance/README.md @@ -31,7 +31,14 @@ provisions the Inter-Hub ops-hub key. `ISSUE_SINK_TYPE` defaults to **`state-hub`** (ACTIVITY-WP-0022; no silent Forgejo issues). Set `rest` only for intentional issue-core projection when the backend is healthy. `ISSUE_CORE_API_KEY` and `FORGEJO_TOKEN` are synced from OpenBao into -`actcore-runtime-secret` by ExternalSecrets: +`actcore-runtime-secret` by ExternalSecrets. + +**Ops run claim queue (ACTIVITY-WP-0026):** ConfigMap sets +`OPS_RUN_QUEUE_ENABLED=true`. After image + migrate job (alembic **0007**), +workers insert claimable `ops_runs` on emit. Full railiance checklist: +`docs/deploy-ops-run-queue-railiance.md`. Keep host timers until REIN-A-0002. +Optional `ACTIVITY_CORE_WORKER_TOKEN` in `actcore-runtime-secret` for harness +claim auth. | ExternalSecret | OpenBao path | Secret key | | --- | --- | --- | diff --git a/k8s/railiance/bootstrap-secrets.sh b/k8s/railiance/bootstrap-secrets.sh index dea48b5..905a8da 100644 --- a/k8s/railiance/bootstrap-secrets.sh +++ b/k8s/railiance/bootstrap-secrets.sh @@ -42,3 +42,7 @@ fi # ISSUE_CORE_API_KEY is merged into actcore-runtime-secret by ExternalSecret # actcore-issue-core-runtime (k8s/railiance/15-externalsecret-issue-core.yaml). # Apply that manifest after ClusterSecretStore openbao-activity-core is Ready. +# +# Optional (ACTIVITY-WP-0026 / REIN-A-0002): patch ACTIVITY_CORE_WORKER_TOKEN and +# ACTIVITY_CORE_OPERATOR_TOKEN into actcore-runtime-secret — see +# docs/deploy-ops-run-queue-railiance.md. Not auto-generated here. diff --git a/workplans/ACTIVITY-WP-0026-ops-run-claim-queue.md b/workplans/ACTIVITY-WP-0026-ops-run-claim-queue.md index 920f20c..1085544 100644 --- a/workplans/ACTIVITY-WP-0026-ops-run-claim-queue.md +++ b/workplans/ACTIVITY-WP-0026-ops-run-claim-queue.md @@ -196,12 +196,19 @@ priority: medium state_hub_task_id: "b1ce221b-edb2-42ed-91a1-e9f144a58d1e" ``` -- Feature flag `OPS_RUN_QUEUE_ENABLED=true` on railiance. -- Deploy migration + API. -- Smoke: trigger FI definition → open ops_run visible. -- Coordinate REIN-A-0002 cutover; keep host timers until harness claim proven. +**Deploy notes + executable checklist (docs only until smoke green):** -**Done when:** prod smoke checklist green; flag documented. +→ [`docs/deploy-ops-run-queue-railiance.md`](../docs/deploy-ops-run-queue-railiance.md) + +Manifest prep (landed with this notes commit): + +- [x] `OPS_RUN_*` keys in `k8s/railiance/20-runtime.yaml` ConfigMap +- [x] Checklist: image → migrate 0007 → rollout → FI trigger smoke → manual claim +- [x] Dual-path residual documented (keep host timers; REIN-A-0002 next) +- [ ] **Operator executes checklist on railiance** (still open) +- [ ] Prod smoke boxes ticked in deploy doc → mark this task `done` + +**Done when:** prod smoke checklist green on railiance; flag live. ## Acceptance @@ -211,7 +218,8 @@ state_hub_task_id: "b1ce221b-edb2-42ed-91a1-e9f144a58d1e" - [x] Docs use Forgejo-only language for self-hosted forge - [x] REIN-A-0002 unblocked (contract in docs/ops-run-queue.md + consumer contract) -**T07 remaining:** railiance migrate + flag + smoke (deploy, not code). +**T07 remaining:** execute `docs/deploy-ops-run-queue-railiance.md` on railiance +(image + migrate + smoke). Notes/checklist + ConfigMap flags are in-repo. ## Out of scope