Document ACTIVITY-WP-0026-T07 railiance deploy checklist.
Add deploy-ops-run-queue-railiance.md (image, migrate 0007, smoke, dual-path residual), wire OPS_RUN_* into runtime ConfigMap, and cross-link runbook/README. T07 stays open until prod smoke is executed.
This commit is contained in:
parent
32f62ac1ac
commit
6ce77a72bb
8 changed files with 339 additions and 12 deletions
300
docs/deploy-ops-run-queue-railiance.md
Normal file
300
docs/deploy-ops-run-queue-railiance.md
Normal file
|
|
@ -0,0 +1,300 @@
|
|||
# Deploy notes — ops_run claim queue on railiance (ACTIVITY-WP-0026-T07)
|
||||
|
||||
**Audience:** operator deploying activity-core after WP-0026 code lands
|
||||
**Spec:** [ops-run-queue.md](ops-run-queue.md) · **ADR:** [ACT-ADR-005](adr/adr-005-ops-runs-vs-dev-work-records.md)
|
||||
**Consumer next:** rein-aharness [REIN-A-0002](https://forgejo.coulomb.social/coulomb/rein-aharness)
|
||||
**Image tag:** `activity-core:railiance01-prod`
|
||||
**Namespace:** `activity-core`
|
||||
|
||||
This is the **prod rollout checklist**, not the implementation. Code path
|
||||
defaults `OPS_RUN_QUEUE_ENABLED=true`; railiance still needs image + migration
|
||||
+ ConfigMap apply before open rows appear.
|
||||
|
||||
---
|
||||
|
||||
## Goals
|
||||
|
||||
1. Ship migration **`0007_create_ops_runs`** to railiance Postgres.
|
||||
2. Expose claim API on **actcore-api** (image that includes WP-0026).
|
||||
3. Worker **emit_tasks** dual-writes open `ops_run` rows.
|
||||
4. Smoke: FI (or Binky) one-shot → `GET /ops-runs?state=open` shows a row.
|
||||
5. **Do not** remove host systemd timers until REIN-A-0002 claim loop is proven.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Implementing rein-aharness claim (REIN-A-0002).
|
||||
- Flipping `ISSUE_SINK_TYPE` to `rest` / Forgejo.
|
||||
- Workstation cron.
|
||||
|
||||
---
|
||||
|
||||
## Pre-flight
|
||||
|
||||
| Check | Command / note |
|
||||
| ----- | -------------- |
|
||||
| Code on `main` | Includes `migrations/versions/0007_create_ops_runs.py`, `ops_run_queue.py`, `ops_runs_api.py` (commit ≥ `15eb3a2`) |
|
||||
| Cluster access | `export KUBECONFIG=…` or `ssh railiance01` with k3s |
|
||||
| API healthy today | `kubectl -n activity-core exec deploy/actcore-api -- … /health` |
|
||||
| FI definition present | Name contains `fi-daily` / Freedom Intelligence; id from inventory |
|
||||
| Host timers still run | Keep FI/Binky user-systemd timers until claim proven |
|
||||
|
||||
Lookup definition id (use for trigger smoke):
|
||||
|
||||
```bash
|
||||
kubectl -n activity-core exec deploy/actcore-api -- \
|
||||
/app/.venv/bin/python3 -c '
|
||||
import urllib.request, json
|
||||
d=json.loads(urllib.request.urlopen("http://127.0.0.1:8010/ops/automations").read())
|
||||
for a in d.get("items") or d.get("automations") or []:
|
||||
name=(a.get("name") or "")
|
||||
if "reedom" in name or "fi-daily" in name.lower() or "FI " in name:
|
||||
print(a.get("id"), name, a.get("enabled"))
|
||||
'
|
||||
```
|
||||
|
||||
If the inventory shape differs, use:
|
||||
|
||||
```bash
|
||||
curl -sS "http://127.0.0.1:8010/ops/automations" | python3 -m json.tool | less
|
||||
# via port-forward or SSO host activity.coulomb.social
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Checklist (execute in order)
|
||||
|
||||
### 1. Build and load image
|
||||
|
||||
From activity-core checkout (workstation or build host):
|
||||
|
||||
```bash
|
||||
cd ~/activity-core
|
||||
git pull --ff-only
|
||||
docker build -t activity-core:railiance01-prod .
|
||||
docker save -o /tmp/activity-core-railiance01-prod.tar activity-core:railiance01-prod
|
||||
scp /tmp/activity-core-railiance01-prod.tar railiance01:/tmp/
|
||||
ssh railiance01 sudo k3s ctr images import /tmp/activity-core-railiance01-prod.tar
|
||||
```
|
||||
|
||||
- [ ] Image import succeeded (`k3s ctr images ls | grep activity-core`)
|
||||
|
||||
### 2. Sync manifests + ConfigMap flags
|
||||
|
||||
`actcore-runtime-config` must include (defaults match code if omitted, but
|
||||
explicit is better for ops):
|
||||
|
||||
| Key | Value | Notes |
|
||||
| --- | ----- | ----- |
|
||||
| `OPS_RUN_QUEUE_ENABLED` | `true` | Create ops_run on emit |
|
||||
| `OPS_RUN_LEASE_SECONDS` | `900` | Optional |
|
||||
| `OPS_RUN_MAX_ATTEMPTS` | `3` | Optional |
|
||||
| `OPS_RUN_SLA_HOURS` | `1` | Status stuck threshold |
|
||||
|
||||
Optional worker auth (recommended before external claim):
|
||||
|
||||
```bash
|
||||
# Generate once; store in secret — do not commit
|
||||
WORKER_TOKEN="$(openssl rand -hex 24)"
|
||||
kubectl -n activity-core patch secret actcore-runtime-secret --type merge \
|
||||
-p "{\"stringData\":{\"ACTIVITY_CORE_WORKER_TOKEN\":\"${WORKER_TOKEN}\"}}"
|
||||
# Record token in operator secret store (OpenBao / password manager), not chat.
|
||||
```
|
||||
|
||||
Apply:
|
||||
|
||||
```bash
|
||||
rsync -a k8s/railiance/ railiance01:activity-core/k8s/railiance/
|
||||
ssh railiance01 'cd ~/activity-core && kubectl apply -f k8s/railiance/20-runtime.yaml'
|
||||
```
|
||||
|
||||
- [ ] ConfigMap shows `OPS_RUN_QUEUE_ENABLED=true`
|
||||
`kubectl -n activity-core get cm actcore-runtime-config -o yaml | grep OPS_RUN`
|
||||
|
||||
### 3. Migration job (0007)
|
||||
|
||||
Jobs are immutable; delete and re-apply so the new image runs:
|
||||
|
||||
```bash
|
||||
kubectl -n activity-core delete job actcore-migrate --ignore-not-found
|
||||
kubectl apply -f k8s/railiance/20-runtime.yaml # or only the Job if split later
|
||||
kubectl -n activity-core wait --for=condition=complete job/actcore-migrate --timeout=180s
|
||||
kubectl -n activity-core logs job/actcore-migrate
|
||||
```
|
||||
|
||||
Expect alembic to reach revision **`0007`** (creates table `ops_runs`).
|
||||
|
||||
Verify table:
|
||||
|
||||
```bash
|
||||
kubectl -n activity-core exec deploy/actcore-api -- \
|
||||
/app/.venv/bin/python3 -c '
|
||||
import asyncio, os
|
||||
from sqlalchemy.ext.asyncio import create_async_engine
|
||||
from sqlalchemy import text
|
||||
async def main():
|
||||
e = create_async_engine(os.environ["ACTCORE_DB_URL"])
|
||||
async with e.connect() as c:
|
||||
r = await c.execute(text(
|
||||
"select to_regclass('\''public.ops_runs'\'') is not null"
|
||||
))
|
||||
print("ops_runs_exists", r.scalar())
|
||||
await e.dispose()
|
||||
asyncio.run(main())
|
||||
'
|
||||
```
|
||||
|
||||
- [ ] Job complete
|
||||
- [ ] `ops_runs_exists True`
|
||||
|
||||
### 4. Roll API + worker (+ event-router)
|
||||
|
||||
```bash
|
||||
kubectl -n activity-core rollout restart deploy/actcore-api deploy/actcore-worker deploy/actcore-event-router
|
||||
kubectl -n activity-core rollout status deploy/actcore-api --timeout=180s
|
||||
kubectl -n activity-core rollout status deploy/actcore-worker --timeout=180s
|
||||
kubectl -n activity-core rollout status deploy/actcore-event-router --timeout=180s
|
||||
```
|
||||
|
||||
Confirm OpenAPI / routes:
|
||||
|
||||
```bash
|
||||
kubectl -n activity-core exec deploy/actcore-api -- \
|
||||
/app/.venv/bin/python3 -c '
|
||||
import urllib.request, json
|
||||
# paths that must exist after WP-0026
|
||||
for path in ["/ops-runs", "/ops/automations/status"]:
|
||||
try:
|
||||
urllib.request.urlopen(f"http://127.0.0.1:8010{path}")
|
||||
print(path, "ok")
|
||||
except Exception as e:
|
||||
# 401 still proves route exists
|
||||
print(path, type(e).__name__, getattr(e, "code", e))
|
||||
'
|
||||
```
|
||||
|
||||
- [ ] Rollouts healthy
|
||||
- [ ] `/ops-runs` reachable (200 or 401, not 404)
|
||||
|
||||
### 5. Smoke — emit open ops_run
|
||||
|
||||
Prefer **operator trigger** (does not wait for next cron):
|
||||
|
||||
```bash
|
||||
# Port-forward if needed
|
||||
kubectl -n activity-core port-forward svc/actcore-api 8010:8010 &
|
||||
# Operator token from secret (break-glass) or SSO via activity.coulomb.social
|
||||
OP_TOKEN=$(kubectl -n activity-core get secret actcore-runtime-secret \
|
||||
-o jsonpath='{.data.ACTIVITY_CORE_OPERATOR_TOKEN}' | base64 -d)
|
||||
|
||||
DEF_ID="<fi-activity-definition-uuid>" # from pre-flight lookup
|
||||
curl -sS -X POST "http://127.0.0.1:8010/ops/automations/${DEF_ID}/trigger" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "X-Operator-Token: ${OP_TOKEN}" \
|
||||
-d '{}' | python3 -m json.tool
|
||||
```
|
||||
|
||||
If FI context has `due=false` (brief already posted today), expect **no** new
|
||||
task/ops_run. Options:
|
||||
|
||||
- Trigger a definition known to emit, or
|
||||
- Temporarily clear today's completion only in a **test** domain, or
|
||||
- Wait for next weekday 07:30 Europe/Berlin fire when due.
|
||||
|
||||
List open runs:
|
||||
|
||||
```bash
|
||||
curl -sS "http://127.0.0.1:8010/ops-runs?state=open" \
|
||||
-H "X-Operator-Token: ${OP_TOKEN}" | python3 -m json.tool
|
||||
```
|
||||
|
||||
Status block:
|
||||
|
||||
```bash
|
||||
curl -sS "http://127.0.0.1:8010/ops/automations/status?since=today" \
|
||||
| python3 -c "import sys,json; print(json.load(sys.stdin).get('ops_runs'))"
|
||||
```
|
||||
|
||||
- [ ] After a successful emit: at least one `state=open` row with labels including
|
||||
`research-brief` / `automated` (or Binky equivalents)
|
||||
- [ ] `counts.open >= 1` (or expected)
|
||||
- [ ] Dual-write still visible: State Hub `activity_task_spawn` progress (unchanged path)
|
||||
|
||||
### 6. Claim path smoke (manual, no harness yet)
|
||||
|
||||
```bash
|
||||
WORKER_TOKEN=… # from secret if set; else local-dev open auth
|
||||
curl -sS -X POST "http://127.0.0.1:8010/ops-runs/claim" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "X-Worker-Token: ${WORKER_TOKEN}" \
|
||||
-d '{"worker_id":"smoke@railiance01","labels":["automated"],"limit":1,"lease_seconds":120}' \
|
||||
| python3 -m json.tool
|
||||
```
|
||||
|
||||
Then **complete or fail with reopen** so the run does not sit claimed forever:
|
||||
|
||||
```bash
|
||||
RUN_ID=… # from claim response
|
||||
# Prefer fail+reopen if you did not execute real work:
|
||||
curl -sS -X POST "http://127.0.0.1:8010/ops-runs/${RUN_ID}/fail" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "X-Worker-Token: ${WORKER_TOKEN}" \
|
||||
-d '{"worker_id":"smoke@railiance01","error":"T07 smoke only","reopen":true}'
|
||||
```
|
||||
|
||||
- [ ] Claim returns the open run
|
||||
- [ ] Fail+reopen returns `state=open` (or complete if intentional)
|
||||
|
||||
### 7. Dual-path residual (keep until REIN-A-0002)
|
||||
|
||||
| Path | Status after T07 |
|
||||
| ---- | ---------------- |
|
||||
| Temporal schedule → emit → **ops_run** | **On** (this checklist) |
|
||||
| State Hub `activity_task_spawn` dual-write | **On** (visibility) |
|
||||
| railiance host timers → rein-aharness | **Keep** until claim loop proven |
|
||||
| issue-core / Forgejo as claim queue | **Off** (never for FI/Binky ops) |
|
||||
|
||||
- [ ] Confirmed timers not deleted
|
||||
- [ ] REIN-A-0002 unblocked (API contract live)
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
| Situation | Action |
|
||||
| --------- | ------ |
|
||||
| Bad API/worker image | Roll back image tag / previous tar; restart deploys |
|
||||
| Migration issues | Table is additive; app old image ignores `ops_runs`. To disable only: set `OPS_RUN_QUEUE_ENABLED=false` on ConfigMap + restart worker |
|
||||
| Claim spam / bad consumers | Set worker token; revoke old token; fail open runs manually |
|
||||
|
||||
Disable without redeploying code:
|
||||
|
||||
```bash
|
||||
kubectl -n activity-core set env deploy/actcore-worker OPS_RUN_QUEUE_ENABLED=false
|
||||
kubectl -n activity-core set env deploy/actcore-api OPS_RUN_QUEUE_ENABLED=false
|
||||
# Prefer ConfigMap edit + rollout so Job/migrate stay consistent
|
||||
```
|
||||
|
||||
Dropping `ops_runs` is **not** required for rollback; leave the table.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance (T07 done when)
|
||||
|
||||
- [ ] Migration 0007 applied on railiance
|
||||
- [ ] `OPS_RUN_QUEUE_ENABLED=true` on runtime ConfigMap
|
||||
- [ ] API serves `/ops-runs/*`
|
||||
- [ ] At least one real emit produced `state=open` (or documented due=false skip + schedule observation)
|
||||
- [ ] Manual claim smoke succeeded
|
||||
- [ ] Host timers still in place; REIN-A-0002 noted as next
|
||||
|
||||
Mark **ACTIVITY-WP-0026-T07** `done` and WP status `done` only after the boxes above.
|
||||
|
||||
---
|
||||
|
||||
## Related
|
||||
|
||||
- `k8s/railiance/README.md` — full stack deploy
|
||||
- `docs/runbook.md` — ops_run curl cheatsheet
|
||||
- `docs/task-emission-consumer-contract.md` — harness contract
|
||||
- `docs/recurring-automations-playbook.md` — organizer layers
|
||||
- REIN-A-0002 — claim loop + approach table + timer cutover
|
||||
Loading…
Add table
Add a link
Reference in a new issue