diff --git a/workplans/ACTIVITY-WP-0027-llm-connect-and-run-artifacts.md b/workplans/ACTIVITY-WP-0027-llm-connect-and-run-artifacts.md new file mode 100644 index 0000000..4d9b205 --- /dev/null +++ b/workplans/ACTIVITY-WP-0027-llm-connect-and-run-artifacts.md @@ -0,0 +1,339 @@ +--- +id: ACTIVITY-WP-0027 +type: workplan +title: "Hardened llm-connect access + run→artifact links in ops UI" +domain: infotech +repo: activity-core +status: ready +owner: grok +topic_slug: activity-core +priority: high +created: "2026-08-05" +updated: "2026-08-05" +depends_on: + - ACTIVITY-WP-0024 + - ACTIVITY-WP-0025 + - ACTIVITY-WP-0026 +related: + - ACT-ADR-005 + - REIN-A-0002 + - ACTIVITY-WP-0010 + - ACTIVITY-WP-0021 +--- + +# ACTIVITY-WP-0027 — Hardened llm-connect access + run→artifact links + +## Origin + +2026-08-05 operator session after FI automation diagnosis and the ops_run +per-fire idempotency fix (commit `21dc228`). Schedules and claim loop work on +railiance01, but two operator-visible gaps remain: + +1. **llm-connect for host executors is fragile.** rein-aharness on the host + reaches llm-connect via `LLM_CONNECT_URL=http://127.0.0.1:8080` backed by a + long-lived `kubectl port-forward` process. That is not a production access + path: reboots, PF death, or kubectl session loss break FI/Binky briefs while + in-cluster `actcore-worker` still works via ClusterIP Service DNS. +2. **Ops UI cannot show deliverables.** `activity.coulomb.social/ops/ui` lists + `activity_runs` (run_id, fired_at, tasks_spawned) but does not surface + `ops_runs.result` paths (e.g. `briefs/2026/08/2026-08-05.md`), Forgejo + links, commit SHAs, or hub completion events. Operators cannot review + results without SSH/git. + +Also carried forward from the same review (lower priority in this plan): + +- Edge-relay **503** on required POST resolvers (e.g. consistency sweep) when + State Hub upstream is degraded — high-frequency jobs hard-fail. +- Binky host dual-clock timers still enabled after FI timer was disabled. + +## Goal + +1. Make **llm-connect durable for rein-aharness on railiance01** without a + manual port-forward dependency. +2. Make **run history in the ops UI a review surface**: each fire links to + the artefacts it produced (repo path, Forgejo URL when known, ops_run + result summary, completion evidence). +3. Document and, where small, harden the remaining reliability caveats so + automated work remains reviewable and self-hosted when the workstation is + offline. + +## Non-goals + +- Building a full Forgejo browser or second wiki inside activity-core. +- Storing raw LLM prompts or full model transcripts in the ops API/UI. +- Replacing Temporal UI for deep workflow debugging. +- Moving domain brief *generation* into activity-core (still rein-aharness). +- Public unauthenticated access to artefact content (SSO / operator policy). + +## Architecture sketch + +### A. llm-connect access (T01–T02) + +```text +Today (broken path): + rein-aharness@host → 127.0.0.1:8080 → kubectl port-forward → svc/llm-connect + +Target (pick one in T01, implement in T02): + Option H1 Host NetworkPolicy-friendly Service + NodePort/LoadBalancer on loopback-bound proxy + Option H2 Dedicated HostPath/systemd unit: socat or nginx stream proxy to ClusterIP (managed) + Option H3 rein-aharness runs claim/execute *in-cluster* as a Deployment (same net as worker) + Option H4 Documented metalLB / Traefik internal route llm.activity-core.svc (cluster-only DNS + via CoreDNS host stub) — only if host resolvers stay private + +Preference order for design decision: + 1. No secrets in unit files beyond existing worker token + 2. Survives k3s node reboot without operator re-login + 3. Does not expose llm-connect on the public Internet + 4. Same URL shape for claim-loop and break-glass CLI +``` + +Claim-loop env (`~/.config/rein-aharness/claim-loop.env`) must load without +bash-breaking JSON (`AGENT_HARNESS_REPO_MAP=...`). + +### B. Run → artefact contract (T03–T05) + +```text +activity_runs (when fire happened) + │ + ├─ join ops_runs (what was claimed / result JSON) + │ result.path, result.head_after, result.committed, result.ok, … + │ + ├─ optional progress evidence (fi_daily_brief, activity_task_spawn) + │ + └─ artefact links (computed, not secrets) + - repo-relative path + - forgejo blob URL (configured base + target_repo + path + ref) + - ops_run id deep-link in ops UI +``` + +**UI (activity.coulomb.social/ops/ui):** + +| Surface | Behaviour | +| ------- | --------- | +| Automation detail — Recent runs | New column **Artifacts** with link(s) or “none” | +| Run detail page (new) | Full result JSON (sanitized), artefact links, ops_run state, Temporal deep-link if known | +| Optional | Safe markdown preview for text artefacts (size-capped fetch from Forgejo or cached result summary) — only if SSO already gates the page | + +### C. Evidence shape (no secrets) + +`ops_runs.result` already carries paths. Normalize a small public contract: + +```json +{ + "ok": true, + "approach": "fi-research-brief", + "path": "briefs/2026/08/2026-08-05.md", + "target_repo": "freedom-intelligence", + "head_after": "33a5aff…", + "committed": true, + "pushed": true, + "artifact_urls": [ + { + "kind": "forgejo_blob", + "label": "FI brief 2026-08-05", + "url": "https://forgejo.coulomb.social/coulomb/freedom-intelligence/src/commit//briefs/…" + } + ] +} +``` + +activity-core may **compute** `artifact_urls` at read time from path + +`head_after` + env `FORGEJO_WEB_BASE` if the executor did not store them. +Never put tokens in URLs. + +## Dependencies + +- railiance01 k3s + rein-aharness claim loop (REIN-A-0002 live). +- Forgejo public browse for artefact links (same as repo remotes today). +- SSO path for ops UI (ACTIVITY-WP-0025). +- Optional: rein-aharness result schema bump (coordinate REIN-A follow-up). + +## Tasks + +## Task: Design llm-connect host access (decision) + +```task +id: ACTIVITY-WP-0027-T01 +status: todo +priority: high +``` + +1. Inventory current railiance paths: claim-loop env, PF processes, in-cluster + Service `llm-connect.activity-core.svc.cluster.local:8080`, worker env. +2. Choose among H1–H4 (or hybrid) with explicit reject of “long-lived kubectl + port-forward as production.” +3. Write short decision note in this workplan or `docs/llm-connect-host-access.md`: + threat model (no public LLM proxy), reboot survival, who owns the unit. +4. Credential custody via `warden route` / OpenBao if any new secret appears — + never git. + +**Done when:** one option is locked with owner (activity-core vs rein-aharness +vs railiance host unit) and acceptance tests listed. + +## Task: Implement durable llm-connect access + claim-loop env hygiene + +```task +id: ACTIVITY-WP-0027-T02 +status: todo +priority: high +``` + +Depends on T01. + +1. Implement the chosen access path on railiance01 (systemd unit, k8s Service + type, or in-cluster worker) so `rein-aharness fi-research-brief` and the + claim loop reach llm-connect after reboot **without** manual port-forward. +2. Fix claim-loop env loading so JSON map values do not break shell `source` + (wrapper already preferred; ensure docs + install script match reality). +3. Document break-glass and health check: + `curl -sS "$LLM_CONNECT_URL/health"` from the host executor context. +4. Smoke: one FI (or Binky) claim path completion using the new URL. + +**Done when:** reboot-safe path documented + smoke evidence in this workplan; +old PF-only path marked deprecated. + +## Task: Define run↔ops_run join and artefact link contract + +```task +id: ACTIVITY-WP-0027-T03 +status: todo +priority: high +``` + +1. Spec how `activity_runs` joins to `ops_runs` (by time window + + `activity_definition_id`, `triggering_event_id` ∈ {run_id, workflow key, + manual-*}, and/or `idempotency_key` prefix). Document ambiguity rules when + multiple ops_runs match (smoke storms). +2. Spec `artifact_urls` computation from `result.path` + `head_after` + + `target_repo` + configurable Forgejo base. +3. Extend `GET /ops/automations/{id}/runs` (and/or new + `GET /ops/automations/{id}/runs/{run_id}`) to include: + - linked `ops_runs[]` (id, state, title, result summary) + - `artifacts[]` with label + url + kind + - no secrets / no raw prompts +4. Unit tests for join helpers and URL builder. + +**Done when:** JSON contract documented in `docs/ops-run-queue.md` (or +`docs/run-artifacts.md`) and covered by tests. + +## Task: Ops UI — artifacts column + run detail page + +```task +id: ACTIVITY-WP-0027-T04 +status: todo +priority: high +``` + +Depends on T03. + +1. On `/ops/ui/automations/{id}` Recent runs table: add **Artifacts** column + with one or more clickable links (open in new tab). Show “none” / “pending” + when no ops_run result yet. +2. Add `/ops/ui/automations/{id}/runs/{run_id}` (or equivalent) showing: + - fire times, tasks_spawned, version + - ops_run state machine summary + - artefact links + - sanitized result JSON + - optional Temporal UI deep-link if workflow id known +3. Keep SSO-only (existing ACTIVITY-WP-0025 posture); no new public routes for + private content. +4. Manual acceptance: open FI definition on + `https://activity.coulomb.social/ops/ui`, click through to today’s brief on + Forgejo without SSH. + +**Done when:** operator can review FI (and Binky) deliverables from the SSO UI +alone after a successful claim. + +## Task: Executor completion payload completeness (rein-aharness) + +```task +id: ACTIVITY-WP-0027-T05 +status: todo +priority: medium +``` + +Coordinate with rein-aharness (REIN-A follow-up or small patch in that repo): + +1. On complete, always set `result.path`, `result.head_after`, + `result.target_repo`, `result.committed`, `result.ok`. +2. Prefer also `result.pushed` and/or precomputed `artifact_urls` when Forgejo + base is known. +3. Post domain completion events (`fi_daily_brief`, …) so hub evidence stays + aligned with artefacts. +4. If activity-core can derive URLs alone, rein-aharness changes stay minimal — + still require stable `path` + `head_after`. + +**Done when:** at least one production claim after the change shows artefact +links in ops UI without hand-editing DB rows. + +## Task: Edge-relay / hub-offline behaviour for required resolvers + +```task +id: ACTIVITY-WP-0027-T06 +status: todo +priority: medium +``` + +1. Inventory which `state-hub` resolvers **must** hard-fail vs may degrade + (consistency sweep POST vs `fi_brief_status` GET cache). +2. Document edge-relay outbox + stale-read behaviour for operators. +3. Prefer: non-critical automations fail soft with explicit + `validation_failed` / warning in automation-status rather than Temporal + retry storms when upstream is 503. +4. Small code changes only when clearly scoped; larger redesign → split WP. + +**Done when:** runbook section exists and high-frequency 503 noise is either +fixed or explicitly accepted with a follow-up id. + +## Task: Retire remaining dual-clock host timers (Binky) + +```task +id: ACTIVITY-WP-0027-T07 +status: todo +priority: low +``` + +Depends on T02 (llm path solid) and proven claim-loop days for Binky. + +1. After ≥2 clean Binky claim-loop business days, disable + `binky-rhythm-daily.timer` (and document mail/review timers). +2. Update `docs/recurring-automations-playbook.md` + Binky recurrence docs. +3. Keep units on disk as break-glass; do not delete without operator ack. + +**Done when:** timers disabled or consciously retained with written reason. + +## Task: Deploy, smoke, and operator walkthrough + +```task +id: ACTIVITY-WP-0027-T08 +status: todo +priority: high +``` + +Depends on T02 + T04 (minimum). + +1. Deploy activity-core image + any host units to railiance01. +2. Smoke matrix: + - llm-connect health from host executor context + - trigger or wait for FI → ops_run succeeded → artefact link in UI + - cold check: no kubectl port-forward process required +3. Record non-secret evidence (URLs, run ids, commit SHAs) in this workplan. +4. Close with `statehub fix-consistency` after status updates. + +**Done when:** walkthrough steps work for a second operator on SSO UI only. + +## Acceptance (workplan level) + +- [ ] rein-aharness on railiance01 does not depend on ad-hoc `kubectl port-forward` for llm-connect +- [ ] `https://activity.coulomb.social/ops/ui` shows clickable artefact links for recent FI/Binky successes +- [ ] Contract tests cover join + URL builder; no secrets in responses +- [ ] Remaining caveats (edge 503, Binky timers) either fixed or explicitly deferred with owner + +## References + +- Ops console: ACTIVITY-WP-0024 / SSO ACTIVITY-WP-0025 +- Claim queue: ACTIVITY-WP-0026 · `docs/ops-run-queue.md` · ACT-ADR-005 +- llm-connect in-cluster: ACTIVITY-WP-0010 +- FI recurrence: `freedom-intelligence/docs/recurrence-ops.md` +- UI entry: `src/activity_core/ops_api.py` (`/ops/ui/automations/{id}`) +- Runs API: `src/activity_core/ops_console.py` (`ops_runs`)