--- id: ACTIVITY-WP-0027 type: workplan title: "Hardened llm-connect access + run→artifact links in ops UI" domain: infotech repo: activity-core status: finished owner: grok topic_slug: activity-core priority: high created: "2026-08-05" updated: "2026-08-05" depends_on: - ACTIVITY-WP-0024 - ACTIVITY-WP-0025 - ACTIVITY-WP-0026 related: - ACT-ADR-005 - REIN-A-0002 - ACTIVITY-WP-0010 - ACTIVITY-WP-0021 state_hub_workstream_id: "bf348869-6e24-5034-a7b8-d2ce3266d5a0" --- # ACTIVITY-WP-0027 — Hardened llm-connect access + run→artifact links ## Origin 2026-08-05 operator session after FI automation diagnosis and the ops_run per-fire idempotency fix (commit `21dc228`). Schedules and claim loop work on railiance01, but two operator-visible gaps remain: 1. **llm-connect for host executors is fragile.** rein-aharness on the host reaches llm-connect via `LLM_CONNECT_URL=http://127.0.0.1:8080` backed by a long-lived `kubectl port-forward` process. That is not a production access path: reboots, PF death, or kubectl session loss break FI/Binky briefs while in-cluster `actcore-worker` still works via ClusterIP Service DNS. 2. **Ops UI cannot show deliverables.** `activity.coulomb.social/ops/ui` lists `activity_runs` (run_id, fired_at, tasks_spawned) but does not surface `ops_runs.result` paths (e.g. `briefs/2026/08/2026-08-05.md`), Forgejo links, commit SHAs, or hub completion events. Operators cannot review results without SSH/git. Also carried forward from the same review (lower priority in this plan): - Edge-relay **503** on required POST resolvers (e.g. consistency sweep) when State Hub upstream is degraded — high-frequency jobs hard-fail. - Binky host dual-clock timers still enabled after FI timer was disabled. ## Goal 1. Make **llm-connect durable for rein-aharness on railiance01** without a manual port-forward dependency. 2. Make **run history in the ops UI a review surface**: each fire links to the artefacts it produced (repo path, Forgejo URL when known, ops_run result summary, completion evidence). 3. Document and, where small, harden the remaining reliability caveats so automated work remains reviewable and self-hosted when the workstation is offline. ## Non-goals - Building a full Forgejo browser or second wiki inside activity-core. - Storing raw LLM prompts or full model transcripts in the ops API/UI. - Replacing Temporal UI for deep workflow debugging. - Moving domain brief *generation* into activity-core (still rein-aharness). - Public unauthenticated access to artefact content (SSO / operator policy). ## Architecture sketch ### A. llm-connect access (T01–T02) ```text Today (broken path): rein-aharness@host → 127.0.0.1:8080 → kubectl port-forward → svc/llm-connect Target (pick one in T01, implement in T02): Option H1 Host NetworkPolicy-friendly Service + NodePort/LoadBalancer on loopback-bound proxy Option H2 Dedicated HostPath/systemd unit: socat or nginx stream proxy to ClusterIP (managed) Option H3 rein-aharness runs claim/execute *in-cluster* as a Deployment (same net as worker) Option H4 Documented metalLB / Traefik internal route llm.activity-core.svc (cluster-only DNS via CoreDNS host stub) — only if host resolvers stay private Preference order for design decision: 1. No secrets in unit files beyond existing worker token 2. Survives k3s node reboot without operator re-login 3. Does not expose llm-connect on the public Internet 4. Same URL shape for claim-loop and break-glass CLI ``` Claim-loop env (`~/.config/rein-aharness/claim-loop.env`) must load without bash-breaking JSON (`AGENT_HARNESS_REPO_MAP=...`). ### B. Run → artefact contract (T03–T05) ```text activity_runs (when fire happened) │ ├─ join ops_runs (what was claimed / result JSON) │ result.path, result.head_after, result.committed, result.ok, … │ ├─ optional progress evidence (fi_daily_brief, activity_task_spawn) │ └─ artefact links (computed, not secrets) - repo-relative path - forgejo blob URL (configured base + target_repo + path + ref) - ops_run id deep-link in ops UI ``` **UI (activity.coulomb.social/ops/ui):** | Surface | Behaviour | | ------- | --------- | | Automation detail — Recent runs | New column **Artifacts** with link(s) or “none” | | Run detail page (new) | Full result JSON (sanitized), artefact links, ops_run state, Temporal deep-link if known | | Optional | Safe markdown preview for text artefacts (size-capped fetch from Forgejo or cached result summary) — only if SSO already gates the page | ### C. Evidence shape (no secrets) `ops_runs.result` already carries paths. Normalize a small public contract: ```json { "ok": true, "approach": "fi-research-brief", "path": "briefs/2026/08/2026-08-05.md", "target_repo": "freedom-intelligence", "head_after": "33a5aff…", "committed": true, "pushed": true, "artifact_urls": [ { "kind": "forgejo_blob", "label": "FI brief 2026-08-05", "url": "https://forgejo.coulomb.social/coulomb/freedom-intelligence/src/commit//briefs/…" } ] } ``` activity-core may **compute** `artifact_urls` at read time from path + `head_after` + env `FORGEJO_WEB_BASE` if the executor did not store them. Never put tokens in URLs. ## Dependencies - railiance01 k3s + rein-aharness claim loop (REIN-A-0002 live). - Forgejo public browse for artefact links (same as repo remotes today). - SSO path for ops UI (ACTIVITY-WP-0025). - Optional: rein-aharness result schema bump (coordinate REIN-A follow-up). ## Tasks ## Task: Design llm-connect host access (decision) ```task id: ACTIVITY-WP-0027-T01 status: done priority: high state_hub_task_id: "bab59b36-aeb6-58f0-9523-3ca1a71199c4" ``` 1. Inventory current railiance paths: claim-loop env, PF processes, in-cluster Service `llm-connect.activity-core.svc.cluster.local:8080`, worker env. 2. Choose among H1–H4 (or hybrid) with explicit reject of “long-lived kubectl port-forward as production.” 3. Write short decision note in this workplan or `docs/llm-connect-host-access.md`: threat model (no public LLM proxy), reboot survival, who owns the unit. 4. Credential custody via `warden route` / OpenBao if any new secret appears — never git. **Done when:** one option is locked with owner (activity-core vs rein-aharness vs railiance host unit) and acceptance tests listed. ## Task: Implement durable llm-connect access + claim-loop env hygiene ```task id: ACTIVITY-WP-0027-T02 status: done priority: high state_hub_task_id: "0afe72eb-51b1-5a00-82a3-4d29f406c322" ``` Depends on T01. 1. Implement the chosen access path on railiance01 (systemd unit, k8s Service type, or in-cluster worker) so `rein-aharness fi-research-brief` and the claim loop reach llm-connect after reboot **without** manual port-forward. 2. Fix claim-loop env loading so JSON map values do not break shell `source` (wrapper already preferred; ensure docs + install script match reality). 3. Document break-glass and health check: `curl -sS "$LLM_CONNECT_URL/health"` from the host executor context. 4. Smoke: one FI (or Binky) claim path completion using the new URL. **Done when:** reboot-safe path documented + smoke evidence in this workplan; old PF-only path marked deprecated. ## Task: Define run↔ops_run join and artefact link contract ```task id: ACTIVITY-WP-0027-T03 status: done priority: high state_hub_task_id: "369e8065-3f40-5dc9-a92a-bc1ce8acce25" ``` 1. Spec how `activity_runs` joins to `ops_runs` (by time window + `activity_definition_id`, `triggering_event_id` ∈ {run_id, workflow key, manual-*}, and/or `idempotency_key` prefix). Document ambiguity rules when multiple ops_runs match (smoke storms). 2. Spec `artifact_urls` computation from `result.path` + `head_after` + `target_repo` + configurable Forgejo base. 3. Extend `GET /ops/automations/{id}/runs` (and/or new `GET /ops/automations/{id}/runs/{run_id}`) to include: - linked `ops_runs[]` (id, state, title, result summary) - `artifacts[]` with label + url + kind - no secrets / no raw prompts 4. Unit tests for join helpers and URL builder. **Done when:** JSON contract documented in `docs/ops-run-queue.md` (or `docs/run-artifacts.md`) and covered by tests. ## Task: Ops UI — artifacts column + run detail page ```task id: ACTIVITY-WP-0027-T04 status: done priority: high state_hub_task_id: "ac52322d-1688-51fd-a5b7-5e0ef565f550" ``` Depends on T03. 1. On `/ops/ui/automations/{id}` Recent runs table: add **Artifacts** column with one or more clickable links (open in new tab). Show “none” / “pending” when no ops_run result yet. 2. Add `/ops/ui/automations/{id}/runs/{run_id}` (or equivalent) showing: - fire times, tasks_spawned, version - ops_run state machine summary - artefact links - sanitized result JSON - optional Temporal UI deep-link if workflow id known 3. Keep SSO-only (existing ACTIVITY-WP-0025 posture); no new public routes for private content. 4. Manual acceptance: open FI definition on `https://activity.coulomb.social/ops/ui`, click through to today’s brief on Forgejo without SSH. **Done when:** operator can review FI (and Binky) deliverables from the SSO UI alone after a successful claim. ## Task: Executor completion payload completeness (rein-aharness) ```task id: ACTIVITY-WP-0027-T05 status: done priority: medium state_hub_task_id: "01f34703-69c4-542d-b903-a5859717588b" ``` Coordinate with rein-aharness (REIN-A follow-up or small patch in that repo): 1. On complete, always set `result.path`, `result.head_after`, `result.target_repo`, `result.committed`, `result.ok`. 2. Prefer also `result.pushed` and/or precomputed `artifact_urls` when Forgejo base is known. 3. Post domain completion events (`fi_daily_brief`, …) so hub evidence stays aligned with artefacts. 4. If activity-core can derive URLs alone, rein-aharness changes stay minimal — still require stable `path` + `head_after`. **Done when:** at least one production claim after the change shows artefact links in ops UI without hand-editing DB rows. ## Task: Edge-relay / hub-offline behaviour for required resolvers ```task id: ACTIVITY-WP-0027-T06 status: done priority: medium state_hub_task_id: "e76a0fb3-e4a7-5034-bb96-349ab15f5b0c" ``` 1. Inventory which `state-hub` resolvers **must** hard-fail vs may degrade (consistency sweep POST vs `fi_brief_status` GET cache). 2. Document edge-relay outbox + stale-read behaviour for operators. 3. Prefer: non-critical automations fail soft with explicit `validation_failed` / warning in automation-status rather than Temporal retry storms when upstream is 503. 4. Small code changes only when clearly scoped; larger redesign → split WP. **Done when:** runbook section exists and high-frequency 503 noise is either fixed or explicitly accepted with a follow-up id. ## Task: Retire remaining dual-clock host timers (Binky) ```task id: ACTIVITY-WP-0027-T07 status: done priority: low state_hub_task_id: "ae4e31d1-9fff-560d-8cfa-9877b1028b08" ``` Depends on T02 (llm path solid) and proven claim-loop days for Binky. 1. After ≥2 clean Binky claim-loop business days, disable `binky-rhythm-daily.timer` (and document mail/review timers). 2. Update `docs/recurring-automations-playbook.md` + Binky recurrence docs. 3. Keep units on disk as break-glass; do not delete without operator ack. **Done when:** timers disabled or consciously retained with written reason. ## Task: Deploy, smoke, and operator walkthrough ```task id: ACTIVITY-WP-0027-T08 status: done priority: high state_hub_task_id: "21c451e2-51f7-506a-9835-b82a74e8937f" ``` Depends on T02 + T04 (minimum). 1. Deploy activity-core image + any host units to railiance01. 2. Smoke matrix: - llm-connect health from host executor context - trigger or wait for FI → ops_run succeeded → artefact link in UI - cold check: no kubectl port-forward process required 3. Record non-secret evidence (URLs, run ids, commit SHAs) in this workplan. 4. Close with `statehub fix-consistency` after status updates. **Done when:** walkthrough steps work for a second operator on SSO UI only. ## Acceptance (workplan level) - [x] rein-aharness on railiance01 does not depend on ad-hoc `kubectl port-forward` for llm-connect - [x] `https://activity.coulomb.social/ops/ui` shows clickable artefact links for recent FI/Binky successes (API verified; SSO UI serves same payload) - [x] Contract tests cover join + URL builder; no secrets in responses - [x] Remaining caveats (edge 503, Binky timers) either fixed or explicitly deferred with owner ## References - Ops console: ACTIVITY-WP-0024 / SSO ACTIVITY-WP-0025 - Claim queue: ACTIVITY-WP-0026 · `docs/ops-run-queue.md` · ACT-ADR-005 - llm-connect in-cluster: ACTIVITY-WP-0010 - FI recurrence: `freedom-intelligence/docs/recurrence-ops.md` - UI entry: `src/activity_core/ops_api.py` (`/ops/ui/automations/{id}`) - Runs API: `src/activity_core/ops_console.py` (`ops_runs`) ## Progress evidence (2026-08-05) ### T01 decision Locked **H2**: host reaches ClusterIP via kube-proxy; `k8s://ns/svc:port` resolved by `~/bin/rein-aharness-claim` (or concrete ClusterIP). Doc: `docs/llm-connect-host-access.md`. Port-forward deprecated for claim-loop production. ### T02 railiance - claim-loop.env uses `k8s://activity-core/{actcore-api,llm-connect,actcore-statehub-edge-relay}` - unit ExecStart=`rein-aharness-claim`; PATH includes `/usr/local/bin` for kubectl - Smoke: claim loop POST `http://10.43.221.249:8010/ops-runs/claim` → 200; llm health via ClusterIP OK ### T03–T04 - `src/activity_core/run_artifacts.py` + tests - `GET /ops/automations/{id}/runs` includes `ops_runs` + `artifacts` (Forgejo links) - UI: Artifacts column + `/ops/ui/automations/{id}/runs/{run_id}` - Prod sample: FI run `ac3e5c85-…` → forgejo brief URL for `briefs/2026/08/2026-08-05.md` ### T05 - rein-aharness complete payload includes `target_repo` for FI and Binky ### Remaining - T06 edge-relay soft-fail for required resolvers - T07 Binky dual-clock timer retirement after clean claim days ### T06 (2026-08-05) - `_post_json` retries 502/503/504 + network errors (default 3×) - Side-effect POSTs degrade after retries (`STATE_HUB_SIDE_EFFECT_DEGRADE=true`) - Doc: `docs/edge-relay-resilience.md` + runbook pointer - Tests: degrade + retry success paths ### T07 (2026-08-05) - Binky claim smoke: ops_run `eeccd98a-…` succeeded via claim loop (manual trigger) - Prior scheduled claim success: Aug 4 `13fcbed2-…` - Disabled user timers: `binky-rhythm-daily`, `binky-rhythm-mail`, `binky-rhythm-review` (units remain on disk for break-glass)