activity-core/workplans/ACTIVITY-WP-0027-llm-connect-and-run-artifacts.md

---
id: ACTIVITY-WP-0027
type: workplan
title: "Hardened llm-connect access + run→artifact links in ops UI"
domain: infotech
repo: activity-core
status: finished
owner: grok
topic_slug: activity-core
priority: high
created: "2026-08-05"
updated: "2026-08-05"
depends_on:
  - ACTIVITY-WP-0024
  - ACTIVITY-WP-0025
  - ACTIVITY-WP-0026
related:
  - ACT-ADR-005
  - REIN-A-0002
  - ACTIVITY-WP-0010
  - ACTIVITY-WP-0021
state_hub_workstream_id: "bf348869-6e24-5034-a7b8-d2ce3266d5a0"
---

# ACTIVITY-WP-0027 — Hardened llm-connect access + run→artifact links

## Origin

2026-08-05 operator session after FI automation diagnosis and the ops_run
per-fire idempotency fix (commit `21dc228`). Schedules and claim loop work on
railiance01, but two operator-visible gaps remain:

1. **llm-connect for host executors is fragile.** rein-aharness on the host
   reaches llm-connect via `LLM_CONNECT_URL=http://127.0.0.1:8080` backed by a
   long-lived `kubectl port-forward` process. That is not a production access
   path: reboots, PF death, or kubectl session loss break FI/Binky briefs while
   in-cluster `actcore-worker` still works via ClusterIP Service DNS.
2. **Ops UI cannot show deliverables.** `activity.coulomb.social/ops/ui` lists
   `activity_runs` (run_id, fired_at, tasks_spawned) but does not surface
   `ops_runs.result` paths (e.g. `briefs/2026/08/2026-08-05.md`), Forgejo
   links, commit SHAs, or hub completion events. Operators cannot review
   results without SSH/git.

Also carried forward from the same review (lower priority in this plan):

- Edge-relay **503** on required POST resolvers (e.g. consistency sweep) when
  State Hub upstream is degraded — high-frequency jobs hard-fail.
- Binky host dual-clock timers still enabled after FI timer was disabled.

## Goal

1. Make **llm-connect durable for rein-aharness on railiance01** without a
   manual port-forward dependency.
2. Make **run history in the ops UI a review surface**: each fire links to
   the artefacts it produced (repo path, Forgejo URL when known, ops_run
   result summary, completion evidence).
3. Document and, where small, harden the remaining reliability caveats so
   automated work remains reviewable and self-hosted when the workstation is
   offline.

## Non-goals

- Building a full Forgejo browser or second wiki inside activity-core.
- Storing raw LLM prompts or full model transcripts in the ops API/UI.
- Replacing Temporal UI for deep workflow debugging.
- Moving domain brief *generation* into activity-core (still rein-aharness).
- Public unauthenticated access to artefact content (SSO / operator policy).

## Architecture sketch

### A. llm-connect access (T01–T02)

```text
Today (broken path):
  rein-aharness@host → 127.0.0.1:8080 → kubectl port-forward → svc/llm-connect

Target (pick one in T01, implement in T02):
  Option H1  Host NetworkPolicy-friendly Service + NodePort/LoadBalancer on loopback-bound proxy
  Option H2  Dedicated HostPath/systemd unit: socat or nginx stream proxy to ClusterIP (managed)
  Option H3  rein-aharness runs claim/execute *in-cluster* as a Deployment (same net as worker)
  Option H4  Documented metalLB / Traefik internal route llm.activity-core.svc (cluster-only DNS
             via CoreDNS host stub) — only if host resolvers stay private

Preference order for design decision:
  1. No secrets in unit files beyond existing worker token
  2. Survives k3s node reboot without operator re-login
  3. Does not expose llm-connect on the public Internet
  4. Same URL shape for claim-loop and break-glass CLI
```

Claim-loop env (`~/.config/rein-aharness/claim-loop.env`) must load without
bash-breaking JSON (`AGENT_HARNESS_REPO_MAP=...`).

### B. Run → artefact contract (T03–T05)

```text
activity_runs (when fire happened)
     │
     ├─ join ops_runs (what was claimed / result JSON)
     │     result.path, result.head_after, result.committed, result.ok, …
     │
     ├─ optional progress evidence (fi_daily_brief, activity_task_spawn)
     │
     └─ artefact links (computed, not secrets)
           - repo-relative path
           - forgejo blob URL (configured base + target_repo + path + ref)
           - ops_run id deep-link in ops UI
```

**UI (activity.coulomb.social/ops/ui):**

| Surface | Behaviour |
| ------- | --------- |
| Automation detail — Recent runs | New column **Artifacts** with link(s) or “none” |
| Run detail page (new) | Full result JSON (sanitized), artefact links, ops_run state, Temporal deep-link if known |
| Optional | Safe markdown preview for text artefacts (size-capped fetch from Forgejo or cached result summary) — only if SSO already gates the page |

### C. Evidence shape (no secrets)

`ops_runs.result` already carries paths. Normalize a small public contract:

```json
{
  "ok": true,
  "approach": "fi-research-brief",
  "path": "briefs/2026/08/2026-08-05.md",
  "target_repo": "freedom-intelligence",
  "head_after": "33a5aff…",
  "committed": true,
  "pushed": true,
  "artifact_urls": [
    {
      "kind": "forgejo_blob",
      "label": "FI brief 2026-08-05",
      "url": "https://forgejo.coulomb.social/coulomb/freedom-intelligence/src/commit/<sha>/briefs/…"
    }
  ]
}
```

activity-core may **compute** `artifact_urls` at read time from path +
`head_after` + env `FORGEJO_WEB_BASE` if the executor did not store them.
Never put tokens in URLs.

## Dependencies

- railiance01 k3s + rein-aharness claim loop (REIN-A-0002 live).
- Forgejo public browse for artefact links (same as repo remotes today).
- SSO path for ops UI (ACTIVITY-WP-0025).
- Optional: rein-aharness result schema bump (coordinate REIN-A follow-up).

## Tasks

## Task: Design llm-connect host access (decision)

```task
id: ACTIVITY-WP-0027-T01
status: done
priority: high
state_hub_task_id: "bab59b36-aeb6-58f0-9523-3ca1a71199c4"
```

1. Inventory current railiance paths: claim-loop env, PF processes, in-cluster
   Service `llm-connect.activity-core.svc.cluster.local:8080`, worker env.
2. Choose among H1–H4 (or hybrid) with explicit reject of “long-lived kubectl
   port-forward as production.”
3. Write short decision note in this workplan or `docs/llm-connect-host-access.md`:
   threat model (no public LLM proxy), reboot survival, who owns the unit.
4. Credential custody via `warden route` / OpenBao if any new secret appears —
   never git.

**Done when:** one option is locked with owner (activity-core vs rein-aharness
vs railiance host unit) and acceptance tests listed.

## Task: Implement durable llm-connect access + claim-loop env hygiene

```task
id: ACTIVITY-WP-0027-T02
status: done
priority: high
state_hub_task_id: "0afe72eb-51b1-5a00-82a3-4d29f406c322"
```

Depends on T01.

1. Implement the chosen access path on railiance01 (systemd unit, k8s Service
   type, or in-cluster worker) so `rein-aharness fi-research-brief` and the
   claim loop reach llm-connect after reboot **without** manual port-forward.
2. Fix claim-loop env loading so JSON map values do not break shell `source`
   (wrapper already preferred; ensure docs + install script match reality).
3. Document break-glass and health check:
   `curl -sS "$LLM_CONNECT_URL/health"` from the host executor context.
4. Smoke: one FI (or Binky) claim path completion using the new URL.

**Done when:** reboot-safe path documented + smoke evidence in this workplan;
old PF-only path marked deprecated.

## Task: Define run↔ops_run join and artefact link contract

```task
id: ACTIVITY-WP-0027-T03
status: done
priority: high
state_hub_task_id: "369e8065-3f40-5dc9-a92a-bc1ce8acce25"
```

1. Spec how `activity_runs` joins to `ops_runs` (by time window +
   `activity_definition_id`, `triggering_event_id` ∈ {run_id, workflow key,
   manual-*}, and/or `idempotency_key` prefix). Document ambiguity rules when
   multiple ops_runs match (smoke storms).
2. Spec `artifact_urls` computation from `result.path` + `head_after` +
   `target_repo` + configurable Forgejo base.
3. Extend `GET /ops/automations/{id}/runs` (and/or new
   `GET /ops/automations/{id}/runs/{run_id}`) to include:
   - linked `ops_runs[]` (id, state, title, result summary)
   - `artifacts[]` with label + url + kind
   - no secrets / no raw prompts
4. Unit tests for join helpers and URL builder.

**Done when:** JSON contract documented in `docs/ops-run-queue.md` (or
`docs/run-artifacts.md`) and covered by tests.

## Task: Ops UI — artifacts column + run detail page

```task
id: ACTIVITY-WP-0027-T04
status: done
priority: high
state_hub_task_id: "ac52322d-1688-51fd-a5b7-5e0ef565f550"
```

Depends on T03.

1. On `/ops/ui/automations/{id}` Recent runs table: add **Artifacts** column
   with one or more clickable links (open in new tab). Show “none” / “pending”
   when no ops_run result yet.
2. Add `/ops/ui/automations/{id}/runs/{run_id}` (or equivalent) showing:
   - fire times, tasks_spawned, version
   - ops_run state machine summary
   - artefact links
   - sanitized result JSON
   - optional Temporal UI deep-link if workflow id known
3. Keep SSO-only (existing ACTIVITY-WP-0025 posture); no new public routes for
   private content.
4. Manual acceptance: open FI definition on
   `https://activity.coulomb.social/ops/ui`, click through to today’s brief on
   Forgejo without SSH.

**Done when:** operator can review FI (and Binky) deliverables from the SSO UI
alone after a successful claim.

## Task: Executor completion payload completeness (rein-aharness)

```task
id: ACTIVITY-WP-0027-T05
status: done
priority: medium
state_hub_task_id: "01f34703-69c4-542d-b903-a5859717588b"
```

Coordinate with rein-aharness (REIN-A follow-up or small patch in that repo):

1. On complete, always set `result.path`, `result.head_after`,
   `result.target_repo`, `result.committed`, `result.ok`.
2. Prefer also `result.pushed` and/or precomputed `artifact_urls` when Forgejo
   base is known.
3. Post domain completion events (`fi_daily_brief`, …) so hub evidence stays
   aligned with artefacts.
4. If activity-core can derive URLs alone, rein-aharness changes stay minimal —
   still require stable `path` + `head_after`.

**Done when:** at least one production claim after the change shows artefact
links in ops UI without hand-editing DB rows.

## Task: Edge-relay / hub-offline behaviour for required resolvers

```task
id: ACTIVITY-WP-0027-T06
status: done
priority: medium
state_hub_task_id: "e76a0fb3-e4a7-5034-bb96-349ab15f5b0c"
```

1. Inventory which `state-hub` resolvers **must** hard-fail vs may degrade
   (consistency sweep POST vs `fi_brief_status` GET cache).
2. Document edge-relay outbox + stale-read behaviour for operators.
3. Prefer: non-critical automations fail soft with explicit
   `validation_failed` / warning in automation-status rather than Temporal
   retry storms when upstream is 503.
4. Small code changes only when clearly scoped; larger redesign → split WP.

**Done when:** runbook section exists and high-frequency 503 noise is either
fixed or explicitly accepted with a follow-up id.

## Task: Retire remaining dual-clock host timers (Binky)

```task
id: ACTIVITY-WP-0027-T07
status: done
priority: low
state_hub_task_id: "ae4e31d1-9fff-560d-8cfa-9877b1028b08"
```

Depends on T02 (llm path solid) and proven claim-loop days for Binky.

1. After ≥2 clean Binky claim-loop business days, disable
   `binky-rhythm-daily.timer` (and document mail/review timers).
2. Update `docs/recurring-automations-playbook.md` + Binky recurrence docs.
3. Keep units on disk as break-glass; do not delete without operator ack.

**Done when:** timers disabled or consciously retained with written reason.

## Task: Deploy, smoke, and operator walkthrough

```task
id: ACTIVITY-WP-0027-T08
status: done
priority: high
state_hub_task_id: "21c451e2-51f7-506a-9835-b82a74e8937f"
```

Depends on T02 + T04 (minimum).

1. Deploy activity-core image + any host units to railiance01.
2. Smoke matrix:
   - llm-connect health from host executor context
   - trigger or wait for FI → ops_run succeeded → artefact link in UI
   - cold check: no kubectl port-forward process required
3. Record non-secret evidence (URLs, run ids, commit SHAs) in this workplan.
4. Close with `statehub fix-consistency` after status updates.

**Done when:** walkthrough steps work for a second operator on SSO UI only.

## Acceptance (workplan level)

- [x] rein-aharness on railiance01 does not depend on ad-hoc `kubectl port-forward` for llm-connect
- [x] `https://activity.coulomb.social/ops/ui` shows clickable artefact links for recent FI/Binky successes (API verified; SSO UI serves same payload)
- [x] Contract tests cover join + URL builder; no secrets in responses
- [x] Remaining caveats (edge 503, Binky timers) either fixed or explicitly deferred with owner

## References

- Ops console: ACTIVITY-WP-0024 / SSO ACTIVITY-WP-0025
- Claim queue: ACTIVITY-WP-0026 · `docs/ops-run-queue.md` · ACT-ADR-005
- llm-connect in-cluster: ACTIVITY-WP-0010
- FI recurrence: `freedom-intelligence/docs/recurrence-ops.md`
- UI entry: `src/activity_core/ops_api.py` (`/ops/ui/automations/{id}`)
- Runs API: `src/activity_core/ops_console.py` (`ops_runs`)

## Progress evidence (2026-08-05)

### T01 decision
Locked **H2**: host reaches ClusterIP via kube-proxy; `k8s://ns/svc:port` resolved by
`~/bin/rein-aharness-claim` (or concrete ClusterIP). Doc: `docs/llm-connect-host-access.md`.
Port-forward deprecated for claim-loop production.

### T02 railiance
- claim-loop.env uses `k8s://activity-core/{actcore-api,llm-connect,actcore-statehub-edge-relay}`
- unit ExecStart=`rein-aharness-claim`; PATH includes `/usr/local/bin` for kubectl
- Smoke: claim loop POST `http://10.43.221.249:8010/ops-runs/claim` → 200; llm health via ClusterIP OK

### T03–T04
- `src/activity_core/run_artifacts.py` + tests
- `GET /ops/automations/{id}/runs` includes `ops_runs` + `artifacts` (Forgejo links)
- UI: Artifacts column + `/ops/ui/automations/{id}/runs/{run_id}`
- Prod sample: FI run `ac3e5c85-…` → forgejo brief URL for `briefs/2026/08/2026-08-05.md`

### T05
- rein-aharness complete payload includes `target_repo` for FI and Binky

### Remaining
- T06 edge-relay soft-fail for required resolvers
- T07 Binky dual-clock timer retirement after clean claim days

### T06 (2026-08-05)
- `_post_json` retries 502/503/504 + network errors (default 3×)
- Side-effect POSTs degrade after retries (`STATE_HUB_SIDE_EFFECT_DEGRADE=true`)
- Doc: `docs/edge-relay-resilience.md` + runbook pointer
- Tests: degrade + retry success paths

### T07 (2026-08-05)
- Binky claim smoke: ops_run `eeccd98a-…` succeeded via claim loop (manual trigger)
- Prior scheduled claim success: Aug 4 `13fcbed2-…`
- Disabled user timers: `binky-rhythm-daily`, `binky-rhythm-mail`, `binky-rhythm-review`
  (units remain on disk for break-glass)