activity-core/workplans/ACTIVITY-WP-0027-llm-connect-and-run-artifacts.md

id	type	title	domain	repo	status	owner	topic_slug	priority	created	updated	depends_on	related	state_hub_workstream_id
ACTIVITY-WP-0027	workplan	Hardened llm-connect access + run→artifact links in ops UI	infotech	activity-core	finished	grok	activity-core	high	2026-08-05	2026-08-05	ACTIVITY-WP-0024 ACTIVITY-WP-0025 ACTIVITY-WP-0026	ACT-ADR-005 REIN-A-0002 ACTIVITY-WP-0010 ACTIVITY-WP-0021	bf348869-6e24-5034-a7b8-d2ce3266d5a0

ACTIVITY-WP-0027 — Hardened llm-connect access + run→artifact links

Origin

2026-08-05 operator session after FI automation diagnosis and the ops_run per-fire idempotency fix (commit 21dc228). Schedules and claim loop work on railiance01, but two operator-visible gaps remain:

1. llm-connect for host executors is fragile. rein-aharness on the host reaches llm-connect via LLM_CONNECT_URL=http://127.0.0.1:8080 backed by a long-lived kubectl port-forward process. That is not a production access path: reboots, PF death, or kubectl session loss break FI/Binky briefs while in-cluster actcore-worker still works via ClusterIP Service DNS.
2. Ops UI cannot show deliverables. activity.coulomb.social/ops/ui lists activity_runs (run_id, fired_at, tasks_spawned) but does not surface ops_runs.result paths (e.g. briefs/2026/08/2026-08-05.md), Forgejo links, commit SHAs, or hub completion events. Operators cannot review results without SSH/git.

Also carried forward from the same review (lower priority in this plan):

• Edge-relay 503 on required POST resolvers (e.g. consistency sweep) when State Hub upstream is degraded — high-frequency jobs hard-fail.
• Binky host dual-clock timers still enabled after FI timer was disabled.

Goal

1. Make llm-connect durable for rein-aharness on railiance01 without a manual port-forward dependency.
2. Make run history in the ops UI a review surface: each fire links to the artefacts it produced (repo path, Forgejo URL when known, ops_run result summary, completion evidence).
3. Document and, where small, harden the remaining reliability caveats so automated work remains reviewable and self-hosted when the workstation is offline.

Non-goals

• Building a full Forgejo browser or second wiki inside activity-core.
• Storing raw LLM prompts or full model transcripts in the ops API/UI.
• Replacing Temporal UI for deep workflow debugging.
• Moving domain brief generation into activity-core (still rein-aharness).
• Public unauthenticated access to artefact content (SSO / operator policy).

Architecture sketch

A. llm-connect access (T01–T02)

Today (broken path):
  rein-aharness@host → 127.0.0.1:8080 → kubectl port-forward → svc/llm-connect

Target (pick one in T01, implement in T02):
  Option H1  Host NetworkPolicy-friendly Service + NodePort/LoadBalancer on loopback-bound proxy
  Option H2  Dedicated HostPath/systemd unit: socat or nginx stream proxy to ClusterIP (managed)
  Option H3  rein-aharness runs claim/execute *in-cluster* as a Deployment (same net as worker)
  Option H4  Documented metalLB / Traefik internal route llm.activity-core.svc (cluster-only DNS
             via CoreDNS host stub) — only if host resolvers stay private

Preference order for design decision:
  1. No secrets in unit files beyond existing worker token
  2. Survives k3s node reboot without operator re-login
  3. Does not expose llm-connect on the public Internet
  4. Same URL shape for claim-loop and break-glass CLI

Claim-loop env (~/.config/rein-aharness/claim-loop.env) must load without bash-breaking JSON (AGENT_HARNESS_REPO_MAP=...).

B. Run → artefact contract (T03–T05)

activity_runs (when fire happened)
     │
     ├─ join ops_runs (what was claimed / result JSON)
     │     result.path, result.head_after, result.committed, result.ok, …
     │
     ├─ optional progress evidence (fi_daily_brief, activity_task_spawn)
     │
     └─ artefact links (computed, not secrets)
           - repo-relative path
           - forgejo blob URL (configured base + target_repo + path + ref)
           - ops_run id deep-link in ops UI

UI (activity.coulomb.social/ops/ui):

Surface	Behaviour
Automation detail — Recent runs	New column Artifacts with link(s) or “none”
Run detail page (new)	Full result JSON (sanitized), artefact links, ops_run state, Temporal deep-link if known
Optional	Safe markdown preview for text artefacts (size-capped fetch from Forgejo or cached result summary) — only if SSO already gates the page

C. Evidence shape (no secrets)

ops_runs.result already carries paths. Normalize a small public contract:

{
  "ok": true,
  "approach": "fi-research-brief",
  "path": "briefs/2026/08/2026-08-05.md",
  "target_repo": "freedom-intelligence",
  "head_after": "33a5aff…",
  "committed": true,
  "pushed": true,
  "artifact_urls": [
    {
      "kind": "forgejo_blob",
      "label": "FI brief 2026-08-05",
      "url": "https://forgejo.coulomb.social/coulomb/freedom-intelligence/src/commit/<sha>/briefs/…"
    }
  ]
}

activity-core may compute artifact_urls at read time from path + head_after + env FORGEJO_WEB_BASE if the executor did not store them. Never put tokens in URLs.

Dependencies

• railiance01 k3s + rein-aharness claim loop (REIN-A-0002 live).
• Forgejo public browse for artefact links (same as repo remotes today).
• SSO path for ops UI (ACTIVITY-WP-0025).
• Optional: rein-aharness result schema bump (coordinate REIN-A follow-up).

Tasks

Task: Design llm-connect host access (decision)

id: ACTIVITY-WP-0027-T01
status: done
priority: high
state_hub_task_id: "bab59b36-aeb6-58f0-9523-3ca1a71199c4"

1. Inventory current railiance paths: claim-loop env, PF processes, in-cluster Service llm-connect.activity-core.svc.cluster.local:8080, worker env.
2. Choose among H1–H4 (or hybrid) with explicit reject of “long-lived kubectl port-forward as production.”
3. Write short decision note in this workplan or docs/llm-connect-host-access.md: threat model (no public LLM proxy), reboot survival, who owns the unit.
4. Credential custody via warden route / OpenBao if any new secret appears — never git.

Done when: one option is locked with owner (activity-core vs rein-aharness vs railiance host unit) and acceptance tests listed.

Task: Implement durable llm-connect access + claim-loop env hygiene

id: ACTIVITY-WP-0027-T02
status: done
priority: high
state_hub_task_id: "0afe72eb-51b1-5a00-82a3-4d29f406c322"

Depends on T01.

1. Implement the chosen access path on railiance01 (systemd unit, k8s Service type, or in-cluster worker) so rein-aharness fi-research-brief and the claim loop reach llm-connect after reboot without manual port-forward.
2. Fix claim-loop env loading so JSON map values do not break shell source (wrapper already preferred; ensure docs + install script match reality).
3. Document break-glass and health check: curl -sS "$LLM_CONNECT_URL/health" from the host executor context.
4. Smoke: one FI (or Binky) claim path completion using the new URL.

Done when: reboot-safe path documented + smoke evidence in this workplan; old PF-only path marked deprecated.

Task: Define run↔ops_run join and artefact link contract

id: ACTIVITY-WP-0027-T03
status: done
priority: high
state_hub_task_id: "369e8065-3f40-5dc9-a92a-bc1ce8acce25"

1. Spec how activity_runs joins to ops_runs (by time window + activity_definition_id, triggering_event_id ∈ {run_id, workflow key, manual-*}, and/or idempotency_key prefix). Document ambiguity rules when multiple ops_runs match (smoke storms).
2. Spec artifact_urls computation from result.path + head_after + target_repo + configurable Forgejo base.
3. Extend GET /ops/automations/{id}/runs (and/or new GET /ops/automations/{id}/runs/{run_id}) to include:
   • linked ops_runs[] (id, state, title, result summary)
   • artifacts[] with label + url + kind
   • no secrets / no raw prompts
4. Unit tests for join helpers and URL builder.

Done when: JSON contract documented in docs/ops-run-queue.md (or docs/run-artifacts.md) and covered by tests.

Task: Ops UI — artifacts column + run detail page

id: ACTIVITY-WP-0027-T04
status: done
priority: high
state_hub_task_id: "ac52322d-1688-51fd-a5b7-5e0ef565f550"

Depends on T03.

1. On /ops/ui/automations/{id} Recent runs table: add Artifacts column with one or more clickable links (open in new tab). Show “none” / “pending” when no ops_run result yet.
2. Add /ops/ui/automations/{id}/runs/{run_id} (or equivalent) showing:
   • fire times, tasks_spawned, version
   • ops_run state machine summary
   • artefact links
   • sanitized result JSON
   • optional Temporal UI deep-link if workflow id known
3. Keep SSO-only (existing ACTIVITY-WP-0025 posture); no new public routes for private content.
4. Manual acceptance: open FI definition on https://activity.coulomb.social/ops/ui, click through to today’s brief on Forgejo without SSH.

Done when: operator can review FI (and Binky) deliverables from the SSO UI alone after a successful claim.

Task: Executor completion payload completeness (rein-aharness)

id: ACTIVITY-WP-0027-T05
status: done
priority: medium
state_hub_task_id: "01f34703-69c4-542d-b903-a5859717588b"

Coordinate with rein-aharness (REIN-A follow-up or small patch in that repo):

1. On complete, always set result.path, result.head_after, result.target_repo, result.committed, result.ok.
2. Prefer also result.pushed and/or precomputed artifact_urls when Forgejo base is known.
3. Post domain completion events (fi_daily_brief, …) so hub evidence stays aligned with artefacts.
4. If activity-core can derive URLs alone, rein-aharness changes stay minimal — still require stable path + head_after.

Done when: at least one production claim after the change shows artefact links in ops UI without hand-editing DB rows.

Task: Edge-relay / hub-offline behaviour for required resolvers

id: ACTIVITY-WP-0027-T06
status: done
priority: medium
state_hub_task_id: "e76a0fb3-e4a7-5034-bb96-349ab15f5b0c"

1. Inventory which state-hub resolvers must hard-fail vs may degrade (consistency sweep POST vs fi_brief_status GET cache).
2. Document edge-relay outbox + stale-read behaviour for operators.
3. Prefer: non-critical automations fail soft with explicit validation_failed / warning in automation-status rather than Temporal retry storms when upstream is 503.
4. Small code changes only when clearly scoped; larger redesign → split WP.

Done when: runbook section exists and high-frequency 503 noise is either fixed or explicitly accepted with a follow-up id.

Task: Retire remaining dual-clock host timers (Binky)

id: ACTIVITY-WP-0027-T07
status: done
priority: low
state_hub_task_id: "ae4e31d1-9fff-560d-8cfa-9877b1028b08"

Depends on T02 (llm path solid) and proven claim-loop days for Binky.

1. After ≥2 clean Binky claim-loop business days, disable binky-rhythm-daily.timer (and document mail/review timers).
2. Update docs/recurring-automations-playbook.md + Binky recurrence docs.
3. Keep units on disk as break-glass; do not delete without operator ack.

Done when: timers disabled or consciously retained with written reason.

Task: Deploy, smoke, and operator walkthrough

id: ACTIVITY-WP-0027-T08
status: done
priority: high
state_hub_task_id: "21c451e2-51f7-506a-9835-b82a74e8937f"

Depends on T02 + T04 (minimum).

1. Deploy activity-core image + any host units to railiance01.
2. Smoke matrix:
   • llm-connect health from host executor context
   • trigger or wait for FI → ops_run succeeded → artefact link in UI
   • cold check: no kubectl port-forward process required
3. Record non-secret evidence (URLs, run ids, commit SHAs) in this workplan.
4. Close with statehub fix-consistency after status updates.

Done when: walkthrough steps work for a second operator on SSO UI only.

Acceptance (workplan level)

• rein-aharness on railiance01 does not depend on ad-hoc kubectl port-forward for llm-connect
• https://activity.coulomb.social/ops/ui shows clickable artefact links for recent FI/Binky successes (API verified; SSO UI serves same payload)
• Contract tests cover join + URL builder; no secrets in responses
• Remaining caveats (edge 503, Binky timers) either fixed or explicitly deferred with owner

References

• Ops console: ACTIVITY-WP-0024 / SSO ACTIVITY-WP-0025
• Claim queue: ACTIVITY-WP-0026 · docs/ops-run-queue.md · ACT-ADR-005
• llm-connect in-cluster: ACTIVITY-WP-0010
• FI recurrence: freedom-intelligence/docs/recurrence-ops.md
• UI entry: src/activity_core/ops_api.py (/ops/ui/automations/{id})
• Runs API: src/activity_core/ops_console.py (ops_runs)

Progress evidence (2026-08-05)

T01 decision

Locked H2: host reaches ClusterIP via kube-proxy; k8s://ns/svc:port resolved by ~/bin/rein-aharness-claim (or concrete ClusterIP). Doc: docs/llm-connect-host-access.md. Port-forward deprecated for claim-loop production.

T02 railiance

• claim-loop.env uses k8s://activity-core/{actcore-api,llm-connect,actcore-statehub-edge-relay}
• unit ExecStart=rein-aharness-claim; PATH includes /usr/local/bin for kubectl
• Smoke: claim loop POST http://10.43.221.249:8010/ops-runs/claim → 200; llm health via ClusterIP OK

T03–T04

• src/activity_core/run_artifacts.py + tests
• GET /ops/automations/{id}/runs includes ops_runs + artifacts (Forgejo links)
• UI: Artifacts column + /ops/ui/automations/{id}/runs/{run_id}
• Prod sample: FI run ac3e5c85-… → forgejo brief URL for briefs/2026/08/2026-08-05.md

T05

• rein-aharness complete payload includes target_repo for FI and Binky

Remaining

• T06 edge-relay soft-fail for required resolvers
• T07 Binky dual-clock timer retirement after clean claim days

T06 (2026-08-05)

• _post_json retries 502/503/504 + network errors (default 3×)
• Side-effect POSTs degrade after retries (STATE_HUB_SIDE_EFFECT_DEGRADE=true)
• Doc: docs/edge-relay-resilience.md + runbook pointer
• Tests: degrade + retry success paths

T07 (2026-08-05)

• Binky claim smoke: ops_run eeccd98a-… succeeded via claim loop (manual trigger)
• Prior scheduled claim success: Aug 4 13fcbed2-…
• Disabled user timers: binky-rhythm-daily, binky-rhythm-mail, binky-rhythm-review (units remain on disk for break-glass)