workplan: ACTIVITY-WP-0027 llm-connect access + run artefacts in ops UI
Capture post-FI-cutover reliability caveats: durable host llm-connect for rein-aharness, and operator review of deliverables from activity.coulomb.social without SSH.
This commit is contained in:
parent
3b2528ab84
commit
d22b57f087
1 changed files with 339 additions and 0 deletions
339
workplans/ACTIVITY-WP-0027-llm-connect-and-run-artifacts.md
Normal file
339
workplans/ACTIVITY-WP-0027-llm-connect-and-run-artifacts.md
Normal file
|
|
@ -0,0 +1,339 @@
|
|||
---
|
||||
id: ACTIVITY-WP-0027
|
||||
type: workplan
|
||||
title: "Hardened llm-connect access + run→artifact links in ops UI"
|
||||
domain: infotech
|
||||
repo: activity-core
|
||||
status: ready
|
||||
owner: grok
|
||||
topic_slug: activity-core
|
||||
priority: high
|
||||
created: "2026-08-05"
|
||||
updated: "2026-08-05"
|
||||
depends_on:
|
||||
- ACTIVITY-WP-0024
|
||||
- ACTIVITY-WP-0025
|
||||
- ACTIVITY-WP-0026
|
||||
related:
|
||||
- ACT-ADR-005
|
||||
- REIN-A-0002
|
||||
- ACTIVITY-WP-0010
|
||||
- ACTIVITY-WP-0021
|
||||
---
|
||||
|
||||
# ACTIVITY-WP-0027 — Hardened llm-connect access + run→artifact links
|
||||
|
||||
## Origin
|
||||
|
||||
2026-08-05 operator session after FI automation diagnosis and the ops_run
|
||||
per-fire idempotency fix (commit `21dc228`). Schedules and claim loop work on
|
||||
railiance01, but two operator-visible gaps remain:
|
||||
|
||||
1. **llm-connect for host executors is fragile.** rein-aharness on the host
|
||||
reaches llm-connect via `LLM_CONNECT_URL=http://127.0.0.1:8080` backed by a
|
||||
long-lived `kubectl port-forward` process. That is not a production access
|
||||
path: reboots, PF death, or kubectl session loss break FI/Binky briefs while
|
||||
in-cluster `actcore-worker` still works via ClusterIP Service DNS.
|
||||
2. **Ops UI cannot show deliverables.** `activity.coulomb.social/ops/ui` lists
|
||||
`activity_runs` (run_id, fired_at, tasks_spawned) but does not surface
|
||||
`ops_runs.result` paths (e.g. `briefs/2026/08/2026-08-05.md`), Forgejo
|
||||
links, commit SHAs, or hub completion events. Operators cannot review
|
||||
results without SSH/git.
|
||||
|
||||
Also carried forward from the same review (lower priority in this plan):
|
||||
|
||||
- Edge-relay **503** on required POST resolvers (e.g. consistency sweep) when
|
||||
State Hub upstream is degraded — high-frequency jobs hard-fail.
|
||||
- Binky host dual-clock timers still enabled after FI timer was disabled.
|
||||
|
||||
## Goal
|
||||
|
||||
1. Make **llm-connect durable for rein-aharness on railiance01** without a
|
||||
manual port-forward dependency.
|
||||
2. Make **run history in the ops UI a review surface**: each fire links to
|
||||
the artefacts it produced (repo path, Forgejo URL when known, ops_run
|
||||
result summary, completion evidence).
|
||||
3. Document and, where small, harden the remaining reliability caveats so
|
||||
automated work remains reviewable and self-hosted when the workstation is
|
||||
offline.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Building a full Forgejo browser or second wiki inside activity-core.
|
||||
- Storing raw LLM prompts or full model transcripts in the ops API/UI.
|
||||
- Replacing Temporal UI for deep workflow debugging.
|
||||
- Moving domain brief *generation* into activity-core (still rein-aharness).
|
||||
- Public unauthenticated access to artefact content (SSO / operator policy).
|
||||
|
||||
## Architecture sketch
|
||||
|
||||
### A. llm-connect access (T01–T02)
|
||||
|
||||
```text
|
||||
Today (broken path):
|
||||
rein-aharness@host → 127.0.0.1:8080 → kubectl port-forward → svc/llm-connect
|
||||
|
||||
Target (pick one in T01, implement in T02):
|
||||
Option H1 Host NetworkPolicy-friendly Service + NodePort/LoadBalancer on loopback-bound proxy
|
||||
Option H2 Dedicated HostPath/systemd unit: socat or nginx stream proxy to ClusterIP (managed)
|
||||
Option H3 rein-aharness runs claim/execute *in-cluster* as a Deployment (same net as worker)
|
||||
Option H4 Documented metalLB / Traefik internal route llm.activity-core.svc (cluster-only DNS
|
||||
via CoreDNS host stub) — only if host resolvers stay private
|
||||
|
||||
Preference order for design decision:
|
||||
1. No secrets in unit files beyond existing worker token
|
||||
2. Survives k3s node reboot without operator re-login
|
||||
3. Does not expose llm-connect on the public Internet
|
||||
4. Same URL shape for claim-loop and break-glass CLI
|
||||
```
|
||||
|
||||
Claim-loop env (`~/.config/rein-aharness/claim-loop.env`) must load without
|
||||
bash-breaking JSON (`AGENT_HARNESS_REPO_MAP=...`).
|
||||
|
||||
### B. Run → artefact contract (T03–T05)
|
||||
|
||||
```text
|
||||
activity_runs (when fire happened)
|
||||
│
|
||||
├─ join ops_runs (what was claimed / result JSON)
|
||||
│ result.path, result.head_after, result.committed, result.ok, …
|
||||
│
|
||||
├─ optional progress evidence (fi_daily_brief, activity_task_spawn)
|
||||
│
|
||||
└─ artefact links (computed, not secrets)
|
||||
- repo-relative path
|
||||
- forgejo blob URL (configured base + target_repo + path + ref)
|
||||
- ops_run id deep-link in ops UI
|
||||
```
|
||||
|
||||
**UI (activity.coulomb.social/ops/ui):**
|
||||
|
||||
| Surface | Behaviour |
|
||||
| ------- | --------- |
|
||||
| Automation detail — Recent runs | New column **Artifacts** with link(s) or “none” |
|
||||
| Run detail page (new) | Full result JSON (sanitized), artefact links, ops_run state, Temporal deep-link if known |
|
||||
| Optional | Safe markdown preview for text artefacts (size-capped fetch from Forgejo or cached result summary) — only if SSO already gates the page |
|
||||
|
||||
### C. Evidence shape (no secrets)
|
||||
|
||||
`ops_runs.result` already carries paths. Normalize a small public contract:
|
||||
|
||||
```json
|
||||
{
|
||||
"ok": true,
|
||||
"approach": "fi-research-brief",
|
||||
"path": "briefs/2026/08/2026-08-05.md",
|
||||
"target_repo": "freedom-intelligence",
|
||||
"head_after": "33a5aff…",
|
||||
"committed": true,
|
||||
"pushed": true,
|
||||
"artifact_urls": [
|
||||
{
|
||||
"kind": "forgejo_blob",
|
||||
"label": "FI brief 2026-08-05",
|
||||
"url": "https://forgejo.coulomb.social/coulomb/freedom-intelligence/src/commit/<sha>/briefs/…"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
activity-core may **compute** `artifact_urls` at read time from path +
|
||||
`head_after` + env `FORGEJO_WEB_BASE` if the executor did not store them.
|
||||
Never put tokens in URLs.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- railiance01 k3s + rein-aharness claim loop (REIN-A-0002 live).
|
||||
- Forgejo public browse for artefact links (same as repo remotes today).
|
||||
- SSO path for ops UI (ACTIVITY-WP-0025).
|
||||
- Optional: rein-aharness result schema bump (coordinate REIN-A follow-up).
|
||||
|
||||
## Tasks
|
||||
|
||||
## Task: Design llm-connect host access (decision)
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0027-T01
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
1. Inventory current railiance paths: claim-loop env, PF processes, in-cluster
|
||||
Service `llm-connect.activity-core.svc.cluster.local:8080`, worker env.
|
||||
2. Choose among H1–H4 (or hybrid) with explicit reject of “long-lived kubectl
|
||||
port-forward as production.”
|
||||
3. Write short decision note in this workplan or `docs/llm-connect-host-access.md`:
|
||||
threat model (no public LLM proxy), reboot survival, who owns the unit.
|
||||
4. Credential custody via `warden route` / OpenBao if any new secret appears —
|
||||
never git.
|
||||
|
||||
**Done when:** one option is locked with owner (activity-core vs rein-aharness
|
||||
vs railiance host unit) and acceptance tests listed.
|
||||
|
||||
## Task: Implement durable llm-connect access + claim-loop env hygiene
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0027-T02
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Depends on T01.
|
||||
|
||||
1. Implement the chosen access path on railiance01 (systemd unit, k8s Service
|
||||
type, or in-cluster worker) so `rein-aharness fi-research-brief` and the
|
||||
claim loop reach llm-connect after reboot **without** manual port-forward.
|
||||
2. Fix claim-loop env loading so JSON map values do not break shell `source`
|
||||
(wrapper already preferred; ensure docs + install script match reality).
|
||||
3. Document break-glass and health check:
|
||||
`curl -sS "$LLM_CONNECT_URL/health"` from the host executor context.
|
||||
4. Smoke: one FI (or Binky) claim path completion using the new URL.
|
||||
|
||||
**Done when:** reboot-safe path documented + smoke evidence in this workplan;
|
||||
old PF-only path marked deprecated.
|
||||
|
||||
## Task: Define run↔ops_run join and artefact link contract
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0027-T03
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
1. Spec how `activity_runs` joins to `ops_runs` (by time window +
|
||||
`activity_definition_id`, `triggering_event_id` ∈ {run_id, workflow key,
|
||||
manual-*}, and/or `idempotency_key` prefix). Document ambiguity rules when
|
||||
multiple ops_runs match (smoke storms).
|
||||
2. Spec `artifact_urls` computation from `result.path` + `head_after` +
|
||||
`target_repo` + configurable Forgejo base.
|
||||
3. Extend `GET /ops/automations/{id}/runs` (and/or new
|
||||
`GET /ops/automations/{id}/runs/{run_id}`) to include:
|
||||
- linked `ops_runs[]` (id, state, title, result summary)
|
||||
- `artifacts[]` with label + url + kind
|
||||
- no secrets / no raw prompts
|
||||
4. Unit tests for join helpers and URL builder.
|
||||
|
||||
**Done when:** JSON contract documented in `docs/ops-run-queue.md` (or
|
||||
`docs/run-artifacts.md`) and covered by tests.
|
||||
|
||||
## Task: Ops UI — artifacts column + run detail page
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0027-T04
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Depends on T03.
|
||||
|
||||
1. On `/ops/ui/automations/{id}` Recent runs table: add **Artifacts** column
|
||||
with one or more clickable links (open in new tab). Show “none” / “pending”
|
||||
when no ops_run result yet.
|
||||
2. Add `/ops/ui/automations/{id}/runs/{run_id}` (or equivalent) showing:
|
||||
- fire times, tasks_spawned, version
|
||||
- ops_run state machine summary
|
||||
- artefact links
|
||||
- sanitized result JSON
|
||||
- optional Temporal UI deep-link if workflow id known
|
||||
3. Keep SSO-only (existing ACTIVITY-WP-0025 posture); no new public routes for
|
||||
private content.
|
||||
4. Manual acceptance: open FI definition on
|
||||
`https://activity.coulomb.social/ops/ui`, click through to today’s brief on
|
||||
Forgejo without SSH.
|
||||
|
||||
**Done when:** operator can review FI (and Binky) deliverables from the SSO UI
|
||||
alone after a successful claim.
|
||||
|
||||
## Task: Executor completion payload completeness (rein-aharness)
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0027-T05
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Coordinate with rein-aharness (REIN-A follow-up or small patch in that repo):
|
||||
|
||||
1. On complete, always set `result.path`, `result.head_after`,
|
||||
`result.target_repo`, `result.committed`, `result.ok`.
|
||||
2. Prefer also `result.pushed` and/or precomputed `artifact_urls` when Forgejo
|
||||
base is known.
|
||||
3. Post domain completion events (`fi_daily_brief`, …) so hub evidence stays
|
||||
aligned with artefacts.
|
||||
4. If activity-core can derive URLs alone, rein-aharness changes stay minimal —
|
||||
still require stable `path` + `head_after`.
|
||||
|
||||
**Done when:** at least one production claim after the change shows artefact
|
||||
links in ops UI without hand-editing DB rows.
|
||||
|
||||
## Task: Edge-relay / hub-offline behaviour for required resolvers
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0027-T06
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
1. Inventory which `state-hub` resolvers **must** hard-fail vs may degrade
|
||||
(consistency sweep POST vs `fi_brief_status` GET cache).
|
||||
2. Document edge-relay outbox + stale-read behaviour for operators.
|
||||
3. Prefer: non-critical automations fail soft with explicit
|
||||
`validation_failed` / warning in automation-status rather than Temporal
|
||||
retry storms when upstream is 503.
|
||||
4. Small code changes only when clearly scoped; larger redesign → split WP.
|
||||
|
||||
**Done when:** runbook section exists and high-frequency 503 noise is either
|
||||
fixed or explicitly accepted with a follow-up id.
|
||||
|
||||
## Task: Retire remaining dual-clock host timers (Binky)
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0027-T07
|
||||
status: todo
|
||||
priority: low
|
||||
```
|
||||
|
||||
Depends on T02 (llm path solid) and proven claim-loop days for Binky.
|
||||
|
||||
1. After ≥2 clean Binky claim-loop business days, disable
|
||||
`binky-rhythm-daily.timer` (and document mail/review timers).
|
||||
2. Update `docs/recurring-automations-playbook.md` + Binky recurrence docs.
|
||||
3. Keep units on disk as break-glass; do not delete without operator ack.
|
||||
|
||||
**Done when:** timers disabled or consciously retained with written reason.
|
||||
|
||||
## Task: Deploy, smoke, and operator walkthrough
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0027-T08
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Depends on T02 + T04 (minimum).
|
||||
|
||||
1. Deploy activity-core image + any host units to railiance01.
|
||||
2. Smoke matrix:
|
||||
- llm-connect health from host executor context
|
||||
- trigger or wait for FI → ops_run succeeded → artefact link in UI
|
||||
- cold check: no kubectl port-forward process required
|
||||
3. Record non-secret evidence (URLs, run ids, commit SHAs) in this workplan.
|
||||
4. Close with `statehub fix-consistency` after status updates.
|
||||
|
||||
**Done when:** walkthrough steps work for a second operator on SSO UI only.
|
||||
|
||||
## Acceptance (workplan level)
|
||||
|
||||
- [ ] rein-aharness on railiance01 does not depend on ad-hoc `kubectl port-forward` for llm-connect
|
||||
- [ ] `https://activity.coulomb.social/ops/ui` shows clickable artefact links for recent FI/Binky successes
|
||||
- [ ] Contract tests cover join + URL builder; no secrets in responses
|
||||
- [ ] Remaining caveats (edge 503, Binky timers) either fixed or explicitly deferred with owner
|
||||
|
||||
## References
|
||||
|
||||
- Ops console: ACTIVITY-WP-0024 / SSO ACTIVITY-WP-0025
|
||||
- Claim queue: ACTIVITY-WP-0026 · `docs/ops-run-queue.md` · ACT-ADR-005
|
||||
- llm-connect in-cluster: ACTIVITY-WP-0010
|
||||
- FI recurrence: `freedom-intelligence/docs/recurrence-ops.md`
|
||||
- UI entry: `src/activity_core/ops_api.py` (`/ops/ui/automations/{id}`)
|
||||
- Runs API: `src/activity_core/ops_console.py` (`ops_runs`)
|
||||
Loading…
Add table
Add a link
Reference in a new issue