coulomb/frontend-patterns
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Route live research through llm-connect and record pilot findings

Browse source 
Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a0cffe-206f-7323-89d3-b20ca4cf4bf9
This commit is contained in:
tegwick 2026-09-24 15:55:31 +02:00
parent 1f95f6389a
commit 2ec23bb22c
39 changed files with 2259 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

58
research/bench/README.md
View file

@ -8,7 +8,8 @@ workspace; `canon/`, `patterns/` and the research program remain authoritative.
The current workstation already has the bench environment installed. Override
`BENCH_PY` with another Python interpreter where `infospace-bench`, PyYAML and
pytest are installed. The sibling package has local dependencies on markitect-tool,
pytest and llm-connect are installed.
If needed: `uv pip install --python "$BENCH_PY" -e ../llm-connect`. The sibling package has local dependencies on markitect-tool,
artifact-store and llm-connect; this repo does not duplicate their installation.
```bash
@ -50,20 +51,33 @@ pilot_workspace=$(mktemp -d /tmp/frontend-patterns-bench.XXXXXX)
research/bench/infospaces/carniato-review-01
```
4. For a live run, configure `OPENROUTER_API_KEY` using the credential owner described
in `AGENTS.md`, choose a model explicitly, then run:
4. Use the deployed llm-connect service. On this workstation, forward its port
in a separate terminal (leave it running for the investigation):
```bash
kubectl -n activity-core port-forward svc/llm-connect 18081:8080
```
Then run in a fresh workspace:
```bash
../infospace-bench/.venv/bin/python scripts/research_bench.py run \
research/bench/infospaces/carniato-review-01 \
--provider openrouter --model YOUR_MODEL_ID --max-calls 4
--llm-connect-url http://127.0.0.1:18081 \
--model custodian-triage-balanced --max-tokens 12000 --max-calls 4
```
Live use transmits the selected source and intermediate drafts to that provider.
There is no default provider or model. The four-call ceiling is enforced before and
during execution, with automatic OpenRouter retries disabled. It is **not a dollar
or token budget**; actual usage metadata appears in the bench run record. No live
provider call was made during setup.
In-cluster consumers can use `http://llm-connect.activity-core.svc.cluster.local:8080`.
The service owns credentials and provider routing. `--model` names a service runtime
profile or requests a model; the resolved model is recorded from the response.
The bench uses llm-connect's `RunConfig` and `/execute` contract. It makes no direct
provider requests and has no client retries. Sources and drafts reach the service's
configured provider. The four-call ceiling bounds **service requests**, not internal
provider retries, tokens or dollars. `--max-tokens` is a requested per-response cap.
The service may apply profile settings. Actual model, usage, finish reason and
selected routing metadata are retained in receipts and artifact provenance; cost
remains unknown unless independently measured. Empty or incomplete responses stop
the workflow and retain diagnostic metadata. Successful runs remain unreviewed.
5. Review `artifacts/generated/claims.md`, `candidates.md`, `relation-proposals.md`
and `output/evaluations/source-review.md`. The review receives original source
@ -99,8 +113,8 @@ provider call was made during setup.
| Investigation | Input readiness | Work record |
| --- | --- | --- |
| Carniato first source case | Initial manual review recorded; provider attempt failed | FEP-WP-0002-T02 |
| Contrasting local-first source | Initial manual review recorded; provider attempt failed | FEP-WP-0002-T02 |
| Carniato first source case | Manual review plus completed llm-connect draft; admission rejected | FEP-WP-0002-T02 |
| Contrasting local-first source | Manual review; llm-connect claims response truncated | FEP-WP-0002-T02 |
| VC-01 and VC-02 synthesis | Depends on reviewed source claims and candidate patterns | FEP-WP-0002-T03 |
Do not ingest the existing Carniato research stub as if it were the original article.
@ -122,9 +136,27 @@ for full source review, independent known uses or the two architecture walkthrou
Both primary-source runs attempted Gemini 2.5 Flash through OpenRouter and failed
before producing research artifacts. A separate credential probe returned HTTP 401.
Their failed receipts remain immutable. Read [AuthorityPilot](../reviews/AuthorityPilot.md)
for the manual review that continued independently. New runs require a working
credential authorized for this client and fresh workspace slugs.
for the manual review that continued independently. Subsequent live runs use the llm-connect service and fresh workspace slugs;
no workstation provider credential is required.
Failure receipts now capture adapter/model and safe structured diagnostic fields;
raw provider error bodies, headers and credential values are never persisted.
The initial failed receipts predate this improvement; the review records their model.
## llm-connect service pilot — 2026-09-24
Live execution now uses llm-connect exclusively. The service's
`custodian-triage-balanced` profile resolved to `google/gemini-2.5-flash`:
| Workspace | Outcome | Service requests | Reported total tokens |
| --- | --- | --- | --- |
| carniato-authority-02 | Claims truncated at 4,000 completion tokens | 1 | 4,470 |
| carniato-authority-03 | Four stages completed; unreviewed generated artifacts | 4 | 12,638 |
| local-first-authority-02 | Claims truncated at 12,000 completion tokens | 1 | 12,489 |
All receipts are retained. Token totals include failed responses and are not dollar
costs. No additional local-first retry was made. See
[the service pilot review](../reviews/LLMConnectPilot.md) for rejected proposals and
follow-up work. The current profile is a triage profile; research suitability has
not been established. Successful service execution is not evidence acceptance.