frontend-patterns/research/bench/README.md

# Frontend research bench

Repository-owned workflows executed by the sibling `infospace-bench` package.
Verified against its checkout `be872bf` on 2026-09-24. This is a source investigation
workspace; `canon/`, `patterns/` and the research program remain authoritative.

## Quick start

The current workstation already has the bench environment installed. Override
`BENCH_PY` with another Python interpreter where `infospace-bench`, PyYAML and
pytest and llm-connect are installed.
If needed: `uv pip install --python "$BENCH_PY" -e ../llm-connect`. The sibling package has local dependencies on markitect-tool,
artifact-store and llm-connect; this repo does not duplicate their installation.

```bash
make bench-test
make bench-plan
```

`bench-plan` inspects the checked-in synthetic example without calling a model.
`infospaces/fixture-smoke/` contains its completed four-stage run, prompts, fixture
responses, provenance and review. It verifies workflow plumbing only. Its candidate
does not count toward the research corpus, evidence or M1 acceptance.

To repeat the fixture in a fresh temporary workspace:

```bash
BENCH_PY=../infospace-bench/.venv/bin/python
pilot_workspace=$(mktemp -d /tmp/frontend-patterns-bench.XXXXXX)
"$BENCH_PY" scripts/research_bench.py init research/bench/fixtures/source.md \
  --workspace "$pilot_workspace" --slug fixture
"$BENCH_PY" scripts/research_bench.py run "$pilot_workspace/infospaces/fixture" \
  --fixture-responses research/bench/fixtures/responses.yaml
```

## Investigate a real source

1. Obtain and review an original source using the [research protocol](../plan/ResearchProtocol.md).
   Prepare a bounded normalized Markdown excerpt with stable section/paragraph
   locators. Record any omitted context. No source acquisition or URL fetching is
   performed by this adapter.
2. Add the metadata shown in [SourceInput](SourceInput.md). Identify the original
   author, URL/bibliographic locator, version and access date. This is required
   provenance, not a claim that the source was independently verified.
3. Initialize a uniquely named workspace and inspect its plan:

```bash
../infospace-bench/.venv/bin/python scripts/research_bench.py init /path/to/source.md \
  --slug carniato-review-01
../infospace-bench/.venv/bin/python scripts/research_bench.py plan \
  research/bench/infospaces/carniato-review-01
```

4. Use the deployed llm-connect service. On this workstation, forward its port
   in a separate terminal (leave it running for the investigation):

```bash
kubectl -n activity-core port-forward svc/llm-connect 18081:8080
```

Then run in a fresh workspace:

```bash
../infospace-bench/.venv/bin/python scripts/research_bench.py run \
  research/bench/infospaces/carniato-review-01 \
  --llm-connect-url http://127.0.0.1:18081 \
  --model custodian-triage-balanced --max-tokens 12000 --max-calls 4
```

In-cluster consumers can use `http://llm-connect.activity-core.svc.cluster.local:8080`.
The service owns credentials and provider routing. `--model` names a service runtime
profile or requests a model; the resolved model is recorded from the response.
The bench uses llm-connect's `RunConfig` and `/execute` contract. It makes no direct
provider requests and has no client retries. Sources and drafts reach the service's
configured provider. The four-call ceiling bounds **service requests**, not internal
provider retries, tokens or dollars. `--max-tokens` is a requested per-response cap.
The service may apply profile settings. Actual model, usage, finish reason and
selected routing metadata are retained in receipts and artifact provenance; cost
remains unknown unless independently measured. Empty or incomplete responses stop
the workflow and retain diagnostic metadata. Successful runs remain unreviewed.

5. Review `artifacts/generated/claims.md`, `candidates.md`, `relation-proposals.md`
   and `output/evaluations/source-review.md`. The review receives original source
   text plus all previous outputs. It is an assisted self-review; a successful
   run never marks it independently reviewed.
6. Update the source/candidate catalogs and question ledger after review. Use the
   [admission procedure](Admission.md) before copying anything to `patterns/`.
   Then compare the two source investigations in VC-01/VC-02 walkthroughs.

## Execution and evidence boundaries

- One source per workspace. The inspected upstream runner keeps one intermediate
  result per stage, not per source. The adapter refuses multiple sources so stages
  cannot mix documents. Cross-source synthesis is a subsequent explicit review.
- Workflow order: claims → candidates → relation proposals → critique. Each stage
  retains the original source. Empty candidate/relation proposals are valid findings.
- Candidates and relations remain generated Markdown proposals. Native bench
  entity splitting requires at least one candidate and would reject legitimate
  negative results. Typed graph admission therefore follows review, not generation.
- Source digests are checked before running. Workflow/prompt digests and a start
  receipt are written before any call; failures remain visible. Reusing an attempted
  workspace is refused. Make a new revision workspace instead of overwriting results.
- Use this adapter's `plan` and `run` for the custom workflow. The generic `generate`
  pipeline selects its own workflow names; native dry-run rendering cannot resolve
  downstream references to assisted outputs that have not been generated yet.
- Bench `validate` checks its registered artifact contracts. Generated research
  proposals have no automatic semantic validation here. Structural metrics and an
  empty validation error list do not prove evidence quality or pattern validity.
- Fixture responses are allowed only for synthetic sources through the CLI.
  Fixture provenance remains visible in manifests and run records.

## Current source queue

| Investigation | Input readiness | Work record |
| --- | --- | --- |
| Carniato first source case | Manual review plus completed llm-connect draft; admission rejected | FEP-WP-0002-T02 |
| Contrasting local-first source | Manual review; llm-connect claims response truncated | FEP-WP-0002-T02 |
| VC-01 and VC-02 synthesis | Depends on reviewed source claims and candidate patterns | FEP-WP-0002-T03 |

Do not ingest the existing Carniato research stub as if it were the original article.


Primary-source plans (four calls each; each run failed on its first call):

```bash
../infospace-bench/.venv/bin/python scripts/research_bench.py plan research/bench/infospaces/carniato-authority-01
../infospace-bench/.venv/bin/python scripts/research_bench.py plan research/bench/infospaces/local-first-authority-01
```

These short packets isolate the first comparison question. They do not substitute
for full source review, independent known uses or the two architecture walkthroughs.


## Live pilot outcome — 2026-09-24

Both primary-source runs attempted Gemini 2.5 Flash through OpenRouter and failed
before producing research artifacts. A separate credential probe returned HTTP 401.
Their failed receipts remain immutable. Read [AuthorityPilot](../reviews/AuthorityPilot.md)
for the manual review that continued independently. Subsequent live runs use the llm-connect service and fresh workspace slugs;
no workstation provider credential is required.

Failure receipts now capture adapter/model and safe structured diagnostic fields;
raw provider error bodies, headers and credential values are never persisted.
The initial failed receipts predate this improvement; the review records their model.


## llm-connect service pilot — 2026-09-24

Live execution now uses llm-connect exclusively. The service's
`custodian-triage-balanced` profile resolved to `google/gemini-2.5-flash`:

| Workspace | Outcome | Service requests | Reported total tokens |
| --- | --- | --- | --- |
| carniato-authority-02 | Claims truncated at 4,000 completion tokens | 1 | 4,470 |
| carniato-authority-03 | Four stages completed; unreviewed generated artifacts | 4 | 12,638 |
| local-first-authority-02 | Claims truncated at 12,000 completion tokens | 1 | 12,489 |

All receipts are retained. Token totals include failed responses and are not dollar
costs. No additional local-first retry was made. See
[the service pilot review](../reviews/LLMConnectPilot.md) for rejected proposals and
follow-up work. The current profile is a triage profile; research suitability has
not been established. Successful service execution is not evidence acceptance.