frontend-patterns/research/bench/README.md

# Frontend research bench

Repository-owned workflows executed by the sibling `infospace-bench` package.
Verified against its checkout `be872bf` on 2026-09-24. This is a source investigation
workspace; `canon/`, `patterns/` and the research program remain authoritative.

## Quick start

The current workstation already has the bench environment installed. Override
`BENCH_PY` with another Python interpreter where `infospace-bench`, PyYAML and
pytest and llm-connect are installed.
If needed: `uv pip install --python "$BENCH_PY" -e ../llm-connect`. The sibling package has local dependencies on markitect-tool,
artifact-store and llm-connect; this repo does not duplicate their installation.

```bash
make bench-test
make bench-plan
```

`bench-plan` inspects the checked-in synthetic example without calling a model.
`infospaces/fixture-smoke/` contains its completed four-stage run, prompts, fixture
responses, provenance and review. It verifies workflow plumbing only. Its candidate
does not count toward the research corpus, evidence or M1 acceptance.

To repeat the fixture in a fresh temporary workspace:

```bash
BENCH_PY=../infospace-bench/.venv/bin/python
pilot_workspace=$(mktemp -d /tmp/frontend-patterns-bench.XXXXXX)
"$BENCH_PY" scripts/research_bench.py init research/bench/fixtures/source.md \
  --workspace "$pilot_workspace" --slug fixture
"$BENCH_PY" scripts/research_bench.py run "$pilot_workspace/infospaces/fixture" \
  --fixture-responses research/bench/fixtures/responses.yaml
```

## Investigate a real source

1. Obtain and review an original source using the [research protocol](../plan/ResearchProtocol.md).
   Prepare a bounded normalized Markdown excerpt with stable section/paragraph
   locators. Record any omitted context. No source acquisition or URL fetching is
   performed by this adapter.
2. Add the metadata shown in [SourceInput](SourceInput.md). Identify the original
   author, URL/bibliographic locator, version and access date. This is required
   provenance, not a claim that the source was independently verified.
3. Initialize a uniquely named workspace and inspect its plan:

```bash
../infospace-bench/.venv/bin/python scripts/research_bench.py init /path/to/source.md \
  --slug carniato-review-01
../infospace-bench/.venv/bin/python scripts/research_bench.py plan \
  research/bench/infospaces/carniato-review-01
```

4. Use the deployed llm-connect service. On this workstation, forward its port
   in a separate terminal (leave it running for the investigation):

```bash
kubectl -n activity-core port-forward svc/llm-connect 18081:8080
```

Then run in a fresh workspace:

```bash
../infospace-bench/.venv/bin/python scripts/research_bench.py run \
  research/bench/infospaces/carniato-review-01 \
  --llm-connect-url http://127.0.0.1:18081 \
  --model custodian-triage-balanced --diagnostics \
  --max-tokens 3000 --max-calls 4
```

In-cluster consumers can use `http://llm-connect.activity-core.svc.cluster.local:8080`.
The service owns credentials and provider routing. `--model` names a service runtime
profile or requests a model; the resolved model is recorded from the response.
The bench uses llm-connect's `RunConfig` and `/execute` contract. It makes no direct
provider requests and has no client retries. Sources and drafts reach the service's
configured provider. The four-call ceiling bounds **service requests**, not internal
provider retries, tokens or dollars. `--max-tokens` is a requested per-response cap.
The service may apply profile settings. Actual model, usage, finish reason and
selected routing metadata are retained in receipts and artifact provenance; cost
remains unknown unless independently measured. Empty or incomplete responses stop
the workflow and retain diagnostic metadata. Successful runs remain unreviewed.

5. Review `artifacts/generated/claims.md`, `candidates.md`, `relation-proposals.md`
   and `output/evaluations/source-review.md`. The review receives original source
   text plus all previous outputs. It is an assisted self-review; a successful
   run never marks it independently reviewed.
6. Update the source/candidate catalogs and question ledger after review. Use the
   [admission procedure](Admission.md) before copying anything to `patterns/`.
   Then compare the two source investigations in VC-01/VC-02 walkthroughs.

## Execution and evidence boundaries

- One source per workspace. The inspected upstream runner keeps one intermediate
  result per stage, not per source. The adapter refuses multiple sources so stages
  cannot mix documents. Cross-source synthesis is a subsequent explicit review.
- Workflow order: claims → candidates → relation proposals → critique. Each stage
  retains the original source. Empty candidate/relation proposals are valid findings.
- Candidates and relations remain generated Markdown proposals. Native bench
  entity splitting requires at least one candidate and would reject legitimate
  negative results. Typed graph admission therefore follows review, not generation.
- Source digests are checked before running. Workflow/prompt digests and a start
  receipt are written before any call; failures remain visible. Reusing an attempted
  workspace is refused. Make a new revision workspace instead of overwriting results.
- Use this adapter's `plan` and `run` for the custom workflow. The generic `generate`
  pipeline selects its own workflow names; native dry-run rendering cannot resolve
  downstream references to assisted outputs that have not been generated yet.
- Bench `validate` checks its registered artifact contracts. Generated research
  proposals have no automatic semantic validation here. Structural metrics and an
  empty validation error list do not prove evidence quality or pattern validity.
- Fixture responses are allowed only for synthetic sources through the CLI.
  Fixture provenance remains visible in manifests and run records.

## Current source queue

| Investigation | Input readiness | Work record |
| --- | --- | --- |
| Carniato first source case | Manual review plus completed llm-connect draft; admission rejected | FEP-WP-0002-T02 |
| Contrasting local-first source | Manual plus completed bounded live review; no candidates justified | FEP-WP-0002-T02 |
| VC-01 and VC-02 synthesis | Depends on reviewed source claims and candidate patterns | FEP-WP-0002-T03 |

Do not ingest the existing Carniato research stub as if it were the original article.


Primary-source plans (four calls each; each run failed on its first call):

```bash
../infospace-bench/.venv/bin/python scripts/research_bench.py plan research/bench/infospaces/carniato-authority-01
../infospace-bench/.venv/bin/python scripts/research_bench.py plan research/bench/infospaces/local-first-authority-01
```

These short packets isolate the first comparison question. They do not substitute
for full source review, independent known uses or the two architecture walkthroughs.


## Live pilot outcome — 2026-09-24

Both primary-source runs attempted Gemini 2.5 Flash through OpenRouter and failed
before producing research artifacts. A separate credential probe returned HTTP 401.
Their failed receipts remain immutable. Read [AuthorityPilot](../reviews/AuthorityPilot.md)
for the manual review that continued independently. Subsequent live runs use the llm-connect service and fresh workspace slugs;
no workstation provider credential is required.

Failure receipts now capture adapter/model and safe structured diagnostic fields;
raw provider error bodies, headers and credential values are never persisted.
The initial failed receipts predate this improvement; the review records their model.


## llm-connect service pilot — 2026-09-24

Live execution now uses llm-connect exclusively. The service's
`custodian-triage-balanced` profile resolved to `google/gemini-2.5-flash`:

| Workspace | Outcome | Service requests | Reported total tokens |
| --- | --- | --- | --- |
| carniato-authority-02 | Claims truncated at 4,000 completion tokens | 1 | 4,470 |
| carniato-authority-03 | Four stages completed; unreviewed generated artifacts | 4 | 12,638 |
| local-first-authority-02 | Claims truncated at 12,000 completion tokens | 1 | 12,489 |

All receipts are retained. Token totals include failed responses and are not dollar
costs. No additional local-first retry was made. See
[the service pilot review](../reviews/LLMConnectPilot.md) for rejected proposals and
follow-up work. The current profile is a triage profile; research suitability has
not been established. Successful service execution is not evidence acceptance.


## Bounded refinement — 2026-09-25

The current prompts bound stage output length, distinguish claim IDs from candidate
IDs, and require a JSON evidence ledger. Run `scripts/research_admission.py` before
editorial admission review (see Admission.md). Historical workspaces preserve their
original prompts and are not silently migrated.

The 2026-09-25 local-first attempt used a requested low reasoning effort and 6,000 maximum
completion tokens per response, four service requests maximum, with no client retry
or automatic rerun. This is at most 24,000 requested completion tokens across four
responses; input tokens, internal retries and dollar cost are outside that bound.
The service owns profile resolution and provider support for reasoning settings;
a recorded request is not proof the provider honored every setting. Shorter prompts
and changed reasoning settings are a combined experiment, not an isolated causal test.


That attempt (`local-first-authority-03`) truncated at claims: one request, 6,525
reported total tokens. No further retry was made. See
[refinement review](../reviews/ResearchPipelineRefinement.md) for the remaining
execution limitation and the captured-failure screening report.


## Execution diagnostics and compact output

`--diagnostics` requests llm-connect's per-call debug capture but stores only an
allowlisted summary: numeric generation settings, presence of reasoning fields,
numeric usage/cost and reasoning-token counts. It never writes raw debug bodies,
headers, prompts or internal reasoning text. The service response may contain those
fields transiently; this consumer discards them. Availability depends on the server.
Normal model-generated incomplete content is quarantined separately as
`reports/rejected-response.md`, with a hash and rejected status in the receipt;
it is not registered as a completed research artifact.

The inspected llm-connect OpenRouter adapter drops `reasoning_effort`. Live diagnostic
capture confirmed its absence at the provider boundary. Do not use that flag as a
working control for this route; it remains available for other service implementations.
The local-first diagnostic output was a table header followed by tens of thousands
of spaces. Current prompts use compact records and explicitly forbid padded tables.
See the [truncation diagnosis](../reviews/TruncationDiagnosis.md) for measured outcomes.


Local-first `-06` subsequently completed all four stages at the 3,000-token response
cap using compact records. Its empty candidate/relation outcome passes structural
screening; editorial review admits no patterns. The next work is source expansion,
not another truncation retry.