frontend-patterns/research/bench/README.md

Frontend research bench

Repository-owned workflows executed by the sibling infospace-bench package. Verified against its checkout be872bf on 2026-09-24. This is a source investigation workspace; canon/, patterns/ and the research program remain authoritative.

Quick start

The current workstation already has the bench environment installed. Override BENCH_PY with another Python interpreter where infospace-bench, PyYAML and pytest and llm-connect are installed. If needed: uv pip install --python "$BENCH_PY" -e ../llm-connect. The sibling package has local dependencies on markitect-tool, artifact-store and llm-connect; this repo does not duplicate their installation.

make bench-test
make bench-plan

bench-plan inspects the checked-in synthetic example without calling a model. infospaces/fixture-smoke/ contains its completed four-stage run, prompts, fixture responses, provenance and review. It verifies workflow plumbing only. Its candidate does not count toward the research corpus, evidence or M1 acceptance.

To repeat the fixture in a fresh temporary workspace:

BENCH_PY=../infospace-bench/.venv/bin/python
pilot_workspace=$(mktemp -d /tmp/frontend-patterns-bench.XXXXXX)
"$BENCH_PY" scripts/research_bench.py init research/bench/fixtures/source.md \
  --workspace "$pilot_workspace" --slug fixture
"$BENCH_PY" scripts/research_bench.py run "$pilot_workspace/infospaces/fixture" \
  --fixture-responses research/bench/fixtures/responses.yaml

Investigate a real source

1. Obtain and review an original source using the research protocol. Prepare a bounded normalized Markdown excerpt with stable section/paragraph locators. Record any omitted context. No source acquisition or URL fetching is performed by this adapter.
2. Add the metadata shown in SourceInput. Identify the original author, URL/bibliographic locator, version and access date. This is required provenance, not a claim that the source was independently verified.
3. Initialize a uniquely named workspace and inspect its plan:

../infospace-bench/.venv/bin/python scripts/research_bench.py init /path/to/source.md \
  --slug carniato-review-01
../infospace-bench/.venv/bin/python scripts/research_bench.py plan \
  research/bench/infospaces/carniato-review-01

4. Use the deployed llm-connect service. On this workstation, forward its port in a separate terminal (leave it running for the investigation):

kubectl -n activity-core port-forward svc/llm-connect 18081:8080

Then run in a fresh workspace:

../infospace-bench/.venv/bin/python scripts/research_bench.py run \
  research/bench/infospaces/carniato-review-01 \
  --llm-connect-url http://127.0.0.1:18081 \
  --model custodian-triage-balanced --diagnostics \
  --max-tokens 3000 --max-calls 4

In-cluster consumers can use http://llm-connect.activity-core.svc.cluster.local:8080. The service owns credentials and provider routing. --model names a service runtime profile or requests a model; the resolved model is recorded from the response. The bench uses llm-connect's RunConfig and /execute contract. It makes no direct provider requests and has no client retries. Sources and drafts reach the service's configured provider. The four-call ceiling bounds service requests, not internal provider retries, tokens or dollars. --max-tokens is a requested per-response cap. The service may apply profile settings. Actual model, usage, finish reason and selected routing metadata are retained in receipts and artifact provenance; cost remains unknown unless independently measured. Empty or incomplete responses stop the workflow and retain diagnostic metadata. Successful runs remain unreviewed.

5. Review artifacts/generated/claims.md, candidates.md, relation-proposals.md and output/evaluations/source-review.md. The review receives original source text plus all previous outputs. It is an assisted self-review; a successful run never marks it independently reviewed.
6. Update the source/candidate catalogs and question ledger after review. Use the admission procedure before copying anything to patterns/. Then compare the two source investigations in VC-01/VC-02 walkthroughs.

Execution and evidence boundaries

• One source per workspace. The inspected upstream runner keeps one intermediate result per stage, not per source. The adapter refuses multiple sources so stages cannot mix documents. Cross-source synthesis is a subsequent explicit review.
• Workflow order: claims → candidates → relation proposals → critique. Each stage retains the original source. Empty candidate/relation proposals are valid findings.
• Candidates and relations remain generated Markdown proposals. Native bench entity splitting requires at least one candidate and would reject legitimate negative results. Typed graph admission therefore follows review, not generation.
• Source digests are checked before running. Workflow/prompt digests and a start receipt are written before any call; failures remain visible. Reusing an attempted workspace is refused. Make a new revision workspace instead of overwriting results.
• Use this adapter's plan and run for the custom workflow. The generic generate pipeline selects its own workflow names; native dry-run rendering cannot resolve downstream references to assisted outputs that have not been generated yet.
• Bench validate checks its registered artifact contracts. Generated research proposals have no automatic semantic validation here. Structural metrics and an empty validation error list do not prove evidence quality or pattern validity.
• Fixture responses are allowed only for synthetic sources through the CLI. Fixture provenance remains visible in manifests and run records.

Current source queue

Investigation	Input readiness	Work record
Carniato first source case	Manual review plus completed llm-connect draft; admission rejected	FEP-WP-0002-T02
Contrasting local-first source	Manual plus completed bounded live review; no candidates justified	FEP-WP-0002-T02
VC-01 and VC-02 synthesis	Depends on reviewed source claims and candidate patterns	FEP-WP-0002-T03

Do not ingest the existing Carniato research stub as if it were the original article.

Primary-source plans (four calls each; each run failed on its first call):

../infospace-bench/.venv/bin/python scripts/research_bench.py plan research/bench/infospaces/carniato-authority-01
../infospace-bench/.venv/bin/python scripts/research_bench.py plan research/bench/infospaces/local-first-authority-01

These short packets isolate the first comparison question. They do not substitute for full source review, independent known uses or the two architecture walkthroughs.

Live pilot outcome — 2026-09-24

Both primary-source runs attempted Gemini 2.5 Flash through OpenRouter and failed before producing research artifacts. A separate credential probe returned HTTP 401. Their failed receipts remain immutable. Read AuthorityPilot for the manual review that continued independently. Subsequent live runs use the llm-connect service and fresh workspace slugs; no workstation provider credential is required.

Failure receipts now capture adapter/model and safe structured diagnostic fields; raw provider error bodies, headers and credential values are never persisted. The initial failed receipts predate this improvement; the review records their model.

llm-connect service pilot — 2026-09-24

Live execution now uses llm-connect exclusively. The service's custodian-triage-balanced profile resolved to google/gemini-2.5-flash:

Workspace	Outcome	Service requests	Reported total tokens
carniato-authority-02	Claims truncated at 4,000 completion tokens	1	4,470
carniato-authority-03	Four stages completed; unreviewed generated artifacts	4	12,638
local-first-authority-02	Claims truncated at 12,000 completion tokens	1	12,489

All receipts are retained. Token totals include failed responses and are not dollar costs. No additional local-first retry was made. See the service pilot review for rejected proposals and follow-up work. The current profile is a triage profile; research suitability has not been established. Successful service execution is not evidence acceptance.

Bounded refinement — 2026-09-25

The current prompts bound stage output length, distinguish claim IDs from candidate IDs, and require a JSON evidence ledger. Run scripts/research_admission.py before editorial admission review (see Admission.md). Historical workspaces preserve their original prompts and are not silently migrated.

The 2026-09-25 local-first attempt used a requested low reasoning effort and 6,000 maximum completion tokens per response, four service requests maximum, with no client retry or automatic rerun. This is at most 24,000 requested completion tokens across four responses; input tokens, internal retries and dollar cost are outside that bound. The service owns profile resolution and provider support for reasoning settings; a recorded request is not proof the provider honored every setting. Shorter prompts and changed reasoning settings are a combined experiment, not an isolated causal test.

That attempt (local-first-authority-03) truncated at claims: one request, 6,525 reported total tokens. No further retry was made. See refinement review for the remaining execution limitation and the captured-failure screening report.

Execution diagnostics and compact output

--diagnostics requests llm-connect's per-call debug capture but stores only an allowlisted summary: numeric generation settings, presence of reasoning fields, numeric usage/cost and reasoning-token counts. It never writes raw debug bodies, headers, prompts or internal reasoning text. The service response may contain those fields transiently; this consumer discards them. Availability depends on the server. Normal model-generated incomplete content is quarantined separately as reports/rejected-response.md, with a hash and rejected status in the receipt; it is not registered as a completed research artifact.

The inspected llm-connect OpenRouter adapter drops reasoning_effort. Live diagnostic capture confirmed its absence at the provider boundary. Do not use that flag as a working control for this route; it remains available for other service implementations. The local-first diagnostic output was a table header followed by tens of thousands of spaces. Current prompts use compact records and explicitly forbid padded tables. See the truncation diagnosis for measured outcomes.

Local-first -06 subsequently completed all four stages at the 3,000-token response cap using compact records. Its empty candidate/relation outcome passes structural screening; editorial review admits no patterns. The next work is source expansion, not another truncation retry.