frontend-patterns/research/bench

Frontend research bench

Repository-owned workflows executed by the sibling infospace-bench package. Verified against its checkout be872bf on 2026-09-24. This is a source investigation workspace; canon/, patterns/ and the research program remain authoritative.

Quick start

The current workstation already has the bench environment installed. Override BENCH_PY with another Python interpreter where infospace-bench, PyYAML and pytest are installed. The sibling package has local dependencies on markitect-tool, artifact-store and llm-connect; this repo does not duplicate their installation.

make bench-test
make bench-plan

bench-plan inspects the checked-in synthetic example without calling a model. infospaces/fixture-smoke/ contains its completed four-stage run, prompts, fixture responses, provenance and review. It verifies workflow plumbing only. Its candidate does not count toward the research corpus, evidence or M1 acceptance.

To repeat the fixture in a fresh temporary workspace:

BENCH_PY=../infospace-bench/.venv/bin/python
pilot_workspace=$(mktemp -d /tmp/frontend-patterns-bench.XXXXXX)
"$BENCH_PY" scripts/research_bench.py init research/bench/fixtures/source.md \
  --workspace "$pilot_workspace" --slug fixture
"$BENCH_PY" scripts/research_bench.py run "$pilot_workspace/infospaces/fixture" \
  --fixture-responses research/bench/fixtures/responses.yaml

Investigate a real source

1. Obtain and review an original source using the research protocol. Prepare a bounded normalized Markdown excerpt with stable section/paragraph locators. Record any omitted context. No source acquisition or URL fetching is performed by this adapter.
2. Add the metadata shown in SourceInput. Identify the original author, URL/bibliographic locator, version and access date. This is required provenance, not a claim that the source was independently verified.
3. Initialize a uniquely named workspace and inspect its plan:

../infospace-bench/.venv/bin/python scripts/research_bench.py init /path/to/source.md \
  --slug carniato-review-01
../infospace-bench/.venv/bin/python scripts/research_bench.py plan \
  research/bench/infospaces/carniato-review-01

4. For a live run, configure OPENROUTER_API_KEY using the credential owner described in AGENTS.md, choose a model explicitly, then run:

../infospace-bench/.venv/bin/python scripts/research_bench.py run \
  research/bench/infospaces/carniato-review-01 \
  --provider openrouter --model YOUR_MODEL_ID --max-calls 4

Live use transmits the selected source and intermediate drafts to that provider. There is no default provider or model. The four-call ceiling is enforced before and during execution, with automatic OpenRouter retries disabled. It is not a dollar or token budget; actual usage metadata appears in the bench run record. No live provider call was made during setup.

5. Review artifacts/generated/claims.md, candidates.md, relation-proposals.md and output/evaluations/source-review.md. The review receives original source text plus all previous outputs. It is an assisted self-review; a successful run never marks it independently reviewed.
6. Update the source/candidate catalogs and question ledger after review. Use the admission procedure before copying anything to patterns/. Then compare the two source investigations in VC-01/VC-02 walkthroughs.

Execution and evidence boundaries

• One source per workspace. The inspected upstream runner keeps one intermediate result per stage, not per source. The adapter refuses multiple sources so stages cannot mix documents. Cross-source synthesis is a subsequent explicit review.
• Workflow order: claims → candidates → relation proposals → critique. Each stage retains the original source. Empty candidate/relation proposals are valid findings.
• Candidates and relations remain generated Markdown proposals. Native bench entity splitting requires at least one candidate and would reject legitimate negative results. Typed graph admission therefore follows review, not generation.
• Source digests are checked before running. Workflow/prompt digests and a start receipt are written before any call; failures remain visible. Reusing an attempted workspace is refused. Make a new revision workspace instead of overwriting results.
• Use this adapter's plan and run for the custom workflow. The generic generate pipeline selects its own workflow names; native dry-run rendering cannot resolve downstream references to assisted outputs that have not been generated yet.
• Bench validate checks its registered artifact contracts. Generated research proposals have no automatic semantic validation here. Structural metrics and an empty validation error list do not prove evidence quality or pattern validity.
• Fixture responses are allowed only for synthetic sources through the CLI. Fixture provenance remains visible in manifests and run records.

Current source queue

Investigation	Input readiness	Work record
Carniato first source case	Initial manual review recorded; provider attempt failed	FEP-WP-0002-T02
Contrasting local-first source	Initial manual review recorded; provider attempt failed	FEP-WP-0002-T02
VC-01 and VC-02 synthesis	Depends on reviewed source claims and candidate patterns	FEP-WP-0002-T03

Do not ingest the existing Carniato research stub as if it were the original article.

Primary-source plans (four calls each; each run failed on its first call):

../infospace-bench/.venv/bin/python scripts/research_bench.py plan research/bench/infospaces/carniato-authority-01
../infospace-bench/.venv/bin/python scripts/research_bench.py plan research/bench/infospaces/local-first-authority-01

These short packets isolate the first comparison question. They do not substitute for full source review, independent known uses or the two architecture walkthroughs.

Live pilot outcome — 2026-09-24

Both primary-source runs attempted Gemini 2.5 Flash through OpenRouter and failed before producing research artifacts. A separate credential probe returned HTTP 401. Their failed receipts remain immutable. Read AuthorityPilot for the manual review that continued independently. New runs require a working credential authorized for this client and fresh workspace slugs.

Failure receipts now capture adapter/model and safe structured diagnostic fields; raw provider error bodies, headers and credential values are never persisted. The initial failed receipts predate this improvement; the review records their model.