frontend-patterns/research/bench/README.md

# Frontend research bench

Repository-owned workflows executed by the sibling `infospace-bench` package.
Verified against its checkout `be872bf` on 2026-09-24. This is a source investigation
workspace; `canon/`, `patterns/` and the research program remain authoritative.

## Quick start

The current workstation already has the bench environment installed. Override
`BENCH_PY` with another Python interpreter where `infospace-bench`, PyYAML and
pytest are installed. The sibling package has local dependencies on markitect-tool,
artifact-store and llm-connect; this repo does not duplicate their installation.

```bash
make bench-test
make bench-plan
```

`bench-plan` inspects the checked-in synthetic example without calling a model.
`infospaces/fixture-smoke/` contains its completed four-stage run, prompts, fixture
responses, provenance and review. It verifies workflow plumbing only. Its candidate
does not count toward the research corpus, evidence or M1 acceptance.

To repeat the fixture in a fresh temporary workspace:

```bash
BENCH_PY=../infospace-bench/.venv/bin/python
pilot_workspace=$(mktemp -d /tmp/frontend-patterns-bench.XXXXXX)
"$BENCH_PY" scripts/research_bench.py init research/bench/fixtures/source.md \
  --workspace "$pilot_workspace" --slug fixture
"$BENCH_PY" scripts/research_bench.py run "$pilot_workspace/infospaces/fixture" \
  --fixture-responses research/bench/fixtures/responses.yaml
```

## Investigate a real source

1. Obtain and review an original source using the [research protocol](../plan/ResearchProtocol.md).
   Prepare a bounded normalized Markdown excerpt with stable section/paragraph
   locators. Record any omitted context. No source acquisition or URL fetching is
   performed by this adapter.
2. Add the metadata shown in [SourceInput](SourceInput.md). Identify the original
   author, URL/bibliographic locator, version and access date. This is required
   provenance, not a claim that the source was independently verified.
3. Initialize a uniquely named workspace and inspect its plan:

```bash
../infospace-bench/.venv/bin/python scripts/research_bench.py init /path/to/source.md \
  --slug carniato-review-01
../infospace-bench/.venv/bin/python scripts/research_bench.py plan \
  research/bench/infospaces/carniato-review-01
```

4. For a live run, configure `OPENROUTER_API_KEY` using the credential owner described
   in `AGENTS.md`, choose a model explicitly, then run:

```bash
../infospace-bench/.venv/bin/python scripts/research_bench.py run \
  research/bench/infospaces/carniato-review-01 \
  --provider openrouter --model YOUR_MODEL_ID --max-calls 4
```

Live use transmits the selected source and intermediate drafts to that provider.
There is no default provider or model. The four-call ceiling is enforced before and
during execution, with automatic OpenRouter retries disabled. It is **not a dollar
or token budget**; actual usage metadata appears in the bench run record. No live
provider call was made during setup.

5. Review `artifacts/generated/claims.md`, `candidates.md`, `relation-proposals.md`
   and `output/evaluations/source-review.md`. The review receives original source
   text plus all previous outputs. It is an assisted self-review; a successful
   run never marks it independently reviewed.
6. Update the source/candidate catalogs and question ledger after review. Use the
   [admission procedure](Admission.md) before copying anything to `patterns/`.
   Then compare the two source investigations in VC-01/VC-02 walkthroughs.

## Execution and evidence boundaries

- One source per workspace. The inspected upstream runner keeps one intermediate
  result per stage, not per source. The adapter refuses multiple sources so stages
  cannot mix documents. Cross-source synthesis is a subsequent explicit review.
- Workflow order: claims → candidates → relation proposals → critique. Each stage
  retains the original source. Empty candidate/relation proposals are valid findings.
- Candidates and relations remain generated Markdown proposals. Native bench
  entity splitting requires at least one candidate and would reject legitimate
  negative results. Typed graph admission therefore follows review, not generation.
- Source digests are checked before running. Workflow/prompt digests and a start
  receipt are written before any call; failures remain visible. Reusing an attempted
  workspace is refused. Make a new revision workspace instead of overwriting results.
- Use this adapter's `plan` and `run` for the custom workflow. The generic `generate`
  pipeline selects its own workflow names; native dry-run rendering cannot resolve
  downstream references to assisted outputs that have not been generated yet.
- Bench `validate` checks its registered artifact contracts. Generated research
  proposals have no automatic semantic validation here. Structural metrics and an
  empty validation error list do not prove evidence quality or pattern validity.
- Fixture responses are allowed only for synthetic sources through the CLI.
  Fixture provenance remains visible in manifests and run records.

## Current source queue

| Investigation | Input readiness | Work record |
| --- | --- | --- |
| Carniato first source case | Initial manual review recorded; provider attempt failed | FEP-WP-0002-T02 |
| Contrasting local-first source | Initial manual review recorded; provider attempt failed | FEP-WP-0002-T02 |
| VC-01 and VC-02 synthesis | Depends on reviewed source claims and candidate patterns | FEP-WP-0002-T03 |

Do not ingest the existing Carniato research stub as if it were the original article.


Primary-source plans (four calls each; each run failed on its first call):

```bash
../infospace-bench/.venv/bin/python scripts/research_bench.py plan research/bench/infospaces/carniato-authority-01
../infospace-bench/.venv/bin/python scripts/research_bench.py plan research/bench/infospaces/local-first-authority-01
```

These short packets isolate the first comparison question. They do not substitute
for full source review, independent known uses or the two architecture walkthroughs.


## Live pilot outcome — 2026-09-24

Both primary-source runs attempted Gemini 2.5 Flash through OpenRouter and failed
before producing research artifacts. A separate credential probe returned HTTP 401.
Their failed receipts remain immutable. Read [AuthorityPilot](../reviews/AuthorityPilot.md)
for the manual review that continued independently. New runs require a working
credential authorized for this client and fresh workspace slugs.

Failure receipts now capture adapter/model and safe structured diagnostic fields;
raw provider error bodies, headers and credential values are never persisted.
The initial failed receipts predate this improvement; the review records their model.