frontend-patterns/research/bench

Frontend research bench

Repository-owned source investigations executed through infospace-bench. New workspaces use conditional-v1; historical workspaces retain their recorded policy. Live calls use llm-connect exclusively. Canonical admission remains editorial.

Environment and checks

BENCH_PY defaults to ../infospace-bench/.venv/bin/python. It needs infospace-bench, PyYAML, pytest and llm-connect (uv pip install --python "$BENCH_PY" -e ../llm-connect if the local package is missing). Run make bench-test schema-test bench-plan. The final command inspects the historical synthetic fixture without calling a model.

A question-driven investigation

1. State the unresolved question, hypothesis and possible counterexample in the source packet's editorial section. Consult the walkthrough's evidence gaps before collecting more sources. Follow SourceInput; summaries are not original evidence.
2. Initialize one immutable source per workspace, then inspect the plan:

../infospace-bench/.venv/bin/python scripts/research_bench.py init /path/to/source.md --slug question-01
../infospace-bench/.venv/bin/python scripts/research_bench.py plan research/bench/infospaces/question-01

3. For live execution, keep a service port-forward running in a separate terminal:

kubectl -n activity-core port-forward svc/llm-connect 18081:8080

../infospace-bench/.venv/bin/python scripts/research_bench.py run \
  research/bench/infospaces/question-01 \
  --llm-connect-url http://127.0.0.1:18081 --model custodian-triage-balanced \
  --diagnostics --max-tokens 3000 --max-calls 3

The service owns provider credentials/routing. The model argument may name a runtime profile; actual model is recorded. Sources/drafts reach its provider. The inspected OpenRouter route drops reasoning_effort; it is not a working control for that route. A service-request ceiling does not bound internal retries or dollars. Token caps are requested per response, not a total input-plus-output budget.

Conditional execution

The copied profile/execution.json opts new workspaces into this policy:

• Claims and candidates use compact source-grounded model outputs. Candidate evidence ledgers must pass structural screening before further work.
• With fewer than two candidates, the adapter writes an empty relation array with deterministic provenance and makes no relation call.
• With multiple candidates, one model call proposes relations. Endpoints must resolve to candidate IDs and every edge must satisfy the structural contract. Invalid proposals are quarantined in reports/rejected-stage.json; execution stops.
• The historical stage ID critique now writes a deterministic screen and editorial checklist to output/evaluations/source-review.md. No automatic self-critique call. This compatibility filename does not mean editorial review has happened.

Successful uncached runs therefore use two or three adapter calls, compared with four historically. plan.provider_calls is the uncached upper bound; the minimum assumes successful execution without reuse. A lower --max-calls is enforced during execution: if a needed call exceeds it, the attempted workspace fails and remains immutable. Nothing automatically retries, promotes artifacts, or marks them reviewed.

Explicit reuse

Create a fresh workspace for the same source and use --reuse-from:

../infospace-bench/.venv/bin/python scripts/research_bench.py run \
  research/bench/infospaces/question-02 \
  --llm-connect-url http://127.0.0.1:18081 --model custodian-triage-balanced \
  --diagnostics --max-tokens 3000 --max-calls 0 \
  --reuse-from research/bench/infospaces/question-01

Only completed conditional runs with a matching cache-file digest are eligible. Keys include source hash, every profile file, workflow, rendered prompt, stage, input ID, adapter/settings and execution/screen code hashes. Changed inputs/settings miss the cache. A zero-call ceiling prevents paid fallback. Legacy receipts without a stage cache cannot be replayed this way; cite their artifacts directly instead.

Reuse is an explicit replay of recorded evidence, not a fresh provider observation. Remote profile changes cannot be detected from a local fingerprint. Omit reuse when a new model/profile observation is the research question. Reused artifacts carry reused_from and current_provider_call: false; their original usage remains historical provenance. Use the research receipt's current-call/current-token totals for cost accounting, not a sum over reused artifact metadata.

Review and cycle metrics

Read reports/research-run.json: actual attempted calls, cache hits, deterministic stages, safe response usage/diagnostics, and structural rejection counts. Live reported_tokens_this_run excludes reused usage; missing provider usage is not a billing measurement. rejected_assertions counts structural findings, not a semantic truth score. editorial_corrections: null means not yet assessed, never zero corrections.

Record a named editorial review using ResearchReviewMetrics, with individual rejected assertions and corrections linked to artifacts. Check source fidelity, scope, known-use independence, counterexamples and applicability. Apply Admission before moving any proposal into canonical patterns.

--diagnostics retains only allowlisted numeric settings/usage and reasoning-field presence. Raw debug bodies, headers and internal reasoning text are discarded. Incomplete normal generated text is quarantined in reports/rejected-response.md, with digest and rejection status. Immutable receipts preserve failures.

Verified examples and limitations

• fixture-smoke: historical four-stage synthetic execution, never research evidence.
• conditional-smoke-01: two fixture calls plus deterministic relations/review.
• conditional-reuse-01: same inputs reused with zero new calls. These demonstrate routing and provenance, not measured live token savings or improved research quality.
• Truncation diagnosis: live compact-output repair.
• Implementation evidence: completed Joplin and Obsidian investigations; the latter's invalid relations were rejected editorially.
• Optimization review: policy checks and VC-02 result.

One source per workspace remains necessary because upstream stage outputs are keyed by stage rather than source. Cross-source synthesis is separate editorial work. Structural screening cannot establish source truth, independent recurrence, durability or semantic validity. No live model call is needed merely to rephrase unchanged evidence.