frontend-patterns/research/bench/README.md

# Frontend research bench

Repository-owned source investigations executed through infospace-bench. New
workspaces use `conditional-v1`; historical workspaces retain their recorded policy.
Live calls use llm-connect exclusively. Canonical admission remains editorial.

## Environment and checks

`BENCH_PY` defaults to `../infospace-bench/.venv/bin/python`. It needs infospace-bench,
PyYAML, pytest and llm-connect (`uv pip install --python "$BENCH_PY" -e ../llm-connect`
if the local package is missing). Run `make bench-test schema-test bench-plan`.
The final command inspects the historical synthetic fixture without calling a model.

## A question-driven investigation

1. State the unresolved question, hypothesis and possible counterexample in the source
   packet's editorial section. Consult the walkthrough's evidence gaps before collecting
   more sources. Follow [SourceInput](SourceInput.md); summaries are not original evidence.
2. Initialize one immutable source per workspace, then inspect the plan:

```bash
../infospace-bench/.venv/bin/python scripts/research_bench.py init /path/to/source.md --slug question-01
../infospace-bench/.venv/bin/python scripts/research_bench.py plan research/bench/infospaces/question-01
```

3. For live execution, keep a service port-forward running in a separate terminal:

```bash
kubectl -n activity-core port-forward svc/llm-connect 18081:8080
```

```bash
../infospace-bench/.venv/bin/python scripts/research_bench.py run \
  research/bench/infospaces/question-01 \
  --llm-connect-url http://127.0.0.1:18081 --model custodian-triage-balanced \
  --diagnostics --max-tokens 3000 --max-calls 3
```

The service owns provider credentials/routing. The model argument may name a runtime
profile; actual model is recorded. Sources/drafts reach its provider. The inspected
OpenRouter route drops `reasoning_effort`; it is not a working control for that route.
A service-request ceiling does not bound internal retries or dollars. Token caps are
requested per response, not a total input-plus-output budget.

## Conditional execution

The copied `profile/execution.json` opts new workspaces into this policy:

- Claims and candidates use compact source-grounded model outputs. Candidate evidence
  ledgers must pass structural screening before further work.
- With fewer than two candidates, the adapter writes an empty relation array with
  deterministic provenance and makes no relation call.
- With multiple candidates, one model call proposes relations. Endpoints must resolve
  to candidate IDs and every edge must satisfy the structural contract. Invalid
  proposals are quarantined in `reports/rejected-stage.json`; execution stops.
- The historical stage ID `critique` now writes a deterministic screen and editorial
  checklist to `output/evaluations/source-review.md`. No automatic self-critique call.
  This compatibility filename does not mean editorial review has happened.

Successful uncached runs therefore use two or three adapter calls, compared with
four historically. `plan.provider_calls` is the uncached upper bound; the minimum
assumes successful execution without reuse. A lower `--max-calls` is enforced during
execution: if a needed call exceeds it, the attempted workspace fails and remains
immutable. Nothing automatically retries, promotes artifacts, or marks them reviewed.

## Explicit reuse

Create a fresh workspace for the same source and use `--reuse-from`:

```bash
../infospace-bench/.venv/bin/python scripts/research_bench.py run \
  research/bench/infospaces/question-02 \
  --llm-connect-url http://127.0.0.1:18081 --model custodian-triage-balanced \
  --diagnostics --max-tokens 3000 --max-calls 0 \
  --reuse-from research/bench/infospaces/question-01
```

Only completed conditional runs with a matching cache-file digest are eligible.
Keys include source hash, every profile file, workflow, rendered prompt, stage,
input ID, adapter/settings and execution/screen code hashes. Changed inputs/settings
miss the cache. A zero-call ceiling prevents paid fallback. Legacy receipts without
a stage cache cannot be replayed this way; cite their artifacts directly instead.

Reuse is an explicit replay of recorded evidence, not a fresh provider observation.
Remote profile changes cannot be detected from a local fingerprint. Omit reuse when
a new model/profile observation is the research question. Reused artifacts carry
`reused_from` and `current_provider_call: false`; their original usage remains historical
provenance. Use the research receipt's current-call/current-token totals for cost
accounting, not a sum over reused artifact metadata.

## Review and cycle metrics

Read `reports/research-run.json`: actual attempted calls, cache hits, deterministic
stages, safe response usage/diagnostics, and structural rejection counts. Live
`reported_tokens_this_run` excludes reused usage; missing provider usage is not a
billing measurement. `rejected_assertions` counts structural findings, not a semantic
truth score. `editorial_corrections: null` means not yet assessed, never zero corrections.

Record a named editorial review using [ResearchReviewMetrics](../../templates/ResearchReviewMetrics.yaml),
with individual rejected assertions and corrections linked to artifacts. Check source
fidelity, scope, known-use independence, counterexamples and applicability. Apply
[Admission](Admission.md) before moving any proposal into canonical patterns.

`--diagnostics` retains only allowlisted numeric settings/usage and reasoning-field
presence. Raw debug bodies, headers and internal reasoning text are discarded.
Incomplete normal generated text is quarantined in `reports/rejected-response.md`,
with digest and rejection status. Immutable receipts preserve failures.

## Verified examples and limitations

- `fixture-smoke`: historical four-stage synthetic execution, never research evidence.
- `conditional-smoke-01`: two fixture calls plus deterministic relations/review.
- `conditional-reuse-01`: same inputs reused with zero new calls. These demonstrate
  routing and provenance, not measured live token savings or improved research quality.
- [Truncation diagnosis](../reviews/TruncationDiagnosis.md): live compact-output repair.
- [Implementation evidence](../reviews/LocalFirstEvidenceExpansion.md): completed Joplin
  and Obsidian investigations; the latter's invalid relations were rejected editorially.
- [Optimization review](../reviews/ResearchOptimization.md): policy checks and VC-02 result.

One source per workspace remains necessary because upstream stage outputs are keyed
by stage rather than source. Cross-source synthesis is separate editorial work.
Structural screening cannot establish source truth, independent recurrence, durability
or semantic validity. No live model call is needed merely to rephrase unchanged evidence.