Set up source-grounded frontend research workflows with infospace-bench
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a0cffe-206f-7323-89d3-b20ca4cf4bf9
This commit is contained in:
parent
debf9e0aa0
commit
fc8a7dbb49
40 changed files with 1947 additions and 8 deletions
106
research/bench/README.md
Normal file
106
research/bench/README.md
Normal file
|
|
@ -0,0 +1,106 @@
|
|||
# Frontend research bench
|
||||
|
||||
Repository-owned workflows executed by the sibling `infospace-bench` package.
|
||||
Verified against its checkout `be872bf` on 2026-09-24. This is a source investigation
|
||||
workspace; `canon/`, `patterns/` and the research program remain authoritative.
|
||||
|
||||
## Quick start
|
||||
|
||||
The current workstation already has the bench environment installed. Override
|
||||
`BENCH_PY` with another Python interpreter where `infospace-bench`, PyYAML and
|
||||
pytest are installed. The sibling package has local dependencies on markitect-tool,
|
||||
artifact-store and llm-connect; this repo does not duplicate their installation.
|
||||
|
||||
```bash
|
||||
make bench-test
|
||||
make bench-plan
|
||||
```
|
||||
|
||||
`bench-plan` inspects the checked-in synthetic example without calling a model.
|
||||
`infospaces/fixture-smoke/` contains its completed four-stage run, prompts, fixture
|
||||
responses, provenance and review. It verifies workflow plumbing only. Its candidate
|
||||
does not count toward the research corpus, evidence or M1 acceptance.
|
||||
|
||||
To repeat the fixture in a fresh temporary workspace:
|
||||
|
||||
```bash
|
||||
BENCH_PY=../infospace-bench/.venv/bin/python
|
||||
pilot_workspace=$(mktemp -d /tmp/frontend-patterns-bench.XXXXXX)
|
||||
"$BENCH_PY" scripts/research_bench.py init research/bench/fixtures/source.md \
|
||||
--workspace "$pilot_workspace" --slug fixture
|
||||
"$BENCH_PY" scripts/research_bench.py run "$pilot_workspace/infospaces/fixture" \
|
||||
--fixture-responses research/bench/fixtures/responses.yaml
|
||||
```
|
||||
|
||||
## Investigate a real source
|
||||
|
||||
1. Obtain and review an original source using the [research protocol](../plan/ResearchProtocol.md).
|
||||
Prepare a bounded normalized Markdown excerpt with stable section/paragraph
|
||||
locators. Record any omitted context. No source acquisition or URL fetching is
|
||||
performed by this adapter.
|
||||
2. Add the metadata shown in [SourceInput](SourceInput.md). Identify the original
|
||||
author, URL/bibliographic locator, version and access date. This is required
|
||||
provenance, not a claim that the source was independently verified.
|
||||
3. Initialize a uniquely named workspace and inspect its plan:
|
||||
|
||||
```bash
|
||||
../infospace-bench/.venv/bin/python scripts/research_bench.py init /path/to/source.md \
|
||||
--slug carniato-review-01
|
||||
../infospace-bench/.venv/bin/python scripts/research_bench.py plan \
|
||||
research/bench/infospaces/carniato-review-01
|
||||
```
|
||||
|
||||
4. For a live run, configure `OPENROUTER_API_KEY` using the credential owner described
|
||||
in `AGENTS.md`, choose a model explicitly, then run:
|
||||
|
||||
```bash
|
||||
../infospace-bench/.venv/bin/python scripts/research_bench.py run \
|
||||
research/bench/infospaces/carniato-review-01 \
|
||||
--provider openrouter --model YOUR_MODEL_ID --max-calls 4
|
||||
```
|
||||
|
||||
Live use transmits the selected source and intermediate drafts to that provider.
|
||||
There is no default provider or model. The four-call ceiling is enforced before and
|
||||
during execution, with automatic OpenRouter retries disabled. It is **not a dollar
|
||||
or token budget**; actual usage metadata appears in the bench run record. No live
|
||||
provider call was made during setup.
|
||||
|
||||
5. Review `artifacts/generated/claims.md`, `candidates.md`, `relation-proposals.md`
|
||||
and `output/evaluations/source-review.md`. The review receives original source
|
||||
text plus all previous outputs. It is an assisted self-review; a successful
|
||||
run never marks it independently reviewed.
|
||||
6. Update the source/candidate catalogs and question ledger after review. Use the
|
||||
[admission procedure](Admission.md) before copying anything to `patterns/`.
|
||||
Then compare the two source investigations in VC-01/VC-02 walkthroughs.
|
||||
|
||||
## Execution and evidence boundaries
|
||||
|
||||
- One source per workspace. The inspected upstream runner keeps one intermediate
|
||||
result per stage, not per source. The adapter refuses multiple sources so stages
|
||||
cannot mix documents. Cross-source synthesis is a subsequent explicit review.
|
||||
- Workflow order: claims → candidates → relation proposals → critique. Each stage
|
||||
retains the original source. Empty candidate/relation proposals are valid findings.
|
||||
- Candidates and relations remain generated Markdown proposals. Native bench
|
||||
entity splitting requires at least one candidate and would reject legitimate
|
||||
negative results. Typed graph admission therefore follows review, not generation.
|
||||
- Source digests are checked before running. Workflow/prompt digests and a start
|
||||
receipt are written before any call; failures remain visible. Reusing an attempted
|
||||
workspace is refused. Make a new revision workspace instead of overwriting results.
|
||||
- Use this adapter's `plan` and `run` for the custom workflow. The generic `generate`
|
||||
pipeline selects its own workflow names; native dry-run rendering cannot resolve
|
||||
downstream references to assisted outputs that have not been generated yet.
|
||||
- Bench `validate` checks its registered artifact contracts. Generated research
|
||||
proposals have no automatic semantic validation here. Structural metrics and an
|
||||
empty validation error list do not prove evidence quality or pattern validity.
|
||||
- Fixture responses are allowed only for synthetic sources through the CLI.
|
||||
Fixture provenance remains visible in manifests and run records.
|
||||
|
||||
## Current source queue
|
||||
|
||||
| Investigation | Input readiness | Work record |
|
||||
| --- | --- | --- |
|
||||
| Carniato first source case | Original source and bounded normalized excerpt still needed | FEP-WP-0002-T02 |
|
||||
| Contrasting local-first source | Primary source selection and excerpt still needed | FEP-WP-0002-T02 |
|
||||
| VC-01 and VC-02 synthesis | Depends on reviewed source claims and candidate patterns | FEP-WP-0002-T03 |
|
||||
|
||||
Do not ingest the existing Carniato research stub as if it were the original article.
|
||||
Loading…
Add table
Add a link
Reference in a new issue