Reduce research calls with screened reuse and draft VC-02 walkthrough
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a0cffe-206f-7323-89d3-b20ca4cf4bf9
This commit is contained in:
parent
ad6f8fa600
commit
cfa249ab13
49 changed files with 3284 additions and 207 deletions
|
|
@ -1,220 +1,123 @@
|
|||
# Frontend research bench
|
||||
|
||||
Repository-owned workflows executed by the sibling `infospace-bench` package.
|
||||
Verified against its checkout `be872bf` on 2026-09-24. This is a source investigation
|
||||
workspace; `canon/`, `patterns/` and the research program remain authoritative.
|
||||
Repository-owned source investigations executed through infospace-bench. New
|
||||
workspaces use `conditional-v1`; historical workspaces retain their recorded policy.
|
||||
Live calls use llm-connect exclusively. Canonical admission remains editorial.
|
||||
|
||||
## Quick start
|
||||
## Environment and checks
|
||||
|
||||
The current workstation already has the bench environment installed. Override
|
||||
`BENCH_PY` with another Python interpreter where `infospace-bench`, PyYAML and
|
||||
pytest and llm-connect are installed.
|
||||
If needed: `uv pip install --python "$BENCH_PY" -e ../llm-connect`. The sibling package has local dependencies on markitect-tool,
|
||||
artifact-store and llm-connect; this repo does not duplicate their installation.
|
||||
`BENCH_PY` defaults to `../infospace-bench/.venv/bin/python`. It needs infospace-bench,
|
||||
PyYAML, pytest and llm-connect (`uv pip install --python "$BENCH_PY" -e ../llm-connect`
|
||||
if the local package is missing). Run `make bench-test schema-test bench-plan`.
|
||||
The final command inspects the historical synthetic fixture without calling a model.
|
||||
|
||||
## A question-driven investigation
|
||||
|
||||
1. State the unresolved question, hypothesis and possible counterexample in the source
|
||||
packet's editorial section. Consult the walkthrough's evidence gaps before collecting
|
||||
more sources. Follow [SourceInput](SourceInput.md); summaries are not original evidence.
|
||||
2. Initialize one immutable source per workspace, then inspect the plan:
|
||||
|
||||
```bash
|
||||
make bench-test
|
||||
make bench-plan
|
||||
../infospace-bench/.venv/bin/python scripts/research_bench.py init /path/to/source.md --slug question-01
|
||||
../infospace-bench/.venv/bin/python scripts/research_bench.py plan research/bench/infospaces/question-01
|
||||
```
|
||||
|
||||
`bench-plan` inspects the checked-in synthetic example without calling a model.
|
||||
`infospaces/fixture-smoke/` contains its completed four-stage run, prompts, fixture
|
||||
responses, provenance and review. It verifies workflow plumbing only. Its candidate
|
||||
does not count toward the research corpus, evidence or M1 acceptance.
|
||||
|
||||
To repeat the fixture in a fresh temporary workspace:
|
||||
|
||||
```bash
|
||||
BENCH_PY=../infospace-bench/.venv/bin/python
|
||||
pilot_workspace=$(mktemp -d /tmp/frontend-patterns-bench.XXXXXX)
|
||||
"$BENCH_PY" scripts/research_bench.py init research/bench/fixtures/source.md \
|
||||
--workspace "$pilot_workspace" --slug fixture
|
||||
"$BENCH_PY" scripts/research_bench.py run "$pilot_workspace/infospaces/fixture" \
|
||||
--fixture-responses research/bench/fixtures/responses.yaml
|
||||
```
|
||||
|
||||
## Investigate a real source
|
||||
|
||||
1. Obtain and review an original source using the [research protocol](../plan/ResearchProtocol.md).
|
||||
Prepare a bounded normalized Markdown excerpt with stable section/paragraph
|
||||
locators. Record any omitted context. No source acquisition or URL fetching is
|
||||
performed by this adapter.
|
||||
2. Add the metadata shown in [SourceInput](SourceInput.md). Identify the original
|
||||
author, URL/bibliographic locator, version and access date. This is required
|
||||
provenance, not a claim that the source was independently verified.
|
||||
3. Initialize a uniquely named workspace and inspect its plan:
|
||||
|
||||
```bash
|
||||
../infospace-bench/.venv/bin/python scripts/research_bench.py init /path/to/source.md \
|
||||
--slug carniato-review-01
|
||||
../infospace-bench/.venv/bin/python scripts/research_bench.py plan \
|
||||
research/bench/infospaces/carniato-review-01
|
||||
```
|
||||
|
||||
4. Use the deployed llm-connect service. On this workstation, forward its port
|
||||
in a separate terminal (leave it running for the investigation):
|
||||
3. For live execution, keep a service port-forward running in a separate terminal:
|
||||
|
||||
```bash
|
||||
kubectl -n activity-core port-forward svc/llm-connect 18081:8080
|
||||
```
|
||||
|
||||
Then run in a fresh workspace:
|
||||
```bash
|
||||
../infospace-bench/.venv/bin/python scripts/research_bench.py run \
|
||||
research/bench/infospaces/question-01 \
|
||||
--llm-connect-url http://127.0.0.1:18081 --model custodian-triage-balanced \
|
||||
--diagnostics --max-tokens 3000 --max-calls 3
|
||||
```
|
||||
|
||||
The service owns provider credentials/routing. The model argument may name a runtime
|
||||
profile; actual model is recorded. Sources/drafts reach its provider. The inspected
|
||||
OpenRouter route drops `reasoning_effort`; it is not a working control for that route.
|
||||
A service-request ceiling does not bound internal retries or dollars. Token caps are
|
||||
requested per response, not a total input-plus-output budget.
|
||||
|
||||
## Conditional execution
|
||||
|
||||
The copied `profile/execution.json` opts new workspaces into this policy:
|
||||
|
||||
- Claims and candidates use compact source-grounded model outputs. Candidate evidence
|
||||
ledgers must pass structural screening before further work.
|
||||
- With fewer than two candidates, the adapter writes an empty relation array with
|
||||
deterministic provenance and makes no relation call.
|
||||
- With multiple candidates, one model call proposes relations. Endpoints must resolve
|
||||
to candidate IDs and every edge must satisfy the structural contract. Invalid
|
||||
proposals are quarantined in `reports/rejected-stage.json`; execution stops.
|
||||
- The historical stage ID `critique` now writes a deterministic screen and editorial
|
||||
checklist to `output/evaluations/source-review.md`. No automatic self-critique call.
|
||||
This compatibility filename does not mean editorial review has happened.
|
||||
|
||||
Successful uncached runs therefore use two or three adapter calls, compared with
|
||||
four historically. `plan.provider_calls` is the uncached upper bound; the minimum
|
||||
assumes successful execution without reuse. A lower `--max-calls` is enforced during
|
||||
execution: if a needed call exceeds it, the attempted workspace fails and remains
|
||||
immutable. Nothing automatically retries, promotes artifacts, or marks them reviewed.
|
||||
|
||||
## Explicit reuse
|
||||
|
||||
Create a fresh workspace for the same source and use `--reuse-from`:
|
||||
|
||||
```bash
|
||||
../infospace-bench/.venv/bin/python scripts/research_bench.py run \
|
||||
research/bench/infospaces/carniato-review-01 \
|
||||
--llm-connect-url http://127.0.0.1:18081 \
|
||||
--model custodian-triage-balanced --diagnostics \
|
||||
--max-tokens 3000 --max-calls 4
|
||||
research/bench/infospaces/question-02 \
|
||||
--llm-connect-url http://127.0.0.1:18081 --model custodian-triage-balanced \
|
||||
--diagnostics --max-tokens 3000 --max-calls 0 \
|
||||
--reuse-from research/bench/infospaces/question-01
|
||||
```
|
||||
|
||||
In-cluster consumers can use `http://llm-connect.activity-core.svc.cluster.local:8080`.
|
||||
The service owns credentials and provider routing. `--model` names a service runtime
|
||||
profile or requests a model; the resolved model is recorded from the response.
|
||||
The bench uses llm-connect's `RunConfig` and `/execute` contract. It makes no direct
|
||||
provider requests and has no client retries. Sources and drafts reach the service's
|
||||
configured provider. The four-call ceiling bounds **service requests**, not internal
|
||||
provider retries, tokens or dollars. `--max-tokens` is a requested per-response cap.
|
||||
The service may apply profile settings. Actual model, usage, finish reason and
|
||||
selected routing metadata are retained in receipts and artifact provenance; cost
|
||||
remains unknown unless independently measured. Empty or incomplete responses stop
|
||||
the workflow and retain diagnostic metadata. Successful runs remain unreviewed.
|
||||
Only completed conditional runs with a matching cache-file digest are eligible.
|
||||
Keys include source hash, every profile file, workflow, rendered prompt, stage,
|
||||
input ID, adapter/settings and execution/screen code hashes. Changed inputs/settings
|
||||
miss the cache. A zero-call ceiling prevents paid fallback. Legacy receipts without
|
||||
a stage cache cannot be replayed this way; cite their artifacts directly instead.
|
||||
|
||||
5. Review `artifacts/generated/claims.md`, `candidates.md`, `relation-proposals.md`
|
||||
and `output/evaluations/source-review.md`. The review receives original source
|
||||
text plus all previous outputs. It is an assisted self-review; a successful
|
||||
run never marks it independently reviewed.
|
||||
6. Update the source/candidate catalogs and question ledger after review. Use the
|
||||
[admission procedure](Admission.md) before copying anything to `patterns/`.
|
||||
Then compare the two source investigations in VC-01/VC-02 walkthroughs.
|
||||
Reuse is an explicit replay of recorded evidence, not a fresh provider observation.
|
||||
Remote profile changes cannot be detected from a local fingerprint. Omit reuse when
|
||||
a new model/profile observation is the research question. Reused artifacts carry
|
||||
`reused_from` and `current_provider_call: false`; their original usage remains historical
|
||||
provenance. Use the research receipt's current-call/current-token totals for cost
|
||||
accounting, not a sum over reused artifact metadata.
|
||||
|
||||
## Execution and evidence boundaries
|
||||
## Review and cycle metrics
|
||||
|
||||
- One source per workspace. The inspected upstream runner keeps one intermediate
|
||||
result per stage, not per source. The adapter refuses multiple sources so stages
|
||||
cannot mix documents. Cross-source synthesis is a subsequent explicit review.
|
||||
- Workflow order: claims → candidates → relation proposals → critique. Each stage
|
||||
retains the original source. Empty candidate/relation proposals are valid findings.
|
||||
- Candidates and relations remain generated Markdown proposals. Native bench
|
||||
entity splitting requires at least one candidate and would reject legitimate
|
||||
negative results. Typed graph admission therefore follows review, not generation.
|
||||
- Source digests are checked before running. Workflow/prompt digests and a start
|
||||
receipt are written before any call; failures remain visible. Reusing an attempted
|
||||
workspace is refused. Make a new revision workspace instead of overwriting results.
|
||||
- Use this adapter's `plan` and `run` for the custom workflow. The generic `generate`
|
||||
pipeline selects its own workflow names; native dry-run rendering cannot resolve
|
||||
downstream references to assisted outputs that have not been generated yet.
|
||||
- Bench `validate` checks its registered artifact contracts. Generated research
|
||||
proposals have no automatic semantic validation here. Structural metrics and an
|
||||
empty validation error list do not prove evidence quality or pattern validity.
|
||||
- Fixture responses are allowed only for synthetic sources through the CLI.
|
||||
Fixture provenance remains visible in manifests and run records.
|
||||
Read `reports/research-run.json`: actual attempted calls, cache hits, deterministic
|
||||
stages, safe response usage/diagnostics, and structural rejection counts. Live
|
||||
`reported_tokens_this_run` excludes reused usage; missing provider usage is not a
|
||||
billing measurement. `rejected_assertions` counts structural findings, not a semantic
|
||||
truth score. `editorial_corrections: null` means not yet assessed, never zero corrections.
|
||||
|
||||
## Current source queue
|
||||
Record a named editorial review using [ResearchReviewMetrics](../../templates/ResearchReviewMetrics.yaml),
|
||||
with individual rejected assertions and corrections linked to artifacts. Check source
|
||||
fidelity, scope, known-use independence, counterexamples and applicability. Apply
|
||||
[Admission](Admission.md) before moving any proposal into canonical patterns.
|
||||
|
||||
| Investigation | Input readiness | Work record |
|
||||
| --- | --- | --- |
|
||||
| Carniato first source case | Manual review plus completed llm-connect draft; admission rejected | FEP-WP-0002-T02 |
|
||||
| Contrasting local-first source | Manual plus completed bounded live review; no candidates justified | FEP-WP-0002-T02 |
|
||||
| VC-01 and VC-02 synthesis | Depends on reviewed source claims and candidate patterns | FEP-WP-0002-T03 |
|
||||
`--diagnostics` retains only allowlisted numeric settings/usage and reasoning-field
|
||||
presence. Raw debug bodies, headers and internal reasoning text are discarded.
|
||||
Incomplete normal generated text is quarantined in `reports/rejected-response.md`,
|
||||
with digest and rejection status. Immutable receipts preserve failures.
|
||||
|
||||
Do not ingest the existing Carniato research stub as if it were the original article.
|
||||
## Verified examples and limitations
|
||||
|
||||
- `fixture-smoke`: historical four-stage synthetic execution, never research evidence.
|
||||
- `conditional-smoke-01`: two fixture calls plus deterministic relations/review.
|
||||
- `conditional-reuse-01`: same inputs reused with zero new calls. These demonstrate
|
||||
routing and provenance, not measured live token savings or improved research quality.
|
||||
- [Truncation diagnosis](../reviews/TruncationDiagnosis.md): live compact-output repair.
|
||||
- [Implementation evidence](../reviews/LocalFirstEvidenceExpansion.md): completed Joplin
|
||||
and Obsidian investigations; the latter's invalid relations were rejected editorially.
|
||||
- [Optimization review](../reviews/ResearchOptimization.md): policy checks and VC-02 result.
|
||||
|
||||
Primary-source plans (four calls each; each run failed on its first call):
|
||||
|
||||
```bash
|
||||
../infospace-bench/.venv/bin/python scripts/research_bench.py plan research/bench/infospaces/carniato-authority-01
|
||||
../infospace-bench/.venv/bin/python scripts/research_bench.py plan research/bench/infospaces/local-first-authority-01
|
||||
```
|
||||
|
||||
These short packets isolate the first comparison question. They do not substitute
|
||||
for full source review, independent known uses or the two architecture walkthroughs.
|
||||
|
||||
|
||||
## Live pilot outcome — 2026-09-24
|
||||
|
||||
Both primary-source runs attempted Gemini 2.5 Flash through OpenRouter and failed
|
||||
before producing research artifacts. A separate credential probe returned HTTP 401.
|
||||
Their failed receipts remain immutable. Read [AuthorityPilot](../reviews/AuthorityPilot.md)
|
||||
for the manual review that continued independently. Subsequent live runs use the llm-connect service and fresh workspace slugs;
|
||||
no workstation provider credential is required.
|
||||
|
||||
Failure receipts now capture adapter/model and safe structured diagnostic fields;
|
||||
raw provider error bodies, headers and credential values are never persisted.
|
||||
The initial failed receipts predate this improvement; the review records their model.
|
||||
|
||||
|
||||
## llm-connect service pilot — 2026-09-24
|
||||
|
||||
Live execution now uses llm-connect exclusively. The service's
|
||||
`custodian-triage-balanced` profile resolved to `google/gemini-2.5-flash`:
|
||||
|
||||
| Workspace | Outcome | Service requests | Reported total tokens |
|
||||
| --- | --- | --- | --- |
|
||||
| carniato-authority-02 | Claims truncated at 4,000 completion tokens | 1 | 4,470 |
|
||||
| carniato-authority-03 | Four stages completed; unreviewed generated artifacts | 4 | 12,638 |
|
||||
| local-first-authority-02 | Claims truncated at 12,000 completion tokens | 1 | 12,489 |
|
||||
|
||||
All receipts are retained. Token totals include failed responses and are not dollar
|
||||
costs. No additional local-first retry was made. See
|
||||
[the service pilot review](../reviews/LLMConnectPilot.md) for rejected proposals and
|
||||
follow-up work. The current profile is a triage profile; research suitability has
|
||||
not been established. Successful service execution is not evidence acceptance.
|
||||
|
||||
|
||||
## Bounded refinement — 2026-09-25
|
||||
|
||||
The current prompts bound stage output length, distinguish claim IDs from candidate
|
||||
IDs, and require a JSON evidence ledger. Run `scripts/research_admission.py` before
|
||||
editorial admission review (see Admission.md). Historical workspaces preserve their
|
||||
original prompts and are not silently migrated.
|
||||
|
||||
The 2026-09-25 local-first attempt used a requested low reasoning effort and 6,000 maximum
|
||||
completion tokens per response, four service requests maximum, with no client retry
|
||||
or automatic rerun. This is at most 24,000 requested completion tokens across four
|
||||
responses; input tokens, internal retries and dollar cost are outside that bound.
|
||||
The service owns profile resolution and provider support for reasoning settings;
|
||||
a recorded request is not proof the provider honored every setting. Shorter prompts
|
||||
and changed reasoning settings are a combined experiment, not an isolated causal test.
|
||||
|
||||
|
||||
That attempt (`local-first-authority-03`) truncated at claims: one request, 6,525
|
||||
reported total tokens. No further retry was made. See
|
||||
[refinement review](../reviews/ResearchPipelineRefinement.md) for the remaining
|
||||
execution limitation and the captured-failure screening report.
|
||||
|
||||
|
||||
## Execution diagnostics and compact output
|
||||
|
||||
`--diagnostics` requests llm-connect's per-call debug capture but stores only an
|
||||
allowlisted summary: numeric generation settings, presence of reasoning fields,
|
||||
numeric usage/cost and reasoning-token counts. It never writes raw debug bodies,
|
||||
headers, prompts or internal reasoning text. The service response may contain those
|
||||
fields transiently; this consumer discards them. Availability depends on the server.
|
||||
Normal model-generated incomplete content is quarantined separately as
|
||||
`reports/rejected-response.md`, with a hash and rejected status in the receipt;
|
||||
it is not registered as a completed research artifact.
|
||||
|
||||
The inspected llm-connect OpenRouter adapter drops `reasoning_effort`. Live diagnostic
|
||||
capture confirmed its absence at the provider boundary. Do not use that flag as a
|
||||
working control for this route; it remains available for other service implementations.
|
||||
The local-first diagnostic output was a table header followed by tens of thousands
|
||||
of spaces. Current prompts use compact records and explicitly forbid padded tables.
|
||||
See the [truncation diagnosis](../reviews/TruncationDiagnosis.md) for measured outcomes.
|
||||
|
||||
|
||||
Local-first `-06` subsequently completed all four stages at the 3,000-token response
|
||||
cap using compact records. Its empty candidate/relation outcome passes structural
|
||||
screening; editorial review admits no patterns. The next work is source expansion,
|
||||
not another truncation retry.
|
||||
|
||||
|
||||
## Implementation evidence — 2026-09-25
|
||||
|
||||
`joplin-conflicts-01` and `obsidian-conflicts-01` completed four stages each using
|
||||
llm-connect with compact prompts and a 3,000-token response cap. Joplin passes
|
||||
structural screening; Obsidian's invalid claim-ID relations are blocked. Editorial
|
||||
synthesis is [Conflict Copy](../candidates/ConflictCopy.md), a hypothesis research
|
||||
draft with two documented uses. See [expanded review](../reviews/LocalFirstEvidenceExpansion.md)
|
||||
for scope, source locators, rejected guarantees and remaining research work.
|
||||
One source per workspace remains necessary because upstream stage outputs are keyed
|
||||
by stage rather than source. Cross-source synthesis is separate editorial work.
|
||||
Structural screening cannot establish source truth, independent recurrence, durability
|
||||
or semantic validity. No live model call is needed merely to rephrase unchanged evidence.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue