tegwick
509e86d9c5
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a0cffe-206f-7323-89d3-b20ca4cf4bf9
57 lines
3.3 KiB
Markdown
57 lines
3.3 KiB
Markdown
# Research pipeline refinement
|
|
|
|
Date: 2026-09-25. Reviewer: codex, self-review. Work owner: FEP-WP-0002-T02.
|
|
|
|
## Changes and verification
|
|
|
|
Current prompts require bounded stage output, unknown recurrence when evidence is
|
|
missing, located implementation evidence for known uses, and a JSON candidate
|
|
ledger. Relations must resolve Pnn endpoints against that ledger; Cnn claims cannot
|
|
be endpoints. The assisted critique explicitly challenges these failure modes.
|
|
|
|
`scripts/research_admission.py` screens the ledger and relations without granting
|
|
admission. Its tests exercise the unchanged Carniato `-03` artifacts, unsourced
|
|
known uses, unsupported recurrence, duplicate IDs, malformed contracts, claim
|
|
endpoints and valid empty outcomes. The captured Carniato case is blocked: missing
|
|
ledger, and unresolved C01/C02 endpoints. A missing ledger also prevents resolving
|
|
its P01 endpoint under the new contract; this is format incompatibility, not proof
|
|
that P01 never existed. Report: [screening JSON](CarniatoAdmissionRegression.json).
|
|
|
|
The screen does not interpret prose or establish citation truth, independent uses,
|
|
source fidelity or semantic prerequisites. Passing it never grants admission.
|
|
The new generation prompts have not yet been demonstrated end-to-end in a successful
|
|
live run; offline checks establish the contract, not model compliance.
|
|
|
|
## Bounded live experiment
|
|
|
|
Fresh workspace: `research/bench/infospaces/local-first-authority-03`.
|
|
Original source bytes are unchanged from `-01`; revised prompts are copied and
|
|
hashed. Requested llm-connect profile: custodian-triage-balanced; reasoning effort:
|
|
low; maximum completion tokens: 6,000 per response; service call ceiling: four.
|
|
No automatic client retries or repeat workspace runs. Provider retries are unknown.
|
|
|
|
Result: **failed at claims**, finish_reason `length`, resolved model
|
|
`google/gemini-2.5-flash`; 525 prompt tokens + 6,000 completion tokens = 6,525 reported
|
|
total tokens. No generated research artifacts. The remaining three service calls
|
|
were not made. Dollar cost is unknown. Receipt:
|
|
[run record](../bench/infospaces/local-first-authority-03/reports/research-run.json).
|
|
|
|
This combined prompt/settings intervention did not resolve truncation. It does not
|
|
show whether the requested reasoning setting was honored or whether token use was
|
|
visible output, internal reasoning, or repetition. Existing receipts lack that
|
|
breakdown. No causal conclusion or model-quality conclusion follows. Failed runs
|
|
remain immutable; no further live retry was made in this cycle.
|
|
|
|
## Next work
|
|
|
|
T02 remains in progress. Before another paid attempt, establish service-side
|
|
visibility into effective reasoning settings and completion-token breakdown, or
|
|
select a documented research profile with predictable output behavior. Future client
|
|
receipts now retain content length and hash to distinguish empty output from a
|
|
nonempty incomplete response; this cannot retroactively diagnose existing failures.
|
|
Do not retrieve workload credentials or change a deployment to obtain diagnostics.
|
|
|
|
Once a run completes, apply structural screening, then editorial source/prose review.
|
|
Known uses and recurrence remain unknown until independently verified. T03 still
|
|
owns the server-authoritative and offline-field-note walkthroughs and draft corpus;
|
|
this tooling cycle does not admit patterns or complete source research.
|