frontend-patterns/research/evaluations/language-pilot-01

Matched presentation pilot — 2026-09-26

Outcome: inconclusive. Twelve isolated llm-connect sessions completed. Initial condition-masked Codex scoring found a small descriptive difference, with errors in both presentations. Independent human review remains pending. No canonical admission or claim of proven decision usefulness follows.

Update: the user-submitted review and adjudication changes four binary ratings, giving 18/24 obligations met in each condition. The worksheet declares model authorship and exposure to the aggregate summary; independent human validation is still unfulfilled. The figures below preserve the initial scoring, not the latest adjudicated totals.

For independent scoring, begin with the human handoff before reading the results or assignment key. The scorer who prepared this report also authored the materials, so masking does not make the assessment independent.

Design and execution

Corpus baseline 78bd951; protocol, equivalent packs, six scenarios and rubric frozen in preregistration commit 6acbe6d. Each model session received only common instructions, one pack and one scenario brief. No scoring criteria, other answers or conversation history were supplied. All eight guidance bodies and evidence limits are shared; named headings and named connections are the treatment. Pack sizes: 689 versus 663 words. This isolates presentation more narrowly than comparing a full knowledge base with an uninformed baseline.

Six pairs, counterbalanced condition order (three each); one sample per cell. Every call used llm-connect profile custodian-triage-balanced, resolved to google/gemini-2.5-flash, temperature 0.2 and 1,400 requested output tokens. Twelve calls, no retries, no failed/truncated generations. One pattern-condition answer exceeded the requested 220 words; all stopped normally within token caps.

Masked judgments were committed in 3690af1 before opening the assignment mapping. Assessor read U/V answers without receipts/packs/key; no direct pattern-name leakage was noticed. Presentation could still be inferred. The scoring record contains quotations/locators, explanations and ambiguity flags. Raw answers were never edited. Summary is derived by summarize.py, which verifies prompts and output hashes before counting.

Initial descriptive results

Measure	Named patterns	Equivalent prose
Completed sessions	6	6
Required obligations met	21 / 24	19 / 24
Missed obligations	3	5
Constraint items met (C1 + C2)	11 / 12	11 / 12
Recovery items met (R)	5 / 6	3 / 6
Missing-contract items met (Q)	5 / 6	5 / 6
Unsupported claims flagged	2	2
Prompt tokens	6,297	6,051
Completion tokens	1,137	1,192
Total reported tokens	7,434	7,243
Summed request wall time	11.997 s	11.317 s
Provider-reported cost	$0.0047316	$0.0047953

Total: 14,677 reported tokens, $0.0095269 provider-reported cost. These are diagnostics, not a billing audit. Service latency is not human review time. Request order, provider variance and the slightly longer pattern pack confound efficiency claims. No significance tests or calibrated confidence estimates are warranted.

Scenario	Obligations: patterns / prose	Descriptive advantage
S01 Late receipt/new draft/opaque replica version	4 / 3	Patterns
S02 Identity lost across device change	4 / 4	Tie
S03 Revoked access and confidential local content	3 / 3	Tie
S04 Cancellation of an uncertain booking	4 / 3	Patterns
S05 Obsolete upload in flight after reconciliation	3 / 3	Tie
S06 Power failure, device loss and recovery copies	3 / 3	Prose: fewer unsupported claims

“Advantage” is the preregistered dominance rule on fulfilled obligations and flagged claims, not a declaration of superiority. Two pattern advantages, one prose advantage and three ties are a mixed, small result on purpose-built development variants. These scenarios are now consumed evaluation material; do not call tuned reruns held-out.

What the errors teach us

Both conditions left meaningful gaps. In S03, answers asked about revocation policy but also made unconditional retention or access statements before receiving it. In S05 neither fully specified atomic retirement of P with c/Q despite that guidance being present. In S06 neither asked for an independently recoverable backup/restore contract. Clarification language alone does not ensure the recommendation actually respects the missing information.

S01 prose asserted that a different opaque version was stale. S04 prose warned about duplication but did not provide reconciliation before suggesting a fresh identity. S06 patterns correctly denied a survival guarantee, then overstated that the storage could not recover at all. Lack of evidence and evidence of impossibility differ.

Several judgments need adjudication: S03's retention phrasing versus its later policy question; S04's fresh-identity suggestion versus its duplicate warning; S05's implicit scope of the old receipt; S06's negative capability assertion and backup wording. The original scores remain fixed. Alternative reasonable interpretations could change pair counts; do not resolve ambiguity in favor of a preferred presentation.

Decision and next step

Retain the eight hypotheses and both presentations. Do not expand the schema or promote patterns based on this pilot. Obtain human ratings using the existing frozen rubric, retain disagreement, then decide whether shared guidance needs clearer rules for policy-dependent recovery, atomic supersession and evidence limits. Any revised materials require fresh scenarios and replication before a usefulness claim.

Human review is queued under FEP-WP-0002 T03, with an empty score sheet. It has not happened; no human agreement or independent validation is claimed.

Reproduction and audit

From the repository root:

PYTHONDONTWRITEBYTECODE=1 ../infospace-bench/.venv/bin/python research/evaluations/language-pilot-01/run.py validate
python3 research/evaluations/language-pilot-01/summarize.py

These commands make no model calls. Validation checks the frozen material against the working checkout; use the pilot snapshot if later corpus changes are present. The inference runner refuses to overwrite an existing session directory. A new replication needs a new workspace, protocol and explicit budget; no silent replay. The assignment key is public in Git, so future masking is procedural, not enforced.