frontend-patterns/research/evaluations/language-pilot-01/protocol.md

Presentation pilot protocol — frozen before execution

2026-09-26. Owner: FEP-WP-0002 T03. Corpus baseline: 78bd951. Question: does named pattern organization improve recommendations compared with plain prose containing the same guidance? This is a small exploratory model pilot, not human validation, a retrieval benchmark or a general efficacy estimate.

Conditions and isolation

Two packs share eight byte-identical guidance bodies in the same order and identical introductory/evidence caveats. The named condition adds candidate headings and names in a conditional relationship paragraph; prose expresses equivalent relationships without names. Word counts: 689 versus 663 (about 4% difference). See equivalence mapping. No extra implementation facts are supplied in either condition.

Six new briefs follow the frozen corpus, each in two independent single-request sessions: 12 calls total. The model sees one pack, common instructions and only the scenario brief. It never receives criteria, other answers, assignment mapping or conversation history. Same profile (custodian-triage-balanced), temperature 0.2, 1,400 completion-token cap and requested 220-word maximum for every call. Record actual resolved model, numeric diagnostics, usage and elapsed time. No tools are available to the model. All inference goes through llm-connect.

Alternate which condition runs first across the six pairs (three each). Randomize U/V answer labels within each pair independently before execution; persist the key. No optional stopping, selective retries, output edits or prompt changes after freeze. A failed generation remains failed and its pair is excluded from quality comparison; report failures and costs. Do not infer a dollar ceiling from call caps. Response-length noncompliance is recorded separately, not a quality score penalty.

Predefined scoring

Each scenario in scenarios.json contains four required decision criteria: C1/C2 = constraint handling; R = recovery correctness; Q = identifying the material missing contract. Score each 1 when explicitly and correctly addressed, otherwise 0, with an answer quotation/locator and explanation. No partial credit. A contradiction of a requirement scores 0 even if another sentence mentions it. Concise semantically equivalent wording qualifies. Do not reward names or unnecessary questions.

Separately record each materially unsupported guarantee or prohibited recommendation with its quoted evidence. Mere mention of an unsafe alternative while rejecting it is not a violation. Omitting a requirement is a miss, not automatically an unsafe claim. Do not convert absent source citations into errors: no browsing is requested. Assess all answers against the same prewritten rubric; unavailable answers are null, never 0. Report per-criterion counts, total missed obligations, unsafe claims and per-scenario paired differences. These are descriptive counts, not calibrated scores.

A condition has a descriptive pair advantage only if it meets more obligations with no additional unsafe claims, or has fewer unsafe claims with no fewer obligations. Trade-offs are mixed; equal counts are tied. No significance, population confidence, causal generalization or canonical promotion is justified by six pairs and one sample/model per cell. No evidence of difference is not evidence of equivalence. Report token use and elapsed time separately; variable service latency is not human review speed. Training knowledge and single-model variance remain uncontrolled.

Masking and human follow-up

Generation is asked not to mention headings/names. Score U/V views containing only answer text with the rubric before reading the assignment key or raw receipts. Persist and commit initial scores before unmasking. This is condition-masked Codex editorial scoring; the same author prepared the materials, so it is not independent or perfectly blind. Wording may reveal a condition. Do not redact semantic content or claim the assessor cannot infer it. Record suspected leakage.

Human review is a separate pending deliverable. Provide masked answers, rubric and an empty scoring sheet without the key or initial model scores. Request assessment of missed obligations and unsafe claims plus disagreements; leave it unfilled until a person supplies judgments. Do not substitute a second model for the human gate. The public Git key makes future blind review procedural rather than access-enforced.

Decision rule

If differences are small, tied or inconsistent, retain hypotheses and report the pilot as inconclusive. Material errors common to both packs suggest revising shared guidance or scenario clarity, not declaring one presentation superior. A consistent advantage without extra unsafe claims warrants replication with independent scenarios and reviewers, not adoption as established fact. Do not tune these six briefs and reuse them as held-out evaluation after reading outputs.