frontend-patterns/research/evaluations/language-pilot-01/protocol.md

# Presentation pilot protocol — frozen before execution

2026-09-26. Owner: FEP-WP-0002 T03. Corpus baseline: `78bd951`.
Question: does named pattern organization improve recommendations compared with
plain prose containing the same guidance? This is a small exploratory model pilot,
not human validation, a retrieval benchmark or a general efficacy estimate.

## Conditions and isolation

Two packs share eight byte-identical guidance bodies in the same order and identical
introductory/evidence caveats. The named condition adds candidate headings and names
in a conditional relationship paragraph; prose expresses equivalent relationships
without names. Word counts: 689 versus 663 (about 4% difference). See equivalence
mapping. No extra implementation facts are supplied in either condition.

Six new briefs follow the frozen corpus, each in two independent single-request
sessions: 12 calls total. The model sees one pack, common instructions and only the
scenario brief. It never receives criteria, other answers, assignment mapping or
conversation history. Same profile (custodian-triage-balanced), temperature 0.2,
1,400 completion-token cap and requested 220-word maximum for every call. Record
actual resolved model, numeric diagnostics, usage and elapsed time. No tools are
available to the model. All inference goes through llm-connect.

Alternate which condition runs first across the six pairs (three each). Randomize
U/V answer labels within each pair independently before execution; persist the key.
No optional stopping, selective retries, output edits or prompt changes after
freeze. A failed generation remains failed and its pair is excluded from quality
comparison; report failures and costs. Do not infer a dollar ceiling from call caps.
Response-length noncompliance is recorded separately, not a quality score penalty.

## Predefined scoring

Each scenario in scenarios.json contains four required decision criteria:
C1/C2 = constraint handling; R = recovery correctness; Q = identifying the material
missing contract. Score each 1 when explicitly and correctly addressed, otherwise 0,
with an answer quotation/locator and explanation. No partial credit. A contradiction
of a requirement scores 0 even if another sentence mentions it. Concise semantically
equivalent wording qualifies. Do not reward names or unnecessary questions.

Separately record each materially unsupported guarantee or prohibited recommendation
with its quoted evidence. Mere mention of an unsafe alternative while rejecting it
is not a violation. Omitting a requirement is a miss, not automatically an unsafe
claim. Do not convert absent source citations into errors: no browsing is requested.
Assess all answers against the same prewritten rubric; unavailable answers are null,
never 0. Report per-criterion counts, total missed obligations, unsafe claims and
per-scenario paired differences. These are descriptive counts, not calibrated scores.

A condition has a descriptive pair advantage only if it meets more obligations with
no additional unsafe claims, or has fewer unsafe claims with no fewer obligations.
Trade-offs are mixed; equal counts are tied. No significance, population confidence,
causal generalization or canonical promotion is justified by six pairs and one
sample/model per cell. No evidence of difference is not evidence of equivalence.
Report token use and elapsed time separately; variable service latency is not human
review speed. Training knowledge and single-model variance remain uncontrolled.

## Masking and human follow-up

Generation is asked not to mention headings/names. Score U/V views containing only
answer text with the rubric before reading the assignment key or raw receipts.
Persist and commit initial scores before unmasking. This is condition-masked Codex
editorial scoring; the same author prepared the materials, so it is not independent
or perfectly blind. Wording may reveal a condition. Do not redact semantic content
or claim the assessor cannot infer it. Record suspected leakage.

Human review is a separate pending deliverable. Provide masked answers, rubric and
an empty scoring sheet without the key or initial model scores. Request assessment
of missed obligations and unsafe claims plus disagreements; leave it unfilled until
a person supplies judgments. Do not substitute a second model for the human gate.
The public Git key makes future blind review procedural rather than access-enforced.

## Decision rule

If differences are small, tied or inconsistent, retain hypotheses and report the
pilot as inconclusive. Material errors common to both packs suggest revising shared
guidance or scenario clarity, not declaring one presentation superior. A consistent
advantage without extra unsafe claims warrants replication with independent scenarios
and reviewers, not adoption as established fact. Do not tune these six briefs and
reuse them as held-out evaluation after reading outputs.