frontend-patterns/research/evaluations/language-replication-03/protocol.md

Clarified-criteria replication protocol — 2026-10-20

Owner: FEP-WP-0009 T02. Author: Claude Code (author-model) at the owner's request. Preparation uses no model calls. This packet supersedes the unexecuted replication-02 packet for T02 without modifying it: replication-02's twelve run identities are spent and stay frozen. This is not held-out M3 retrieval or production use.

Owner decisions applied (2026-10-20)

• D1 Reuse briefs: yes. The six replication-02 briefs and prohibited lists are reused byte-identically. No solver has seen them; all replication-02 attempts failed without answers. The author has seen them. Pilot-01 briefs remain consumed.
• D2 Overstated caution: apply G5. Criterion S04-A4 no longer penalises declaring recovery impossible; such statements are recorded as observations.
• D3 Reviewers: both. The owner and an external human reviewer each rate all answers independently before either sees the other's sheet, the assignment key or any model scores.
• D4 Execution: not authorized. It needs llm-connect reachable from this workstation and a fresh explicit call-budget authorization.

Question and scope

Unchanged from replication-02. Does named organization change constraint/recovery recommendations compared with the same guidance under neutral numbered headings? Guidance, instructions, pack construction, headings-only variation and the immutable reference 0.1 pin are reused unchanged. What changes is how the criteria are written, so that two reviewers can apply them the same way.

Criteria and scoring rules

Six scenarios × six binary obligations = 36 per condition. Each criterion carries a requirement, a placement, illustrative qualifies / does_not_qualify examples and its change from replication-02, derived from the pilot-01 human adjudication.

• G1 Placement. A recommendation criterion is met only inside the recommended state or action; an anywhere criterion may be met in any section.
• G2 Named subject. Contract-identification criteria require the guarantee's subject; a generic "define the contract" does not qualify. Alternatives joined by "or" are each sufficient.
• G3 Examples are illustrative. Equivalent wording qualifies.
• G4 Unknown facts. Asserting an unspecified fact as known is an unsupported claim.
• G5 Overstated caution. Unproven negative certainty is recorded in ambiguity as an observation, not as an unsupported claim or a criterion miss.
• G6 Ambiguity. A reviewer who finds a criterion ambiguous still records a binary value and adds an ambiguity entry naming the criterion. Ambiguity counts are reported per criterion.

Award 1 only for an explicit, correct obligation without a contradicting recommendation, with a quote or locator and a reason. No partial credit or reward for names. Null means unavailable or unreviewed, never zero. Unsafe recommendations are listed separately; rejecting an unsafe alternative is not an unsafe claim.

Execution and budget

As replication-02: freeze and commit before inference; 12 independent single-request sessions; alternated condition order; independently randomized U/V labels; prompts contain only the instructions, one pack and one brief; same llm-connect profile, temperature 0.2 and 1,400-token cap. The runner reserves each attempt before calling, enforces an aggregate 12-call ceiling and never retries. Run make inference-preflight first; a failed preflight means no run. Paid execution requires explicit user authorization; a prepared packet implies none.

Review and interpretation

Each reviewer works from the masked answers and the reviewer worksheet, which gives the plain-language reviewer_context for each scenario (written before any answer existed) and each criterion with its examples. Interview-mediated review is allowed if the mediator supplies no scores or hints. Both sheets are committed before unmasking. Disagreements between the two human sheets, and with any model sheet, are adjudicated against this rubric, keeping all original judgements.

Interpretation rules are unchanged: equal counts tie; a descriptive advantage needs more obligations with no more unsafe claims, or fewer unsafe claims with no fewer obligations. Six pairs prove neither benefit nor equivalence. Never auto-promote or mark M1 passed; T03 weighs the result, reviewer independence and ambiguity counts.