Freeze replication-03 with clarified criteria (FEP-WP-0009-T02)

Owner decisions D1-D3 applied; execution not authorized.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 182330@bnt-lap001
Assistant-Session: 242866cf-789d-4e24-ac5e-0a1d7b47a701

research/evaluations/language-replication-03/protocol.md

@@ -0,0 +1,75 @@
+# Clarified-criteria replication protocol — 2026-10-20
+
+Owner: FEP-WP-0009 T02. Author: Claude Code (author-model) at the owner's request.
+Preparation uses no model calls. This packet supersedes the unexecuted replication-02
+packet for T02 without modifying it: replication-02's twelve run identities are spent
+and stay frozen. This is not held-out M3 retrieval or production use.
+
+## Owner decisions applied (2026-10-20)
+
+- **D1 Reuse briefs: yes.** The six replication-02 briefs and prohibited lists are
+  reused byte-identically. No solver has seen them; all replication-02 attempts failed
+  without answers. The author has seen them. Pilot-01 briefs remain consumed.
+- **D2 Overstated caution: apply G5.** Criterion S04-A4 no longer penalises declaring
+  recovery impossible; such statements are recorded as observations.
+- **D3 Reviewers: both.** The owner and an external human reviewer each rate all
+  answers independently before either sees the other's sheet, the assignment key or
+  any model scores.
+- **D4 Execution: not authorized.** It needs llm-connect reachable from this
+  workstation and a fresh explicit call-budget authorization.
+
+## Question and scope
+
+Unchanged from replication-02. Does named organization change constraint/recovery
+recommendations compared with the same guidance under neutral numbered headings?
+Guidance, instructions, pack construction, headings-only variation and the immutable
+reference 0.1 pin are reused unchanged. What changes is how the criteria are written,
+so that two reviewers can apply them the same way.
+
+## Criteria and scoring rules
+
+Six scenarios × six binary obligations = 36 per condition. Each criterion carries a
+requirement, a placement, illustrative `qualifies` / `does_not_qualify` examples and
+its change from replication-02, derived from the pilot-01 human adjudication.
+
+- **G1 Placement.** A `recommendation` criterion is met only inside the recommended
+  state or action; an `anywhere` criterion may be met in any section.
+- **G2 Named subject.** Contract-identification criteria require the guarantee's
+  subject; a generic "define the contract" does not qualify. Alternatives joined by
+  "or" are each sufficient.
+- **G3 Examples are illustrative.** Equivalent wording qualifies.
+- **G4 Unknown facts.** Asserting an unspecified fact as known is an unsupported claim.
+- **G5 Overstated caution.** Unproven negative certainty is recorded in `ambiguity`
+  as an observation, not as an unsupported claim or a criterion miss.
+- **G6 Ambiguity.** A reviewer who finds a criterion ambiguous still records a binary
+  value and adds an `ambiguity` entry naming the criterion. Ambiguity counts are
+  reported per criterion.
+
+Award 1 only for an explicit, correct obligation without a contradicting
+recommendation, with a quote or locator and a reason. No partial credit or reward for
+names. Null means unavailable or unreviewed, never zero. Unsafe recommendations are
+listed separately; rejecting an unsafe alternative is not an unsafe claim.
+
+## Execution and budget
+
+As replication-02: freeze and commit before inference; 12 independent single-request
+sessions; alternated condition order; independently randomized U/V labels; prompts
+contain only the instructions, one pack and one brief; same llm-connect profile,
+temperature 0.2 and 1,400-token cap. The runner reserves each attempt before calling,
+enforces an aggregate 12-call ceiling and never retries. Run `make inference-preflight`
+first; a failed preflight means no run. Paid execution requires explicit user
+authorization; a prepared packet implies none.
+
+## Review and interpretation
+
+Each reviewer works from the masked answers and the reviewer worksheet, which gives
+the plain-language `reviewer_context` for each scenario (written before any answer
+existed) and each criterion with its examples. Interview-mediated review is allowed
+if the mediator supplies no scores or hints. Both sheets are committed before
+unmasking. Disagreements between the two human sheets, and with any model sheet, are
+adjudicated against this rubric, keeping all original judgements.
+
+Interpretation rules are unchanged: equal counts tie; a descriptive advantage needs
+more obligations with no more unsafe claims, or fewer unsafe claims with no fewer
+obligations. Six pairs prove neither benefit nor equivalence. Never auto-promote or
+mark M1 passed; T03 weighs the result, reviewer independence and ambiguity counts.