frontend-patterns/research/evaluations/language-replication-03/README.md

Replication 03 — clarified criteria

Status: frozen 2026-10-20, not authorized for execution. Owner: FEP-WP-0009 T02. Prepared 2026-10-20 by Claude Code (author-model) at the owner's request, using no model calls. It applies the results of the pilot-01 human review and adjudication (review, adjudication). Replication-02 and its spent run identities stay unchanged.

Why a revision

Replication-02 already uses six single-obligation criteria per scenario. The pilot-01 adjudication found 11 of 16 disagreements were caused by rubric wording, not by the reviewers. Four kinds of ambiguity recur, and several replication-02 criteria still admit them:

Ambiguity found in pilot-01	Example	Replication-02 criteria affected
Generic vs specific contract naming	S01/S02/S04 Q	S01-A3, S02-A6, S03-A5, S05-A6, S06-A6
Condition stated elsewhere vs in the recommendation	S03 R	S01-A4, S01-A5
End state vs stated atomic step	S05 C1	S03-A1
Flat assertion of an unknown fact	S01-V "stale"; S03-V "must be retained"	S01-A6, S06-A5

The reviewer's recurring guidance was to pair abstract wording with a concrete example. Each criterion in scenarios.json therefore carries qualifies and does_not_qualify examples, a placement and a change_from_r02 note. Briefs and prohibited lists are copied unchanged.

Scoring rules (summary)

• G1 Placement. A recommendation criterion is met only inside the recommended state or action. A matching statement only in a later questions section does not count. An anywhere criterion may be met in any section.
• G2 Named subject. A contract-identification criterion requires the guarantee's subject (for example lookup, replay window, read-after-write). A generic "define the contract" does not qualify. Listed alternatives joined by "or" are each sufficient.
• G3 Examples are illustrative. Equivalent wording qualifies; the examples show where the line is, not required phrases.
• G4 Unknown facts. Asserting an unspecified fact as known (stale, permanent, succeeded) is recorded as an unsupported claim.
• G5 Overstated caution (for example "cannot recover" where it is only untested) is recorded as a separate observation, not an unsupported claim, following the owner's S06-V adjudication.
• G6 Ambiguity is a valid result. A reviewer may mark a criterion ambiguous (X) instead of forcing a score; ambiguity counts are reported per criterion.

Reviewer support

The pilot-01 human review worked only in interview form, with plain-language context per scenario. A replication-03 worksheet should include that context, written before any answers exist, without hints about any answer.

Owner decisions (2026-10-20)

D1 reuse briefs: yes. D2 overstated-caution rule G5: yes. D3 reviewers: owner and an external human reviewer, independently. D4 execution: not yet authorized. Details and the binding rules are in the protocol; G6 there supersedes the draft wording above (a binary value is still required alongside an ambiguity entry).

Commands

From the repository root, without inference:

make replication-03-check replication-03-test

Execution needs, in order: make inference-preflight passing against a reachable llm-connect, fresh explicit authorization of up to 12 capped calls with no retries, and a committed freeze. Then:

PYTHONDONTWRITEBYTECODE=1 ../infospace-bench/.venv/bin/python scripts/language_replication_03.py run --max-calls 12

After answers exist, build the reviewer worksheet from masked/ plus each scenario's reviewer_context and criteria examples; each reviewer copies scores-template.json. scripts/language_replication_03.py scores-check|summarize --scores <file> validate and summarise a sheet. Replication-02 stays frozen and verifiable with make replication-check.