frontend-patterns/research/evaluations/language-replication-02/protocol.md

Fresh atomic-criteria replication protocol — 2026-10-04

Owner: FEP-WP-0009 T02. Author/reviewer: Codex, author-model; no independent human validation. Preparation uses no model calls. This is a fresh authored scenario replication of the presentation question, not held-out M3 retrieval or production use. The preparing author has seen the original pilot and rubric; evaluated solver sessions have not yet received these six new briefs. Old pilot-01 materials/results stay fixed.

Question and frozen scope

Does named organization change constraint/recovery recommendations compared with the same guidance under neutral numbered headings, on six fresh variants of the existing failure mechanisms? All eight bodies, introductory caveat, order and closing relationships are byte-identical. Only headings vary. Both conditions receive the same facts. Guidance summarizes the eight core hypotheses from immutable preview 0.1; exact corpus/packaging/manifest pins and source candidate hashes are in freeze.json. No claim about subsequent live research, general model competence or independent full-pattern recurrence follows. A protocol check establishes traceability, not semantic truth. Human reviewers may flag rubric/guidance ambiguity rather than force a score.

Atomic criteria and scope limits

Six scenarios have six binary atomic obligations each: 36 per condition. Award 1 only for an explicit correct obligation without a contradicting recommendation; otherwise 0 with an explanation. No partial credit, reward for names, inferred implementation guarantee or multi-clause implicit completion. Keep scores null for unavailable/unreviewed answers, never default them to zero. Quotes or answer locators and reasons are required for reviewed values. Unsafe recommendations are listed separately; rejecting an unsafe alternative is not an unsafe claim. Missing an obligation is not automatically an unsafe claim.

These scenarios target policy-dependent preservation/export, cancellation versus reversal, atomic supersession, backup/restore, expired receipt lookup and accepted operation versus projection recovery. They deliberately challenge guidance shared by both packs. Thirty-six obligations are not directly comparable with pilot-01's 24; report individual scenarios/criteria, contradictions, failures and ambiguity, not an aggregate improvement claim across changed protocols.

Execution and budget

Freeze and commit inputs, assignment and exact isolated prompts before any inference. Six pairs × two conditions = 12 independent single-request sessions. Alternate the condition run first (three each) and independently randomize U/V labels before execution. Each prompt contains only common instructions, one pack and one brief: no criteria, prohibited list, other answer, assignment key or conversation history. No tools are available. Same llm-connect profile, temperature 0.2 and 1,400 requested output-token cap for every session. Paid execution requires explicit user call-budget authorization; no authorization is implied by a prepared prompt or command. No dollar ceiling is inferred from a call cap. Record actual model/usage, latency, output cap and failures.

No optional stopping, regeneration or selective retries. Reserve each attempt in its immutable session directory before calling. A failed response remains failed; its pair cannot enter fulfilled-obligation comparison. Length violations are recorded separately. Do not repair prompts or answers after observing results; corrections get a new packet. No worker or paid execution is enabled by preparation.

The user explicitly authorized up to 12 capped calls with no retries on October 4. The runner enforces an aggregate 12-reservation ceiling, refuses a second attempt for any reserved session, and can resume only unstarted sessions. Commit verification checks the packet and runner before any request. Report failures or interruption rather than silently regenerate. Active research effort for preparation is unknown because no timer was started; do not infer it from commits or elapsed chat time. Execution seconds are measured separately per request; cost/tokens only from retained receipts.

Scoring, masking and interpretation

The sheet identifies U/V answers without condition metadata. Persist first ratings before reading assignments; retain exposure disclosures and ambiguous criteria. The public Git assignment makes masking procedural, not access-enforced, and wording may reveal a condition. A declaration of human authorship is provenance to review, not an automatic identity/independence verification. Model review does not fill the human gate.

Use a complete frozen rubric with evidence-bearing binary ratings and separate unsafe claims. Equal counts are a tie, fewer unsafe claims with no fewer fulfilled obligations or more obligations with no more unsafe claims is a descriptive advantage; conflicting movements are mixed. Six pairs are not statistical proof of benefit, equivalence or a plateau. Service latency is not human decision time. Retain hypotheses unless stronger independent evidence warrants review; never auto-promote or mark M1 passed.

Handoff

make replication-check replication-test verifies the packet and score-record structure without inference. WP-0009 T02 remains open until authorized execution, retained outputs and scoped replication review. T01 retains independent human pilot-01 ratings; T03 retains M1 re-assessment after both native evidence obligations. No empirical learning cycle or consumer outcome is produced by this preparation.