frontend-patterns/research/evaluations/language-replication-02/protocol.md

# Fresh atomic-criteria replication protocol — 2026-10-04

Owner: FEP-WP-0009 T02. Author/reviewer: Codex, author-model; no independent human
validation. Preparation uses no model calls. This is a fresh authored scenario
replication of the presentation question, not held-out M3 retrieval or production use.
The preparing author has seen the original pilot and rubric; evaluated solver sessions
have not yet received these six new briefs. Old pilot-01 materials/results stay fixed.

## Question and frozen scope

Does named organization change constraint/recovery recommendations compared with the
same guidance under neutral numbered headings, on six fresh variants of the existing
failure mechanisms? All eight bodies, introductory caveat, order and closing relationships
are byte-identical. Only headings vary. Both conditions receive the same facts.
Guidance summarizes the eight core hypotheses from immutable preview 0.1; exact
corpus/packaging/manifest pins and source candidate hashes are in freeze.json.
No claim about subsequent live research, general model competence or independent
full-pattern recurrence follows. A protocol check establishes traceability, not semantic
truth. Human reviewers may flag rubric/guidance ambiguity rather than force a score.

## Atomic criteria and scope limits

Six scenarios have six binary atomic obligations each: 36 per condition. Award 1
only for an explicit correct obligation without a contradicting recommendation;
otherwise 0 with an explanation. No partial credit, reward for names, inferred
implementation guarantee or multi-clause implicit completion. Keep scores null for
unavailable/unreviewed answers, never default them to zero. Quotes or answer locators
and reasons are required for reviewed values. Unsafe recommendations are listed
separately; rejecting an unsafe alternative is not an unsafe claim. Missing an
obligation is not automatically an unsafe claim.

These scenarios target policy-dependent preservation/export, cancellation versus
reversal, atomic supersession, backup/restore, expired receipt lookup and accepted
operation versus projection recovery. They deliberately challenge guidance shared by
both packs. Thirty-six obligations are not directly comparable with pilot-01's 24;
report individual scenarios/criteria, contradictions, failures and ambiguity, not an
aggregate improvement claim across changed protocols.

## Execution and budget

Freeze and commit inputs, assignment and exact isolated prompts before any inference.
Six pairs × two conditions = 12 independent single-request sessions. Alternate the
condition run first (three each) and independently randomize U/V labels before execution.
Each prompt contains only common instructions, one pack and one brief: no criteria,
prohibited list, other answer, assignment key or conversation history. No tools are
available. Same llm-connect profile, temperature 0.2 and 1,400 requested output-token cap
for every session. Paid execution requires explicit user call-budget authorization;
no authorization is implied by a prepared prompt or command. No dollar ceiling is
inferred from a call cap. Record actual model/usage, latency, output cap and failures.

No optional stopping, regeneration or selective retries. Reserve each attempt in its
immutable session directory before calling. A failed response remains failed; its pair
cannot enter fulfilled-obligation comparison. Length violations are recorded separately.
Do not repair prompts or answers after observing results; corrections get a new packet.
No worker or paid execution is enabled by preparation.

The user explicitly authorized up to 12 capped calls with no retries on October 4.
The runner enforces an aggregate 12-reservation ceiling, refuses a second attempt for
any reserved session, and can resume only unstarted sessions. Commit verification
checks the packet and runner before any request. Report failures or interruption rather
than silently regenerate. Active research effort for preparation is unknown because
no timer was started; do not infer it from commits or elapsed chat time. Execution
seconds are measured separately per request; cost/tokens only from retained receipts.

## Scoring, masking and interpretation

The sheet identifies U/V answers without condition metadata. Persist first ratings
before reading assignments; retain exposure disclosures and ambiguous criteria. The
public Git assignment makes masking procedural, not access-enforced, and wording may
reveal a condition. A declaration of human authorship is provenance to review, not an
automatic identity/independence verification. Model review does not fill the human gate.

Use a complete frozen rubric with evidence-bearing binary ratings and separate unsafe
claims. Equal counts are a tie, fewer unsafe claims with no fewer fulfilled obligations
or more obligations with no more unsafe claims is a descriptive advantage; conflicting
movements are mixed. Six pairs are not statistical proof of benefit, equivalence or a
plateau. Service latency is not human decision time. Retain hypotheses unless stronger
independent evidence warrants review; never auto-promote or mark M1 passed.

## Handoff

`make replication-check replication-test` verifies the packet and score-record structure
without inference. WP-0009 T02 remains open until authorized execution, retained outputs
and scoped replication review. T01 retains independent human pilot-01 ratings; T03
retains M1 re-assessment after both native evidence obligations. No empirical learning
cycle or consumer outcome is produced by this preparation.