76 lines
4.3 KiB
Markdown
76 lines
4.3 KiB
Markdown
|
# Clarified-criteria replication protocol — 2026-10-20
|
|||
|
|
|
|||
|
|
Owner: FEP-WP-0009 T02. Author: Claude Code (author-model) at the owner's request.
|
|||
|
|
Preparation uses no model calls. This packet supersedes the unexecuted replication-02
|
|||
|
|
packet for T02 without modifying it: replication-02's twelve run identities are spent
|
|||
|
|
and stay frozen. This is not held-out M3 retrieval or production use.
|
|||
|
|
|
|||
|
|
## Owner decisions applied (2026-10-20)
|
|||
|
|
|
|||
|
|
- **D1 Reuse briefs: yes.** The six replication-02 briefs and prohibited lists are
|
|||
|
|
reused byte-identically. No solver has seen them; all replication-02 attempts failed
|
|||
|
|
without answers. The author has seen them. Pilot-01 briefs remain consumed.
|
|||
|
|
- **D2 Overstated caution: apply G5.** Criterion S04-A4 no longer penalises declaring
|
|||
|
|
recovery impossible; such statements are recorded as observations.
|
|||
|
|
- **D3 Reviewers: both.** The owner and an external human reviewer each rate all
|
|||
|
|
answers independently before either sees the other's sheet, the assignment key or
|
|||
|
|
any model scores.
|
|||
|
|
- **D4 Execution: not authorized.** It needs llm-connect reachable from this
|
|||
|
|
workstation and a fresh explicit call-budget authorization.
|
|||
|
|
|
|||
|
|
## Question and scope
|
|||
|
|
|
|||
|
|
Unchanged from replication-02. Does named organization change constraint/recovery
|
|||
|
|
recommendations compared with the same guidance under neutral numbered headings?
|
|||
|
|
Guidance, instructions, pack construction, headings-only variation and the immutable
|
|||
|
|
reference 0.1 pin are reused unchanged. What changes is how the criteria are written,
|
|||
|
|
so that two reviewers can apply them the same way.
|
|||
|
|
|
|||
|
|
## Criteria and scoring rules
|
|||
|
|
|
|||
|
|
Six scenarios × six binary obligations = 36 per condition. Each criterion carries a
|
|||
|
|
requirement, a placement, illustrative `qualifies` / `does_not_qualify` examples and
|
|||
|
|
its change from replication-02, derived from the pilot-01 human adjudication.
|
|||
|
|
|
|||
|
|
- **G1 Placement.** A `recommendation` criterion is met only inside the recommended
|
|||
|
|
state or action; an `anywhere` criterion may be met in any section.
|
|||
|
|
- **G2 Named subject.** Contract-identification criteria require the guarantee's
|
|||
|
|
subject; a generic "define the contract" does not qualify. Alternatives joined by
|
|||
|
|
"or" are each sufficient.
|
|||
|
|
- **G3 Examples are illustrative.** Equivalent wording qualifies.
|
|||
|
|
- **G4 Unknown facts.** Asserting an unspecified fact as known is an unsupported claim.
|
|||
|
|
- **G5 Overstated caution.** Unproven negative certainty is recorded in `ambiguity`
|
|||
|
|
as an observation, not as an unsupported claim or a criterion miss.
|
|||
|
|
- **G6 Ambiguity.** A reviewer who finds a criterion ambiguous still records a binary
|
|||
|
|
value and adds an `ambiguity` entry naming the criterion. Ambiguity counts are
|
|||
|
|
reported per criterion.
|
|||
|
|
|
|||
|
|
Award 1 only for an explicit, correct obligation without a contradicting
|
|||
|
|
recommendation, with a quote or locator and a reason. No partial credit or reward for
|
|||
|
|
names. Null means unavailable or unreviewed, never zero. Unsafe recommendations are
|
|||
|
|
listed separately; rejecting an unsafe alternative is not an unsafe claim.
|
|||
|
|
|
|||
|
|
## Execution and budget
|
|||
|
|
|
|||
|
|
As replication-02: freeze and commit before inference; 12 independent single-request
|
|||
|
|
sessions; alternated condition order; independently randomized U/V labels; prompts
|
|||
|
|
contain only the instructions, one pack and one brief; same llm-connect profile,
|
|||
|
|
temperature 0.2 and 1,400-token cap. The runner reserves each attempt before calling,
|
|||
|
|
enforces an aggregate 12-call ceiling and never retries. Run `make inference-preflight`
|
|||
|
|
first; a failed preflight means no run. Paid execution requires explicit user
|
|||
|
|
authorization; a prepared packet implies none.
|
|||
|
|
|
|||
|
|
## Review and interpretation
|
|||
|
|
|
|||
|
|
Each reviewer works from the masked answers and the reviewer worksheet, which gives
|
|||
|
|
the plain-language `reviewer_context` for each scenario (written before any answer
|
|||
|
|
existed) and each criterion with its examples. Interview-mediated review is allowed
|
|||
|
|
if the mediator supplies no scores or hints. Both sheets are committed before
|
|||
|
|
unmasking. Disagreements between the two human sheets, and with any model sheet, are
|
|||
|
|
adjudicated against this rubric, keeping all original judgements.
|
|||
|
|
|
|||
|
|
Interpretation rules are unchanged: equal counts tie; a descriptive advantage needs
|
|||
|
|
more obligations with no more unsafe claims, or fewer unsafe claims with no fewer
|
|||
|
|
obligations. Six pairs prove neither benefit nor equivalence. Never auto-promote or
|
|||
|
|
mark M1 passed; T03 weighs the result, reviewer independence and ambiguity counts.
|