tegwick
f8c3ce3938
Owner decisions D1-D3 applied; execution not authorized. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 182330@bnt-lap001 Assistant-Session: 242866cf-789d-4e24-ac5e-0a1d7b47a701
75 lines
4.3 KiB
Markdown
75 lines
4.3 KiB
Markdown
# Clarified-criteria replication protocol — 2026-10-20
|
||
|
||
Owner: FEP-WP-0009 T02. Author: Claude Code (author-model) at the owner's request.
|
||
Preparation uses no model calls. This packet supersedes the unexecuted replication-02
|
||
packet for T02 without modifying it: replication-02's twelve run identities are spent
|
||
and stay frozen. This is not held-out M3 retrieval or production use.
|
||
|
||
## Owner decisions applied (2026-10-20)
|
||
|
||
- **D1 Reuse briefs: yes.** The six replication-02 briefs and prohibited lists are
|
||
reused byte-identically. No solver has seen them; all replication-02 attempts failed
|
||
without answers. The author has seen them. Pilot-01 briefs remain consumed.
|
||
- **D2 Overstated caution: apply G5.** Criterion S04-A4 no longer penalises declaring
|
||
recovery impossible; such statements are recorded as observations.
|
||
- **D3 Reviewers: both.** The owner and an external human reviewer each rate all
|
||
answers independently before either sees the other's sheet, the assignment key or
|
||
any model scores.
|
||
- **D4 Execution: not authorized.** It needs llm-connect reachable from this
|
||
workstation and a fresh explicit call-budget authorization.
|
||
|
||
## Question and scope
|
||
|
||
Unchanged from replication-02. Does named organization change constraint/recovery
|
||
recommendations compared with the same guidance under neutral numbered headings?
|
||
Guidance, instructions, pack construction, headings-only variation and the immutable
|
||
reference 0.1 pin are reused unchanged. What changes is how the criteria are written,
|
||
so that two reviewers can apply them the same way.
|
||
|
||
## Criteria and scoring rules
|
||
|
||
Six scenarios × six binary obligations = 36 per condition. Each criterion carries a
|
||
requirement, a placement, illustrative `qualifies` / `does_not_qualify` examples and
|
||
its change from replication-02, derived from the pilot-01 human adjudication.
|
||
|
||
- **G1 Placement.** A `recommendation` criterion is met only inside the recommended
|
||
state or action; an `anywhere` criterion may be met in any section.
|
||
- **G2 Named subject.** Contract-identification criteria require the guarantee's
|
||
subject; a generic "define the contract" does not qualify. Alternatives joined by
|
||
"or" are each sufficient.
|
||
- **G3 Examples are illustrative.** Equivalent wording qualifies.
|
||
- **G4 Unknown facts.** Asserting an unspecified fact as known is an unsupported claim.
|
||
- **G5 Overstated caution.** Unproven negative certainty is recorded in `ambiguity`
|
||
as an observation, not as an unsupported claim or a criterion miss.
|
||
- **G6 Ambiguity.** A reviewer who finds a criterion ambiguous still records a binary
|
||
value and adds an `ambiguity` entry naming the criterion. Ambiguity counts are
|
||
reported per criterion.
|
||
|
||
Award 1 only for an explicit, correct obligation without a contradicting
|
||
recommendation, with a quote or locator and a reason. No partial credit or reward for
|
||
names. Null means unavailable or unreviewed, never zero. Unsafe recommendations are
|
||
listed separately; rejecting an unsafe alternative is not an unsafe claim.
|
||
|
||
## Execution and budget
|
||
|
||
As replication-02: freeze and commit before inference; 12 independent single-request
|
||
sessions; alternated condition order; independently randomized U/V labels; prompts
|
||
contain only the instructions, one pack and one brief; same llm-connect profile,
|
||
temperature 0.2 and 1,400-token cap. The runner reserves each attempt before calling,
|
||
enforces an aggregate 12-call ceiling and never retries. Run `make inference-preflight`
|
||
first; a failed preflight means no run. Paid execution requires explicit user
|
||
authorization; a prepared packet implies none.
|
||
|
||
## Review and interpretation
|
||
|
||
Each reviewer works from the masked answers and the reviewer worksheet, which gives
|
||
the plain-language `reviewer_context` for each scenario (written before any answer
|
||
existed) and each criterion with its examples. Interview-mediated review is allowed
|
||
if the mediator supplies no scores or hints. Both sheets are committed before
|
||
unmasking. Disagreements between the two human sheets, and with any model sheet, are
|
||
adjudicated against this rubric, keeping all original judgements.
|
||
|
||
Interpretation rules are unchanged: equal counts tie; a descriptive advantage needs
|
||
more obligations with no more unsafe claims, or fewer unsafe claims with no fewer
|
||
obligations. Six pairs prove neither benefit nor equivalence. Never auto-promote or
|
||
mark M1 passed; T03 weighs the result, reviewer independence and ambiguity counts.
|