87 lines
5.5 KiB
Markdown
87 lines
5.5 KiB
Markdown
|
# Fresh atomic-criteria replication protocol — 2026-10-04
|
|||
|
|
|
|||
|
|
Owner: FEP-WP-0009 T02. Author/reviewer: Codex, author-model; no independent human
|
|||
|
|
validation. Preparation uses no model calls. This is a fresh authored scenario
|
|||
|
|
replication of the presentation question, not held-out M3 retrieval or production use.
|
|||
|
|
The preparing author has seen the original pilot and rubric; evaluated solver sessions
|
|||
|
|
have not yet received these six new briefs. Old pilot-01 materials/results stay fixed.
|
|||
|
|
|
|||
|
|
## Question and frozen scope
|
|||
|
|
|
|||
|
|
Does named organization change constraint/recovery recommendations compared with the
|
|||
|
|
same guidance under neutral numbered headings, on six fresh variants of the existing
|
|||
|
|
failure mechanisms? All eight bodies, introductory caveat, order and closing relationships
|
|||
|
|
are byte-identical. Only headings vary. Both conditions receive the same facts.
|
|||
|
|
Guidance summarizes the eight core hypotheses from immutable preview 0.1; exact
|
|||
|
|
corpus/packaging/manifest pins and source candidate hashes are in freeze.json.
|
|||
|
|
No claim about subsequent live research, general model competence or independent
|
|||
|
|
full-pattern recurrence follows. A protocol check establishes traceability, not semantic
|
|||
|
|
truth. Human reviewers may flag rubric/guidance ambiguity rather than force a score.
|
|||
|
|
|
|||
|
|
## Atomic criteria and scope limits
|
|||
|
|
|
|||
|
|
Six scenarios have six binary atomic obligations each: 36 per condition. Award 1
|
|||
|
|
only for an explicit correct obligation without a contradicting recommendation;
|
|||
|
|
otherwise 0 with an explanation. No partial credit, reward for names, inferred
|
|||
|
|
implementation guarantee or multi-clause implicit completion. Keep scores null for
|
|||
|
|
unavailable/unreviewed answers, never default them to zero. Quotes or answer locators
|
|||
|
|
and reasons are required for reviewed values. Unsafe recommendations are listed
|
|||
|
|
separately; rejecting an unsafe alternative is not an unsafe claim. Missing an
|
|||
|
|
obligation is not automatically an unsafe claim.
|
|||
|
|
|
|||
|
|
These scenarios target policy-dependent preservation/export, cancellation versus
|
|||
|
|
reversal, atomic supersession, backup/restore, expired receipt lookup and accepted
|
|||
|
|
operation versus projection recovery. They deliberately challenge guidance shared by
|
|||
|
|
both packs. Thirty-six obligations are not directly comparable with pilot-01's 24;
|
|||
|
|
report individual scenarios/criteria, contradictions, failures and ambiguity, not an
|
|||
|
|
aggregate improvement claim across changed protocols.
|
|||
|
|
|
|||
|
|
## Execution and budget
|
|||
|
|
|
|||
|
|
Freeze and commit inputs, assignment and exact isolated prompts before any inference.
|
|||
|
|
Six pairs × two conditions = 12 independent single-request sessions. Alternate the
|
|||
|
|
condition run first (three each) and independently randomize U/V labels before execution.
|
|||
|
|
Each prompt contains only common instructions, one pack and one brief: no criteria,
|
|||
|
|
prohibited list, other answer, assignment key or conversation history. No tools are
|
|||
|
|
available. Same llm-connect profile, temperature 0.2 and 1,400 requested output-token cap
|
|||
|
|
for every session. Paid execution requires explicit user call-budget authorization;
|
|||
|
|
no authorization is implied by a prepared prompt or command. No dollar ceiling is
|
|||
|
|
inferred from a call cap. Record actual model/usage, latency, output cap and failures.
|
|||
|
|
|
|||
|
|
No optional stopping, regeneration or selective retries. Reserve each attempt in its
|
|||
|
|
immutable session directory before calling. A failed response remains failed; its pair
|
|||
|
|
cannot enter fulfilled-obligation comparison. Length violations are recorded separately.
|
|||
|
|
Do not repair prompts or answers after observing results; corrections get a new packet.
|
|||
|
|
No worker or paid execution is enabled by preparation.
|
|||
|
|
|
|||
|
|
The user explicitly authorized up to 12 capped calls with no retries on October 4.
|
|||
|
|
The runner enforces an aggregate 12-reservation ceiling, refuses a second attempt for
|
|||
|
|
any reserved session, and can resume only unstarted sessions. Commit verification
|
|||
|
|
checks the packet and runner before any request. Report failures or interruption rather
|
|||
|
|
than silently regenerate. Active research effort for preparation is unknown because
|
|||
|
|
no timer was started; do not infer it from commits or elapsed chat time. Execution
|
|||
|
|
seconds are measured separately per request; cost/tokens only from retained receipts.
|
|||
|
|
|
|||
|
|
## Scoring, masking and interpretation
|
|||
|
|
|
|||
|
|
The sheet identifies U/V answers without condition metadata. Persist first ratings
|
|||
|
|
before reading assignments; retain exposure disclosures and ambiguous criteria. The
|
|||
|
|
public Git assignment makes masking procedural, not access-enforced, and wording may
|
|||
|
|
reveal a condition. A declaration of human authorship is provenance to review, not an
|
|||
|
|
automatic identity/independence verification. Model review does not fill the human gate.
|
|||
|
|
|
|||
|
|
Use a complete frozen rubric with evidence-bearing binary ratings and separate unsafe
|
|||
|
|
claims. Equal counts are a tie, fewer unsafe claims with no fewer fulfilled obligations
|
|||
|
|
or more obligations with no more unsafe claims is a descriptive advantage; conflicting
|
|||
|
|
movements are mixed. Six pairs are not statistical proof of benefit, equivalence or a
|
|||
|
|
plateau. Service latency is not human decision time. Retain hypotheses unless stronger
|
|||
|
|
independent evidence warrants review; never auto-promote or mark M1 passed.
|
|||
|
|
|
|||
|
|
## Handoff
|
|||
|
|
|
|||
|
|
`make replication-check replication-test` verifies the packet and score-record structure
|
|||
|
|
without inference. WP-0009 T02 remains open until authorized execution, retained outputs
|
|||
|
|
and scoped replication review. T01 retains independent human pilot-01 ratings; T03
|
|||
|
|
retains M1 re-assessment after both native evidence obligations. No empirical learning
|
|||
|
|
cycle or consumer outcome is produced by this preparation.
|