tegwick
f8c3ce3938
Owner decisions D1-D3 applied; execution not authorized. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 182330@bnt-lap001 Assistant-Session: 242866cf-789d-4e24-ac5e-0a1d7b47a701
80 lines
4 KiB
Markdown
80 lines
4 KiB
Markdown
# Replication 03 — draft criteria revision
|
|
|
|
Status: **frozen 2026-10-20, not authorized for execution.** Owner: FEP-WP-0009 T02.
|
|
Prepared 2026-10-20 by Claude Code (author-model) at the owner's request, using no
|
|
model calls. It applies the results of the pilot-01 human review and adjudication
|
|
([review](../language-pilot-01/human-review/review-bernd-worsch-2026-10-19.json),
|
|
[adjudication](../language-pilot-01/human-review/disagreements-2026-10-19.json)).
|
|
Replication-02 and its spent run identities stay unchanged.
|
|
|
|
## Why a revision
|
|
|
|
Replication-02 already uses six single-obligation criteria per scenario. The pilot-01
|
|
adjudication found 11 of 16 disagreements were caused by rubric wording, not by the
|
|
reviewers. Four kinds of ambiguity recur, and several replication-02 criteria still
|
|
admit them:
|
|
|
|
| Ambiguity found in pilot-01 | Example | Replication-02 criteria affected |
|
|
| --- | --- | --- |
|
|
| Generic vs specific contract naming | S01/S02/S04 Q | S01-A3, S02-A6, S03-A5, S05-A6, S06-A6 |
|
|
| Condition stated elsewhere vs in the recommendation | S03 R | S01-A4, S01-A5 |
|
|
| End state vs stated atomic step | S05 C1 | S03-A1 |
|
|
| Flat assertion of an unknown fact | S01-V "stale"; S03-V "must be retained" | S01-A6, S06-A5 |
|
|
|
|
The reviewer's recurring guidance was to pair abstract wording with a concrete
|
|
example. Each criterion in [scenarios.draft.json](scenarios.draft.json) therefore
|
|
carries `qualifies` and `does_not_qualify` examples, a `placement` and a
|
|
`change_from_r02` note. Briefs and prohibited lists are copied unchanged.
|
|
|
|
## Draft scoring rules
|
|
|
|
- **G1 Placement.** A `recommendation` criterion is met only inside the recommended
|
|
state or action. A matching statement only in a later questions section does not count.
|
|
An `anywhere` criterion may be met in any section.
|
|
- **G2 Named subject.** A contract-identification criterion requires the guarantee's
|
|
subject (for example lookup, replay window, read-after-write). A generic "define the
|
|
contract" does not qualify. Listed alternatives joined by "or" are each sufficient.
|
|
- **G3 Examples are illustrative.** Equivalent wording qualifies; the examples show
|
|
where the line is, not required phrases.
|
|
- **G4 Unknown facts.** Asserting an unspecified fact as known (stale, permanent,
|
|
succeeded) is recorded as an unsupported claim.
|
|
- **G5 Overstated caution** (for example "cannot recover" where it is only untested)
|
|
is recorded as a separate observation, not an unsupported claim, following the
|
|
owner's S06-V adjudication.
|
|
- **G6 Ambiguity is a valid result.** A reviewer may mark a criterion ambiguous (X)
|
|
instead of forcing a score; ambiguity counts are reported per criterion.
|
|
|
|
## Reviewer support
|
|
|
|
The pilot-01 human review worked only in interview form, with plain-language context
|
|
per scenario. A replication-03 worksheet should include that context, written before
|
|
any answers exist, without hints about any answer.
|
|
|
|
## Owner decisions (2026-10-20)
|
|
|
|
D1 reuse briefs: yes. D2 overstated-caution rule G5: yes. D3 reviewers: owner and an
|
|
external human reviewer, independently. D4 execution: not yet authorized. Details and
|
|
the binding rules are in the [protocol](protocol.md); G6 there supersedes the draft
|
|
wording above (a binary value is still required alongside an ambiguity entry).
|
|
|
|
## Commands
|
|
|
|
From the repository root, without inference:
|
|
|
|
```bash
|
|
make replication-03-check replication-03-test
|
|
```
|
|
|
|
Execution needs, in order: `make inference-preflight` passing against a reachable
|
|
llm-connect, fresh explicit authorization of up to 12 capped calls with no retries,
|
|
and a committed freeze. Then:
|
|
|
|
```bash
|
|
PYTHONDONTWRITEBYTECODE=1 ../infospace-bench/.venv/bin/python scripts/language_replication_03.py run --max-calls 12
|
|
```
|
|
|
|
After answers exist, build the reviewer worksheet from `masked/` plus each scenario's
|
|
`reviewer_context` and criteria examples; each reviewer copies `scores-template.json`.
|
|
`scripts/language_replication_03.py scores-check|summarize --scores <file>` validate
|
|
and summarise a sheet. Replication-02 stays frozen and verifiable with
|
|
`make replication-check`.
|