45 lines
2.3 KiB
Markdown
45 lines
2.3 KiB
Markdown
|
# Fresh presentation replication 02
|
||
|
|
|
||
|
|
WP-0009 T02 implementation: six new authored variants, six atomic criteria per
|
||
|
|
scenario, two equivalent guidance presentations and twelve isolated prompts.
|
||
|
|
The prior consumed pilot remains separate and unchanged. This replication is not
|
||
|
|
held-out M3 retrieval, a real consumer use or independent human validation.
|
||
|
|
|
||
|
|
The [protocol](protocol.md) defines interpretation and budgets;
|
||
|
|
[scenarios](scenarios.json) define the frozen obligations and prohibited claims.
|
||
|
|
`pack-equivalence.json` records shared body/source hashes. `freeze.json` pins all
|
||
|
|
inputs, prompts, assignment and code plus immutable reference version/corpus/package.
|
||
|
|
The public assignment supports auditing but makes future masking procedural.
|
||
|
|
|
||
|
|
Commands from the repository root:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
make replication-check replication-test
|
||
|
|
```
|
||
|
|
|
||
|
|
The user authorized up to twelve llm-connect calls at a requested 1,400-output-token
|
||
|
|
cap per call with no retries. Execution occurs only after a committed freeze:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
PYTHONDONTWRITEBYTECODE=1 ../infospace-bench/.venv/bin/python scripts/language_replication.py run --max-calls 12
|
||
|
|
```
|
||
|
|
|
||
|
|
The runner reserves attempts before requests. A second invocation skips every reserved
|
||
|
|
session and can call only unstarted sessions within the aggregate ceiling; it never
|
||
|
|
retries a failed/interrupted attempt. Raw receipts preserve usage, model identity,
|
||
|
|
output hashes and failed responses. Scoring reads `masked/` before opening the key.
|
||
|
|
Copy the empty `scores-template.json`, disclose reviewer role and prior exposure,
|
||
|
|
record binary scores with quotes/locators and reasons, and list unsupported claims
|
||
|
|
separately. Null means unavailable or unreviewed; it is never a default zero.
|
||
|
|
|
||
|
|
```bash
|
||
|
|
python3 scripts/language_replication.py scores-check --scores <review-file.json> --complete
|
||
|
|
python3 scripts/language_replication.py summarize --scores <review-file.json>
|
||
|
|
```
|
||
|
|
|
||
|
|
Validation checks structure and retained quotations/hashes; it does not assess semantic
|
||
|
|
correctness or independently authenticate a declared reviewer. Keep initial ratings,
|
||
|
|
disagreements and adjudication. A failed pair is excluded from quality counts, and no
|
||
|
|
score automatically completes human review or M1. Actual execution/editorial outcomes
|
||
|
|
are recorded here after the authorized run; independent-human scores remain pending.
|