tegwick
1e2741a959
Assistant: codex Assistant-Model: gpt-6.1-sol Assistant-Session: 01a1045b-933d-7aa1-b2db-104557487a79 |
||
|---|---|---|
| .. | ||
| requests | ||
| assignment.json | ||
| freeze.json | ||
| guidance.json | ||
| instructions.txt | ||
| pack-equivalence.json | ||
| pack-patterns.md | ||
| pack-prose.md | ||
| protocol.md | ||
| README.md | ||
| scenarios.json | ||
| scores-template.json | ||
Fresh presentation replication 02
WP-0009 T02 implementation: six new authored variants, six atomic criteria per scenario, two equivalent guidance presentations and twelve isolated prompts. The prior consumed pilot remains separate and unchanged. This replication is not held-out M3 retrieval, a real consumer use or independent human validation.
The protocol defines interpretation and budgets;
scenarios define the frozen obligations and prohibited claims.
pack-equivalence.json records shared body/source hashes. freeze.json pins all
inputs, prompts, assignment and code plus immutable reference version/corpus/package.
The public assignment supports auditing but makes future masking procedural.
Commands from the repository root:
make replication-check replication-test
The user authorized up to twelve llm-connect calls at a requested 1,400-output-token cap per call with no retries. Execution occurs only after a committed freeze:
PYTHONDONTWRITEBYTECODE=1 ../infospace-bench/.venv/bin/python scripts/language_replication.py run --max-calls 12
The runner reserves attempts before requests. A second invocation skips every reserved
session and can call only unstarted sessions within the aggregate ceiling; it never
retries a failed/interrupted attempt. Raw receipts preserve usage, model identity,
output hashes and failed responses. Scoring reads masked/ before opening the key.
Copy the empty scores-template.json, disclose reviewer role and prior exposure,
record binary scores with quotes/locators and reasons, and list unsupported claims
separately. Null means unavailable or unreviewed; it is never a default zero.
python3 scripts/language_replication.py scores-check --scores <review-file.json> --complete
python3 scripts/language_replication.py summarize --scores <review-file.json>
Validation checks structure and retained quotations/hashes; it does not assess semantic correctness or independently authenticate a declared reviewer. Keep initial ratings, disagreements and adjudication. A failed pair is excluded from quality counts, and no score automatically completes human review or M1. Actual execution/editorial outcomes are recorded here after the authorized run; independent-human scores remain pending.