coulomb/frontend-patterns
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Preregister fresh atomic-criteria replication with 12-call budget

Browse source 
Assistant: codex
Assistant-Model: gpt-6.1-sol
Assistant-Session: 01a1045b-933d-7aa1-b2db-104557487a79
This commit is contained in:
tegwick 2026-10-04 11:08:31 +02:00
parent bd85fcd179
commit 1e2741a959
30 changed files with 2229 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

44
research/evaluations/language-replication-02/README.md Normal file
View file

@ -0,0 +1,44 @@
# Fresh presentation replication 02
WP-0009 T02 implementation: six new authored variants, six atomic criteria per
scenario, two equivalent guidance presentations and twelve isolated prompts.
The prior consumed pilot remains separate and unchanged. This replication is not
held-out M3 retrieval, a real consumer use or independent human validation.
The [protocol](protocol.md) defines interpretation and budgets;
[scenarios](scenarios.json) define the frozen obligations and prohibited claims.
`pack-equivalence.json` records shared body/source hashes. `freeze.json` pins all
inputs, prompts, assignment and code plus immutable reference version/corpus/package.
The public assignment supports auditing but makes future masking procedural.
Commands from the repository root:
```bash
make replication-check replication-test
```
The user authorized up to twelve llm-connect calls at a requested 1,400-output-token
cap per call with no retries. Execution occurs only after a committed freeze:
```bash
PYTHONDONTWRITEBYTECODE=1 ../infospace-bench/.venv/bin/python scripts/language_replication.py run --max-calls 12
```
The runner reserves attempts before requests. A second invocation skips every reserved
session and can call only unstarted sessions within the aggregate ceiling; it never
retries a failed/interrupted attempt. Raw receipts preserve usage, model identity,
output hashes and failed responses. Scoring reads `masked/` before opening the key.
Copy the empty `scores-template.json`, disclose reviewer role and prior exposure,
record binary scores with quotes/locators and reasons, and list unsupported claims
separately. Null means unavailable or unreviewed; it is never a default zero.
```bash
python3 scripts/language_replication.py scores-check --scores <review-file.json> --complete
python3 scripts/language_replication.py summarize --scores <review-file.json>
```
Validation checks structure and retained quotations/hashes; it does not assess semantic
correctness or independently authenticate a declared reviewer. Keep initial ratings,
disagreements and adjudication. A failed pair is excluded from quality counts, and no
score automatically completes human review or M1. Actual execution/editorial outcomes
are recorded here after the authorized run; independent-human scores remain pending.