frontend-patterns/research/evaluations/language-replication-02

Fresh presentation replication 02

WP-0009 T02 implementation: six new authored variants, six atomic criteria per scenario, two equivalent guidance presentations and twelve isolated prompts. The prior consumed pilot remains separate and unchanged. This replication is not held-out M3 retrieval, a real consumer use or independent human validation.

The protocol defines interpretation and budgets; scenarios define the frozen obligations and prohibited claims. pack-equivalence.json records shared body/source hashes. freeze.json pins all inputs, prompts, assignment and code plus immutable reference version/corpus/package. The public assignment supports auditing but makes future masking procedural.

Commands from the repository root:

make replication-check replication-test

The user authorized up to twelve llm-connect calls at a requested 1,400-output-token cap per call with no retries. Execution occurs only after a committed freeze:

PYTHONDONTWRITEBYTECODE=1 ../infospace-bench/.venv/bin/python scripts/language_replication.py run --max-calls 12

The runner reserves attempts before requests. A second invocation skips every reserved session and can call only unstarted sessions within the aggregate ceiling; it never retries a failed/interrupted attempt. Raw receipts preserve usage, model identity, output hashes and failed responses. Scoring reads masked/ before opening the key. Copy the empty scores-template.json, disclose reviewer role and prior exposure, record binary scores with quotes/locators and reasons, and list unsupported claims separately. Null means unavailable or unreviewed; it is never a default zero.

python3 scripts/language_replication.py scores-check --scores <review-file.json> --complete
python3 scripts/language_replication.py summarize --scores <review-file.json>

Validation checks structure and retained quotations/hashes; it does not assess semantic correctness or independently authenticate a declared reviewer. Keep initial ratings, disagreements and adjudication. A failed pair is excluded from quality counts, and no score automatically completes human review or M1. Actual execution/editorial outcomes are recorded here after the authorized run; independent-human scores remain pending.

Authorized execution outcome — 2026-10-04

Preregistration commit: 1e2741a95909341441da05f86a4b6721eae29b60, synced before execution. Twelve attempts were reserved and made once each; all failed, with no retained answers. Raw records retain every reservation and receipt. Execution summary records elapsed request time and failed health checks inside/outside the sandbox. The runner records InfospaceError, not its subtype; endpoint unavailability was verified afterward. Provider execution, token usage and billing are unknown. No retries were made.

Availability review marks all answers unavailable and all atomic ratings null. No masked answer review occurred; reading the first receipt exposed its assignment, which is disclosed. Comparison summary excludes all six pairs. Its zero totals have zero denominators and are inventory counts, not quality scores or evidence of equivalence. Neither presentation has an observed result. Original pilot-01 evidence remains unchanged.

The packet and tested execution/scoring tools are usable, but WP-0009 T02 is waiting for a usable fresh replication. Do not reuse the reserved run identities or modify the frozen packet. Any future execution needs service restoration, a new preregistered run identity and explicit fresh authorization. Human validation remains WP-0009 T01; consumer discovery/integration remains WP-0008. LEARN-0019 retains this failure rather than dropping it from research history; it contributes no pattern-quality conclusion.