tegwick
5da8b2405c
Assistant: codex Assistant-Model: gpt-6.1-sol Assistant-Session: 01a1045b-933d-7aa1-b2db-104557487a79
68 lines
3.8 KiB
Markdown
68 lines
3.8 KiB
Markdown
# Fresh presentation replication 02
|
|
|
|
WP-0009 T02 implementation: six new authored variants, six atomic criteria per
|
|
scenario, two equivalent guidance presentations and twelve isolated prompts.
|
|
The prior consumed pilot remains separate and unchanged. This replication is not
|
|
held-out M3 retrieval, a real consumer use or independent human validation.
|
|
|
|
The [protocol](protocol.md) defines interpretation and budgets;
|
|
[scenarios](scenarios.json) define the frozen obligations and prohibited claims.
|
|
`pack-equivalence.json` records shared body/source hashes. `freeze.json` pins all
|
|
inputs, prompts, assignment and code plus immutable reference version/corpus/package.
|
|
The public assignment supports auditing but makes future masking procedural.
|
|
|
|
Commands from the repository root:
|
|
|
|
```bash
|
|
make replication-check replication-test
|
|
```
|
|
|
|
The user authorized up to twelve llm-connect calls at a requested 1,400-output-token
|
|
cap per call with no retries. Execution occurs only after a committed freeze:
|
|
|
|
```bash
|
|
PYTHONDONTWRITEBYTECODE=1 ../infospace-bench/.venv/bin/python scripts/language_replication.py run --max-calls 12
|
|
```
|
|
|
|
The runner reserves attempts before requests. A second invocation skips every reserved
|
|
session and can call only unstarted sessions within the aggregate ceiling; it never
|
|
retries a failed/interrupted attempt. Raw receipts preserve usage, model identity,
|
|
output hashes and failed responses. Scoring reads `masked/` before opening the key.
|
|
Copy the empty `scores-template.json`, disclose reviewer role and prior exposure,
|
|
record binary scores with quotes/locators and reasons, and list unsupported claims
|
|
separately. Null means unavailable or unreviewed; it is never a default zero.
|
|
|
|
```bash
|
|
python3 scripts/language_replication.py scores-check --scores <review-file.json> --complete
|
|
python3 scripts/language_replication.py summarize --scores <review-file.json>
|
|
```
|
|
|
|
Validation checks structure and retained quotations/hashes; it does not assess semantic
|
|
correctness or independently authenticate a declared reviewer. Keep initial ratings,
|
|
disagreements and adjudication. A failed pair is excluded from quality counts, and no
|
|
score automatically completes human review or M1. Actual execution/editorial outcomes
|
|
are recorded here after the authorized run; independent-human scores remain pending.
|
|
|
|
## Authorized execution outcome — 2026-10-04
|
|
|
|
Preregistration commit: `1e2741a95909341441da05f86a4b6721eae29b60`, synced before
|
|
execution. Twelve attempts were reserved and made once each; **all failed**, with
|
|
no retained answers. [Raw records](raw/) retain every reservation and receipt.
|
|
[Execution summary](execution-summary.json) records elapsed request time and failed
|
|
health checks inside/outside the sandbox. The runner records `InfospaceError`, not
|
|
its subtype; endpoint unavailability was verified afterward. Provider execution,
|
|
token usage and billing are unknown. No retries were made.
|
|
|
|
[Availability review](availability-review.json) marks all answers unavailable and
|
|
all atomic ratings null. No masked answer review occurred; reading the first receipt
|
|
exposed its assignment, which is disclosed. [Comparison summary](comparison-summary.json)
|
|
excludes all six pairs. Its zero totals have zero denominators and are inventory
|
|
counts, not quality scores or evidence of equivalence. Neither presentation has an
|
|
observed result. Original pilot-01 evidence remains unchanged.
|
|
|
|
The packet and tested execution/scoring tools are usable, but WP-0009 T02 is waiting
|
|
for a usable fresh replication. Do not reuse the reserved run identities or modify
|
|
the frozen packet. Any future execution needs service restoration, a new preregistered
|
|
run identity and explicit fresh authorization. Human validation remains WP-0009 T01;
|
|
consumer discovery/integration remains WP-0008. LEARN-0019 retains this failure rather
|
|
than dropping it from research history; it contributes no pattern-quality conclusion.
|