frontend-patterns/research/evaluations/language-pilot-01/submitted-review-01/adjudication.md

Submitted review and adjudication — 2026-09-26

The user's completed worksheet is incorporated as a user-submitted model-assisted review. Its declared author is ChatGPT (GPT-5.6 Sol), explicitly not a human validator. It reports no assignment-key access, but exposure to the aggregate pilot summary before scoring. That exposure prevents a claim of fully blind review. No new model calls were made to import or adjudicate this submission; the external review's execution route was not supplied and is not inferred.

The original worksheet is preserved byte-for-byte with its hash and provenance. All twelve answer texts and all 24 rubric rows match the frozen materials. Parsed scores preserve the submitted ratings; adjudication data records decisions separately. Neither the original Codex ratings nor the frozen pilot results have been overwritten.

Four rating differences

Codex adjudication is unmasked and not independent human validation. All four proposed changes are accepted under the existing explicit-coverage/no-partial-credit rule. The other 44 of 48 binary ratings agree. This is descriptive agreement between two model assessments, not evidence of human reliability.

Answer / criterion	Initial → submitted → adjudicated	Reason
S02-V / Q	1 → 0 → 0	Retention alone omits lookup or recovery of the old operation/device identity required by the compound criterion.
S04-U / R	1 → 0 → 0	Escalation is present, but a separate authorized remote reversal after confirmed acceptance is not.
S04-U / Q	1 → 0 → 0	Lookup and retention are named; the remote cancellation contract is omitted.
S04-V / Q	1 → 0 → 0	Duplicate handling/retention omit lookup/history and cancellation semantics.

The initial scoring was too generous toward partial answers on these compound criteria. Do not change the rubric after seeing this result. In a future preregistered pilot, split compound criteria into atomic obligations so omissions and disagreements are easier to audit; use fresh scenarios rather than rescoring to a new rubric here.

Updated descriptive totals

Measure	Named patterns	Equivalent prose
Required obligations met	18 / 24	18 / 24
Missed obligations	6	6
Constraint items (C1 + C2)	11 / 12	11 / 12
Recovery items (R)	4 / 6	3 / 6
Missing-contract items (Q)	3 / 6	4 / 6

Per scenario, obligations met (patterns/prose): S01 4/3, S02 3/4, S03 3/3, S04 2/2, S05 3/3, S06 3/3. These totals are not pair-dominance results, because that rule also considers unsafe claims. Equal totals do not establish equivalence or remove the uncertainty from a six-pair, single-model pilot.

Narrative flags and disagreements

• S01-U: accept the additional caution about “stale replica data.” Superseding an earlier request is a valid local gate and still earns R credit; it does not prove the returned data is older. Apply that distinction to both presentations. The initial Codex review flagged only V's unsupported stale assertion.
• S03: both reviews identify policy assumptions before the missing policy is known. Retain R=0 for both. For V, local export is phrased tentatively and the final sentence conditions options on policy; mandatory retention is the clearer defect. Do not silently turn each phrase into another independently counted violation.
• S04-V: accept the unsafe-sequencing concern about a fresh identity before reconciliation. Its explicit duplicate warning still meets the narrow C2 criterion, but it does not provide a safe recovery sequence; R remains 0. No duplicate-free promise was made. The initial record noted this ambiguity without counting an additional unsafe claim; preserve both interpretations.
• S05: both miss an explicit atomic supersession/crash boundary despite discussing receiver preconditions. S06 similarly omits independent backup/restore evidence.
• S06-V: retain the warning that untested recovery was described as impossible. Rejecting a guarantee does not justify a categorical claim of incapability.

The submission gives narrative flags, not a numerical count with a common unit for compound claims. Preserve them qualitatively rather than inventing a new unsafe-claim count. The original two-per-condition counts and original pair advantages remain historical initial findings, not the adjudicated conclusion.

Conclusion and follow-up

No presentation benefit has been demonstrated. The original two-obligation aggregate difference disappears under the stricter submitted ratings. Retain the eight pattern hypotheses; do not promote them or claim equivalence with prose.

This satisfies incorporation of the user's submitted review. It does not satisfy the preregistered independent-human-validation gate, given the worksheet's own author and exposure disclosures. T03 keeps that evidence limit explicit rather than treating submission by a human as human authorship of the judgments.

Next useful revision targets are shared by both presentations: make policy-dependent recommendations conditional throughout, specify cancellation/reconciliation/reversal separately, and spell out atomic supersession and independent backup obligations. Change guidance in a new version after this frozen pilot; these answers are consumed evaluation material. A later usefulness study should use atomic criteria, new briefs and an independent reviewer.