frontend-patterns/research/evaluations/language-pilot-01/submitted-review-01/adjudication.md

# Submitted review and adjudication — 2026-09-26

The user's completed worksheet is incorporated as a **user-submitted model-assisted
review**. Its declared author is ChatGPT (GPT-5.6 Sol), explicitly not a human
validator. It reports no assignment-key access, but exposure to the aggregate pilot
summary before scoring. That exposure prevents a claim of fully blind review.
No new model calls were made to import or adjudicate this submission; the external
review's execution route was not supplied and is not inferred.

The [original worksheet](worksheet-completed.md) is preserved byte-for-byte with its
[hash and provenance](provenance.json). All twelve answer texts and all 24 rubric
rows match the frozen materials. [Parsed scores](scores.json) preserve the submitted
ratings; [adjudication data](adjudication.json) records decisions separately. Neither
the original Codex ratings nor the frozen pilot results have been overwritten.

## Four rating differences

Codex adjudication is unmasked and not independent human validation. All four proposed
changes are accepted under the existing explicit-coverage/no-partial-credit rule.
The other 44 of 48 binary ratings agree. This is descriptive agreement between two
model assessments, not evidence of human reliability.

| Answer / criterion | Initial → submitted → adjudicated | Reason |
| --- | --- | --- |
| S02-V / Q | 1 → 0 → 0 | Retention alone omits lookup or recovery of the old operation/device identity required by the compound criterion. |
| S04-U / R | 1 → 0 → 0 | Escalation is present, but a separate authorized remote reversal after confirmed acceptance is not. |
| S04-U / Q | 1 → 0 → 0 | Lookup and retention are named; the remote cancellation contract is omitted. |
| S04-V / Q | 1 → 0 → 0 | Duplicate handling/retention omit lookup/history and cancellation semantics. |

The initial scoring was too generous toward partial answers on these compound
criteria. Do not change the rubric after seeing this result. In a future preregistered
pilot, split compound criteria into atomic obligations so omissions and disagreements
are easier to audit; use fresh scenarios rather than rescoring to a new rubric here.

## Updated descriptive totals

| Measure | Named patterns | Equivalent prose |
| --- | ---: | ---: |
| Required obligations met | 18 / 24 | 18 / 24 |
| Missed obligations | 6 | 6 |
| Constraint items (C1 + C2) | 11 / 12 | 11 / 12 |
| Recovery items (R) | 4 / 6 | 3 / 6 |
| Missing-contract items (Q) | 3 / 6 | 4 / 6 |

Per scenario, obligations met (patterns/prose): S01 4/3, S02 3/4, S03 3/3,
S04 2/2, S05 3/3, S06 3/3. These totals are not pair-dominance results, because that
rule also considers unsafe claims. Equal totals do not establish equivalence or
remove the uncertainty from a six-pair, single-model pilot.

## Narrative flags and disagreements

- **S01-U:** accept the additional caution about “stale replica data.” Superseding
  an earlier request is a valid local gate and still earns R credit; it does not
  prove the returned data is older. Apply that distinction to both presentations.
  The initial Codex review flagged only V's unsupported stale assertion.
- **S03:** both reviews identify policy assumptions before the missing policy is
  known. Retain R=0 for both. For V, local export is phrased tentatively and the final
  sentence conditions options on policy; mandatory retention is the clearer defect.
  Do not silently turn each phrase into another independently counted violation.
- **S04-V:** accept the unsafe-sequencing concern about a fresh identity before
  reconciliation. Its explicit duplicate warning still meets the narrow C2 criterion,
  but it does not provide a safe recovery sequence; R remains 0. No duplicate-free
  promise was made. The initial record noted this ambiguity without counting an
  additional unsafe claim; preserve both interpretations.
- **S05:** both miss an explicit atomic supersession/crash boundary despite discussing
  receiver preconditions. S06 similarly omits independent backup/restore evidence.
- **S06-V:** retain the warning that untested recovery was described as impossible.
  Rejecting a guarantee does not justify a categorical claim of incapability.

The submission gives narrative flags, not a numerical count with a common unit for
compound claims. Preserve them qualitatively rather than inventing a new unsafe-claim
count. The original two-per-condition counts and original pair advantages remain
historical initial findings, not the adjudicated conclusion.

## Conclusion and follow-up

No presentation benefit has been demonstrated. The original two-obligation aggregate
difference disappears under the stricter submitted ratings. Retain the eight pattern
hypotheses; do not promote them or claim equivalence with prose.

This satisfies incorporation of the user's submitted review. It does **not** satisfy
the preregistered independent-human-validation gate, given the worksheet's own author
and exposure disclosures. T03 keeps that evidence limit explicit rather than treating
submission by a human as human authorship of the judgments.

Next useful revision targets are shared by both presentations: make policy-dependent
recommendations conditional throughout, specify cancellation/reconciliation/reversal
separately, and spell out atomic supersession and independent backup obligations.
Change guidance in a new version after this frozen pilot; these answers are consumed
evaluation material. A later usefulness study should use atomic criteria, new briefs
and an independent reviewer.