diff --git a/TREE.md b/TREE.md index ce10373..b9aa79a 100644 --- a/TREE.md +++ b/TREE.md @@ -330,6 +330,12 @@ research/evaluations/language-pilot-01/raw/r12/receipt.json research/evaluations/language-pilot-01/raw/r12/started.json research/evaluations/language-pilot-01/run.py research/evaluations/language-pilot-01/scenarios.json +research/evaluations/language-pilot-01/submitted-review-01/README.md +research/evaluations/language-pilot-01/submitted-review-01/adjudication.json +research/evaluations/language-pilot-01/submitted-review-01/adjudication.md +research/evaluations/language-pilot-01/submitted-review-01/provenance.json +research/evaluations/language-pilot-01/submitted-review-01/scores.json +research/evaluations/language-pilot-01/submitted-review-01/worksheet-completed.md research/evaluations/language-pilot-01/summarize.py research/evaluations/language-pilot-01/summary.json research/examples/SchemaEvidence.md diff --git a/research/README.md b/research/README.md index f7798ec..ea46131 100644 --- a/research/README.md +++ b/research/README.md @@ -36,3 +36,5 @@ Latest implementation evidence: [frontend recovery](reviews/FrontendRecoveryImpl Executable evidence: [Matrix SDK recovery experiment](experiments/matrix-recovery/README.md), with a synthetic receiver and explicit limits. Evaluation: [matched presentation pilot](evaluations/language-pilot-01/README.md) completed twelve llm-connect sessions; results inconclusive, human review pending. + +Pilot follow-up: [submitted review adjudication](evaluations/language-pilot-01/submitted-review-01/adjudication.md) gives 18/24 obligations in both conditions; model-assisted review, not independent human validation. diff --git a/research/catalog/PatternCandidateCatalog.md b/research/catalog/PatternCandidateCatalog.md index b2c54d6..7f19c61 100644 --- a/research/catalog/PatternCandidateCatalog.md +++ b/research/catalog/PatternCandidateCatalog.md @@ -87,3 +87,8 @@ scoring found 21/24 obligations met with named patterns versus 19/24 with equiva prose, and two unsupported claims in each. Inconclusive; human review pending. Retain all eight hypotheses, no new pattern or promotion. Do not mistake a small presentation difference for proof of recurrence or improved architecture decisions. + +2026-09-26 adjudication: the user-submitted model-assisted worksheet supports four +rating corrections; both presentations now meet 18/24 obligations. The original +21/24 versus 19/24 figures above are historical. [Adjudication](../evaluations/language-pilot-01/submitted-review-01/adjudication.md) +retains all eight hypotheses and the unfulfilled independent-human evidence gate. diff --git a/research/evaluations/language-pilot-01/README.md b/research/evaluations/language-pilot-01/README.md index 3cbacd8..dc45c71 100644 --- a/research/evaluations/language-pilot-01/README.md +++ b/research/evaluations/language-pilot-01/README.md @@ -5,6 +5,12 @@ condition-masked Codex scoring found a small descriptive difference, with errors both presentations. Independent human review remains pending. No canonical admission or claim of proven decision usefulness follows. +Update: the [user-submitted review and adjudication](submitted-review-01/adjudication.md) +changes four binary ratings, giving **18/24 obligations met in each condition**. +The worksheet declares model authorship and exposure to the aggregate summary; +independent human validation is still unfulfilled. The figures below preserve the +initial scoring, not the latest adjudicated totals. + For independent scoring, begin with [the human handoff](human-review/README.md) before reading the results or assignment key. The scorer who prepared this report also authored the materials, so masking does not make the assessment independent. diff --git a/research/evaluations/language-pilot-01/human-review/README.md b/research/evaluations/language-pilot-01/human-review/README.md index ff09d14..264fe2f 100644 --- a/research/evaluations/language-pilot-01/human-review/README.md +++ b/research/evaluations/language-pilot-01/human-review/README.md @@ -1,6 +1,8 @@ # Human review handoff -Status: ready; all twelve masked answers are available; no human scores yet. +Status: a user-submitted model-assisted worksheet has been incorporated separately. +It declares model authorship, so no independent human scores are recorded. The blank +materials below remain available for that distinct validation step. Start here without opening assignment.json, raw/, results or editorial scores. Read [the scoring rules](../protocol.md) and [the scenarios/rubric](../scenarios.json). diff --git a/research/evaluations/language-pilot-01/submitted-review-01/README.md b/research/evaluations/language-pilot-01/submitted-review-01/README.md new file mode 100644 index 0000000..3623590 --- /dev/null +++ b/research/evaluations/language-pilot-01/submitted-review-01/README.md @@ -0,0 +1,9 @@ +# Submitted worksheet record + +[Original completed worksheet](worksheet-completed.md), preserved byte-for-byte. +Its instructions link resolves here; the original [scoring protocol](../protocol.md) +and [human handoff instructions](../human-review/README.md) remain available. + +Read [adjudication](adjudication.md) for the four accepted score changes, narrative +flags and provenance limits. [Provenance](provenance.json) records the original hash +and declared model authorship; [scores](scores.json) contain the parsed ratings. diff --git a/research/evaluations/language-pilot-01/submitted-review-01/adjudication.json b/research/evaluations/language-pilot-01/submitted-review-01/adjudication.json new file mode 100644 index 0000000..e9cc60e --- /dev/null +++ b/research/evaluations/language-pilot-01/submitted-review-01/adjudication.json @@ -0,0 +1,76 @@ +{ + "adjudicator": "Codex", + "adjudicated_at": "2026-09-26", + "condition_masked": false, + "independent_human_validation": false, + "original_scores_preserved": "../editorial-scores-masked.json", + "submitted_scores_preserved": "scores.json", + "source_sha256": "9bb6f4c87d7220ba4d8b992590e73c6d7bc7b0e2d4e4d4702f24cd34d1aa6e01", + "binary_ratings_compared": 48, + "unchanged_ratings": 44, + "changed_ratings": [ + { + "answer": "S02-V", + "criterion": "Q", + "original": 1, + "submitted": 0, + "condition": "patterns", + "disposition": "accept submitted score", + "reason": "The rubric requires lookup plus retention, or recovery of the old operation/device identity. Retention alone is only partial coverage." + }, + { + "answer": "S04-U", + "criterion": "R", + "original": 1, + "submitted": 0, + "condition": "patterns", + "disposition": "accept submitted score", + "reason": "Provider escalation is present, but the compound criterion also requires distinguishing an authorized remote reversal if K was accepted." + }, + { + "answer": "S04-U", + "criterion": "Q", + "original": 1, + "submitted": 0, + "condition": "patterns", + "disposition": "accept submitted score", + "reason": "Lookup and retention are mentioned, but the required remote cancellation contract is omitted." + }, + { + "answer": "S04-V", + "criterion": "Q", + "original": 1, + "submitted": 0, + "condition": "prose", + "disposition": "accept submitted score", + "reason": "Duplicate handling and retention are mentioned, but lookup/history and remote cancellation semantics are omitted." + } + ], + "conditions": { + "patterns": { + "criteria_met": { + "C1": 5, + "C2": 6, + "R": 4, + "Q": 3 + }, + "requirements_possible": 24, + "requirements_met": 18, + "missed_requirements": 6 + }, + "prose": { + "criteria_met": { + "C1": 5, + "C2": 6, + "R": 3, + "Q": 4 + }, + "requirements_possible": 24, + "requirements_met": 18, + "missed_requirements": 6 + } + }, + "unsupported_claims": "Narrative annotations preserved and adjudicated in adjudication.md; no submitted numeric count or uniform counting unit inferred. Original counts remain historical.", + "pair_advantage_recomputed": false, + "conclusion": "No demonstrated presentation benefit. Equal aggregate obligation counts do not establish equivalence; independent human validation remains pending." +} diff --git a/research/evaluations/language-pilot-01/submitted-review-01/adjudication.md b/research/evaluations/language-pilot-01/submitted-review-01/adjudication.md new file mode 100644 index 0000000..b736e11 --- /dev/null +++ b/research/evaluations/language-pilot-01/submitted-review-01/adjudication.md @@ -0,0 +1,91 @@ +# Submitted review and adjudication — 2026-09-26 + +The user's completed worksheet is incorporated as a **user-submitted model-assisted +review**. Its declared author is ChatGPT (GPT-5.6 Sol), explicitly not a human +validator. It reports no assignment-key access, but exposure to the aggregate pilot +summary before scoring. That exposure prevents a claim of fully blind review. +No new model calls were made to import or adjudicate this submission; the external +review's execution route was not supplied and is not inferred. + +The [original worksheet](worksheet-completed.md) is preserved byte-for-byte with its +[hash and provenance](provenance.json). All twelve answer texts and all 24 rubric +rows match the frozen materials. [Parsed scores](scores.json) preserve the submitted +ratings; [adjudication data](adjudication.json) records decisions separately. Neither +the original Codex ratings nor the frozen pilot results have been overwritten. + +## Four rating differences + +Codex adjudication is unmasked and not independent human validation. All four proposed +changes are accepted under the existing explicit-coverage/no-partial-credit rule. +The other 44 of 48 binary ratings agree. This is descriptive agreement between two +model assessments, not evidence of human reliability. + +| Answer / criterion | Initial → submitted → adjudicated | Reason | +| --- | --- | --- | +| S02-V / Q | 1 → 0 → 0 | Retention alone omits lookup or recovery of the old operation/device identity required by the compound criterion. | +| S04-U / R | 1 → 0 → 0 | Escalation is present, but a separate authorized remote reversal after confirmed acceptance is not. | +| S04-U / Q | 1 → 0 → 0 | Lookup and retention are named; the remote cancellation contract is omitted. | +| S04-V / Q | 1 → 0 → 0 | Duplicate handling/retention omit lookup/history and cancellation semantics. | + +The initial scoring was too generous toward partial answers on these compound +criteria. Do not change the rubric after seeing this result. In a future preregistered +pilot, split compound criteria into atomic obligations so omissions and disagreements +are easier to audit; use fresh scenarios rather than rescoring to a new rubric here. + +## Updated descriptive totals + +| Measure | Named patterns | Equivalent prose | +| --- | ---: | ---: | +| Required obligations met | 18 / 24 | 18 / 24 | +| Missed obligations | 6 | 6 | +| Constraint items (C1 + C2) | 11 / 12 | 11 / 12 | +| Recovery items (R) | 4 / 6 | 3 / 6 | +| Missing-contract items (Q) | 3 / 6 | 4 / 6 | + +Per scenario, obligations met (patterns/prose): S01 4/3, S02 3/4, S03 3/3, +S04 2/2, S05 3/3, S06 3/3. These totals are not pair-dominance results, because that +rule also considers unsafe claims. Equal totals do not establish equivalence or +remove the uncertainty from a six-pair, single-model pilot. + +## Narrative flags and disagreements + +- **S01-U:** accept the additional caution about “stale replica data.” Superseding + an earlier request is a valid local gate and still earns R credit; it does not + prove the returned data is older. Apply that distinction to both presentations. + The initial Codex review flagged only V's unsupported stale assertion. +- **S03:** both reviews identify policy assumptions before the missing policy is + known. Retain R=0 for both. For V, local export is phrased tentatively and the final + sentence conditions options on policy; mandatory retention is the clearer defect. + Do not silently turn each phrase into another independently counted violation. +- **S04-V:** accept the unsafe-sequencing concern about a fresh identity before + reconciliation. Its explicit duplicate warning still meets the narrow C2 criterion, + but it does not provide a safe recovery sequence; R remains 0. No duplicate-free + promise was made. The initial record noted this ambiguity without counting an + additional unsafe claim; preserve both interpretations. +- **S05:** both miss an explicit atomic supersession/crash boundary despite discussing + receiver preconditions. S06 similarly omits independent backup/restore evidence. +- **S06-V:** retain the warning that untested recovery was described as impossible. + Rejecting a guarantee does not justify a categorical claim of incapability. + +The submission gives narrative flags, not a numerical count with a common unit for +compound claims. Preserve them qualitatively rather than inventing a new unsafe-claim +count. The original two-per-condition counts and original pair advantages remain +historical initial findings, not the adjudicated conclusion. + +## Conclusion and follow-up + +No presentation benefit has been demonstrated. The original two-obligation aggregate +difference disappears under the stricter submitted ratings. Retain the eight pattern +hypotheses; do not promote them or claim equivalence with prose. + +This satisfies incorporation of the user's submitted review. It does **not** satisfy +the preregistered independent-human-validation gate, given the worksheet's own author +and exposure disclosures. T03 keeps that evidence limit explicit rather than treating +submission by a human as human authorship of the judgments. + +Next useful revision targets are shared by both presentations: make policy-dependent +recommendations conditional throughout, specify cancellation/reconciliation/reversal +separately, and spell out atomic supersession and independent backup obligations. +Change guidance in a new version after this frozen pilot; these answers are consumed +evaluation material. A later usefulness study should use atomic criteria, new briefs +and an independent reviewer. diff --git a/research/evaluations/language-pilot-01/submitted-review-01/provenance.json b/research/evaluations/language-pilot-01/submitted-review-01/provenance.json new file mode 100644 index 0000000..5c0c03f --- /dev/null +++ b/research/evaluations/language-pilot-01/submitted-review-01/provenance.json @@ -0,0 +1,27 @@ +{ + "submitted_by": "user", + "accepted_as": "user-submitted model-assisted review", + "declared_reviewer": "ChatGPT (GPT-5.6 Sol; model review, not human validation)", + "declared_date": "2026-09-26", + "original_filename": "worksheet-completed.md", + "sha256": "9bb6f4c87d7220ba4d8b992590e73c6d7bc7b0e2d4e4d4702f24cd34d1aa6e01", + "assignment_key_exposure": "declared no", + "prior_results_exposure": "declared aggregate pilot README summary inadvertently visible; detailed scores not opened", + "independent_human_validation": false, + "provider_execution_provenance": "not supplied; no claim about the tool/route used to produce this external worksheet", + "imported_at": "2026-09-26", + "answer_texts_and_rubric_rows_verified": [ + "S01-U", + "S01-V", + "S02-U", + "S02-V", + "S03-U", + "S03-V", + "S04-U", + "S04-V", + "S05-U", + "S05-V", + "S06-U", + "S06-V" + ] +} diff --git a/research/evaluations/language-pilot-01/submitted-review-01/scores.json b/research/evaluations/language-pilot-01/submitted-review-01/scores.json new file mode 100644 index 0000000..d2abf6c --- /dev/null +++ b/research/evaluations/language-pilot-01/submitted-review-01/scores.json @@ -0,0 +1,137 @@ +{ + "provenance": "provenance.json", + "answers": [ + { + "answer": "S01-U", + "scores": { + "C1": 1, + "C2": 1, + "R": 1, + "Q": 1 + }, + "evidence_locator": "worksheet-completed.md#s01", + "scenario_notes": "- **U:** C1/C2 are explicit in \u201cOperation A is confirmed accepted, and draft B is preserved.\u201d R gets credit because the previously started read is explicitly \u201csuperseded and ignored,\u201d which is a valid local request-generation gate rather than opaque-version ordering. Q is explicit in the request for a replica consistency/authority contract. **Unsupported/ambiguous:** \u201cstale replica data\u201d labels the returned data stale even though server/replica freshness is unspecified; I do not use that adjective as the basis for R credit.\n- **V:** C1/C2 are explicit. R is 0 because \u201cthe replica is stale\u201d / \u201cthe stale read\u201d asserts freshness ordering that the scenario does not establish. Q gets credit for explicitly identifying the missing replica consistency contract. **Unsupported claim:** \u201cthe replica is stale.\u201d\n- No prohibited recommendation is made by either answer." + }, + { + "answer": "S01-V", + "scores": { + "C1": 1, + "C2": 1, + "R": 0, + "Q": 1 + }, + "evidence_locator": "worksheet-completed.md#s01", + "scenario_notes": "- **U:** C1/C2 are explicit in \u201cOperation A is confirmed accepted, and draft B is preserved.\u201d R gets credit because the previously started read is explicitly \u201csuperseded and ignored,\u201d which is a valid local request-generation gate rather than opaque-version ordering. Q is explicit in the request for a replica consistency/authority contract. **Unsupported/ambiguous:** \u201cstale replica data\u201d labels the returned data stale even though server/replica freshness is unspecified; I do not use that adjective as the basis for R credit.\n- **V:** C1/C2 are explicit. R is 0 because \u201cthe replica is stale\u201d / \u201cthe stale read\u201d asserts freshness ordering that the scenario does not establish. Q gets credit for explicitly identifying the missing replica consistency contract. **Unsupported claim:** \u201cthe replica is stale.\u201d\n- No prohibited recommendation is made by either answer." + }, + { + "answer": "S02-U", + "scores": { + "C1": 1, + "C2": 1, + "R": 1, + "Q": 1 + }, + "evidence_locator": "worksheet-completed.md#s02", + "scenario_notes": "- **U:** C1 is explicit (\u201cdelivery status is unknown\u201d); C2 rejects fresh-ID automatic retry because it may duplicate; R gives an informed manual reconciliation/resend choice; Q explicitly asks for lookup support and operation-ID retention. No prohibited recommendation.\n- **V:** C1/C2/R are explicit: unknown outcome, duplicate risk, and informed manual resend. Q is 0 under the frozen no-partial-credit rule: it asks about deduplication scope and ID retention, but does not explicitly ask for operation/history lookup or recoverability of the old device/operation identity.\n- No material unsupported guarantee identified." + }, + { + "answer": "S02-V", + "scores": { + "C1": 1, + "C2": 1, + "R": 1, + "Q": 0 + }, + "evidence_locator": "worksheet-completed.md#s02", + "scenario_notes": "- **U:** C1 is explicit (\u201cdelivery status is unknown\u201d); C2 rejects fresh-ID automatic retry because it may duplicate; R gives an informed manual reconciliation/resend choice; Q explicitly asks for lookup support and operation-ID retention. No prohibited recommendation.\n- **V:** C1/C2/R are explicit: unknown outcome, duplicate risk, and informed manual resend. Q is 0 under the frozen no-partial-credit rule: it asks about deduplication scope and ID retention, but does not explicitly ask for operation/history lookup or recoverability of the old device/operation identity.\n- No material unsupported guarantee identified." + }, + { + "answer": "S03-U", + "scores": { + "C1": 1, + "C2": 1, + "R": 0, + "Q": 1 + }, + "evidence_locator": "worksheet-completed.md#s03", + "scenario_notes": "- **U:** C1 distinguishes \u201clocally committed\u201d from an upload \u201crejected due to authorization failure.\u201d C2 rejects retrying under the same unresolved authorization problem. R is 0 because recovery is prescribed before the missing confidentiality policy is known: \u201cThe notes remain editable locally,\u201d the full content \u201cmust be retained,\u201d and the user \u201cshould be presented\u201d with export/save options. Q correctly asks whether retention, view, edit, export or purge is allowed after revocation.\n- **V:** C1/C2 are met. R is 0 because \u201cmust be retained\u201d and proposed local save/export options are asserted before the revocation policy is known, even though the last bullet correctly asks for that policy. Q is met.\n- **Unsupported policy assumptions:** U: \u201cThe notes remain editable locally\u201d; \u201cmust be retained\u201d; export/save as an offered recovery. V: \u201cmust be retained\u201d; possible local save/export before policy resolution." + }, + { + "answer": "S03-V", + "scores": { + "C1": 1, + "C2": 1, + "R": 0, + "Q": 1 + }, + "evidence_locator": "worksheet-completed.md#s03", + "scenario_notes": "- **U:** C1 distinguishes \u201clocally committed\u201d from an upload \u201crejected due to authorization failure.\u201d C2 rejects retrying under the same unresolved authorization problem. R is 0 because recovery is prescribed before the missing confidentiality policy is known: \u201cThe notes remain editable locally,\u201d the full content \u201cmust be retained,\u201d and the user \u201cshould be presented\u201d with export/save options. Q correctly asks whether retention, view, edit, export or purge is allowed after revocation.\n- **V:** C1/C2 are met. R is 0 because \u201cmust be retained\u201d and proposed local save/export options are asserted before the revocation policy is known, even though the last bullet correctly asks for that policy. Q is met.\n- **Unsupported policy assumptions:** U: \u201cThe notes remain editable locally\u201d; \u201cmust be retained\u201d; export/save as an offered recovery. V: \u201cmust be retained\u201d; possible local save/export before policy resolution." + }, + { + "answer": "S04-U", + "scores": { + "C1": 1, + "C2": 1, + "R": 0, + "Q": 0 + }, + "evidence_locator": "worksheet-completed.md#s04", + "scenario_notes": "- **U:** C1 is explicit: K may have been accepted despite timeout and local cancellation. C2 is met by \u201cA new submission is unsafe.\u201d R is 0 because checking with the provider is a valid escalation path, but the answer does not explicitly state that an accepted K requires a separate authorized remote reversal/cancellation action. Q is 0 because it identifies lookup and deduplication-retention gaps but does not ask for the remote cancellation contract/semantics.\n- **V:** C1 is met and C2 is met because it explicitly says a new submission cannot be assumed duplicate-safe. R is 0: \u201cshould be treated as a new, distinct operation with a fresh identity\u201d is suggested before authoritative reconciliation of K. Q is 0 because it asks about duplicate handling/retention but not operation-status lookup/history or remote cancellation semantics.\n- **Unsafe sequencing (V):** recommending a fresh-identity replacement while K remains unresolved. It does not, however, falsely promise that this is duplicate-safe." + }, + { + "answer": "S04-V", + "scores": { + "C1": 1, + "C2": 1, + "R": 0, + "Q": 0 + }, + "evidence_locator": "worksheet-completed.md#s04", + "scenario_notes": "- **U:** C1 is explicit: K may have been accepted despite timeout and local cancellation. C2 is met by \u201cA new submission is unsafe.\u201d R is 0 because checking with the provider is a valid escalation path, but the answer does not explicitly state that an accepted K requires a separate authorized remote reversal/cancellation action. Q is 0 because it identifies lookup and deduplication-retention gaps but does not ask for the remote cancellation contract/semantics.\n- **V:** C1 is met and C2 is met because it explicitly says a new submission cannot be assumed duplicate-safe. R is 0: \u201cshould be treated as a new, distinct operation with a fresh identity\u201d is suggested before authoritative reconciliation of K. Q is 0 because it asks about duplicate handling/retention but not operation-status lookup/history or remote cancellation semantics.\n- **Unsafe sequencing (V):** recommending a fresh-identity replacement while K remains unresolved. It does not, however, falsely promise that this is duplicate-safe." + }, + { + "answer": "S05-U", + "scores": { + "C1": 0, + "C2": 1, + "R": 1, + "Q": 1 + }, + "evidence_locator": "worksheet-completed.md#s05", + "scenario_notes": "- **U:** C1 is 0: it preserves c/Q across restart, but never explicitly says that recording c and Q atomically supersedes/blocks P in the local queue. C2 is explicit: the receipt updates P history only and not active c or Q. R is met because the answer treats P as already in flight and requires receiver ancestry/precondition behavior rather than assuming local retirement can stop it. Q explicitly identifies that receiver contract as missing.\n- **V:** C1 is 0 for the same reason and additionally lacks an explicit crash-boundary transition for atomically recording c/Q while retiring P. C2 is met by updating P to accepted without reapplying it to c/Q. R/Q are met by explicitly calling out receiver concurrent-update/precondition behavior as necessary before making guarantees.\n- No prohibited guarantee that local cancellation can stop in-flight P; neither answer marks Q synchronized from P\u2019s receipt." + }, + { + "answer": "S05-V", + "scores": { + "C1": 0, + "C2": 1, + "R": 1, + "Q": 1 + }, + "evidence_locator": "worksheet-completed.md#s05", + "scenario_notes": "- **U:** C1 is 0: it preserves c/Q across restart, but never explicitly says that recording c and Q atomically supersedes/blocks P in the local queue. C2 is explicit: the receipt updates P history only and not active c or Q. R is met because the answer treats P as already in flight and requires receiver ancestry/precondition behavior rather than assuming local retirement can stop it. Q explicitly identifies that receiver contract as missing.\n- **V:** C1 is 0 for the same reason and additionally lacks an explicit crash-boundary transition for atomically recording c/Q while retiring P. C2 is met by updating P to accepted without reapplying it to c/Q. R/Q are met by explicitly calling out receiver concurrent-update/precondition behavior as necessary before making guarantees.\n- No prohibited guarantee that local cancellation can stop in-flight P; neither answer marks Q synchronized from P\u2019s receipt." + }, + { + "answer": "S06-U", + "scores": { + "C1": 1, + "C2": 1, + "R": 1, + "Q": 0 + }, + "evidence_locator": "worksheet-completed.md#s06", + "scenario_notes": "- **U:** C1 is explicit: restart testing cannot support power-loss/device-loss guarantees. C2 is met because the answer recognizes same-device copies plus deletion-propagating sync as insufficient/risky recovery evidence. R is met by rejecting removal of keyboard conflict review because users still need to resolve divergent versions. Q is 0: it asks for stronger storage/deletion semantics but not an independently recoverable backup/restore contract.\n- **V:** C1/C2/R are met for the same reasons. Q is 0 because no independently recoverable backup/restore contract is requested. **Unsupported negative capability claim:** \u201cThe current local transaction only handles process restarts; it cannot recover from power or device loss.\u201d The evidence only establishes that those failure modes were not tested, not that recovery is impossible.\n- Neither answer treats synchronization or same-device copies as sufficient independent backup." + }, + { + "answer": "S06-V", + "scores": { + "C1": 1, + "C2": 1, + "R": 1, + "Q": 0 + }, + "evidence_locator": "worksheet-completed.md#s06", + "scenario_notes": "- **U:** C1 is explicit: restart testing cannot support power-loss/device-loss guarantees. C2 is met because the answer recognizes same-device copies plus deletion-propagating sync as insufficient/risky recovery evidence. R is met by rejecting removal of keyboard conflict review because users still need to resolve divergent versions. Q is 0: it asks for stronger storage/deletion semantics but not an independently recoverable backup/restore contract.\n- **V:** C1/C2/R are met for the same reasons. Q is 0 because no independently recoverable backup/restore contract is requested. **Unsupported negative capability claim:** \u201cThe current local transaction only handles process restarts; it cannot recover from power or device loss.\u201d The evidence only establishes that those failure modes were not tested, not that recovery is impossible.\n- Neither answer treats synchronization or same-device copies as sufficient independent backup." + } + ] +} diff --git a/research/evaluations/language-pilot-01/submitted-review-01/worksheet-completed.md b/research/evaluations/language-pilot-01/submitted-review-01/worksheet-completed.md new file mode 100644 index 0000000..e611caf --- /dev/null +++ b/research/evaluations/language-pilot-01/submitted-review-01/worksheet-completed.md @@ -0,0 +1,271 @@ +# Human scoring worksheet + +Reviewer: ChatGPT (GPT-5.6 Sol; model review, not human validation) Date: 2026-09-26 + +Have you seen the assignment key or prior scores? Assignment key: no. Prior scores: a high-level summary was inadvertently visible in the top-level pilot README before scoring; assignment.json and score files were not opened, and that summary was not used as scoring evidence. + +For each criterion mark 1 if explicitly and correctly met, otherwise 0. Leave blanks +until reviewed. Record unsupported guarantees separately with a quote. Do not reward +pattern names. See [instructions](README.md). This sheet contains no condition labels +or prior ratings. Keep all original answers unchanged. + +## S01 — Late receipt and a newer draft + +A server-validated record editor submits operation A. Its request promise rejects, but an authenticated server event identifying A and its accepted revision arrives. The user has already typed draft B. A previously started replica read then returns a different opaque version; the replica consistency contract is unspecified. Explain the correct visible states and next actions. + +### Answer U + +This architecture should result in the following: + +* **Outcome/State:** Operation A is confirmed accepted, and draft B is preserved. The replica read is superseded and ignored. +* **Preserved Information:** The accepted revision of A, the full content of draft B, and the identity of operation A are retained. +* **Recovery Action and Rejected Alternative:** The system should refresh observations based on the accepted revision of A, while keeping draft B visible. Blindly overwriting draft B with the stale replica data is incorrect. +* **Missing Contract or Clarification:** The replica consistency contract needs to be explicitly defined to determine how and when its data should be considered authoritative or how to reconcile it with confirmed server state. + +### Answer V + +* **Outcome/State:** Operation A is accepted, but the replica is stale, and draft B is present. +* **Preserved Information:** Accepted revision of A, draft B, and the original identity of A. +* **Recovery Action:** Refresh the affected observations while preserving draft B. Do not retry A with a new identity. +* **Missing Contract/Clarification:** The replica consistency contract needs to be explicitly defined to understand how to handle the stale read. + +### Your assessment + +| Criterion | Requirement | U (0/1) | V (0/1) | +| --- | --- | --- | --- | +| C1 | Keep A accepted despite the rejected promise. | 1 | 1 | +| C2 | Preserve draft B; receipt for A cannot clear it. | 1 | 1 | +| R | Do not replace current data based on opaque-version ordering or equate acceptance with freshness; mark freshness unresolved or gate stale reads. | 1 | 0 | +| Q | Identify the missing read-after-write/server ordering contract before asserting freshness. | 1 | 1 | + +Prohibited recommendations to check: + +- Treat the rejected promise as proof A failed. +- Erase B because A completed. +- Numerically or lexically order opaque versions as proof of freshness. + +Evidence for your ratings / unsupported claims / ambiguities: + +- **U:** C1/C2 are explicit in “Operation A is confirmed accepted, and draft B is preserved.” R gets credit because the previously started read is explicitly “superseded and ignored,” which is a valid local request-generation gate rather than opaque-version ordering. Q is explicit in the request for a replica consistency/authority contract. **Unsupported/ambiguous:** “stale replica data” labels the returned data stale even though server/replica freshness is unspecified; I do not use that adjective as the basis for R credit. +- **V:** C1/C2 are explicit. R is 0 because “the replica is stale” / “the stale read” asserts freshness ordering that the scenario does not establish. Q gets credit for explicitly identifying the missing replica consistency contract. **Unsupported claim:** “the replica is stale.” +- No prohibited recommendation is made by either answer. + +## S02 — Retry after losing operation identity + +A mobile message composer queued an immutable payload and sent it. The response was lost. After logout, reinstall and login as a new device, only the payload remains; the original operation ID is lost. The server deduplication scope is device and endpoint. Lookup support and retention are unspecified. The product suggests generating a fresh ID and automatically retrying. Evaluate that suggestion and give a recovery plan. + +### Answer U + +Here's a recommendation for the scenario: + +* **Outcome/State:** The original message was sent, but its delivery status is unknown due to a lost response. The local operation ID is gone, and the server's deduplication scope is device-specific. + +* **Preserved Information:** Only the immutable message payload remains locally. The original operation identity and any server-side acceptance evidence are lost to the client. + +* **Recovery Action and Rejected Alternative:** The product's suggestion to generate a fresh ID and retry is problematic. This risks sending a duplicate message if the original was accepted. Instead, the user should be prompted to manually reconcile the message. This could involve presenting the message to the user and asking if they want to resend it, acknowledging potential duplication. + +* **Missing Contract or Clarification:** The server's lookup support and retention for operation IDs are critical. Without these, the client cannot determine if the original message was processed. A contract for server-side reconciliation or query by payload content would be beneficial. + +### Answer V + +Here's an evaluation and recovery plan: + +* **Outcome/State:** The original message payload is present, but its operation identity is lost, leading to an unknown server outcome for the initial send attempt. +* **Preserved Information:** Only the immutable message payload itself is retained across device changes and reinstallation. +* **Recovery Action and Rejected Alternative:** The product's suggestion to generate a fresh ID and automatically retry is flawed. This risks duplicating the message if the original send was successful. Instead, the user must be prompted to manually confirm re-sending the message, acknowledging the potential for duplication. +* **Missing Contract or Clarification:** The server's deduplication scope and retention for operation IDs are critical but unspecified. Without this, it's impossible to know if the original message was processed or if a fresh ID would truly be safe. + +### Your assessment + +| Criterion | Requirement | U (0/1) | V (0/1) | +| --- | --- | --- | --- | +| C1 | Outcome remains unknown; same payload does not establish rejection or identity. | 1 | 1 | +| C2 | A fresh ID/new device cannot be claimed duplicate-safe; do not automatically resend as recovery. | 1 | 1 | +| R | Preserve the payload separately and resolve/check the old outcome before a new send, or require an explicit informed reconciliation decision. | 1 | 1 | +| Q | Ask about operation/history lookup and retention or recoverability of the old device/operation identity. | 1 | 0 | + +Prohibited recommendations to check: + +- Claim identical content guarantees idempotency across devices. +- Promise automatic fresh-ID retry cannot duplicate. + +Evidence for your ratings / unsupported claims / ambiguities: + +- **U:** C1 is explicit (“delivery status is unknown”); C2 rejects fresh-ID automatic retry because it may duplicate; R gives an informed manual reconciliation/resend choice; Q explicitly asks for lookup support and operation-ID retention. No prohibited recommendation. +- **V:** C1/C2/R are explicit: unknown outcome, duplicate risk, and informed manual resend. Q is 0 under the frozen no-partial-credit rule: it asks about deduplication scope and ID retention, but does not explicitly ask for operation/history lookup or recoverability of the old device/operation identity. +- No material unsupported guarantee identified. + +## S03 — Revoked access during offline work + +An offline field editor has a completed local commit of confidential notes and a pending upload. On reconnect, the server explicitly denies write authorization. The product requirement only says "never lose edits"; it says nothing about whether revoked users may retain, view or export local copies. Design the next states and recovery behavior without inventing an access policy. + +### Answer U + +Here's a recommendation for the frontend architecture: + +* **Outcome/State:** The editor should display the confidential notes as locally committed but with a clear, persistent indication that the pending upload was explicitly rejected due to authorization failure. The notes remain editable locally. + +* **Preserved Information:** The full content of the confidential notes, their local commit identity, and the identity of the rejected upload attempt must be retained. The explicit server rejection message should also be preserved and associated with the failed upload. + +* **Recovery Action and Rejected Alternative:** The user should be presented with options to either discard the local changes (if they choose to, understanding the implications) or attempt to export/save the content locally. Blindly retrying the upload with a new identity is rejected as it doesn't address the authorization issue and could lead to data duplication if the server's rejection was transient. + +* **Missing Contract or Clarification:** A clear contract is needed regarding the retention and access policies for locally committed data when server authorization is revoked. Specifically, can the user continue to view, edit, or export these notes, or should they be purged? + +### Answer V + +This architecture needs refinement to handle authorization denial gracefully. + +* **Outcome/State:** The local commit remains, but the pending upload is explicitly rejected due to authorization. The editor is in a state of "accepted locally, rejected remotely." +* **Preserved Information:** The full local commit of confidential notes, including its revision and operation identity, must be retained. The server's explicit denial and the reason (authorization failure) should also be recorded. +* **Recovery Action and Rejected Alternative:** The system should prompt the user to explicitly reconcile the rejected upload, perhaps by allowing them to save the notes locally or export them. Blindly discarding the local commit or attempting a re-upload with a new identity is inappropriate as it would lose data and violate the "never lose edits" requirement. +* **Missing Contract or Clarification:** A clear contract is needed regarding the retention, viewing, and export of local data for users whose write authorization has been revoked. This will dictate the specific recovery options presented to the user. + +### Your assessment + +| Criterion | Requirement | U (0/1) | V (0/1) | +| --- | --- | --- | --- | +| C1 | Distinguish saved locally from uploaded/accepted remotely. | 1 | 1 | +| C2 | Block unauthorized synchronization and do not bypass revocation or endlessly retry unchanged credentials. | 1 | 1 | +| R | Make recovery conditional on authorized access/policy; do not promise unconditional export, permanent retention or silent destructive deletion. | 0 | 0 | +| Q | Explicitly request the policy for retaining, viewing or deleting/exporting confidential copies after revocation. | 1 | 1 | + +Prohibited recommendations to check: + +- Assert never-lose-edits authorizes export despite revocation. +- Unconditionally wipe or retain all confidential content as if policy were known. + +Evidence for your ratings / unsupported claims / ambiguities: + +- **U:** C1 distinguishes “locally committed” from an upload “rejected due to authorization failure.” C2 rejects retrying under the same unresolved authorization problem. R is 0 because recovery is prescribed before the missing confidentiality policy is known: “The notes remain editable locally,” the full content “must be retained,” and the user “should be presented” with export/save options. Q correctly asks whether retention, view, edit, export or purge is allowed after revocation. +- **V:** C1/C2 are met. R is 0 because “must be retained” and proposed local save/export options are asserted before the revocation policy is known, even though the last bullet correctly asks for that policy. Q is met. +- **Unsupported policy assumptions:** U: “The notes remain editable locally”; “must be retained”; export/save as an offered recovery. V: “must be retained”; possible local save/export before policy resolution. + +## S04 — Local cancellation after uncertain booking + +A server-confirmed booking UI times out after submitting operation K. Before receiving authoritative evidence, the user presses Cancel; the client removes its pending entry. The user then asks to submit the same booking again. The server exposes no documented operation-status lookup or deduplication retention guarantee. Specify what Cancel can honestly mean and whether a new submission is safe. + +### Answer U + +The proposed architecture has a critical flaw for this scenario. + +* **Outcome/State:** The booking's final state is unknown. The server may have accepted operation K despite the timeout and local cancellation. +* **Preserved Information:** The client has discarded the original operation K's identity and intent. +* **Recovery Action:** The only safe action is to inform the user that the previous booking's status is unknown and they should check with the service provider. A new submission is unsafe. +* **Missing Contract:** The server lacks a documented operation-status lookup or deduplication retention guarantee, which is essential for safely retrying or confirming operations after client-side uncertainty. + +### Answer V + +Here's a recommendation for the scenario: + +* **Outcome/State:** The client should not remove the pending entry for operation K upon local "Cancel." The server's status for K is unknown. +* **Preserved Information:** The client must retain the original operation K's identity and payload, along with its "pending" status, even after local cancellation. +* **Recovery Action and Rejected Alternative:** The user's request to submit the "same booking again" should be treated as a new, distinct operation with a fresh identity. Blindly retrying K with its original identity is rejected because the server lacks deduplication guarantees. +* **Missing Contract or Clarification:** A critical missing contract is the server's behavior regarding duplicate submissions and its retention window for operation identities. Without this, the client cannot safely assume a new submission will not result in a duplicate booking if K was, in fact, accepted. + +### Your assessment + +| Criterion | Requirement | U (0/1) | V (0/1) | +| --- | --- | --- | --- | +| C1 | Local cancellation does not establish remote cancellation or rejection of K. | 1 | 1 | +| C2 | Do not claim a new submission is duplicate-safe while K is unresolved. | 1 | 1 | +| R | Use an authoritative reconciliation/escalation path before replacement; remote reversal is a separate authorized action if K was accepted. | 0 | 0 | +| Q | Ask about cancellation, lookup/history and retry-retention contracts needed to resolve K. | 0 | 0 | + +Prohibited recommendations to check: + +- Tell the user the booking was undone by removing local state. +- Automatically use a new ID and promise no duplicate booking. + +Evidence for your ratings / unsupported claims / ambiguities: + +- **U:** C1 is explicit: K may have been accepted despite timeout and local cancellation. C2 is met by “A new submission is unsafe.” R is 0 because checking with the provider is a valid escalation path, but the answer does not explicitly state that an accepted K requires a separate authorized remote reversal/cancellation action. Q is 0 because it identifies lookup and deduplication-retention gaps but does not ask for the remote cancellation contract/semantics. +- **V:** C1 is met and C2 is met because it explicitly says a new submission cannot be assumed duplicate-safe. R is 0: “should be treated as a new, distinct operation with a fresh identity” is suggested before authoritative reconciliation of K. Q is 0 because it asks about duplicate handling/retention but not operation-status lookup/history or remote cancellation semantics. +- **Unsafe sequencing (V):** recommending a fresh-identity replacement while K remains unresolved. It does not, however, falsely promise that this is duplicate-safe. + +## S05 — An old upload survives conflict resolution + +Device A durably recorded offline revision a with pending operation P. Another device uploaded divergent revision b. A preserves both and resolves them into revision c with new pending operation Q. P was already in flight. Later a receipt for P arrives. Local commits survive process restart, but receiver ancestry/precondition behavior is unspecified. Specify queue transitions, crash boundaries and what the late receipt may change. + +### Answer U + +Here's a recommendation for the frontend architecture based on the scenario: + +* **Outcome/State:** The system should transition to a state where revision `c` is the active local content, with pending operation `Q` awaiting remote acceptance. The late receipt for `P` should update its own operation history, marking `P` as remotely accepted, but it should not alter the active content or the status of `Q`. + +* **Preserved Information:** Both divergent revisions `a` and `b` should be preserved with their identities and provenance until `c` is fully committed and resolved. The original pending operation `P` and its identity must also be retained, along with the new pending operation `Q`. + +* **Recovery Action and Rejected Alternative:** Upon process restart, the system must recover `c` as the active content and `Q` as the pending operation. The late receipt for `P` should not trigger a re-application of `P` or a rollback to `a`, as this would erase the user's explicit resolution into `c`. + +* **Missing Contract or Clarification:** The architecture needs a clear contract for the receiver's ancestry and precondition behavior. Specifically, it must define how the receiver handles an in-flight operation (`P`) when a superseding operation (`Q` derived from `c`) is subsequently sent, and how it would respond to `P` if `b` was already accepted. + +### Answer V + +This architecture has several issues that could lead to data loss or corruption. + +* **Outcome/State:** The system will transition from having pending operations P and Q, and revisions a, b, and c, to a state where P's receipt arrives. The system needs to define how the late receipt for P interacts with the already-resolved revision c and the new pending operation Q. +* **Preserved Information:** The system must preserve revisions a, b, and c, along with the identities of pending operations P and Q, and the fact that P was in flight. The receipt for P, when it arrives, must also be preserved with its associated operation identity. +* **Recovery Action and Rejected Alternative:** Upon receiving the late receipt for P, the system should update P's status to "accepted" but **not** reapply P's changes to revision c or Q. Blindly applying P's changes would overwrite the explicit reconciliation that produced c. Instead, the system should acknowledge P's acceptance and potentially flag that P's original intent was superseded by the reconciliation. +* **Missing Contract or Clarification:** The critical missing contract is the receiver's ancestry and precondition behavior. Without knowing how the receiver handles concurrent updates and what preconditions it enforces, it's impossible to guarantee that P's original intent wasn't already superseded on the server, or how the server would react to a re-submission of P after c was uploaded. + +### Your assessment + +| Criterion | Requirement | U (0/1) | V (0/1) | +| --- | --- | --- | --- | +| C1 | Block/supersede P atomically with the resolved revision and Q; maintain restart-safe pending intent. | 0 | 0 | +| C2 | A receipt for P updates P history only, not Q status or the active resolved revision. | 1 | 1 | +| R | Do not assume local queue retirement stops an in-flight P; require receiver preconditions/ancestry to protect c. | 1 | 1 | +| Q | Explicitly identify receiver ancestry/version handling as a missing contract requiring verification. | 1 | 1 | + +Prohibited recommendations to check: + +- Mark Q synchronized because P was acknowledged. +- Guarantee local cancellation prevents P from overwriting remotely. + +Evidence for your ratings / unsupported claims / ambiguities: + +- **U:** C1 is 0: it preserves c/Q across restart, but never explicitly says that recording c and Q atomically supersedes/blocks P in the local queue. C2 is explicit: the receipt updates P history only and not active c or Q. R is met because the answer treats P as already in flight and requires receiver ancestry/precondition behavior rather than assuming local retirement can stop it. Q explicitly identifies that receiver contract as missing. +- **V:** C1 is 0 for the same reason and additionally lacks an explicit crash-boundary transition for atomically recording c/Q while retiring P. C2 is met by updating P to accepted without reapplying it to c/Q. R/Q are met by explicitly calling out receiver concurrent-update/precondition behavior as necessary before making guarantees. +- No prohibited guarantee that local cancellation can stop in-flight P; neither answer marks Q synchronized from P’s receipt. + +## S06 — Storage guarantee beyond the evidence + +A field-note product promises edits survive device loss and complete power failure. The chosen local transaction has only been tested against ordinary process restart. Two copies of a conflict are stored on the same device, and the sync protocol propagates deletions. The team proposes declaring recovery solved and removing conflict review from the keyboard interface. Assess that design and state what can actually be promised. + +### Answer U + +Here's an assessment of the proposed design: + +* **Outcome/State:** The design cannot guarantee edits survive device loss or complete power failure, nor can it claim recovery is solved. +* **Preserved Information:** Only two copies of a conflict are stored on the same device, and the sync protocol propagates deletions, which risks data loss. +* **Recovery Action and Rejected Alternative:** The proposal to remove conflict review from the keyboard interface is premature and dangerous, as it eliminates the user's ability to resolve divergent versions. +* **Missing Contract or Clarification:** There is no documented contract for restart-safe offline editing that covers device loss or power failure, nor is there a clear definition of how deletions are handled during conflict resolution. + +### Answer V + +Here's an assessment of the proposed frontend architecture: + +* **Outcome/State:** The product cannot guarantee survival of edits through device loss or complete power failure, nor can it claim conflict resolution is solved. +* **Preserved Information:** Only two copies of a conflict are retained on the same device, and the sync protocol propagates deletions, which is insufficient for robust recovery. +* **Recovery Action and Rejected Alternative:** The current local transaction only handles process restarts; it cannot recover from power or device loss. Removing conflict review from the keyboard interface is premature and risky. +* **Missing Contract or Clarification:** There's no documented contract for restart-safe storage beyond process restart, nor for how deletions are handled during conflict resolution or synchronization across devices. + +### Your assessment + +| Criterion | Requirement | U (0/1) | V (0/1) | +| --- | --- | --- | --- | +| C1 | Ordinary restart evidence cannot establish power-loss or device-loss survival. | 1 | 1 | +| C2 | Copies on one device and deletion-propagating sync are not independent backup. | 1 | 1 | +| R | Retain accessible discoverable comparison/resolution; storing conflict copies alone does not resolve intent. | 1 | 1 | +| Q | Request storage flush/power-loss evidence and an independently recoverable backup/restore contract. | 0 | 0 | + +Prohibited recommendations to check: + +- Claim local transaction or same-device copies guarantee device-loss recovery. +- Treat synchronization as sufficient independent backup. +- Approve inaccessible conflict review because bytes are preserved. + +Evidence for your ratings / unsupported claims / ambiguities: + +- **U:** C1 is explicit: restart testing cannot support power-loss/device-loss guarantees. C2 is met because the answer recognizes same-device copies plus deletion-propagating sync as insufficient/risky recovery evidence. R is met by rejecting removal of keyboard conflict review because users still need to resolve divergent versions. Q is 0: it asks for stronger storage/deletion semantics but not an independently recoverable backup/restore contract. +- **V:** C1/C2/R are met for the same reasons. Q is 0 because no independently recoverable backup/restore contract is requested. **Unsupported negative capability claim:** “The current local transaction only handles process restarts; it cannot recover from power or device loss.” The evidence only establishes that those failure modes were not tested, not that recovery is impossible. +- Neither answer treats synchronization or same-device copies as sufficient independent backup. diff --git a/research/plan/OpenQuestions.md b/research/plan/OpenQuestions.md index 3eb4b90..d79698b 100644 --- a/research/plan/OpenQuestions.md +++ b/research/plan/OpenQuestions.md @@ -65,3 +65,9 @@ under two equivalent-information conditions. Q04 remains open: small mixed descr differences, same-author editorial scoring and pending human adjudication do not establish usefulness. The six briefs are consumed pilot material, not future held-out items. Next: independent ratings before changing guidance or replicating. + +2026-09-26 submitted-review update: [adjudicated totals](../evaluations/language-pilot-01/submitted-review-01/adjudication.md) +are 18/24 for each condition after four corrections. No presentation benefit is +established; equal totals do not establish equivalence. Worksheet declares model +authorship and prior aggregate-result exposure. Q04 remains open; future rubrics +should split compound obligations before execution. diff --git a/workplans/FEP-WP-0002-seed-language-validation.md b/workplans/FEP-WP-0002-seed-language-validation.md index 88cab48..c2ab533 100644 --- a/workplans/FEP-WP-0002-seed-language-validation.md +++ b/workplans/FEP-WP-0002-seed-language-validation.md @@ -304,3 +304,23 @@ the frozen rubric. No human scores may be inferred or filled by a model. Shared weaknesses concern policy-dependent retention, atomic supersession and backup/restore. After human review, revise shared guidance if warranted and replicate on fresh scenarios. These six briefs are consumed. T02/T03 remain progress; no M1 completion. + +## User-submitted worksheet adjudicated — 2026-09-26 + +T03: imported worksheet-completed.md byte-for-byte under the pilot's +submitted-review-01 directory; verified all twelve answer texts and rubric rows. +The worksheet identifies GPT-5.6 Sol as reviewer, explicitly model review, with +aggregate-result exposure. Accepted as the user's submitted review without claiming +independent human validation or inventing external inference provenance. + +Accepted all four binary-rating corrections under the frozen no-partial-credit +rubric: 44/48 ratings unchanged; updated totals 18/24 for both named and prose packs. +Original scores/summary remain immutable. Narrative unsafe-claim annotations are +retained with adjudication, not converted to an unsupported numeric count. +No presentation benefit demonstrated; equal counts do not establish equivalence. + +User-review incorporation is complete. T03 remains progress: independent-human +validation is still unfulfilled under the frozen protocol, and later studies need +atomic criteria and fresh scenarios. Target shared guidance at policy-dependent +recovery, cancellation/reversal contracts, atomic supersession and backup/restore. +See research/evaluations/language-pilot-01/submitted-review-01/adjudication.md.