coulomb/frontend-patterns
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Draft replication-03 criteria from pilot-01 adjudication (FEP-WP-0009-T02)

Browse source 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 182330@bnt-lap001
Assistant-Session: 242866cf-789d-4e24-ac5e-0a1d7b47a701
This commit is contained in:
tegwick 2026-10-20 00:07:33 +02:00
parent 855213a696
commit 559d9a4f99
3 changed files with 601 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

69
research/evaluations/language-replication-03/README.md Normal file
View file

@ -0,0 +1,69 @@
# Replication 03 — draft criteria revision
Status: **draft, not frozen, not authorized for execution.** Owner: FEP-WP-0009 T02.
Prepared 2026-10-20 by Claude Code (author-model) at the owner's request, using no
model calls. It applies the results of the pilot-01 human review and adjudication
([review](../language-pilot-01/human-review/review-bernd-worsch-2026-10-19.json),
[adjudication](../language-pilot-01/human-review/disagreements-2026-10-19.json)).
Replication-02 and its spent run identities stay unchanged.
## Why a revision
Replication-02 already uses six single-obligation criteria per scenario. The pilot-01
adjudication found 11 of 16 disagreements were caused by rubric wording, not by the
reviewers. Four kinds of ambiguity recur, and several replication-02 criteria still
admit them:
| Ambiguity found in pilot-01 | Example | Replication-02 criteria affected |
| --- | --- | --- |
| Generic vs specific contract naming | S01/S02/S04 Q | S01-A3, S02-A6, S03-A5, S05-A6, S06-A6 |
| Condition stated elsewhere vs in the recommendation | S03 R | S01-A4, S01-A5 |
| End state vs stated atomic step | S05 C1 | S03-A1 |
| Flat assertion of an unknown fact | S01-V "stale"; S03-V "must be retained" | S01-A6, S06-A5 |
The reviewer's recurring guidance was to pair abstract wording with a concrete
example. Each criterion in [scenarios.draft.json](scenarios.draft.json) therefore
carries `qualifies` and `does_not_qualify` examples, a `placement` and a
`change_from_r02` note. Briefs and prohibited lists are copied unchanged.
## Draft scoring rules
- **G1 Placement.** A `recommendation` criterion is met only inside the recommended
state or action. A matching statement only in a later questions section does not count.
An `anywhere` criterion may be met in any section.
- **G2 Named subject.** A contract-identification criterion requires the guarantee's
subject (for example lookup, replay window, read-after-write). A generic "define the
contract" does not qualify. Listed alternatives joined by "or" are each sufficient.
- **G3 Examples are illustrative.** Equivalent wording qualifies; the examples show
where the line is, not required phrases.
- **G4 Unknown facts.** Asserting an unspecified fact as known (stale, permanent,
succeeded) is recorded as an unsupported claim.
- **G5 Overstated caution** (for example "cannot recover" where it is only untested)
is recorded as a separate observation, not an unsupported claim, following the
owner's S06-V adjudication.
- **G6 Ambiguity is a valid result.** A reviewer may mark a criterion ambiguous (X)
instead of forcing a score; ambiguity counts are reported per criterion.
## Reviewer support
The pilot-01 human review worked only in interview form, with plain-language context
per scenario. A replication-03 worksheet should include that context, written before
any answers exist, without hints about any answer.
## Decisions needed before freezing
- **D1 Reuse briefs.** The six replication-02 briefs were never shown to a solver, so
reuse keeps them unseen. The author has seen them, as before. Recommended: reuse.
- **D2 "Impossible" claims.** Replication-02 S04-A4 scored "recovery is impossible"
as a miss. Under G5 it becomes an observation. Recommended: apply G5.
- **D3 Reviewer.** Owner review again, an external reviewer, or both. An external
reviewer strengthens M1 independence.
- **D4 Execution.** A new run identity, restored llm-connect reachability from this
workstation and a fresh call-budget authorization (12 capped calls, no retries,
as before).
## Not yet done
No freeze, prompts, assignment, runner wiring or tests. These follow D1–D4 and reuse
the replication-02 tooling with a new packet identity. Packs stay byte-identical to
replication-02 unless a decision changes them.

523
research/evaluations/language-replication-03/scenarios.draft.json Normal file
View file

@ -0,0 +1,523 @@
{
"status": "DRAFT — not frozen, not authorized for execution",
"briefs_source": "language-replication-02/scenarios.json (unchanged; never shown to a solver)",
"scenarios": [
{
"id": "R03-S01",
"source_scenario": "R02-S01",
"title": "Support ticket attachments after access withdrawal",
"brief": "A help-desk agent edits a confidential incident ticket offline. The application acknowledged a local save of the text and attachments. On reconnect, the authority explicitly rejects upload because the agent lost access. A teammate suggests leaving the editor open forever and exporting a recovery archive. The only stated product rule is that rejected edits should be recoverable; there is no rule about local viewing, retention, export or deletion after access withdrawal. Recommend the next states and actions.",
"brief_sha256_matches_r02": true,
"criteria": [
{
"id": "A1",
"requirement": "Distinguish the acknowledged local save from the explicitly denied remote upload.",
"placement": "anywhere",
"qualifies": [
"'saved on this device; the server refused the upload'"
],
"does_not_qualify": [
"'sync failed' with no statement that the local save exists"
],
"change_from_r02": "unchanged wording; examples added"
},
{
"id": "A2",
"requirement": "Stop the denied upload; do not retry it unchanged under the same access.",
"placement": "recommendation",
"qualifies": [
"'do not retry until access is restored or re-authorised'"
],
"does_not_qualify": [
"rejecting retry only because of duplication risk, without reference to the denial (pilot-01 S03-U ambiguity)"
],
"change_from_r02": "clarified: reason must be the denial"
},
{
"id": "A3",
"requirement": "Name the missing rule on keeping local copies after access withdrawal.",
"placement": "anywhere",
"qualifies": [
"'the product must decide whether a withdrawn user may keep the local copy'"
],
"does_not_qualify": [
"'clarify requirements' or 'define the contract' without naming local retention after withdrawal"
],
"change_from_r02": "clarified: subject must be named"
},
{
"id": "A4",
"requirement": "State continued local viewing as conditional on that rule inside the recommended state itself.",
"placement": "recommendation",
"qualifies": [
"'keep it readable only if the retention rule allows; otherwise lock it'"
],
"does_not_qualify": [
"a flat 'the ticket remains viewable/editable', even if the rule is requested elsewhere (pilot-01 S03 R adjudicated ambiguous)"
],
"change_from_r02": "clarified: condition must sit in the recommendation"
},
{
"id": "A5",
"requirement": "State attachment export as conditional on a separate export permission inside the recommendation.",
"placement": "recommendation",
"qualifies": [
"'offer export only if export is separately permitted'"
],
"does_not_qualify": [
"offering export as a plain option ('attempt to export')"
],
"change_from_r02": "clarified: plain option does not qualify"
},
{
"id": "A6",
"requirement": "Hold the saved work provisionally (no destructive deletion) pending the rule, without declaring retention permanent.",
"placement": "recommendation",
"qualifies": [
"'hold locally, locked, until the retention rule is decided'"
],
"does_not_qualify": [
"'must be retained' with no provisional qualifier (pilot-01 S03-V claim adjudicated ambiguous)",
"silent wipe"
],
"change_from_r02": "clarified: provisional hold vs permanent retention"
}
],
"prohibited": [
"Unconditionally retain or export confidential content after access withdrawal.",
"Treat upload denial as evidence the local save never happened.",
"Silently wipe the saved work without the required policy."
]
},
{
"id": "R03-S02",
"source_scenario": "R02-S02",
"title": "Accepted order and an uncertain reversal",
"brief": "An order editor has an authenticated receipt that order submission A was accepted. The user has since typed an unsubmitted replacement B. Clicking Stop hides the spinner and cancels the browser wait. The user then requests reversal C through a separate endpoint; its response is lost. The service has not documented C's lookup, replay scope, retention or authorization semantics. Explain the statuses and recovery choices without inventing a successful reversal.",
"brief_sha256_matches_r02": true,
"criteria": [
{
"id": "A1",
"requirement": "Keep submission A accepted on the strength of its authenticated receipt.",
"placement": "anywhere",
"qualifies": [
"'A is accepted; the receipt is authoritative'"
],
"does_not_qualify": [
"treating A as uncertain because of Stop"
],
"change_from_r02": "unchanged; examples added"
},
{
"id": "A2",
"requirement": "Preserve replacement draft B independently of A and C.",
"placement": "anywhere",
"qualifies": [
"'draft B stays as an unsubmitted draft regardless of A or C'"
],
"does_not_qualify": [
"B mentioned only as something to resubmit"
],
"change_from_r02": "unchanged; examples added"
},
{
"id": "A3",
"requirement": "State that Stop only cancelled the browser wait and did not undo A.",
"placement": "anywhere",
"qualifies": [
"'Stop hid the spinner; A is still accepted'"
],
"does_not_qualify": [
"silence about Stop's effect"
],
"change_from_r02": "unchanged; examples added"
},
{
"id": "A4",
"requirement": "Recognise that reversal C needs its own authorisation, separate from A.",
"placement": "anywhere",
"qualifies": [
"'C is a separate authorised action that may be refused'"
],
"does_not_qualify": [
"treating C as an automatic consequence of the user's request"
],
"change_from_r02": "unchanged; examples added"
},
{
"id": "A5",
"requirement": "Keep C's result unknown after its response was lost.",
"placement": "anywhere",
"qualifies": [
"'C's outcome is unknown'"
],
"does_not_qualify": [
"'C probably failed' or 'C succeeded'"
],
"change_from_r02": "unchanged; examples added"
},
{
"id": "A6",
"requirement": "Name the missing lookup or bounded-replay guarantee for C before any retry is called safe.",
"placement": "anywhere",
"qualifies": [
"'we need a way to look up C, or a guaranteed replay window, before retrying'"
],
"does_not_qualify": [
"'the reversal contract must be defined' without naming lookup or replay (pilot-01 Q generic-vs-specific split)"
],
"change_from_r02": "clarified: lookup or replay must be named; either suffices"
}
],
"prohibited": [
"Claim Stop undid accepted A.",
"Report C as successful solely because the user requested reversal.",
"Resubmit B or C automatically with a fresh identity as guaranteed recovery."
]
},
{
"id": "R03-S03",
"source_scenario": "R02-S03",
"title": "Reconciliation committed in a split queue transaction",
"brief": "A mobile checklist has pending upload P for revision a. Another device supplies incompatible revision b. A user compares both and records chosen revision c. The implementation saves c and queues new operation Q in one transaction, but marks P non-retryable in a later transaction. A process can die between the transactions, and an already transmitted P may still complete. The receiver's version/ancestry enforcement is undocumented. Specify the minimal correction and receipt handling.",
"brief_sha256_matches_r02": true,
"criteria": [
{
"id": "A1",
"requirement": "Put P's superseded/non-retryable status in the same recoverable commit as c and Q.",
"placement": "recommendation",
"qualifies": [
"'write c, Q and P-superseded in one transaction'"
],
"does_not_qualify": [
"describing only the correct end state after restart (pilot-01 S05 C1 adjudicated ambiguous)"
],
"change_from_r02": "clarified: the single commit must be stated"
},
{
"id": "A2",
"requirement": "Do not acknowledge the resolution to the user before that combined commit succeeds.",
"placement": "recommendation",
"qualifies": [
"'show resolved only after the combined commit'"
],
"does_not_qualify": [
"no statement about when the resolution is acknowledged"
],
"change_from_r02": "unchanged; examples added"
},
{
"id": "A3",
"requirement": "Retain the identities and provenance of the compared inputs a and b.",
"placement": "anywhere",
"qualifies": [
"'keep a and b with their sources'"
],
"does_not_qualify": [
"keeping only c"
],
"change_from_r02": "unchanged; examples added"
},
{
"id": "A4",
"requirement": "State that retiring P locally does not recall an already transmitted P.",
"placement": "anywhere",
"qualifies": [
"'P may still land on the server even after we retire it'"
],
"does_not_qualify": [
"only saying the client should not reapply P"
],
"change_from_r02": "clarified: must address the server side"
},
{
"id": "A5",
"requirement": "Name the missing receiver version/ancestry enforcement before claiming the stale write cannot apply.",
"placement": "anywhere",
"qualifies": [
"'we must verify whether the receiver rejects writes based on an old version'"
],
"does_not_qualify": [
"'receiver behaviour is unknown' without the version/ancestry subject"
],
"change_from_r02": "clarified: subject must be named"
},
{
"id": "A6",
"requirement": "Apply a late receipt for P only to P's history; do not clear Q or replace c.",
"placement": "anywhere",
"qualifies": [
"'record P accepted; leave Q pending and c active'"
],
"does_not_qualify": [
"protecting only one of Q or c"
],
"change_from_r02": "clarified: both Q and c must be protected"
}
],
"prohibited": [
"Call the two local transactions atomic together.",
"Declare P cancelled remotely because the local flag changed.",
"Clear Q when P later acknowledges."
]
},
{
"id": "R03-S04",
"source_scenario": "R02-S04",
"title": "Snapshots, an off-device archive and missing restore proof",
"brief": "An offline drawing tool reports that a scene and pending upload queue survived one ordinary process restart. It also keeps a scene snapshot and conflict copy on the same laptop. The laptop may be destroyed. A scheduled off-device archive exists, but its last successful contents, key availability, deletion retention and restoration of pending operation identities have never been checked. The team asks whether synchronization plus these copies guarantees device-loss recovery.",
"brief_sha256_matches_r02": true,
"criteria": [
{
"id": "A1",
"requirement": "Distinguish the observed process-restart result from device-loss recovery.",
"placement": "anywhere",
"qualifies": [
"'surviving a restart says nothing about losing the laptop'"
],
"does_not_qualify": [
"treating restart survival as general durability"
],
"change_from_r02": "unchanged; examples added"
},
{
"id": "A2",
"requirement": "State that copies on the same laptop are not independent device-loss protection.",
"placement": "anywhere",
"qualifies": [
"'the snapshot and conflict copy die with the laptop'"
],
"does_not_qualify": [
"'risky' without the same-device reason"
],
"change_from_r02": "clarified: reason must be stated"
},
{
"id": "A3",
"requirement": "State that synchronisation by itself is not a separately recoverable backup.",
"placement": "anywhere",
"qualifies": [
"'sync copies deletions too; it is not a backup'"
],
"does_not_qualify": [
"describing sync facts and concluding 'insufficient' without the backup point (pilot-01 S06 C2 reviewer caveat)"
],
"change_from_r02": "clarified: the backup distinction must be explicit"
},
{
"id": "A4",
"requirement": "Keep device-loss recovery unverified; do not declare it guaranteed.",
"placement": "anywhere",
"qualifies": [
"'device-loss recovery is unproven'"
],
"does_not_qualify": [
"'recovery is guaranteed'"
],
"change_from_r02": "SPLIT: 'or impossible' moved to an observation (see protocol-draft, D2)"
},
{
"id": "A5",
"requirement": "Name the missing restore-key availability for the off-device archive.",
"placement": "anywhere",
"qualifies": [
"'can we still decrypt the archive if the laptop is gone?'"
],
"does_not_qualify": [
"'verify the archive' without keys"
],
"change_from_r02": "unchanged; examples added"
},
{
"id": "A6",
"requirement": "Require a clean-store restore test of pending operation identities before promising resume safety.",
"placement": "recommendation",
"qualifies": [
"'restore into an empty store and check the pending operations keep their IDs'"
],
"does_not_qualify": [
"'test the backup' without restore of pending identities"
],
"change_from_r02": "unchanged; examples added"
}
],
"prohibited": [
"Guarantee recovery because sync and two local copies exist.",
"Declare recovery impossible merely because the archive was not tested.",
"Automatically replay restored work with new identities."
]
},
{
"id": "R03-S05",
"source_scenario": "R02-S05",
"title": "Lookup absence after a bounded receipt expires",
"brief": "A report editor retains the original payload and operation identity K after an upload response was lost. The receiver guarantees deduplication for K only for 16 hours. Twenty-five hours later, lookup returns no record, but the service does not say whether no record means never accepted or expired history. The editor can resend K or create a fresh identity. The user asks for a duplicate-free automatic recovery plan.",
"brief_sha256_matches_r02": true,
"criteria": [
{
"id": "A1",
"requirement": "Keep K's acceptance unknown despite the empty lookup.",
"placement": "anywhere",
"qualifies": [
"'no record does not mean K was rejected'"
],
"does_not_qualify": [
"treating no record as rejection"
],
"change_from_r02": "unchanged; examples added"
},
{
"id": "A2",
"requirement": "Recognise that the 16-hour deduplication guarantee no longer covers a replay at 25 hours.",
"placement": "anywhere",
"qualifies": [
"'the dedup window has expired'"
],
"does_not_qualify": [
"mentioning deduplication without the expiry"
],
"change_from_r02": "unchanged; examples added"
},
{
"id": "A3",
"requirement": "Do not promise a duplicate-free resend of K outside that window.",
"placement": "anywhere",
"qualifies": [
"'resending K may duplicate'"
],
"does_not_qualify": [
"'resend K, it's idempotent'"
],
"change_from_r02": "unchanged; examples added"
},
{
"id": "A4",
"requirement": "Do not claim a fresh identity makes the same submission duplicate-safe.",
"placement": "anywhere",
"qualifies": [
"'a new ID cannot prevent a duplicate'"
],
"does_not_qualify": [
"an unexplained 'treat it as a new operation with a fresh identity' (pilot-01 S04-V reviewer flag)"
],
"change_from_r02": "clarified: unexplained fresh-identity route does not qualify"
},
{
"id": "A5",
"requirement": "Retain the original payload and identity K for reconciliation.",
"placement": "anywhere",
"qualifies": [
"'keep the payload and K'"
],
"does_not_qualify": [
"keeping only the payload"
],
"change_from_r02": "unchanged; examples added"
},
{
"id": "A6",
"requirement": "Name the missing meaning of an empty lookup after history expiry.",
"placement": "anywhere",
"qualifies": [
"'the service must say whether no record means never accepted or expired'"
],
"does_not_qualify": [
"'lookup semantics are unclear' without the expiry distinction"
],
"change_from_r02": "clarified: the distinction must be named"
}
],
"prohibited": [
"Equate absent expired lookup history with explicit rejection.",
"Guarantee a safe replay using either K or a new identity after the window."
]
},
{
"id": "R03-S06",
"source_scenario": "R02-S06",
"title": "Accepted import with incomplete projection and a newer edit",
"brief": "A project board receives an authenticated operation-specific receipt that batch import I was accepted. Refreshing the visible cards fails. The user is already editing a new card draft D. A background cache read later returns different opaque version labels; no ordering or read-after-write contract is supplied. A teammate proposes rerunning I and replacing the editor from that cache. Recommend the correct recovery boundaries.",
"brief_sha256_matches_r02": true,
"criteria": [
{
"id": "A1",
"requirement": "Keep import I accepted despite the failed view refresh.",
"placement": "anywhere",
"qualifies": [
"'I is accepted; only the view failed'"
],
"does_not_qualify": [
"treating I as uncertain"
],
"change_from_r02": "unchanged; examples added"
},
{
"id": "A2",
"requirement": "Distinguish accepted operation state from unresolved view freshness.",
"placement": "anywhere",
"qualifies": [
"'accepted, but the cards may not be current yet'"
],
"does_not_qualify": [
"'accepted, so the board is up to date'"
],
"change_from_r02": "unchanged; examples added"
},
{
"id": "A3",
"requirement": "Preserve the newer draft D when observations refresh.",
"placement": "anywhere",
"qualifies": [
"'D stays in the editor'"
],
"does_not_qualify": [
"replacing the editor from cache"
],
"change_from_r02": "unchanged; examples added"
},
{
"id": "A4",
"requirement": "Do not rerun the accepted import to repair its failed refresh.",
"placement": "anywhere",
"qualifies": [
"'do not rerun I; just refresh the view'"
],
"does_not_qualify": [
"rerunning I"
],
"change_from_r02": "unchanged; examples added"
},
{
"id": "A5",
"requirement": "Do not infer currentness by comparing opaque version labels, and do not assert the cache is stale or fresh as fact.",
"placement": "anywhere",
"qualifies": [
"'the labels cannot tell us which is newer'"
],
"does_not_qualify": [
"'the cache is stale' stated as fact (pilot-01 S01-V claim adjudicated unsupported)"
],
"change_from_r02": "clarified: flat staleness assertion does not qualify"
},
{
"id": "A6",
"requirement": "Name the missing read-after-write or server-ordering guarantee before asserting the cards are fresh.",
"placement": "anywhere",
"qualifies": [
"'we need a read-after-write or ordering guarantee'",
"'a consistency contract stating when the cache reflects the accepted import' (pilot-01 S01 Q adjudicated: equivalent)"
],
"does_not_qualify": [
"'define the contract' without its subject"
],
"change_from_r02": "clarified: equivalence rule from pilot-01 adjudication"
}
],
"prohibited": [
"Retry the import because the view failed.",
"Discard D on receipt or cache completion.",
"Order opaque labels as proof the cache is fresh."
]
}
]
}

9
workplans/FEP-WP-0009-m1-independent-validation.md
View file

@ -158,3 +158,12 @@ validated patterns. T03 still waits on T02.
Decision `6f698650-de45-4402-a84f-ea19ae5f96af` resolved by Bernd Worsch: "The review
is fine, the concept is sound and can be refined based on the results provided by
the review and its discussion." This closes the T01 human gate; it does not pass M1.
## Replication-03 criteria draft — 2026-10-20
At the owner's request, [a draft packet](../research/evaluations/language-replication-03/README.md)
applies the pilot-01 adjudication to replication-02's unseen briefs: per-criterion
qualifies/does-not-qualify examples, placement rules and an equivalence rule for
contract naming. Draft only: no freeze, prompts, run identity or execution authority.
Open decisions D1–D4 (brief reuse, overstated-caution rule, reviewer, execution) precede
freezing. T02 remains wait.