coulomb/frontend-patterns
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Report inconclusive presentation pilot and prepare human adjudication

Browse source 
Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a0cffe-206f-7323-89d3-b20ca4cf4bf9
This commit is contained in:
tegwick 2026-09-26 20:07:55 +02:00
parent 3690af12a5
commit 629bc2e880
10 changed files with 932 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

78
TREE.md
View file

@ -1,6 +1,6 @@
# Repository file inventory
Project files, including pilot receipts and reviews; updated 2026-09-25.
Project files, including pilot receipts and reviews; updated 2026-09-26.
```text
.custodian-brief.md
@ -256,6 +256,82 @@ research/catalog/SourceCatalog.md
research/compendium/FrontendArchitectureCompendium.md
research/concepts/GenerativeSequence.md
research/concepts/ResponsibilityModel.md
research/evaluations/language-pilot-01/README.md
research/evaluations/language-pilot-01/assignment.json
research/evaluations/language-pilot-01/editorial-scores-masked.json
research/evaluations/language-pilot-01/freeze.json
research/evaluations/language-pilot-01/human-review/README.md
research/evaluations/language-pilot-01/human-review/scores-template.json
research/evaluations/language-pilot-01/human-review/worksheet.md
research/evaluations/language-pilot-01/instructions.txt
research/evaluations/language-pilot-01/masked/S01-U.md
research/evaluations/language-pilot-01/masked/S01-V.md
research/evaluations/language-pilot-01/masked/S02-U.md
research/evaluations/language-pilot-01/masked/S02-V.md
research/evaluations/language-pilot-01/masked/S03-U.md
research/evaluations/language-pilot-01/masked/S03-V.md
research/evaluations/language-pilot-01/masked/S04-U.md
research/evaluations/language-pilot-01/masked/S04-V.md
research/evaluations/language-pilot-01/masked/S05-U.md
research/evaluations/language-pilot-01/masked/S05-V.md
research/evaluations/language-pilot-01/masked/S06-U.md
research/evaluations/language-pilot-01/masked/S06-V.md
research/evaluations/language-pilot-01/pack-equivalence.json
research/evaluations/language-pilot-01/pack-patterns.md
research/evaluations/language-pilot-01/pack-prose.md
research/evaluations/language-pilot-01/protocol.md
research/evaluations/language-pilot-01/raw/r01/answer.md
research/evaluations/language-pilot-01/raw/r01/prompt.txt
research/evaluations/language-pilot-01/raw/r01/receipt.json
research/evaluations/language-pilot-01/raw/r01/started.json
research/evaluations/language-pilot-01/raw/r02/answer.md
research/evaluations/language-pilot-01/raw/r02/prompt.txt
research/evaluations/language-pilot-01/raw/r02/receipt.json
research/evaluations/language-pilot-01/raw/r02/started.json
research/evaluations/language-pilot-01/raw/r03/answer.md
research/evaluations/language-pilot-01/raw/r03/prompt.txt
research/evaluations/language-pilot-01/raw/r03/receipt.json
research/evaluations/language-pilot-01/raw/r03/started.json
research/evaluations/language-pilot-01/raw/r04/answer.md
research/evaluations/language-pilot-01/raw/r04/prompt.txt
research/evaluations/language-pilot-01/raw/r04/receipt.json
research/evaluations/language-pilot-01/raw/r04/started.json
research/evaluations/language-pilot-01/raw/r05/answer.md
research/evaluations/language-pilot-01/raw/r05/prompt.txt
research/evaluations/language-pilot-01/raw/r05/receipt.json
research/evaluations/language-pilot-01/raw/r05/started.json
research/evaluations/language-pilot-01/raw/r06/answer.md
research/evaluations/language-pilot-01/raw/r06/prompt.txt
research/evaluations/language-pilot-01/raw/r06/receipt.json
research/evaluations/language-pilot-01/raw/r06/started.json
research/evaluations/language-pilot-01/raw/r07/answer.md
research/evaluations/language-pilot-01/raw/r07/prompt.txt
research/evaluations/language-pilot-01/raw/r07/receipt.json
research/evaluations/language-pilot-01/raw/r07/started.json
research/evaluations/language-pilot-01/raw/r08/answer.md
research/evaluations/language-pilot-01/raw/r08/prompt.txt
research/evaluations/language-pilot-01/raw/r08/receipt.json
research/evaluations/language-pilot-01/raw/r08/started.json
research/evaluations/language-pilot-01/raw/r09/answer.md
research/evaluations/language-pilot-01/raw/r09/prompt.txt
research/evaluations/language-pilot-01/raw/r09/receipt.json
research/evaluations/language-pilot-01/raw/r09/started.json
research/evaluations/language-pilot-01/raw/r10/answer.md
research/evaluations/language-pilot-01/raw/r10/prompt.txt
research/evaluations/language-pilot-01/raw/r10/receipt.json
research/evaluations/language-pilot-01/raw/r10/started.json
research/evaluations/language-pilot-01/raw/r11/answer.md
research/evaluations/language-pilot-01/raw/r11/prompt.txt
research/evaluations/language-pilot-01/raw/r11/receipt.json
research/evaluations/language-pilot-01/raw/r11/started.json
research/evaluations/language-pilot-01/raw/r12/answer.md
research/evaluations/language-pilot-01/raw/r12/prompt.txt
research/evaluations/language-pilot-01/raw/r12/receipt.json
research/evaluations/language-pilot-01/raw/r12/started.json
research/evaluations/language-pilot-01/run.py
research/evaluations/language-pilot-01/scenarios.json
research/evaluations/language-pilot-01/summarize.py
research/evaluations/language-pilot-01/summary.json
research/examples/SchemaEvidence.md
research/examples/schema-contract.yaml
research/experiments/matrix-recovery/README.md

2
research/README.md
View file

@ -34,3 +34,5 @@ Current contrast: [VC-01 server-validated editing](walkthroughs/VC01-Transaction
Latest implementation evidence: [frontend recovery](reviews/FrontendRecoveryImplementations.md) and [walkthrough correction review](reviews/RecoveryWalkthroughReview.md).
Executable evidence: [Matrix SDK recovery experiment](experiments/matrix-recovery/README.md), with a synthetic receiver and explicit limits.
Evaluation: [matched presentation pilot](evaluations/language-pilot-01/README.md) completed twelve llm-connect sessions; results inconclusive, human review pending.

8
research/catalog/PatternCandidateCatalog.md
View file

@ -79,3 +79,11 @@ Four additional hypothesis drafts: [Local Draft State](../candidates/LocalDraftS
The last elaborates the existing seed name. Eight developed research drafts now exist;
none has canonical admission. Server Authority remains context; Optimistic Overlay
remains deferred. See [dispositions and evidence](../reviews/TransactionalEvidenceExpansion.md).
## Presentation pilot — 2026-09-26
[Six-pair pilot](../evaluations/language-pilot-01/README.md): initial masked editorial
scoring found 21/24 obligations met with named patterns versus 19/24 with equivalent
prose, and two unsupported claims in each. Inconclusive; human review pending.
Retain all eight hypotheses, no new pattern or promotion. Do not mistake a small
presentation difference for proof of recurrence or improved architecture decisions.

115
research/evaluations/language-pilot-01/README.md Normal file
View file

@ -0,0 +1,115 @@
# Matched presentation pilot — 2026-09-26
**Outcome: inconclusive.** Twelve isolated llm-connect sessions completed. Initial
condition-masked Codex scoring found a small descriptive difference, with errors in
both presentations. Independent human review remains pending. No canonical admission
or claim of proven decision usefulness follows.
For independent scoring, begin with [the human handoff](human-review/README.md)
before reading the results or assignment key. The scorer who prepared this report
also authored the materials, so masking does not make the assessment independent.
## Design and execution
Corpus baseline `78bd951`; [protocol](protocol.md), equivalent packs, six scenarios
and rubric frozen in preregistration commit `6acbe6d`. Each model session received
only common instructions, one pack and one scenario brief. No scoring criteria,
other answers or conversation history were supplied. All eight guidance bodies and
evidence limits are shared; named headings and named connections are the treatment.
Pack sizes: 689 versus 663 words. This isolates presentation more narrowly than
comparing a full knowledge base with an uninformed baseline.
Six pairs, counterbalanced condition order (three each); one sample per cell. Every
call used llm-connect profile custodian-triage-balanced, resolved to
`google/gemini-2.5-flash`, temperature 0.2 and 1,400 requested output tokens.
Twelve calls, no retries, no failed/truncated generations. One pattern-condition
answer exceeded the requested 220 words; all stopped normally within token caps.
[Masked judgments](editorial-scores-masked.json) were committed in `3690af1` before
opening the assignment mapping. Assessor read U/V answers without receipts/packs/key;
no direct pattern-name leakage was noticed. Presentation could still be inferred.
The scoring record contains quotations/locators, explanations and ambiguity flags.
Raw answers were never edited. [Summary](summary.json) is derived by
[summarize.py](summarize.py), which verifies prompts and output hashes before counting.
## Initial descriptive results
| Measure | Named patterns | Equivalent prose |
| --- | ---: | ---: |
| Completed sessions | 6 | 6 |
| Required obligations met | 21 / 24 | 19 / 24 |
| Missed obligations | 3 | 5 |
| Constraint items met (C1 + C2) | 11 / 12 | 11 / 12 |
| Recovery items met (R) | 5 / 6 | 3 / 6 |
| Missing-contract items met (Q) | 5 / 6 | 5 / 6 |
| Unsupported claims flagged | 2 | 2 |
| Prompt tokens | 6,297 | 6,051 |
| Completion tokens | 1,137 | 1,192 |
| Total reported tokens | 7,434 | 7,243 |
| Summed request wall time | 11.997 s | 11.317 s |
| Provider-reported cost | $0.0047316 | $0.0047953 |
Total: 14,677 reported tokens, $0.0095269 provider-reported cost. These are diagnostics,
not a billing audit. Service latency is not human review time. Request order,
provider variance and the slightly longer pattern pack confound efficiency claims.
No significance tests or calibrated confidence estimates are warranted.
| Scenario | Obligations: patterns / prose | Descriptive advantage |
| --- | --- | --- |
| S01 Late receipt/new draft/opaque replica version | 4 / 3 | Patterns |
| S02 Identity lost across device change | 4 / 4 | Tie |
| S03 Revoked access and confidential local content | 3 / 3 | Tie |
| S04 Cancellation of an uncertain booking | 4 / 3 | Patterns |
| S05 Obsolete upload in flight after reconciliation | 3 / 3 | Tie |
| S06 Power failure, device loss and recovery copies | 3 / 3 | Prose: fewer unsupported claims |
“Advantage” is the preregistered dominance rule on fulfilled obligations and flagged
claims, not a declaration of superiority. Two pattern advantages, one prose advantage
and three ties are a mixed, small result on purpose-built development variants.
These scenarios are now consumed evaluation material; do not call tuned reruns held-out.
## What the errors teach us
Both conditions left meaningful gaps. In S03, answers asked about revocation policy
but also made unconditional retention or access statements before receiving it.
In S05 neither fully specified atomic retirement of P with c/Q despite that guidance
being present. In S06 neither asked for an independently recoverable backup/restore
contract. Clarification language alone does not ensure the recommendation actually
respects the missing information.
S01 prose asserted that a different opaque version was stale. S04 prose warned about
duplication but did not provide reconciliation before suggesting a fresh identity.
S06 patterns correctly denied a survival guarantee, then overstated that the storage
could not recover at all. Lack of evidence and evidence of impossibility differ.
Several judgments need adjudication: S03's retention phrasing versus its later policy
question; S04's fresh-identity suggestion versus its duplicate warning; S05's implicit
scope of the old receipt; S06's negative capability assertion and backup wording.
The original scores remain fixed. Alternative reasonable interpretations could change
pair counts; do not resolve ambiguity in favor of a preferred presentation.
## Decision and next step
Retain the eight hypotheses and both presentations. Do not expand the schema or
promote patterns based on this pilot. Obtain human ratings using the existing frozen
rubric, retain disagreement, then decide whether shared guidance needs clearer rules
for policy-dependent recovery, atomic supersession and evidence limits. Any revised
materials require fresh scenarios and replication before a usefulness claim.
Human review is queued under FEP-WP-0002 T03, with [an empty score sheet](human-review/scores-template.json).
It has not happened; no human agreement or independent validation is claimed.
## Reproduction and audit
From the repository root:
```sh
PYTHONDONTWRITEBYTECODE=1 ../infospace-bench/.venv/bin/python research/evaluations/language-pilot-01/run.py validate
python3 research/evaluations/language-pilot-01/summarize.py
```
These commands make no model calls. Validation checks the frozen material against
the working checkout; use the pilot snapshot if later corpus changes are present.
The inference runner refuses to overwrite an existing session directory. A new
replication needs a new workspace, protocol and explicit budget; no silent replay.
The assignment key is public in Git, so future masking is procedural, not enforced.

5
research/evaluations/language-pilot-01/human-review/README.md
View file

@ -1,13 +1,14 @@
# Human review handoff
Status: ready after masked answers are generated; no human scores yet.
Status: ready; all twelve masked answers are available; no human scores yet.
Start here without opening assignment.json, raw/, results or editorial scores.
Read [the scoring rules](../protocol.md) and [the scenarios/rubric](../scenarios.json).
Each scenario has U and V responses in [masked](../masked/). These are the actual
model answers without condition metadata; no semantic redaction was applied.
Copy [the empty scoring sheet](scores-template.json) to a new file. Enter your name,
Use the [combined worksheet](worksheet.md) for a readable form with all briefs and
answers, or copy [the empty scoring sheet](scores-template.json) to a new file. Enter your name,
date and whether you saw the assignment or prior scores. For each answer, score
C1, C2, R and Q as 1 or 0 using the frozen rubric, with quotations or locators.
List unsupported guarantees separately (use [] only after checking). No credit for

259
research/evaluations/language-pilot-01/human-review/worksheet.md Normal file
View file

@ -0,0 +1,259 @@
# Human scoring worksheet
Reviewer: __________ Date: __________
Have you seen the assignment key or prior scores? __________
For each criterion mark 1 if explicitly and correctly met, otherwise 0. Leave blanks
until reviewed. Record unsupported guarantees separately with a quote. Do not reward
pattern names. See [instructions](README.md). This sheet contains no condition labels
or prior ratings. Keep all original answers unchanged.
## S01 — Late receipt and a newer draft
A server-validated record editor submits operation A. Its request promise rejects, but an authenticated server event identifying A and its accepted revision arrives. The user has already typed draft B. A previously started replica read then returns a different opaque version; the replica consistency contract is unspecified. Explain the correct visible states and next actions.
### Answer U
This architecture should result in the following:
* **Outcome/State:** Operation A is confirmed accepted, and draft B is preserved. The replica read is superseded and ignored.
* **Preserved Information:** The accepted revision of A, the full content of draft B, and the identity of operation A are retained.
* **Recovery Action and Rejected Alternative:** The system should refresh observations based on the accepted revision of A, while keeping draft B visible. Blindly overwriting draft B with the stale replica data is incorrect.
* **Missing Contract or Clarification:** The replica consistency contract needs to be explicitly defined to determine how and when its data should be considered authoritative or how to reconcile it with confirmed server state.
### Answer V
* **Outcome/State:** Operation A is accepted, but the replica is stale, and draft B is present.
* **Preserved Information:** Accepted revision of A, draft B, and the original identity of A.
* **Recovery Action:** Refresh the affected observations while preserving draft B. Do not retry A with a new identity.
* **Missing Contract/Clarification:** The replica consistency contract needs to be explicitly defined to understand how to handle the stale read.
### Your assessment
| Criterion | Requirement | U (0/1) | V (0/1) |
| --- | --- | --- | --- |
| C1 | Keep A accepted despite the rejected promise. | | |
| C2 | Preserve draft B; receipt for A cannot clear it. | | |
| R | Do not replace current data based on opaque-version ordering or equate acceptance with freshness; mark freshness unresolved or gate stale reads. | | |
| Q | Identify the missing read-after-write/server ordering contract before asserting freshness. | | |
Prohibited recommendations to check:
- Treat the rejected promise as proof A failed.
- Erase B because A completed.
- Numerically or lexically order opaque versions as proof of freshness.
Evidence for your ratings / unsupported claims / ambiguities:
____________________
## S02 — Retry after losing operation identity
A mobile message composer queued an immutable payload and sent it. The response was lost. After logout, reinstall and login as a new device, only the payload remains; the original operation ID is lost. The server deduplication scope is device and endpoint. Lookup support and retention are unspecified. The product suggests generating a fresh ID and automatically retrying. Evaluate that suggestion and give a recovery plan.
### Answer U
Here's a recommendation for the scenario:
* **Outcome/State:** The original message was sent, but its delivery status is unknown due to a lost response. The local operation ID is gone, and the server's deduplication scope is device-specific.
* **Preserved Information:** Only the immutable message payload remains locally. The original operation identity and any server-side acceptance evidence are lost to the client.
* **Recovery Action and Rejected Alternative:** The product's suggestion to generate a fresh ID and retry is problematic. This risks sending a duplicate message if the original was accepted. Instead, the user should be prompted to manually reconcile the message. This could involve presenting the message to the user and asking if they want to resend it, acknowledging potential duplication.
* **Missing Contract or Clarification:** The server's lookup support and retention for operation IDs are critical. Without these, the client cannot determine if the original message was processed. A contract for server-side reconciliation or query by payload content would be beneficial.
### Answer V
Here's an evaluation and recovery plan:
* **Outcome/State:** The original message payload is present, but its operation identity is lost, leading to an unknown server outcome for the initial send attempt.
* **Preserved Information:** Only the immutable message payload itself is retained across device changes and reinstallation.
* **Recovery Action and Rejected Alternative:** The product's suggestion to generate a fresh ID and automatically retry is flawed. This risks duplicating the message if the original send was successful. Instead, the user must be prompted to manually confirm re-sending the message, acknowledging the potential for duplication.
* **Missing Contract or Clarification:** The server's deduplication scope and retention for operation IDs are critical but unspecified. Without this, it's impossible to know if the original message was processed or if a fresh ID would truly be safe.
### Your assessment
| Criterion | Requirement | U (0/1) | V (0/1) |
| --- | --- | --- | --- |
| C1 | Outcome remains unknown; same payload does not establish rejection or identity. | | |
| C2 | A fresh ID/new device cannot be claimed duplicate-safe; do not automatically resend as recovery. | | |
| R | Preserve the payload separately and resolve/check the old outcome before a new send, or require an explicit informed reconciliation decision. | | |
| Q | Ask about operation/history lookup and retention or recoverability of the old device/operation identity. | | |
Prohibited recommendations to check:
- Claim identical content guarantees idempotency across devices.
- Promise automatic fresh-ID retry cannot duplicate.
Evidence for your ratings / unsupported claims / ambiguities:
____________________
## S03 — Revoked access during offline work
An offline field editor has a completed local commit of confidential notes and a pending upload. On reconnect, the server explicitly denies write authorization. The product requirement only says "never lose edits"; it says nothing about whether revoked users may retain, view or export local copies. Design the next states and recovery behavior without inventing an access policy.
### Answer U
Here's a recommendation for the frontend architecture:
* **Outcome/State:** The editor should display the confidential notes as locally committed but with a clear, persistent indication that the pending upload was explicitly rejected due to authorization failure. The notes remain editable locally.
* **Preserved Information:** The full content of the confidential notes, their local commit identity, and the identity of the rejected upload attempt must be retained. The explicit server rejection message should also be preserved and associated with the failed upload.
* **Recovery Action and Rejected Alternative:** The user should be presented with options to either discard the local changes (if they choose to, understanding the implications) or attempt to export/save the content locally. Blindly retrying the upload with a new identity is rejected as it doesn't address the authorization issue and could lead to data duplication if the server's rejection was transient.
* **Missing Contract or Clarification:** A clear contract is needed regarding the retention and access policies for locally committed data when server authorization is revoked. Specifically, can the user continue to view, edit, or export these notes, or should they be purged?
### Answer V
This architecture needs refinement to handle authorization denial gracefully.
* **Outcome/State:** The local commit remains, but the pending upload is explicitly rejected due to authorization. The editor is in a state of "accepted locally, rejected remotely."
* **Preserved Information:** The full local commit of confidential notes, including its revision and operation identity, must be retained. The server's explicit denial and the reason (authorization failure) should also be recorded.
* **Recovery Action and Rejected Alternative:** The system should prompt the user to explicitly reconcile the rejected upload, perhaps by allowing them to save the notes locally or export them. Blindly discarding the local commit or attempting a re-upload with a new identity is inappropriate as it would lose data and violate the "never lose edits" requirement.
* **Missing Contract or Clarification:** A clear contract is needed regarding the retention, viewing, and export of local data for users whose write authorization has been revoked. This will dictate the specific recovery options presented to the user.
### Your assessment
| Criterion | Requirement | U (0/1) | V (0/1) |
| --- | --- | --- | --- |
| C1 | Distinguish saved locally from uploaded/accepted remotely. | | |
| C2 | Block unauthorized synchronization and do not bypass revocation or endlessly retry unchanged credentials. | | |
| R | Make recovery conditional on authorized access/policy; do not promise unconditional export, permanent retention or silent destructive deletion. | | |
| Q | Explicitly request the policy for retaining, viewing or deleting/exporting confidential copies after revocation. | | |
Prohibited recommendations to check:
- Assert never-lose-edits authorizes export despite revocation.
- Unconditionally wipe or retain all confidential content as if policy were known.
Evidence for your ratings / unsupported claims / ambiguities:
____________________
## S04 — Local cancellation after uncertain booking
A server-confirmed booking UI times out after submitting operation K. Before receiving authoritative evidence, the user presses Cancel; the client removes its pending entry. The user then asks to submit the same booking again. The server exposes no documented operation-status lookup or deduplication retention guarantee. Specify what Cancel can honestly mean and whether a new submission is safe.
### Answer U
The proposed architecture has a critical flaw for this scenario.
* **Outcome/State:** The booking's final state is unknown. The server may have accepted operation K despite the timeout and local cancellation.
* **Preserved Information:** The client has discarded the original operation K's identity and intent.
* **Recovery Action:** The only safe action is to inform the user that the previous booking's status is unknown and they should check with the service provider. A new submission is unsafe.
* **Missing Contract:** The server lacks a documented operation-status lookup or deduplication retention guarantee, which is essential for safely retrying or confirming operations after client-side uncertainty.
### Answer V
Here's a recommendation for the scenario:
* **Outcome/State:** The client should not remove the pending entry for operation K upon local "Cancel." The server's status for K is unknown.
* **Preserved Information:** The client must retain the original operation K's identity and payload, along with its "pending" status, even after local cancellation.
* **Recovery Action and Rejected Alternative:** The user's request to submit the "same booking again" should be treated as a new, distinct operation with a fresh identity. Blindly retrying K with its original identity is rejected because the server lacks deduplication guarantees.
* **Missing Contract or Clarification:** A critical missing contract is the server's behavior regarding duplicate submissions and its retention window for operation identities. Without this, the client cannot safely assume a new submission will not result in a duplicate booking if K was, in fact, accepted.
### Your assessment
| Criterion | Requirement | U (0/1) | V (0/1) |
| --- | --- | --- | --- |
| C1 | Local cancellation does not establish remote cancellation or rejection of K. | | |
| C2 | Do not claim a new submission is duplicate-safe while K is unresolved. | | |
| R | Use an authoritative reconciliation/escalation path before replacement; remote reversal is a separate authorized action if K was accepted. | | |
| Q | Ask about cancellation, lookup/history and retry-retention contracts needed to resolve K. | | |
Prohibited recommendations to check:
- Tell the user the booking was undone by removing local state.
- Automatically use a new ID and promise no duplicate booking.
Evidence for your ratings / unsupported claims / ambiguities:
____________________
## S05 — An old upload survives conflict resolution
Device A durably recorded offline revision a with pending operation P. Another device uploaded divergent revision b. A preserves both and resolves them into revision c with new pending operation Q. P was already in flight. Later a receipt for P arrives. Local commits survive process restart, but receiver ancestry/precondition behavior is unspecified. Specify queue transitions, crash boundaries and what the late receipt may change.
### Answer U
Here's a recommendation for the frontend architecture based on the scenario:
* **Outcome/State:** The system should transition to a state where revision `c` is the active local content, with pending operation `Q` awaiting remote acceptance. The late receipt for `P` should update its own operation history, marking `P` as remotely accepted, but it should not alter the active content or the status of `Q`.
* **Preserved Information:** Both divergent revisions `a` and `b` should be preserved with their identities and provenance until `c` is fully committed and resolved. The original pending operation `P` and its identity must also be retained, along with the new pending operation `Q`.
* **Recovery Action and Rejected Alternative:** Upon process restart, the system must recover `c` as the active content and `Q` as the pending operation. The late receipt for `P` should not trigger a re-application of `P` or a rollback to `a`, as this would erase the user's explicit resolution into `c`.
* **Missing Contract or Clarification:** The architecture needs a clear contract for the receiver's ancestry and precondition behavior. Specifically, it must define how the receiver handles an in-flight operation (`P`) when a superseding operation (`Q` derived from `c`) is subsequently sent, and how it would respond to `P` if `b` was already accepted.
### Answer V
This architecture has several issues that could lead to data loss or corruption.
* **Outcome/State:** The system will transition from having pending operations P and Q, and revisions a, b, and c, to a state where P's receipt arrives. The system needs to define how the late receipt for P interacts with the already-resolved revision c and the new pending operation Q.
* **Preserved Information:** The system must preserve revisions a, b, and c, along with the identities of pending operations P and Q, and the fact that P was in flight. The receipt for P, when it arrives, must also be preserved with its associated operation identity.
* **Recovery Action and Rejected Alternative:** Upon receiving the late receipt for P, the system should update P's status to "accepted" but **not** reapply P's changes to revision c or Q. Blindly applying P's changes would overwrite the explicit reconciliation that produced c. Instead, the system should acknowledge P's acceptance and potentially flag that P's original intent was superseded by the reconciliation.
* **Missing Contract or Clarification:** The critical missing contract is the receiver's ancestry and precondition behavior. Without knowing how the receiver handles concurrent updates and what preconditions it enforces, it's impossible to guarantee that P's original intent wasn't already superseded on the server, or how the server would react to a re-submission of P after c was uploaded.
### Your assessment
| Criterion | Requirement | U (0/1) | V (0/1) |
| --- | --- | --- | --- |
| C1 | Block/supersede P atomically with the resolved revision and Q; maintain restart-safe pending intent. | | |
| C2 | A receipt for P updates P history only, not Q status or the active resolved revision. | | |
| R | Do not assume local queue retirement stops an in-flight P; require receiver preconditions/ancestry to protect c. | | |
| Q | Explicitly identify receiver ancestry/version handling as a missing contract requiring verification. | | |
Prohibited recommendations to check:
- Mark Q synchronized because P was acknowledged.
- Guarantee local cancellation prevents P from overwriting remotely.
Evidence for your ratings / unsupported claims / ambiguities:
____________________
## S06 — Storage guarantee beyond the evidence
A field-note product promises edits survive device loss and complete power failure. The chosen local transaction has only been tested against ordinary process restart. Two copies of a conflict are stored on the same device, and the sync protocol propagates deletions. The team proposes declaring recovery solved and removing conflict review from the keyboard interface. Assess that design and state what can actually be promised.
### Answer U
Here's an assessment of the proposed design:
* **Outcome/State:** The design cannot guarantee edits survive device loss or complete power failure, nor can it claim recovery is solved.
* **Preserved Information:** Only two copies of a conflict are stored on the same device, and the sync protocol propagates deletions, which risks data loss.
* **Recovery Action and Rejected Alternative:** The proposal to remove conflict review from the keyboard interface is premature and dangerous, as it eliminates the user's ability to resolve divergent versions.
* **Missing Contract or Clarification:** There is no documented contract for restart-safe offline editing that covers device loss or power failure, nor is there a clear definition of how deletions are handled during conflict resolution.
### Answer V
Here's an assessment of the proposed frontend architecture:
* **Outcome/State:** The product cannot guarantee survival of edits through device loss or complete power failure, nor can it claim conflict resolution is solved.
* **Preserved Information:** Only two copies of a conflict are retained on the same device, and the sync protocol propagates deletions, which is insufficient for robust recovery.
* **Recovery Action and Rejected Alternative:** The current local transaction only handles process restarts; it cannot recover from power or device loss. Removing conflict review from the keyboard interface is premature and risky.
* **Missing Contract or Clarification:** There's no documented contract for restart-safe storage beyond process restart, nor for how deletions are handled during conflict resolution or synchronization across devices.
### Your assessment
| Criterion | Requirement | U (0/1) | V (0/1) |
| --- | --- | --- | --- |
| C1 | Ordinary restart evidence cannot establish power-loss or device-loss survival. | | |
| C2 | Copies on one device and deletion-propagating sync are not independent backup. | | |
| R | Retain accessible discoverable comparison/resolution; storing conflict copies alone does not resolve intent. | | |
| Q | Request storage flush/power-loss evidence and an independently recoverable backup/restore contract. | | |
Prohibited recommendations to check:
- Claim local transaction or same-device copies guarantee device-loss recovery.
- Treat synchronization as sufficient independent backup.
- Approve inaccessible conflict review because bytes are preserved.
Evidence for your ratings / unsupported claims / ambiguities:
____________________

75
research/evaluations/language-pilot-01/summarize.py Normal file
View file

@ -0,0 +1,75 @@
"""Unmask frozen editorial judgments and verify all evidence before aggregation."""
import json
from pathlib import Path
from hashlib import sha256
ROOT = Path(__file__).resolve().parent
def read(name):
return json.loads((ROOT/name).read_text())
def digest(path):
return sha256(path.read_bytes()).hexdigest()
scores = read('editorial-scores-masked.json')
score_by_id = {a['answer']: a for a in scores['answers']}
assert len(score_by_id) == 12
rubric = {s['id']: s for s in read('scenarios.json')}
result = {'preregistration_commit':'6acbe6d', 'masked_scoring_commit':'3690af1',
'independent_human_review':'pending', 'interpretation':'exploratory descriptive counts only',
'score_sha256':digest(ROOT/'editorial-scores-masked.json'), 'conditions':{}, 'pairs':{}}
for condition in ['patterns','prose']:
result['conditions'][condition] = {'completed':0, 'failed':0, 'requirements_met':0,
'requirements_possible':0, 'criteria_met':dict.fromkeys(['C1','C2','R','Q'],0),
'unsupported_claims':0, 'calls':0, 'prompt_tokens':0, 'completion_tokens':0,
'total_tokens':0, 'elapsed_seconds':0, 'provider_reported_cost_usd':0,
'over_requested_word_limit':0, 'resolved_models':[], 'answers':[]}
for a in read('assignment.json'):
raw = ROOT/'raw'/a['run']; receipt = json.loads((raw/'receipt.json').read_text())
c = result['conditions'][a['condition']]
assert receipt['condition'] == a['condition'] and receipt['scenario'] == a['scenario']
assert receipt['freeze_sha256'] == digest(ROOT/'freeze.json')
assert receipt['prompt_sha256'] == digest(raw/'prompt.txt')
expected = ((ROOT/'instructions.txt').read_text()+'\nREFERENCE\n'+
(ROOT/f"pack-{a['condition']}.md").read_text()+'\nSCENARIO\n'+rubric[a['scenario']]['brief'])
assert expected == (raw/'prompt.txt').read_text(), 'Prompt contains unexpected input'
c['calls'] += receipt['service_requests_attempted']
c['elapsed_seconds'] += receipt['elapsed_seconds']
if receipt['status'] != 'completed':
c['failed'] += 1
continue
c['completed'] += 1
score = score_by_id[a['blind_id']]
assert digest(raw/'answer.md') == digest(ROOT/'masked'/(a['blind_id']+'.md')) == score['answer_sha256']
assert len(receipt['responses']) == 1
response = receipt['responses'][0]
assert response['finish_reason'] == 'stop'
assert response['content_sha256'] == digest(raw/'answer.md')
for key in ['prompt_tokens','completion_tokens','total_tokens']:
c[key] += response['usage'][key]
c['provider_reported_cost_usd'] += response['diagnostics']['provider_usage']['cost']
if response['model'] not in c['resolved_models']: c['resolved_models'].append(response['model'])
c['over_requested_word_limit'] += int(receipt['output_words'] > 220)
values = {k:v['score'] for k,v in score['criteria'].items()}
assert set(values)=={'C1','C2','R','Q'} and all(v in [0,1] for v in values.values())
met = sum(values.values()); unsafe=len(score['unsupported_claims'])
c['requirements_met'] += met; c['requirements_possible'] += 4
c['unsupported_claims'] += unsafe
for k,v in values.items(): c['criteria_met'][k] += v
record={'answer':a['blind_id'], 'run':a['run'], 'met':met, 'unsupported':unsafe, 'criteria':values}
c['answers'].append(record)
result['pairs'].setdefault(a['scenario'],{})[a['condition']]=record
for scenario,pair in result['pairs'].items():
p,b=pair['patterns'],pair['prose']
pm,bm=p['met'],b['met']; pu,bu=p['unsupported'],b['unsupported']
if pm==bm and pu==bu: outcome='tie'
elif pm>=bm and pu<=bu: outcome='patterns'
elif bm>=pm and bu<=pu: outcome='prose'
else: outcome='mixed'
pair['descriptive_advantage']=outcome
for c in result['conditions'].values():
c['missed_requirements']=c['requirements_possible']-c['requirements_met']
c['elapsed_seconds']=round(c['elapsed_seconds'],3)
c['provider_reported_cost_usd']=round(c['provider_reported_cost_usd'],7)
(ROOT/'summary.json').write_text(json.dumps(result,indent=2)+'\n')
print(json.dumps(result,indent=2))

369
research/evaluations/language-pilot-01/summary.json Normal file
View file

@ -0,0 +1,369 @@
{
"preregistration_commit": "6acbe6d",
"masked_scoring_commit": "3690af1",
"independent_human_review": "pending",
"interpretation": "exploratory descriptive counts only",
"score_sha256": "d7d7fa0b688f5db93709821a076157acb2723791c99736fc296f08e1627b73b9",
"conditions": {
"patterns": {
"completed": 6,
"failed": 0,
"requirements_met": 21,
"requirements_possible": 24,
"criteria_met": {
"C1": 5,
"C2": 6,
"R": 5,
"Q": 5
},
"unsupported_claims": 2,
"calls": 6,
"prompt_tokens": 6297,
"completion_tokens": 1137,
"total_tokens": 7434,
"elapsed_seconds": 11.997,
"provider_reported_cost_usd": 0.0047316,
"over_requested_word_limit": 1,
"resolved_models": [
"google/gemini-2.5-flash"
],
"answers": [
{
"answer": "S01-U",
"run": "r01",
"met": 4,
"unsupported": 0,
"criteria": {
"C1": 1,
"C2": 1,
"R": 1,
"Q": 1
}
},
{
"answer": "S02-V",
"run": "r04",
"met": 4,
"unsupported": 0,
"criteria": {
"C1": 1,
"C2": 1,
"R": 1,
"Q": 1
}
},
{
"answer": "S03-V",
"run": "r05",
"met": 3,
"unsupported": 1,
"criteria": {
"C1": 1,
"C2": 1,
"R": 0,
"Q": 1
}
},
{
"answer": "S04-U",
"run": "r08",
"met": 4,
"unsupported": 0,
"criteria": {
"C1": 1,
"C2": 1,
"R": 1,
"Q": 1
}
},
{
"answer": "S05-V",
"run": "r09",
"met": 3,
"unsupported": 0,
"criteria": {
"C1": 0,
"C2": 1,
"R": 1,
"Q": 1
}
},
{
"answer": "S06-V",
"run": "r12",
"met": 3,
"unsupported": 1,
"criteria": {
"C1": 1,
"C2": 1,
"R": 1,
"Q": 0
}
}
],
"missed_requirements": 3
},
"prose": {
"completed": 6,
"failed": 0,
"requirements_met": 19,
"requirements_possible": 24,
"criteria_met": {
"C1": 5,
"C2": 6,
"R": 3,
"Q": 5
},
"unsupported_claims": 2,
"calls": 6,
"prompt_tokens": 6051,
"completion_tokens": 1192,
"total_tokens": 7243,
"elapsed_seconds": 11.317,
"provider_reported_cost_usd": 0.0047953,
"over_requested_word_limit": 0,
"resolved_models": [
"google/gemini-2.5-flash"
],
"answers": [
{
"answer": "S01-V",
"run": "r02",
"met": 3,
"unsupported": 1,
"criteria": {
"C1": 1,
"C2": 1,
"R": 0,
"Q": 1
}
},
{
"answer": "S02-U",
"run": "r03",
"met": 4,
"unsupported": 0,
"criteria": {
"C1": 1,
"C2": 1,
"R": 1,
"Q": 1
}
},
{
"answer": "S03-U",
"run": "r06",
"met": 3,
"unsupported": 1,
"criteria": {
"C1": 1,
"C2": 1,
"R": 0,
"Q": 1
}
},
{
"answer": "S04-V",
"run": "r07",
"met": 3,
"unsupported": 0,
"criteria": {
"C1": 1,
"C2": 1,
"R": 0,
"Q": 1
}
},
{
"answer": "S05-U",
"run": "r10",
"met": 3,
"unsupported": 0,
"criteria": {
"C1": 0,
"C2": 1,
"R": 1,
"Q": 1
}
},
{
"answer": "S06-U",
"run": "r11",
"met": 3,
"unsupported": 0,
"criteria": {
"C1": 1,
"C2": 1,
"R": 1,
"Q": 0
}
}
],
"missed_requirements": 5
}
},
"pairs": {
"S01": {
"patterns": {
"answer": "S01-U",
"run": "r01",
"met": 4,
"unsupported": 0,
"criteria": {
"C1": 1,
"C2": 1,
"R": 1,
"Q": 1
}
},
"prose": {
"answer": "S01-V",
"run": "r02",
"met": 3,
"unsupported": 1,
"criteria": {
"C1": 1,
"C2": 1,
"R": 0,
"Q": 1
}
},
"descriptive_advantage": "patterns"
},
"S02": {
"prose": {
"answer": "S02-U",
"run": "r03",
"met": 4,
"unsupported": 0,
"criteria": {
"C1": 1,
"C2": 1,
"R": 1,
"Q": 1
}
},
"patterns": {
"answer": "S02-V",
"run": "r04",
"met": 4,
"unsupported": 0,
"criteria": {
"C1": 1,
"C2": 1,
"R": 1,
"Q": 1
}
},
"descriptive_advantage": "tie"
},
"S03": {
"patterns": {
"answer": "S03-V",
"run": "r05",
"met": 3,
"unsupported": 1,
"criteria": {
"C1": 1,
"C2": 1,
"R": 0,
"Q": 1
}
},
"prose": {
"answer": "S03-U",
"run": "r06",
"met": 3,
"unsupported": 1,
"criteria": {
"C1": 1,
"C2": 1,
"R": 0,
"Q": 1
}
},
"descriptive_advantage": "tie"
},
"S04": {
"prose": {
"answer": "S04-V",
"run": "r07",
"met": 3,
"unsupported": 0,
"criteria": {
"C1": 1,
"C2": 1,
"R": 0,
"Q": 1
}
},
"patterns": {
"answer": "S04-U",
"run": "r08",
"met": 4,
"unsupported": 0,
"criteria": {
"C1": 1,
"C2": 1,
"R": 1,
"Q": 1
}
},
"descriptive_advantage": "patterns"
},
"S05": {
"patterns": {
"answer": "S05-V",
"run": "r09",
"met": 3,
"unsupported": 0,
"criteria": {
"C1": 0,
"C2": 1,
"R": 1,
"Q": 1
}
},
"prose": {
"answer": "S05-U",
"run": "r10",
"met": 3,
"unsupported": 0,
"criteria": {
"C1": 0,
"C2": 1,
"R": 1,
"Q": 1
}
},
"descriptive_advantage": "tie"
},
"S06": {
"prose": {
"answer": "S06-U",
"run": "r11",
"met": 3,
"unsupported": 0,
"criteria": {
"C1": 1,
"C2": 1,
"R": 1,
"Q": 0
}
},
"patterns": {
"answer": "S06-V",
"run": "r12",
"met": 3,
"unsupported": 1,
"criteria": {
"C1": 1,
"C2": 1,
"R": 1,
"Q": 0
}
},
"descriptive_advantage": "prose"
}
}
}

8
research/plan/OpenQuestions.md
View file

@ -57,3 +57,11 @@ The failed provider runs are execution failures, not negative evidence for the m
[VC-01/VC-02 assessment](../reviews/VC01-VC02-Comparison.md) sharpens Q02: local drafts do not imply local authority. Q03 lacks independent frontend uses; Q04 remains unknown despite two complete synthetic narratives. No question is closed by pattern count.
2026-09-25 follow-up: [pinned frontend code](../reviews/FrontendRecoveryImplementations.md) supports two partial draft-separation uses, but not full-pattern recurrence. [Review attempt](../reviews/RecoveryWalkthroughReview.md) yielded no adopted model fixes; editorial lifecycle corrections do not resolve Q04 usefulness.
## Q04 presentation pilot — 2026-09-26
[Matched pilot](../evaluations/language-pilot-01/README.md) completed six new briefs
under two equivalent-information conditions. Q04 remains open: small mixed descriptive
differences, same-author editorial scoring and pending human adjudication do not
establish usefulness. The six briefs are consumed pilot material, not future held-out
items. Next: independent ratings before changing guidance or replicating.

16
workplans/FEP-WP-0002-seed-language-validation.md
View file

@ -288,3 +288,19 @@ condition-masked editorial scores before unmasking. Do not claim human validatio
Human follow-up remains part of T03: prepare masked answers/rubric/empty score sheet
for independent human review, record disagreements and adjudication before claiming
usefulness. No canonical admission or M1 completion follows from the model pilot.
## Presentation pilot results and human handoff — 2026-09-26
T03: all 12 isolated sessions completed through llm-connect, 14,677 reported tokens,
no retries or truncation. Preregistration 6acbe6d; masked editorial scores committed
in 3690af1 before unmasking. Named presentation met 21/24 obligations versus 19/24
for equivalent prose; each had two unsupported claims. Pair comparison: two named
advantages, one prose advantage, three ties. Inconclusive, no human validation.
Human follow-up is explicitly pending within T03: use
research/evaluations/language-pilot-01/human-review/README.md and its empty score
sheet to obtain independent ratings, retain disagreements and adjudicate against
the frozen rubric. No human scores may be inferred or filled by a model. Shared
weaknesses concern policy-dependent retention, atomic supersession and backup/restore.
After human review, revise shared guidance if warranted and replicate on fresh
scenarios. These six briefs are consumed. T02/T03 remain progress; no M1 completion.