tegwick
21030784b1
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a0cffe-206f-7323-89d3-b20ca4cf4bf9
121 lines
6.6 KiB
Markdown
121 lines
6.6 KiB
Markdown
# Matched presentation pilot — 2026-09-26
|
|
|
|
**Outcome: inconclusive.** Twelve isolated llm-connect sessions completed. Initial
|
|
condition-masked Codex scoring found a small descriptive difference, with errors in
|
|
both presentations. Independent human review remains pending. No canonical admission
|
|
or claim of proven decision usefulness follows.
|
|
|
|
Update: the [user-submitted review and adjudication](submitted-review-01/adjudication.md)
|
|
changes four binary ratings, giving **18/24 obligations met in each condition**.
|
|
The worksheet declares model authorship and exposure to the aggregate summary;
|
|
independent human validation is still unfulfilled. The figures below preserve the
|
|
initial scoring, not the latest adjudicated totals.
|
|
|
|
For independent scoring, begin with [the human handoff](human-review/README.md)
|
|
before reading the results or assignment key. The scorer who prepared this report
|
|
also authored the materials, so masking does not make the assessment independent.
|
|
|
|
## Design and execution
|
|
|
|
Corpus baseline `78bd951`; [protocol](protocol.md), equivalent packs, six scenarios
|
|
and rubric frozen in preregistration commit `6acbe6d`. Each model session received
|
|
only common instructions, one pack and one scenario brief. No scoring criteria,
|
|
other answers or conversation history were supplied. All eight guidance bodies and
|
|
evidence limits are shared; named headings and named connections are the treatment.
|
|
Pack sizes: 689 versus 663 words. This isolates presentation more narrowly than
|
|
comparing a full knowledge base with an uninformed baseline.
|
|
|
|
Six pairs, counterbalanced condition order (three each); one sample per cell. Every
|
|
call used llm-connect profile custodian-triage-balanced, resolved to
|
|
`google/gemini-2.5-flash`, temperature 0.2 and 1,400 requested output tokens.
|
|
Twelve calls, no retries, no failed/truncated generations. One pattern-condition
|
|
answer exceeded the requested 220 words; all stopped normally within token caps.
|
|
|
|
[Masked judgments](editorial-scores-masked.json) were committed in `3690af1` before
|
|
opening the assignment mapping. Assessor read U/V answers without receipts/packs/key;
|
|
no direct pattern-name leakage was noticed. Presentation could still be inferred.
|
|
The scoring record contains quotations/locators, explanations and ambiguity flags.
|
|
Raw answers were never edited. [Summary](summary.json) is derived by
|
|
[summarize.py](summarize.py), which verifies prompts and output hashes before counting.
|
|
|
|
## Initial descriptive results
|
|
|
|
| Measure | Named patterns | Equivalent prose |
|
|
| --- | ---: | ---: |
|
|
| Completed sessions | 6 | 6 |
|
|
| Required obligations met | 21 / 24 | 19 / 24 |
|
|
| Missed obligations | 3 | 5 |
|
|
| Constraint items met (C1 + C2) | 11 / 12 | 11 / 12 |
|
|
| Recovery items met (R) | 5 / 6 | 3 / 6 |
|
|
| Missing-contract items met (Q) | 5 / 6 | 5 / 6 |
|
|
| Unsupported claims flagged | 2 | 2 |
|
|
| Prompt tokens | 6,297 | 6,051 |
|
|
| Completion tokens | 1,137 | 1,192 |
|
|
| Total reported tokens | 7,434 | 7,243 |
|
|
| Summed request wall time | 11.997 s | 11.317 s |
|
|
| Provider-reported cost | $0.0047316 | $0.0047953 |
|
|
|
|
Total: 14,677 reported tokens, $0.0095269 provider-reported cost. These are diagnostics,
|
|
not a billing audit. Service latency is not human review time. Request order,
|
|
provider variance and the slightly longer pattern pack confound efficiency claims.
|
|
No significance tests or calibrated confidence estimates are warranted.
|
|
|
|
| Scenario | Obligations: patterns / prose | Descriptive advantage |
|
|
| --- | --- | --- |
|
|
| S01 Late receipt/new draft/opaque replica version | 4 / 3 | Patterns |
|
|
| S02 Identity lost across device change | 4 / 4 | Tie |
|
|
| S03 Revoked access and confidential local content | 3 / 3 | Tie |
|
|
| S04 Cancellation of an uncertain booking | 4 / 3 | Patterns |
|
|
| S05 Obsolete upload in flight after reconciliation | 3 / 3 | Tie |
|
|
| S06 Power failure, device loss and recovery copies | 3 / 3 | Prose: fewer unsupported claims |
|
|
|
|
“Advantage” is the preregistered dominance rule on fulfilled obligations and flagged
|
|
claims, not a declaration of superiority. Two pattern advantages, one prose advantage
|
|
and three ties are a mixed, small result on purpose-built development variants.
|
|
These scenarios are now consumed evaluation material; do not call tuned reruns held-out.
|
|
|
|
## What the errors teach us
|
|
|
|
Both conditions left meaningful gaps. In S03, answers asked about revocation policy
|
|
but also made unconditional retention or access statements before receiving it.
|
|
In S05 neither fully specified atomic retirement of P with c/Q despite that guidance
|
|
being present. In S06 neither asked for an independently recoverable backup/restore
|
|
contract. Clarification language alone does not ensure the recommendation actually
|
|
respects the missing information.
|
|
|
|
S01 prose asserted that a different opaque version was stale. S04 prose warned about
|
|
duplication but did not provide reconciliation before suggesting a fresh identity.
|
|
S06 patterns correctly denied a survival guarantee, then overstated that the storage
|
|
could not recover at all. Lack of evidence and evidence of impossibility differ.
|
|
|
|
Several judgments need adjudication: S03's retention phrasing versus its later policy
|
|
question; S04's fresh-identity suggestion versus its duplicate warning; S05's implicit
|
|
scope of the old receipt; S06's negative capability assertion and backup wording.
|
|
The original scores remain fixed. Alternative reasonable interpretations could change
|
|
pair counts; do not resolve ambiguity in favor of a preferred presentation.
|
|
|
|
## Decision and next step
|
|
|
|
Retain the eight hypotheses and both presentations. Do not expand the schema or
|
|
promote patterns based on this pilot. Obtain human ratings using the existing frozen
|
|
rubric, retain disagreement, then decide whether shared guidance needs clearer rules
|
|
for policy-dependent recovery, atomic supersession and evidence limits. Any revised
|
|
materials require fresh scenarios and replication before a usefulness claim.
|
|
|
|
Human review is queued under FEP-WP-0002 T03, with [an empty score sheet](human-review/scores-template.json).
|
|
It has not happened; no human agreement or independent validation is claimed.
|
|
|
|
## Reproduction and audit
|
|
|
|
From the repository root:
|
|
|
|
```sh
|
|
PYTHONDONTWRITEBYTECODE=1 ../infospace-bench/.venv/bin/python research/evaluations/language-pilot-01/run.py validate
|
|
python3 research/evaluations/language-pilot-01/summarize.py
|
|
```
|
|
|
|
These commands make no model calls. Validation checks the frozen material against
|
|
the working checkout; use the pilot snapshot if later corpus changes are present.
|
|
The inference runner refuses to overwrite an existing session directory. A new
|
|
replication needs a new workspace, protocol and explicit budget; no silent replay.
|
|
The assignment key is public in Git, so future masking is procedural, not enforced.
|