tegwick
6808aa3594
IR-WP-0007-T03 can now record a capture-linked publication separately from generated drafts. The field-outcome design is written, and credential, billing, and website-pull gates were rechecked and left waiting. Assistant: grok Assistant-Session: 01a106a5-fc07-78c2-b64f-8e04cea25daf
58 lines
3.3 KiB
Markdown
58 lines
3.3 KiB
Markdown
# IR-WP-0007-T02 — field sampling frame and failure taxonomy
|
|
|
|
Written 2026-10-04. This is the design portion T02 allows before paid execution.
|
|
It does not collect outcomes, settle a campaign, or make the 20 pilot cases representative.
|
|
|
|
Paid execution still waits on IR-WP-0005-T02 and IR-WP-0006-T03. IR-WP-0007-T02
|
|
stays `wait`. IR-WP-0007-T04 stays `wait` until matched held-out outcomes exist.
|
|
|
|
## What the pilot already is
|
|
|
|
`examples/work-sample-suite.json` holds 20 cases: four close variants in each of
|
|
five evidence-processing families. The 2026-09-14 review keeps them as pilot
|
|
evidence for execution and accounting. They are not the field frame. Variants
|
|
that share a family are one dependence group. A later count is by group, not
|
|
by variant.
|
|
|
|
## Field-task frame
|
|
|
|
Freeze this frame before any prompt or verifier is tuned on field tasks.
|
|
|
|
| Rule | Meaning |
|
|
| --- | --- |
|
|
| Origin | A field task is an operator request, or one case copied from the pilot suite, named before the run. The origin is part of the outcome note. |
|
|
| Inclusion | The task has an acceptance check a person can apply without asking the model that produced the answer. Cost stays inside the existing USD 10 campaign. |
|
|
| Exclusion | The task was used to write the prompt or the verifier. A synthetic advisor fixture is not a field task. A provider catalog row is not a task. |
|
|
| Hold-out | At least one dependence group is named before the first field run and is not used to change prompts, verifiers, or profile choice. |
|
|
| Dependence | Tasks that share a source document or a verifier are one group. Report the group, then the runs inside it. |
|
|
|
|
No group list is frozen here. Freezing names before the first paid field run is
|
|
part of the execution that T02 still cannot start.
|
|
|
|
## Failure taxonomy
|
|
|
|
Record one primary class per run. A run can note a second class when both happened.
|
|
|
|
| Class | When it applies |
|
|
| --- | --- |
|
|
| `execution` | The provider returned an error, timed out, or produced no output. |
|
|
| `verification` | An output exists and the independent verifier rejects it. |
|
|
| `verifier_dependence` | The verifier can be satisfied without the task being done, including a score that the producing system can game, or a checker that repeats the producer's error. |
|
|
| `refusal` | The run stopped on a policy or safety refusal before an answer existed. |
|
|
| `budget` | The campaign cap or the case cap stopped the run. |
|
|
| `evidence_gap` | The run finished and `independent_verification` is false or the verifier artifact is missing. |
|
|
|
|
`verifier_dependence` is the class for reward hacking and for a verifier that
|
|
shares the generator's errors. An outcome with that class is not a success,
|
|
even if the harness score passed. Outcome records already require
|
|
`independent_verification` and `verifier_evidence`. This taxonomy names how a
|
|
later field log should use those fields. It does not change the outcome schema
|
|
and it does not authorize a run.
|
|
|
|
## What a later field report may say
|
|
|
|
Counts by dependence group. Execution failures separate from verification
|
|
failures. Actual USD and runtime for each run. The hold-out group named, and
|
|
whether any prompt changed after it was frozen. No cross-task success interval
|
|
and no cost or runtime quantile until a separate analysis says which sample
|
|
justifies them. The pilot's 20 cases stay labeled pilot.
|