intelligence-radar/docs/evidence/2026-10-04-ir-wp-0007-t02-sampling-frame.md

IR-WP-0007-T02 — field sampling frame and failure taxonomy

Written 2026-10-04. This is the design portion T02 allows before paid execution. It does not collect outcomes, settle a campaign, or make the 20 pilot cases representative.

Paid execution still waits on IR-WP-0005-T02 and IR-WP-0006-T03. IR-WP-0007-T02 stays wait. IR-WP-0007-T04 stays wait until matched held-out outcomes exist.

What the pilot already is

examples/work-sample-suite.json holds 20 cases: four close variants in each of five evidence-processing families. The 2026-09-14 review keeps them as pilot evidence for execution and accounting. They are not the field frame. Variants that share a family are one dependence group. A later count is by group, not by variant.

Field-task frame

Freeze this frame before any prompt or verifier is tuned on field tasks.

Rule	Meaning
Origin	A field task is an operator request, or one case copied from the pilot suite, named before the run. The origin is part of the outcome note.
Inclusion	The task has an acceptance check a person can apply without asking the model that produced the answer. Cost stays inside the existing USD 10 campaign.
Exclusion	The task was used to write the prompt or the verifier. A synthetic advisor fixture is not a field task. A provider catalog row is not a task.
Hold-out	At least one dependence group is named before the first field run and is not used to change prompts, verifiers, or profile choice.
Dependence	Tasks that share a source document or a verifier are one group. Report the group, then the runs inside it.

No group list is frozen here. Freezing names before the first paid field run is part of the execution that T02 still cannot start.

Failure taxonomy

Record one primary class per run. A run can note a second class when both happened.

Class	When it applies
execution	The provider returned an error, timed out, or produced no output.
verification	An output exists and the independent verifier rejects it.
verifier_dependence	The verifier can be satisfied without the task being done, including a score that the producing system can game, or a checker that repeats the producer's error.
refusal	The run stopped on a policy or safety refusal before an answer existed.
budget	The campaign cap or the case cap stopped the run.
evidence_gap	The run finished and independent_verification is false or the verifier artifact is missing.

verifier_dependence is the class for reward hacking and for a verifier that shares the generator's errors. An outcome with that class is not a success, even if the harness score passed. Outcome records already require independent_verification and verifier_evidence. This taxonomy names how a later field log should use those fields. It does not change the outcome schema and it does not authorize a run.

What a later field report may say

Counts by dependence group. Execution failures separate from verification failures. Actual USD and runtime for each run. The hold-out group named, and whether any prompt changed after it was frozen. No cross-task success interval and no cost or runtime quantile until a separate analysis says which sample justifies them. The pilot's 20 cases stay labeled pilot.