coulomb/intelligence-radar
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Define reviewed publications and record the field-sampling frame.

Browse source 
IR-WP-0007-T03 can now record a capture-linked publication separately from generated drafts. The field-outcome design is written, and credential, billing, and website-pull gates were rechecked and left waiting.

Assistant: grok
Assistant-Session: 01a106a5-fc07-78c2-b64f-8e04cea25daf
This commit is contained in:
tegwick 2026-10-04 14:51:01 +02:00
parent a25cc867dd
commit 6808aa3594
11 changed files with 296 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

13
docs/data-contracts.md
View file

@ -109,6 +109,19 @@ retains it. New observation drafts list retained briefs for the same period.
Existing digest files are still not overwritten.
## Reviewed publications
```bash
python3 -m intelligence_radar publish /path/to/publication.json
```
A reviewed publication is a separate record from a daily, weekly, or external
draft. The command writes `digests/reviewed/<revision>.md` and refuses to treat
a generated digest as that record. Each included claim names a capture that is
already in the ledger. An excluded claim names the reason and may name a
capture. A correction sets `supersedes` to an earlier publication revision.
`reviewed_at` requires a timezone. No publication is issued by the scheduler.
## Portable evidence bundles
```bash

58
docs/evidence/2026-10-04-ir-wp-0007-t02-sampling-frame.md Normal file
View file

@ -0,0 +1,58 @@
# IR-WP-0007-T02 — field sampling frame and failure taxonomy
Written 2026-10-04. This is the design portion T02 allows before paid execution.
It does not collect outcomes, settle a campaign, or make the 20 pilot cases representative.
Paid execution still waits on IR-WP-0005-T02 and IR-WP-0006-T03. IR-WP-0007-T02
stays `wait`. IR-WP-0007-T04 stays `wait` until matched held-out outcomes exist.
## What the pilot already is
`examples/work-sample-suite.json` holds 20 cases: four close variants in each of
five evidence-processing families. The 2026-09-14 review keeps them as pilot
evidence for execution and accounting. They are not the field frame. Variants
that share a family are one dependence group. A later count is by group, not
by variant.
## Field-task frame
Freeze this frame before any prompt or verifier is tuned on field tasks.
| Rule | Meaning |
| --- | --- |
| Origin | A field task is an operator request, or one case copied from the pilot suite, named before the run. The origin is part of the outcome note. |
| Inclusion | The task has an acceptance check a person can apply without asking the model that produced the answer. Cost stays inside the existing USD 10 campaign. |
| Exclusion | The task was used to write the prompt or the verifier. A synthetic advisor fixture is not a field task. A provider catalog row is not a task. |
| Hold-out | At least one dependence group is named before the first field run and is not used to change prompts, verifiers, or profile choice. |
| Dependence | Tasks that share a source document or a verifier are one group. Report the group, then the runs inside it. |
No group list is frozen here. Freezing names before the first paid field run is
part of the execution that T02 still cannot start.
## Failure taxonomy
Record one primary class per run. A run can note a second class when both happened.
| Class | When it applies |
| --- | --- |
| `execution` | The provider returned an error, timed out, or produced no output. |
| `verification` | An output exists and the independent verifier rejects it. |
| `verifier_dependence` | The verifier can be satisfied without the task being done, including a score that the producing system can game, or a checker that repeats the producer's error. |
| `refusal` | The run stopped on a policy or safety refusal before an answer existed. |
| `budget` | The campaign cap or the case cap stopped the run. |
| `evidence_gap` | The run finished and `independent_verification` is false or the verifier artifact is missing. |
`verifier_dependence` is the class for reward hacking and for a verifier that
shares the generator's errors. An outcome with that class is not a success,
even if the harness score passed. Outcome records already require
`independent_verification` and `verifier_evidence`. This taxonomy names how a
later field log should use those fields. It does not change the outcome schema
and it does not authorize a run.
## What a later field report may say
Counts by dependence group. Execution failures separate from verification
failures. Actual USD and runtime for each run. The hold-out group named, and
whether any prompt changed after it was frozen. No cross-task success interval
and no cost or runtime quantile until a separate analysis says which sample
justifies them. The pilot's 20 cases stay labeled pilot.