coulomb/intelligence-radar
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Define reviewed publications and record the field-sampling frame.

Browse source 
IR-WP-0007-T03 can now record a capture-linked publication separately from generated drafts. The field-outcome design is written, and credential, billing, and website-pull gates were rechecked and left waiting.

Assistant: grok
Assistant-Session: 01a106a5-fc07-78c2-b64f-8e04cea25daf
This commit is contained in:
tegwick 2026-10-04 14:51:01 +02:00
parent a25cc867dd
commit 6808aa3594
11 changed files with 296 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

13
docs/data-contracts.md
View file

@ -109,6 +109,19 @@ retains it. New observation drafts list retained briefs for the same period.
Existing digest files are still not overwritten.
## Reviewed publications
```bash
python3 -m intelligence_radar publish /path/to/publication.json
```
A reviewed publication is a separate record from a daily, weekly, or external
draft. The command writes `digests/reviewed/<revision>.md` and refuses to treat
a generated digest as that record. Each included claim names a capture that is
already in the ledger. An excluded claim names the reason and may name a
capture. A correction sets `supersedes` to an earlier publication revision.
`reviewed_at` requires a timezone. No publication is issued by the scheduler.
## Portable evidence bundles
```bash

58
docs/evidence/2026-10-04-ir-wp-0007-t02-sampling-frame.md Normal file
View file

@ -0,0 +1,58 @@
# IR-WP-0007-T02 — field sampling frame and failure taxonomy
Written 2026-10-04. This is the design portion T02 allows before paid execution.
It does not collect outcomes, settle a campaign, or make the 20 pilot cases representative.
Paid execution still waits on IR-WP-0005-T02 and IR-WP-0006-T03. IR-WP-0007-T02
stays `wait`. IR-WP-0007-T04 stays `wait` until matched held-out outcomes exist.
## What the pilot already is
`examples/work-sample-suite.json` holds 20 cases: four close variants in each of
five evidence-processing families. The 2026-09-14 review keeps them as pilot
evidence for execution and accounting. They are not the field frame. Variants
that share a family are one dependence group. A later count is by group, not
by variant.
## Field-task frame
Freeze this frame before any prompt or verifier is tuned on field tasks.
| Rule | Meaning |
| --- | --- |
| Origin | A field task is an operator request, or one case copied from the pilot suite, named before the run. The origin is part of the outcome note. |
| Inclusion | The task has an acceptance check a person can apply without asking the model that produced the answer. Cost stays inside the existing USD 10 campaign. |
| Exclusion | The task was used to write the prompt or the verifier. A synthetic advisor fixture is not a field task. A provider catalog row is not a task. |
| Hold-out | At least one dependence group is named before the first field run and is not used to change prompts, verifiers, or profile choice. |
| Dependence | Tasks that share a source document or a verifier are one group. Report the group, then the runs inside it. |
No group list is frozen here. Freezing names before the first paid field run is
part of the execution that T02 still cannot start.
## Failure taxonomy
Record one primary class per run. A run can note a second class when both happened.
| Class | When it applies |
| --- | --- |
| `execution` | The provider returned an error, timed out, or produced no output. |
| `verification` | An output exists and the independent verifier rejects it. |
| `verifier_dependence` | The verifier can be satisfied without the task being done, including a score that the producing system can game, or a checker that repeats the producer's error. |
| `refusal` | The run stopped on a policy or safety refusal before an answer existed. |
| `budget` | The campaign cap or the case cap stopped the run. |
| `evidence_gap` | The run finished and `independent_verification` is false or the verifier artifact is missing. |
`verifier_dependence` is the class for reward hacking and for a verifier that
shares the generator's errors. An outcome with that class is not a success,
even if the harness score passed. Outcome records already require
`independent_verification` and `verifier_evidence`. This taxonomy names how a
later field log should use those fields. It does not change the outcome schema
and it does not authorize a run.
## What a later field report may say
Counts by dependence group. Execution failures separate from verification
failures. Actual USD and runtime for each run. The hold-out group named, and
whether any prompt changed after it was frozen. No cross-task success interval
and no cost or runtime quantile until a separate analysis says which sample
justifies them. The pilot's 20 cases stay labeled pilot.

7
intelligence_radar/__main__.py
View file

@ -154,6 +154,8 @@ def check(root):
all_records(root, kind)
from .external_briefs import verify
verify(root)
from .publications import verify as verify_publications
verify_publications(root)
from .advisor import replay
for ident, _ in all_records(root, 'recommendations'):
replay(root, ident)
@ -189,6 +191,8 @@ def main(argv=None):
briefs_command = commands.add_parser('import-briefs', help='retain unarchived ChatGPT and Grok briefs from the local inbox')
briefs_command.add_argument('--config', type=Path, default=Path('ops/schedule.json'))
briefs_command.add_argument('--inbox', type=Path, help='override the configured inbox for this run')
publish_command = commands.add_parser('publish', help='record a reviewed publication linked to captures')
publish_command.add_argument('input', type=Path)
commands.add_parser('health', help='show durable scheduler health')
export_command = commands.add_parser('export', help='export a recommendation and source dependencies as a tar bundle')
export_command.add_argument('id')
@ -249,6 +253,9 @@ def main(argv=None):
print(json.dumps(result, indent=2))
external = result.get('external_briefs') or {}
return 0 if result['ok'] and external.get('ok', True) else 1
elif args.command == 'publish':
from .publications import publish
print(publish(args.root, json.loads(args.input.read_text())))
elif args.command == 'import-briefs':
from .external_briefs import import_inbox, native_status, resolve_inbox
spec = json.loads(args.config.read_text()).get('external_briefs', {})

113
intelligence_radar/publications.py Normal file
View file

@ -0,0 +1,113 @@
"""Reviewed publications. A generated digest is not one of these records."""
from datetime import datetime
from pathlib import Path
from .store import all_records, get, put
FIELDS = {'schema_version', 'title', 'reviewer', 'reviewed_at', 'claims', 'supersedes'}
CLAIM_FIELDS = {'statement', 'disposition', 'capture_id', 'exclusion_reason'}
def publish(root, value):
root = Path(root)
record = _record(root, value)
ident = put(root, 'publications', record)
_render(root, ident, record)
return ident
def verify(root):
root = Path(root)
directory = root / 'data' / 'publications'
if not directory.exists():
return 0
reviewed = (root / 'digests' / 'reviewed').resolve()
count = 0
for ident, record in all_records(root, 'publications'):
_record(root, record, existing=ident)
path = (reviewed / f'{ident}.md').resolve()
if path.parent != reviewed or not path.is_file():
raise ValueError(f'reviewed publication file missing: {ident}')
text = path.read_text(encoding='utf-8')
if not text.startswith('# Reviewed publication\n') or ident not in text:
raise ValueError(f'reviewed publication rendering mismatch: {ident}')
count += 1
return count
def _record(root, value, existing=None):
if not isinstance(value, dict) or set(value) - FIELDS or FIELDS - {'supersedes'} - set(value):
raise ValueError('publication has missing or unknown fields')
if type(value['schema_version']) is not int or value['schema_version'] != 1:
raise ValueError('schema_version must be 1')
for key in ('title', 'reviewer'):
if not isinstance(value[key], str) or not value[key].strip():
raise ValueError(f'{key} must be nonempty text')
reviewed = datetime.fromisoformat(value['reviewed_at'].replace('Z', '+00:00'))
if reviewed.tzinfo is None:
raise ValueError('reviewed_at requires a timezone')
if not isinstance(value['claims'], list) or not value['claims']:
raise ValueError('publication requires at least one claim')
record = {'schema_version': 1, 'title': value['title'].strip(),
'reviewer': value['reviewer'].strip(), 'reviewed_at': value['reviewed_at'],
'claims': [_claim(root, claim) for claim in value['claims']]}
if 'supersedes' in value:
parent = value['supersedes']
if not isinstance(parent, str) or parent == existing:
raise ValueError('supersedes must name an earlier publication')
_chain(root, parent, set())
record['supersedes'] = parent
return record
def _claim(root, claim):
if not isinstance(claim, dict) or set(claim) - CLAIM_FIELDS or {'statement', 'disposition'} - set(claim):
raise ValueError('invalid publication claim')
if not isinstance(claim['statement'], str) or not claim['statement'].strip():
raise ValueError('claim statement must be nonempty text')
if claim['disposition'] not in {'included', 'excluded'}:
raise ValueError('claim disposition must be included or excluded')
item = {'statement': claim['statement'].strip(), 'disposition': claim['disposition']}
if 'capture_id' in claim:
get(root, 'captures', claim['capture_id'])
item['capture_id'] = claim['capture_id']
if claim['disposition'] == 'included':
if 'capture_id' not in item or 'exclusion_reason' in claim:
raise ValueError('an included claim requires a capture and no exclusion reason')
else:
reason = claim.get('exclusion_reason')
if not isinstance(reason, str) or not reason.strip():
raise ValueError('an excluded claim requires an exclusion reason')
item['exclusion_reason'] = reason.strip()
return item
def _chain(root, ident, seen):
if ident in seen:
raise ValueError('publication correction cycle')
seen.add(ident)
try:
record = get(root, 'publications', ident)
except OSError as exc:
raise ValueError('supersedes must name an earlier publication') from exc
if 'supersedes' in record:
_chain(root, record['supersedes'], seen)
def _render(root, ident, record):
lines = ['# Reviewed publication', '',
'This file is a reviewed publication. It is not a generated daily, weekly, or external draft.',
'', f'- Revision: `{ident}`', f"- Title: {record['title']}",
f"- Reviewer: {record['reviewer']}", f"- Reviewed: {record['reviewed_at']}"]
if 'supersedes' in record:
lines.append(f"- Supersedes: `{record['supersedes']}`")
lines += ['', '## Claims', '']
for claim in record['claims']:
lines.append(f"- {claim['disposition']}: {claim['statement']}")
if 'capture_id' in claim:
lines.append(f" - Capture: `{claim['capture_id']}`")
if 'exclusion_reason' in claim:
lines.append(f" - Excluded because: {claim['exclusion_reason']}")
path = Path(root) / 'digests' / 'reviewed' / f'{ident}.md'
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text('\n'.join(lines) + '\n', encoding='utf-8')

4
nudges/2026-10-04-reward-hacking-in-taxonomy.md
View file

@ -6,4 +6,6 @@ Derived from [compendium/04-socher-eureka-machine.md](../compendium/04-socher-eu
The operator's Gemini notes call reward hacking the hard problem in autonomous science: an objective harness that measures discovery rather than a model exploit. Pant's review makes the same demand in laboratory language. The checker has to be able to disagree with the system that produced the result.
Outcome records already require `independent_verification`. [IR-WP-0007-T02](../workplans/IR-WP-0007-evidence-and-publication.md) already owes a failure taxonomy before any paid sample is called representative. When that taxonomy is written, reward hacking and a verifier that shares the generator's errors are worth a place in it. This note does not edit that workplan.
Outcome records already require `independent_verification`. [IR-WP-0007-T02](../workplans/IR-WP-0007-evidence-and-publication.md) already owes a failure taxonomy before any paid sample is called representative. When that taxonomy is written, reward hacking and a verifier that shares the generator's errors are worth a place in it. 2026-10-04: the class is named `verifier_dependence` in
`docs/evidence/2026-10-04-ir-wp-0007-t02-sampling-frame.md`. IR-WP-0007-T02
stays wait. This note is still not a task.

2
ops/RUNBOOK.md
View file

@ -36,6 +36,8 @@ Run commands at the repository root with Python 3.11+. No installation is needed
are retained by `python3 -m intelligence_radar import-briefs` and by
`run-due`. Read them from `digests/external/`. They are not reviewed digests.
The inbox `archive/` directory is the already-captured set and is not read.
A reviewed publication is `python3 -m intelligence_radar publish`. It is
not produced by `digest` or `run-due`.
6. For recommendations, complete task-router/TEMPLATE.md under the governing method.
Preserve unknowns. Real task outcomes later support calibration.
7. Run `make check`, review changes, commit the evidence and artifacts. Update

59
tests/test_publications.py Normal file
View file

@ -0,0 +1,59 @@
from pathlib import Path
import tempfile
import unittest
from intelligence_radar.publications import publish, verify
from intelligence_radar.store import put
class PublicationTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
self.capture = put(self.root, 'captures', {'source': 'example', 'content_sha256': 'abc'})
def publication(self, **extra):
value = {'schema_version': 1, 'title': 'Two profiles, no winner',
'reviewer': 'ada', 'reviewed_at': '2026-10-04T12:00:00+00:00',
'claims': [{'statement': 'The capture lists a price.', 'disposition': 'included',
'capture_id': self.capture},
{'statement': 'No access evidence.', 'disposition': 'excluded',
'exclusion_reason': 'Requester access was not observed.'}]}
value.update(extra)
return value
def test_publication_links_captures_and_is_not_a_draft(self):
ident = publish(self.root, self.publication())
self.assertEqual(publish(self.root, self.publication()), ident)
text = (self.root / 'digests' / 'reviewed' / f'{ident}.md').read_text(encoding='utf-8')
self.assertTrue(text.startswith('# Reviewed publication\n'))
self.assertIn(self.capture, text)
self.assertIn('Requester access was not observed.', text)
self.assertFalse((self.root / 'digests' / 'daily').exists())
self.assertEqual(verify(self.root), 1)
def test_included_claim_without_capture_is_refused(self):
value = self.publication()
value['claims'] = [{'statement': 'Unsourced.', 'disposition': 'included'}]
with self.assertRaisesRegex(ValueError, 'capture'):
publish(self.root, value)
def test_correction_links_an_earlier_revision(self):
original = publish(self.root, self.publication())
correction = publish(self.root, self.publication(
title='Corrected', supersedes=original,
claims=[{'statement': 'The price claim was withdrawn.', 'disposition': 'excluded',
'exclusion_reason': 'The capture did not show that price.'}]))
self.assertIn(original, (self.root / 'digests' / 'reviewed' / f'{correction}.md').read_text())
self.assertEqual(verify(self.root), 2)
with self.assertRaisesRegex(ValueError, 'earlier publication'):
publish(self.root, self.publication(supersedes='0' * 64))
def test_review_time_requires_a_timezone(self):
with self.assertRaisesRegex(ValueError, 'timezone'):
publish(self.root, self.publication(reviewed_at='2026-10-04T12:00:00'))
if __name__ == '__main__':
unittest.main()

7
workplans/IR-WP-0005-credential-delivery.md
View file

@ -11,7 +11,7 @@ topic_slug: custodian
origin: residual
origin_ref: IR-WP-0004
created: "2026-09-14"
updated: "2026-10-03"
updated: "2026-10-04"
state_hub_workstream_id: "624750a5-d613-5e32-a656-595de03ef73c"
---
@ -60,3 +60,8 @@ pinned `intelligence-radar-key-check` recipient; Radar requested secrets-engine
`docs/evidence/2026-09-27-ir-wp-0005-t01-native-delivery.md`.
2026-10-03: human gate routed to hub decision 62aa95e4-3ccc-4867-8806-5e3db005b483 (D1: Attended OpenBao provisioning window: one session clears the pending credential lanes) by CUST-WP-0075; resolving that decision raises C-40 here.
Rechecked 2026-10-04: decision `62aa95e4-3ccc-4867-8806-5e3db005b483` is still
`open` (`decided_at` null, last hub update 2026-10-03). The radar inbox has no
new secrets-engine message. No delivery check was run and no credential was
read. Stays wait.

7
workplans/IR-WP-0006-billing-reconciliation.md
View file

@ -11,7 +11,7 @@ topic_slug: custodian
origin: residual
origin_ref: IR-WP-0004
created: "2026-09-14"
updated: "2026-09-14"
updated: "2026-10-04"
state_hub_workstream_id: "b877f5e2-0c0e-5bca-9082-873eab857ada"
---
@ -70,3 +70,8 @@ Blocked: the mechanism is ready (T01/T02 done), but no authoritative provider bi
exists for the pending run_id `29a46a1d6c2d9ef6cfce8de6efef75cddcfecee346c1b4babf57119f1a241d71`
yet. Applying `reconcile` here without that evidence would be exactly the fabrication this
workplan forbids. Waits on external evidence (likely via the IR-WP-0005 admitted read-only path).
Rechecked 2026-10-04: the ledger still has the one `reserved` event for
`29a46a1d6c2d9ef6cfce8de6efef75cddcfecee346c1b4babf57119f1a241d71` and no
provider billing artifact. IR-WP-0005-T02 is still wait, so the read-only path
is still closed. `reconcile` was not called. Stays wait.

28
workplans/IR-WP-0007-evidence-and-publication.md
View file

@ -11,7 +11,7 @@ topic_slug: custodian
origin: residual
origin_ref: IR-WP-0004
created: "2026-09-14"
updated: "2026-09-14"
updated: "2026-10-04"
state_hub_workstream_id: "e75cf5ae-3df2-507e-8fa3-8de687137623"
---
@ -39,6 +39,12 @@ review (`docs/evidence/2026-09-14-ir-wp-0004-review.md`) and found to require a
dedicated, sourced measurement campaign rather than a redo inside this session —
inventing pinned versions without primary-source verification would violate P11.
Rechecked 2026-10-04: the instrument mismatch in
`docs/evidence/2026-09-14-ir-wp-0004-review.md` is unchanged, and requester
access is still absent because IR-WP-0005-T02 is waiting on open decision
`62aa95e4-3ccc-4867-8806-5e3db005b483`. No benchmark versions were pinned and
no Frontier/Efficient comparison was produced. Stays todo.
## Collect and assess representative field outcomes
```task
@ -55,14 +61,20 @@ Residual IR-WP-0004-T02. Paid execution depends on BOTH IR-WP-0005 credential re
Blocked, 2026-09-27: paid execution still needs IR-WP-0005-T02 (wait) and IR-WP-0006-T03
(wait) both closed. The sampling-frame/verifier design portion this task allows to proceed
earlier is nontrivial methodology work in its own right (real task origins, held-out sets,
verifiers, failure taxonomy) and was not attempted in this session to avoid a rushed
design under time pressure; it remains open, undecomposed, for a dedicated pass.
verifiers, failure taxonomy) and was not attempted in the 2026-09-27 pass.
Design portion recorded 2026-10-04 in
`docs/evidence/2026-10-04-ir-wp-0007-t02-sampling-frame.md`: origin,
inclusion, exclusion, hold-out, dependence groups, and a failure taxonomy
that includes verifier dependence (reward hacking and a checker that shares
the generator's errors). No field tasks were frozen and no runs were made.
Paid execution still waits. Status stays wait.
## Complete reviewed publication and operational disposition
```task
id: IR-WP-0007-T03
status: todo
status: done
priority: high
state_hub_task_id: "314f79fd-d95a-5e02-b67d-67e35e4334ed"
```
@ -75,6 +87,14 @@ definition (claim-to-capture references, exclusions, revision identity, correcti
distinct from a generated draft) is still open; it is a schema/format design decision
better made deliberately than folded into this pass. Stays todo.
Done 2026-10-04: `python3 -m intelligence_radar publish` records a reviewed
publication in `data/publications/` and renders `digests/reviewed/<revision>.md`.
An included claim must name a capture already in the ledger. An excluded claim
must name the reason. `supersedes` links a correction to an earlier revision.
The scheduler does not write these files, and a generated draft is not accepted
as the record. No publication was issued; nothing had been reviewed. The local
timer decision remains ADR 0003.
## Compare decomposition with measured monolithic execution
```task

5
workplans/IR-WP-0008-external-brief-intake.md
View file

@ -71,3 +71,8 @@ path remains the brief inbox: an export saved there, including one saved from
Grok's email delivery, is retained by T02. Pulling the websites themselves
waits on a real export or an authorized account feed that does not guess
private endpoints.
Rechecked 2026-10-04: the inbox has no new export. It still holds the empty
3 October Grok file and the 27 September ChatGPT file whose archive name is
already taken by different bytes. email-connect has no Intelligence Radar
feed. No session was sent to chatgpt.com or grok.com. Stays wait.