diff --git a/docs/data-contracts.md b/docs/data-contracts.md index 0246ce7..93afc1c 100644 --- a/docs/data-contracts.md +++ b/docs/data-contracts.md @@ -109,6 +109,19 @@ retains it. New observation drafts list retained briefs for the same period. Existing digest files are still not overwritten. +## Reviewed publications + +```bash +python3 -m intelligence_radar publish /path/to/publication.json +``` + +A reviewed publication is a separate record from a daily, weekly, or external +draft. The command writes `digests/reviewed/.md` and refuses to treat +a generated digest as that record. Each included claim names a capture that is +already in the ledger. An excluded claim names the reason and may name a +capture. A correction sets `supersedes` to an earlier publication revision. +`reviewed_at` requires a timezone. No publication is issued by the scheduler. + ## Portable evidence bundles ```bash diff --git a/docs/evidence/2026-10-04-ir-wp-0007-t02-sampling-frame.md b/docs/evidence/2026-10-04-ir-wp-0007-t02-sampling-frame.md new file mode 100644 index 0000000..54039b6 --- /dev/null +++ b/docs/evidence/2026-10-04-ir-wp-0007-t02-sampling-frame.md @@ -0,0 +1,58 @@ +# IR-WP-0007-T02 — field sampling frame and failure taxonomy + +Written 2026-10-04. This is the design portion T02 allows before paid execution. +It does not collect outcomes, settle a campaign, or make the 20 pilot cases representative. + +Paid execution still waits on IR-WP-0005-T02 and IR-WP-0006-T03. IR-WP-0007-T02 +stays `wait`. IR-WP-0007-T04 stays `wait` until matched held-out outcomes exist. + +## What the pilot already is + +`examples/work-sample-suite.json` holds 20 cases: four close variants in each of +five evidence-processing families. The 2026-09-14 review keeps them as pilot +evidence for execution and accounting. They are not the field frame. Variants +that share a family are one dependence group. A later count is by group, not +by variant. + +## Field-task frame + +Freeze this frame before any prompt or verifier is tuned on field tasks. + +| Rule | Meaning | +| --- | --- | +| Origin | A field task is an operator request, or one case copied from the pilot suite, named before the run. The origin is part of the outcome note. | +| Inclusion | The task has an acceptance check a person can apply without asking the model that produced the answer. Cost stays inside the existing USD 10 campaign. | +| Exclusion | The task was used to write the prompt or the verifier. A synthetic advisor fixture is not a field task. A provider catalog row is not a task. | +| Hold-out | At least one dependence group is named before the first field run and is not used to change prompts, verifiers, or profile choice. | +| Dependence | Tasks that share a source document or a verifier are one group. Report the group, then the runs inside it. | + +No group list is frozen here. Freezing names before the first paid field run is +part of the execution that T02 still cannot start. + +## Failure taxonomy + +Record one primary class per run. A run can note a second class when both happened. + +| Class | When it applies | +| --- | --- | +| `execution` | The provider returned an error, timed out, or produced no output. | +| `verification` | An output exists and the independent verifier rejects it. | +| `verifier_dependence` | The verifier can be satisfied without the task being done, including a score that the producing system can game, or a checker that repeats the producer's error. | +| `refusal` | The run stopped on a policy or safety refusal before an answer existed. | +| `budget` | The campaign cap or the case cap stopped the run. | +| `evidence_gap` | The run finished and `independent_verification` is false or the verifier artifact is missing. | + +`verifier_dependence` is the class for reward hacking and for a verifier that +shares the generator's errors. An outcome with that class is not a success, +even if the harness score passed. Outcome records already require +`independent_verification` and `verifier_evidence`. This taxonomy names how a +later field log should use those fields. It does not change the outcome schema +and it does not authorize a run. + +## What a later field report may say + +Counts by dependence group. Execution failures separate from verification +failures. Actual USD and runtime for each run. The hold-out group named, and +whether any prompt changed after it was frozen. No cross-task success interval +and no cost or runtime quantile until a separate analysis says which sample +justifies them. The pilot's 20 cases stay labeled pilot. diff --git a/intelligence_radar/__main__.py b/intelligence_radar/__main__.py index a802dce..e6dc17d 100644 --- a/intelligence_radar/__main__.py +++ b/intelligence_radar/__main__.py @@ -154,6 +154,8 @@ def check(root): all_records(root, kind) from .external_briefs import verify verify(root) + from .publications import verify as verify_publications + verify_publications(root) from .advisor import replay for ident, _ in all_records(root, 'recommendations'): replay(root, ident) @@ -189,6 +191,8 @@ def main(argv=None): briefs_command = commands.add_parser('import-briefs', help='retain unarchived ChatGPT and Grok briefs from the local inbox') briefs_command.add_argument('--config', type=Path, default=Path('ops/schedule.json')) briefs_command.add_argument('--inbox', type=Path, help='override the configured inbox for this run') + publish_command = commands.add_parser('publish', help='record a reviewed publication linked to captures') + publish_command.add_argument('input', type=Path) commands.add_parser('health', help='show durable scheduler health') export_command = commands.add_parser('export', help='export a recommendation and source dependencies as a tar bundle') export_command.add_argument('id') @@ -249,6 +253,9 @@ def main(argv=None): print(json.dumps(result, indent=2)) external = result.get('external_briefs') or {} return 0 if result['ok'] and external.get('ok', True) else 1 + elif args.command == 'publish': + from .publications import publish + print(publish(args.root, json.loads(args.input.read_text()))) elif args.command == 'import-briefs': from .external_briefs import import_inbox, native_status, resolve_inbox spec = json.loads(args.config.read_text()).get('external_briefs', {}) diff --git a/intelligence_radar/publications.py b/intelligence_radar/publications.py new file mode 100644 index 0000000..8723dff --- /dev/null +++ b/intelligence_radar/publications.py @@ -0,0 +1,113 @@ +"""Reviewed publications. A generated digest is not one of these records.""" +from datetime import datetime +from pathlib import Path + +from .store import all_records, get, put + +FIELDS = {'schema_version', 'title', 'reviewer', 'reviewed_at', 'claims', 'supersedes'} +CLAIM_FIELDS = {'statement', 'disposition', 'capture_id', 'exclusion_reason'} + + +def publish(root, value): + root = Path(root) + record = _record(root, value) + ident = put(root, 'publications', record) + _render(root, ident, record) + return ident + + +def verify(root): + root = Path(root) + directory = root / 'data' / 'publications' + if not directory.exists(): + return 0 + reviewed = (root / 'digests' / 'reviewed').resolve() + count = 0 + for ident, record in all_records(root, 'publications'): + _record(root, record, existing=ident) + path = (reviewed / f'{ident}.md').resolve() + if path.parent != reviewed or not path.is_file(): + raise ValueError(f'reviewed publication file missing: {ident}') + text = path.read_text(encoding='utf-8') + if not text.startswith('# Reviewed publication\n') or ident not in text: + raise ValueError(f'reviewed publication rendering mismatch: {ident}') + count += 1 + return count + + +def _record(root, value, existing=None): + if not isinstance(value, dict) or set(value) - FIELDS or FIELDS - {'supersedes'} - set(value): + raise ValueError('publication has missing or unknown fields') + if type(value['schema_version']) is not int or value['schema_version'] != 1: + raise ValueError('schema_version must be 1') + for key in ('title', 'reviewer'): + if not isinstance(value[key], str) or not value[key].strip(): + raise ValueError(f'{key} must be nonempty text') + reviewed = datetime.fromisoformat(value['reviewed_at'].replace('Z', '+00:00')) + if reviewed.tzinfo is None: + raise ValueError('reviewed_at requires a timezone') + if not isinstance(value['claims'], list) or not value['claims']: + raise ValueError('publication requires at least one claim') + record = {'schema_version': 1, 'title': value['title'].strip(), + 'reviewer': value['reviewer'].strip(), 'reviewed_at': value['reviewed_at'], + 'claims': [_claim(root, claim) for claim in value['claims']]} + if 'supersedes' in value: + parent = value['supersedes'] + if not isinstance(parent, str) or parent == existing: + raise ValueError('supersedes must name an earlier publication') + _chain(root, parent, set()) + record['supersedes'] = parent + return record + + +def _claim(root, claim): + if not isinstance(claim, dict) or set(claim) - CLAIM_FIELDS or {'statement', 'disposition'} - set(claim): + raise ValueError('invalid publication claim') + if not isinstance(claim['statement'], str) or not claim['statement'].strip(): + raise ValueError('claim statement must be nonempty text') + if claim['disposition'] not in {'included', 'excluded'}: + raise ValueError('claim disposition must be included or excluded') + item = {'statement': claim['statement'].strip(), 'disposition': claim['disposition']} + if 'capture_id' in claim: + get(root, 'captures', claim['capture_id']) + item['capture_id'] = claim['capture_id'] + if claim['disposition'] == 'included': + if 'capture_id' not in item or 'exclusion_reason' in claim: + raise ValueError('an included claim requires a capture and no exclusion reason') + else: + reason = claim.get('exclusion_reason') + if not isinstance(reason, str) or not reason.strip(): + raise ValueError('an excluded claim requires an exclusion reason') + item['exclusion_reason'] = reason.strip() + return item + + +def _chain(root, ident, seen): + if ident in seen: + raise ValueError('publication correction cycle') + seen.add(ident) + try: + record = get(root, 'publications', ident) + except OSError as exc: + raise ValueError('supersedes must name an earlier publication') from exc + if 'supersedes' in record: + _chain(root, record['supersedes'], seen) + + +def _render(root, ident, record): + lines = ['# Reviewed publication', '', + 'This file is a reviewed publication. It is not a generated daily, weekly, or external draft.', + '', f'- Revision: `{ident}`', f"- Title: {record['title']}", + f"- Reviewer: {record['reviewer']}", f"- Reviewed: {record['reviewed_at']}"] + if 'supersedes' in record: + lines.append(f"- Supersedes: `{record['supersedes']}`") + lines += ['', '## Claims', ''] + for claim in record['claims']: + lines.append(f"- {claim['disposition']}: {claim['statement']}") + if 'capture_id' in claim: + lines.append(f" - Capture: `{claim['capture_id']}`") + if 'exclusion_reason' in claim: + lines.append(f" - Excluded because: {claim['exclusion_reason']}") + path = Path(root) / 'digests' / 'reviewed' / f'{ident}.md' + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text('\n'.join(lines) + '\n', encoding='utf-8') diff --git a/nudges/2026-10-04-reward-hacking-in-taxonomy.md b/nudges/2026-10-04-reward-hacking-in-taxonomy.md index faa194b..6364124 100644 --- a/nudges/2026-10-04-reward-hacking-in-taxonomy.md +++ b/nudges/2026-10-04-reward-hacking-in-taxonomy.md @@ -6,4 +6,6 @@ Derived from [compendium/04-socher-eureka-machine.md](../compendium/04-socher-eu The operator's Gemini notes call reward hacking the hard problem in autonomous science: an objective harness that measures discovery rather than a model exploit. Pant's review makes the same demand in laboratory language. The checker has to be able to disagree with the system that produced the result. -Outcome records already require `independent_verification`. [IR-WP-0007-T02](../workplans/IR-WP-0007-evidence-and-publication.md) already owes a failure taxonomy before any paid sample is called representative. When that taxonomy is written, reward hacking and a verifier that shares the generator's errors are worth a place in it. This note does not edit that workplan. +Outcome records already require `independent_verification`. [IR-WP-0007-T02](../workplans/IR-WP-0007-evidence-and-publication.md) already owes a failure taxonomy before any paid sample is called representative. When that taxonomy is written, reward hacking and a verifier that shares the generator's errors are worth a place in it. 2026-10-04: the class is named `verifier_dependence` in +`docs/evidence/2026-10-04-ir-wp-0007-t02-sampling-frame.md`. IR-WP-0007-T02 +stays wait. This note is still not a task. diff --git a/ops/RUNBOOK.md b/ops/RUNBOOK.md index 58f6b25..056a41f 100644 --- a/ops/RUNBOOK.md +++ b/ops/RUNBOOK.md @@ -36,6 +36,8 @@ Run commands at the repository root with Python 3.11+. No installation is needed are retained by `python3 -m intelligence_radar import-briefs` and by `run-due`. Read them from `digests/external/`. They are not reviewed digests. The inbox `archive/` directory is the already-captured set and is not read. + A reviewed publication is `python3 -m intelligence_radar publish`. It is + not produced by `digest` or `run-due`. 6. For recommendations, complete task-router/TEMPLATE.md under the governing method. Preserve unknowns. Real task outcomes later support calibration. 7. Run `make check`, review changes, commit the evidence and artifacts. Update diff --git a/tests/test_publications.py b/tests/test_publications.py new file mode 100644 index 0000000..e96f22b --- /dev/null +++ b/tests/test_publications.py @@ -0,0 +1,59 @@ +from pathlib import Path +import tempfile +import unittest + +from intelligence_radar.publications import publish, verify +from intelligence_radar.store import put + + +class PublicationTests(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory() + self.addCleanup(self.temp.cleanup) + self.root = Path(self.temp.name) + self.capture = put(self.root, 'captures', {'source': 'example', 'content_sha256': 'abc'}) + + def publication(self, **extra): + value = {'schema_version': 1, 'title': 'Two profiles, no winner', + 'reviewer': 'ada', 'reviewed_at': '2026-10-04T12:00:00+00:00', + 'claims': [{'statement': 'The capture lists a price.', 'disposition': 'included', + 'capture_id': self.capture}, + {'statement': 'No access evidence.', 'disposition': 'excluded', + 'exclusion_reason': 'Requester access was not observed.'}]} + value.update(extra) + return value + + def test_publication_links_captures_and_is_not_a_draft(self): + ident = publish(self.root, self.publication()) + self.assertEqual(publish(self.root, self.publication()), ident) + text = (self.root / 'digests' / 'reviewed' / f'{ident}.md').read_text(encoding='utf-8') + self.assertTrue(text.startswith('# Reviewed publication\n')) + self.assertIn(self.capture, text) + self.assertIn('Requester access was not observed.', text) + self.assertFalse((self.root / 'digests' / 'daily').exists()) + self.assertEqual(verify(self.root), 1) + + def test_included_claim_without_capture_is_refused(self): + value = self.publication() + value['claims'] = [{'statement': 'Unsourced.', 'disposition': 'included'}] + with self.assertRaisesRegex(ValueError, 'capture'): + publish(self.root, value) + + def test_correction_links_an_earlier_revision(self): + original = publish(self.root, self.publication()) + correction = publish(self.root, self.publication( + title='Corrected', supersedes=original, + claims=[{'statement': 'The price claim was withdrawn.', 'disposition': 'excluded', + 'exclusion_reason': 'The capture did not show that price.'}])) + self.assertIn(original, (self.root / 'digests' / 'reviewed' / f'{correction}.md').read_text()) + self.assertEqual(verify(self.root), 2) + with self.assertRaisesRegex(ValueError, 'earlier publication'): + publish(self.root, self.publication(supersedes='0' * 64)) + + def test_review_time_requires_a_timezone(self): + with self.assertRaisesRegex(ValueError, 'timezone'): + publish(self.root, self.publication(reviewed_at='2026-10-04T12:00:00')) + + +if __name__ == '__main__': + unittest.main() diff --git a/workplans/IR-WP-0005-credential-delivery.md b/workplans/IR-WP-0005-credential-delivery.md index 9f7f5fb..a000425 100644 --- a/workplans/IR-WP-0005-credential-delivery.md +++ b/workplans/IR-WP-0005-credential-delivery.md @@ -11,7 +11,7 @@ topic_slug: custodian origin: residual origin_ref: IR-WP-0004 created: "2026-09-14" -updated: "2026-10-03" +updated: "2026-10-04" state_hub_workstream_id: "624750a5-d613-5e32-a656-595de03ef73c" --- @@ -60,3 +60,8 @@ pinned `intelligence-radar-key-check` recipient; Radar requested secrets-engine `docs/evidence/2026-09-27-ir-wp-0005-t01-native-delivery.md`. 2026-10-03: human gate routed to hub decision 62aa95e4-3ccc-4867-8806-5e3db005b483 (D1: Attended OpenBao provisioning window: one session clears the pending credential lanes) by CUST-WP-0075; resolving that decision raises C-40 here. + +Rechecked 2026-10-04: decision `62aa95e4-3ccc-4867-8806-5e3db005b483` is still +`open` (`decided_at` null, last hub update 2026-10-03). The radar inbox has no +new secrets-engine message. No delivery check was run and no credential was +read. Stays wait. diff --git a/workplans/IR-WP-0006-billing-reconciliation.md b/workplans/IR-WP-0006-billing-reconciliation.md index f76a0bc..a4016b3 100644 --- a/workplans/IR-WP-0006-billing-reconciliation.md +++ b/workplans/IR-WP-0006-billing-reconciliation.md @@ -11,7 +11,7 @@ topic_slug: custodian origin: residual origin_ref: IR-WP-0004 created: "2026-09-14" -updated: "2026-09-14" +updated: "2026-10-04" state_hub_workstream_id: "b877f5e2-0c0e-5bca-9082-873eab857ada" --- @@ -70,3 +70,8 @@ Blocked: the mechanism is ready (T01/T02 done), but no authoritative provider bi exists for the pending run_id `29a46a1d6c2d9ef6cfce8de6efef75cddcfecee346c1b4babf57119f1a241d71` yet. Applying `reconcile` here without that evidence would be exactly the fabrication this workplan forbids. Waits on external evidence (likely via the IR-WP-0005 admitted read-only path). + +Rechecked 2026-10-04: the ledger still has the one `reserved` event for +`29a46a1d6c2d9ef6cfce8de6efef75cddcfecee346c1b4babf57119f1a241d71` and no +provider billing artifact. IR-WP-0005-T02 is still wait, so the read-only path +is still closed. `reconcile` was not called. Stays wait. diff --git a/workplans/IR-WP-0007-evidence-and-publication.md b/workplans/IR-WP-0007-evidence-and-publication.md index 9d854c7..319d078 100644 --- a/workplans/IR-WP-0007-evidence-and-publication.md +++ b/workplans/IR-WP-0007-evidence-and-publication.md @@ -11,7 +11,7 @@ topic_slug: custodian origin: residual origin_ref: IR-WP-0004 created: "2026-09-14" -updated: "2026-09-14" +updated: "2026-10-04" state_hub_workstream_id: "e75cf5ae-3df2-507e-8fa3-8de687137623" --- @@ -39,6 +39,12 @@ review (`docs/evidence/2026-09-14-ir-wp-0004-review.md`) and found to require a dedicated, sourced measurement campaign rather than a redo inside this session — inventing pinned versions without primary-source verification would violate P11. +Rechecked 2026-10-04: the instrument mismatch in +`docs/evidence/2026-09-14-ir-wp-0004-review.md` is unchanged, and requester +access is still absent because IR-WP-0005-T02 is waiting on open decision +`62aa95e4-3ccc-4867-8806-5e3db005b483`. No benchmark versions were pinned and +no Frontier/Efficient comparison was produced. Stays todo. + ## Collect and assess representative field outcomes ```task @@ -55,14 +61,20 @@ Residual IR-WP-0004-T02. Paid execution depends on BOTH IR-WP-0005 credential re Blocked, 2026-09-27: paid execution still needs IR-WP-0005-T02 (wait) and IR-WP-0006-T03 (wait) both closed. The sampling-frame/verifier design portion this task allows to proceed earlier is nontrivial methodology work in its own right (real task origins, held-out sets, -verifiers, failure taxonomy) and was not attempted in this session to avoid a rushed -design under time pressure; it remains open, undecomposed, for a dedicated pass. +verifiers, failure taxonomy) and was not attempted in the 2026-09-27 pass. + +Design portion recorded 2026-10-04 in +`docs/evidence/2026-10-04-ir-wp-0007-t02-sampling-frame.md`: origin, +inclusion, exclusion, hold-out, dependence groups, and a failure taxonomy +that includes verifier dependence (reward hacking and a checker that shares +the generator's errors). No field tasks were frozen and no runs were made. +Paid execution still waits. Status stays wait. ## Complete reviewed publication and operational disposition ```task id: IR-WP-0007-T03 -status: todo +status: done priority: high state_hub_task_id: "314f79fd-d95a-5e02-b67d-67e35e4334ed" ``` @@ -75,6 +87,14 @@ definition (claim-to-capture references, exclusions, revision identity, correcti distinct from a generated draft) is still open; it is a schema/format design decision better made deliberately than folded into this pass. Stays todo. +Done 2026-10-04: `python3 -m intelligence_radar publish` records a reviewed +publication in `data/publications/` and renders `digests/reviewed/.md`. +An included claim must name a capture already in the ledger. An excluded claim +must name the reason. `supersedes` links a correction to an earlier revision. +The scheduler does not write these files, and a generated draft is not accepted +as the record. No publication was issued; nothing had been reviewed. The local +timer decision remains ADR 0003. + ## Compare decomposition with measured monolithic execution ```task diff --git a/workplans/IR-WP-0008-external-brief-intake.md b/workplans/IR-WP-0008-external-brief-intake.md index 3386562..1f8c811 100644 --- a/workplans/IR-WP-0008-external-brief-intake.md +++ b/workplans/IR-WP-0008-external-brief-intake.md @@ -71,3 +71,8 @@ path remains the brief inbox: an export saved there, including one saved from Grok's email delivery, is retained by T02. Pulling the websites themselves waits on a real export or an authorized account feed that does not guess private endpoints. + +Rechecked 2026-10-04: the inbox has no new export. It still holds the empty +3 October Grok file and the 27 September ChatGPT file whose archive name is +already taken by different bytes. email-connect has no Intelligence Radar +feed. No session was sent to chatgpt.com or grok.com. Stays wait.