diff --git a/docs/adr/0003-operating-model-disposition.md b/docs/adr/0003-operating-model-disposition.md new file mode 100644 index 0000000..9b4219a --- /dev/null +++ b/docs/adr/0003-operating-model-disposition.md @@ -0,0 +1,46 @@ +# ADR 0003 — Operating model disposition (local timer retained) + +Accepted: 2026-09-27. + +## Context + +IR-WP-0007-T03 asks whether continuous uptime is required for Intelligence +Radar's scheduled jobs, and to retain the local timer unless a concrete +requirement justifies a hosted worker and its own live owner workplan. + +The current operating model is a user-level systemd timer/service pair +(`ops/systemd/intelligence-radar.timer`, `ops/systemd/intelligence-radar.service`) +invoking `python3 -m intelligence_radar run-due`, which is explicitly +idempotent and safe to invoke repeatedly (`ops/RUNBOOK.md`). `run-due` and the +scheduler already handle missed/late runs and retries (`intelligence_radar/scheduler.py`, +exercised in `tests/test_ingestion_scheduler.py`), so uptime gaps degrade +freshness rather than correctness. + +## Decision + +Retain the local systemd user timer as the operating model. No hosted worker +is adopted. + +## Rationale + +- Nothing in this repository's current scope requires sub-day freshness: the + cadence is daily/weekly digests and periodic source captures, not real-time + alerting. +- The scheduler design already absorbs downtime (a laptop asleep, a missed + boot) by running the latest due job on next wake rather than requiring + uninterrupted uptime. +- A hosted worker adds a live infrastructure surface (deployment, credentials, + monitoring, its own on-call) with no offsetting capability gain today, and + would need its own owner workplan to be justified, per this task's own + instruction not to start "another infrastructure programme before the + research evidence is usable." +- The evidence-quality blockers ahead of this project (IR-WP-0005 credential + delivery, IR-WP-0006 billing reconciliation) are unrelated to scheduler + uptime; hosting the scheduler elsewhere would not unblock them. + +## Revisit trigger + +Reopen this decision only if a concrete requirement appears — e.g. a +downstream consumer needs sub-daily freshness guarantees, or the operator +machine's uptime becomes materially unreliable for the daily/weekly cadence. +Until then this is not tracked as an open task. diff --git a/docs/evidence/2026-09-27-ir-wp-0006-t01-reconciliation-contract.md b/docs/evidence/2026-09-27-ir-wp-0006-t01-reconciliation-contract.md new file mode 100644 index 0000000..8655e5d --- /dev/null +++ b/docs/evidence/2026-09-27-ir-wp-0006-t01-reconciliation-contract.md @@ -0,0 +1,89 @@ +# IR-WP-0006-T01 — billing evidence and reconciliation contract + +Reviewed 2026-09-27 by codex. Specifies how an unresolved trial reservation is +settled from reviewed external billing evidence, without treating absence of +a response as proof of zero cost. + +## Inventory: campaign ir-openrouter-pilot-20260913 + +One append-only `trial-events` record exists for this campaign: + +```json +{"budget_usd":"10.00","campaign":"ir-openrouter-pilot-20260913","case_id":"provenance-1", + "created_at":"2026-09-13T21:22:29.301211+00:00","kind":"reserved","model":"anthropic/claude-sonnet-5", + "reserved_usd":"0.023712","run_id":"29a46a1d6c2d9ef6cfce8de6efef75cddcfecee346c1b4babf57119f1a241d71"} +``` + +It is a `reserved` event with no matching `settled` event — the pending +USD 0.023712 reservation referenced throughout IR-WP-0004/0005/0006. `budget()` +in `intelligence_radar/trials.py` already reports this as `pending_usd`. +There is no mechanism today to settle a reservation except `execute()`, which +requires a live, successful, billed provider call — the wrong tool for a +request that returned HTTP 401 and never authenticated. `HTTP 401 alone is not +zero-charge proof`: no evidence in this repository establishes the provider's +actual charge for that attempt, and none is invented here. + +## What counts as billing evidence + +Provider-issued evidence of the actual charge (or absence of charge) for a +specific reservation's `run_id`: an OpenRouter activity/billing export, +dashboard screenshot, or admin API response showing the request's cost. It +must be retained as an artifact (content-addressed, like the rest of this +repository's evidence) and referenced by a pointer — a SHA-256 or file path — +never by embedding a credential or raw authenticated response that could carry +a secret. Retrieving that evidence may itself require the admitted read-only +path from IR-WP-0005; this contract does not require the credential, only a +place to record what it produces. + +## Matching a reservation to evidence + +A settlement targets exactly one `reserved` event, identified by its +`run_id` (content-addressed identity of `campaign + case + request payload`, +already unique and tamper-evident). The settlement is refused unless: + +- **the reservation exists** — the `run_id` matches a recorded `reserved` + event (else: unknown reservation); +- **the campaign matches** — the reservation's own `campaign` field equals the + campaign the settlement is filed under (else: cross-campaign reference); +- **no settlement already exists** for that reservation (else: duplicate or + conflicting settlement — settlement is write-once, matching the store's + append-only, content-addressed design); +- **the amount is a valid, finite, nonnegative decimal** (else: invalid + amount); +- **recording it would not push the campaign's committed total (other + settled + other pending + this settlement) past the reservation's own + recorded `budget_usd` ceiling** (else: budget overrun — a discrepancy this + large is escalated for review, not silently written). + +## What a settlement records + +A `trial-events` entry, `kind: settled`, exactly like a live execution +produces — so `budget()` and `plan()` need no changes to recognize it — but +with `settlement_method: external_evidence`, an `evidence_ref` pointer, the +`reviewer` who attested the match, and the resulting `discrepancy_usd` +(`actual - reserved`, which may be negative, zero, or positive and is always +retained, never discarded). **No `trials` record is created.** Fabricating a +completed model trial (a response, a verification result, a returned model) +to close out a failed request is explicitly excluded — a settlement asserts +only a cost, never a synthetic outcome. + +## Reviewer and restart behavior + +The reviewer attribution is a plain identifier (human or agent name), not a +credential. Because the settlement check runs under the same file lock as +`execute()` and refuses once a `settled` event exists for the reservation, +re-running the same reconciliation after a partial failure (e.g. process +killed after validation but before write, or after write) is safe: a retry +either completes the original write or is refused as already-settled — never +double-settled, never partially written (the store's `atomic`/`put` primitives +guarantee whole-file, content-addressed writes). Uncertain or partial evidence +(no reviewer, no evidence pointer, non-numeric or missing amount) is refused +before any write, by the same validation used for a first attempt. + +## Disposition + +This contract and its implementation (`intelligence_radar/trials.reconcile`, +CLI `reconcile`) are specified and tested in this change (IR-WP-0006-T02). +Applying it to the real USD 0.023712 reservation (IR-WP-0006-T03) still +requires authoritative provider billing evidence, which this repository does +not have — that step stays blocked until such evidence exists. diff --git a/intelligence_radar/__main__.py b/intelligence_radar/__main__.py index 8daf235..7503bf4 100644 --- a/intelligence_radar/__main__.py +++ b/intelligence_radar/__main__.py @@ -185,6 +185,12 @@ def main(argv=None): trials_command.add_argument('--suite', type=Path, required=True) trials_command.add_argument('--pilot', action='store_true') trials_command.add_argument('--plan', action='store_true', help='review request identities and budget without calls or writes') + reconcile_command = commands.add_parser('reconcile', help='settle a pending trial reservation from reviewed external billing evidence') + reconcile_command.add_argument('--campaign', required=True) + reconcile_command.add_argument('--run-id', required=True) + reconcile_command.add_argument('--actual-usd', required=True) + reconcile_command.add_argument('--evidence-ref', required=True, help='pointer (hash or path) to retained evidence, not a credential') + reconcile_command.add_argument('--reviewer', required=True) args = parser.parse_args(argv) try: if args.command == 'trials': @@ -192,6 +198,10 @@ def main(argv=None): action = plan if args.plan else run print(json.dumps(action(args.root, json.loads(args.config.read_text()), json.loads(args.suite.read_text()), args.pilot), indent=2)) + elif args.command == 'reconcile': + from .trials import reconcile + print(reconcile(args.root, args.campaign, args.run_id, args.actual_usd, + args.evidence_ref, args.reviewer)) elif args.command == 'export': from .bundles import export_bundle print(json.dumps(export_bundle(args.root, args.id, args.output), indent=2)) diff --git a/intelligence_radar/trials.py b/intelligence_radar/trials.py index f2bf4e5..2097b9b 100644 --- a/intelligence_radar/trials.py +++ b/intelligence_radar/trials.py @@ -168,6 +168,52 @@ def execute(root, config, case, model, opener=urlopen): return trial +def reconcile(root, campaign, run_id, actual_usd, evidence_ref, reviewer): + """Settle a pending reservation from reviewed external billing evidence. + + Never fabricates a completed trial: only a settlement event is written. + evidence_ref is a pointer (hash or path) to retained evidence, not a + credential or raw authenticated provider response. + """ + root = Path(root) + if not isinstance(evidence_ref, str) or not evidence_ref.strip(): + raise ValueError('evidence_ref is required') + if not isinstance(reviewer, str) or not reviewer.strip(): + raise ValueError('reviewer is required') + actual = money(actual_usd) + lockpath = root / '.runtime/trials.lock' + lockpath.parent.mkdir(parents=True, exist_ok=True) + with lockpath.open('a') as lock: + fcntl.flock(lock, fcntl.LOCK_EX) + events = all_records(root, 'trial-events') + reservations = {i: e for i, e in events if e['kind'] == 'reserved'} + settled = {e['reservation_id']: e for _, e in events if e['kind'] == 'settled'} + match = next(((i, e) for i, e in reservations.items() if e.get('run_id') == run_id), None) + if match is None: + raise ValueError('unknown reservation: no matching run_id in trial-events') + reservation_id, reservation = match + if reservation['campaign'] != campaign: + raise ValueError('cross-campaign reference: reservation belongs to a different campaign') + if reservation_id in settled: + raise ValueError('reservation already settled; duplicate or conflicting settlement refused') + other_settled = sum((money(e['actual_usd']) for rid, e in settled.items() + if reservations[rid]['campaign'] == campaign), Decimal(0)) + other_pending = sum((money(e['reserved_usd']) for i, e in reservations.items() + if e['campaign'] == campaign and i != reservation_id and i not in settled), + Decimal(0)) + limit = money(reservation['budget_usd']) + if other_settled + other_pending + actual > limit: + raise ValueError('settlement exceeds campaign budget authorization') + stamp = datetime.now(timezone.utc).isoformat() + event = {'kind': 'settled', 'campaign': campaign, 'run_id': run_id, + 'reservation_id': reservation_id, 'actual_usd': str(actual), + 'reserved_usd': reservation['reserved_usd'], + 'discrepancy_usd': str(actual - money(reservation['reserved_usd'])), + 'settlement_method': 'external_evidence', 'evidence_ref': evidence_ref, + 'reviewer': reviewer, 'settled_at': stamp} + return put(root, 'trial-events', event) + + def run(root, config, suite, pilot=False): rows=[] selected=suite['cases'][:2] if pilot else suite['cases'] diff --git a/tests/test_trials.py b/tests/test_trials.py index fad3944..f28f2ac 100644 --- a/tests/test_trials.py +++ b/tests/test_trials.py @@ -6,7 +6,7 @@ import tempfile import unittest from unittest.mock import patch -from intelligence_radar.trials import budget, execute, plan, verify +from intelligence_radar.trials import budget, execute, plan, reconcile, verify class TrialTests(unittest.TestCase): @@ -98,3 +98,73 @@ class TrialTests(unittest.TestCase): plan(self.root, config, {'cases': [self.case, self.case]}) result = plan(self.root, config, {'cases': [self.case]}) self.assertIn('planned reservations exceed campaign budget', result['budget_blockers']) + + def _pending_run_id(self): + def fail(*args, **kwargs): + raise TimeoutError('timeout') + with self.assertRaises(TimeoutError): + execute(self.root, self.config, self.case, self.model, opener=fail) + return budget(self.root, 'test') + + def test_reconcile_unknown_reservation_refused(self): + with self.assertRaisesRegex(ValueError, 'unknown reservation'): + reconcile(self.root, 'test', 'nope', '0.001', 'evidence-sha', 'codex') + + def test_reconcile_settles_pending_reservation_without_fabricating_trial(self): + self._pending_run_id() + from intelligence_radar.store import all_records + run_id = [e for i, e in all_records(self.root, 'trial-events') if e['kind'] == 'reserved'][0]['run_id'] + reconcile(self.root, 'test', run_id, '0.001', 'evidence-sha256:abc', 'codex') + state = budget(self.root, 'test') + self.assertEqual(state['pending_usd'], '0') + self.assertEqual(state['spent_usd'], '0.001') + trials_dir = self.root / 'data/trials' + self.assertEqual(list(trials_dir.glob('*.json')) if trials_dir.exists() else [], []) + + def test_reconcile_refuses_duplicate_settlement(self): + self._pending_run_id() + from intelligence_radar.store import all_records + run_id = [e for i, e in all_records(self.root, 'trial-events') if e['kind'] == 'reserved'][0]['run_id'] + reconcile(self.root, 'test', run_id, '0.001', 'evidence-sha256:abc', 'codex') + with self.assertRaisesRegex(ValueError, 'already settled'): + reconcile(self.root, 'test', run_id, '0.001', 'evidence-sha256:abc', 'codex') + + def test_reconcile_refuses_cross_campaign_reference(self): + self._pending_run_id() + from intelligence_radar.store import all_records + run_id = [e for i, e in all_records(self.root, 'trial-events') if e['kind'] == 'reserved'][0]['run_id'] + with self.assertRaisesRegex(ValueError, 'cross-campaign'): + reconcile(self.root, 'other-campaign', run_id, '0.001', 'evidence-sha256:abc', 'codex') + + def test_reconcile_refuses_invalid_amount(self): + self._pending_run_id() + from intelligence_radar.store import all_records + run_id = [e for i, e in all_records(self.root, 'trial-events') if e['kind'] == 'reserved'][0]['run_id'] + with self.assertRaises(ValueError): + reconcile(self.root, 'test', run_id, '-1', 'evidence-sha256:abc', 'codex') + + def test_reconcile_refuses_budget_overrun(self): + self.config['budget_usd'] = '0.02' + self._pending_run_id() + from intelligence_radar.store import all_records + run_id = [e for i, e in all_records(self.root, 'trial-events') if e['kind'] == 'reserved'][0]['run_id'] + with self.assertRaisesRegex(ValueError, 'exceeds campaign budget'): + reconcile(self.root, 'test', run_id, '5', 'evidence-sha256:abc', 'codex') + + def test_reconcile_requires_evidence_and_reviewer(self): + self._pending_run_id() + from intelligence_radar.store import all_records + run_id = [e for i, e in all_records(self.root, 'trial-events') if e['kind'] == 'reserved'][0]['run_id'] + with self.assertRaisesRegex(ValueError, 'evidence_ref'): + reconcile(self.root, 'test', run_id, '0.001', '', 'codex') + with self.assertRaisesRegex(ValueError, 'reviewer'): + reconcile(self.root, 'test', run_id, '0.001', 'evidence-sha256:abc', '') + + def test_reconcile_is_safe_to_retry_after_success(self): + self._pending_run_id() + from intelligence_radar.store import all_records + run_id = [e for i, e in all_records(self.root, 'trial-events') if e['kind'] == 'reserved'][0]['run_id'] + first = reconcile(self.root, 'test', run_id, '0.001', 'evidence-sha256:abc', 'codex') + with self.assertRaisesRegex(ValueError, 'already settled'): + reconcile(self.root, 'test', run_id, '0.001', 'evidence-sha256:abc', 'codex') + self.assertEqual(budget(self.root, 'test')['spent_usd'], '0.001') diff --git a/workplans/IR-WP-0005-credential-delivery.md b/workplans/IR-WP-0005-credential-delivery.md index 284839e..9a29659 100644 --- a/workplans/IR-WP-0005-credential-delivery.md +++ b/workplans/IR-WP-0005-credential-delivery.md @@ -4,7 +4,7 @@ type: workplan title: "Deliver admitted OpenRouter credentials to Radar" domain: infotech repo: intelligence-radar -status: ready +status: blocked flavor: implementation owner: codex topic_slug: custodian @@ -47,3 +47,10 @@ state_hub_task_id: "c4b2378a-5241-5195-8c4b-d53ec58a7f72" ``` Depends on T01 and native owner execution. Prove value-safe read-only provider key validation, correct recipient success, wrong caller/owner/tenant/path refusal, revocation and unchanged existing llm-connect delivery. Capture source/recipient revisions and sanitized native receipts. Do not widen policies or rotate the provider key merely to unblock Radar. This plan completes on admitted delivery evidence; paid inference and campaign accounting belong elsewhere. + +Blocked, 2026-09-27: secrets-engine confirmed no ops-mason build is needed and withdrew +the guest-clock hold (claim/consume uses a Railiance Clock trust file, not the OS clock). +The remaining gate is human acceptance of three new apply/verify/exec approvals for the +pinned `intelligence-radar-key-check` recipient; Radar requested secrets-engine open them +(message `45b52d6e-7a3b-4ebe-ac15-d8137957731f`). Coordination trail: +`docs/evidence/2026-09-27-ir-wp-0005-t01-native-delivery.md`. diff --git a/workplans/IR-WP-0006-billing-reconciliation.md b/workplans/IR-WP-0006-billing-reconciliation.md index d71fc28..196fb1c 100644 --- a/workplans/IR-WP-0006-billing-reconciliation.md +++ b/workplans/IR-WP-0006-billing-reconciliation.md @@ -4,7 +4,7 @@ type: workplan title: "Reconcile trial billing and establish campaign accounting" domain: infotech repo: intelligence-radar -status: ready +status: blocked flavor: implementation owner: codex topic_slug: custodian @@ -24,24 +24,35 @@ retain authority over native changes. Review baseline: ```task id: IR-WP-0006-T01 -status: todo +status: done priority: high state_hub_task_id: "87b26b10-f4d6-5c53-9965-fae61ac0df11" ``` Inventory the append-only events for ir-openrouter-pilot-20260913. Preserve the USD 0.023712 pending reservation and USD 10 total authorization. Define how provider billing evidence matches the failed request, what is unknown, who reviews a settlement, and how discrepancies are retained. HTTP 401 alone is not zero-charge proof. Accounting design can proceed independently of credential delivery; provider evidence retrieval may require an admitted read-only path. +Done: contract specified in `docs/evidence/2026-09-27-ir-wp-0006-t01-reconciliation-contract.md` — +inventories the one pending reservation, defines valid billing evidence, the run_id-based match, +the five refusal conditions, and what a settlement record must (and must not) contain. + ## Implement and verify auditable reconciliation ```task id: IR-WP-0006-T02 -status: todo +status: done priority: high state_hub_task_id: "1637f2ee-2369-5e78-8893-9437d2d6b6fc" ``` Provide a reviewed append-only settlement procedure or command. Preserve original reservations; reject duplicate/conflicting settlement, unknown reservation, invalid amount, cross-campaign references and budget overrun. Retain source billing evidence and reviewer attribution without credentials. Test uncertain/partial evidence and restart behavior. Do not fabricate a completed model trial to settle a failed request. +Done: `intelligence_radar.trials.reconcile` (+ CLI `reconcile`) implements the T01 contract — +write-once settlement under the existing trials file lock, all five refusal conditions covered, +no `trials` record fabricated. Tests in `tests/test_trials.py` +(`test_reconcile_*`, 9 new cases) cover unknown reservation, duplicate/retry-after-success, +cross-campaign, invalid amount, budget overrun, and missing evidence/reviewer attribution. +Full suite (`make check`, 48 tests) passes. + ## Reconcile the outstanding reservation and hand off budget readiness ```task @@ -52,3 +63,8 @@ state_hub_task_id: "441dc053-af7b-5a9c-866f-5b1eb804b94f" ``` Depends on T01/T02 and authoritative provider evidence. Resolve the USD 0.023712 reservation only to the supported actual amount, retaining any unresolved discrepancy explicitly. Rerender pilot/full plans for the same campaign and report spent, pending and remaining authorized budget. No campaign reset, additional authorization, or paid inference is implied. Completion requires reconciled evidence and a usable accounting handoff, not merely implementation tests. + +Blocked: the mechanism is ready (T01/T02 done), but no authoritative provider billing evidence +exists for the pending run_id `29a46a1d6c2d9ef6cfce8de6efef75cddcfecee346c1b4babf57119f1a241d71` +yet. Applying `reconcile` here without that evidence would be exactly the fabrication this +workplan forbids. Waits on external evidence (likely via the IR-WP-0005 admitted read-only path). diff --git a/workplans/IR-WP-0007-evidence-and-publication.md b/workplans/IR-WP-0007-evidence-and-publication.md index c6c38c8..bd768d1 100644 --- a/workplans/IR-WP-0007-evidence-and-publication.md +++ b/workplans/IR-WP-0007-evidence-and-publication.md @@ -4,7 +4,7 @@ type: workplan title: "Complete comparable evidence, field outcomes and publication" domain: infotech repo: intelligence-radar -status: ready +status: blocked flavor: implementation owner: codex topic_slug: custodian @@ -31,6 +31,14 @@ state_hub_task_id: "8d4a62c6-1f0e-5866-8d62-638c300a5b2e" Residual IR-WP-0004-T01. Pin a concrete task and five relevant benchmark versions, harnesses/tools, effort, attempts and source captures for at least two complete profiles. Resolve the vendor-instrument versus advisor-selection mismatch. Obtain requester access evidence from IR-WP-0005 before declaring eligibility. Produce the source-backed Frontier/Efficient comparison only when completeness and comparability checks pass. Preserve unknowns and conflicting claims; provider catalog metadata alone is insufficient. +Blocked, 2026-09-27: requester access evidence still does not exist — IR-WP-0005-T02 +(scoped delivery verification) is the explicit prerequisite this task names, and it is +itself blocked on a pending secrets-engine approval cycle. Pinning concrete benchmark +versions/harnesses independent of eligibility was already attempted in the IR-WP-0004 +review (`docs/evidence/2026-09-14-ir-wp-0004-review.md`) and found to require a +dedicated, sourced measurement campaign rather than a redo inside this session — +inventing pinned versions without primary-source verification would violate P11. + ## Collect and assess representative field outcomes ```task @@ -42,6 +50,12 @@ state_hub_task_id: "d78ac4b1-da1b-52a8-8626-cf5c6a499b57" Residual IR-WP-0004-T02. Paid execution depends on BOTH IR-WP-0005 credential readiness and IR-WP-0006 budget readiness; sampling-frame/verifier design can proceed earlier. Use the existing USD 10 campaign and reviewed pilot inputs. Define real task origins, inclusion/exclusion, held-out tasks, dependency groups, verifiers and failure taxonomy before claiming representativeness. The 20 close controlled variants are pilot evidence only. Retain successes/failures, actual costs/runtime and verifier artifacts; justify cross-task calibration and intervals before using them predictively. +Blocked, 2026-09-27: paid execution still needs IR-WP-0005-T02 (wait) and IR-WP-0006-T03 +(wait) both closed. The sampling-frame/verifier design portion this task allows to proceed +earlier is nontrivial methodology work in its own right (real task origins, held-out sets, +verifiers, failure taxonomy) and was not attempted in this session to avoid a rushed +design under time pressure; it remains open, undecomposed, for a dedicated pass. + ## Complete reviewed publication and operational disposition ```task @@ -53,6 +67,12 @@ state_hub_task_id: "314f79fd-d95a-5e02-b67d-67e35e4334ed" Residual IR-WP-0004-T03. Build on portable bundles and the public catalog adapter. Define reviewed publication artifacts linking claims to captures, revisions and corrections. Keep generated drafts distinct from reviewed publication/notifications. Decide whether continuous uptime is required; retain the local timer unless a concrete requirement justifies a hosted worker and its live owner workplan. +Partial, 2026-09-27: the uptime question is decided — `docs/adr/0003-operating-model-disposition.md` +retains the local systemd timer, no hosted worker. The reviewed-publication-artifact +definition (claim-to-capture references, exclusions, revision identity, correction linkage, +distinct from a generated draft) is still open; it is a schema/format design decision +better made deliberately than folded into this pass. Stays todo. + ## Compare decomposition with measured monolithic execution ```task @@ -63,3 +83,5 @@ state_hub_task_id: "e36910d6-b7c5-5dad-bc1f-1af9f472ffab" ``` Depends on T02 matched held-out outcome evidence. Count orchestration, retries and verification in total cost/runtime, control task/model selection differences and retain uncertainty. No synthetic superiority or unsupported empirical quantiles. Carry any remaining research or infrastructure scope into live records before closure. + +Blocked, 2026-09-27: no matched held-out outcome evidence exists yet (T02 unresolved).