Review IR-WP-0004 evidence gaps and capture provider profile metadata

Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a09cbc-910e-77a0-b336-3130194350b3

catalog/sources.json

@@ -19,6 +19,11 @@
       "id": "anthropic-sonnet-launch",
       "publisher": "Anthropic",
       "url": "https://www.anthropic.com/news/claude-sonnet-5"
+    },
+    {
+      "id": "openrouter-models",
+      "publisher": "OpenRouter",
+      "url": "https://openrouter.ai/api/v1/models"
     }
   ]
 }

data/captures/f86bd0eabdae693391c5ca957523067f85426b24a349a7f0a271a2bb3622741c.json

@@ -0,0 +1 @@
+{"attempts":1,"bytes":735812,"changed":true,"content_sha256":"a76ee26133eb686338e5b8bb4b1d9be0f40615a04312520e8d513e222337331c","final_url":"https://openrouter.ai/api/v1/models","media_type":"application/json","publisher":"OpenRouter","retrieved_at":"2026-09-14T13:44:03.965875+00:00","schema_version":1,"source_id":"openrouter-models","status":"captured_unreviewed","url":"https://openrouter.ai/api/v1/models"}

data/observations/89f64b28ca9dbaba4add4087fb3648dee48dcb6959e5f94773416f4919f7c768.json

@@ -0,0 +1,16 @@
+{
+  "id": "89f64b28ca9dbaba4add4087fb3648dee48dcb6959e5f94773416f4919f7c768",
+  "record": {
+    "schema_version": 1,
+    "title": "Source update: openrouter-models",
+    "claim": "Source content was captured or changed. Extracted claims require editorial review.",
+    "source_url": "https://openrouter.ai/api/v1/models",
+    "source_kind": "primary",
+    "content_sha256": "a76ee26133eb686338e5b8bb4b1d9be0f40615a04312520e8d513e222337331c",
+    "observed_at": "2026-09-14T13:44:03.965875+00:00",
+    "event_date": "2026-09-14",
+    "status": "reported",
+    "observer": "intelligence-radar/capture-v0.2",
+    "notes": "Capture ID: f86bd0eabdae693391c5ca957523067f85426b24a349a7f0a271a2bb3622741c"
+  }
+}

docs/data-contracts.md

@@ -152,3 +152,18 @@ and input bytes; plan config/suite hashes use canonical JSON. The owner must
 install and admit a fixed recipient separately, then rerender against current
 billing before execution. Mutable ledger state and provider availability may
 change after planning.
+
+### Public provider catalog review
+
+```bash
+python3 -m intelligence_radar.profile_review \
+  --capture f86bd0eabdae693391c5ca957523067f85426b24a349a7f0a271a2bb3622741c \
+  --config ops/openrouter-pilot.json
+```
+
+This offline adapter verifies the retained capture/artifact hashes and transcribes
+configured model IDs, provider canonical slugs, context, parameters and base token
+prices. Missing/invalid prices remain unknown. It does not establish account access,
+endpoint pricing, comparable benchmarks or execution authorization. Capture a fresh
+public model catalog through the existing source collector before each review;
+retain old captures and use the new capture ID explicitly. No credential is needed.

docs/evidence/2026-09-14-ir-wp-0004-review.md

@@ -0,0 +1,96 @@
+# IR-WP-0004 evidence and completion review
+
+Reviewed 2026-09-14 by codex. Workplan remains active. No paid calls in this review.
+
+## What the current evidence supports
+
+The retained [public catalog transcription](2026-09-14-openrouter-public-review.json)
+links to the original content-addressed OpenRouter response and campaign config hash.
+The offline adapter can reproduce the transcription without credentials or network.
+
+| Campaign model | Advertised prompt/completion USD per million tokens | Within campaign base-price caps |
+| --- | --- | --- |
+| anthropic/claude-sonnet-5 | 2 / 10 | yes |
+| anthropic/claude-haiku-4.5 | 1 / 5 | yes |
+| deepseek/deepseek-v4-flash | 0.08554 / 0.17108 | yes |
+
+All three IDs were listed in the captured [OpenRouter catalog](https://openrouter.ai/api/v1/models).
+The response also supplies canonical slugs and context limits. These are advertised
+provider metadata at capture time, not account-access proofs, immutable serving
+identities, benchmark results or guaranteed routed prices. Extra charges and routing
+need endpoint review before execution. Existing campaign and price caps are unchanged.
+
+## T01 — comparable profiles
+
+The five vendor-reported instruments in
+`catalog/anthropic-reported-benchmarks-2026-09-13.json` are SWE-bench Pro,
+Terminal-Bench, BrowseComp, HLE with tools and OSWorld. They describe different
+harnesses/tools/effort. The earlier advisor task instead selects SciCode,
+Terminal-Bench, BrowseComp, BFCL and SWE-bench Verified. Pro and Verified are
+not interchangeable. Model names alone cannot join these two evidence sets.
+
+There are therefore two distinct unresolved steps:
+
+1. Select a concrete task and five relevant instruments, pin each benchmark version,
+   scoring protocol, harness/tools, effort and attempts, and retain comparable rows
+   for at least two complete execution profiles. Keep missing settings unknown.
+2. Establish requester access and the relationship between measured profiles and
+   the actual deployment. Produce the Frontier/Efficient comparison only if the
+   eligibility and complete-evidence checks pass. Today's price/listing review
+   supplies provider metadata only.
+
+Do not fill the existing default-API profile's scores from the vendor transcription.
+Do not rename controlled work samples as public benchmark runs. If exact public
+comparability cannot be established, document that outcome and propose a scoped
+measurement campaign; changing acceptance requires an explicit workplan revision.
+
+## T02 — real outcomes and calibration
+
+The 20 existing cases are four close variants in each of five evidence-processing
+families. They can verify execution/accounting and reveal family-specific failures,
+but cannot satisfy representative production calibration by themselves.
+
+Execution order:
+
+1. SECRETS-WP-0010-T03 returns admitted native delivery evidence. Preserve the three
+   completed human approvals; validate their current native usability rather than
+   repeating or extending them. This plan does not depend on finishing RCLK.
+2. Reconcile the USD 0.023712 pending reservation with provider billing evidence,
+   appending a settlement through a reviewed accounting procedure. No assumption
+   that HTTP 401 or lack of a response proves zero cost.
+3. Rerender the existing campaign plans and recheck endpoint price/parameter support.
+   Use the authorized pilot first, verify native response/usage/actual cost and
+   retained verifier artifacts, then continue only within the same USD 10 total.
+4. Before calling observations representative, freeze a field-task sampling frame:
+   real task origin and inclusion/exclusion criteria, family/dependence grouping,
+   acceptance verifier and failure taxonomy. Hold out tasks not used to tune the
+   prompts; record rejected/failed tasks and the model-selection process.
+5. Report counts by independent task/family, execution failures separately from
+   verification failures, actual cost/runtime, and limitations. No cross-task
+   Wilson probability or cost/runtime quantiles until a justified analysis exists.
+
+The paid pilot's outcome data should remain outside advisor production calibration
+until this review is satisfied. Synthetic tests cannot discharge these gates.
+
+## T03 — publication and operational reach
+
+Completed increments: portable bundles; public-catalog adapter and retained
+transcription; this reviewed, source-linked editorial assessment. Existing generated
+source-change digests still require editorial review.
+
+Remaining delivery:
+
+- Define a reviewed publication artifact with claim-to-capture references, exclusions,
+  revision identity and correction linkage; a generated digest is only a draft.
+- Compare decomposition against measured monolithic outcomes on matched held-out
+  tasks, counting orchestration, retries and verification in total cost/runtime.
+  This depends on T02 evidence; no synthetic superiority claim.
+- Decide whether continuous uptime is needed. The current local timer remains the
+  delivered operating model. A hosted worker is conditional, not an excuse to start
+  another infrastructure programme before the research evidence is usable.
+
+## Closure rule
+
+Keep T01/T03 in progress and T02 waiting. Complete the acceptance evidence above,
+or explicitly move deferred scope into identified live successor work records
+before proposing a narrower closure. No scope is silently discarded here.

docs/evidence/2026-09-14-openrouter-public-review.json

@@ -0,0 +1,126 @@
+{
+  "schema_version": 1,
+  "kind": "public_catalog_transcription",
+  "capture_id": "f86bd0eabdae693391c5ca957523067f85426b24a349a7f0a271a2bb3622741c",
+  "artifact_sha256": "a76ee26133eb686338e5b8bb4b1d9be0f40615a04312520e8d513e222337331c",
+  "observed_at": "2026-09-14T13:44:03.965875+00:00",
+  "config_sha256": "6cd62c6f05f08e2be09af34b033c1edee1a24f94c4f81b524921eb12d2276582",
+  "campaign": "ir-openrouter-pilot-20260913",
+  "execution_authorized": false,
+  "limitations": [
+    "Catalog listing does not prove requester access or backend availability",
+    "Base prices exclude extra charges and do not guarantee routed endpoint prices",
+    "Canonical slug is provider metadata, not an immutable serving-weight attestation",
+    "This review supplies no benchmark score, calibration or spending authorization"
+  ],
+  "models": [
+    {
+      "model_id": "anthropic/claude-sonnet-5",
+      "listed": true,
+      "requester_access_verified": null,
+      "benchmark_comparability_verified": null,
+      "canonical_slug": "anthropic/claude-sonnet-5-20260630",
+      "advertised_context_length": 1000000,
+      "supported_parameters": [
+        "include_reasoning",
+        "max_completion_tokens",
+        "max_tokens",
+        "reasoning",
+        "reasoning_effort",
+        "response_format",
+        "stop",
+        "structured_outputs",
+        "tool_choice",
+        "tools",
+        "verbosity"
+      ],
+      "base_prices": {
+        "prompt": {
+          "advertised_usd_per_million": "2.000000",
+          "within_configured_cap": true
+        },
+        "completion": {
+          "advertised_usd_per_million": "10.00000",
+          "within_configured_cap": true
+        }
+      },
+      "pricing_requires_endpoint_review": false
+    },
+    {
+      "model_id": "anthropic/claude-haiku-4.5",
+      "listed": true,
+      "requester_access_verified": null,
+      "benchmark_comparability_verified": null,
+      "canonical_slug": "anthropic/claude-4.5-haiku-20251001",
+      "advertised_context_length": 200000,
+      "supported_parameters": [
+        "include_reasoning",
+        "max_completion_tokens",
+        "max_tokens",
+        "reasoning",
+        "response_format",
+        "stop",
+        "structured_outputs",
+        "temperature",
+        "tool_choice",
+        "tools",
+        "top_k",
+        "top_p"
+      ],
+      "base_prices": {
+        "prompt": {
+          "advertised_usd_per_million": "1.000000",
+          "within_configured_cap": true
+        },
+        "completion": {
+          "advertised_usd_per_million": "5.000000",
+          "within_configured_cap": true
+        }
+      },
+      "pricing_requires_endpoint_review": false
+    },
+    {
+      "model_id": "deepseek/deepseek-v4-flash",
+      "listed": true,
+      "requester_access_verified": null,
+      "benchmark_comparability_verified": null,
+      "canonical_slug": "deepseek/deepseek-v4-flash-20260423",
+      "advertised_context_length": 1048576,
+      "supported_parameters": [
+        "frequency_penalty",
+        "include_reasoning",
+        "logit_bias",
+        "logprobs",
+        "max_completion_tokens",
+        "max_tokens",
+        "min_p",
+        "presence_penalty",
+        "reasoning",
+        "reasoning_effort",
+        "repetition_penalty",
+        "response_format",
+        "seed",
+        "stop",
+        "structured_outputs",
+        "temperature",
+        "tool_choice",
+        "tools",
+        "top_a",
+        "top_k",
+        "top_logprobs",
+        "top_p"
+      ],
+      "base_prices": {
+        "prompt": {
+          "advertised_usd_per_million": "0.08554000000",
+          "within_configured_cap": true
+        },
+        "completion": {
+          "advertised_usd_per_million": "0.17108000000",
+          "within_configured_cap": true
+        }
+      },
+      "pricing_requires_endpoint_review": false
+    }
+  ]
+}

intelligence_radar/profile_review.py

@@ -0,0 +1,77 @@
+"""Offline OpenRouter catalog transcription; never grants trial eligibility."""
+import argparse
+from decimal import Decimal, InvalidOperation
+import hashlib
+import json
+from pathlib import Path
+
+from .store import canonical, get
+
+SOURCE = 'https://openrouter.ai/api/v1/models'
+
+
+def review(root, capture_id, config):
+    receipt = get(root, 'captures', capture_id)
+    if receipt['url'] != SOURCE or receipt['final_url'] != SOURCE:
+        raise ValueError('expected the OpenRouter public model catalog')
+    raw = (Path(root) / 'data/artifacts' / receipt['content_sha256']).read_bytes()
+    if hashlib.sha256(raw).hexdigest() != receipt['content_sha256']:
+        raise ValueError('catalog artifact integrity mismatch')
+    entries = json.loads(raw)['data']
+    ids = [item['id'] for item in entries]
+    if len(ids) != len(set(ids)):
+        raise ValueError('ambiguous duplicate catalog model ID')
+    configured = [item['id'] for item in config['models']]
+    if len(configured) != len(set(configured)):
+        raise ValueError('duplicate configured model ID')
+    index = {item['id']: item for item in entries}
+    rows = []
+    for model in config['models']:
+        item = index.get(model['id'])
+        row = {'model_id': model['id'], 'listed': item is not None,
+               'requester_access_verified': None, 'benchmark_comparability_verified': None}
+        if item is not None:
+            prices = {}
+            for field in ('prompt', 'completion'):
+                try:
+                    rate = Decimal(str(item.get('pricing', {}).get(field))) * 1000000
+                    cap = Decimal(str(model['max_usd_per_million'][field]))
+                    if not rate.is_finite() or rate < 0 or not cap.is_finite() or cap < 0:
+                        raise ValueError('nonfinite or negative price')
+                except (InvalidOperation, ValueError, KeyError):
+                    prices[field] = {'advertised_usd_per_million': None, 'within_configured_cap': None}
+                else:
+                    prices[field] = {'advertised_usd_per_million': str(rate),
+                                     'within_configured_cap': rate <= cap}
+            row.update(canonical_slug=item.get('canonical_slug'),
+                       advertised_context_length=item.get('context_length'),
+                       supported_parameters=item.get('supported_parameters', []),
+                       base_prices=prices,
+                       pricing_requires_endpoint_review=bool(item.get('pricing', {}).get('overrides')))
+        rows.append(row)
+    return {'schema_version': 1, 'kind': 'public_catalog_transcription',
+            'capture_id': capture_id, 'artifact_sha256': receipt['content_sha256'],
+            'observed_at': receipt['retrieved_at'],
+            'config_sha256': hashlib.sha256(canonical(config)).hexdigest(),
+            'campaign': config['campaign'], 'execution_authorized': False,
+            'limitations': ['Catalog listing does not prove requester access or backend availability',
+                            'Base prices exclude extra charges and do not guarantee routed endpoint prices',
+                            'Canonical slug is provider metadata, not an immutable serving-weight attestation',
+                            'This review supplies no benchmark score, calibration or spending authorization'],
+            'models': rows}
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--root', type=Path, default=Path.cwd())
+    parser.add_argument('--capture', required=True)
+    parser.add_argument('--config', type=Path, required=True)
+    args = parser.parse_args()
+    try:
+        print(json.dumps(review(args.root, args.capture, json.loads(args.config.read_text())), indent=2))
+    except (OSError, ValueError, KeyError, TypeError) as exc:
+        parser.exit(1, f'Catalog review failed: {exc}\n')
+
+
+if __name__ == '__main__':
+    main()

tests/test_profile_review.py

@@ -0,0 +1,64 @@
+import hashlib
+import json
+from pathlib import Path
+import tempfile
+import unittest
+
+from intelligence_radar.profile_review import SOURCE, review
+from intelligence_radar.store import put
+
+
+class ProfileReviewTests(unittest.TestCase):
+    def setUp(self):
+        self.tmp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.tmp.cleanup)
+        self.root = Path(self.tmp.name)
+        self.config = {'campaign': 'test', 'models': [
+            {'id': 'model-a', 'max_usd_per_million': {'prompt': '2', 'completion': '10'}},
+            {'id': 'missing', 'max_usd_per_million': {'prompt': '2', 'completion': '10'}}]}
+        self.item = {'id': 'model-a', 'canonical_slug': 'model-a-version',
+                     'pricing': {'prompt': '0.000002', 'completion': '0.000011'}}
+
+    def capture(self, entries, url=SOURCE):
+        raw = json.dumps({'data': entries}).encode()
+        sha = hashlib.sha256(raw).hexdigest()
+        path = self.root / 'data/artifacts' / sha
+        path.parent.mkdir(parents=True, exist_ok=True)
+        path.write_bytes(raw)
+        return put(self.root, 'captures', {'url': url, 'final_url': url,
+                   'content_sha256': sha, 'retrieved_at': '2026-09-14T00:00:00Z'}), path
+
+    def test_prices_and_listing_do_not_establish_access(self):
+        ident, _ = self.capture([self.item])
+        result = review(self.root, ident, self.config)
+        self.assertFalse(result['execution_authorized'])
+        row = result['models'][0]
+        self.assertIsNone(row['requester_access_verified'])
+        self.assertIsNone(row['benchmark_comparability_verified'])
+        self.assertTrue(row['base_prices']['prompt']['within_configured_cap'])
+        self.assertFalse(row['base_prices']['completion']['within_configured_cap'])
+        self.assertFalse(result['models'][1]['listed'])
+
+    def test_invalid_or_missing_prices_remain_unknown(self):
+        for price in ['NaN', '-1', None]:
+            self.item['pricing'] = {'prompt': price}
+            ident, _ = self.capture([self.item])
+            row = review(self.root, ident, self.config)['models'][0]
+            for field in ['prompt', 'completion']:
+                self.assertIsNone(row['base_prices'][field]['within_configured_cap'])
+
+    def test_artifact_tamper_refused(self):
+        ident, path = self.capture([self.item])
+        path.write_text('{}')
+        with self.assertRaisesRegex(ValueError, 'integrity'):
+            review(self.root, ident, self.config)
+
+    def test_duplicate_models_refused(self):
+        ident, _ = self.capture([self.item, self.item])
+        with self.assertRaisesRegex(ValueError, 'duplicate'):
+            review(self.root, ident, self.config)
+
+    def test_wrong_source_refused(self):
+        ident, _ = self.capture([self.item], 'https://example.com/models')
+        with self.assertRaisesRegex(ValueError, 'expected'):
+            review(self.root, ident, self.config)

workplans/IR-WP-0004-evidence-coverage-and-field-calibration.md

@@ -110,3 +110,14 @@ that managed configuration. Restore domain connectivity (for example corporate
 VPN), resync and prove stable time before attended administration.
 Receipt: docs/evidence/2026-09-14-native-access-clock-check.json. No provider
 request, approval consumption or persistent clock configuration change occurred.
+
+### Evidence and completion review — 2026-09-14
+
+`docs/evidence/2026-09-14-ir-wp-0004-review.md` maps each remaining acceptance
+step to evidence and dependency. Added a public OpenRouter catalog adapter and
+retained live capture/transcription: all three configured IDs listed and base
+prices within caps, but account access and benchmark comparability stay unknown.
+The five vendor instruments do not match the original task selection or default
+API harness. The 20 correlated controlled cases are a pilot, not representative
+field calibration. T01/T03 remain progress; T02 remains wait on native delivery
+and billing. No campaign/request changes, paid inference calls, or scope removals.