Build reproducible capability advisor and scheduled source capture
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a09c77-b1a9-7eb3-b501-6cb24b458c97
This commit is contained in:
parent
a1edefd7c5
commit
29ceb77938
68 changed files with 2431 additions and 85 deletions
|
|
@ -1,5 +1,6 @@
|
|||
# Stack and commands
|
||||
|
||||
Python 3.11+ standard library. No install required. make check runs validation and
|
||||
tests; make demo executes a temporary synthetic workflow; make sync reconciles
|
||||
Statehub. python3 -m intelligence_radar --help lists operations. No deployed service.
|
||||
Python 3.11+ standard library. No install required. make check runs tests and integrity
|
||||
checks. make demo and make demo-advisor use isolated synthetic data. make sync pushes
|
||||
committed work and reconciles Statehub. CLI help lists source ingestion, advisor,
|
||||
outcomes, replay, reports, due jobs and health. User timer units live in ops/systemd/.
|
||||
|
|
|
|||
|
|
@ -1,24 +1,21 @@
|
|||
# Custodian Brief — intelligence-radar
|
||||
|
||||
Updated: 2026-09-13. Local offline orientation maintained from canonical workplans.
|
||||
|
||||
Domain: infotech. Topic: custodian (`cee7bedf-2b48-46ef-8601-006474f2ad7a`).
|
||||
Repo ID: `3130bd49-a43f-4682-a25d-68b4d4de9b1f`. Prefix: `IR-WP-`.
|
||||
Statehub: http://127.0.0.1:8000 (primary railiance01).
|
||||
Updated: 2026-09-13. Domain infotech; topic custodian
|
||||
(`cee7bedf-2b48-46ef-8601-006474f2ad7a`); prefix IR-WP-.
|
||||
Repo ID: `3130bd49-a43f-4682-a25d-68b4d4de9b1f`.
|
||||
Statehub: http://127.0.0.1:8000, primary railiance01.
|
||||
|
||||
## Current state
|
||||
|
||||
The two founding explorations are integrated. Local evidence recording, validation,
|
||||
and daily/weekly draft generation work. Advisor execution is manual. Source archive
|
||||
claims are unverified; no live model database or scheduler is installed.
|
||||
v0.2 runs source capture, task-specific advice, outcome recording and frozen replay.
|
||||
Daily/weekly jobs are configured; the bnt-lap001 user timer is enabled and its first
|
||||
service invocation succeeded. Four primary pages and a dated two-profile price
|
||||
comparison are retained. Real benchmark/access gaps are explicit; no production
|
||||
success calibration is claimed. The complete ranking demo is synthetic and isolated.
|
||||
|
||||
IR-WP-0001 bootstrap and IR-WP-0002 operational baseline are finished.
|
||||
IR-WP-0003 is backlog: verify source evidence, implement calibrated advisory
|
||||
capability and outcome records, then automate ingestion/cadence.
|
||||
IR-WP-0001 through IR-WP-0003 are finished. IR-WP-0004 is backlog: broaden verified
|
||||
profile evidence, collect real field outcomes, extend editorial/hosted operation.
|
||||
|
||||
## Session entry
|
||||
|
||||
Read INTENT.md → GOAL.md → SCOPE.md → AGENTS.md → workplans/.
|
||||
`make check` validates; `make demo` demonstrates with temporary synthetic data.
|
||||
Commit workplan changes, then `make sync` pushes and reconciles primary Statehub.
|
||||
See docs/evidence/bootstrap.md for the initial projection receipt and test evidence.
|
||||
Read INTENT → GOAL → SCOPE → AGENTS → workplans. docs/data-contracts.md describes the
|
||||
CLI. make check validates; make demo-advisor demonstrates; make sync pushes committed
|
||||
work and projects it to Statehub. See docs/evidence/v0.2-validation.md.
|
||||
|
|
|
|||
1
.gitignore
vendored
1
.gitignore
vendored
|
|
@ -5,3 +5,4 @@ __pycache__/
|
|||
build/
|
||||
dist/
|
||||
*.egg-info/
|
||||
.runtime/
|
||||
|
|
|
|||
|
|
@ -214,9 +214,9 @@ To create a new workplan:
|
|||
Read INTENT.md → GOAL.md → SCOPE.md → docs/integration-review.md.
|
||||
Python 3.11+ standard library; no installation required. `make check` runs tests
|
||||
and integrity checks; `make demo` runs isolated synthetic evidence; `make sync`
|
||||
uses Repo Manager to push and project workplans. No service is deployed.
|
||||
uses Repo Manager to push and project workplans. The local user timer runs source-capture/digest jobs; no hosted API is deployed.
|
||||
|
||||
Preserve history ZIP bytes. Treat imported catalogs and dated prose as unverified
|
||||
research leads. Append corrections as observations; never overwrite old records.
|
||||
Use TaskModelEvaluationMethod.md and versioned policies as normative; task-router
|
||||
is its manual interface. Do not equate benchmark fitness with task success.
|
||||
is its editorial interface. Read docs/data-contracts.md for executable v0.2 policy. Do not equate benchmark fitness with task success.
|
||||
|
|
|
|||
15
GOAL.md
15
GOAL.md
|
|
@ -1,12 +1,11 @@
|
|||
# Current goal
|
||||
|
||||
Establish a reproducible, manually operated v0.1 research facility from the two
|
||||
2026-09-13 explorations, with source provenance, one coherent advisory method,
|
||||
working evidence/digest tooling, and Coulomb/Statehub lifecycle integration.
|
||||
Ship v0.2: source capture and artifact retention, reproducible task-specific advisory
|
||||
reports, outcome-based empirical bounds, and local scheduled daily/weekly operations.
|
||||
|
||||
Acceptance: archives retain checksums; governing documents have explicit precedence;
|
||||
evidence can be recorded and checked; daily and weekly draft digests can be generated;
|
||||
tests pass; repo identity and file-backed workplans are registered and synced.
|
||||
Acceptance: live source receipts; a sourced comparison preserving evidence gaps;
|
||||
synthetic full-ranking/replay demonstration; tested outcome deduplication, stale and
|
||||
missing evidence handling, and durable scheduler retries; Statehub projection current.
|
||||
|
||||
Next stage: IR-WP-0003 covers verified source ingestion, calibrated advisory data,
|
||||
and unattended scheduling. The larger architecture remains a target.
|
||||
Next: broaden verified benchmark/profile coverage and collect representative actual
|
||||
outcomes. These research and calibration tasks continue in IR-WP-0004.
|
||||
|
|
|
|||
9
Makefile
9
Makefile
|
|
@ -10,3 +10,12 @@ demo:
|
|||
|
||||
sync:
|
||||
uv run --project /home/worsch/repo-manager rmgr sync --path . --push
|
||||
|
||||
demo-advisor:
|
||||
python3 tools/demo_advisor.py
|
||||
|
||||
install-timer:
|
||||
install -D -m 644 ops/systemd/intelligence-radar.service $(HOME)/.config/systemd/user/intelligence-radar.service
|
||||
install -D -m 644 ops/systemd/intelligence-radar.timer $(HOME)/.config/systemd/user/intelligence-radar.timer
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable --now intelligence-radar.timer
|
||||
|
|
|
|||
30
README.md
30
README.md
|
|
@ -2,26 +2,32 @@
|
|||
|
||||
Epistemic observatory and task-specific machine intelligence capability and economics advisor.
|
||||
|
||||
The initial operational version combines a **Compendium**, a manually curated
|
||||
**Radar**, and an on-demand **Capability Advisor** (also called Task Router).
|
||||
Read [INTENT](INTENT.md), [current goal](GOAL.md), and [scope](SCOPE.md).
|
||||
The v0.2 implementation combines a **Compendium**, a source-capture **Radar**, and
|
||||
an executable **Capability Advisor**. Read [INTENT](INTENT.md), [scope](SCOPE.md),
|
||||
and [the operating contracts](docs/data-contracts.md).
|
||||
|
||||
## Run locally
|
||||
|
||||
Requires Python 3.11+ and Make. No service, API key, or package installation is needed.
|
||||
Requires Python 3.11+ and Make. No package installation or API key is needed.
|
||||
|
||||
```bash
|
||||
make check
|
||||
make demo
|
||||
make demo-advisor
|
||||
python3 -m intelligence_radar ingest catalog/sources.json
|
||||
python3 -m intelligence_radar advise examples/source-backed-task.json \
|
||||
--catalog catalog/reviewed-2026-09-13.json --as-of 2026-09-13
|
||||
python3 -m intelligence_radar --help
|
||||
```
|
||||
|
||||
`make demo` exercises an isolated temporary ledger using synthetic evidence.
|
||||
For real work, follow [the runbook](ops/RUNBOOK.md): capture source evidence,
|
||||
record an observation, produce a dated digest draft, review it, and evaluate tasks
|
||||
with [the routing method](task-router/METHOD.md) and [ticket](task-router/TEMPLATE.md).
|
||||
The CLI supports recording and validating evidence and drafting daily/weekly digests.
|
||||
Model ranking and task calibration are currently analyst workflows.
|
||||
The advisor selects five measurements, checks constraints, ranks evidenced profiles,
|
||||
estimates workload cost scenarios, and records outcomes for empirical success bounds.
|
||||
Saved recommendations can be rendered with `report` and verified with `replay`.
|
||||
The source-backed example makes evidence gaps visible; the synthetic demo exercises
|
||||
complete ranking without claiming real model performance.
|
||||
|
||||
Source captures retain original bytes and retrieval receipts. Changed sources create
|
||||
review-needed observations. Daily/weekly jobs run through `run-due`; `health` shows
|
||||
results. See [scheduling and data contracts](docs/data-contracts.md).
|
||||
|
||||
## Knowledge and operations
|
||||
|
||||
|
|
@ -40,7 +46,7 @@ Model ranking and task calibration are currently analyst workflows.
|
|||
| `workplans/` | Canonical Coulomb work records projected into Statehub |
|
||||
|
||||
Imported September 2026 rankings and prices are research leads, not validated
|
||||
recommendations. No external automation has been activated by this repository.
|
||||
recommendations. The local user timer is managed separately from the historical external automation IDs.
|
||||
|
||||
Statehub: domain `infotech`, topic `custodian`, prefix `IR-WP-`.
|
||||
Commit workplan edits, then run `make sync` (pushes commits and reconciles Statehub). See [agent instructions](AGENTS.md).
|
||||
|
|
|
|||
45
SCOPE.md
45
SCOPE.md
|
|
@ -1,23 +1,34 @@
|
|||
# Scope
|
||||
|
||||
## Implemented
|
||||
## Implemented v0.2
|
||||
|
||||
- Governing intent, concept model, architecture, research program, benchmark and source catalogs.
|
||||
- Compendium, watchlist, historical draft digests and frontier snapshots from both explorations.
|
||||
- One authoritative task evaluation method, policy baseline, and manual ticket workflow.
|
||||
- Python CLI: validate and record immutable observation envelopes; validate the ledger;
|
||||
generate daily or ISO-weekly digest drafts by observation event date.
|
||||
- Local checks and a synthetic end-to-end demonstration.
|
||||
- Coulomb workplans, agent entry points, classification and capability registry.
|
||||
- Compendium, research program and historical explorations with explicit provenance.
|
||||
- Bounded HTTPS source capture, retained content-addressed artifacts, change receipts,
|
||||
review-needed observations, deduplication and per-source ingestion health.
|
||||
- Task-specific five-instrument selection, exact profile/deployment identity,
|
||||
constraints, benchmark score normalization, fitness ranking and workload cost scenarios.
|
||||
- Empirical outcome recording with unique runs, exact-task/profile Wilson intervals,
|
||||
provisional sufficiency thresholds and explicit missing evidence.
|
||||
- Frozen recommendations, deterministic historical replay and Markdown reports.
|
||||
- Configured Europe/Berlin daily/weekly job runner, retries, locking, health receipts,
|
||||
and user systemd timer units for the local workstation.
|
||||
- Twenty-one behavior tests and isolated ledger/advisor demonstrations.
|
||||
- Coulomb workplans and Statehub integration.
|
||||
|
||||
## Boundaries
|
||||
## Current evidence boundary
|
||||
|
||||
Statehub tracks repository work; `data/observations/` holds research evidence.
|
||||
A digest records what was observed; editorial review determines whether a claim
|
||||
is supported. A content hash proves local integrity, not truth or source authenticity.
|
||||
Imported claims and source catalog status labels remain unverified research leads.
|
||||
Four primary source pages have been captured. A reviewed two-profile price comparison
|
||||
is retained, but requester access and exact benchmark results are not evidenced.
|
||||
The initial real comparison therefore withholds Frontier/Efficient/Budget winners.
|
||||
Other imported claims remain unverified research leads. Real task outcomes are absent;
|
||||
full ranking and calibration behavior are demonstrated with isolated synthetic fixtures.
|
||||
|
||||
The advisor is manually operated. No calibrated success predictor, provider adapter,
|
||||
automated benchmark selector, live price feed, API server, PostgreSQL deployment,
|
||||
scheduled worker, or production UI exists yet. These belong to IR-WP-0003.
|
||||
No live model recommendation is warranted by the synthetic demo.
|
||||
The implementation is a local CLI and curator workflow, not a hosted service or UI.
|
||||
It does not execute model calls, learn cross-task calibration, extract trustworthy
|
||||
facts automatically, optimize decomposition, or estimate empirical cost/runtime
|
||||
quantiles. Published costs are explicit workload scenarios. Catalog expansion and
|
||||
actual outcome collection continue in IR-WP-0004.
|
||||
|
||||
Statehub holds work projections, not research evidence. Artifact hashes prove local
|
||||
integrity, not truth. Scheduler operation depends on this workstation and its user
|
||||
service manager; publication still requires editorial synthesis/review.
|
||||
|
|
|
|||
185
catalog/reviewed-2026-09-13.json
Normal file
185
catalog/reviewed-2026-09-13.json
Normal file
|
|
@ -0,0 +1,185 @@
|
|||
{
|
||||
"synthetic": false,
|
||||
"benchmarks": [
|
||||
{
|
||||
"id": "swe-bench-verified",
|
||||
"version": "pending-verification",
|
||||
"constructs": [
|
||||
"coding"
|
||||
],
|
||||
"health": {
|
||||
"ecological_validity": 0.5,
|
||||
"measurement_reliability": 0.5,
|
||||
"discriminative_power": 0.5,
|
||||
"freshness": 0.5,
|
||||
"execution_profile_similarity": 0.5,
|
||||
"reproducibility": 0.5,
|
||||
"contamination_resistance": 0.5
|
||||
},
|
||||
"scale": {
|
||||
"minimum": 0,
|
||||
"maximum": 100
|
||||
},
|
||||
"limitations": "Candidate instrument only. Exact benchmark version, health and profile-matched result need review. Health values are neutral curator placeholders."
|
||||
},
|
||||
{
|
||||
"id": "terminal-bench",
|
||||
"version": "pending-verification",
|
||||
"constructs": [
|
||||
"coding",
|
||||
"tool_use"
|
||||
],
|
||||
"health": {
|
||||
"ecological_validity": 0.5,
|
||||
"measurement_reliability": 0.5,
|
||||
"discriminative_power": 0.5,
|
||||
"freshness": 0.5,
|
||||
"execution_profile_similarity": 0.5,
|
||||
"reproducibility": 0.5,
|
||||
"contamination_resistance": 0.5
|
||||
},
|
||||
"scale": {
|
||||
"minimum": 0,
|
||||
"maximum": 100
|
||||
},
|
||||
"limitations": "Candidate instrument only. Exact benchmark version, health and profile-matched result need review. Health values are neutral curator placeholders."
|
||||
},
|
||||
{
|
||||
"id": "bfcl",
|
||||
"version": "pending-verification",
|
||||
"constructs": [
|
||||
"tool_use"
|
||||
],
|
||||
"health": {
|
||||
"ecological_validity": 0.5,
|
||||
"measurement_reliability": 0.5,
|
||||
"discriminative_power": 0.5,
|
||||
"freshness": 0.5,
|
||||
"execution_profile_similarity": 0.5,
|
||||
"reproducibility": 0.5,
|
||||
"contamination_resistance": 0.5
|
||||
},
|
||||
"scale": {
|
||||
"minimum": 0,
|
||||
"maximum": 100
|
||||
},
|
||||
"limitations": "Candidate instrument only. Exact benchmark version, health and profile-matched result need review. Health values are neutral curator placeholders."
|
||||
},
|
||||
{
|
||||
"id": "browsecomp",
|
||||
"version": "pending-verification",
|
||||
"constructs": [
|
||||
"web_research"
|
||||
],
|
||||
"health": {
|
||||
"ecological_validity": 0.5,
|
||||
"measurement_reliability": 0.5,
|
||||
"discriminative_power": 0.5,
|
||||
"freshness": 0.5,
|
||||
"execution_profile_similarity": 0.5,
|
||||
"reproducibility": 0.5,
|
||||
"contamination_resistance": 0.5
|
||||
},
|
||||
"scale": {
|
||||
"minimum": 0,
|
||||
"maximum": 100
|
||||
},
|
||||
"limitations": "Candidate instrument only. Exact benchmark version, health and profile-matched result need review. Health values are neutral curator placeholders."
|
||||
},
|
||||
{
|
||||
"id": "scicode",
|
||||
"version": "pending-verification",
|
||||
"constructs": [
|
||||
"coding",
|
||||
"reasoning"
|
||||
],
|
||||
"health": {
|
||||
"ecological_validity": 0.5,
|
||||
"measurement_reliability": 0.5,
|
||||
"discriminative_power": 0.5,
|
||||
"freshness": 0.5,
|
||||
"execution_profile_similarity": 0.5,
|
||||
"reproducibility": 0.5,
|
||||
"contamination_resistance": 0.5
|
||||
},
|
||||
"scale": {
|
||||
"minimum": 0,
|
||||
"maximum": 100
|
||||
},
|
||||
"limitations": "Candidate instrument only. Exact benchmark version, health and profile-matched result need review. Health values are neutral curator placeholders."
|
||||
}
|
||||
],
|
||||
"profiles": [
|
||||
{
|
||||
"id": "claude-sonnet-5-api-default",
|
||||
"model": "claude-sonnet-5",
|
||||
"harness": "direct-api-no-tools",
|
||||
"effort": "provider-default-unspecified",
|
||||
"provider": "Anthropic",
|
||||
"deployment": "Claude Platform/global/standard",
|
||||
"attributes": {
|
||||
"requester_access": false
|
||||
},
|
||||
"availability": {
|
||||
"observed_at": "2026-09-13T20:49:30.828142+00:00",
|
||||
"capture_id": "74b6b3868ee49b78e44df9ab1c088042ad36bb45627edf3079c81a8942370dda",
|
||||
"review_status": "reviewed",
|
||||
"reviewer": "codex",
|
||||
"available": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "claude-haiku-4-5-api-default",
|
||||
"model": "claude-haiku-4-5",
|
||||
"harness": "direct-api-no-tools",
|
||||
"effort": "provider-default-unspecified",
|
||||
"provider": "Anthropic",
|
||||
"deployment": "Claude Platform/global/standard",
|
||||
"attributes": {
|
||||
"requester_access": false
|
||||
},
|
||||
"availability": {
|
||||
"observed_at": "2026-09-13T20:49:30.528597+00:00",
|
||||
"capture_id": "42c09fd59fada0dd8b1e77dde13e5a33d0ae696a8675d39af691941bd8c8028e",
|
||||
"review_status": "reviewed",
|
||||
"reviewer": "codex",
|
||||
"available": true
|
||||
}
|
||||
}
|
||||
],
|
||||
"results": [],
|
||||
"prices": [
|
||||
{
|
||||
"observed_at": "2026-09-13T20:49:30.528597+00:00",
|
||||
"capture_id": "42c09fd59fada0dd8b1e77dde13e5a33d0ae696a8675d39af691941bd8c8028e",
|
||||
"review_status": "reviewed",
|
||||
"reviewer": "codex",
|
||||
"profile_id": "claude-sonnet-5-api-default",
|
||||
"currency": "USD",
|
||||
"valid_from": "2026-09-13",
|
||||
"valid_until": "2026-09-13",
|
||||
"usd_per_million": {
|
||||
"input": 2,
|
||||
"output": 10,
|
||||
"cached_input": 0.2,
|
||||
"cache_write": 2.5
|
||||
}
|
||||
},
|
||||
{
|
||||
"observed_at": "2026-09-13T20:49:30.528597+00:00",
|
||||
"capture_id": "42c09fd59fada0dd8b1e77dde13e5a33d0ae696a8675d39af691941bd8c8028e",
|
||||
"review_status": "reviewed",
|
||||
"reviewer": "codex",
|
||||
"profile_id": "claude-haiku-4-5-api-default",
|
||||
"currency": "USD",
|
||||
"valid_from": "2026-09-13",
|
||||
"valid_until": "2026-09-13",
|
||||
"usd_per_million": {
|
||||
"input": 1,
|
||||
"output": 5,
|
||||
"cached_input": 0.1,
|
||||
"cache_write": 1.25
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
24
catalog/sources.json
Normal file
24
catalog/sources.json
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
{
|
||||
"sources": [
|
||||
{
|
||||
"id": "swe-bench",
|
||||
"publisher": "SWE-bench maintainers",
|
||||
"url": "https://www.swebench.com/"
|
||||
},
|
||||
{
|
||||
"id": "anthropic-pricing",
|
||||
"publisher": "Anthropic",
|
||||
"url": "https://claude.com/pricing"
|
||||
},
|
||||
{
|
||||
"id": "anthropic-sonnet",
|
||||
"publisher": "Anthropic",
|
||||
"url": "https://www.anthropic.com/claude/sonnet"
|
||||
},
|
||||
{
|
||||
"id": "anthropic-sonnet-launch",
|
||||
"publisher": "Anthropic",
|
||||
"url": "https://www.anthropic.com/news/claude-sonnet-5"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -8,3 +8,15 @@ with `supersedes` pointing to the prior observation ID, leaving history intact.
|
|||
|
||||
No model scores or prices from the explorations have been accepted as observations.
|
||||
Synthetic fixtures live in `examples/` and are used only in temporary demo storage.
|
||||
|
||||
## v0.2 stores
|
||||
|
||||
- captures: immutable retrieval metadata; artifacts: exact original source bytes.
|
||||
- ingestion-runs: per-source collection health.
|
||||
- recommendations: frozen task/catalog/policy/outcomes/report snapshots.
|
||||
- outcomes: unique real or explicitly synthetic execution results.
|
||||
- scheduler-runs: durable daily/weekly job receipts.
|
||||
|
||||
Synthetic advisor demonstrations use temporary directories. The retained real
|
||||
comparison explicitly lacks verified benchmark results and requester access.
|
||||
Preserve artifacts with captures; replay verifies their content hashes.
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1 @@
|
|||
{"attempts":1,"bytes":2392125,"changed":false,"content_sha256":"c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433","final_url":"https://www.swebench.com/","media_type":"text/html; charset=utf-8","publisher":"SWE-bench maintainers","retrieved_at":"2026-09-13T20:53:36.147505+00:00","schema_version":1,"source_id":"swe-bench","status":"captured_unreviewed","url":"https://www.swebench.com/"}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"attempts":1,"bytes":1147265,"changed":true,"content_sha256":"b78140937fbc6401068a829ba4b69493c544ece2c6c4e6b547a1e6deab0edd36","final_url":"https://claude.com/pricing","media_type":"text/html; charset=utf-8","publisher":"Anthropic","retrieved_at":"2026-09-13T20:49:30.528597+00:00","schema_version":1,"source_id":"anthropic-pricing","status":"captured_unreviewed","url":"https://claude.com/pricing"}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"attempts":1,"bytes":2392125,"changed":true,"content_sha256":"c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433","final_url":"https://www.swebench.com/","media_type":"text/html; charset=utf-8","publisher":"SWE-bench maintainers","retrieved_at":"2026-09-13T20:49:30.217693+00:00","schema_version":1,"source_id":"swe-bench","status":"captured_unreviewed","url":"https://www.swebench.com/"}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"attempts":1,"bytes":264372,"changed":true,"content_sha256":"87cdcea7dd7bfc4e554d3fe4e8d3377ca8a47110130537414c82cc9ac61fdd80","final_url":"https://www.anthropic.com/news/claude-sonnet-5","media_type":"text/html; charset=utf-8","publisher":"Anthropic","retrieved_at":"2026-09-13T20:52:47.496347+00:00","schema_version":1,"source_id":"anthropic-sonnet-launch","status":"captured_unreviewed","url":"https://www.anthropic.com/news/claude-sonnet-5"}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"attempts":1,"bytes":413640,"changed":true,"content_sha256":"a8961211fa5b55587d3989516eaab6f50cac759e16337ac0f30077bb8ad2a025","final_url":"https://www.anthropic.com/claude/sonnet","media_type":"text/html; charset=utf-8","publisher":"Anthropic","retrieved_at":"2026-09-13T20:49:30.828142+00:00","schema_version":1,"source_id":"anthropic-sonnet","status":"captured_unreviewed","url":"https://www.anthropic.com/claude/sonnet"}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"attempts":1,"bytes":413640,"changed":false,"content_sha256":"a8961211fa5b55587d3989516eaab6f50cac759e16337ac0f30077bb8ad2a025","final_url":"https://www.anthropic.com/claude/sonnet","media_type":"text/html; charset=utf-8","publisher":"Anthropic","retrieved_at":"2026-09-13T20:53:38.068491+00:00","schema_version":1,"source_id":"anthropic-sonnet","status":"captured_unreviewed","url":"https://www.anthropic.com/claude/sonnet"}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"attempts":1,"bytes":264372,"changed":false,"content_sha256":"87cdcea7dd7bfc4e554d3fe4e8d3377ca8a47110130537414c82cc9ac61fdd80","final_url":"https://www.anthropic.com/news/claude-sonnet-5","media_type":"text/html; charset=utf-8","publisher":"Anthropic","retrieved_at":"2026-09-13T20:53:38.392839+00:00","schema_version":1,"source_id":"anthropic-sonnet-launch","status":"captured_unreviewed","url":"https://www.anthropic.com/news/claude-sonnet-5"}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"attempts":1,"bytes":413640,"changed":false,"content_sha256":"a8961211fa5b55587d3989516eaab6f50cac759e16337ac0f30077bb8ad2a025","final_url":"https://www.anthropic.com/claude/sonnet","media_type":"text/html; charset=utf-8","publisher":"Anthropic","retrieved_at":"2026-09-13T20:53:36.898896+00:00","schema_version":1,"source_id":"anthropic-sonnet","status":"captured_unreviewed","url":"https://www.anthropic.com/claude/sonnet"}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"attempts":1,"bytes":2392125,"changed":false,"content_sha256":"c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433","final_url":"https://www.swebench.com/","media_type":"text/html; charset=utf-8","publisher":"SWE-bench maintainers","retrieved_at":"2026-09-13T20:53:37.530744+00:00","schema_version":1,"source_id":"swe-bench","status":"captured_unreviewed","url":"https://www.swebench.com/"}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"attempts":1,"bytes":1147265,"changed":true,"content_sha256":"872698ed9a658d9d5ca28a9ccb387886c695bdbe25d801027ff54baddb96bb71","final_url":"https://claude.com/pricing","media_type":"text/html; charset=utf-8","publisher":"Anthropic","retrieved_at":"2026-09-13T20:53:36.519455+00:00","schema_version":1,"source_id":"anthropic-pricing","status":"captured_unreviewed","url":"https://claude.com/pricing"}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"attempts":1,"bytes":1147341,"changed":true,"content_sha256":"f3463af48e1feeeab0a656cb7e7af15f8fabcde2853f18d0b42f905bfaea7177","final_url":"https://claude.com/pricing","media_type":"text/html; charset=utf-8","publisher":"Anthropic","retrieved_at":"2026-09-13T20:53:37.736979+00:00","schema_version":1,"source_id":"anthropic-pricing","status":"captured_unreviewed","url":"https://claude.com/pricing"}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"attempts":1,"bytes":264372,"changed":false,"content_sha256":"87cdcea7dd7bfc4e554d3fe4e8d3377ca8a47110130537414c82cc9ac61fdd80","final_url":"https://www.anthropic.com/news/claude-sonnet-5","media_type":"text/html; charset=utf-8","publisher":"Anthropic","retrieved_at":"2026-09-13T20:53:37.227224+00:00","schema_version":1,"source_id":"anthropic-sonnet-launch","status":"captured_unreviewed","url":"https://www.anthropic.com/news/claude-sonnet-5"}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"observed_at":"2026-09-13T20:49:31.160878+00:00","ok":true,"sources":[{"capture_id":"4a99589c743803e93ec0595344901836e6f4eb3facc2326e2d2d0109dab1c4e8","changed":true,"ok":true,"source_id":"swe-bench"},{"capture_id":"42c09fd59fada0dd8b1e77dde13e5a33d0ae696a8675d39af691941bd8c8028e","changed":true,"ok":true,"source_id":"anthropic-pricing"},{"capture_id":"74b6b3868ee49b78e44df9ab1c088042ad36bb45627edf3079c81a8942370dda","changed":true,"ok":true,"source_id":"anthropic-sonnet"}]}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"observed_at":"2026-09-13T20:53:37.510668+00:00","ok":true,"sources":[{"capture_id":"01444678f75427d4b4b108a4ea3599b69135885eaaaa3e125eaea2988541bddf","changed":false,"ok":true,"source_id":"swe-bench"},{"capture_id":"d146ef54d2ca96e129b380a3d7dd8d22450c209dacf76ac2095fa88ec69001e5","changed":true,"ok":true,"source_id":"anthropic-pricing"},{"capture_id":"8d3e4d9d673d5159ecaf1e2b1042eaadb6d159405022078598990b62f8c1c061","changed":false,"ok":true,"source_id":"anthropic-sonnet"},{"capture_id":"fcc84b5bb21c41b5d6588daffd41bd6d558a262480491a4e27b88d217d891f40","changed":false,"ok":true,"source_id":"anthropic-sonnet-launch"}]}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"observed_at":"2026-09-13T20:53:38.706600+00:00","ok":true,"sources":[{"capture_id":"ca6e94f8b9b2bb2102601d103fa31e1fc5000615c751358ea2dae795b040c37f","changed":false,"ok":true,"source_id":"swe-bench"},{"capture_id":"db008dc2625cd93d6836cf0848291508e304ea8c5004b9cb51d9b29a39c631a8","changed":true,"ok":true,"source_id":"anthropic-pricing"},{"capture_id":"7609eb261b17e5098147042e0dcc162c26c3d1b13e4226fd31322e0c4dda4f6b","changed":false,"ok":true,"source_id":"anthropic-sonnet"},{"capture_id":"85d278ea018ba72b78502873729b9102a984e4169ab3281945659ca0546736fc","changed":false,"ok":true,"source_id":"anthropic-sonnet-launch"}]}
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
{
|
||||
"id": "31f3a9aaae5476b20819fd469286cb7a198a16e1d27610d288afb1922c584ad1",
|
||||
"record": {
|
||||
"schema_version": 1,
|
||||
"title": "Primary source captured: anthropic-sonnet",
|
||||
"claim": "Primary-source content retained for review; capture is not independent validation of its claims.",
|
||||
"source_url": "https://www.anthropic.com/claude/sonnet",
|
||||
"source_kind": "primary",
|
||||
"content_sha256": "a8961211fa5b55587d3989516eaab6f50cac759e16337ac0f30077bb8ad2a025",
|
||||
"observed_at": "2026-09-13T20:49:30.828142+00:00",
|
||||
"event_date": "2026-09-13",
|
||||
"status": "reported",
|
||||
"observer": "codex",
|
||||
"notes": "Capture ID: 74b6b3868ee49b78e44df9ab1c088042ad36bb45627edf3079c81a8942370dda"
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
{
|
||||
"id": "510f6809bc59adb08295d2025a36c1bb40195d974e1b50563132ab99f1673aba",
|
||||
"record": {
|
||||
"schema_version": 1,
|
||||
"title": "Source update: anthropic-pricing",
|
||||
"claim": "Source content was captured or changed. Extracted claims require editorial review.",
|
||||
"source_url": "https://claude.com/pricing",
|
||||
"source_kind": "primary",
|
||||
"content_sha256": "872698ed9a658d9d5ca28a9ccb387886c695bdbe25d801027ff54baddb96bb71",
|
||||
"observed_at": "2026-09-13T20:53:36.519455+00:00",
|
||||
"event_date": "2026-09-13",
|
||||
"status": "reported",
|
||||
"observer": "intelligence-radar/capture-v0.2",
|
||||
"notes": "Capture ID: d146ef54d2ca96e129b380a3d7dd8d22450c209dacf76ac2095fa88ec69001e5"
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
{
|
||||
"id": "9e50dfd4a4b47541c9fc352c2084b016af052c56b879db69a6e370c1bff0fcfc",
|
||||
"record": {
|
||||
"schema_version": 1,
|
||||
"title": "Source update: anthropic-pricing",
|
||||
"claim": "Source content was captured or changed. Extracted claims require editorial review.",
|
||||
"source_url": "https://claude.com/pricing",
|
||||
"source_kind": "primary",
|
||||
"content_sha256": "f3463af48e1feeeab0a656cb7e7af15f8fabcde2853f18d0b42f905bfaea7177",
|
||||
"observed_at": "2026-09-13T20:53:37.736979+00:00",
|
||||
"event_date": "2026-09-13",
|
||||
"status": "reported",
|
||||
"observer": "intelligence-radar/capture-v0.2",
|
||||
"notes": "Capture ID: db008dc2625cd93d6836cf0848291508e304ea8c5004b9cb51d9b29a39c631a8"
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
{
|
||||
"id": "bb2c9cd9dafa8644b5744df615f8a38defa8debbe03ef50ff620a0478044db5c",
|
||||
"record": {
|
||||
"schema_version": 1,
|
||||
"title": "Primary source captured: anthropic-pricing",
|
||||
"claim": "Primary-source content retained for review; capture is not independent validation of its claims.",
|
||||
"source_url": "https://claude.com/pricing",
|
||||
"source_kind": "primary",
|
||||
"content_sha256": "b78140937fbc6401068a829ba4b69493c544ece2c6c4e6b547a1e6deab0edd36",
|
||||
"observed_at": "2026-09-13T20:49:30.528597+00:00",
|
||||
"event_date": "2026-09-13",
|
||||
"status": "reported",
|
||||
"observer": "codex",
|
||||
"notes": "Capture ID: 42c09fd59fada0dd8b1e77dde13e5a33d0ae696a8675d39af691941bd8c8028e"
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
{
|
||||
"id": "c8fdf59ceb504995bc48c3e6f99d6159bf28e7480e11c7a35daa8c0fc5caae27",
|
||||
"record": {
|
||||
"schema_version": 1,
|
||||
"title": "Primary source captured: swe-bench",
|
||||
"claim": "Primary-source content retained for review; capture is not independent validation of its claims.",
|
||||
"source_url": "https://www.swebench.com/",
|
||||
"source_kind": "primary",
|
||||
"content_sha256": "c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433",
|
||||
"observed_at": "2026-09-13T20:49:30.217693+00:00",
|
||||
"event_date": "2026-09-13",
|
||||
"status": "reported",
|
||||
"observer": "codex",
|
||||
"notes": "Capture ID: 4a99589c743803e93ec0595344901836e6f4eb3facc2326e2d2d0109dab1c4e8"
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
{
|
||||
"id": "eaf5ca53710c20cd9a2935e961209023b0b84058772f6e8a7bb833ac7dadf221",
|
||||
"record": {
|
||||
"schema_version": 1,
|
||||
"title": "Source update: anthropic-sonnet-launch",
|
||||
"claim": "Source content was captured or changed. Extracted claims require editorial review.",
|
||||
"source_url": "https://www.anthropic.com/news/claude-sonnet-5",
|
||||
"source_kind": "primary",
|
||||
"content_sha256": "87cdcea7dd7bfc4e554d3fe4e8d3377ca8a47110130537414c82cc9ac61fdd80",
|
||||
"observed_at": "2026-09-13T20:52:47.496347+00:00",
|
||||
"event_date": "2026-09-13",
|
||||
"status": "reported",
|
||||
"observer": "intelligence-radar/capture-v0.2",
|
||||
"notes": "Capture ID: 5fb451cd647f5b2f235023cb61b4a8f1a45bde150c45d452782453251e9e8d2c"
|
||||
}
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
|
|
@ -0,0 +1 @@
|
|||
{"digest":"digests/weekly/2026-W37.md","digest_status":"existing_preserved","finished_at":"2026-09-13T22:53:36.143928+02:00","job_id":"radar-weekly","ok":true,"period":"2026-W37"}
|
||||
|
|
@ -0,0 +1 @@
|
|||
{"digest":"digests/daily/2026-09-13.md","digest_status":"existing_preserved","finished_at":"2026-09-13T22:53:36.143928+02:00","job_id":"radar-daily","ok":true,"period":"2026-09-13"}
|
||||
47
docs/adr/0002-reproducible-advisor-and-source-capture.md
Normal file
47
docs/adr/0002-reproducible-advisor-and-source-capture.md
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
# ADR 0002 — Reproducible advisor and source capture
|
||||
|
||||
Accepted: 2026-09-13.
|
||||
|
||||
The v0.2 CLI uses standard-library Python and content-addressed JSON. Source bytes
|
||||
are retained by hash. Capture receipts record requested/final URLs, publisher,
|
||||
retrieval time, media type and change detection; capture does not certify a claim.
|
||||
Changed sources create review-needed observations rather than invented extracted facts.
|
||||
|
||||
`advisor-v0.2.json` is the executable policy. It carries forward the baseline
|
||||
selection weights, inclusive .90 Topnotch ratio and risk thresholds. It adds a
|
||||
90-day evidence freshness limit, a .15 construct-overlap redundancy penalty,
|
||||
and a minimum of 20 exact-task/profile outcomes. Earlier YAML remains an immutable
|
||||
design baseline. v0.2 is explicitly provisional and curator-driven.
|
||||
|
||||
Five distinct nonaggregate instruments are selected greedily with construct-based
|
||||
task relevance, curator-assigned health and redundancy penalties. Raw scores use
|
||||
an explicit increasing linear scale per benchmark version. Missing scores are not
|
||||
imputed. Nonlinear metrics, Elo, lower-is-better metrics and composite indices need
|
||||
separate adapters before use. Pending benchmark versions cannot support ranking.
|
||||
|
||||
Eligible means all declared task constraints match evidenced deployment attributes.
|
||||
Minima use `min_` keys; other constraints require exact equality. Production catalog
|
||||
attributes require curator review; available does not imply requester access.
|
||||
Fitness is comparative, not calibrated task success. Rankings cover only evidenced,
|
||||
eligible candidates, not the entire model market.
|
||||
|
||||
Costs are explicit workload scenarios: disjoint input/cache/output buckets, tools,
|
||||
declared attempts and verification. Output includes billable reasoning. Unknown or
|
||||
conflicting prices prevent claims of cheapest. These are not estimated p50/p90s.
|
||||
|
||||
Outcomes bind a recommendation, exact task and exact profile snapshot. Unique run
|
||||
IDs prevent duplicate evidence. Wilson bounds are reported over matching observed
|
||||
trials; Sufficient requires the lower bound above the policy threshold, with at
|
||||
least 20 trials. High-assurance also requires independent-verification assertions.
|
||||
This assumes representative independent trials; it is not a guarantee or a
|
||||
learned cross-task calibration model. Real operational outcome data is still absent.
|
||||
|
||||
Recommendations freeze task, catalog, policy, outcomes and report. Canonical key
|
||||
ordering makes replay deterministic. Source captures and bytes remain referenced
|
||||
by immutable IDs. Replay checks algorithm version and rejects divergence.
|
||||
|
||||
The scheduler checks the latest due daily/weekly slot in Europe/Berlin, catches up
|
||||
one slot after downtime, serializes runs, persists successes, and retries failures
|
||||
on later invocations. Each source fetch allows three bounded attempts and 5 MiB.
|
||||
Existing digest files are preserved. Receipt history distinguishes a new draft
|
||||
from an existing preserved artifact. Editorial synthesis remains a human/agent step.
|
||||
94
docs/data-contracts.md
Normal file
94
docs/data-contracts.md
Normal file
|
|
@ -0,0 +1,94 @@
|
|||
# Data contracts and commands
|
||||
|
||||
Python 3.11+; standard library only. Run commands from the checkout root.
|
||||
|
||||
## Source collection
|
||||
|
||||
```bash
|
||||
python3 -m intelligence_radar ingest catalog/sources.json
|
||||
python3 -m intelligence_radar validate
|
||||
make check
|
||||
```
|
||||
|
||||
Source registry: `sources` array with unique `id`, `publisher`, and HTTPS `url`.
|
||||
The registry is trusted operator configuration; review URLs before adding sources.
|
||||
Captures live in `data/captures/<id>.json`; original bytes are in
|
||||
`data/artifacts/<content_sha256>`. Preserve both for replay. `ingestion-runs` records
|
||||
success/failure per source. HTML changes may be incidental, not newsworthy.
|
||||
|
||||
Review actual source content before marking structured facts `reviewed`. Each
|
||||
result, price and availability item requires `capture_id`, timezone-aware
|
||||
`observed_at`, `review_status: reviewed`, and a named `reviewer`. Timestamps cannot
|
||||
precede the supporting capture date. An evidence ID proves linkage, not correct
|
||||
extraction. The curated initial catalog documents its missing measurement data.
|
||||
|
||||
## Capability Advisor
|
||||
|
||||
```bash
|
||||
python3 -m intelligence_radar advise examples/source-backed-task.json \
|
||||
--catalog catalog/reviewed-2026-09-13.json --as-of 2026-09-13
|
||||
python3 -m intelligence_radar report <recommendation-id>
|
||||
python3 -m intelligence_radar replay <recommendation-id>
|
||||
```
|
||||
|
||||
The source-backed example deliberately withholds winners. Its candidate benchmarks
|
||||
lack verified versions/results and its deployment access has not been established.
|
||||
This is distinct from the synthetic demonstration:
|
||||
|
||||
```bash
|
||||
make demo-advisor
|
||||
```
|
||||
|
||||
Task JSON: objective, acceptance, environment, risk, capability weights (0–1), hard
|
||||
constraints and workload. See examples/task.json. Input and cache token buckets are
|
||||
disjoint; output includes reasoning. Attempts are a declared scenario, never `1/fitness`.
|
||||
|
||||
Catalog JSON: benchmark definitions (id/version/constructs/health/limitations/scale),
|
||||
exact profiles (id/model/harness/effort/provider/deployment/attributes/availability),
|
||||
results (profile_id/benchmark_id/benchmark_version/raw_score/evidence), and prices
|
||||
(profile_id/currency/valid_from/valid_until/usd_per_million/evidence). See
|
||||
examples/advisor-catalog.json for a complete synthetic shape. A synthetic catalog
|
||||
requires explicit `--synthetic` and never mixes with production evidence.
|
||||
|
||||
For benchmark health, provide all non-task-relevance weights named in the active
|
||||
policy. Values are curator judgments, not automated validation. Duplicate IDs are
|
||||
rejected. Missing, stale, future or conflicting results remain unknown. Price
|
||||
validity is date-inclusive and was deliberately limited to the review day in the
|
||||
initial live catalog. New days need new observations.
|
||||
|
||||
## Actual outcomes
|
||||
|
||||
```bash
|
||||
python3 -m intelligence_radar outcome /tmp/outcome.json
|
||||
```
|
||||
|
||||
An outcome contains recommendation_id, profile_id, unique run_id, accepted (boolean),
|
||||
independent_verification (boolean), verifier_evidence (artifact reference or notes),
|
||||
actual_usd, runtime_seconds, and completed_at (timezone-aware timestamp).
|
||||
Outcomes are immutable and run IDs are idempotent; changed duplicate runs are rejected.
|
||||
Keep the verifier artifact where its reference can be resolved. These are operator
|
||||
assertions. No actual provider execution or paid inference happens in this command.
|
||||
|
||||
A subsequent recommendation uses matching outcomes available at its cutoff.
|
||||
A historical replay uses its original frozen set. The current baseline reports
|
||||
95% Wilson intervals, not a learned cross-task success predictor.
|
||||
|
||||
## Scheduled operation
|
||||
|
||||
```bash
|
||||
python3 -m intelligence_radar run-due
|
||||
python3 -m intelligence_radar health
|
||||
systemctl --user status intelligence-radar.timer
|
||||
```
|
||||
|
||||
`ops/schedule.json` configures daily 21:30 and Sunday 20:00 Europe/Berlin.
|
||||
The shipped user timer checks every 15 minutes while this workstation's user service
|
||||
manager is running. It does not promise execution while the workstation is off.
|
||||
Each job is at most once per successful slot; failures remain retryable. Only the
|
||||
latest missed slot is caught up. Runtime state lives in ignored `.runtime/`;
|
||||
`data/scheduler-runs/` retains auditable receipts. Existing digests are preserved,
|
||||
so review the receipt's digest_status before assuming a new draft was produced.
|
||||
|
||||
Install the reviewed units with `make install-timer`; uninstall/disable with
|
||||
`systemctl --user disable --now intelligence-radar.timer`. That command leaves
|
||||
source evidence and publications intact. No external app notifications are sent.
|
||||
28
docs/evidence/source-comparison-2026-09-13.md
Normal file
28
docs/evidence/source-comparison-2026-09-13.md
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
# Capability comparison — 2026-09-13
|
||||
|
||||
Source-backed comparison; evidence gaps remain explicit.
|
||||
|
||||
Recommendation: `024a468396eb7d73840df04271cff4c42cd14ced1b249810c399b5a565dea5f1`
|
||||
Policy: `advisor-v0.2`
|
||||
|
||||
| Profile | Eligible | Fitness proxy | Scenario USD | Outcome trials |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| claude-sonnet-5-api-default | False | unknown | 0.0700 | 0 |
|
||||
| claude-haiku-4-5-api-default | False | unknown | 0.0500 | 0 |
|
||||
|
||||
Frontier: undetermined
|
||||
Efficient: undetermined
|
||||
Budget: undetermined
|
||||
|
||||
## Evidence gaps
|
||||
|
||||
- Fewer than five relevant distinct measurements; ranking withheld.
|
||||
- No eligible profile has complete benchmark evidence.
|
||||
- no_evidenced_sufficient_model
|
||||
- claude-sonnet-5-api-default: constraint min_context not satisfied; constraint requester_access not satisfied
|
||||
- claude-sonnet-5-api-default: missing scicode, terminal-bench, browsecomp, bfcl, swe-bench-verified
|
||||
- claude-haiku-4-5-api-default: constraint min_context not satisfied; constraint requester_access not satisfied
|
||||
- claude-haiku-4-5-api-default: missing scicode, terminal-bench, browsecomp, bfcl, swe-bench-verified
|
||||
|
||||
Cost is a declared token/tool/retry/verification scenario, not a measured success-cost distribution.
|
||||
Source review does not prove requester access or task success. See saved inputs for capture references.
|
||||
32
docs/evidence/v0.2-validation.md
Normal file
32
docs/evidence/v0.2-validation.md
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
# v0.2 implementation evidence — 2026-09-13
|
||||
|
||||
`make check`: 21 behavior tests cover rank/eligibility, missing or stale evidence,
|
||||
price conflicts, numeric validation, aggregate exclusion, outcome deduplication,
|
||||
empirical bounds, historical replay including JSON ordering, source integrity,
|
||||
retry/size limits, scheduler idempotence and ISO-week handling. Repository checks
|
||||
validate archive hashes, observation envelopes, retained captures and report replay.
|
||||
|
||||
`make demo-advisor`: full synthetic ranking returns the fixture Frontier and
|
||||
Efficient, withholds Budget without outcomes, and reproduces the saved report.
|
||||
The fixture remains isolated from the operational ledger.
|
||||
|
||||
Four primary pages were captured from SWE-bench and Anthropic. Original bytes,
|
||||
retrieval receipts and ingestion results are retained under data/. A source-backed
|
||||
comparison is in source-comparison-2026-09-13.md. It explicitly withholds winners
|
||||
because matching benchmark measurements and requester access are not evidenced.
|
||||
|
||||
The initial price search contained stale promotion information. Inspection of
|
||||
[Anthropic's launch-page changelog](https://www.anthropic.com/news/claude-sonnet-5)
|
||||
resolved it: the August update superseded the previously scheduled price increase.
|
||||
The reviewed price observation reflects the current captured
|
||||
[pricing page](https://claude.com/pricing), with a one-day validity policy.
|
||||
|
||||
`run-due` completed both 2026-09-13 daily and 2026-W37 weekly jobs successfully.
|
||||
Their receipts say `existing_preserved`, accurately distinguishing the imported
|
||||
historical draft from a newly generated publication. A repeat invocation performed
|
||||
no completed jobs. No external app automation or notifications were used.
|
||||
|
||||
The user timer was installed on bnt-lap001. `systemctl --user` reports timer
|
||||
ActiveState=active, SubState=waiting; an explicit service invocation returned
|
||||
Result=success and ExecMainStatus=0. The timer checks every 15 minutes while the
|
||||
workstation user manager runs; it is not a continuously available hosted worker.
|
||||
323
examples/advisor-catalog.json
Normal file
323
examples/advisor-catalog.json
Normal file
|
|
@ -0,0 +1,323 @@
|
|||
{
|
||||
"synthetic": true,
|
||||
"benchmarks": [
|
||||
{
|
||||
"id": "fixture-1",
|
||||
"version": "1",
|
||||
"constructs": [
|
||||
"coding"
|
||||
],
|
||||
"health": {
|
||||
"ecological_validity": 0.8,
|
||||
"measurement_reliability": 0.8,
|
||||
"discriminative_power": 0.8,
|
||||
"freshness": 0.8,
|
||||
"execution_profile_similarity": 0.8,
|
||||
"reproducibility": 0.8,
|
||||
"contamination_resistance": 0.8
|
||||
},
|
||||
"limitations": "Synthetic instrument for behavior tests only.",
|
||||
"scale": {
|
||||
"minimum": 0,
|
||||
"maximum": 100
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "fixture-2",
|
||||
"version": "1",
|
||||
"constructs": [
|
||||
"tool_use"
|
||||
],
|
||||
"health": {
|
||||
"ecological_validity": 0.8,
|
||||
"measurement_reliability": 0.8,
|
||||
"discriminative_power": 0.8,
|
||||
"freshness": 0.8,
|
||||
"execution_profile_similarity": 0.8,
|
||||
"reproducibility": 0.8,
|
||||
"contamination_resistance": 0.8
|
||||
},
|
||||
"limitations": "Synthetic instrument for behavior tests only.",
|
||||
"scale": {
|
||||
"minimum": 0,
|
||||
"maximum": 100
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "fixture-3",
|
||||
"version": "1",
|
||||
"constructs": [
|
||||
"reasoning"
|
||||
],
|
||||
"health": {
|
||||
"ecological_validity": 0.8,
|
||||
"measurement_reliability": 0.8,
|
||||
"discriminative_power": 0.8,
|
||||
"freshness": 0.8,
|
||||
"execution_profile_similarity": 0.8,
|
||||
"reproducibility": 0.8,
|
||||
"contamination_resistance": 0.8
|
||||
},
|
||||
"limitations": "Synthetic instrument for behavior tests only.",
|
||||
"scale": {
|
||||
"minimum": 0,
|
||||
"maximum": 100
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "fixture-4",
|
||||
"version": "1",
|
||||
"constructs": [
|
||||
"web_research"
|
||||
],
|
||||
"health": {
|
||||
"ecological_validity": 0.8,
|
||||
"measurement_reliability": 0.8,
|
||||
"discriminative_power": 0.8,
|
||||
"freshness": 0.8,
|
||||
"execution_profile_similarity": 0.8,
|
||||
"reproducibility": 0.8,
|
||||
"contamination_resistance": 0.8
|
||||
},
|
||||
"limitations": "Synthetic instrument for behavior tests only.",
|
||||
"scale": {
|
||||
"minimum": 0,
|
||||
"maximum": 100
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "fixture-5",
|
||||
"version": "1",
|
||||
"constructs": [
|
||||
"artifact_creation"
|
||||
],
|
||||
"health": {
|
||||
"ecological_validity": 0.8,
|
||||
"measurement_reliability": 0.8,
|
||||
"discriminative_power": 0.8,
|
||||
"freshness": 0.8,
|
||||
"execution_profile_similarity": 0.8,
|
||||
"reproducibility": 0.8,
|
||||
"contamination_resistance": 0.8
|
||||
},
|
||||
"limitations": "Synthetic instrument for behavior tests only.",
|
||||
"scale": {
|
||||
"minimum": 0,
|
||||
"maximum": 100
|
||||
}
|
||||
}
|
||||
],
|
||||
"profiles": [
|
||||
{
|
||||
"id": "frontier-fixture",
|
||||
"model": "frontier-fixture",
|
||||
"harness": "fixture-harness-v1",
|
||||
"effort": "default",
|
||||
"provider": "fixture",
|
||||
"deployment": "fixture-local",
|
||||
"attributes": {
|
||||
"requester_access": true,
|
||||
"min_context": 200000
|
||||
},
|
||||
"availability": {
|
||||
"evidence_kind": "synthetic",
|
||||
"observed_at": "2026-09-13T10:00:00Z",
|
||||
"available": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "efficient-fixture",
|
||||
"model": "efficient-fixture",
|
||||
"harness": "fixture-harness-v1",
|
||||
"effort": "default",
|
||||
"provider": "fixture",
|
||||
"deployment": "fixture-local",
|
||||
"attributes": {
|
||||
"requester_access": true,
|
||||
"min_context": 200000
|
||||
},
|
||||
"availability": {
|
||||
"evidence_kind": "synthetic",
|
||||
"observed_at": "2026-09-13T10:00:00Z",
|
||||
"available": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "budget-fixture",
|
||||
"model": "budget-fixture",
|
||||
"harness": "fixture-harness-v1",
|
||||
"effort": "default",
|
||||
"provider": "fixture",
|
||||
"deployment": "fixture-local",
|
||||
"attributes": {
|
||||
"requester_access": true,
|
||||
"min_context": 200000
|
||||
},
|
||||
"availability": {
|
||||
"evidence_kind": "synthetic",
|
||||
"observed_at": "2026-09-13T10:00:00Z",
|
||||
"available": true
|
||||
}
|
||||
}
|
||||
],
|
||||
"results": [
|
||||
{
|
||||
"evidence_kind": "synthetic",
|
||||
"observed_at": "2026-09-13T10:00:00Z",
|
||||
"profile_id": "frontier-fixture",
|
||||
"benchmark_id": "fixture-1",
|
||||
"benchmark_version": "1",
|
||||
"raw_score": 90
|
||||
},
|
||||
{
|
||||
"evidence_kind": "synthetic",
|
||||
"observed_at": "2026-09-13T10:00:00Z",
|
||||
"profile_id": "frontier-fixture",
|
||||
"benchmark_id": "fixture-2",
|
||||
"benchmark_version": "1",
|
||||
"raw_score": 90
|
||||
},
|
||||
{
|
||||
"evidence_kind": "synthetic",
|
||||
"observed_at": "2026-09-13T10:00:00Z",
|
||||
"profile_id": "frontier-fixture",
|
||||
"benchmark_id": "fixture-3",
|
||||
"benchmark_version": "1",
|
||||
"raw_score": 90
|
||||
},
|
||||
{
|
||||
"evidence_kind": "synthetic",
|
||||
"observed_at": "2026-09-13T10:00:00Z",
|
||||
"profile_id": "frontier-fixture",
|
||||
"benchmark_id": "fixture-4",
|
||||
"benchmark_version": "1",
|
||||
"raw_score": 90
|
||||
},
|
||||
{
|
||||
"evidence_kind": "synthetic",
|
||||
"observed_at": "2026-09-13T10:00:00Z",
|
||||
"profile_id": "frontier-fixture",
|
||||
"benchmark_id": "fixture-5",
|
||||
"benchmark_version": "1",
|
||||
"raw_score": 90
|
||||
},
|
||||
{
|
||||
"evidence_kind": "synthetic",
|
||||
"observed_at": "2026-09-13T10:00:00Z",
|
||||
"profile_id": "efficient-fixture",
|
||||
"benchmark_id": "fixture-1",
|
||||
"benchmark_version": "1",
|
||||
"raw_score": 85
|
||||
},
|
||||
{
|
||||
"evidence_kind": "synthetic",
|
||||
"observed_at": "2026-09-13T10:00:00Z",
|
||||
"profile_id": "efficient-fixture",
|
||||
"benchmark_id": "fixture-2",
|
||||
"benchmark_version": "1",
|
||||
"raw_score": 85
|
||||
},
|
||||
{
|
||||
"evidence_kind": "synthetic",
|
||||
"observed_at": "2026-09-13T10:00:00Z",
|
||||
"profile_id": "efficient-fixture",
|
||||
"benchmark_id": "fixture-3",
|
||||
"benchmark_version": "1",
|
||||
"raw_score": 85
|
||||
},
|
||||
{
|
||||
"evidence_kind": "synthetic",
|
||||
"observed_at": "2026-09-13T10:00:00Z",
|
||||
"profile_id": "efficient-fixture",
|
||||
"benchmark_id": "fixture-4",
|
||||
"benchmark_version": "1",
|
||||
"raw_score": 85
|
||||
},
|
||||
{
|
||||
"evidence_kind": "synthetic",
|
||||
"observed_at": "2026-09-13T10:00:00Z",
|
||||
"profile_id": "efficient-fixture",
|
||||
"benchmark_id": "fixture-5",
|
||||
"benchmark_version": "1",
|
||||
"raw_score": 85
|
||||
},
|
||||
{
|
||||
"evidence_kind": "synthetic",
|
||||
"observed_at": "2026-09-13T10:00:00Z",
|
||||
"profile_id": "budget-fixture",
|
||||
"benchmark_id": "fixture-1",
|
||||
"benchmark_version": "1",
|
||||
"raw_score": 70
|
||||
},
|
||||
{
|
||||
"evidence_kind": "synthetic",
|
||||
"observed_at": "2026-09-13T10:00:00Z",
|
||||
"profile_id": "budget-fixture",
|
||||
"benchmark_id": "fixture-2",
|
||||
"benchmark_version": "1",
|
||||
"raw_score": 70
|
||||
},
|
||||
{
|
||||
"evidence_kind": "synthetic",
|
||||
"observed_at": "2026-09-13T10:00:00Z",
|
||||
"profile_id": "budget-fixture",
|
||||
"benchmark_id": "fixture-3",
|
||||
"benchmark_version": "1",
|
||||
"raw_score": 70
|
||||
},
|
||||
{
|
||||
"evidence_kind": "synthetic",
|
||||
"observed_at": "2026-09-13T10:00:00Z",
|
||||
"profile_id": "budget-fixture",
|
||||
"benchmark_id": "fixture-4",
|
||||
"benchmark_version": "1",
|
||||
"raw_score": 70
|
||||
},
|
||||
{
|
||||
"evidence_kind": "synthetic",
|
||||
"observed_at": "2026-09-13T10:00:00Z",
|
||||
"profile_id": "budget-fixture",
|
||||
"benchmark_id": "fixture-5",
|
||||
"benchmark_version": "1",
|
||||
"raw_score": 70
|
||||
}
|
||||
],
|
||||
"prices": [
|
||||
{
|
||||
"evidence_kind": "synthetic",
|
||||
"observed_at": "2026-09-13T10:00:00Z",
|
||||
"profile_id": "frontier-fixture",
|
||||
"currency": "USD",
|
||||
"valid_from": "2026-09-13",
|
||||
"valid_until": "2026-10-01",
|
||||
"usd_per_million": {
|
||||
"input": 10,
|
||||
"output": 50
|
||||
}
|
||||
},
|
||||
{
|
||||
"evidence_kind": "synthetic",
|
||||
"observed_at": "2026-09-13T10:00:00Z",
|
||||
"profile_id": "efficient-fixture",
|
||||
"currency": "USD",
|
||||
"valid_from": "2026-09-13",
|
||||
"valid_until": "2026-10-01",
|
||||
"usd_per_million": {
|
||||
"input": 2,
|
||||
"output": 10
|
||||
}
|
||||
},
|
||||
{
|
||||
"evidence_kind": "synthetic",
|
||||
"observed_at": "2026-09-13T10:00:00Z",
|
||||
"profile_id": "budget-fixture",
|
||||
"currency": "USD",
|
||||
"valid_from": "2026-09-13",
|
||||
"valid_until": "2026-10-01",
|
||||
"usd_per_million": {
|
||||
"input": 0.2,
|
||||
"output": 1.0
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
24
examples/source-backed-task.json
Normal file
24
examples/source-backed-task.json
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
{
|
||||
"objective": "Compare candidate API deployments for repository issue resolution.",
|
||||
"acceptance": "Repository acceptance tests pass and an independent reviewer accepts the change.",
|
||||
"environment": "Proposed Claude Platform API integration; requester access and tool harness not yet evidenced.",
|
||||
"risk": "routine",
|
||||
"capabilities": {
|
||||
"coding": 1,
|
||||
"tool_use": 1,
|
||||
"reasoning": 1,
|
||||
"web_research": 1,
|
||||
"artifact_creation": 1
|
||||
},
|
||||
"constraints": {
|
||||
"requester_access": true,
|
||||
"min_context": 32000
|
||||
},
|
||||
"workload": {
|
||||
"input": 10000,
|
||||
"output": 2000,
|
||||
"tool_usd": 0.01,
|
||||
"attempts": 1,
|
||||
"verification_usd": 0.02
|
||||
}
|
||||
}
|
||||
24
examples/task.json
Normal file
24
examples/task.json
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
{
|
||||
"objective": "Resolve a representative repository issue with a sourced change explanation.",
|
||||
"acceptance": "Repository acceptance tests pass and an independent reviewer accepts the change.",
|
||||
"environment": "Fixture sandbox with shell and browser",
|
||||
"risk": "routine",
|
||||
"capabilities": {
|
||||
"coding": 1,
|
||||
"tool_use": 1,
|
||||
"reasoning": 1,
|
||||
"web_research": 1,
|
||||
"artifact_creation": 1
|
||||
},
|
||||
"constraints": {
|
||||
"requester_access": true,
|
||||
"min_context": 32000
|
||||
},
|
||||
"workload": {
|
||||
"input": 10000,
|
||||
"output": 2000,
|
||||
"tool_usd": 0.01,
|
||||
"attempts": 1,
|
||||
"verification_usd": 0.02
|
||||
}
|
||||
}
|
||||
|
|
@ -80,12 +80,8 @@ def record(root, value):
|
|||
path = root / 'data/observations' / f'{ident}.json'
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
envelope = {'id': ident, 'record': value}
|
||||
try:
|
||||
with path.open('x') as stream:
|
||||
stream.write(json.dumps(envelope, indent=2, ensure_ascii=False) + '\n')
|
||||
except FileExistsError:
|
||||
if json.loads(path.read_text()) != envelope:
|
||||
raise ValueError('existing observation differs; refusing overwrite') from None
|
||||
from .store import atomic
|
||||
atomic(path, (json.dumps(envelope, indent=2, ensure_ascii=False) + '\n').encode(), immutable=True)
|
||||
return ident
|
||||
|
||||
|
||||
|
|
@ -141,6 +137,14 @@ def check(root):
|
|||
raise ValueError('archive path escapes history')
|
||||
if hashlib.sha256(path.read_bytes()).hexdigest() != item['sha256']:
|
||||
raise ValueError(f'archive checksum mismatch: {path}')
|
||||
from .ingest import check_artifacts
|
||||
from .store import all_records
|
||||
check_artifacts(root)
|
||||
for kind in ('recommendations', 'outcomes', 'ingestion-runs', 'scheduler-runs'):
|
||||
all_records(root, kind)
|
||||
from .advisor import replay
|
||||
for ident, _ in all_records(root, 'recommendations'):
|
||||
replay(root, ident)
|
||||
return len(ledger(root))
|
||||
|
||||
|
||||
|
|
@ -154,9 +158,54 @@ def main(argv=None):
|
|||
add.add_argument('input', type=Path)
|
||||
render = commands.add_parser('digest', help='create a daily or ISO-week draft, refusing overwrite')
|
||||
render.add_argument('period')
|
||||
ingest_command = commands.add_parser('ingest', help='capture registered primary-source artifacts')
|
||||
ingest_command.add_argument('registry', type=Path)
|
||||
advisor_command = commands.add_parser('advise', help='save a reproducible task-specific recommendation')
|
||||
advisor_command.add_argument('task', type=Path)
|
||||
advisor_command.add_argument('--catalog', type=Path, required=True)
|
||||
advisor_command.add_argument('--policy', type=Path, default=Path('policies/advisor-v0.2.json'))
|
||||
advisor_command.add_argument('--as-of', required=True)
|
||||
advisor_command.add_argument('--synthetic', action='store_true')
|
||||
report_command = commands.add_parser('report', help='render a saved comparison as Markdown')
|
||||
report_command.add_argument('id')
|
||||
replay_command = commands.add_parser('replay', help='verify a saved recommendation using its frozen inputs')
|
||||
replay_command.add_argument('id')
|
||||
outcome_command = commands.add_parser('outcome', help='record an actual task execution and acceptance result')
|
||||
outcome_command.add_argument('input', type=Path)
|
||||
schedule_command = commands.add_parser('run-due', help='run latest due daily/weekly jobs; safe to invoke repeatedly')
|
||||
schedule_command.add_argument('--config', type=Path, default=Path('ops/schedule.json'))
|
||||
commands.add_parser('health', help='show durable scheduler health')
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
if args.command == 'record':
|
||||
if args.command == 'ingest':
|
||||
from .ingest import ingest
|
||||
result = ingest(args.root, json.loads(args.registry.read_text()))
|
||||
print(json.dumps(result, indent=2))
|
||||
return 0 if result['ok'] else 1
|
||||
elif args.command == 'advise':
|
||||
from .advisor import recommend
|
||||
ident, report = recommend(args.root, json.loads(args.task.read_text()),
|
||||
json.loads(args.catalog.read_text()), json.loads(args.policy.read_text()),
|
||||
args.as_of, args.synthetic)
|
||||
print(json.dumps({'recommendation_id': ident, **report}, indent=2))
|
||||
elif args.command == 'report':
|
||||
from .advisor import render_report
|
||||
print(render_report(args.root, args.id))
|
||||
elif args.command == 'replay':
|
||||
from .advisor import replay
|
||||
print(json.dumps(replay(args.root, args.id), indent=2))
|
||||
elif args.command == 'outcome':
|
||||
from .advisor import outcome
|
||||
print(outcome(args.root, json.loads(args.input.read_text())))
|
||||
elif args.command == 'run-due':
|
||||
from .scheduler import run_due
|
||||
result = run_due(args.root, json.loads(args.config.read_text()))
|
||||
print(json.dumps(result, indent=2))
|
||||
return 0 if result['ok'] else 1
|
||||
elif args.command == 'health':
|
||||
path = args.root / '.runtime/scheduler.json'
|
||||
print(path.read_text() if path.exists() else '{"status": "not_run"}')
|
||||
elif args.command == 'record':
|
||||
print(record(args.root, json.loads(args.input.read_text())))
|
||||
elif args.command == 'digest':
|
||||
print(draft(args.root, args.period))
|
||||
|
|
|
|||
294
intelligence_radar/advisor.py
Normal file
294
intelligence_radar/advisor.py
Normal file
|
|
@ -0,0 +1,294 @@
|
|||
"""Deterministic task-fit ranking with explicit gaps and empirical outcome bounds."""
|
||||
from datetime import date, datetime
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
|
||||
from .store import all_records, canonical, get, put
|
||||
|
||||
|
||||
def number(value, name, low=0, high=None):
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float)) or not math.isfinite(value):
|
||||
raise ValueError(f'{name} must be a finite number')
|
||||
if value < low or (high is not None and value > high):
|
||||
raise ValueError(f'{name} outside allowed range')
|
||||
return value
|
||||
|
||||
|
||||
def unique(items, label):
|
||||
ids = [x['id'] for x in items]
|
||||
if len(ids) != len(set(ids)):
|
||||
raise ValueError(f'duplicate {label} IDs')
|
||||
|
||||
|
||||
def task_key(task):
|
||||
# Includes acceptance, constraints, environment, workload and capability requirements.
|
||||
return hashlib.sha256(canonical(task)).hexdigest()
|
||||
|
||||
|
||||
def wilson(successes, total, z):
|
||||
p = successes / total
|
||||
center = p + z*z / (2*total)
|
||||
delta = z * math.sqrt(p*(1-p)/total + z*z/(4*total*total))
|
||||
denominator = 1 + z*z/total
|
||||
return [max(0, (center-delta)/denominator), min(1, (center+delta)/denominator)]
|
||||
|
||||
|
||||
def select(task, benchmarks, policy):
|
||||
capabilities = task['capabilities']
|
||||
if not capabilities or sum(capabilities.values()) <= 0:
|
||||
raise ValueError('task requires positive capability weights')
|
||||
for k, v in capabilities.items():
|
||||
number(v, k, high=1)
|
||||
weights = policy['selection_weights']
|
||||
if not math.isclose(sum(weights.values()), 1):
|
||||
raise ValueError('selection policy weights must sum to one')
|
||||
candidates = []
|
||||
for b in benchmarks:
|
||||
if b.get('aggregate', False):
|
||||
continue
|
||||
if not b.get('version') or not b.get('limitations'):
|
||||
raise ValueError('benchmarks require version and limitations')
|
||||
relevance = sum(v for k, v in capabilities.items() if k in b['constructs']) / sum(capabilities.values())
|
||||
health = b['health']
|
||||
base = weights['task_relevance'] * relevance
|
||||
for key, weight in weights.items():
|
||||
if key != 'task_relevance':
|
||||
base += weight * number(health[key], key, high=1)
|
||||
base -= number(b.get('known_task_bias', 0), 'known_task_bias', high=1)
|
||||
candidates.append(dict(b, relevance=relevance, selection_score=base))
|
||||
chosen = []
|
||||
while candidates and len(chosen) < policy['selection_count']:
|
||||
def marginal(b):
|
||||
overlap = max(len(set(b['constructs']) & set(s['constructs'])) /
|
||||
max(1, len(set(b['constructs']) | set(s['constructs']))) for s in chosen) if chosen else 0
|
||||
return b['selection_score'] - policy['redundancy_penalty'] * overlap
|
||||
candidates.sort(key=lambda b: (-marginal(b), b['id']))
|
||||
picked = candidates.pop(0)
|
||||
picked['marginal_score'] = marginal(picked)
|
||||
chosen.append(picked)
|
||||
total = sum(b['relevance'] for b in chosen)
|
||||
for b in chosen:
|
||||
b['weight'] = b['relevance'] / total if total else 0
|
||||
return chosen
|
||||
|
||||
|
||||
def evidence_ok(root, item, cutoff, max_age, synthetic):
|
||||
observed = datetime.fromisoformat(item['observed_at'].replace('Z', '+00:00'))
|
||||
if observed.tzinfo is None:
|
||||
raise ValueError('evidence timestamps require timezone')
|
||||
age = (cutoff - observed.date()).days
|
||||
if age < 0 or age > max_age:
|
||||
return False
|
||||
if synthetic:
|
||||
return item.get('evidence_kind') == 'synthetic'
|
||||
if not item.get('capture_id'):
|
||||
return False
|
||||
capture = get(root, 'captures', item['capture_id'])
|
||||
if not datetime.fromisoformat(capture['retrieved_at']).date() <= observed.date() <= cutoff:
|
||||
return False
|
||||
artifact = Path(root) / 'data/artifacts' / capture['content_sha256']
|
||||
if hashlib.sha256(artifact.read_bytes()).hexdigest() != capture['content_sha256']:
|
||||
raise ValueError('capture artifact integrity mismatch')
|
||||
return item.get('review_status') == 'reviewed' and bool(item.get('reviewer'))
|
||||
|
||||
|
||||
def estimate_cost(task, price):
|
||||
if price is None:
|
||||
return None
|
||||
workload = task['workload']
|
||||
rates = price['usd_per_million']
|
||||
cost = 0
|
||||
for key in ('input', 'cached_input', 'cache_write', 'output'):
|
||||
tokens = number(workload.get(key, 0), key)
|
||||
if tokens and key not in rates:
|
||||
return None
|
||||
cost += tokens * number(rates.get(key, 0), key + ' rate') / 1_000_000
|
||||
# Output includes billable reasoning; uncached/cache buckets are disjoint.
|
||||
cost += number(workload['tool_usd'], 'tool_usd')
|
||||
cost *= number(workload['attempts'], 'attempts', low=1)
|
||||
cost += number(workload['verification_usd'], 'verification_usd')
|
||||
return {'scenario_usd': cost, 'basis': 'declared workload; not a probability-derived estimate',
|
||||
'price_observed_at': price['observed_at']}
|
||||
|
||||
|
||||
def advise(root, task, catalog, policy, as_of, synthetic=False, saved_outcomes=None):
|
||||
task, catalog, policy = (json.loads(canonical(x)) for x in (task, catalog, policy))
|
||||
cutoff = date.fromisoformat(as_of)
|
||||
number(policy['topnotch_ratio'], 'topnotch ratio', high=1)
|
||||
number(policy['redundancy_penalty'], 'redundancy penalty', high=1)
|
||||
number(policy['minimum_outcomes'], 'minimum outcomes', low=1)
|
||||
number(policy['max_evidence_age_days'], 'maximum evidence age')
|
||||
number(policy['wilson_z'], 'Wilson z', low=0.01)
|
||||
for v in policy['thresholds'].values():
|
||||
number(v, 'sufficiency threshold', high=1)
|
||||
if date.fromisoformat(policy['effective_from']) > cutoff:
|
||||
raise ValueError('policy not effective at evidence cutoff')
|
||||
if synthetic != bool(catalog.get('synthetic', False)):
|
||||
raise ValueError('synthetic catalogs require --synthetic and cannot mix with live catalogs')
|
||||
if policy['selection_count'] != 5:
|
||||
raise ValueError('advisor requires five measurement candidates')
|
||||
for key in ('objective', 'acceptance', 'environment'):
|
||||
if not isinstance(task[key], str) or not task[key].strip():
|
||||
raise ValueError(f'task {key} required')
|
||||
if task['risk'] not in policy['thresholds']:
|
||||
raise ValueError('unknown task risk')
|
||||
for collection in ('benchmarks', 'profiles'):
|
||||
unique(catalog[collection], collection)
|
||||
selected = select(task, catalog['benchmarks'], policy)
|
||||
complete_selection = len(selected) == 5 and all(b['relevance'] > 0 and b['version'] != 'pending-verification' for b in selected)
|
||||
gaps = [] if complete_selection else ['Fewer than five relevant distinct measurements; ranking withheld.']
|
||||
max_age = policy['max_evidence_age_days']
|
||||
rows = []
|
||||
outcomes = [r for _, r in (all_records(root, 'outcomes') if saved_outcomes is None else saved_outcomes)]
|
||||
for profile in catalog['profiles']:
|
||||
for field in ('model', 'harness', 'effort', 'provider', 'deployment'):
|
||||
if not profile.get(field):
|
||||
raise ValueError(f'profile {field} required')
|
||||
reasons = []
|
||||
available = profile.get('availability')
|
||||
if not available or not evidence_ok(root, available, cutoff, max_age, synthetic) or available['available'] is not True:
|
||||
reasons.append('availability unknown, stale, or unavailable')
|
||||
for key, required in task['constraints'].items():
|
||||
actual = profile.get('attributes', {}).get(key)
|
||||
if key.startswith('min_'):
|
||||
if actual is None or number(actual, key) < number(required, key):
|
||||
reasons.append(f'constraint {key} not satisfied')
|
||||
elif actual != required:
|
||||
reasons.append(f'constraint {key} not satisfied')
|
||||
scores = []
|
||||
for b in selected:
|
||||
matches = [r for r in catalog['results'] if r['profile_id'] == profile['id'] and
|
||||
r['benchmark_id'] == b['id'] and r['benchmark_version'] == b['version'] and
|
||||
evidence_ok(root, r, cutoff, max_age, synthetic)]
|
||||
if len(matches) != 1:
|
||||
scores.append(None)
|
||||
continue
|
||||
scale = b['scale']
|
||||
lower = number(scale['minimum'], 'scale minimum')
|
||||
upper = number(scale['maximum'], 'scale maximum')
|
||||
if upper <= lower:
|
||||
raise ValueError('benchmark scale maximum must exceed minimum')
|
||||
raw = number(matches[0]['raw_score'], 'raw score', low=lower, high=upper)
|
||||
scores.append((raw-lower)/(upper-lower))
|
||||
fit = sum(b['weight'] * score for b, score in zip(selected, scores)) if complete_selection and all(s is not None for s in scores) else None
|
||||
prices = [p for p in catalog['prices'] if p['profile_id'] == profile['id'] and
|
||||
p['currency'] == 'USD' and p['valid_from'] <= as_of <= p['valid_until'] and
|
||||
evidence_ok(root, p, cutoff, max_age, synthetic)]
|
||||
price = prices[0] if len(prices) == 1 else None
|
||||
cost = estimate_cost(task, price)
|
||||
eligible_outcomes = [o for o in outcomes if o['task_key'] == task_key(task) and
|
||||
o['profile_hash'] == hashlib.sha256(canonical(profile)).hexdigest() and
|
||||
o['completed_at'][:10] <= as_of and o['synthetic'] == synthetic]
|
||||
n = len(eligible_outcomes)
|
||||
interval = wilson(sum(o['accepted'] for o in eligible_outcomes), n, policy['wilson_z']) if n else None
|
||||
sufficient = n >= policy['minimum_outcomes'] and interval[0] >= policy['thresholds'][task['risk']]
|
||||
if task['risk'] == 'high_assurance':
|
||||
sufficient = sufficient and all(o['independent_verification'] for o in eligible_outcomes)
|
||||
rows.append({'id': profile['id'], 'eligible': not reasons, 'excluded_reasons': reasons,
|
||||
'fitness_proxy': fit, 'missing_measurements': [b['id'] for b, s in zip(selected, scores) if s is None],
|
||||
'cost': cost, 'price_status': 'usable' if price else 'missing_stale_or_conflicting',
|
||||
'calibration': {'trials': n, 'success_interval_95': interval},
|
||||
'sufficient': bool(sufficient and not reasons)})
|
||||
ranked = sorted([r for r in rows if r['eligible'] and r['fitness_proxy'] is not None],
|
||||
key=lambda r: (-r['fitness_proxy'], r['id']))
|
||||
frontier = ranked[0]['id'] if ranked else None
|
||||
topnotch = [r for r in ranked if r['fitness_proxy'] >= ranked[0]['fitness_proxy'] * policy['topnotch_ratio']]
|
||||
sufficient = [r for r in rows if r['sufficient']]
|
||||
def cheapest(items):
|
||||
# Missing prices prevent a defensible claim of cheapest in the eligible set.
|
||||
if not items or any(r['cost'] is None for r in items):
|
||||
return None
|
||||
return min(items, key=lambda r: (r['cost']['scenario_usd'], r['id']))['id']
|
||||
if not ranked:
|
||||
gaps.append('No eligible profile has complete benchmark evidence.')
|
||||
if not sufficient:
|
||||
gaps.append('no_evidenced_sufficient_model')
|
||||
return {'schema_version': 1, 'algorithm_version': 'advisor-0.2', 'as_of': as_of, 'synthetic': synthetic,
|
||||
'task_key': task_key(task), 'policy_id': policy['id'], 'selected_benchmarks': selected,
|
||||
'profiles': rows, 'frontier': frontier, 'topnotch': [r['id'] for r in topnotch],
|
||||
'efficient': cheapest(topnotch), 'sufficient': [r['id'] for r in sufficient],
|
||||
'budget': cheapest(sufficient), 'evidence_gaps': gaps,
|
||||
'ranking_scope': 'eligible profiles with complete selected evidence; unknown competitors are not proven inferior',
|
||||
'confidence': 'curator-assigned benchmark health; empirical bounds only for exactly matching task and profile; fitness is a proxy'}
|
||||
|
||||
|
||||
def recommend(root, task, catalog, policy, as_of, synthetic=False):
|
||||
saved = all_records(root, 'outcomes')
|
||||
report = advise(root, task, catalog, policy, as_of, synthetic, saved)
|
||||
snapshot = {'task': task, 'catalog': catalog, 'policy': policy,
|
||||
'outcomes': saved, 'report': report}
|
||||
return put(root, 'recommendations', snapshot), report
|
||||
|
||||
|
||||
def _outcome(root, value):
|
||||
recommendation = get(root, 'recommendations', value['recommendation_id'])
|
||||
task = recommendation['task']
|
||||
profiles = [p for p in recommendation['catalog']['profiles'] if p['id'] == value['profile_id']]
|
||||
if len(profiles) != 1:
|
||||
raise ValueError('outcome profile absent from recommendation')
|
||||
for key in ('accepted', 'independent_verification'):
|
||||
if type(value[key]) is not bool:
|
||||
raise ValueError(f'{key} must be boolean')
|
||||
for key in ('actual_usd', 'runtime_seconds'):
|
||||
number(value[key], key)
|
||||
completed = datetime.fromisoformat(value['completed_at'].replace('Z', '+00:00'))
|
||||
if not completed.tzinfo or completed.date() < date.fromisoformat(recommendation['report']['as_of']):
|
||||
raise ValueError('invalid outcome timestamp')
|
||||
if not value.get('run_id') or not value.get('verifier_evidence'):
|
||||
raise ValueError('run_id and verifier_evidence required')
|
||||
result = dict(value, task_key=task_key(task), synthetic=recommendation['report']['synthetic'],
|
||||
profile_hash=hashlib.sha256(canonical(profiles[0])).hexdigest())
|
||||
for ident, existing in all_records(root, 'outcomes'):
|
||||
if existing['run_id'] == value['run_id']:
|
||||
if existing != result:
|
||||
raise ValueError('run_id already recorded with different outcome')
|
||||
return ident
|
||||
return put(root, 'outcomes', result)
|
||||
|
||||
|
||||
def replay(root, ident):
|
||||
snapshot = get(root, 'recommendations', ident)
|
||||
report = snapshot['report']
|
||||
if report.get('algorithm_version') != 'advisor-0.2':
|
||||
raise ValueError('unsupported historical advisor version')
|
||||
rebuilt = advise(root, snapshot['task'], snapshot['catalog'], snapshot['policy'],
|
||||
report['as_of'], report['synthetic'], snapshot['outcomes'])
|
||||
if rebuilt != report:
|
||||
raise ValueError('recommendation replay mismatch')
|
||||
return rebuilt
|
||||
|
||||
|
||||
def outcome(root, value):
|
||||
import fcntl
|
||||
lockpath = Path(root) / '.runtime/outcomes.lock'
|
||||
lockpath.parent.mkdir(parents=True, exist_ok=True)
|
||||
with lockpath.open('a') as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX)
|
||||
return _outcome(root, value)
|
||||
|
||||
|
||||
def render_report(root, ident):
|
||||
report = get(root, 'recommendations', ident)['report']
|
||||
lines = [f"# Capability comparison — {report['as_of']}", '',
|
||||
'Synthetic demonstration.' if report['synthetic'] else 'Source-backed comparison; evidence gaps remain explicit.', '',
|
||||
f"Recommendation: `{ident}`", f"Policy: `{report['policy_id']}`", '',
|
||||
'| Profile | Eligible | Fitness proxy | Scenario USD | Outcome trials |',
|
||||
'| --- | --- | --- | --- | --- |']
|
||||
for row in report['profiles']:
|
||||
cost = f"{row['cost']['scenario_usd']:.4f}" if row['cost'] else 'unknown'
|
||||
fit = f"{row['fitness_proxy']:.4f}" if row['fitness_proxy'] is not None else 'unknown'
|
||||
lines.append(f"| {row['id']} | {row['eligible']} | {fit} | {cost} | {row['calibration']['trials']} |")
|
||||
lines += ['', f"Frontier: {report['frontier'] or 'undetermined'}",
|
||||
f"Efficient: {report['efficient'] or 'undetermined'}",
|
||||
f"Budget: {report['budget'] or 'undetermined'}", '', '## Evidence gaps', '']
|
||||
lines.extend('- '+gap for gap in report['evidence_gaps'])
|
||||
for row in report['profiles']:
|
||||
if row['excluded_reasons']:
|
||||
lines.append(f"- {row['id']}: " + '; '.join(row['excluded_reasons']))
|
||||
if row['missing_measurements']:
|
||||
lines.append(f"- {row['id']}: missing " + ', '.join(row['missing_measurements']))
|
||||
lines += ['', 'Cost is a declared token/tool/retry/verification scenario, not a measured success-cost distribution.',
|
||||
'Source review does not prove requester access or task success. See saved inputs for capture references.', '']
|
||||
return '\n'.join(lines)
|
||||
90
intelligence_radar/ingest.py
Normal file
90
intelligence_radar/ingest.py
Normal file
|
|
@ -0,0 +1,90 @@
|
|||
"""Bounded public HTTP capture; fetched pages are evidence, not verified claims."""
|
||||
from datetime import datetime, timezone
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import time
|
||||
from urllib.parse import urlparse
|
||||
from urllib.request import Request, urlopen
|
||||
|
||||
from .store import all_records, atomic, put
|
||||
|
||||
MAX_BYTES = 5 * 1024 * 1024
|
||||
|
||||
|
||||
def capture(root, source, opener=urlopen, now=None):
|
||||
url = source['url']
|
||||
parsed = urlparse(url)
|
||||
if parsed.scheme != 'https' or not parsed.hostname or parsed.username or parsed.password:
|
||||
raise ValueError('sources require public HTTPS URLs without credentials')
|
||||
if not source.get('id') or not source.get('publisher'):
|
||||
raise ValueError('source id and publisher required')
|
||||
timestamp = (now or datetime.now(timezone.utc)).isoformat()
|
||||
attempts = 0
|
||||
for attempt in range(3):
|
||||
attempts += 1
|
||||
try:
|
||||
request = Request(url, headers={'User-Agent': 'IntelligenceRadar/0.2 (research source monitor)',
|
||||
'Accept': 'text/html,application/json,application/xml,text/plain'})
|
||||
with opener(request, timeout=20) as response:
|
||||
final_url = response.geturl()
|
||||
if urlparse(final_url).scheme != 'https':
|
||||
raise ValueError('source redirected away from HTTPS')
|
||||
body = response.read(MAX_BYTES + 1)
|
||||
media_type = response.headers.get('Content-Type', 'application/octet-stream')
|
||||
break
|
||||
except OSError:
|
||||
if attempt == 2:
|
||||
raise
|
||||
time.sleep(0.25 * (2 ** attempt))
|
||||
if len(body) > MAX_BYTES:
|
||||
raise ValueError('source exceeds 5 MiB capture limit')
|
||||
content_hash = hashlib.sha256(body).hexdigest()
|
||||
artifact = Path(root) / 'data/artifacts' / content_hash
|
||||
atomic(artifact, body, immutable=True)
|
||||
previous = [r for _, r in all_records(root, 'captures') if r['source_id'] == source['id']]
|
||||
previous.sort(key=lambda r: r['retrieved_at'])
|
||||
changed = not previous or previous[-1]['content_sha256'] != content_hash
|
||||
receipt = {'schema_version': 1, 'source_id': source['id'], 'publisher': source['publisher'],
|
||||
'url': url, 'final_url': final_url, 'retrieved_at': timestamp,
|
||||
'content_sha256': content_hash, 'media_type': media_type,
|
||||
'bytes': len(body), 'changed': changed, 'attempts': attempts,
|
||||
'status': 'captured_unreviewed'}
|
||||
ident = put(root, 'captures', receipt)
|
||||
if changed:
|
||||
from .__main__ import record
|
||||
record(root, {'schema_version': 1, 'title': f"Source update: {source['id']}",
|
||||
'claim': 'Source content was captured or changed. Extracted claims require editorial review.',
|
||||
'source_url': final_url, 'source_kind': 'primary',
|
||||
'content_sha256': content_hash, 'observed_at': timestamp,
|
||||
'event_date': timestamp[:10], 'status': 'reported',
|
||||
'observer': 'intelligence-radar/capture-v0.2', 'notes': f'Capture ID: {ident}'})
|
||||
return ident, receipt
|
||||
|
||||
|
||||
def ingest(root, registry):
|
||||
sources = registry['sources']
|
||||
ids = [s['id'] for s in sources]
|
||||
if len(set(ids)) != len(ids):
|
||||
raise ValueError('duplicate source IDs')
|
||||
results = []
|
||||
for source in sources:
|
||||
try:
|
||||
ident, receipt = capture(root, source)
|
||||
results.append({'source_id': source['id'], 'capture_id': ident,
|
||||
'ok': True, 'changed': receipt['changed']})
|
||||
except (ValueError, OSError) as exc:
|
||||
results.append({'source_id': source['id'], 'ok': False, 'error': str(exc)})
|
||||
run = {'observed_at': datetime.now(timezone.utc).isoformat(), 'sources': results,
|
||||
'ok': all(r['ok'] for r in results)}
|
||||
put(root, 'ingestion-runs', run)
|
||||
return run
|
||||
|
||||
|
||||
def check_artifacts(root):
|
||||
captures = all_records(root, 'captures')
|
||||
for _, value in captures:
|
||||
path = Path(root) / 'data/artifacts' / value['content_sha256']
|
||||
if hashlib.sha256(path.read_bytes()).hexdigest() != value['content_sha256']:
|
||||
raise ValueError(f'artifact integrity mismatch: {path}')
|
||||
return len(captures)
|
||||
74
intelligence_radar/scheduler.py
Normal file
74
intelligence_radar/scheduler.py
Normal file
|
|
@ -0,0 +1,74 @@
|
|||
"""Idempotent daily/weekly source capture and digest jobs, with durable run health."""
|
||||
from datetime import datetime, timedelta
|
||||
import fcntl
|
||||
import json
|
||||
from pathlib import Path
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
from .ingest import ingest
|
||||
from .store import atomic, put
|
||||
|
||||
|
||||
def run_due(root, config, now=None, runner=ingest):
|
||||
from .__main__ import draft
|
||||
root = Path(root)
|
||||
zone = ZoneInfo(config['timezone'])
|
||||
now = now or datetime.now(zone)
|
||||
if now.tzinfo is None:
|
||||
raise ValueError('scheduler time requires a timezone')
|
||||
local = now.astimezone(zone)
|
||||
state_dir = root / '.runtime'
|
||||
state_dir.mkdir(parents=True, exist_ok=True)
|
||||
with (state_dir / 'scheduler.lock').open('a') as lock:
|
||||
try:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError:
|
||||
return {'ok': True, 'status': 'already_running', 'jobs': []}
|
||||
state_path = state_dir / 'scheduler.json'
|
||||
state = json.loads(state_path.read_text()) if state_path.exists() else {}
|
||||
results = []
|
||||
for job in config['jobs']:
|
||||
hour, minute = (int(x) for x in job['at'].split(':'))
|
||||
if not (0 <= hour < 24 and 0 <= minute < 60):
|
||||
raise ValueError('invalid schedule time')
|
||||
due = local.replace(hour=hour, minute=minute, second=0, microsecond=0)
|
||||
if job['cadence'] == 'daily':
|
||||
if due > local:
|
||||
due -= timedelta(days=1)
|
||||
period = due.date().isoformat()
|
||||
elif job['cadence'] == 'weekly':
|
||||
due -= timedelta(days=(due.weekday()-job['weekday']) % 7)
|
||||
if due > local:
|
||||
due -= timedelta(days=7)
|
||||
year, week, _ = due.isocalendar()
|
||||
period = f'{year}-W{week:02}'
|
||||
else:
|
||||
raise ValueError('unsupported cadence')
|
||||
key = job['id'] + ':' + period
|
||||
if state.get(key, {}).get('ok'):
|
||||
continue
|
||||
try:
|
||||
registry_path = (root / job['sources']).resolve()
|
||||
if not registry_path.is_relative_to(root.resolve()):
|
||||
raise ValueError('source registry escapes repository')
|
||||
registry = json.loads(registry_path.read_text())
|
||||
result = runner(root, registry)
|
||||
if not result['ok']:
|
||||
raise ValueError('one or more sources failed; see ingestion-runs')
|
||||
# Capture-only changes remain unreviewed. Do not fabricate factual events.
|
||||
try:
|
||||
output = draft(root, period)
|
||||
digest_status = 'created'
|
||||
except FileExistsError:
|
||||
output = root / 'digests' / job['cadence'] / f'{period}.md'
|
||||
digest_status = 'existing_preserved'
|
||||
entry = {'ok': True, 'finished_at': local.isoformat(), 'period': period,
|
||||
'digest': str(output.relative_to(root)), 'digest_status': digest_status}
|
||||
except (ValueError, OSError, KeyError) as exc:
|
||||
entry = {'ok': False, 'finished_at': local.isoformat(), 'period': period,
|
||||
'error': str(exc)}
|
||||
state[key] = entry
|
||||
put(root, 'scheduler-runs', {'job_id': job['id'], **entry})
|
||||
atomic(state_path, json.dumps(state, indent=2).encode() + b'\n')
|
||||
results.append({'job_id': job['id'], **entry})
|
||||
return {'ok': all(r['ok'] for r in results), 'status': 'checked', 'jobs': results}
|
||||
53
intelligence_radar/store.py
Normal file
53
intelligence_radar/store.py
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
"""Atomic content-addressed JSON and source artifact storage."""
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
|
||||
|
||||
def canonical(value):
|
||||
return json.dumps(value, sort_keys=True, separators=(',', ':'), ensure_ascii=False,
|
||||
allow_nan=False).encode('utf-8')
|
||||
|
||||
|
||||
def atomic(path, data, immutable=False):
|
||||
path = Path(path)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
fd, name = tempfile.mkstemp(dir=path.parent, prefix='.write-')
|
||||
try:
|
||||
with os.fdopen(fd, 'wb') as stream:
|
||||
stream.write(data)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
if immutable:
|
||||
try:
|
||||
os.link(name, path)
|
||||
except FileExistsError:
|
||||
if path.read_bytes() != data:
|
||||
raise ValueError(f'immutable content mismatch: {path}') from None
|
||||
else:
|
||||
os.replace(name, path)
|
||||
finally:
|
||||
Path(name).unlink(missing_ok=True)
|
||||
|
||||
|
||||
def put(root, kind, value):
|
||||
content = canonical(value)
|
||||
ident = hashlib.sha256(content).hexdigest()
|
||||
atomic(Path(root) / 'data' / kind / f'{ident}.json', content + b'\n', immutable=True)
|
||||
return ident
|
||||
|
||||
|
||||
def get(root, kind, ident):
|
||||
if len(ident) != 64 or any(c not in '0123456789abcdef' for c in ident):
|
||||
raise ValueError('invalid content ID')
|
||||
value = json.loads((Path(root) / 'data' / kind / f'{ident}.json').read_text())
|
||||
if hashlib.sha256(canonical(value)).hexdigest() != ident:
|
||||
raise ValueError(f'integrity mismatch: {kind}/{ident}')
|
||||
return value
|
||||
|
||||
|
||||
def all_records(root, kind):
|
||||
return [(p.stem, get(root, kind, p.stem))
|
||||
for p in sorted((Path(root) / 'data' / kind).glob('*.json'))]
|
||||
|
|
@ -1,8 +1,8 @@
|
|||
# Scheduling status and prompts
|
||||
|
||||
Repository scheduling status: **not configured**. The source Grok archive contains
|
||||
Repository scheduling: configured local daily/weekly runner and user timer units. The source Grok archive contains
|
||||
external task/schedule UUIDs; ownership and execution were not verified. They are
|
||||
historical references only. No app notifications or email have been enabled here.
|
||||
historical references only. The local timer sends no app notifications or email.
|
||||
|
||||
Daily operator prompt: Review the watchlist for primary-source changes since the
|
||||
previous pass. Record evidence with provenance, publication/event and observation
|
||||
|
|
@ -12,4 +12,4 @@ Weekly operator prompt: Generate the ISO-week draft. Synthesize decision-relevan
|
|||
changes, unresolved contradictions, frontier evidence gaps, compendium edit proposals,
|
||||
and benchmark health. Link observation IDs. Do not pad an empty week with invented news.
|
||||
|
||||
IR-WP-0003 tracks source adapters and an explicitly configured scheduler.
|
||||
See docs/data-contracts.md and ops/schedule.json for the implemented runner.
|
||||
|
|
|
|||
|
|
@ -1,10 +1,10 @@
|
|||
# Operating cadence
|
||||
|
||||
Planned editorial targets; manually operated in v0.1. No scheduler is installed.
|
||||
Editorial targets; source capture and draft creation use the v0.2 scheduler. Synthesis and publication require review.
|
||||
|
||||
## Daily — 21:30 Europe/Berlin
|
||||
|
||||
Planned job name: `intelligence-radar-daily`
|
||||
Editorial job name: `intelligence-radar-daily`
|
||||
|
||||
Produce a brief (800–1,400 words) and store the pattern in `digests/daily/YYYY-MM-DD.md`.
|
||||
|
||||
|
|
@ -22,7 +22,7 @@ Ignore rumor without a primary. Tag confidence per item: confirmed / reported /
|
|||
|
||||
## Weekly — Sunday 20:00 Europe/Berlin
|
||||
|
||||
Planned job name: `intelligence-radar-weekly`
|
||||
Editorial job name: `intelligence-radar-weekly`
|
||||
|
||||
Produce `digests/weekly/YYYY-Www.md` (ISO week).
|
||||
|
||||
|
|
|
|||
|
|
@ -38,5 +38,5 @@ Run commands at the repository root with Python 3.11+. No installation is needed
|
|||
workplan files and run `make sync` for Statehub projection.
|
||||
|
||||
`make demo` uses a temporary directory; its synthetic observation never enters the
|
||||
repository ledger. No crawler, scheduled job, network fetch, or paid inference is
|
||||
triggered by these commands. The planned editorial cadence is in CADENCE.md.
|
||||
repository ledger. These v0.1 evidence commands do not trigger paid inference.
|
||||
For v0.2 source capture, automated jobs, advice and outcomes, see [data contracts](../docs/data-contracts.md). The planned editorial cadence is in CADENCE.md.
|
||||
|
|
|
|||
7
ops/schedule.json
Normal file
7
ops/schedule.json
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
{
|
||||
"timezone": "Europe/Berlin",
|
||||
"jobs": [
|
||||
{"id": "radar-daily", "cadence": "daily", "at": "21:30", "sources": "catalog/sources.json"},
|
||||
{"id": "radar-weekly", "cadence": "weekly", "weekday": 6, "at": "20:00", "sources": "catalog/sources.json"}
|
||||
]
|
||||
}
|
||||
8
ops/systemd/intelligence-radar.service
Normal file
8
ops/systemd/intelligence-radar.service
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
[Unit]
|
||||
Description=Intelligence Radar due-job runner
|
||||
After=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
WorkingDirectory=%h/intelligence-radar
|
||||
ExecStart=/usr/bin/python3 -m intelligence_radar run-due
|
||||
10
ops/systemd/intelligence-radar.timer
Normal file
10
ops/systemd/intelligence-radar.timer
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
[Unit]
|
||||
Description=Check Intelligence Radar daily and weekly jobs
|
||||
|
||||
[Timer]
|
||||
OnBootSec=2min
|
||||
OnUnitActiveSec=15min
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
12
policies/advisor-v0.2.json
Normal file
12
policies/advisor-v0.2.json
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
{
|
||||
"id": "advisor-v0.2",
|
||||
"effective_from": "2026-09-13",
|
||||
"selection_count": 5,
|
||||
"selection_weights": {"task_relevance": 0.30, "ecological_validity": 0.15, "measurement_reliability": 0.12, "discriminative_power": 0.12, "freshness": 0.10, "execution_profile_similarity": 0.10, "reproducibility": 0.06, "contamination_resistance": 0.05},
|
||||
"redundancy_penalty": 0.15,
|
||||
"topnotch_ratio": 0.9,
|
||||
"thresholds": {"exploratory": 0.6, "routine": 0.8, "important": 0.9, "high_assurance": 0.97},
|
||||
"max_evidence_age_days": 90,
|
||||
"minimum_outcomes": 20,
|
||||
"wilson_z": 1.96
|
||||
}
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
# Capability registry
|
||||
|
||||
The local evidence workflow is discoverable through indexes/capabilities.yaml.
|
||||
The Capability Advisor remains manual; no runtime API is advertised.
|
||||
The local source-capture, evidence and advisor workflow is discoverable through
|
||||
indexes/capabilities.yaml. Consumption is CLI and Git artifacts; no hosted API is advertised.
|
||||
|
|
|
|||
|
|
@ -1,21 +1,21 @@
|
|||
---
|
||||
id: capability.infotech.intelligence-radar
|
||||
name: Intelligence Radar evidence workflow
|
||||
summary: Local provenance-preserving observation ledger and draft intelligence digests.
|
||||
summary: Local source capture, reproducible capability advice and draft intelligence digests.
|
||||
owner: intelligence-radar
|
||||
status: draft
|
||||
domain: infotech
|
||||
tags: [research, evidence, evaluation]
|
||||
discovery:
|
||||
intent: Support auditable intelligence research and task-specific capability advice.
|
||||
includes: [observation recording, integrity validation, draft digests, manual advisory method]
|
||||
excludes: [live model rankings, calibrated predictor, hosted API, scheduled crawling]
|
||||
includes: [source capture, observation recording, integrity validation, draft digests, reproducible advisor, outcome records]
|
||||
excludes: [complete market rankings, learned cross-task predictor, hosted API]
|
||||
availability:
|
||||
consumption_modes: [cli, informational]
|
||||
current_artifacts: [intelligence_radar/, ops/RUNBOOK.md, task-router/METHOD.md]
|
||||
evidence:
|
||||
documentation: [README.md, SCOPE.md, ops/RUNBOOK.md]
|
||||
tests: [tests/test_workflow.py]
|
||||
tests: [tests/test_workflow.py, tests/test_advisor.py, tests/test_ingestion_scheduler.py]
|
||||
---
|
||||
|
||||
# Intelligence Radar evidence workflow
|
||||
|
|
|
|||
|
|
@ -21,4 +21,4 @@ Use policy IDs `benchmark-selection-v0.1` and `model-ranking-v0.1`.
|
|||
7. Compare optional decomposition with a monolithic baseline including coordination,
|
||||
verification and failure costs. Record actual outcomes for later calibration.
|
||||
|
||||
Use [TEMPLATE](TEMPLATE.md). This is a manual workflow in v0.1, not an API promise.
|
||||
Use [TEMPLATE](TEMPLATE.md). The v0.2 CLI implements selection, eligibility, ranking, scenario costs, outcomes and replay under advisor-v0.2.json. See ../docs/data-contracts.md; decomposition remains manual.
|
||||
|
|
|
|||
95
tests/test_advisor.py
Normal file
95
tests/test_advisor.py
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
import copy
|
||||
import json
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
from intelligence_radar.advisor import advise, outcome, recommend, replay
|
||||
|
||||
REPO = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
class AdvisorTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.root = Path(self.temp.name)
|
||||
self.task = json.loads((REPO/'examples/task.json').read_text())
|
||||
self.catalog = json.loads((REPO/'examples/advisor-catalog.json').read_text())
|
||||
self.policy = json.loads((REPO/'policies/advisor-v0.2.json').read_text())
|
||||
|
||||
def report(self):
|
||||
return advise(self.root, self.task, self.catalog, self.policy, '2026-09-13', True)
|
||||
|
||||
def test_rank_cost_and_no_unearned_sufficiency(self):
|
||||
r = self.report()
|
||||
self.assertEqual(r['frontier'], 'frontier-fixture')
|
||||
self.assertEqual(r['efficient'], 'efficient-fixture')
|
||||
self.assertIsNone(r['budget'])
|
||||
self.assertEqual(len(r['selected_benchmarks']), 5)
|
||||
self.assertAlmostEqual(sum(b['weight'] for b in r['selected_benchmarks']), 1)
|
||||
efficient = next(x for x in r['profiles'] if x['id']=='efficient-fixture')
|
||||
self.assertAlmostEqual(efficient['cost']['scenario_usd'], .07)
|
||||
|
||||
def test_missing_score_never_imputed(self):
|
||||
self.catalog['results'].pop(0)
|
||||
report = self.report()
|
||||
self.assertEqual(report['frontier'], 'efficient-fixture')
|
||||
row = next(x for x in report['profiles'] if x['id']=='frontier-fixture')
|
||||
self.assertIsNone(row['fitness_proxy'])
|
||||
self.assertEqual(len(row['missing_measurements']), 1)
|
||||
|
||||
def test_constraints_remove_highest_scoring_profile(self):
|
||||
self.catalog['profiles'][0]['attributes']['requester_access'] = False
|
||||
self.assertEqual(self.report()['frontier'], 'efficient-fixture')
|
||||
|
||||
def test_missing_or_conflicting_price_prevents_cheapest_claim(self):
|
||||
self.catalog['prices'].pop(0)
|
||||
self.assertIsNone(self.report()['efficient'])
|
||||
self.catalog['prices'].append(copy.deepcopy(self.catalog['prices'][0]))
|
||||
self.assertIsNone(self.report()['efficient'])
|
||||
|
||||
def test_stale_future_and_wrong_version_results(self):
|
||||
for field, value in [('observed_at','2026-09-14T00:00:00Z'),
|
||||
('observed_at','2025-01-01T00:00:00Z'),
|
||||
('benchmark_version','other')]:
|
||||
with self.subTest(field=field,value=value):
|
||||
original=self.catalog['results'][0].copy()
|
||||
self.catalog['results'][0][field]=value
|
||||
self.assertEqual(self.report()['frontier'], 'efficient-fixture')
|
||||
self.catalog['results'][0]=original
|
||||
|
||||
def test_aggregate_and_short_selection_withhold_ranking(self):
|
||||
self.catalog['benchmarks'][0]['aggregate']=True
|
||||
self.assertIsNone(self.report()['frontier'])
|
||||
|
||||
def test_calibration_duplicate_protection_and_frozen_replay(self):
|
||||
ident, initial = recommend(self.root,self.task,self.catalog,self.policy,'2026-09-13',True)
|
||||
value=dict(recommendation_id=ident,profile_id='budget-fixture',run_id='run-0',
|
||||
accepted=True,independent_verification=True,verifier_evidence='synthetic test run',
|
||||
actual_usd=.01,runtime_seconds=1,completed_at='2026-09-13T22:00:00Z')
|
||||
first=outcome(self.root,value)
|
||||
self.assertEqual(outcome(self.root,value),first)
|
||||
with self.assertRaises(ValueError):
|
||||
outcome(self.root,dict(value,accepted=False))
|
||||
for i in range(1,20):
|
||||
outcome(self.root,dict(value,run_id=f'run-{i}'))
|
||||
self.assertEqual(self.report()['budget'],'budget-fixture')
|
||||
self.assertEqual(replay(self.root,ident), initial)
|
||||
self.task['acceptance']+=' Different acceptance test.'
|
||||
self.assertIsNone(self.report()['budget'])
|
||||
|
||||
def test_synthetic_requires_explicit_opt_in(self):
|
||||
with self.assertRaises(ValueError):
|
||||
advise(self.root,self.task,self.catalog,self.policy,'2026-09-13')
|
||||
|
||||
def test_invalid_numeric_inputs_rejected(self):
|
||||
self.catalog['results'][0]['raw_score']=float('nan')
|
||||
with self.assertRaises(ValueError):
|
||||
self.report()
|
||||
|
||||
def test_replay_is_independent_of_json_object_key_order(self):
|
||||
self.task['constraints']['unavailable_attribute'] = True
|
||||
self.task['capabilities']['coding'] = .31
|
||||
ident, initial = recommend(self.root,self.task,self.catalog,self.policy,'2026-09-13',True)
|
||||
self.assertEqual(replay(self.root,ident), initial)
|
||||
67
tests/test_ingestion_scheduler.py
Normal file
67
tests/test_ingestion_scheduler.py
Normal file
|
|
@ -0,0 +1,67 @@
|
|||
from datetime import datetime, timezone
|
||||
import io
|
||||
import json
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from intelligence_radar.ingest import capture, check_artifacts, MAX_BYTES
|
||||
from intelligence_radar.scheduler import run_due
|
||||
from intelligence_radar.store import all_records
|
||||
|
||||
|
||||
class Response(io.BytesIO):
|
||||
headers={'Content-Type':'text/plain'}
|
||||
def geturl(self):
|
||||
return 'https://example.invalid/source'
|
||||
|
||||
|
||||
class IngestionTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp=tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.root=Path(self.temp.name)
|
||||
self.source={'id':'test','publisher':'test','url':'https://example.invalid/source'}
|
||||
|
||||
def test_capture_deduplication_and_integrity(self):
|
||||
first,r=capture(self.root,self.source,opener=lambda *a,**k:Response(b'body'))
|
||||
second,s=capture(self.root,self.source,opener=lambda *a,**k:Response(b'body'))
|
||||
self.assertTrue(r['changed'])
|
||||
self.assertFalse(s['changed'])
|
||||
self.assertEqual(len(list((self.root/'data/artifacts').iterdir())),1)
|
||||
self.assertEqual(check_artifacts(self.root),2)
|
||||
(self.root/'data/artifacts'/r['content_sha256']).write_bytes(b'tampered')
|
||||
with self.assertRaises(ValueError):
|
||||
check_artifacts(self.root)
|
||||
|
||||
def test_retry_and_capture_limit(self):
|
||||
with patch('intelligence_radar.ingest.time.sleep'), patch('urllib.request.urlopen'):
|
||||
count=[0]
|
||||
def fetch(*args,**kwargs):
|
||||
count[0]+=1
|
||||
if count[0]<3:
|
||||
raise OSError('transient')
|
||||
return Response(b'ok')
|
||||
_,r=capture(self.root,self.source,opener=fetch)
|
||||
self.assertEqual(r['attempts'],3)
|
||||
with self.assertRaises(ValueError):
|
||||
capture(self.root,self.source,opener=lambda *a,**k:Response(b'x'*(MAX_BYTES+1)))
|
||||
|
||||
def test_scheduler_retries_failed_job_and_preserves_existing_digest(self):
|
||||
(self.root/'sources.json').write_text('{"sources": []}')
|
||||
config={'timezone':'Europe/Berlin','jobs':[{'id':'daily','cadence':'daily','at':'21:30','sources':'sources.json'}]}
|
||||
now=datetime(2026,9,13,20,tzinfo=timezone.utc)
|
||||
failed=run_due(self.root,config,now,runner=lambda *a:{'ok':False})
|
||||
self.assertFalse(failed['ok'])
|
||||
passed=run_due(self.root,config,now,runner=lambda *a:{'ok':True})
|
||||
self.assertTrue(passed['ok'])
|
||||
self.assertEqual(len(passed['jobs']),1)
|
||||
self.assertEqual(run_due(self.root,config,now,runner=lambda *a:self.fail('already done'))['jobs'],[])
|
||||
self.assertEqual(len(all_records(self.root,'scheduler-runs')),2)
|
||||
|
||||
def test_weekly_iso_boundary_and_dst_zone(self):
|
||||
(self.root/'sources.json').write_text('{"sources": []}')
|
||||
config={'timezone':'Europe/Berlin','jobs':[{'id':'weekly','cadence':'weekly','weekday':6,'at':'20:00','sources':'sources.json'}]}
|
||||
result=run_due(self.root,config,datetime(2026,1,4,19,tzinfo=timezone.utc),runner=lambda *a:{'ok':True})
|
||||
self.assertEqual(result['jobs'][0]['period'],'2026-W01')
|
||||
22
tools/demo_advisor.py
Normal file
22
tools/demo_advisor.py
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
"""Synthetic advisor and frozen-replay demonstration in temporary storage."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
from intelligence_radar.advisor import recommend, replay, render_report
|
||||
|
||||
repo = Path(__file__).resolve().parents[1]
|
||||
with tempfile.TemporaryDirectory(prefix='radar-advisor-demo-') as directory:
|
||||
root = Path(directory)
|
||||
ident, report = recommend(root, json.loads((repo/'examples/task.json').read_text()),
|
||||
json.loads((repo/'examples/advisor-catalog.json').read_text()),
|
||||
json.loads((repo/'policies/advisor-v0.2.json').read_text()),
|
||||
'2026-09-13', True)
|
||||
assert replay(root, ident) == report
|
||||
assert report['frontier'] == 'frontier-fixture'
|
||||
assert report['efficient'] == 'efficient-fixture'
|
||||
assert report['budget'] is None
|
||||
print(render_report(root, ident))
|
||||
print('Advisor demo passed; synthetic records stayed in temporary storage.')
|
||||
|
|
@ -4,7 +4,7 @@ type: workplan
|
|||
title: "Verified sources and calibrated advisory capability"
|
||||
domain: infotech
|
||||
repo: intelligence-radar
|
||||
status: backlog
|
||||
status: finished
|
||||
owner: codex
|
||||
topic_slug: custodian
|
||||
created: "2026-09-13"
|
||||
|
|
@ -18,7 +18,7 @@ state_hub_workstream_id: "66593995-8a1c-5c41-a36e-e4dc38675dde"
|
|||
|
||||
```task
|
||||
id: IR-WP-0003-T01
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "03ea42ad-0eb8-55ea-93a4-582d9310ca86"
|
||||
```
|
||||
|
|
@ -29,7 +29,7 @@ Residual of IR-WP-0002: validate imported benchmark versions, prices and availab
|
|||
|
||||
```task
|
||||
id: IR-WP-0003-T02
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "a4809775-e432-56a0-8434-4b9164cfe012"
|
||||
```
|
||||
|
|
@ -40,10 +40,21 @@ Residual of IR-WP-0002: implement five-instrument selection, eligibility gates,
|
|||
|
||||
```task
|
||||
id: IR-WP-0003-T03
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "123f455e-eb54-5420-91d0-608ea310e756"
|
||||
```
|
||||
|
||||
Residual of IR-WP-0002: add source adapters, deduplication, retrieval artifacts and a configured scheduler with retry/health evidence. Confirm ownership of any external automation before reuse; repository scheduler must be explicit and testable.
|
||||
|
||||
|
||||
## Completion evidence
|
||||
|
||||
Implemented source capture with retained artifacts and reviewed price observations;
|
||||
produced a source-backed comparison with explicit absent benchmark/access evidence.
|
||||
Implemented executable selection, eligibility, ranking, scenario economics,
|
||||
immutable outcomes, Wilson bounds and frozen historical replay. Configured and ran
|
||||
daily/weekly jobs and installed the local user timer; service exited successfully.
|
||||
Validation: 21 tests, isolated advisor demo, retained source integrity and real report
|
||||
replay. See docs/evidence/v0.2-validation.md. Broader measurement coverage, real outcome
|
||||
collection and hosted/editorial expansion are explicit residuals in IR-WP-0004.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,57 @@
|
|||
---
|
||||
id: IR-WP-0004
|
||||
type: workplan
|
||||
title: "Expand verified evidence and calibrate against field outcomes"
|
||||
domain: infotech
|
||||
repo: intelligence-radar
|
||||
status: backlog
|
||||
owner: codex
|
||||
topic_slug: custodian
|
||||
created: "2026-09-13"
|
||||
updated: "2026-09-13"
|
||||
---
|
||||
|
||||
# Expand verified evidence and calibrate against field outcomes
|
||||
|
||||
Residual research and product scope from IR-WP-0003. v0.2 implements the mechanics;
|
||||
this work grows the evidence and operational reach without inventing calibration.
|
||||
|
||||
## Build complete comparable profile evidence
|
||||
|
||||
```task
|
||||
id: IR-WP-0004-T01
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Review exact versions, harnesses, effort and scores for five task-relevant instruments.
|
||||
Expand beyond the two-profile dated price comparison. Preserve conflicting results
|
||||
and independently verify requester access. Acceptance: at least two eligible profiles
|
||||
with complete comparable evidence and a source-backed Frontier/Efficient comparison.
|
||||
All other imported model/benchmark claims remain research leads until reviewed.
|
||||
|
||||
## Collect representative real task outcomes
|
||||
|
||||
```task
|
||||
id: IR-WP-0004-T02
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Run an explicitly budgeted sample of representative tasks with retained verifier
|
||||
artifacts. Record successes and failures, actual cost and runtime. Evaluate selection
|
||||
bias and dependence before treating Wilson bounds as predictive. Develop versioned
|
||||
cross-task calibration and measured cost/runtime intervals only when justified.
|
||||
|
||||
## Extend publication and hosted operation
|
||||
|
||||
```task
|
||||
id: IR-WP-0004-T03
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Add curated source adapters and editorial synthesis, artifact retention/export,
|
||||
reviewed source coverage, and a deployment-independent worker if continuous uptime
|
||||
is required. Add decomposition comparisons against measured monolithic outcomes.
|
||||
Keep draft creation separate from reviewed publication and any notifications.
|
||||
Loading…
Add table
Add a link
Reference in a new issue