95 lines
4.4 KiB
Markdown
95 lines
4.4 KiB
Markdown
|
# Data contracts and commands
|
|||
|
|
|
|||
|
|
Python 3.11+; standard library only. Run commands from the checkout root.
|
|||
|
|
|
|||
|
|
## Source collection
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
python3 -m intelligence_radar ingest catalog/sources.json
|
|||
|
|
python3 -m intelligence_radar validate
|
|||
|
|
make check
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Source registry: `sources` array with unique `id`, `publisher`, and HTTPS `url`.
|
|||
|
|
The registry is trusted operator configuration; review URLs before adding sources.
|
|||
|
|
Captures live in `data/captures/<id>.json`; original bytes are in
|
|||
|
|
`data/artifacts/<content_sha256>`. Preserve both for replay. `ingestion-runs` records
|
|||
|
|
success/failure per source. HTML changes may be incidental, not newsworthy.
|
|||
|
|
|
|||
|
|
Review actual source content before marking structured facts `reviewed`. Each
|
|||
|
|
result, price and availability item requires `capture_id`, timezone-aware
|
|||
|
|
`observed_at`, `review_status: reviewed`, and a named `reviewer`. Timestamps cannot
|
|||
|
|
precede the supporting capture date. An evidence ID proves linkage, not correct
|
|||
|
|
extraction. The curated initial catalog documents its missing measurement data.
|
|||
|
|
|
|||
|
|
## Capability Advisor
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
python3 -m intelligence_radar advise examples/source-backed-task.json \
|
|||
|
|
--catalog catalog/reviewed-2026-09-13.json --as-of 2026-09-13
|
|||
|
|
python3 -m intelligence_radar report <recommendation-id>
|
|||
|
|
python3 -m intelligence_radar replay <recommendation-id>
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The source-backed example deliberately withholds winners. Its candidate benchmarks
|
|||
|
|
lack verified versions/results and its deployment access has not been established.
|
|||
|
|
This is distinct from the synthetic demonstration:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
make demo-advisor
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Task JSON: objective, acceptance, environment, risk, capability weights (0–1), hard
|
|||
|
|
constraints and workload. See examples/task.json. Input and cache token buckets are
|
|||
|
|
disjoint; output includes reasoning. Attempts are a declared scenario, never `1/fitness`.
|
|||
|
|
|
|||
|
|
Catalog JSON: benchmark definitions (id/version/constructs/health/limitations/scale),
|
|||
|
|
exact profiles (id/model/harness/effort/provider/deployment/attributes/availability),
|
|||
|
|
results (profile_id/benchmark_id/benchmark_version/raw_score/evidence), and prices
|
|||
|
|
(profile_id/currency/valid_from/valid_until/usd_per_million/evidence). See
|
|||
|
|
examples/advisor-catalog.json for a complete synthetic shape. A synthetic catalog
|
|||
|
|
requires explicit `--synthetic` and never mixes with production evidence.
|
|||
|
|
|
|||
|
|
For benchmark health, provide all non-task-relevance weights named in the active
|
|||
|
|
policy. Values are curator judgments, not automated validation. Duplicate IDs are
|
|||
|
|
rejected. Missing, stale, future or conflicting results remain unknown. Price
|
|||
|
|
validity is date-inclusive and was deliberately limited to the review day in the
|
|||
|
|
initial live catalog. New days need new observations.
|
|||
|
|
|
|||
|
|
## Actual outcomes
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
python3 -m intelligence_radar outcome /tmp/outcome.json
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
An outcome contains recommendation_id, profile_id, unique run_id, accepted (boolean),
|
|||
|
|
independent_verification (boolean), verifier_evidence (artifact reference or notes),
|
|||
|
|
actual_usd, runtime_seconds, and completed_at (timezone-aware timestamp).
|
|||
|
|
Outcomes are immutable and run IDs are idempotent; changed duplicate runs are rejected.
|
|||
|
|
Keep the verifier artifact where its reference can be resolved. These are operator
|
|||
|
|
assertions. No actual provider execution or paid inference happens in this command.
|
|||
|
|
|
|||
|
|
A subsequent recommendation uses matching outcomes available at its cutoff.
|
|||
|
|
A historical replay uses its original frozen set. The current baseline reports
|
|||
|
|
95% Wilson intervals, not a learned cross-task success predictor.
|
|||
|
|
|
|||
|
|
## Scheduled operation
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
python3 -m intelligence_radar run-due
|
|||
|
|
python3 -m intelligence_radar health
|
|||
|
|
systemctl --user status intelligence-radar.timer
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
`ops/schedule.json` configures daily 21:30 and Sunday 20:00 Europe/Berlin.
|
|||
|
|
The shipped user timer checks every 15 minutes while this workstation's user service
|
|||
|
|
manager is running. It does not promise execution while the workstation is off.
|
|||
|
|
Each job is at most once per successful slot; failures remain retryable. Only the
|
|||
|
|
latest missed slot is caught up. Runtime state lives in ignored `.runtime/`;
|
|||
|
|
`data/scheduler-runs/` retains auditable receipts. Existing digests are preserved,
|
|||
|
|
so review the receipt's digest_status before assuming a new draft was produced.
|
|||
|
|
|
|||
|
|
Install the reviewed units with `make install-timer`; uninstall/disable with
|
|||
|
|
`systemctl --user disable --now intelligence-radar.timer`. That command leaves
|
|||
|
|
source evidence and publications intact. No external app notifications are sent.
|