Build reproducible capability advisor and scheduled source capture
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a09c77-b1a9-7eb3-b501-6cb24b458c97
This commit is contained in:
parent
a1edefd7c5
commit
29ceb77938
68 changed files with 2431 additions and 85 deletions
94
docs/data-contracts.md
Normal file
94
docs/data-contracts.md
Normal file
|
|
@ -0,0 +1,94 @@
|
|||
# Data contracts and commands
|
||||
|
||||
Python 3.11+; standard library only. Run commands from the checkout root.
|
||||
|
||||
## Source collection
|
||||
|
||||
```bash
|
||||
python3 -m intelligence_radar ingest catalog/sources.json
|
||||
python3 -m intelligence_radar validate
|
||||
make check
|
||||
```
|
||||
|
||||
Source registry: `sources` array with unique `id`, `publisher`, and HTTPS `url`.
|
||||
The registry is trusted operator configuration; review URLs before adding sources.
|
||||
Captures live in `data/captures/<id>.json`; original bytes are in
|
||||
`data/artifacts/<content_sha256>`. Preserve both for replay. `ingestion-runs` records
|
||||
success/failure per source. HTML changes may be incidental, not newsworthy.
|
||||
|
||||
Review actual source content before marking structured facts `reviewed`. Each
|
||||
result, price and availability item requires `capture_id`, timezone-aware
|
||||
`observed_at`, `review_status: reviewed`, and a named `reviewer`. Timestamps cannot
|
||||
precede the supporting capture date. An evidence ID proves linkage, not correct
|
||||
extraction. The curated initial catalog documents its missing measurement data.
|
||||
|
||||
## Capability Advisor
|
||||
|
||||
```bash
|
||||
python3 -m intelligence_radar advise examples/source-backed-task.json \
|
||||
--catalog catalog/reviewed-2026-09-13.json --as-of 2026-09-13
|
||||
python3 -m intelligence_radar report <recommendation-id>
|
||||
python3 -m intelligence_radar replay <recommendation-id>
|
||||
```
|
||||
|
||||
The source-backed example deliberately withholds winners. Its candidate benchmarks
|
||||
lack verified versions/results and its deployment access has not been established.
|
||||
This is distinct from the synthetic demonstration:
|
||||
|
||||
```bash
|
||||
make demo-advisor
|
||||
```
|
||||
|
||||
Task JSON: objective, acceptance, environment, risk, capability weights (0–1), hard
|
||||
constraints and workload. See examples/task.json. Input and cache token buckets are
|
||||
disjoint; output includes reasoning. Attempts are a declared scenario, never `1/fitness`.
|
||||
|
||||
Catalog JSON: benchmark definitions (id/version/constructs/health/limitations/scale),
|
||||
exact profiles (id/model/harness/effort/provider/deployment/attributes/availability),
|
||||
results (profile_id/benchmark_id/benchmark_version/raw_score/evidence), and prices
|
||||
(profile_id/currency/valid_from/valid_until/usd_per_million/evidence). See
|
||||
examples/advisor-catalog.json for a complete synthetic shape. A synthetic catalog
|
||||
requires explicit `--synthetic` and never mixes with production evidence.
|
||||
|
||||
For benchmark health, provide all non-task-relevance weights named in the active
|
||||
policy. Values are curator judgments, not automated validation. Duplicate IDs are
|
||||
rejected. Missing, stale, future or conflicting results remain unknown. Price
|
||||
validity is date-inclusive and was deliberately limited to the review day in the
|
||||
initial live catalog. New days need new observations.
|
||||
|
||||
## Actual outcomes
|
||||
|
||||
```bash
|
||||
python3 -m intelligence_radar outcome /tmp/outcome.json
|
||||
```
|
||||
|
||||
An outcome contains recommendation_id, profile_id, unique run_id, accepted (boolean),
|
||||
independent_verification (boolean), verifier_evidence (artifact reference or notes),
|
||||
actual_usd, runtime_seconds, and completed_at (timezone-aware timestamp).
|
||||
Outcomes are immutable and run IDs are idempotent; changed duplicate runs are rejected.
|
||||
Keep the verifier artifact where its reference can be resolved. These are operator
|
||||
assertions. No actual provider execution or paid inference happens in this command.
|
||||
|
||||
A subsequent recommendation uses matching outcomes available at its cutoff.
|
||||
A historical replay uses its original frozen set. The current baseline reports
|
||||
95% Wilson intervals, not a learned cross-task success predictor.
|
||||
|
||||
## Scheduled operation
|
||||
|
||||
```bash
|
||||
python3 -m intelligence_radar run-due
|
||||
python3 -m intelligence_radar health
|
||||
systemctl --user status intelligence-radar.timer
|
||||
```
|
||||
|
||||
`ops/schedule.json` configures daily 21:30 and Sunday 20:00 Europe/Berlin.
|
||||
The shipped user timer checks every 15 minutes while this workstation's user service
|
||||
manager is running. It does not promise execution while the workstation is off.
|
||||
Each job is at most once per successful slot; failures remain retryable. Only the
|
||||
latest missed slot is caught up. Runtime state lives in ignored `.runtime/`;
|
||||
`data/scheduler-runs/` retains auditable receipts. Existing digests are preserved,
|
||||
so review the receipt's digest_status before assuming a new draft was produced.
|
||||
|
||||
Install the reviewed units with `make install-timer`; uninstall/disable with
|
||||
`systemctl --user disable --now intelligence-radar.timer`. That command leaves
|
||||
source evidence and publications intact. No external app notifications are sent.
|
||||
Loading…
Add table
Add a link
Reference in a new issue