coulomb/intelligence-radar
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Build reproducible capability advisor and scheduled source capture

Browse source 
Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a09c77-b1a9-7eb3-b501-6cb24b458c97
This commit is contained in:
tegwick 2026-09-13 23:07:41 +02:00
parent a1edefd7c5
commit 29ceb77938
68 changed files with 2431 additions and 85 deletions
Download patch file Download diff file Expand all files Collapse all files

47
docs/adr/0002-reproducible-advisor-and-source-capture.md Normal file
View file

@ -0,0 +1,47 @@
# ADR 0002 — Reproducible advisor and source capture
Accepted: 2026-09-13.
The v0.2 CLI uses standard-library Python and content-addressed JSON. Source bytes
are retained by hash. Capture receipts record requested/final URLs, publisher,
retrieval time, media type and change detection; capture does not certify a claim.
Changed sources create review-needed observations rather than invented extracted facts.
`advisor-v0.2.json` is the executable policy. It carries forward the baseline
selection weights, inclusive .90 Topnotch ratio and risk thresholds. It adds a
90-day evidence freshness limit, a .15 construct-overlap redundancy penalty,
and a minimum of 20 exact-task/profile outcomes. Earlier YAML remains an immutable
design baseline. v0.2 is explicitly provisional and curator-driven.
Five distinct nonaggregate instruments are selected greedily with construct-based
task relevance, curator-assigned health and redundancy penalties. Raw scores use
an explicit increasing linear scale per benchmark version. Missing scores are not
imputed. Nonlinear metrics, Elo, lower-is-better metrics and composite indices need
separate adapters before use. Pending benchmark versions cannot support ranking.
Eligible means all declared task constraints match evidenced deployment attributes.
Minima use `min_` keys; other constraints require exact equality. Production catalog
attributes require curator review; available does not imply requester access.
Fitness is comparative, not calibrated task success. Rankings cover only evidenced,
eligible candidates, not the entire model market.
Costs are explicit workload scenarios: disjoint input/cache/output buckets, tools,
declared attempts and verification. Output includes billable reasoning. Unknown or
conflicting prices prevent claims of cheapest. These are not estimated p50/p90s.
Outcomes bind a recommendation, exact task and exact profile snapshot. Unique run
IDs prevent duplicate evidence. Wilson bounds are reported over matching observed
trials; Sufficient requires the lower bound above the policy threshold, with at
least 20 trials. High-assurance also requires independent-verification assertions.
This assumes representative independent trials; it is not a guarantee or a
learned cross-task calibration model. Real operational outcome data is still absent.
Recommendations freeze task, catalog, policy, outcomes and report. Canonical key
ordering makes replay deterministic. Source captures and bytes remain referenced
by immutable IDs. Replay checks algorithm version and rejects divergence.
The scheduler checks the latest due daily/weekly slot in Europe/Berlin, catches up
one slot after downtime, serializes runs, persists successes, and retries failures
on later invocations. Each source fetch allows three bounded attempts and 5 MiB.
Existing digest files are preserved. Receipt history distinguishes a new draft
from an existing preserved artifact. Editorial synthesis remains a human/agent step.