Integrate founding explorations and bootstrap operational radar
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a09c77-b1a9-7eb3-b501-6cb24b458c97
This commit is contained in:
parent
775f93b920
commit
d0e6438609
57 changed files with 4723 additions and 2 deletions
464
ArchitectureBlueprint.md
Normal file
464
ArchitectureBlueprint.md
Normal file
|
|
@ -0,0 +1,464 @@
|
|||
> Target architecture. The implemented local v0.1 boundary is in [SCOPE.md](SCOPE.md) and [ADR 0001](docs/adr/0001-initial-operational-baseline.md).
|
||||
|
||||
# Intelligence Radar — Architecture Blueprint
|
||||
|
||||
**Status:** Draft 0.1
|
||||
**Date:** 2026-09-13
|
||||
|
||||
## 1. Architectural objective
|
||||
|
||||
Build a system that can continuously ingest evidence about intelligence and AI capability, preserve provenance, update a structured knowledge model, and answer task-specific model-selection questions reproducibly.
|
||||
|
||||
The design separates:
|
||||
|
||||
1. **observations** — immutable source-derived facts;
|
||||
2. **knowledge** — claims and relationships built from observations;
|
||||
3. **policy** — versioned interpretation and ranking rules;
|
||||
4. **recommendations** — time-indexed outputs produced from observations + policy;
|
||||
5. **outcomes** — real task results used for calibration.
|
||||
|
||||
## 2. High-level architecture
|
||||
|
||||
```text
|
||||
┌──────────────────────┐
|
||||
│ Primary Sources │
|
||||
│ papers/docs/repos │
|
||||
│ labs/regulators/etc. │
|
||||
└──────────┬───────────┘
|
||||
│
|
||||
discovery / fetch / poll
|
||||
│
|
||||
┌─────────────────▼─────────────────┐
|
||||
│ Ingestion Plane │
|
||||
│ source registry · fetchers │
|
||||
│ dedupe · provenance · extraction │
|
||||
└─────────────────┬─────────────────┘
|
||||
│
|
||||
immutable observations
|
||||
│
|
||||
┌───────────────────────────▼───────────────────────────┐
|
||||
│ Knowledge Plane │
|
||||
│ events · claims · evidence · models · deployments │
|
||||
│ benchmarks · prices · hardware · regulations │
|
||||
│ task outcomes · historical versions │
|
||||
└───────────────┬───────────────────────┬──────────────┘
|
||||
│ │
|
||||
compendium synthesis capability data
|
||||
│ │
|
||||
┌─────────────▼─────────┐ ┌────────▼────────────────┐
|
||||
│ Intelligence │ │ Capability Advisor │
|
||||
│ Compendium │ │ task profiler │
|
||||
│ claim graph │ │ benchmark selector │
|
||||
│ theory maps │ │ model scorer │
|
||||
└─────────────┬────────┘ │ economics estimator │
|
||||
│ │ workplan optimizer │
|
||||
│ └──────────┬───────────────┘
|
||||
│ │
|
||||
┌─────────▼───────────────────────▼────────┐
|
||||
│ Publication Plane │
|
||||
│ daily delta · weekly digest · query API │
|
||||
│ recommendation report · dashboards │
|
||||
└───────────────────┬──────────────────────┘
|
||||
│
|
||||
▼
|
||||
execution / feedback
|
||||
│
|
||||
┌───────────▼───────────┐
|
||||
│ Outcome & Calibration│
|
||||
│ task-performance │
|
||||
│ ledger │
|
||||
└───────────┬───────────┘
|
||||
└──────► model calibration
|
||||
```
|
||||
|
||||
## 3. Data layers
|
||||
|
||||
### 3.1 Source registry
|
||||
|
||||
A versioned registry describing what should be monitored.
|
||||
|
||||
Fields:
|
||||
|
||||
- source id;
|
||||
- publisher/person;
|
||||
- source type;
|
||||
- URL/API/feed;
|
||||
- topic tags;
|
||||
- trust tier;
|
||||
- primary/secondary;
|
||||
- expected cadence;
|
||||
- retrieval method;
|
||||
- license/copyright constraints;
|
||||
- last successful observation;
|
||||
- importance;
|
||||
- notes.
|
||||
|
||||
### 3.2 Raw observation store
|
||||
|
||||
Append-only.
|
||||
|
||||
Stores:
|
||||
|
||||
- retrieval timestamp;
|
||||
- publication/event timestamp;
|
||||
- source URI;
|
||||
- content hash;
|
||||
- document metadata;
|
||||
- extracted structured facts;
|
||||
- retrieval agent/version;
|
||||
- link to raw artifact where legally appropriate;
|
||||
- provenance.
|
||||
|
||||
Never overwrite a historical observation because a source page later changes.
|
||||
|
||||
### 3.3 Event store
|
||||
|
||||
Normalized events such as:
|
||||
|
||||
- model released;
|
||||
- model deprecated;
|
||||
- benchmark version released;
|
||||
- benchmark score published;
|
||||
- price changed;
|
||||
- provider added/removed;
|
||||
- paper published;
|
||||
- regulation enacted/applies;
|
||||
- hardware announced/shipped;
|
||||
- safety incident;
|
||||
- capability claim;
|
||||
- benchmark invalidation/saturation evidence.
|
||||
|
||||
### 3.4 Claim graph
|
||||
|
||||
Compendium entities and evidence relationships described in `IntelligenceConceptModel.md`.
|
||||
|
||||
### 3.5 Capability catalog
|
||||
|
||||
Contains separate entities for:
|
||||
|
||||
- `model`;
|
||||
- `execution_profile`;
|
||||
- `deployment`;
|
||||
- `benchmark`;
|
||||
- `benchmark_version`;
|
||||
- `benchmark_result`;
|
||||
- `price_observation`;
|
||||
- `availability_observation`;
|
||||
- `hardware/system_result`.
|
||||
|
||||
### 3.6 Task-performance ledger
|
||||
|
||||
Stores real task predictions and outcomes.
|
||||
|
||||
This becomes the calibration dataset for task-specific performance prediction.
|
||||
|
||||
## 4. Suggested initial physical implementation
|
||||
|
||||
Start boring and auditable:
|
||||
|
||||
- **Git** for normative Markdown, policies, schemas, hand-curated catalog seeds, and digest artifacts.
|
||||
- **PostgreSQL** for indexed structured observations, claim relationships, benchmark results, model/deployment catalog, recommendation history, and task-outcome ledger.
|
||||
- **Object storage** for larger source artifacts and generated reports.
|
||||
- **Parquet/Iceberg later** when observation volume, historical analytics, or multi-engine processing warrants it.
|
||||
- **Embeddings/vector search** as an index over evidence, never as the canonical source of truth.
|
||||
|
||||
The architecture should not require a graph database initially. PostgreSQL can represent graph edges cleanly; a specialized graph engine can be added when query patterns justify it.
|
||||
|
||||
## 5. Ingestion pipeline
|
||||
|
||||
```text
|
||||
discover
|
||||
→ fetch
|
||||
→ fingerprint
|
||||
→ deduplicate
|
||||
→ classify
|
||||
→ extract candidate facts/events
|
||||
→ resolve entities
|
||||
→ attach provenance
|
||||
→ confidence checks
|
||||
→ append observation
|
||||
→ emit event
|
||||
→ candidate claim update
|
||||
```
|
||||
|
||||
### Agentic extraction rule
|
||||
|
||||
Agents may propose structured facts and claims.
|
||||
|
||||
They must not silently turn their inference into source fact.
|
||||
|
||||
Every extracted field should carry one of:
|
||||
|
||||
- `explicit_source_statement`
|
||||
- `derived_from_source`
|
||||
- `cross_source_inference`
|
||||
- `system_estimate`
|
||||
|
||||
## 6. Daily Radar pipeline
|
||||
|
||||
Daily runs should answer:
|
||||
|
||||
1. What new source artifacts appeared?
|
||||
2. What facts/events are genuinely new?
|
||||
3. What existing observations were corrected or contradicted?
|
||||
4. Did any benchmark materially change?
|
||||
5. Did price, availability, context limit, or model status change?
|
||||
6. Did the capability frontier move?
|
||||
7. Did the cost/performance Pareto frontier move?
|
||||
8. Did any event affect a Compendium claim?
|
||||
9. Did any current recommendation policy become questionable?
|
||||
|
||||
Output should be a **delta**, not a generic summary of the news.
|
||||
|
||||
## 7. Weekly synthesis pipeline
|
||||
|
||||
The weekly digest ranks events by estimated significance.
|
||||
|
||||
Recommended significance dimensions:
|
||||
|
||||
- epistemic impact;
|
||||
- capability impact;
|
||||
- economic impact;
|
||||
- breadth of affected tasks;
|
||||
- confidence;
|
||||
- novelty;
|
||||
- persistence;
|
||||
- governance/social impact.
|
||||
|
||||
The weekly synthesis should explicitly state:
|
||||
|
||||
> What do we have more reason to believe than seven days ago?
|
||||
|
||||
and:
|
||||
|
||||
> What do we have less reason to believe?
|
||||
|
||||
## 8. Capability Advisor pipeline
|
||||
|
||||
```text
|
||||
task
|
||||
↓
|
||||
Task Interpreter
|
||||
↓
|
||||
Clarity + constraints + risk class
|
||||
↓
|
||||
Capability Profile
|
||||
↓
|
||||
Benchmark Candidate Retrieval
|
||||
↓
|
||||
Five-Benchmark Selector
|
||||
↓
|
||||
Result Freshness / Validity Filter
|
||||
↓
|
||||
Execution Profile Candidate Builder
|
||||
↓
|
||||
Task-Fit Estimator
|
||||
↓
|
||||
Availability Filter
|
||||
↓
|
||||
Cost + Runtime Estimator
|
||||
↓
|
||||
Frontier / Topnotch / Efficient / Sufficient / Budget
|
||||
↓
|
||||
Confidence decomposition
|
||||
↓
|
||||
optional Workplan Optimizer
|
||||
```
|
||||
|
||||
## 9. Workplan optimizer
|
||||
|
||||
The optimizer converts a task into a DAG.
|
||||
|
||||
Each node contains:
|
||||
|
||||
- subtask;
|
||||
- inputs/outputs;
|
||||
- capability profile;
|
||||
- dependency edges;
|
||||
- acceptance/verifier;
|
||||
- parallelism constraints;
|
||||
- privacy/tool constraints;
|
||||
- candidate execution profiles;
|
||||
- predicted success;
|
||||
- cost distribution;
|
||||
- runtime distribution.
|
||||
|
||||
It should generate several plans, at minimum:
|
||||
|
||||
1. **Monolithic Frontier**
|
||||
2. **Monolithic Efficient**
|
||||
3. **Decomposed Quality**
|
||||
4. **Decomposed Efficient**
|
||||
5. **Decomposed Budget**
|
||||
|
||||
Optimization objectives can include:
|
||||
|
||||
- maximize predicted success subject to budget;
|
||||
- minimize expected cost subject to success threshold;
|
||||
- minimize p90 runtime subject to quality threshold;
|
||||
- maximize quality-adjusted value.
|
||||
|
||||
### Verification as a first-class subtask
|
||||
|
||||
Do not allocate all budget to generation.
|
||||
|
||||
A cheaper producer plus an independent verifier can outperform an expensive monolithic attempt when errors are detectable.
|
||||
|
||||
## 10. Confidence model
|
||||
|
||||
Store components independently:
|
||||
|
||||
- `task_clarity`
|
||||
- `benchmark_coverage`
|
||||
- `benchmark_validity`
|
||||
- `measurement_reliability`
|
||||
- `result_freshness`
|
||||
- `deployment_match`
|
||||
- `price_confidence`
|
||||
- `task_calibration_evidence`
|
||||
- `overall_recommendation_confidence`
|
||||
|
||||
The overall confidence number is a convenience. The components are the actual explanation.
|
||||
|
||||
## 11. Temporal semantics
|
||||
|
||||
Every mutable external fact must support:
|
||||
|
||||
- `observed_at`
|
||||
- `valid_from` if known
|
||||
- `valid_to` if known
|
||||
- `source_published_at`
|
||||
- `supersedes_observation_id` where applicable.
|
||||
|
||||
A model price without a timestamp is not valid capability-economics data.
|
||||
|
||||
## 12. Policy versioning
|
||||
|
||||
Examples:
|
||||
|
||||
- benchmark selection policy;
|
||||
- task capability ontology;
|
||||
- benchmark-score normalization;
|
||||
- Topnotch threshold;
|
||||
- sufficiency threshold inference;
|
||||
- cost model;
|
||||
- confidence aggregation;
|
||||
- source trust;
|
||||
- model eligibility.
|
||||
|
||||
Every recommendation stores policy hashes/versions.
|
||||
|
||||
Historical recommendations therefore remain reproducible.
|
||||
|
||||
## 13. API surface — first draft
|
||||
|
||||
### `POST /evaluate-task`
|
||||
|
||||
Returns task profile and model recommendation.
|
||||
|
||||
### `POST /optimize-workplan`
|
||||
|
||||
Returns model-allocated DAG alternatives.
|
||||
|
||||
### `POST /record-outcome`
|
||||
|
||||
Adds verified outcome evidence.
|
||||
|
||||
### `GET /models`
|
||||
|
||||
Current model catalog.
|
||||
|
||||
### `GET /deployments`
|
||||
|
||||
Provider-specific availability and economics.
|
||||
|
||||
### `GET /benchmarks`
|
||||
|
||||
Benchmark catalog and health.
|
||||
|
||||
### `GET /radar/events`
|
||||
|
||||
Normalized recent events.
|
||||
|
||||
### `GET /digests/{period}`
|
||||
|
||||
Daily/weekly Radar products.
|
||||
|
||||
### `GET /compendium/claims/{id}`
|
||||
|
||||
Claim + evidence graph.
|
||||
|
||||
## 14. Minimal recommendation record
|
||||
|
||||
```yaml
|
||||
task_id: task-...
|
||||
evaluated_at: 2026-09-13T...
|
||||
policy:
|
||||
task_profile: v0.1
|
||||
benchmark_selection: v0.1
|
||||
model_ranking: v0.1
|
||||
clarity:
|
||||
score: 0.78
|
||||
ambiguities: []
|
||||
benchmarks:
|
||||
- id: ...
|
||||
weight: 0.27
|
||||
frontier:
|
||||
execution_profile: ...
|
||||
topnotch: [...]
|
||||
efficient: ...
|
||||
sufficient: [...]
|
||||
budget: ...
|
||||
economics:
|
||||
currency: USD
|
||||
expected: ...
|
||||
p90: ...
|
||||
runtime:
|
||||
expected_seconds: ...
|
||||
confidence:
|
||||
overall: ...
|
||||
components: ...
|
||||
evidence_gaps: [...]
|
||||
```
|
||||
|
||||
## 15. Initial milestones
|
||||
|
||||
### M0 — Repository kernel
|
||||
|
||||
Documents, schemas, policies, source catalog.
|
||||
|
||||
### M1 — Dated model/economics catalog
|
||||
|
||||
Automated collection of model releases, availability, context limits, and API prices from major providers.
|
||||
|
||||
### M2 — Benchmark health catalog
|
||||
|
||||
Benchmark metadata, versions, results, saturation and contamination notes.
|
||||
|
||||
### M3 — Task profiler
|
||||
|
||||
Produces stable capability profiles and clarity assessment.
|
||||
|
||||
### M4 — Five-benchmark selector
|
||||
|
||||
Produces explainable benchmark mapping with independence penalties.
|
||||
|
||||
### M5 — Model ranking
|
||||
|
||||
Frontier / Topnotch / Efficient / Sufficient / Budget with cost intervals.
|
||||
|
||||
### M6 — Workplan optimizer
|
||||
|
||||
DAG decomposition, heterogeneous model allocation, verification, parallel runtime estimate.
|
||||
|
||||
### M7 — Outcome ledger
|
||||
|
||||
Capture actual task success, cost, runtime, and failure modes.
|
||||
|
||||
### M8 — Calibration engine
|
||||
|
||||
Learn task-cluster performance models from the outcome ledger.
|
||||
|
||||
### M9 — Self-improving Radar
|
||||
|
||||
Detect benchmark drift/saturation and automatically propose policy/catalog changes for review.
|
||||
Loading…
Add table
Add a link
Reference in a new issue