Integrate founding explorations and bootstrap operational radar
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a09c77-b1a9-7eb3-b501-6cb24b458c97
This commit is contained in:
parent
775f93b920
commit
d0e6438609
57 changed files with 4723 additions and 2 deletions
412
ResearchProgram.md
Normal file
412
ResearchProgram.md
Normal file
|
|
@ -0,0 +1,412 @@
|
|||
# Intelligence Radar Research Program
|
||||
|
||||
**Status:** Draft 0.1
|
||||
**Date:** 2026-09-13
|
||||
|
||||
## 1. Research objective
|
||||
|
||||
Maintain a continuously improving map of:
|
||||
|
||||
1. what intelligence is proposed to mean;
|
||||
2. how intelligence is generated and constrained across substrates;
|
||||
3. how it can be measured;
|
||||
4. what artificial systems can currently do;
|
||||
5. what those capabilities cost and where they are available;
|
||||
6. what the consequences are for engineering, science, economics, society, safety, law, and governance.
|
||||
|
||||
The program intentionally joins **slow epistemic research** with **fast operational monitoring**.
|
||||
|
||||
## 2. Research streams
|
||||
|
||||
### R1 — Models of intelligence
|
||||
|
||||
Questions:
|
||||
|
||||
- What definitions and operationalizations exist?
|
||||
- Which phenomena do they explain well?
|
||||
- Which assumptions do they make about goals, representation, embodiment, learning, or agency?
|
||||
- Which theories are complementary versus genuinely contradictory?
|
||||
- What evidence could discriminate among them?
|
||||
|
||||
### R2 — Biological intelligence
|
||||
|
||||
- neural computation;
|
||||
- memory and learning;
|
||||
- animal cognition;
|
||||
- development;
|
||||
- evolution;
|
||||
- perception/action loops;
|
||||
- energetic efficiency;
|
||||
- social cognition.
|
||||
|
||||
### R3 — Systems, cybernetics, and control
|
||||
|
||||
- requisite variety;
|
||||
- regulation;
|
||||
- viability;
|
||||
- feedback;
|
||||
- adaptation;
|
||||
- recursive organization;
|
||||
- observer dependence;
|
||||
- self-reference;
|
||||
- autopoiesis/enaction.
|
||||
|
||||
### R4 — Artificial architectures
|
||||
|
||||
- deep learning;
|
||||
- transformers and alternatives;
|
||||
- recurrent/state-space systems;
|
||||
- world models;
|
||||
- reinforcement learning;
|
||||
- memory;
|
||||
- retrieval;
|
||||
- planning;
|
||||
- tool use;
|
||||
- program synthesis;
|
||||
- neuro-symbolic systems;
|
||||
- robotics;
|
||||
- multi-agent systems.
|
||||
|
||||
### R5 — Capability measurement
|
||||
|
||||
- construct validity;
|
||||
- benchmark design;
|
||||
- contamination;
|
||||
- saturation;
|
||||
- evaluator/judge reliability;
|
||||
- reproducibility;
|
||||
- ecological validity;
|
||||
- long-horizon evaluation;
|
||||
- hidden/private tests;
|
||||
- human baselines;
|
||||
- task distributions.
|
||||
|
||||
### R6 — Capability frontier
|
||||
|
||||
Track:
|
||||
|
||||
- model releases;
|
||||
- execution profiles;
|
||||
- agent scaffolds;
|
||||
- context and memory;
|
||||
- multimodality;
|
||||
- tool/computer use;
|
||||
- reliability;
|
||||
- autonomy horizon;
|
||||
- coding;
|
||||
- research;
|
||||
- scientific reasoning;
|
||||
- professional knowledge work;
|
||||
- security-relevant capability.
|
||||
|
||||
### R7 — Economics of intelligence
|
||||
|
||||
Track:
|
||||
|
||||
- input/output/cache/reasoning token prices;
|
||||
- tool charges;
|
||||
- cost per solved task;
|
||||
- runtime;
|
||||
- throughput;
|
||||
- latency;
|
||||
- energy;
|
||||
- accelerator economics;
|
||||
- self-hosting economics;
|
||||
- open-weight availability;
|
||||
- batching/flex/priority tiers;
|
||||
- geography and access restrictions.
|
||||
|
||||
Goal:
|
||||
|
||||
> Build a quality-adjusted price surface for intelligence rather than a token-price table.
|
||||
|
||||
### R8 — Intelligence engineering
|
||||
|
||||
- inference infrastructure;
|
||||
- accelerators;
|
||||
- memory bandwidth;
|
||||
- networking;
|
||||
- compilers;
|
||||
- quantization;
|
||||
- serving;
|
||||
- speculative decoding;
|
||||
- distributed inference;
|
||||
- edge intelligence;
|
||||
- agent runtimes;
|
||||
- observability;
|
||||
- evaluation harnesses.
|
||||
|
||||
### R9 — Safety, security, and reliability
|
||||
|
||||
- dangerous capabilities;
|
||||
- cybersecurity;
|
||||
- biosecurity;
|
||||
- alignment;
|
||||
- deception/manipulation;
|
||||
- robustness;
|
||||
- privacy;
|
||||
- misuse;
|
||||
- incident evidence;
|
||||
- control and monitoring methods.
|
||||
|
||||
### R10 — Economic and social effects
|
||||
|
||||
- productivity;
|
||||
- task substitution/complementarity;
|
||||
- labor;
|
||||
- organizational design;
|
||||
- education;
|
||||
- science;
|
||||
- inequality;
|
||||
- access;
|
||||
- market concentration;
|
||||
- new business models.
|
||||
|
||||
### R11 — Law, policy, standards, and institutions
|
||||
|
||||
Track original texts and official implementation material for:
|
||||
|
||||
- AI regulation;
|
||||
- safety obligations;
|
||||
- privacy/data law interactions;
|
||||
- copyright;
|
||||
- liability;
|
||||
- competition;
|
||||
- standards;
|
||||
- procurement;
|
||||
- sector regulation;
|
||||
- export controls and compute policy where relevant.
|
||||
|
||||
## 3. Monitoring cadence
|
||||
|
||||
### Daily
|
||||
|
||||
Detect high-value deltas.
|
||||
|
||||
Output:
|
||||
|
||||
- event;
|
||||
- primary source;
|
||||
- what changed;
|
||||
- why it matters;
|
||||
- affected catalog entities;
|
||||
- affected claims/policies;
|
||||
- confidence;
|
||||
- follow-up required.
|
||||
|
||||
### Weekly
|
||||
|
||||
Synthesize rather than concatenate.
|
||||
|
||||
Recommended sections:
|
||||
|
||||
1. **Five most consequential changes**
|
||||
2. **Capability frontier**
|
||||
3. **Cost and availability frontier**
|
||||
4. **Benchmark/evaluation health**
|
||||
5. **Science of intelligence**
|
||||
6. **Agents/software/hardware**
|
||||
7. **Safety and security**
|
||||
8. **Economics and society**
|
||||
9. **Politics, law, regulation, standards**
|
||||
10. **Compendium changes**
|
||||
11. **What became less certain**
|
||||
12. **Watchlist for next week**
|
||||
|
||||
### Monthly
|
||||
|
||||
Perform maintenance that daily/weekly news cannot:
|
||||
|
||||
- benchmark health review;
|
||||
- source quality review;
|
||||
- thinker/source additions/removals;
|
||||
- stale claims;
|
||||
- pricing/catalog reconciliation;
|
||||
- calibration analysis from real tasks;
|
||||
- research-gap prioritization.
|
||||
|
||||
### Quarterly
|
||||
|
||||
Produce a deeper **State of Intelligence** revision:
|
||||
|
||||
- major model/theory changes;
|
||||
- frontier movement;
|
||||
- economics;
|
||||
- evaluation validity;
|
||||
- capability-to-cost curves;
|
||||
- autonomy horizon;
|
||||
- open vs closed availability;
|
||||
- important regulatory changes;
|
||||
- newly emerging research questions.
|
||||
|
||||
## 4. Evidence discipline
|
||||
|
||||
### Tier A — Primary evidence
|
||||
|
||||
Preferred for durable facts:
|
||||
|
||||
- peer-reviewed papers / primary preprints;
|
||||
- official datasets/evaluation repos;
|
||||
- benchmark maintainers;
|
||||
- model/system cards;
|
||||
- provider documentation;
|
||||
- official pricing;
|
||||
- source code/releases;
|
||||
- regulator/legislator texts;
|
||||
- standards bodies;
|
||||
- original experiment data.
|
||||
|
||||
### Tier B — Independent measurement / synthesis
|
||||
|
||||
Examples:
|
||||
|
||||
- independent evaluation organizations;
|
||||
- systematic benchmark comparisons;
|
||||
- high-quality academic syntheses;
|
||||
- major index reports.
|
||||
|
||||
Useful for triangulation and cross-provider comparison.
|
||||
|
||||
### Tier C — Expert interpretation
|
||||
|
||||
Researchers, engineers, economists, legal experts, and other thinkers.
|
||||
|
||||
Useful for hypotheses, interpretation, and discovery.
|
||||
|
||||
Not equivalent to primary evidence.
|
||||
|
||||
### Tier D — General reporting/social signal
|
||||
|
||||
Useful as an early warning or discovery stream.
|
||||
|
||||
A durable Radar claim should normally resolve to stronger evidence.
|
||||
|
||||
## 5. Research item template
|
||||
|
||||
```yaml
|
||||
id: research-item-...
|
||||
title: ...
|
||||
research_streams: [R5, R6]
|
||||
question: ...
|
||||
status: open
|
||||
priority: high
|
||||
claims:
|
||||
- ...
|
||||
evidence_needed:
|
||||
- ...
|
||||
sources:
|
||||
- ...
|
||||
next_actions:
|
||||
- ...
|
||||
last_reviewed: ...
|
||||
```
|
||||
|
||||
## 6. Claim lifecycle
|
||||
|
||||
```text
|
||||
candidate
|
||||
→ sourced
|
||||
→ structured
|
||||
→ challenged
|
||||
→ triangulated
|
||||
→ compendium claim
|
||||
→ periodically re-evaluated
|
||||
```
|
||||
|
||||
No claim becomes durable merely because many secondary articles repeat it.
|
||||
|
||||
## 7. Benchmark research lifecycle
|
||||
|
||||
Every benchmark gets a health record.
|
||||
|
||||
Review:
|
||||
|
||||
- intended construct;
|
||||
- real task distribution;
|
||||
- scoring correctness;
|
||||
- ceiling/saturation;
|
||||
- public-test contamination;
|
||||
- benchmark gaming incentives;
|
||||
- scaffold sensitivity;
|
||||
- judge sensitivity;
|
||||
- version comparability;
|
||||
- human baseline quality;
|
||||
- cost to run;
|
||||
- licensing/access;
|
||||
- predictive correlation with Intelligence Radar task outcomes.
|
||||
|
||||
A benchmark can be:
|
||||
|
||||
- `candidate`
|
||||
- `active`
|
||||
- `watch`
|
||||
- `degraded`
|
||||
- `saturated`
|
||||
- `retired`
|
||||
|
||||
Retirement does not delete historical results.
|
||||
|
||||
## 8. Frontier research question
|
||||
|
||||
Do not ask only:
|
||||
|
||||
> What is the best model?
|
||||
|
||||
Track multiple frontiers:
|
||||
|
||||
- maximal quality;
|
||||
- quality per dollar;
|
||||
- quality per joule;
|
||||
- quality per second;
|
||||
- quality per GB of memory;
|
||||
- quality under local/self-hosted constraints;
|
||||
- quality under privacy/ZDR constraints;
|
||||
- autonomy horizon;
|
||||
- open-weight frontier;
|
||||
- deployable frontier by region/provider;
|
||||
- multimodal frontier;
|
||||
- specialized domain frontiers.
|
||||
|
||||
## 9. Calibration research
|
||||
|
||||
The most strategically important proprietary dataset will be the Task → Model Performance Ledger.
|
||||
|
||||
Research questions:
|
||||
|
||||
- Which public benchmarks best predict which task clusters?
|
||||
- When do model-family/vendor effects remain after benchmark scores?
|
||||
- How stable are rankings across scaffolds?
|
||||
- Which task features predict failures?
|
||||
- How much does decomposition help?
|
||||
- When is a verifier more valuable than a larger generator?
|
||||
- How do costs scale with ambiguity?
|
||||
- How well does model self-estimated confidence predict actual success?
|
||||
- Which benchmarks lose predictive value first as the frontier moves?
|
||||
|
||||
## 10. Anti-goals
|
||||
|
||||
Avoid:
|
||||
|
||||
- benchmark tourism;
|
||||
- leaderboard worship;
|
||||
- treating press-release evaluation tables as neutral evidence;
|
||||
- silently mixing model-only and agent-scaffold results;
|
||||
- using stale prices;
|
||||
- measuring task success only with model judges;
|
||||
- collapsing heterogeneous scientific concepts into a single AGI narrative;
|
||||
- overreacting to every daily release.
|
||||
|
||||
## 11. First twelve research tasks
|
||||
|
||||
1. Build a genealogy of definitions of intelligence across psychometrics, cybernetics, cognitive science, neuroscience, AI, and collective intelligence.
|
||||
2. Establish a benchmark construct taxonomy.
|
||||
3. Create a benchmark health scorecard.
|
||||
4. Model `execution_profile` explicitly and normalize existing leaderboard evidence into it.
|
||||
5. Build price-observation ingestion for major providers.
|
||||
6. Build a first capability-vector ontology from 50 representative real tasks.
|
||||
7. Test the five-benchmark selector manually on those 50 tasks.
|
||||
8. Establish initial success-probability calibration using repeated model trials.
|
||||
9. Compare monolithic vs decomposed execution on at least 20 complex tasks.
|
||||
10. Measure benchmark correlation and redundancy.
|
||||
11. Define availability/privacy constraints as first-class ranking filters.
|
||||
12. Publish the first quarterly State of Intelligence report from accumulated Radar evidence.
|
||||
Loading…
Add table
Add a link
Reference in a new issue