59 lines
3.7 KiB
Markdown
59 lines
3.7 KiB
Markdown
|
> Imported exploration, 2026-09-13. Source claims have not been independently verified by this repository. See [integration review](../docs/integration-review.md).
|
|||
|
|
|
|||
|
|
# Measurement of intelligence
|
|||
|
|
|
|||
|
|
Version 0.1 · 13 September 2026
|
|||
|
|
|
|||
|
|
If the maps in `01` are theories, this file is the **instrument panel**. Intelligence Radar refuses to rank models from a single number, but it also refuses to treat every eval as equally informative.
|
|||
|
|
|
|||
|
|
## Human measurement (background)
|
|||
|
|
|
|||
|
|
- **Psychometric batteries** — WAIS-class tests, Raven matrices, specialized factor batteries. Human *g* research lives here.
|
|||
|
|
- **Expert work samples** — the closest human analogue to GDPval / OSWorld: can a person produce the artifact a job requires.
|
|||
|
|
- **Animal batteries** — delayed match, tool use, social inference. Relevant when someone claims “animal-level AGI.”
|
|||
|
|
|
|||
|
|
These are monitored for conceptual transfer, not for daily model ranking.
|
|||
|
|
|
|||
|
|
## Artificial measurement — what still discriminates in September 2026
|
|||
|
|
|
|||
|
|
Classic exams (MMLU, GSM8K, HumanEval, HellaSwag, ARC-Challenge) are largely **saturated** at the frontier. They remain useful as regression tests and as historical series, not as rankers.
|
|||
|
|
|
|||
|
|
Instruments that still move, with caveats:
|
|||
|
|
|
|||
|
|
| Instrument | Layer it hits | Caveat |
|
|||
|
|
| --- | --- | --- |
|
|||
|
|
| Humanity’s Last Exam (HLE) | expert knowledge + reasoning | tool-use variants change the number |
|
|||
|
|
| FrontierMath / FrontierMath Erdős | deep mathematical research | tiny N; one problem moves the headline |
|
|||
|
|
| GPQA Diamond | graduate science QA | approaching saturation at the very top |
|
|||
|
|
| SciCode | scientific coding | scaffold-sensitive |
|
|||
|
|
| SWE-bench Verified / Pro, DeepSWE, Frontier Code | software engineering agents | harness and test-contamination risk |
|
|||
|
|
| Terminal-Bench 4.0 | CLI / sysadmin / research agents | version jumps rewrite history |
|
|||
|
|
| OSWorld 2.0 | long-horizon computer use | live desktop; expensive to run |
|
|||
|
|
| ARC-AGI-3 | abstract interactive reasoning | wrapper/harness can swing scores by tens of points |
|
|||
|
|
| GDPval-AA v2, GDP.pdf, AA-Briefcase | professional knowledge work | closer to economic value |
|
|||
|
|
| AutomationBench-AA | agentic office workflows | private set; good contamination resistance |
|
|||
|
|
| AA-Omniscience, SimpleQA, CritPt | factuality / calibration | “smart but wrong” detector |
|
|||
|
|
| Artificial Analysis Intelligence Index v4.3 | composite of the above class | re-bases; never compare raw scores across index versions |
|
|||
|
|
| Epoch Capabilities Index (ECI) | long-run capability scale | best for *history*, not shopping |
|
|||
|
|
| LMSYS / Arena-style preference | human taste | not a substitute for task success |
|
|||
|
|
|
|||
|
|
## Rules for using scores
|
|||
|
|
|
|||
|
|
1. **Same harness, same date, same effort setting.** “Max” vs “medium” reasoning is a different product.
|
|||
|
|
2. **Price per completed task beats price per million tokens** when the model burns hidden reasoning tokens.
|
|||
|
|
3. **Availability is part of the score.** A preview-only or capacity-gated model is tagged, not treated as generally usable.
|
|||
|
|
4. **Open weights vs API** are different goods (latency, privacy, sovereignty, fine-tune rights).
|
|||
|
|
5. **A 10% degradation** in Task Router language means: on the *task-weighted* blend of the five mapped benchmarks, comparative task-fit proxy is within 10% relative of the Frontier model (see `task-router/METHOD.md`). It is not “10 Elo” and it is not “10 index points.”
|
|||
|
|
|
|||
|
|
## What we do not pretend to measure well
|
|||
|
|
|
|||
|
|
- Scientific novelty (a model can ace HLE and still not find a new theorem).
|
|||
|
|
- Taste, aesthetics, and institutional judgment.
|
|||
|
|
- Long-run reliability in production (weeks of agent autonomy).
|
|||
|
|
- Alignment, refusal quality, and dual-use except via lab preparedness designations and independent red-team reports.
|
|||
|
|
- Consciousness.
|
|||
|
|
|
|||
|
|
## Changelog
|
|||
|
|
|
|||
|
|
- 2026-09-13: v0.1 instrument panel.
|