> Imported exploration, 2026-09-13. Source claims have not been independently verified by this repository. See [integration review](../docs/integration-review.md).

# Measurement of intelligence

Version 0.1 · 13 September 2026

If the maps in `01` are theories, this file is the **instrument panel**. Intelligence Radar refuses to rank models from a single number, but it also refuses to treat every eval as equally informative.

## Human measurement (background)

- **Psychometric batteries** — WAIS-class tests, Raven matrices, specialized factor batteries. Human *g* research lives here.
- **Expert work samples** — the closest human analogue to GDPval / OSWorld: can a person produce the artifact a job requires.
- **Animal batteries** — delayed match, tool use, social inference. Relevant when someone claims “animal-level AGI.”

These are monitored for conceptual transfer, not for daily model ranking.

## Artificial measurement — what still discriminates in September 2026

Classic exams (MMLU, GSM8K, HumanEval, HellaSwag, ARC-Challenge) are largely **saturated** at the frontier. They remain useful as regression tests and as historical series, not as rankers.

Instruments that still move, with caveats:

| Instrument | Layer it hits | Caveat |
| --- | --- | --- |
| Humanity’s Last Exam (HLE) | expert knowledge + reasoning | tool-use variants change the number |
| FrontierMath / FrontierMath Erdős | deep mathematical research | tiny N; one problem moves the headline |
| GPQA Diamond | graduate science QA | approaching saturation at the very top |
| SciCode | scientific coding | scaffold-sensitive |
| SWE-bench Verified / Pro, DeepSWE, Frontier Code | software engineering agents | harness and test-contamination risk |
| Terminal-Bench 4.0 | CLI / sysadmin / research agents | version jumps rewrite history |
| OSWorld 2.0 | long-horizon computer use | live desktop; expensive to run |
| ARC-AGI-3 | abstract interactive reasoning | wrapper/harness can swing scores by tens of points |
| GDPval-AA v2, GDP.pdf, AA-Briefcase | professional knowledge work | closer to economic value |
| AutomationBench-AA | agentic office workflows | private set; good contamination resistance |
| AA-Omniscience, SimpleQA, CritPt | factuality / calibration | “smart but wrong” detector |
| Artificial Analysis Intelligence Index v4.3 | composite of the above class | re-bases; never compare raw scores across index versions |
| Epoch Capabilities Index (ECI) | long-run capability scale | best for *history*, not shopping |
| LMSYS / Arena-style preference | human taste | not a substitute for task success |

## Rules for using scores

1. **Same harness, same date, same effort setting.** “Max” vs “medium” reasoning is a different product.
2. **Price per completed task beats price per million tokens** when the model burns hidden reasoning tokens.
3. **Availability is part of the score.** A preview-only or capacity-gated model is tagged, not treated as generally usable.
4. **Open weights vs API** are different goods (latency, privacy, sovereignty, fine-tune rights).
5. **A 10% degradation** in Task Router language means: on the *task-weighted* blend of the five mapped benchmarks, comparative task-fit proxy is within 10% relative of the Frontier model (see `task-router/METHOD.md`). It is not “10 Elo” and it is not “10 index points.”

## What we do not pretend to measure well

- Scientific novelty (a model can ace HLE and still not find a new theorem).
- Taste, aesthetics, and institutional judgment.
- Long-run reliability in production (weeks of agent autonomy).
- Alignment, refusal quality, and dual-use except via lab preparedness designations and independent red-team reports.
- Consciousness.

## Changelog

- 2026-09-13: v0.1 instrument panel.