tegwick
d0e6438609
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a09c77-b1a9-7eb3-b501-6cb24b458c97
3.7 KiB
3.7 KiB
Imported exploration, 2026-09-13. Source claims have not been independently verified by this repository. See integration review.
Measurement of intelligence
Version 0.1 · 13 September 2026
If the maps in 01 are theories, this file is the instrument panel. Intelligence Radar refuses to rank models from a single number, but it also refuses to treat every eval as equally informative.
Human measurement (background)
- Psychometric batteries — WAIS-class tests, Raven matrices, specialized factor batteries. Human g research lives here.
- Expert work samples — the closest human analogue to GDPval / OSWorld: can a person produce the artifact a job requires.
- Animal batteries — delayed match, tool use, social inference. Relevant when someone claims “animal-level AGI.”
These are monitored for conceptual transfer, not for daily model ranking.
Artificial measurement — what still discriminates in September 2026
Classic exams (MMLU, GSM8K, HumanEval, HellaSwag, ARC-Challenge) are largely saturated at the frontier. They remain useful as regression tests and as historical series, not as rankers.
Instruments that still move, with caveats:
| Instrument | Layer it hits | Caveat |
|---|---|---|
| Humanity’s Last Exam (HLE) | expert knowledge + reasoning | tool-use variants change the number |
| FrontierMath / FrontierMath Erdős | deep mathematical research | tiny N; one problem moves the headline |
| GPQA Diamond | graduate science QA | approaching saturation at the very top |
| SciCode | scientific coding | scaffold-sensitive |
| SWE-bench Verified / Pro, DeepSWE, Frontier Code | software engineering agents | harness and test-contamination risk |
| Terminal-Bench 4.0 | CLI / sysadmin / research agents | version jumps rewrite history |
| OSWorld 2.0 | long-horizon computer use | live desktop; expensive to run |
| ARC-AGI-3 | abstract interactive reasoning | wrapper/harness can swing scores by tens of points |
| GDPval-AA v2, GDP.pdf, AA-Briefcase | professional knowledge work | closer to economic value |
| AutomationBench-AA | agentic office workflows | private set; good contamination resistance |
| AA-Omniscience, SimpleQA, CritPt | factuality / calibration | “smart but wrong” detector |
| Artificial Analysis Intelligence Index v4.3 | composite of the above class | re-bases; never compare raw scores across index versions |
| Epoch Capabilities Index (ECI) | long-run capability scale | best for history, not shopping |
| LMSYS / Arena-style preference | human taste | not a substitute for task success |
Rules for using scores
- Same harness, same date, same effort setting. “Max” vs “medium” reasoning is a different product.
- Price per completed task beats price per million tokens when the model burns hidden reasoning tokens.
- Availability is part of the score. A preview-only or capacity-gated model is tagged, not treated as generally usable.
- Open weights vs API are different goods (latency, privacy, sovereignty, fine-tune rights).
- A 10% degradation in Task Router language means: on the task-weighted blend of the five mapped benchmarks, comparative task-fit proxy is within 10% relative of the Frontier model (see
task-router/METHOD.md). It is not “10 Elo” and it is not “10 index points.”
What we do not pretend to measure well
- Scientific novelty (a model can ace HLE and still not find a new theorem).
- Taste, aesthetics, and institutional judgment.
- Long-run reliability in production (weeks of agent autonomy).
- Alignment, refusal quality, and dual-use except via lab preparedness designations and independent red-team reports.
- Consciousness.
Changelog
- 2026-09-13: v0.1 instrument panel.