tegwick
d0e6438609
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a09c77-b1a9-7eb3-b501-6cb24b458c97
3 KiB
3 KiB
Imported discovery draft; recipes are non-normative. Verify versions and use METHOD.md for selection.
Benchmark atlas — Task Router
Version 0.1 · 13 September 2026
Pick five. Recipes at the bottom.
Living instruments (preferred)
| ID | Benchmark | Best-used-for layers | Still discriminating? |
|---|---|---|---|
| HLE | Humanity’s Last Exam | know + reason (expert) | yes |
| FMath | FrontierMath / Erdős | reason (research math) | yes, tiny N |
| GPQA | GPQA Diamond | know + reason (grad science) | fading at the very top |
| SciCode | SciCode | reason + tool (scientific code) | yes |
| SWE-V | SWE-bench Verified | tool + plan (repo repair) | yes, some saturation at top |
| SWE-P | SWE-bench Pro | tool + plan (harder engineering) | yes |
| DeepSWE | DeepSWE | tool + plan (agent coding) | yes |
| FCode | Frontier Code | tool + plan (front of coding agents) | yes |
| Term4 | Terminal-Bench 4.0 | tool + plan (CLI / research ops) | yes (v4 reset) |
| OSW2 | OSWorld 2.0 | perceive + tool + plan (desktop) | yes, lots of headroom |
| ARC3 | ARC-AGI-3 | reason (interactive abstraction) | yes; harness-sensitive |
| GDPv | GDPval-AA v2 | know + reason (occupation work) | yes |
| GDPpdf | GDP.pdf | know + reason (long professional docs) | yes |
| Brief | AA-Briefcase | know + reason (knowledge work pack) | yes |
| AutoB | AutomationBench-AA | tool + plan (office agents) | yes |
| Omni | AA-Omniscience / SimpleQA | know + reliable (facts) | yes |
| Crit | CritPt | reliable (critique / catch errors) | yes |
| LCR | AA-LCR | reason (long-context) | yes |
| AAIX | AA Intelligence Index (current ver.) | composite | yes, version-locked |
| ECI | Epoch Capabilities Index | long-run general capability | yes, for history |
| LiveC | LiveCodeBench | tool (contest coding, contamination-aware) | yes |
| Browse | BrowseComp / web-research evals | tool + know (search) | use when task is open-web |
Saturated / secondary (at most one of five)
MMLU-Pro, MATH-500, AIME (year-locked), HumanEval, IFEval, Arena Elo.
Default five-benchmark recipes
| Task family | Five |
|---|---|
| Hard general question / research memo | HLE, GDPpdf, Brief, Omni, Crit |
| Graduate science explanation | GPQA, HLE, SciCode, Crit, Omni |
| Research mathematics | FMath, HLE, SciCode, LCR, Crit |
| Production software change | SWE-P, DeepSWE, Term4, LiveC, Crit |
| Greenfield coding / scripts | LiveC, SWE-V, Term4, Crit, Omni |
| Computer-use / “do this in the GUI” | OSW2, Term4, AutoB, ARC3, Crit |
| Office automation / agent workflow | AutoB, GDPv, Term4, OSW2, Brief |
| Long-document diligence | GDPpdf, LCR, Brief, Crit, Omni |
| Open-web investigation | Browse, Omni, Crit, Brief, HLE |
| Multimodal perception + act | OSW2, ARC3, AutoB, GDPv, Crit |
| Cheap classification / extraction | Omni, Crit, Brief, GDPv, AAIX |
| “What model should we buy?” meta | AAIX, GDPv, SWE-P, Term4, OSW2 |
Changelog
- 2026-09-13: v0.1 atlas. Rebuild whenever a major eval saturates or an index rebases.