tegwick
d0e6438609
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a09c77-b1a9-7eb3-b501-6cb24b458c97
58 lines
3 KiB
Markdown
58 lines
3 KiB
Markdown
> Imported discovery draft; recipes are non-normative. Verify versions and use METHOD.md for selection.
|
||
|
||
# Benchmark atlas — Task Router
|
||
|
||
Version 0.1 · 13 September 2026
|
||
Pick five. Recipes at the bottom.
|
||
|
||
## Living instruments (preferred)
|
||
|
||
| ID | Benchmark | Best-used-for layers | Still discriminating? |
|
||
| --- | --- | --- | --- |
|
||
| HLE | Humanity’s Last Exam | know + reason (expert) | yes |
|
||
| FMath | FrontierMath / Erdős | reason (research math) | yes, tiny N |
|
||
| GPQA | GPQA Diamond | know + reason (grad science) | fading at the very top |
|
||
| SciCode | SciCode | reason + tool (scientific code) | yes |
|
||
| SWE-V | SWE-bench Verified | tool + plan (repo repair) | yes, some saturation at top |
|
||
| SWE-P | SWE-bench Pro | tool + plan (harder engineering) | yes |
|
||
| DeepSWE | DeepSWE | tool + plan (agent coding) | yes |
|
||
| FCode | Frontier Code | tool + plan (front of coding agents) | yes |
|
||
| Term4 | Terminal-Bench 4.0 | tool + plan (CLI / research ops) | yes (v4 reset) |
|
||
| OSW2 | OSWorld 2.0 | perceive + tool + plan (desktop) | yes, lots of headroom |
|
||
| ARC3 | ARC-AGI-3 | reason (interactive abstraction) | yes; **harness-sensitive** |
|
||
| GDPv | GDPval-AA v2 | know + reason (occupation work) | yes |
|
||
| GDPpdf | GDP.pdf | know + reason (long professional docs) | yes |
|
||
| Brief | AA-Briefcase | know + reason (knowledge work pack) | yes |
|
||
| AutoB | AutomationBench-AA | tool + plan (office agents) | yes |
|
||
| Omni | AA-Omniscience / SimpleQA | know + reliable (facts) | yes |
|
||
| Crit | CritPt | reliable (critique / catch errors) | yes |
|
||
| LCR | AA-LCR | reason (long-context) | yes |
|
||
| AAIX | AA Intelligence Index (current ver.) | composite | yes, version-locked |
|
||
| ECI | Epoch Capabilities Index | long-run general capability | yes, for history |
|
||
| LiveC | LiveCodeBench | tool (contest coding, contamination-aware) | yes |
|
||
| Browse | BrowseComp / web-research evals | tool + know (search) | use when task is open-web |
|
||
|
||
## Saturated / secondary (at most one of five)
|
||
|
||
MMLU-Pro, MATH-500, AIME (year-locked), HumanEval, IFEval, Arena Elo.
|
||
|
||
## Default five-benchmark recipes
|
||
|
||
| Task family | Five |
|
||
| --- | --- |
|
||
| Hard general question / research memo | HLE, GDPpdf, Brief, Omni, Crit |
|
||
| Graduate science explanation | GPQA, HLE, SciCode, Crit, Omni |
|
||
| Research mathematics | FMath, HLE, SciCode, LCR, Crit |
|
||
| Production software change | SWE-P, DeepSWE, Term4, LiveC, Crit |
|
||
| Greenfield coding / scripts | LiveC, SWE-V, Term4, Crit, Omni |
|
||
| Computer-use / “do this in the GUI” | OSW2, Term4, AutoB, ARC3, Crit |
|
||
| Office automation / agent workflow | AutoB, GDPv, Term4, OSW2, Brief |
|
||
| Long-document diligence | GDPpdf, LCR, Brief, Crit, Omni |
|
||
| Open-web investigation | Browse, Omni, Crit, Brief, HLE |
|
||
| Multimodal perception + act | OSW2, ARC3, AutoB, GDPv, Crit |
|
||
| Cheap classification / extraction | Omni, Crit, Brief, GDPv, AAIX |
|
||
| “What model should we buy?” meta | AAIX, GDPv, SWE-P, Term4, OSW2 |
|
||
|
||
## Changelog
|
||
|
||
- 2026-09-13: v0.1 atlas. Rebuild whenever a major eval saturates or an index rebases.
|