intelligence-radar/BenchmarkCatalog.md

Imported research draft. Benchmark versions, availability, and source claims require verification before operational use.

Benchmark Catalog — Initial Canon

Status: Draft 0.1
Snapshot date: 2026-09-13

1. Purpose

The Benchmark Catalog is not a list of scores. It is a registry of measurement instruments.

Each benchmark record should describe:

• intended construct;
• task distribution;
• modality;
• interaction style;
• agent/tool environment;
• scoring;
• versions;
• freshness;
• saturation;
• contamination risk;
• reproducibility;
• ecological validity;
• scaffold dependence;
• cost to run;
• public/private test status;
• observed correlations with other benchmarks;
• observed predictive value for Intelligence Radar task outcomes.

2. Health dimensions

Suggested 0–1 dimensions:

• construct_relevance
• ecological_validity
• measurement_reliability
• discriminative_power
• freshness
• reproducibility
• contamination_resistance
• execution_profile_similarity
• coverage_breadth

No single health score should replace the components, though a summary can be generated.

3. Initial benchmark candidates

Humanity's Last Exam (HLE)

Constructs: broad expert academic knowledge/reasoning, multimodal closed-ended reasoning.
Strength: deliberately difficult broad frontier evaluation.
Risk: closed-ended academic performance is not equivalent to agentic work; rapid frontier gains can reduce discrimination over time.
Status: active/watch for saturation.

Source: Nature, A benchmark of expert-level academic questions to assess AI capabilities, published 2026-01-28.

ARC-AGI-2

Constructs: abstraction, novel-rule induction, test-time adaptation, sample-efficient generalization.
Strength: novel tasks with minimal prior task-specific knowledge; explicit efficiency orientation.
Risk: narrow task formalism; uncertain transfer to many real workloads.
Status: active.

Source: ARC Prize official ARC-AGI-2 materials and 2026 competition.

SWE-bench Verified

Constructs: repository-level software issue resolution.
Environment: code repository + tests; agent/scaffold heavily matters.
Strength: real GitHub issues; executable verification.
Risk: benchmark and scaffold familiarity; Python-heavy Verified set; high scores reduce discrimination.
Status: active/watch.

Official site states Verified is a human-filtered 500-instance subset.

SWE-bench Multilingual

Constructs: real software engineering across multiple languages.
Strength: 300 curated tasks from 42 repositories across 9 languages.
Risk: still a specific issue-resolution distribution.
Status: active.

Terminal-Bench

Constructs: agent competence on hard terminal tasks.
Strength: execution-based tasks in sandboxed terminal environments.
Risk: version comparability and agent harness effects must be tracked carefully.
Status: active; version must always be recorded.

Artificial Analysis moved its aggregate index to Terminal-Bench v4.0 in September 2026, illustrating rapid benchmark version churn.

Berkeley Function Calling Leaderboard (BFCL) V4

Constructs: tool/function calling and broader agentic tool use.
Strength: explicit function/tool-use evaluation, real-world contributed functions, multi-turn and agentic evolution.
Risk: native function calling vs prompted emulation are distinct execution conditions.
Status: active.

Official leaderboard describes V4 as moving from function calling toward holistic agentic evaluation.

τ³-bench

Constructs: tool-agent-user interaction; knowledge-aware and voice variants.
Strength: evaluates coordination with dynamic users/tools rather than isolated single-turn answers.
Risk: domain/task versioning materially affects comparability.
Status: active.

A July 2026 grading update explicitly made affected pre-1.0.1 and post-1.0.1 results non-comparable.

OSWorld 2.0

Constructs: multimodal computer use in real operating-system environments.
Strength: execution in computer environments.
Risk: environment configuration, UI drift, and agent harness matter.
Status: active.

OSWorld announced version 2.0 in June 2026.

BrowseComp

Constructs: persistent web research/navigation to find difficult information.
Strength: tests search persistence and information discovery.
Risk: short verifiable answers omit much of real research quality, synthesis, ambiguity handling, and citation practice.
Status: active.

The original dataset contains 1,266 difficult browsing questions.

METR Task-Completion Time Horizon

Constructs: autonomy/reliability versus human-expert task duration.
Strength: converts heterogeneous software tasks into an interpretable time-horizon measure at fixed success probability.
Risk: software-task distribution; estimates above the tested horizon become unreliable.
Status: active.

METR's Time Horizon 1.1 page was last updated 2026-05-08 and explicitly warns that measurements above 16 hours are unreliable with the current task suite.

GDPval / GDPval-AA v2

Constructs: economically valuable professional knowledge work across occupations.
Strength: realistic work products including documents, spreadsheets, diagrams, and slides.
Risk: judge-model sensitivity and agent harness effects; GDPval-AA is an independent evaluation implementation rather than the original benchmark alone.
Status: active.

Artificial Analysis' GDPval-AA v2 uses 220 tasks from the larger GDPval dataset, shell/web tools, and pairwise judging.

AA-Briefcase

Constructs: long-horizon agentic knowledge work.
Strength: private evaluation set helps contamination resistance.
Risk: externally non-reproducible/private; methodology provider dependence.
Status: active as independent evidence, not canonical truth.

Included in Artificial Analysis Intelligence Index v4.2+.

AutomationBench-AA

Constructs: agentic business workflow automation.
Strength: hundreds of workflows across simulated business applications with a private question set.
Risk: implementation-specific; private test set.
Status: active as independent evidence.

Added to Artificial Analysis Intelligence Index v4.3 in September 2026.

SciCode

Constructs: scientific reasoning translated into research-level code.
Strength: scientist-curated problems from realistic scientific workflows.
Risk: specialized scientific-programming distribution.
Status: active.

Original benchmark contains research problems across physics, mathematics, materials science, biology, and chemistry.

MLPerf Inference

Constructs: systems-level inference performance and efficiency, not semantic intelligence directly.
Strength: standardized hardware/software serving measurement; indispensable for self-hosting economics.
Risk: workload benchmark results are not direct task-quality rankings.
Status: active.

MLPerf Inference v6.0 added/updated major LLM workloads including GPT-OSS 120B and an interactive DeepSeek-R1 scenario.

4. Benchmarks intentionally not treated as timeless

A benchmark can lose selection priority when:

• frontier systems approach ceiling;
• training contamination becomes plausible;
• test cases become widely optimized against;
• scorer errors are discovered;
• benchmark version changes invalidate comparability;
• the benchmark ceases to predict real Intelligence Radar outcomes;
• a more ecologically valid instrument becomes available.

The 2026 Stanford AI Index explicitly reports rapid benchmark saturation and evaluation-quality concerns. That is a design requirement for Intelligence Radar, not background trivia.

5. Five-benchmark selection rule

Given task T, candidate benchmark b, and already-selected set S:

base(b,T) =
    0.30 relevance_to_task
  + 0.15 ecological_validity
  + 0.12 measurement_reliability
  + 0.12 discriminative_power
  + 0.10 freshness
  + 0.10 execution_profile_similarity
  + 0.06 reproducibility
  + 0.05 contamination_resistance

marginal(b,T,S) =
    base(b,T)
  - redundancy_penalty(b,S)
  - known_bias_penalty(b,T)

Select greedily or via constrained optimization until five measurements are chosen.

Then normalize task weights across the five:

weight_i = positive_relevance_i / Σ positive_relevance

The system must return:

• raw relevance;
• final weight;
• why selected;
• what it fails to measure;
• independence/redundancy notes;
• benchmark version;
• most recent result date used.

6. Capability-to-benchmark examples

Task characteristic	Strong candidate evidence
Repository issue resolution	SWE-bench Verified / Multilingual
Terminal/DevOps work	Terminal-Bench
Tool/API orchestration	BFCL V4
User+tool interactive workflow	τ³-bench
Computer GUI use	OSWorld 2.0
Deep web research	BrowseComp
Long autonomous software tasks	METR time horizon
Professional deliverables	GDPval-AA v2 / AA-Briefcase
Business-app automation	AutomationBench-AA
Scientific programming	SciCode
Novel abstract induction	ARC-AGI-2
Broad expert knowledge	HLE
Self-hosted inference economics	MLPerf Inference

7. Required next catalog work

1. Add machine-readable records for every benchmark.
2. Capture version-specific score semantics.
3. Estimate benchmark correlations from shared model results.
4. Add saturation indicators.
5. Add benchmark-cost estimates.
6. Add contamination/public-test exposure indicators.
7. Record model/scaffold/provider exact configuration for every result.
8. Add predictive validity against the Task → Model Performance Ledger.