intelligence-radar/BenchmarkCatalog.md

> Imported research draft. Benchmark versions, availability, and source claims require verification before operational use.

# Benchmark Catalog — Initial Canon

**Status:** Draft 0.1  
**Snapshot date:** 2026-09-13

## 1. Purpose

The Benchmark Catalog is not a list of scores. It is a registry of **measurement instruments**.

Each benchmark record should describe:

- intended construct;
- task distribution;
- modality;
- interaction style;
- agent/tool environment;
- scoring;
- versions;
- freshness;
- saturation;
- contamination risk;
- reproducibility;
- ecological validity;
- scaffold dependence;
- cost to run;
- public/private test status;
- observed correlations with other benchmarks;
- observed predictive value for Intelligence Radar task outcomes.

## 2. Health dimensions

Suggested 0–1 dimensions:

- `construct_relevance`
- `ecological_validity`
- `measurement_reliability`
- `discriminative_power`
- `freshness`
- `reproducibility`
- `contamination_resistance`
- `execution_profile_similarity`
- `coverage_breadth`

No single health score should replace the components, though a summary can be generated.

## 3. Initial benchmark candidates

### Humanity's Last Exam (HLE)

**Constructs:** broad expert academic knowledge/reasoning, multimodal closed-ended reasoning.  
**Strength:** deliberately difficult broad frontier evaluation.  
**Risk:** closed-ended academic performance is not equivalent to agentic work; rapid frontier gains can reduce discrimination over time.  
**Status:** active/watch for saturation.

Source: Nature, *A benchmark of expert-level academic questions to assess AI capabilities*, published 2026-01-28.

### ARC-AGI-2

**Constructs:** abstraction, novel-rule induction, test-time adaptation, sample-efficient generalization.  
**Strength:** novel tasks with minimal prior task-specific knowledge; explicit efficiency orientation.  
**Risk:** narrow task formalism; uncertain transfer to many real workloads.  
**Status:** active.

Source: ARC Prize official ARC-AGI-2 materials and 2026 competition.

### SWE-bench Verified

**Constructs:** repository-level software issue resolution.  
**Environment:** code repository + tests; agent/scaffold heavily matters.  
**Strength:** real GitHub issues; executable verification.  
**Risk:** benchmark and scaffold familiarity; Python-heavy Verified set; high scores reduce discrimination.  
**Status:** active/watch.

Official site states Verified is a human-filtered 500-instance subset.

### SWE-bench Multilingual

**Constructs:** real software engineering across multiple languages.  
**Strength:** 300 curated tasks from 42 repositories across 9 languages.  
**Risk:** still a specific issue-resolution distribution.  
**Status:** active.

### Terminal-Bench

**Constructs:** agent competence on hard terminal tasks.  
**Strength:** execution-based tasks in sandboxed terminal environments.  
**Risk:** version comparability and agent harness effects must be tracked carefully.  
**Status:** active; version must always be recorded.

Artificial Analysis moved its aggregate index to Terminal-Bench v4.0 in September 2026, illustrating rapid benchmark version churn.

### Berkeley Function Calling Leaderboard (BFCL) V4

**Constructs:** tool/function calling and broader agentic tool use.  
**Strength:** explicit function/tool-use evaluation, real-world contributed functions, multi-turn and agentic evolution.  
**Risk:** native function calling vs prompted emulation are distinct execution conditions.  
**Status:** active.

Official leaderboard describes V4 as moving from function calling toward holistic agentic evaluation.

### τ³-bench

**Constructs:** tool-agent-user interaction; knowledge-aware and voice variants.  
**Strength:** evaluates coordination with dynamic users/tools rather than isolated single-turn answers.  
**Risk:** domain/task versioning materially affects comparability.  
**Status:** active.

A July 2026 grading update explicitly made affected pre-1.0.1 and post-1.0.1 results non-comparable.

### OSWorld 2.0

**Constructs:** multimodal computer use in real operating-system environments.  
**Strength:** execution in computer environments.  
**Risk:** environment configuration, UI drift, and agent harness matter.  
**Status:** active.

OSWorld announced version 2.0 in June 2026.

### BrowseComp

**Constructs:** persistent web research/navigation to find difficult information.  
**Strength:** tests search persistence and information discovery.  
**Risk:** short verifiable answers omit much of real research quality, synthesis, ambiguity handling, and citation practice.  
**Status:** active.

The original dataset contains 1,266 difficult browsing questions.

### METR Task-Completion Time Horizon

**Constructs:** autonomy/reliability versus human-expert task duration.  
**Strength:** converts heterogeneous software tasks into an interpretable time-horizon measure at fixed success probability.  
**Risk:** software-task distribution; estimates above the tested horizon become unreliable.  
**Status:** active.

METR's Time Horizon 1.1 page was last updated 2026-05-08 and explicitly warns that measurements above 16 hours are unreliable with the current task suite.

### GDPval / GDPval-AA v2

**Constructs:** economically valuable professional knowledge work across occupations.  
**Strength:** realistic work products including documents, spreadsheets, diagrams, and slides.  
**Risk:** judge-model sensitivity and agent harness effects; GDPval-AA is an independent evaluation implementation rather than the original benchmark alone.  
**Status:** active.

Artificial Analysis' GDPval-AA v2 uses 220 tasks from the larger GDPval dataset, shell/web tools, and pairwise judging.

### AA-Briefcase

**Constructs:** long-horizon agentic knowledge work.  
**Strength:** private evaluation set helps contamination resistance.  
**Risk:** externally non-reproducible/private; methodology provider dependence.  
**Status:** active as independent evidence, not canonical truth.

Included in Artificial Analysis Intelligence Index v4.2+.

### AutomationBench-AA

**Constructs:** agentic business workflow automation.  
**Strength:** hundreds of workflows across simulated business applications with a private question set.  
**Risk:** implementation-specific; private test set.  
**Status:** active as independent evidence.

Added to Artificial Analysis Intelligence Index v4.3 in September 2026.

### SciCode

**Constructs:** scientific reasoning translated into research-level code.  
**Strength:** scientist-curated problems from realistic scientific workflows.  
**Risk:** specialized scientific-programming distribution.  
**Status:** active.

Original benchmark contains research problems across physics, mathematics, materials science, biology, and chemistry.

### MLPerf Inference

**Constructs:** systems-level inference performance and efficiency, not semantic intelligence directly.  
**Strength:** standardized hardware/software serving measurement; indispensable for self-hosting economics.  
**Risk:** workload benchmark results are not direct task-quality rankings.  
**Status:** active.

MLPerf Inference v6.0 added/updated major LLM workloads including GPT-OSS 120B and an interactive DeepSeek-R1 scenario.

## 4. Benchmarks intentionally not treated as timeless

A benchmark can lose selection priority when:

- frontier systems approach ceiling;
- training contamination becomes plausible;
- test cases become widely optimized against;
- scorer errors are discovered;
- benchmark version changes invalidate comparability;
- the benchmark ceases to predict real Intelligence Radar outcomes;
- a more ecologically valid instrument becomes available.

The 2026 Stanford AI Index explicitly reports rapid benchmark saturation and evaluation-quality concerns. That is a design requirement for Intelligence Radar, not background trivia.

## 5. Five-benchmark selection rule

Given task `T`, candidate benchmark `b`, and already-selected set `S`:

```text
base(b,T) =
    0.30 relevance_to_task
  + 0.15 ecological_validity
  + 0.12 measurement_reliability
  + 0.12 discriminative_power
  + 0.10 freshness
  + 0.10 execution_profile_similarity
  + 0.06 reproducibility
  + 0.05 contamination_resistance

marginal(b,T,S) =
    base(b,T)
  - redundancy_penalty(b,S)
  - known_bias_penalty(b,T)
```

Select greedily or via constrained optimization until five measurements are chosen.

Then normalize task weights across the five:

`weight_i = positive_relevance_i / Σ positive_relevance`

The system must return:

- raw relevance;
- final weight;
- why selected;
- what it fails to measure;
- independence/redundancy notes;
- benchmark version;
- most recent result date used.

## 6. Capability-to-benchmark examples

| Task characteristic | Strong candidate evidence |
|---|---|
| Repository issue resolution | SWE-bench Verified / Multilingual |
| Terminal/DevOps work | Terminal-Bench |
| Tool/API orchestration | BFCL V4 |
| User+tool interactive workflow | τ³-bench |
| Computer GUI use | OSWorld 2.0 |
| Deep web research | BrowseComp |
| Long autonomous software tasks | METR time horizon |
| Professional deliverables | GDPval-AA v2 / AA-Briefcase |
| Business-app automation | AutomationBench-AA |
| Scientific programming | SciCode |
| Novel abstract induction | ARC-AGI-2 |
| Broad expert knowledge | HLE |
| Self-hosted inference economics | MLPerf Inference |

## 7. Required next catalog work

1. Add machine-readable records for every benchmark.
2. Capture version-specific score semantics.
3. Estimate benchmark correlations from shared model results.
4. Add saturation indicators.
5. Add benchmark-cost estimates.
6. Add contamination/public-test exposure indicators.
7. Record model/scaffold/provider exact configuration for every result.
8. Add predictive validity against the Task → Model Performance Ledger.