coulomb/intelligence-radar
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
intelligence-radar/research/2026-09-13-landscape.md
tegwick d0e6438609 Integrate founding explorations and bootstrap operational radar 
Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a09c77-b1a9-7eb3-b501-6cb24b458c97
2026-09-13 22:42:09 +02:00

11 KiB
Raw Blame History

Imported exploration, 2026-09-13. Source claims have not been independently verified by this repository. See integration review.

Intelligence Radar — Initial Landscape Snapshot

As of: 2026-09-13
Purpose: Seed observation for the Capability/Economics Radar.
Important: This is a dated snapshot, not a timeless ranking.

1. Main observation

The current frontier is better described as a dense capability/cost Pareto surface than as one universally dominant model.

At the same time, the measurement regime is changing rapidly: benchmark suites are replacing saturated tests with harder, more agentic, longer-horizon, and increasingly private evaluations.

This directly validates Intelligence Radar's design decision to make:

  • benchmark selection task-specific;
  • benchmark versions explicit;
  • price observations time-indexed;
  • execution profiles distinct from base models;
  • real-task outcomes a future calibration layer.

2. Broad independent capability snapshot

Artificial Analysis announced Intelligence Index v4.3 on 2026-09-07.

Its v4.3 aggregate includes ten evaluations:

  • AA-Briefcase
  • GDPval-AA v2
  • AutomationBench-AA
  • Terminal-Bench v4.0
  • SciCode
  • Humanity's Last Exam
  • GDP.pdf
  • CritPt
  • AA-Omniscience
  • AA-LCR v1.1

Reported top aggregate results in that index:

Execution profile Index v4.3
Claude Fable 5.1, max with fallback 53
GPT-6 Astra, max 53
Claude Opus 5, max 51
Claude Fable 5, with fallback 50
Muse Spark 1.3, max 48
GPT-5.6 Sol, max 47

Interpretation:

  • There is a tie on this particular broad aggregate, not proof of universal task equivalence.
  • Artificial Analysis reports GPT-6 Astra reaching the same aggregate score as Fable 5.1 at roughly 40% of the measured cost per task in its methodology.
  • Therefore Astra sits on a compelling aggregate cost/capability frontier in this measurement regime.
  • Task-specific Frontier/Efficient conclusions can differ.

Sources:

3. OpenAI — current observed deployment economics

OpenAI's current model documentation lists GPT-6 Astra as its most capable model for hard end-to-end work.

Observed API properties:

GPT-6 Astra

  • input: $10 / 1M tokens
  • cached input: $1 / 1M
  • cache write: $12.50 / 1M
  • output: $50 / 1M
  • context: 1.05M
  • maximum output: 128K
  • knowledge cutoff: 2026-04-30
  • reasoning levels include low through max.

Long prompts above 272K input tokens use higher rates according to OpenAI's model documentation.

GPT-5.6 family

Current OpenAI model listing shows:

  • GPT-5.6 Sol: $4 input / $20 output per 1M
  • GPT-5.6 Terra: $2 input / $12 output per 1M
  • GPT-5.6 Luna: $0.20 input / $1.20 output per 1M

GPT-5.6 Sol pricing is currently promotional according to OpenAI documentation and therefore must be stored with validity metadata.

Sources:

4. Anthropic — current observed deployment economics

Anthropic released Claude Fable 5.1 in September 2026.

Observed Fable 5.1 API pricing:

  • input: $10 / 1M tokens
  • output: $50 / 1M
  • cache reads: $0.25 / 1M

Anthropic states that reduced cache-read pricing lowers typical workload cost relative to Fable 5 and can matter even more for highly agentic workloads.

Claude Sonnet 5 is currently documented at:

  • input: $2 / 1M
  • output: $10 / 1M

This is an example of why token list prices alone are insufficient: cache-heavy agents can reverse simple nominal-price comparisons.

Sources:

5. Google — current observed deployment economics

Gemini 3.8 Flash became generally available in September 2026.

Observed properties:

  • 1,048,576 input-token limit;
  • 65,536 output-token limit;
  • text/image/video/audio/PDF input;
  • code execution;
  • computer use in preview;
  • function calling;
  • search grounding;
  • structured outputs;
  • tunable thinking.

Introductory standard API pricing through 2026-12-31:

  • input: $0.75 / 1M
  • output (including thinking tokens): $3.75 / 1M
  • context cache: $0.075 / 1M

Google states standard pricing doubles starting 2027-01-01.

This is exactly the kind of temporary economic regime Intelligence Radar must represent explicitly.

Sources:

6. Benchmark regime is moving toward agentic work

Artificial Analysis' index changes in 2026 are instructive:

  • v4.1 shifted weight toward agentic workloads;
  • IFBench was removed because of saturation;
  • Terminal-Bench and τ-bench variants were upgraded;
  • GDPval methodology was upgraded for longer trajectories and more robust judging;
  • v4.2 added AA-Briefcase and removed GPQA Diamond after saturation;
  • v4.3 moved to Terminal-Bench v4 and added AutomationBench-AA with a private test set.

This suggests a general benchmark lifecycle:

novel → discriminative → widely optimized → saturated/gamed → reduced predictive value → replace or re-scope

Source:

7. Stanford AI Index 2026 reinforces measurement concerns

Stanford HAI's 2026 AI Index technical-performance chapter highlights:

  • rapid saturation of evaluations;
  • convergence among leading model providers on some aggregate measures;
  • benchmark reliability and gaming concerns;
  • large progress in agentic/computer-use systems;
  • continuing jaggedness across apparently simple and hard tasks.

For Intelligence Radar, the important inference is not a specific leaderboard position. It is:

Benchmark health itself is a continuously changing research object.

Source:

8. Long-horizon autonomy is becoming directly measurable

METR's current Time Horizon 1.1 methodology estimates the human-expert task duration at which an AI agent reaches a specified probability of success.

This is valuable because it reframes autonomy from a vague adjective into a reliability-conditioned horizon.

METR explicitly warns that measurements above 16 hours are unreliable with its current task suite.

Intelligence Radar should therefore treat autonomy-horizon measurements with:

  • task-distribution scope;
  • confidence intervals;
  • horizon validity limit;
  • model+agent configuration;
  • version.

Source:

9. Tool use and interactive agents need separate evidence

BFCL V4 explicitly evaluates function/tool calling and describes its evolution toward holistic agentic evaluation.

The τ³-bench family expands interactive evaluation to knowledge-aware and voice interaction. Its July 2026 grading update notes that affected results before and after version 1.0.1 are not comparable.

This strongly supports benchmark versioning as a hard requirement.

Sources:

10. Software engineering evidence is scaffold-dependent

SWE-bench evaluates real software issues, but the leaderboard shows results attached to both a model and an agent/scaffold.

SWE-bench Verified is a 500-instance human-filtered set. SWE-bench Multilingual contains 300 curated tasks across 42 repositories and 9 programming languages.

Intelligence Radar must therefore never transform:

Model X + Agent Y = 79%

into:

Model X = 79% coding ability

without preserving the execution profile.

Sources:

11. Scientific and abstract reasoning remain distinct constructs

SciCode

Scientist-curated scientific-programming tasks test the ability to combine scientific concepts, reasoning, and code synthesis.

Source:

ARC-AGI-2

ARC-AGI-2 deliberately stresses abstraction and novel problem solving under efficiency constraints. The 2026 competition sets 85% on the private evaluation set as a major target.

Source:

Humanity's Last Exam

HLE is an expert-level, broad, multimodal academic benchmark published in Nature in January 2026.

Source:

These should not be collapsed into one “reasoning” number without retaining the constructs they measure.

12. Hardware economics belongs inside capability economics

MLPerf Inference v6.0 substantially updated its inference suite and added/expanded LLM workloads including GPT-OSS 120B and interactive DeepSeek-R1 scenarios.

For hosted proprietary models, provider pricing can approximate economic cost to the user.

For open/self-hosted models, Intelligence Radar needs system-level throughput, latency, power, accelerator utilization, and memory measurements.

MLPerf therefore belongs in the same economic decision graph as model benchmarks.

Sources:

13. Current initial hypotheses

H1 — “Best model” is increasingly task-local

Confidence: high.

Evidence: top models are clustered on broad measures while specialized benchmark and cost differences remain large.

H2 — Scaffold quality is becoming part of measured intelligence

Confidence: high operationally; philosophical interpretation remains open.

Agentic benchmarks increasingly measure model+tools+loop rather than frozen next-token competence.

H3 — Benchmark half-life is shrinking at the frontier

Confidence: medium-high.

Repeated benchmark upgrades/removals due to saturation support the claim, but the exact rate needs measurement.

H4 — Economics will become as important as aggregate capability

Confidence: high.

Top performance is clustering while task cost can differ by multiples.

H5 — Real task outcome data will be more valuable than generic leaderboards for repeated organizational work

Confidence: hypothesis with strong rationale, not yet empirically proven by Intelligence Radar.

This should be tested through the Task → Model Performance Ledger.

14. Immediate catalog consequences

  1. Store benchmark versions.
  2. Store model + scaffold + effort as execution profiles.
  3. Store provider deployment separately.
  4. Timestamp all prices.
  5. Add cache economics.
  6. Add cost-per-task observations where methodology is known.
  7. Track benchmark saturation and replacement events.
  8. Do not globally designate one current Frontier model.
  9. Designate Frontier only relative to task and active policy.
  10. Use this snapshot solely as dated seed evidence.