Integrate founding explorations and bootstrap operational radar

Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a09c77-b1a9-7eb3-b501-6cb24b458c97
2026-09-13 22:42:09 +02:00 · 2026-09-13 22:42:09 +02:00 · d0e6438609
commit d0e6438609
parent 775f93b920
57 changed files with 4723 additions and 2 deletions
--- a/research/2026-09-13-landscape.md
+++ b/research/2026-09-13-landscape.md
@ -0,0 +1,335 @@
+> Imported exploration, 2026-09-13. Source claims have not been independently verified by this repository. See [integration review](../docs/integration-review.md).
+
+# Intelligence Radar — Initial Landscape Snapshot
+
+**As of:** 2026-09-13  
+**Purpose:** Seed observation for the Capability/Economics Radar.  
+**Important:** This is a dated snapshot, not a timeless ranking.
+
+## 1. Main observation
+
+The current frontier is better described as a **dense capability/cost Pareto surface** than as one universally dominant model.
+
+At the same time, the measurement regime is changing rapidly: benchmark suites are replacing saturated tests with harder, more agentic, longer-horizon, and increasingly private evaluations.
+
+This directly validates Intelligence Radar's design decision to make:
+
+- benchmark selection task-specific;
+- benchmark versions explicit;
+- price observations time-indexed;
+- execution profiles distinct from base models;
+- real-task outcomes a future calibration layer.
+
+## 2. Broad independent capability snapshot
+
+Artificial Analysis announced Intelligence Index v4.3 on 2026-09-07.
+
+Its v4.3 aggregate includes ten evaluations:
+
+- AA-Briefcase
+- GDPval-AA v2
+- AutomationBench-AA
+- Terminal-Bench v4.0
+- SciCode
+- Humanity's Last Exam
+- GDP.pdf
+- CritPt
+- AA-Omniscience
+- AA-LCR v1.1
+
+Reported top aggregate results in that index:
+
+| Execution profile | Index v4.3 |
+|---|---:|
+| Claude Fable 5.1, max with fallback | 53 |
+| GPT-6 Astra, max | 53 |
+| Claude Opus 5, max | 51 |
+| Claude Fable 5, with fallback | 50 |
+| Muse Spark 1.3, max | 48 |
+| GPT-5.6 Sol, max | 47 |
+
+Interpretation:
+
+- There is a tie on this particular broad aggregate, not proof of universal task equivalence.
+- Artificial Analysis reports GPT-6 Astra reaching the same aggregate score as Fable 5.1 at roughly 40% of the measured cost per task in its methodology.
+- Therefore Astra sits on a compelling **aggregate cost/capability frontier** in this measurement regime.
+- Task-specific Frontier/Efficient conclusions can differ.
+
+Sources:
+
+- https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-3
+- https://artificialanalysis.ai/articles/benchmarking-gpt-6-astra
+
+## 3. OpenAI — current observed deployment economics
+
+OpenAI's current model documentation lists GPT-6 Astra as its most capable model for hard end-to-end work.
+
+Observed API properties:
+
+### GPT-6 Astra
+
+- input: **$10 / 1M tokens**
+- cached input: **$1 / 1M**
+- cache write: **$12.50 / 1M**
+- output: **$50 / 1M**
+- context: **1.05M**
+- maximum output: **128K**
+- knowledge cutoff: **2026-04-30**
+- reasoning levels include low through max.
+
+Long prompts above 272K input tokens use higher rates according to OpenAI's model documentation.
+
+### GPT-5.6 family
+
+Current OpenAI model listing shows:
+
+- GPT-5.6 Sol: **$4 input / $20 output per 1M**
+- GPT-5.6 Terra: **$2 input / $12 output per 1M**
+- GPT-5.6 Luna: **$0.20 input / $1.20 output per 1M**
+
+GPT-5.6 Sol pricing is currently promotional according to OpenAI documentation and therefore must be stored with validity metadata.
+
+Sources:
+
+- https://developers.openai.com/api/docs/models/gpt-6-astra
+- https://developers.openai.com/api/docs/models
+- https://openai.com/index/gpt-5-6/
+
+## 4. Anthropic — current observed deployment economics
+
+Anthropic released Claude Fable 5.1 in September 2026.
+
+Observed Fable 5.1 API pricing:
+
+- input: **$10 / 1M tokens**
+- output: **$50 / 1M**
+- cache reads: **$0.25 / 1M**
+
+Anthropic states that reduced cache-read pricing lowers typical workload cost relative to Fable 5 and can matter even more for highly agentic workloads.
+
+Claude Sonnet 5 is currently documented at:
+
+- input: **$2 / 1M**
+- output: **$10 / 1M**
+
+This is an example of why token list prices alone are insufficient: cache-heavy agents can reverse simple nominal-price comparisons.
+
+Sources:
+
+- https://www.anthropic.com/claude-fable-and-mythos-5-1
+- https://www.anthropic.com/claude/fable
+- https://www.anthropic.com/news/claude-sonnet-5
+
+## 5. Google — current observed deployment economics
+
+Gemini 3.8 Flash became generally available in September 2026.
+
+Observed properties:
+
+- 1,048,576 input-token limit;
+- 65,536 output-token limit;
+- text/image/video/audio/PDF input;
+- code execution;
+- computer use in preview;
+- function calling;
+- search grounding;
+- structured outputs;
+- tunable thinking.
+
+Introductory standard API pricing through **2026-12-31**:
+
+- input: **$0.75 / 1M**
+- output (including thinking tokens): **$3.75 / 1M**
+- context cache: **$0.075 / 1M**
+
+Google states standard pricing doubles starting 2027-01-01.
+
+This is exactly the kind of temporary economic regime Intelligence Radar must represent explicitly.
+
+Sources:
+
+- https://ai.google.dev/gemini-api/docs/models/gemini-3.8-flash
+- https://ai.google.dev/gemini-api/docs/latest-model
+- https://ai.google.dev/gemini-api/docs/pricing
+
+## 6. Benchmark regime is moving toward agentic work
+
+Artificial Analysis' index changes in 2026 are instructive:
+
+- v4.1 shifted weight toward agentic workloads;
+- IFBench was removed because of saturation;
+- Terminal-Bench and τ-bench variants were upgraded;
+- GDPval methodology was upgraded for longer trajectories and more robust judging;
+- v4.2 added AA-Briefcase and removed GPQA Diamond after saturation;
+- v4.3 moved to Terminal-Bench v4 and added AutomationBench-AA with a private test set.
+
+This suggests a general benchmark lifecycle:
+
+`novel → discriminative → widely optimized → saturated/gamed → reduced predictive value → replace or re-scope`
+
+Source:
+
+- https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-1
+- https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-2
+- https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-3
+
+## 7. Stanford AI Index 2026 reinforces measurement concerns
+
+Stanford HAI's 2026 AI Index technical-performance chapter highlights:
+
+- rapid saturation of evaluations;
+- convergence among leading model providers on some aggregate measures;
+- benchmark reliability and gaming concerns;
+- large progress in agentic/computer-use systems;
+- continuing jaggedness across apparently simple and hard tasks.
+
+For Intelligence Radar, the important inference is not a specific leaderboard position. It is:
+
+> Benchmark health itself is a continuously changing research object.
+
+Source:
+
+- https://hai.stanford.edu/ai-index/2026-ai-index-report/technical-performance
+
+## 8. Long-horizon autonomy is becoming directly measurable
+
+METR's current Time Horizon 1.1 methodology estimates the human-expert task duration at which an AI agent reaches a specified probability of success.
+
+This is valuable because it reframes autonomy from a vague adjective into a reliability-conditioned horizon.
+
+METR explicitly warns that measurements above 16 hours are unreliable with its current task suite.
+
+Intelligence Radar should therefore treat autonomy-horizon measurements with:
+
+- task-distribution scope;
+- confidence intervals;
+- horizon validity limit;
+- model+agent configuration;
+- version.
+
+Source:
+
+- https://metr.org/time-horizons/
+
+## 9. Tool use and interactive agents need separate evidence
+
+BFCL V4 explicitly evaluates function/tool calling and describes its evolution toward holistic agentic evaluation.
+
+The τ³-bench family expands interactive evaluation to knowledge-aware and voice interaction. Its July 2026 grading update notes that affected results before and after version 1.0.1 are not comparable.
+
+This strongly supports benchmark versioning as a hard requirement.
+
+Sources:
+
+- https://gorilla.cs.berkeley.edu/leaderboard
+- https://github.com/sierra-research/tau2-bench
+- https://research.sierra.ai/
+
+## 10. Software engineering evidence is scaffold-dependent
+
+SWE-bench evaluates real software issues, but the leaderboard shows results attached to both a **model and an agent/scaffold**.
+
+SWE-bench Verified is a 500-instance human-filtered set. SWE-bench Multilingual contains 300 curated tasks across 42 repositories and 9 programming languages.
+
+Intelligence Radar must therefore never transform:
+
+`Model X + Agent Y = 79%`
+
+into:
+
+`Model X = 79% coding ability`
+
+without preserving the execution profile.
+
+Sources:
+
+- https://www.swebench.com/
+- https://www.swebench.com/multilingual.html
+
+## 11. Scientific and abstract reasoning remain distinct constructs
+
+### SciCode
+
+Scientist-curated scientific-programming tasks test the ability to combine scientific concepts, reasoning, and code synthesis.
+
+Source:
+
+- https://scicode-bench.github.io/
+
+### ARC-AGI-2
+
+ARC-AGI-2 deliberately stresses abstraction and novel problem solving under efficiency constraints. The 2026 competition sets 85% on the private evaluation set as a major target.
+
+Source:
+
+- https://arcprize.org/competitions/2026/arc-agi-2
+
+### Humanity's Last Exam
+
+HLE is an expert-level, broad, multimodal academic benchmark published in Nature in January 2026.
+
+Source:
+
+- https://www.nature.com/articles/s41586-025-09962-4
+
+These should not be collapsed into one “reasoning” number without retaining the constructs they measure.
+
+## 12. Hardware economics belongs inside capability economics
+
+MLPerf Inference v6.0 substantially updated its inference suite and added/expanded LLM workloads including GPT-OSS 120B and interactive DeepSeek-R1 scenarios.
+
+For hosted proprietary models, provider pricing can approximate economic cost to the user.
+
+For open/self-hosted models, Intelligence Radar needs system-level throughput, latency, power, accelerator utilization, and memory measurements.
+
+MLPerf therefore belongs in the same economic decision graph as model benchmarks.
+
+Sources:
+
+- https://mlcommons.org/2026/04/mlperf-inference-v6-0-results/
+- https://docs.mlcommons.org/inference/index_gh/
+
+## 13. Current initial hypotheses
+
+### H1 — “Best model” is increasingly task-local
+
+**Confidence:** high.
+
+Evidence: top models are clustered on broad measures while specialized benchmark and cost differences remain large.
+
+### H2 — Scaffold quality is becoming part of measured intelligence
+
+**Confidence:** high operationally; philosophical interpretation remains open.
+
+Agentic benchmarks increasingly measure model+tools+loop rather than frozen next-token competence.
+
+### H3 — Benchmark half-life is shrinking at the frontier
+
+**Confidence:** medium-high.
+
+Repeated benchmark upgrades/removals due to saturation support the claim, but the exact rate needs measurement.
+
+### H4 — Economics will become as important as aggregate capability
+
+**Confidence:** high.
+
+Top performance is clustering while task cost can differ by multiples.
+
+### H5 — Real task outcome data will be more valuable than generic leaderboards for repeated organizational work
+
+**Confidence:** hypothesis with strong rationale, not yet empirically proven by Intelligence Radar.
+
+This should be tested through the Task → Model Performance Ledger.
+
+## 14. Immediate catalog consequences
+
+1. Store benchmark versions.
+2. Store model + scaffold + effort as execution profiles.
+3. Store provider deployment separately.
+4. Timestamp all prices.
+5. Add cache economics.
+6. Add cost-per-task observations where methodology is known.
+7. Track benchmark saturation and replacement events.
+8. Do not globally designate one current Frontier model.
+9. Designate Frontier only relative to task and active policy.
+10. Use this snapshot solely as dated seed evidence.