+ + + +
+ +
+
+
+ + + + + +
+
+ + + + + +
+
+ +
+
+ + +
+ + +
+ + + + +
+
+ + +
+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + +
+

Each entry reports % Resolved, the percentage of task instances solved.

+
    +
  • + + Open-weights model +
  • +
  • + + Run performed or directly checked by the SWE-bench team +
  • +
+
+
+
+

SWE-bench

+ 2294 instances +
+

The original benchmark: real GitHub issues from 12 Python repositories.

+ Read the post +
+
+
+

Bash Only

+ 500 instances +
+

The default Verified view: every model in the same mini-SWE-agent environment.

+ Read the post +
+
+
+

Lite

+ 300 instances +
+

A subset curated for less costly evaluation.

+ Read the post +
+
+
+

Verified

+ 500 instances +
+

A human-filtered subset of SWE-bench.

+ Read the post +
+
+
+

Multilingual

+ 300 instances +
+

Tasks from 42 repositories across 9 programming languages.

+ Read the post +
+
+
+

Multimodal

+ 480 instances +
+

Issues described with visual elements.

+ Read the post +
+
+
+ + Analyze results in detail +
+
+ +
+
+

News

+
    +
  1. + ProgramBench icon + May 2026 + We released ProgramBench to benchmark whether models can code meaningful software artifacts from scratch. + Link + +
  2. +
  3. + CodeClash icon + Nov 2025 + Introducing CodeClash, our new eval of LMs as goal (not task) oriented developers. + Link + +
  4. +
  5. + mini-SWE-agent icon + Jul 2025 + mini-SWE-agent scores 65% on SWE-bench Verified in 100 lines of Python. + Link + +
  6. +
  7. + SWE-smith icon + May 2025 + SWE-smith is out! Train your own models for software engineering agents. + Link + +
  8. +
  9. + SWE-agent icon + Mar 2025 + SWE-agent 1.0 is the open source SOTA on SWE-bench Lite. + Link + +
  10. +
  11. + SWE-bench Multimodal icon + Oct 2024 + Introducing SWE-bench Multimodal. + Link + +
  12. +
  13. + OpenAI icon + Aug 2024 + SWE-bench x OpenAI = SWE-bench Verified. + Report + +
  14. +
  15. + SWE-bench icon + Jun 2024 + Docker-ized SWE-bench for easier evaluation. + Report + +
  16. +
  17. + SWE-agent icon + Mar 2024 + Check out SWE-agent (12.47% on SWE-bench). + Link + +
  18. +
  19. + SWE-bench icon + Mar 2024 + Released SWE-bench Lite. + Report + +
  20. +
+
+
+ +
+
+

Acknowledgements

+

+ We thank the following institutions for their generous support: + Open Philanthropy, AWS, Modal, Andreessen Horowitz, OpenAI, and Anthropic. +

+
+
+ + + +