AIEvals

Benchmarks

SWE-Bench Pro V2 HARD

Repository issue resolution on the HARD subset of the public V2 set. Models must edit an existing codebase to satisfy the issue’s verification tests.

Repository codingScale AI / SEAL11 models run by Scale AI / SEALUpdated 22 September 2026Reported by Reflection AI

How it is run

Published September 2026 V2 table. Resolved-task percentage; each row retains its agent scaffold and effort setting.

What it cannot tell you

Full and HARD are distinct task sets. Claude Code, Codex and mini-swe-agent results use different scaffolds and are not a controlled model-only comparison.

Leader
Opus 5 (Claude Code) xhigh98.00% · Scale AI / SEAL results, grouped here by release
Top-ten spread
41.10%Leader minus tenth place, or last place in smaller fields.
Median
86.30%of the 11 rows published
No published result
36of 46 models on this site

Scale AI / SEAL results by release

V2 HARD field, as published. Different agent scaffolds and effort settings are retained per row; this is a model-and-agent comparison.

#ModelScoreEffort
1Opus 5 (Claude Code) xhigh98.00%(Claude Code) xhigh
2Fable 5.1 (Claude Code) high92.20%(Claude Code) high
3GPT-6 Astra (Codex) high90.20%(Codex) high
4Sonnet 5 (Claude Code) xhigh88.20%(Claude Code) xhigh
4Kimi-K3 (mini-swe-agent) max88.20%(mini-swe-agent) max
6GPT-5.6 Terra (Codex) xhigh86.30%(Codex) xhigh
7GLM-5.3 (mini-swe-agent) max84.30%(mini-swe-agent) max
8GPT-5.6 Sol (Codex) xhigh82.40%(Codex) xhigh
9Gemini 3.8 Flash (mini-swe-agent) high58.80%(mini-swe-agent) high
10Inkling (mini-swe-agent) xhigh56.90%(mini-swe-agent) xhigh
11Haiku 4.5 (Claude Code) xhigh25.50%(Claude Code) xhigh
Source: labs.scale.com/leaderboard/swe_bench_pro_public_v2

No published result tracked

36 of the 46 models on this site have no published number on this benchmark, so their cells in the home table stay blank rather than reading as zero.

Step 5 PreviewGPT-6 LunaGLM 5.3 FlashQwen3.8 2.4T A95BQwen3.8-Flash-NextMiMo-V2.6-FlashQwen3.8 Max (0902)Claude Fable 5Grok 4.6Claude Opus 4.8Claude Opus 4.7Gemini 3.7 FlashMuse Spark 1.2GPT-5.4Grok 4.5GPT-5.5DeepSeek V4.1 Flash 1017GLM-LegalClaude Haiku 5.5BeamGemini 4 ArgonClaude Sonnet 5.5Claude Opus 5.5GPT-6 SolGPT-6.1 SolGPT-5.6 LunaGemini 3.1 ProGrok 4.7DeepSeek V4.1 FlashDeepSeek V4 ProQwen3.8 MaxMuse Spark 1.3 MaxMiniMax M3Hunyuan Hy4 PreviewMiMo V2.6 ProMistral Medium 3.5

Read this with

Other versions of the same evaluation. Versions are not comparable: the task set changes between them.

  • SWE-Bench Pro V2 FullSame family. Repository issue resolution on the full public V2 set. Models must edit an existing codebase to satisfy the issue’s verification tests.
  • SWE-Bench Pro (version unspecified)Same category. Resolving long-horizon repository issues requiring large, multi-file code changes.
  • SWE-bench MultimodalSame category. Resolving software-engineering issues that include visual context such as screenshots and design mockups.
  • SWE-Bench Pro v1Same category. Resolving software-engineering issues in existing repositories.
  • SWE-bench MultilingualSame category. Resolving issues in existing software repositories across several programming languages.

As reported by each publisher

The same benchmark, measured by the labs themselves. Compare down a column, never across two publishers.

Reflection AI

Introducing Beam: model capability comparisons · 5 October 2026

RankModelScore
3Beam77.2%
4Inkling56.9%
2GLM 5.384.3%
1Kimi K388.2%

Ranks compare reported scores in the same published row; ties share a rank.

Reflection AI’s four capability tables, read 6 October 2026. All 21 rows and eight model columns are transcribed; NR means not reported and remains blank. The source’s Qwen 3.8 Max column uses this catalogue’s Qwen3.8 Max spelling. Beam results are publisher reports; Reflection AI cites Artificial Analysis and Datacurve for comparison-model evaluations. The announcement does not specify a common harness, effort or trial count for every row. Ranks are derived within each published row from reported values only, with ties sharing a rank; they do not establish an independent or overall model ranking. AutomationBench uses the public split; AA-Omniscience is an index on the public split only, calculated by Reflection AI. BrowseComp and DeepSearchQA use context management. AA-LCR has no stated version. These settings remain separate from other source fields.

reflection.ai/blog/introducing-beam

Sources

Every page used for this benchmark, read on 17 October 2026.