SWE-bench (Verified)

Name: SWE-bench (Verified)
Creator: Princeton and Stanford researchers

Whether a model can write a patch that resolves a real GitHub issue, judged by running the project's own unit tests.

Software engineeringPrinceton and Stanford researchers88 models run by Vals AIUpdated 1 September 2026Reported by Reflection AI

How it is run

500 tasks, each in an isolated container. Vals uses a minimal bash-only harness so the score reflects the model rather than a heavily tuned agent scaffold.

What it cannot tell you

The benchmark measures the harness as much as the model, and labs use different harnesses. Earlier 2026 releases (Gemini 3 Pro) scored higher than newer ones, so a lower number is not automatically a worse model.

Leader: Claude Opus 597.00% · Vals AI's own list
Top-ten spread: 5.00%Leader minus tenth place, or last place in smaller fields.
Median: 86.60%of the 23 models on this site
No published result: 23of 46 models on this site

Independent leaderboard · Vals AI

Top 88 of 88 runs by Vals AI, one harness.

#	Model	Score
1	Claude Opus 5	97.00%
2	DeepSeek V4 Pro	96.40%
3	GPT-5.6 Sol	96.20%
4	Grok 4.6	95.60%
5	GPT-5.6 Terra	95.40%
5	GLM 5.3	95.40%
7	Claude Fable 5	95.00%
8	Kimi K3	93.40%
9	GPT-5.6 Luna	93.00%
10	GLM 5.3 Flash	92.00%
11	Deepseek V4 Flash 0731	88.80%
12	Claude Opus 4.8	88.60%
13	Grok 4.5	86.60%
13	Muse Spark 1.2	86.60%
15	Qwen3.8 27b	86.00%
16	Claude Opus 4 8 Claude Code	85.80%
17	Qwen3.8 Max	85.60%
18	Glm 5.2	82.80%
19	GPT-5.5	82.60%
20	Inkling Small	82.20%
21	Muse Spark 1 1	82.00%
21	Claude Opus 4.7	82.00%
23	Gemini 3.7 Flash	80.80%
24	Gemini 3.8 Flash	80.00%
25	Composer 2.5	79.60%
25	Gemini 3.6 Flash	79.60%
25	Claude Sonnet 5	79.60%
28	Gemini 3.5 Flash	78.80%
28	Gemini 3.1 Pro	78.80%
30	Gpt 5.4 2026 03 05	78.20%
30	Claude Opus 4 6 Thinking	78.20%
30	Kimi K2.7 Code	78.20%
33	Gpt 5.3 Codex	78.00%
34	Inkling	77.60%
35	Claude Sonnet 4 6	77.40%
35	Deepseek V4 Pro	77.40%
37	Gpt 5.5 Codex	76.40%
37	Claude Opus 4 5 20251101 Thinking	76.40%
37	Gemini 3 Pro Preview	76.40%
37	Glm 5.1	76.40%
41	Gpt 5.5 Factory	76.20%
41	Kimi K2.6	76.20%
43	Gpt 5.2 2025 12 11	75.80%
44	Gemini 3 Flash Preview	75.00%
44	Gemini 3.5 Flash Lite	75.00%
44	MiniMax M3	75.00%
47	MiniMax M2.1	74.80%
48	Muse Spark	74.40%
49	MiniMax M2.5	74.20%
50	Mimo V2.5 Pro	74.00%
51	MiniMax M2.7	73.80%
52	Qwen3.6 Plus	73.40%
53	Gpt 5.4 Mini 2026 03 17	73.00%
54	Qwen3.6 Max Preview	72.80%
55	Gpt 5.2 Codex	72.40%
56	Grok 4.20 0309 Reasoning	72.20%
57	Grok 4.3	71.40%
57	Glm 5 Thinking	71.40%
59	Qwen3.5 Plus Thinking	71.20%
60	Mimo V2.5	71.00%
61	Kimi K2.5 Thinking	70.00%
61	Claude Sonnet 4 5 20250929 Thinking	70.00%
61	Qwen3.6 27b	70.00%
64	Gpt 5.4 Nano 2026 03 17	69.80%
64	Gpt 5.1 2025 11 13	69.80%
66	Glm 4.7	69.40%
67	Gpt 5 2025 08 07	69.00%
67	Nemotron 3 Ultra 550b A55b	69.00%
69	Qwen3.7 Max	68.80%
70	Deepseek V3p2 Thinking	67.60%
71	Claude Haiku 4 5 20251001 Thinking	66.60%
72	Mistral Medium 3.5	66.40%
73	Ling 3.0 Flash 2607	65.20%
74	Qwen3.5 Flash	64.40%
75	Gemini 3.1 Flash Lite Preview	62.80%
75	Devstral 2512	62.80%
77	Gpt 5 Mini 2025 08 07	60.80%
78	Kimi K2 Thinking	60.20%
79	Grok 4 0709	57.80%
80	Laguna M.1	57.60%
81	Laguna Xs.2	55.20%
82	Gemini 2.5 Pro	54.40%
83	Nemotron Lightning 3p5 30b A3b	52.80%
84	Grok 4 Fast Reasoning	45.40%
85	Grok 4 1 Fast Reasoning	41.40%
85	Mistral Large 2512	41.40%
87	Gpt Oss 120b	33.60%
88	Command A 03 2025	7.80%

Source: www.vals.ai/benchmarks/swebench

The same model, two sources

Independent and vendor figures for the same model, widest gap first. Run settings differ.

Model	Independent	As published	Gap	Publisher
Inkling	77.60%	77.60%	0.00%	Reflection AI

Gap is the publisher’s figure minus the independent one, in this benchmark’s own points. Both numbers stay visible: averaging them would invent a figure nobody published.

No published result tracked

23 of the 46 models on this site have no published number on this benchmark, so their cells in the home table stay blank rather than reading as zero.

Read this with

Other versions of the same evaluation. Versions are not comparable: the task set changes between them.

As reported by each publisher

The same benchmark, measured by the labs themselves. Compare down a column, never across two publishers.

Reflection AI

Introducing Beam: model capability comparisons · 5 October 2026

Rank	Model	Score
1	Beam	80.9%
2	Inkling	77.6%
3	Nemotron 3 Ultra	70.7%

Ranks compare reported scores in the same published row; ties share a rank.

Reflection AI’s four capability tables, read 6 October 2026. All 21 rows and eight model columns are transcribed; NR means not reported and remains blank. The source’s Qwen 3.8 Max column uses this catalogue’s Qwen3.8 Max spelling. Beam results are publisher reports; Reflection AI cites Artificial Analysis and Datacurve for comparison-model evaluations. The announcement does not specify a common harness, effort or trial count for every row. Ranks are derived within each published row from reported values only, with ties sharing a rank; they do not establish an independent or overall model ranking. AutomationBench uses the public split; AA-Omniscience is an index on the public split only, calculated by Reflection AI. BrowseComp and DeepSearchQA use context management. AA-LCR has no stated version. These settings remain separate from other source fields.

reflection.ai/blog/introducing-beam

Sources

Every page used for this benchmark, read on 17 October 2026.

Benchmark site or the page that publishes itwww.swebench.com/
Independent resultswww.vals.ai/benchmarks/swebench
Reflection AI tablereflection.ai/blog/introducing-beam