AIEvals

Benchmarks

SciCode

Scientist-curated code generation across scientific disciplines.

Scientific codingReported independently by Artificial Analysis144 models run by Artificial AnalysisUpdated 8 October 2026Reported by Reflection AI

How it is run

The benchmark contains 288 test-set subproblems drawn from 80 laboratory problems across 16 scientific disciplines. This table reports AA’s independent results. Highest published effort per release; the table retains its snapshot date.

What it cannot tell you

Effort settings and evaluation versions matter. These source-specific results are kept separate from Vals AI and vendor reports. A missing score stays blank.

Leader
Claude Opus 5.566.90% · Artificial Analysis results, grouped here by release
Top-ten spread
8.10%Leader minus tenth place, or last place in smaller fields.
Median
44.79%of the 144 rows published
No published result
6of 46 models on this site

Artificial Analysis results by release

Highest published effort per model release, grouped here from AA’s public results. Grouping ranks are derived here; source settings and model-publisher reports stay separate.

#ModelScoreEffort
1Claude Opus 5.566.90%max
2Claude Fable 5.163.08%max
3Gemini 4 Argon61.81%high
4Claude Sonnet 5.561.00%max
4Claude Fable 561.00%max
6MiMo-V2.6-Pro60.88%Not specified
7Kimi K359.49%max
8GLM-5.359.03%max
9Step 5 Preview58.91%Not specified
10Muse Spark 1.358.80%max
10Muse Spark 1.158.80%xhigh
12Gemini 3.1 Pro Preview58.68%Not specified
13GPT-6 Sol57.64%max
14Muse Spark 1.257.41%xhigh
14Grok 4.757.41%xhigh
16Gemini 3.7 Flash57.18%high
17GPT-5.6 Sol57.06%max
18Gemini 3.8 Flash56.60%high
19GPT-6 Astra56.48%max
20Claude Opus 556.37%max
21GPT-5.555.79%xhigh
22GPT-5.6 Terra54.98%max
22Grok 4.554.98%high
22Claude Haiku 5.554.98%max
25GPT-6 Luna54.63%max
26Claude Opus 4.854.40%max
27Claude Sonnet 554.28%max
28GPT-6.1 Sol54.17%max
29Qwen3.8 2.4T A95B54.05%Not specified
30Gemini 3.5 Flash53.94%high
31GPT-5.6 Luna53.59%max
32Gemini 3.6 Flash53.36%high
33Qwen3.8 Max53.24%Not specified
34Grok 4.653.01%xhigh
35GPT-5.5 Instant (June 2026)52.55%Not specified
36Qwen3.8 Max (0902)52.08%Not specified
36GPT-5.4 mini52.08%xhigh
38JT-4.1 Flash 236B A21B51.97%Not specified
39DeepSeek V4.1 Flash51.85%max
40GLM 5.3 Flash51.62%max
41Kimi K2.651.50%Not specified
42MiMo-V2.6-Flash51.27%Not specified
43GLM-5.251.16%max
44DeepSeek V4 Pro 081351.04%max
45DeepSeek V4 Pro 042450.81%max
46MiMo-V2.5-Pro50.58%Not specified
46Qwen3.8-Flash-Next50.58%Not specified
48DeepSeek V4 Flash 073150.35%max
49MiniMax-M2.750.12%Not specified
49Claude Sonnet 4.650.12%max
51Inkling Small49.65%Not specified
51DeepSeek V4 Flash Vision49.65%max
53Qwen3.7 Max49.54%Not specified
54Hy348.61%Not specified
55Grok 4.348.26%high
56Quasar 438B48.15%max
57Solar Open2 250B48.03%Not specified
58Kimi K2.7 Code47.80%Not specified
59Solar Mini 447.57%Not specified
60GPT-5.4 nano47.22%xhigh
61MiniMax-M347.11%Not specified
62Inkling46.99%xhigh
63Qwen3.8 27B46.64%xhigh
64Gemini 2.5 Pro46.30%Not specified
65Qwen3.7 Plus46.06%Not specified
66Claude 4.5 Sonnet45.72%Not specified
67Apodex 1.145.49%Not specified
67Gemma 4 31B45.49%Not specified
69DeepSeek V4 Flash 042045.25%max
70Ring-2.6-1T45.02%xhigh
71Muse Glimmer44.91%high
72Qwen3.5 397B A17B44.79%Not specified
72GLM-5.144.79%Not specified
74Solar Pro 444.56%Not specified
75Ling-3.0-flash-VL44.21%Not specified
76MiMo-V2.543.87%Not specified
76Step 3.7 Flash43.87%high
78Gemini 3.1 Flash-Lite43.40%high
79Nex-N2-Pro43.29%Not specified
80K2 Horizon 375B A23B42.94%Not specified
81o3-mini42.82%high
81Qwen3.6 27B42.82%Not specified
83Ling-3.0-flash-Fin42.36%Not specified
84Claude 4.5 Haiku42.25%Not specified
84Motif 342.25%Not specified
86K-EXAONE 2.0 080342.01%Not specified
86Ling 3.0 Flash42.01%Not specified
88Qwen3 235B A22B 2507 (Reasoning)41.44%Not specified
89Gemini 3.5 Flash-Lite41.32%high
90A.X-K240.97%Not specified
91Trinity Large Thinking40.63%Not specified
92Nemotron 3 Ultra 550B A55B40.28%Not specified
93Mistral Medium 3.540.16%high
94K2 Horizon MoVA 36B A4B40.05%Not specified
94Gemma 4 26B A4B40.05%Not specified
96Qwen3.5 122B A10B39.70%Not specified
97GPT-5 mini39.00%high
97DeepSeek V3 032439.00%Not specified
99gpt-oss-20b38.89%high
100North Mini Code38.77%Not specified
100Mistral Small 438.77%Not specified
102Command A+38.54%Not specified
102Mercury 2.538.54%Not specified
104DeepSeek R1 (Jan '25)38.31%Not specified
105DeepSeek V3.1 Terminus37.96%Not specified
106Granite 4.2 30B37.85%Not specified
107Mercury 237.73%high
108G9v3-39A5B36.81%Not specified
109Mistral Large 336.57%Not specified
109Qwen3.6 35B A3B36.57%Not specified
111LongCat 2.036.34%Not specified
112Nemotron 3 Super 120B A12B36.23%Not specified
112Qwen3 Coder Next36.23%Not specified
114Qwen3 32B36.00%Not specified
115DeepSeek V3 (Dec '24)35.76%Not specified
116gpt-oss-120b34.03%high
117Qwen3 30B A3B 2507 (Reasoning)32.99%Not specified
118Devstral 232.75%Not specified
119Mistral Medium 3.132.41%Not specified
119Devstral Small 232.41%Not specified
121Nemotron 3.5 Lightning32.06%Not specified
122Llama 4 Maverick31.71%Not specified
123Granite 4.2 8B31.48%Not specified
124Qwen3 14B30.67%Not specified
125NVIDIA Nemotron 3 Nano 30B A3B30.56%Not specified
126Qwen3.5 9B29.51%Not specified
127Mistral Small 3.228.59%Not specified
128Qwen3 8B27.89%Not specified
129Mistral Small 3.127.78%Not specified
130K2 Horizon 7B27.55%Not specified
131MiniCPM5-2B26.27%Not specified
132Solar Pro 325.46%high
133Granite 4.2 3B25.35%Not specified
134Ling 3.0 Tiny24.19%Not specified
135Ministral 3 14B23.84%Not specified
136Gemma 3 27B Instruct23.26%Not specified
137K2 Horizon 3.7B21.99%Not specified
138Celeris-121.64%Not specified
139Llama 4 Scout21.30%Not specified
140Ministral 3 8B20.72%Not specified
141Gemma 3 12B Instruct16.44%Not specified
142Ministral 3 3B15.28%Not specified
143LFM2.5-2.6B14.35%Not specified
144K2 Horizon 0.9B6.94%Not specified

No published result tracked

6 of the 46 models on this site have no published number on this benchmark, so their cells in the home table stay blank rather than reading as zero.

Read this with

Nothing else on this site shares this benchmark's family or category, so these are other evaluations the same publisher reports.

As reported by each publisher

The same benchmark, measured by the labs themselves. Compare down a column, never across two publishers.

Reflection AI

Introducing Beam: model capability comparisons · 5 October 2026

RankModelScore
5Beam49.7%
6Inkling46.1%
7Nemotron 3 Ultra44.6%
1GLM 5.359.0%
2Kimi K358.7%
3Qwen3.8 Max52.1%
4DeepSeek V4.1 Flash52.0%

Ranks compare reported scores in the same published row; ties share a rank.

Reflection AI’s four capability tables, read 6 October 2026. All 21 rows and eight model columns are transcribed; NR means not reported and remains blank. The source’s Qwen 3.8 Max column uses this catalogue’s Qwen3.8 Max spelling. Beam results are publisher reports; Reflection AI cites Artificial Analysis and Datacurve for comparison-model evaluations. The announcement does not specify a common harness, effort or trial count for every row. Ranks are derived within each published row from reported values only, with ties sharing a rank; they do not establish an independent or overall model ranking. AutomationBench uses the public split; AA-Omniscience is an index on the public split only, calculated by Reflection AI. BrowseComp and DeepSearchQA use context management. AA-LCR has no stated version. These settings remain separate from other source fields.

reflection.ai/blog/introducing-beam

Sources

Every page used for this benchmark, read on 17 October 2026.