IFBench

Name: IFBench
Creator: Allen Institute for AI

Generalization to 58 verifiable output constraints absent from conventional instruction tests.

Instruction followingAllen Institute for AI331 models run by Artificial AnalysisUpdated 30 September 2026Reported by Reflection AI

How it is run

Artificial Analysis independently scores responses against the verifiable constraints.

What it cannot tell you

Constraint compliance does not measure factual correctness or general agent capability.

Leader: Grok 4.20 030982.93% · Artificial Analysis results, grouped here by release
Top-ten spread: 4.15%Leader minus tenth place, or last place in smaller fields.
Median: 42.99%of the 331 rows published
No published result: 36of 46 models on this site

Artificial Analysis results by release

Highest published effort per model release, grouped here from AA’s public results. Grouping ranks are derived here; source settings and model-publisher reports stay separate.

#	Model	Score	Effort
1	Grok 4.20 0309	82.93%	Not specified
2	MiniMax-M3	82.86%	Not specified
3	Nemotron 3 Ultra 550B A55B	81.36%	Not specified
4	Grok 4.3	81.29%	high
5	Qwen3.7 Max	80.54%	Not specified
6	Nemotron Cascade 2 30B A3B	80.41%	Not specified
7	MiMo-V2.5-Pro	79.86%	Not specified
8	DeepSeek V4 Flash 0420	79.18%	max
9	Nova 2.0 Pro Preview	79.05%	medium
10	Qwen3.5 397B A17B	78.78%	Not specified
11	Qwen3.7 Plus	77.96%	Not specified
12	GPT-5.2 Codex	77.62%	xhigh
13	Gemini 3.1 Flash-Lite	77.21%	high
14	Gemini 3.1 Pro Preview	77.14%	Not specified
15	Qwen3.6 Max Preview	76.60%	Not specified
16	DeepSeek V4 Pro 0424	76.46%	max
17	Gemini 3.5 Flash	76.33%	high
18	GLM-5.1	76.26%	Not specified
19	Kimi K2.6	75.99%	Not specified
20	GPT-5.4 nano	75.92%	xhigh
20	Muse Spark	75.92%	Not specified
22	GPT-5.5	75.85%	xhigh
23	MiniMax-M2.7	75.71%	Not specified
24	Qwen3.5 27B	75.58%	Not specified
24	Gemma 4 31B	75.58%	Not specified
26	GPT-5 mini	75.44%	high
26	GPT-5.2	75.44%	xhigh
28	GPT-5.3 Codex	75.37%	xhigh
29	Qwen3.6 Plus	75.17%	Not specified
30	GPT-5 Codex	74.15%	high
31	Command A+	73.95%	Not specified
31	GPT-5.4	73.95%	xhigh
33	GLM-5.2	73.33%	max
34	GPT-5.4 mini	73.27%	xhigh
35	GLM-5-Turbo	73.20%	Not specified
36	GPT-5	73.06%	high
37	GPT-5.1	72.86%	high
38	GPT-5.6 Sol	72.65%	max
39	Qwen3.5 35B A3B	72.52%	Not specified
40	Gemma 4 26B A4B	72.45%	Not specified
41	GLM-5	72.31%	Not specified
41	MiniMax-M2	72.31%	Not specified
43	MiMo-V2-Flash (Feb 2026)	71.84%	Not specified
44	MiniMax-M2.5	71.63%	Not specified
45	Nemotron 3 Super 120B A12B	71.50%	Not specified
45	GPT-5.5 Instant (May 2026)	71.50%	Not specified
47	o3	71.43%	Not specified
48	GPT-5.6 Terra	71.22%	max
49	Solar Pro 3	71.20%	high
50	Nova 2.0 Lite	70.75%	high
50	Qwen3 Max Thinking	70.75%	Not specified
52	Gemini 3 Pro Preview	70.41%	high
53	o1	70.34%	Not specified
54	Kimi K2.5	70.20%	Not specified
55	GPT-5.1 Codex	70.00%	high
56	MiniMax-M2.1	69.86%	Not specified
57	Mercury 2	69.80%	high
58	Apriel-v1.6-15B-Thinker	69.12%	Not specified
59	gpt-oss-120b	68.98%	high
60	MiMo-V2-Pro	68.84%	Not specified
61	Mistral Medium 3.5	68.78%	high
62	o4-mini	68.71%	high
63	KAT-Coder-Pro V1	68.37%	Not specified
64	Kimi K2 Thinking	68.10%	Not specified
65	GLM-4.7	67.89%	Not specified
65	GPT-5.1 Codex mini	67.89%	high
67	GPT-5 nano	67.55%	high
68	MiMo-V2-Omni-0327	67.35%	Not specified
69	Step 3.7 Flash	67.28%	high
70	o3-mini	67.14%	high
70	MiMo-V2.5	67.14%	Not specified
72	KAT Coder Pro V2	66.67%	Not specified
73	Step 3.5 Flash 2603	66.53%	Not specified
74	HyperNova 60B 2605	66.46%	high
75	Nova 2.0 Omni	66.19%	medium
75	Nex-N2-Pro	66.19%	Not specified
77	Olmo 3.1 32B Think	65.99%	Not specified
78	gpt-oss-20b	65.10%	high
79	K-EXAONE	64.69%	Not specified
80	Step 3.5 Flash	64.56%	Not specified
81	DeepSeek V3.2 Speciale	63.88%	Not specified
82	Claude Fable 5	63.47%	max
83	Nemotron 3 Nano Omni 30B A3B Reasoning	63.20%	Not specified
84	Hy3-preview	63.13%	Not specified
84	Kimi K2.7 Code	63.13%	Not specified
86	K2 Think V2	62.79%	high
87	Claude Opus 4.8	62.24%	max
88	Apriel-v1.5-15B-Thinker	61.70%	Not specified
89	GLM 5V Turbo	61.09%	Not specified
90	GLM-4.7-Flash	60.82%	Not specified
91	Qwen3 Next 80B A3B (Reasoning)	60.68%	Not specified
92	K2-V2	60.14%	high
93	DiffusionGemma 26B A4B	59.46%	Not specified
94	Qwen3 VL 32B (Reasoning)	59.39%	Not specified
95	Claude Opus 4.7	58.64%	max
96	NVIDIA Nemotron 3 Nano 4B	58.23%	Not specified
97	Claude Opus 4.5	57.96%	Not specified
97	EXAONE 4.5 33B	57.96%	Not specified
99	Solar Open 100B	57.69%	high
100	North Mini Code	57.55%	Not specified
101	Ling 2.6 Flash	57.41%	Not specified
102	Claude 4.5 Sonnet	57.28%	Not specified
103	DeepSeek V3.1 Terminus	57.01%	Not specified
103	Motif-2-12.7B-Reasoning	57.01%	Not specified
105	Ling-2.6-1T	56.87%	Not specified
106	Claude Sonnet 4.6	56.60%	max
107	Qwen3 VL 235B A22B (Reasoning)	56.46%	Not specified
108	Trinity Large Thinking	56.26%	Not specified
109	LFM2.5-8B-A1B	55.65%	Not specified
110	Claude 4.1 Opus	55.44%	Not specified
111	Gemini 3 Flash Preview	55.10%	Not specified
112	Claude 4 Sonnet	54.69%	Not specified
113	Tri-21B-Think	54.63%	Not specified
114	Falcon-H1R-7B	54.42%	Not specified
115	Claude 4.5 Haiku	54.29%	Not specified
116	Qwen3 Max Thinking (Preview)	53.81%	Not specified
117	Grok 4	53.67%	Not specified
118	MiMo-V2-Omni	53.54%	Not specified
119	Claude Opus 4.6	53.13%	max
120	Grok 4.1 Fast	52.72%	Not specified
121	Jamba Reasoning 3B	52.45%	Not specified
122	Doubao Seed Code	51.43%	Not specified
123	Qwen3 235B A22B 2507 (Reasoning)	51.22%	Not specified
124	Qwen3.5 Omni Plus	51.16%	Not specified
125	Qwen3.5 122B A10B	50.82%	Not specified
126	Qwen3 30B A3B 2507 (Reasoning)	50.68%	Not specified
127	Gemini 2.5 Flash	50.27%	Not specified
128	Step3 VL 10B	50.20%	Not specified
129	Qwen3 4B 2507 (Reasoning)	49.80%	Not specified
130	Grok 4.20 0309 v2	49.32%	Not specified
130	MiniCPM5-1B	49.32%	Not specified
130	Mi:dm K 2.5 Pro	49.32%	Not specified
133	Olmo 3 32B Think	49.12%	Not specified
134	DeepSeek V3.2	49.05%	Not specified
135	Gemini 2.5 Pro	48.71%	Not specified
136	Claude 3.7 Sonnet	48.30%	Not specified
137	Qwen3 Max (Preview)	48.03%	Not specified
138	Tri-21B-think Preview	47.14%	Not specified
139	Llama 3.3 Instruct 70B	47.07%	Not specified
140	Grok 3	46.94%	Not specified
141	Cogito v2.1	46.26%	Not specified
142	Qwen3 235B A22B 2507	46.05%	Not specified
143	LFM2 24B A2B	45.85%	Not specified
143	Grok 3 mini Reasoning	45.85%	high
145	Qwen3.6 27B	45.71%	Not specified
146	Mi:dm K 2.5 Pro Preview	45.58%	Not specified
147	Gemma 4 12B	45.17%	Not specified
148	Qwen3 VL 30B A3B (Reasoning)	45.10%	Not specified
149	GPT-5 (ChatGPT)	45.03%	Not specified
150	Ring-1T	44.56%	Not specified
150	Ring-2.6-1T	44.56%	xhigh
152	Magistral Small 1.2	44.35%	Not specified
152	Granite 4.1 30B	44.35%	Not specified
154	Qwen3 Max	44.15%	Not specified
155	GLM-4.5	44.08%	Not specified
156	LFM2.5-1.2B-Instruct	43.81%	Not specified
157	Gemini 2.5 Flash Preview (Sep '25)	43.54%	Not specified
158	GLM-4.6	43.40%	Not specified
158	Qwen3 Omni 30B A3B (Reasoning)	43.40%	Not specified
160	Ring-flash-2.0	43.27%	Not specified
160	Claude 4 Opus	43.27%	Not specified
162	DeepSeek V3.2 Exp	43.13%	Not specified
163	LongCat Flash Lite	43.06%	Not specified
164	GPT-4.1	42.99%	Not specified
164	Magistral Medium 1.2	42.99%	Not specified
164	Llama 4 Maverick	42.99%	Not specified
167	Claude 3.5 Haiku	42.79%	Not specified
168	Qwen3 VL 235B A22B Instruct	42.65%	Not specified
169	JT-35B-Flash	42.04%	Not specified
170	Seed-OSS-36B-Instruct	41.90%	Not specified
171	LFM2.5-1.2B-Thinking	41.84%	Not specified
171	Gemini 2.5 Flash-Lite Preview (Sep '25)	41.84%	Not specified
173	MiniMax M1 80k	41.77%	Not specified
174	Kimi K2 0905	41.70%	Not specified
175	DeepSeek V3.1	41.50%	Not specified
175	Olmo 3 7B Think	41.50%	Not specified
175	Qwen3 30B A3B	41.50%	Not specified
175	Kimi K2	41.50%	Not specified
179	Grok Code Fast 1	41.36%	Not specified
179	ERNIE 5.0 Thinking Preview	41.36%	Not specified
181	MiniMax M1 40k	41.22%	Not specified
182	DeepSeek V3 0324	41.02%	Not specified
183	Qwen3 14B	40.54%	Not specified
183	Gemma 4 E4B	40.54%	Not specified
185	Qwen3 Coder 480B A35B Instruct	40.48%	Not specified
186	Gemini 2.0 Flash (Feb '25)	40.20%	Not specified
187	MiMo-V2-Flash	39.93%	Not specified
188	Qwen3 VL 8B (Reasoning)	39.86%	Not specified
189	Mistral Medium 3.1	39.80%	Not specified
190	Qwen3 Next 80B A3B Instruct	39.66%	Not specified
191	DeepSeek R1 0528 (May '25)	39.59%	Not specified
192	Llama 4 Scout	39.52%	Not specified
193	Mistral Medium 3	39.25%	Not specified
194	Qwen3 VL 32B Instruct	39.18%	Not specified
194	Olmo 3.1 32B Instruct	39.18%	Not specified
196	ERNIE 4.5 300B A47B	39.12%	Not specified
197	Llama 3.1 Instruct 405B	39.05%	Not specified
198	DeepSeek R1 (Jan '25)	38.98%	Not specified
199	QwQ 32B	38.78%	Not specified
200	Granite 4.1 8B	38.64%	Not specified
201	GPT-4.1 mini	38.30%	Not specified
202	Llama 3.1 Nemotron Ultra 253B v1	38.16%	Not specified
203	Devstral 2	38.10%	Not specified
203	Llama 3.3 Nemotron Super 49B v1	38.10%	Not specified
203	OLMo 2 32B	38.10%	Not specified
203	Nova Pro	38.10%	Not specified
207	Qwen3.5 Omni Flash	37.96%	Not specified
208	HyperCLOVA X SEED Think (32B)	37.89%	Not specified
209	Qwen3.5 9B	37.82%	Not specified
210	Grok 4 Fast	37.69%	Not specified
211	GLM-4.5-Air	37.55%	Not specified
212	NVIDIA Nemotron 3 Nano 30B A3B	37.48%	Not specified
213	Solar Pro 2	37.07%	high
213	Llama 3 Instruct 70B	37.07%	Not specified
215	Qwen2.5 Instruct 72B	36.87%	Not specified
216	Gemma 3 12B Instruct	36.73%	Not specified
217	JT-MINI	36.67%	Not specified
218	Qwen3 235B A22B	36.60%	Not specified
218	Qwen3 VL 4B (Reasoning)	36.60%	Not specified
220	Command A	36.46%	Not specified
221	Qwen3 32B	36.33%	Not specified
222	Mistral Large 3	36.19%	Not specified
222	Qwen3.6 35B A3B	36.19%	Not specified
222	Nova Premier	36.19%	Not specified
225	Claude 3 Haiku	36.12%	Not specified
226	GPT-4o (Aug '24)	35.99%	Not specified
227	Nanbeige4.1-3B	35.44%	Not specified
228	Qwen3 Coder Next	35.24%	Not specified
229	Jamba 1.7 Large	35.17%	Not specified
230	Ling-1T	34.76%	Not specified
230	DeepSeek V3 (Dec '24)	34.76%	Not specified
232	Devstral Small (Jul '25)	34.56%	Not specified
233	Pixtral Large	34.49%	Not specified
234	Llama 3.1 Instruct 70B	34.42%	Not specified
235	Sarvam 105B	34.35%	high
235	Ling-flash-2.0	34.35%	Not specified
237	GPT-4o (Nov '24)	34.29%	Not specified
238	Nova Lite	34.15%	Not specified
239	INTELLECT-3	34.01%	Not specified
240	Granite 4.1 3B	33.67%	Not specified
241	Gemma 4 E2B	33.61%	Not specified
242	Qwen3 4B 2507	33.54%	Not specified
243	EXAONE 4.0 32B	33.47%	Not specified
243	Mistral Small 3.2	33.47%	Not specified
243	Qwen3 8B	33.47%	Not specified
246	Qwen3.5 4B	33.27%	Not specified
247	GPT-4	33.20%	Not specified
248	LFM2.5-VL-1.6B	33.13%	Not specified
248	Qwen3 VL 30B A3B Instruct	33.13%	Not specified
250	Qwen3 30B A3B 2507	33.06%	Not specified
251	Llama Nemotron Super 49B v1.5	32.93%	Not specified
252	Mistral Small 4	32.79%	Not specified
252	Olmo 3 7B Instruct	32.79%	Not specified
254	Hermes 4 - Llama-3.1 405B	32.72%	Not specified
255	Qwen3 Coder 30B A3B Instruct	32.65%	Not specified
256	Qwen3 4B	32.52%	Not specified
257	Qwen3 VL 8B Instruct	32.31%	Not specified
258	Ministral 3 14B	32.04%	Not specified
258	GPT-4.1 nano	32.04%	Not specified
260	NVIDIA Nemotron Nano 12B v2 VL	31.90%	Not specified
261	Gemma 3 27B Instruct	31.84%	Not specified
261	Sarvam M	31.84%	high
261	Qwen3 VL 4B Instruct	31.84%	Not specified
264	Mistral Large 2 (Jul '24)	31.63%	Not specified
264	Devstral Small (May '25)	31.63%	Not specified
266	Gemini 2.5 Flash-Lite	31.50%	Not specified
266	Granite 4.0 H Small	31.50%	Not specified
268	Jamba 1.7 Mini	31.36%	Not specified
269	Mistral Large 2 (Nov '24)	31.22%	Not specified
270	Qwen3 Omni 30B A3B Instruct	31.16%	Not specified
270	Devstral Small 2	31.16%	Not specified
272	GPT-4o mini	30.95%	Not specified
273	Llama 3.1 Nemotron Instruct 70B	30.75%	Not specified
274	Llama 3.2 Instruct 11B (Vision)	30.41%	Not specified
274	Reka Flash 3	30.41%	Not specified
276	Mistral Small 3.1	29.93%	Not specified
276	Devstral Medium	29.93%	Not specified
278	Nova Micro	29.39%	Not specified
279	Qwen3.5 2B	29.12%	Not specified
279	Ministral 3 8B	29.12%	Not specified
281	Hermes 4 - Llama-3.1 70B	28.98%	Not specified
282	GLM-4.5V	28.64%	Not specified
283	Llama 3.1 Instruct 8B	28.57%	Not specified
284	Gemma 3 4B Instruct	28.30%	Not specified
285	Kimi Linear 48B A3B Instruct	28.10%	Not specified
286	GLM-4.6V	27.89%	Not specified
286	Gemma 3n E4B Instruct	27.89%	Not specified
288	DeepSeek R1 Distill Llama 70B	27.55%	Not specified
289	NVIDIA Nemotron Nano 9B V2	27.14%	Not specified
290	Molmo2-8B	26.94%	Not specified
291	Ministral 3 3B	26.80%	Not specified
292	MiniCPM-V 4.6 1.3B	26.73%	Not specified
293	Sarvam 30B	26.46%	high
294	Mistral Small 3	26.39%	Not specified
295	LFM2 8B A1B	26.26%	Not specified
296	Llama 3.2 Instruct 3B	26.19%	Not specified
296	Granite 4.0 H 1B	26.19%	Not specified
298	Apertus 70B Instruct	25.85%	Not specified
299	Llama 3.1 Nemotron Nano 4B v1.1	25.51%	Not specified
300	Exaone 4.0 1.2B	25.31%	Not specified
301	Magistral Medium 1	25.10%	Not specified
302	Magistral Small 1	24.76%	Not specified
302	Granite 4.0 Micro	24.76%	Not specified
304	Llama 3 Instruct 8B	24.56%	Not specified
305	OLMo 2 7B	24.42%	Not specified
306	Phi-3 Mini Instruct 3.8B	23.88%	Not specified
307	Ling-mini-2.0	23.61%	Not specified
308	Phi-4	23.54%	Not specified
309	DeepSeek R1 Distill Qwen 32B	22.93%	Not specified
310	Llama 3.2 Instruct 1B	22.79%	Not specified
311	Granite 3.3 8B	22.45%	Not specified
312	Apertus 8B Instruct	22.38%	Not specified
313	DeepSeek R1 Distill Qwen 14B	22.11%	Not specified
314	Gemma 3n E2B Instruct	22.04%	Not specified
315	LFM2 1.2B	21.97%	Not specified
316	Qwen3 0.6B	21.90%	Not specified
317	Qwen3.5 0.8B	21.56%	Not specified
318	Qwen3 1.7B	21.09%	Not specified
318	Phi-4 Mini Instruct	21.09%	Not specified
320	Granite 4.0 1B	20.54%	Not specified
321	Tiny Aya Global	20.14%	Not specified
322	Mistral 7B Instruct	19.93%	Not specified
322	Gemma 3 1B Instruct	19.93%	Not specified
324	DeepSeek R1 0528 Qwen3 8B	19.86%	Not specified
325	Molmo 7B-D	19.66%	Not specified
326	LFM2 2.6B	19.52%	Not specified
327	DeepSeek R1 Distill Llama 8B	17.55%	Not specified
327	Granite 4.0 H 350M	17.55%	Not specified
329	Granite 4.0 350M	15.92%	Not specified
330	DeepSeek R1 Distill Qwen 1.5B	13.20%	Not specified
331	Gemma 3 270M	12.11%	Not specified

Source: artificialanalysis.ai/evaluations/ifbench

No published result tracked

36 of the 46 models on this site have no published number on this benchmark, so their cells in the home table stay blank rather than reading as zero.

Read this with

Nothing else on this site shares this benchmark's family or category, so these are other evaluations the same publisher reports.

As reported by each publisher

The same benchmark, measured by the labs themselves. Compare down a column, never across two publishers.

Reflection AI

Introducing Beam: model capability comparisons · 5 October 2026

Rank	Model	Score
4	Beam	79.7%
3	Inkling	79.8%
2	Nemotron 3 Ultra	81.7%
5	GLM 5.2	73.3%
1	Qwen3.8 Max	82.8%

Ranks compare reported scores in the same published row; ties share a rank.

Reflection AI’s four capability tables, read 6 October 2026. All 21 rows and eight model columns are transcribed; NR means not reported and remains blank. The source’s Qwen 3.8 Max column uses this catalogue’s Qwen3.8 Max spelling. Beam results are publisher reports; Reflection AI cites Artificial Analysis and Datacurve for comparison-model evaluations. The announcement does not specify a common harness, effort or trial count for every row. Ranks are derived within each published row from reported values only, with ties sharing a rank; they do not establish an independent or overall model ranking. AutomationBench uses the public split; AA-Omniscience is an index on the public split only, calculated by Reflection AI. BrowseComp and DeepSearchQA use context management. AA-LCR has no stated version. These settings remain separate from other source fields.

reflection.ai/blog/introducing-beam

Sources

Every page used for this benchmark, read on 17 October 2026.

Benchmark site or the page that publishes itartificialanalysis.ai/evaluations/ifbench
Independent resultsartificialanalysis.ai/evaluations/ifbench
Reflection AI tablereflection.ai/blog/introducing-beam