AIEvals

Benchmarks

τ³-Banking

Banking support with policy retrieval and multi-step tool calls.

Tool useSierra Research154 models run by Artificial AnalysisUpdated 30 September 2026Reported by Reflection AI

How it is run

AA runs 97 tasks five times and scores pass@1 from the final backend state.

What it cannot tell you

AA labels this a legacy evaluation; it was removed from the current Intelligence Index. User simulation and tool policies affect results.

Leader
Qwen3.8 Max51.34% · Artificial Analysis results, grouped here by release
Top-ten spread
5.98%Leader minus tenth place, or last place in smaller fields.
Median
14.33%of the 154 rows published
No published result
16of 46 models on this site

Artificial Analysis results by release

Highest published effort per model release, grouped here from AA’s public results. Grouping ranks are derived here; source settings and model-publisher reports stay separate.

#ModelScoreEffort
1Qwen3.8 Max51.34%Not specified
2Muse Spark 1.350.52%max
3GLM-5.350.31%max
4Qwen3.8 2.4T A95B49.07%Not specified
5Qwen3.8 27B48.04%xhigh
6Qwen3.8 Max (0902)47.84%Not specified
7Claude Fable 5.147.22%max
7GLM 5.3 Flash47.22%max
9Kimi K345.98%max
10Qwen3.8-Flash-Next45.36%Not specified
11Gemini 3.8 Flash44.95%high
12GPT-5.6 Sol44.33%max
13Grok 4.643.30%xhigh
14Claude Opus 542.06%max
14Grok 4.542.06%high
16GPT-6 Astra41.44%max
17DeepSeek V4 Flash Vision41.03%max
18GPT-5.6 Terra40.21%max
19DeepSeek V4 Pro 081339.59%max
19GPT-5.439.59%xhigh
21DeepSeek V4 Flash 073139.38%max
22GPT-5.538.97%xhigh
23Ling-3.0-flash-Fin38.56%Not specified
24Claude Fable 538.14%max
25Claude Sonnet 537.32%max
26Motif 335.26%Not specified
27Muse Spark 1.234.85%xhigh
28GLM-5.234.64%max
28Claude Opus 4.734.64%max
30Ling-3.0-flash-VL34.43%Not specified
30Claude Sonnet 4.634.43%max
32Claude Opus 4.834.23%max
32K2 Horizon 375B A23B34.23%Not specified
34Gemini 3.7 Flash32.78%high
35Gemini 3.5 Flash32.16%high
36K2 Horizon MoVA 36B A4B31.96%Not specified
37Muse Spark 1.131.75%xhigh
38GPT-5.6 Luna31.13%max
39DeepSeek V4 Flash 042030.93%max
40DeepSeek V4 Pro 042430.10%max
41Gemini 3.6 Flash29.90%high
42Inkling29.07%xhigh
43Quasar 438B28.66%max
44K2 Horizon 7B27.63%Not specified
45GPT-5.4 nano27.42%xhigh
46Ling 3.0 Flash27.22%Not specified
47GPT-5.4 mini25.57%xhigh
48Apodex 1.125.15%Not specified
49Claude 4.5 Sonnet24.54%Not specified
50Muse Glimmer23.51%high
51Kimi K2.623.30%Not specified
51Solar Pro 423.30%Not specified
53Hy322.89%Not specified
54GPT-522.06%high
54G9v3-39A5B22.06%Not specified
56Solar Open2 250B21.65%Not specified
57Gemini 3.1 Pro Preview21.44%Not specified
58DeepSeek V3.1 Terminus21.03%Not specified
59MiniCPM5-2B20.82%Not specified
59Gemini 3 Flash Preview20.82%Not specified
59Ling 3.0 Tiny20.82%Not specified
59Qwen3.6 Plus20.82%Not specified
63Kimi K2.7 Code20.21%Not specified
64K2 Horizon 3.7B19.59%Not specified
65Inkling Small18.76%Not specified
66Ring-2.6-1T17.94%xhigh
67Gemini 3.5 Flash-Lite17.53%high
67Nex-N2-Pro17.53%Not specified
67Qwen3.7 Plus17.53%Not specified
70Claude 4 Sonnet16.70%Not specified
71A.X-K216.08%Not specified
72GPT-5.115.88%high
73GPT-5 mini15.46%high
74MiniMax-M315.26%Not specified
75Mistral Medium 3.515.05%high
76Gemma 4 31B14.85%Not specified
77Granite 4.2 30B14.43%Not specified
78Nemotron 3 Ultra 550B A55B14.23%Not specified
78Kimi K2.514.23%Not specified
80GLM-5.113.61%Not specified
81Grok Build 0.1 061613.40%Not specified
81Qwen3.5 397B A17B13.40%Not specified
81GLM-4.613.40%Not specified
84LongCat 2.013.20%Not specified
85gpt-oss-120b12.78%high
86Grok 4.312.37%high
87GLM-4.712.16%Not specified
88Step 3.7 Flash11.96%high
88Gemma 4 26B A4B11.96%Not specified
90Qwen3.7 Max11.75%Not specified
90GPT-5.5 Instant (June 2026)11.75%Not specified
92K-EXAONE 2.0 080311.55%Not specified
93GPT-5.211.13%Not specified
94Devstral Small 210.72%Not specified
95Devstral 210.52%Not specified
96Qwen3.5 122B A10B10.31%Not specified
96Nemotron 3 Super 120B A12B10.31%Not specified
98MiMo-V2.5-Pro9.90%Not specified
98MiniMax-M2.79.90%Not specified
100Gemini 2.5 Pro9.69%Not specified
100Gemini 3.1 Flash-Lite9.69%high
102Mercury 29.48%high
103Qwen3.6 27B9.28%Not specified
103Claude 4.5 Haiku9.28%Not specified
105Nemotron 3.5 Lightning8.87%Not specified
106MiMo-V2.58.66%Not specified
106Solar Pro 38.66%high
108Mistral Medium 3.18.25%Not specified
109Qwen3 235B A22B 2507 (Reasoning)7.84%Not specified
110Granite 4.2 8B7.63%Not specified
111Mistral Small 3.17.42%Not specified
112LFM2.5-2.6B7.22%Not specified
113Qwen3.5 9B7.01%Not specified
113gpt-oss-20b7.01%high
115Ministral 3 14B6.60%Not specified
116DeepSeek R1 (Jan '25)6.39%Not specified
116North Mini Code6.39%Not specified
118Mistral Small 3.26.19%Not specified
118Qwen3 Next 80B A3B (Reasoning)6.19%Not specified
120Command A+5.98%Not specified
120NVIDIA Nemotron 3 Nano 30B A3B5.98%Not specified
122Mistral Large 35.77%Not specified
122Trinity Large Thinking5.77%Not specified
124Granite 4.2 3B5.57%Not specified
124HyperNova 60B 26055.57%high
124Qwen3 14B5.57%Not specified
124Magistral Medium 1.25.57%Not specified
128Qwen3 30B A3B 2507 (Reasoning)5.36%Not specified
128Qwen3.6 35B A3B5.36%Not specified
128GPT-4.1 mini5.36%Not specified
128Qwen3 32B5.36%Not specified
128Qwen3 Coder Next5.36%Not specified
133o3-mini5.15%high
134Mistral Small 44.95%Not specified
134K2 Horizon 0.9B4.95%Not specified
134Qwen3.5 35B A3B4.95%Not specified
137DeepSeek V3 (Dec '24)4.74%Not specified
137Ministral 3 3B4.74%Not specified
137DeepSeek V3 03244.74%Not specified
137Qwen3 8B4.74%Not specified
141Magistral Small 1.24.54%Not specified
141KAT Coder Pro V24.54%Not specified
143Qwen3.5 4B4.33%Not specified
144Celeris-13.92%Not specified
145Llama 4 Maverick3.71%Not specified
145Ministral 3 8B3.71%Not specified
147GPT-4.1 nano3.51%Not specified
148Llama 4 Scout3.30%Not specified
149Ling 2.6 Flash2.89%Not specified
149GPT-4o mini2.89%Not specified
151Gemma 3 27B Instruct0.82%Not specified
151Gemma 3 12B Instruct0.82%Not specified
153Gemma 3 4B Instruct0.41%Not specified
154Gemma 3n E4B Instruct0.21%Not specified

No published result tracked

16 of the 46 models on this site have no published number on this benchmark, so their cells in the home table stay blank rather than reading as zero.

Read this with

Benchmarks in the same category, which usually disagree with this one on purpose.

As reported by each publisher

The same benchmark, measured by the labs themselves. Compare down a column, never across two publishers.

Reflection AI

Introducing Beam: model capability comparisons · 5 October 2026

RankModelScore
2Beam38.0%
5Inkling25.0%
6Nemotron 3 Ultra22.6%
3GLM 5.237.1%
3Kimi K337.1%
1Qwen3.8 Max55.2%

Ranks compare reported scores in the same published row; ties share a rank.

Reflection AI’s four capability tables, read 6 October 2026. All 21 rows and eight model columns are transcribed; NR means not reported and remains blank. The source’s Qwen 3.8 Max column uses this catalogue’s Qwen3.8 Max spelling. Beam results are publisher reports; Reflection AI cites Artificial Analysis and Datacurve for comparison-model evaluations. The announcement does not specify a common harness, effort or trial count for every row. Ranks are derived within each published row from reported values only, with ties sharing a rank; they do not establish an independent or overall model ranking. AutomationBench uses the public split; AA-Omniscience is an index on the public split only, calculated by Reflection AI. BrowseComp and DeepSearchQA use context management. AA-LCR has no stated version. These settings remain separate from other source fields.

reflection.ai/blog/introducing-beam

Sources

Every page used for this benchmark, read on 17 October 2026.