AIEvals

Benchmarks

AA-Briefcase v1.1

Long-horizon knowledge work of the kind that fills a professional briefcase, scored on an Elo-style scale rather than as a share of tasks passed.

Knowledge workArtificial Analysis142 models run by Artificial AnalysisUpdated 8 October 2026Reported by Anthropic, Anthropic, Anthropic

How it is run

Artificial Analysis independently evaluates professional work and reports Elo ratings. The independent table uses the highest published effort per model release, with its snapshot date shown in the table. Original vendor comparisons remain separate below.

What it cannot tell you

A score, not a percentage, so it can never be averaged into a percentage column, and the whole table comes from one publisher quoting another. Artificial Analysis ran the pre-release Sonnet 5.5 deployment that Anthropic later found had a structured-output bug.

Leader
Claude Opus 5.51,821.85 · Artificial Analysis results, grouped here by release
Top-ten spread
252.99Leader minus tenth place, or last place in smaller fields.
Median
815.43of the 142 rows published
No published result
5of 46 models on this site

Artificial Analysis results by release

Highest published effort per model release, grouped here from AA’s public results. Grouping ranks are derived here; source settings and model-publisher reports stay separate.

#ModelScoreEffort
1Claude Opus 5.51,821.85max
2Claude Sonnet 5.51,810.9max
3Claude Fable 5.11,678.37max
4Claude Opus 51,673.33max
5Grok 4.71,657.2xhigh
6Qwen3.8 Max (0902)1,624.46Not specified
7Qwen3.8-Flash-Next1,587.57Not specified
8Muse Spark 1.31,586.11max
9Claude Haiku 5.51,577.72max
10GPT-6 Astra1,568.86max
11GPT-6.1 Sol1,564.21max
12Grok 4.61,549.27xhigh
13Claude Fable 51,540.79max
14MiMo-V2.6-Pro1,519.54Not specified
15GLM-5.31,511.41max
16Kimi K31,500.5max
17MiMo-V2.6-Flash1,498.59Not specified
18Gemini 4 Argon1,493.84high
19GPT-5.6 Sol1,487.42max
20GPT-6 Sol1,478.97max
21GLM 5.3 Flash1,452.46max
22Qwen3.8 2.4T A95B1,439.67Not specified
23Step 5 Preview1,432.02Not specified
24DeepSeek V4 Flash Vision1,427.66max
25DeepSeek V4.1 Flash1,420.56max
26Qwen3.8 27B1,401.01xhigh
27Qwen3.8 Max1,388.81Not specified
28Claude Sonnet 51,359.19max
29GPT-5.6 Luna1,344.49max
30GPT-6 Luna1,335.51max
31GPT-5.6 Terra1,333.95max
32Muse Spark 1.21,333.32xhigh
33Claude Opus 4.81,320.8max
34K2 Horizon 375B A23B1,295.28Not specified
35Grok 4.51,282.9high
36DeepSeek V4 Pro 08131,257.69max
37DeepSeek V4 Flash 07311,256.59max
38Claude Opus 4.71,253max
39GLM-5.21,230.23max
40Gemini 3.8 Flash1,201.59high
41K2 Horizon MoVA 36B A4B1,167.7Not specified
42GPT-5.51,137.18xhigh
43Quasar 438B1,122.29max
44Gemini 3.7 Flash1,106.74high
45MiniMax-M31,091.65Not specified
46JT-4.1 Flash 236B A21B1,088.46Not specified
47GPT-5.5 Instant (June 2026)1,085.14Not specified
48Claude Sonnet 4.61,062.87max
49K2 Horizon 7B1,018.31Not specified
50Apodex 1.11,008.47Not specified
51Motif 3991.76Not specified
52Ling-3.0-flash-VL984.39Not specified
53GLM-5.1962.91Not specified
54Ling-3.0-flash-Fin962.55Not specified
55Gemini 3.6 Flash950.46high
56Hy3943.85Not specified
57Nex-N2-Pro935.32Not specified
58DeepSeek V4 Pro 0424922.51max
59Qwen3.7 Max914.55Not specified
60Qwen3.7 Plus911.81Not specified
61Inkling Small911.3Not specified
62MiMo-V2.5-Pro881.34Not specified
63Nemotron 3 Ultra 550B A55B876.34Not specified
64Gemini 3.5 Flash871.9high
65Solar Mini 4871.7Not specified
66GPT-5.3 Codex866.16xhigh
67Kimi K2.7 Code856.17Not specified
68Muse Spark 1.1847.79xhigh
69DeepSeek V4 Flash 0420835.91max
70Inkling833.68xhigh
71Kimi K2.6817.7Not specified
72Qwen3.6 27B813.17Not specified
73Ling 3.0 Flash797.05Not specified
74K2 Horizon 3.7B793.23Not specified
75Grok 4.3761.23high
76MiMo-V2.5752.34Not specified
77LongCat 2.0740.07Not specified
78GPT-5.4 mini717.56xhigh
79MiniMax-M2.7716.21Not specified
80Claude 4.5 Sonnet713.4Not specified
81Qwen3.6 35B A3B711.12Not specified
82GPT-5.4 nano671.15xhigh
83Ring-2.6-1T659.16xhigh
84Gemini 3.5 Flash-Lite648.48high
85Muse Spark643.67Not specified
86Claude 4.5 Haiku615.85Not specified
87KAT-Coder-Pro V1600.87Not specified
88Qwen3.5 397B A17B561.37Not specified
89Mistral Medium 3.1527.32Not specified
90Mistral Medium 3.5520.8high
91Nemotron 3.5 Lightning504.51Not specified
92Qwen3.5 122B A10B496Not specified
93Devstral Small 2492.54Not specified
94Devstral 2479.3Not specified
95Muse Glimmer473.58high
96Ling 3.0 Tiny472.5Not specified
97DeepSeek V3.1 Terminus456.68Not specified
98Gemini 3.1 Pro Preview453.35Not specified
99GPT-5 mini427.69high
100MiniCPM5-2B422.58Not specified
101Qwen3 Coder Next404.66Not specified
102Mercury 2.5376.85Not specified
103Gemma 4 31B364.61Not specified
104Command A+354.25Not specified
105Qwen3.5 9B335.84Not specified
106Mistral Small 3.1313.01Not specified
107Granite 4.2 8B307.89Not specified
108Gemini 2.5 Pro302.46Not specified
109Mercury 2264.38high
110Trinity Large Thinking254.79Not specified
111Mistral Small 4242.59Not specified
112Mistral Large 3228.83Not specified
113HyperNova 60B 2605227.55high
114North Mini Code211.08Not specified
115Gemini 3.1 Flash-Lite207.2high
116Qwen3 235B A22B 2507 (Reasoning)171.54Not specified
117Celeris-1152.57Not specified
118o3-mini142.63high
119Ministral 3 14B130.95Not specified
120DeepSeek R1 (Jan '25)129.85Not specified
121Solar Pro 3100.47high
122Granite 4.2 3B98.61Not specified
123NVIDIA Nemotron 3 Nano 30B A3B79.88Not specified
124Ministral 3 8B79.02Not specified
125DeepSeek V3 (Dec '24)78.29Not specified
126DeepSeek V3 032471.21Not specified
127K2 Think V28.42high
128Qwen3 30B A3B 2507 (Reasoning)7.85Not specified
129Gemma 3 27B Instruct0Not specified
129Gemma 3 12B Instruct0Not specified
129gpt-oss-20b0high
129Mistral Small 3.20Not specified
129Qwen3 14B0Not specified
129Nemotron 3 Super 120B A12B0Not specified
129Ministral 3 3B0Not specified
129Llama 4 Maverick0Not specified
129Qwen3 8B0Not specified
129K2 Horizon 0.9B0Not specified
129Llama 4 Scout0Not specified
129Qwen3 32B0Not specified
129Gemma 4 E4B0Not specified
129gpt-oss-120b0high

No published result tracked

5 of the 46 models on this site have no published number on this benchmark, so their cells in the home table stay blank rather than reading as zero.

Read this with

Benchmarks in the same category, which usually disagree with this one on purpose.

As reported by each publisher

The same benchmark, measured by the labs themselves. Compare down a column, never across two publishers.

Anthropic

Claude Haiku 5.5: launch evaluation table · 7 October 2026

RankModelScore (score)
2Claude Haiku 5.51,578
4Claude Haiku 4.5614
3GPT-6 Luna1,336
1Claude Sonnet 5.51,824

Ranks compare reported scores in the same published row; ties share a rank.

AA-Briefcase v1.1 (max). Elo from Artificial Analysis, quoted by Anthropic.

All eight results and four comparison columns from Anthropic’s announcement, read 8 October 2026. Haiku 5.5 uses max effort; Sonnet’s FrontierCode uses xhigh. HLE is split into no-tools and with-tools rows. OSWorld uses partial credit on the official 82-task offline subset, differing from earlier combined-subset reports. AA ran GDPval-AA and AA-Briefcase; these rounded quotes from Anthropic remain separate from the directly imported public AA measurements. GPT results retain the source and configuration described in the system card. Missing cells stay blank; the reported Haiku 4.5 Terminal-Bench score is zero. Each rank compares one published row only.

www.anthropic.com/claude-haiku-5-5

Anthropic

Claude Haiku 5.5: system card capability evaluations · 7 October 2026

RankRowModelScore (score)
2AA-Briefcase v1.1 (max)Claude Haiku 5.51,578
5AA-Briefcase v1.1 (max)Claude Haiku 4.5614
1AA-Briefcase v1.1 (max)Claude Sonnet 5.51,824
3AA-Briefcase v1.1 (max)GPT-6 Luna1,336
4AA-Briefcase v1.1 (max)Gemini 3.8 Flash1,203
1AA-Briefcase v1.1 (medium)Claude Haiku 5.51,372

Ranks compare reported scores in the same published row; ties share a rank.

AA-Briefcase v1.1 (max). §8.10.3: independent AA long-horizon knowledge-work evaluation quoted by Anthropic.

AA-Briefcase v1.1 (medium). §8.10.3: default medium effort, under one quarter of max’s output tokens. Comparison-model medium figures are not stated.

Section 8, pages 111–139 of the Haiku 5.5 system card, read 8 October 2026. Values come from exact prose, tables and labelled figures. Unless specified, Haiku uses adaptive thinking at max effort and five trials with evaluation-specific context up to 1M. Each row retains its subset, metric, effort and source notes; no score is assigned to an unspecified SWE-Bench Pro version. Terminal runs use safeguards without fallback and no internet egress. ProgramBench uses filtered tasks and hidden-test pass rate without a time limit. OSWorld partial and strict results use the official 82-task offline subset. CAD uses 0–1 voxel IoU. Healthcare raw and length-adjusted scores stay separate. Life-science results disable Haiku biology safeguards. Independently run figures quoted by Anthropic remain publisher reports here. Missing comparison values are not inferred from rounded charts.

www.anthropic.com/claude-haiku-5-5-system-card

Anthropic

Claude Sonnet 5.5: evaluation table · 28 September 2026

RankModelScore (score)
2Claude Sonnet 5.51,811
4Claude Sonnet 51,359
1Claude Opus 5.51,822
3GPT-6 Sol1,483

Ranks compare reported scores in the same published row; ties share a rank.

AA-Briefcase v1.1. An Elo-style score from Artificial Analysis's long-horizon knowledge-work benchmark, quoted here as Anthropic published it.

Anthropic publishes this table with Sonnet 5.5, the second model in its Claude 5.5 line, and states that Sonnet 5.5 at Max effort performs comparably to Opus 5.5 on several evaluations while Opus 5.5 remains stronger on open-ended work. The GPT-6 Sol column is quoted from OpenAI, not re-run, and Anthropic notes an OpenAI image-understanding bug that may not yet be reflected in third-party scores. Two rows come from other publishers: Artificial Analysis ran GDPval-AA and AA-Briefcase on a pre-release Sonnet 5.5 deployment that Anthropic later found had a structured-output bug, and Surge AI runs Chartography. Sonnet 5.5 launched with the cyber safeguards first built for Opus-class models, so some security work was completed by a fallback model.

www.anthropic.com/claude-sonnet-5-5

Sources

Every page used for this benchmark, read on 17 October 2026.