AIEvals

Benchmarks

ZeroBench (main set)

An "impossible" visual benchmark: questions written so that no current multimodal model is expected to answer them from the image alone.

MultimodalRoberts et al.Vendor-reported onlyReported by DeepSeek, DeepSeek

How it is run

DeepSeek reports Pass@5 with tools on the main set, which is an agent loop over five attempts rather than the single-shot setting the benchmark was designed around.

What it cannot tell you

Five attempts with image tools is a materially easier measurement than the benchmark's own leaderboard, and DeepSeek reports 49.0 for V4.1 Flash against 53.0 for GPT-5.6 Sol, so this is a comparison inside one publisher's setting only.

Why there is no independent table

No independent suite in this dataset runs ZeroBench (main set). The numbers below are the ones model publishers report about their own models, which is a different kind of evidence: each publisher chooses its own harness and effort level, so a row is readable on its own and not against the other rows.

Read this with

Benchmarks in the same category, which usually disagree with this one on purpose.

  • BabyVisionSame category. Core visual ability rather than knowledge: 388 items in 22 subclasses that a person solves by looking, without needing language to reason about the answer.

As reported by each publisher

Kept separate per publisher, because the settings are not shared.

DeepSeek

DeepSeek-V4.1-Flash-1017: complete release comparison · 17 October 2026

RankModelScore
3DeepSeek V4.1 Flash 101751.0%
4DeepSeek V4.1 Flash49.0%
5Kimi K341.0%
1GPT-5.6 Sol53.0%
2Claude Opus 552.0%

Ranks compare reported scores in the same published row; ties share a rank.

ZeroBench-main (w/tools). Main set, with tools, Pass@5 as stated in DeepSeek’s 17 October release notes. GLM-5.3 is not reported.

The complete 11-row, six-model comparison from DeepSeek’s dated MIT model card, read 17 October 2026 at revision 1bae0e1088993b9b82de08f59de5be5a34e42170. All 63 published comparison cells and both additional parenthesized HLE scores are retained as 65 figures across 12 rows. The HLE text-only scores are split into their own row and benchmark page; GLM-5.3’s dagger-marked 42.0 is text-only and has no full-set score. Unreported values stay null and do not enter ranks. K3 and Opus 5 are displayed as Kimi K3 and Claude Opus 5. Terminal-Bench 4.0 uses Claude Code, DeepSWE v1.1 uses Mini-SWE-V2, and the release notes identify ZeroBench-main as Pass@5. Other effort levels, trial counts and judges are not fully specified. These are DeepSeek publication claims, separate from independent evaluations and earlier publisher snapshots.

huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017

DeepSeek

DeepSeek-V4.1-Flash: comparison with closed and open-source models · 10 September 2026

RankModelScore
2Claude Opus 552.0%
1GPT-5.6 Sol53.0%
4Kimi K341.0%
3DeepSeek V4.1 Flash49.0%

Ranks compare reported scores in the same published row; ties share a rank.

Table 3 of the V4.1 technical report, published with the model release. Every column is the vendor's own run at its maximum reasoning effort, each on that vendor's own harness, so this compares publication claims rather than one shared run. DeepSeek renamed nothing and disclosed no fallback rates; the report instead documents the opposite problem, agents searching for ways to game the harness, and says its visual-agent rows were run with a 512k-token context. The benchmark names are reproduced as DeepSeek wrote them — including rows this site keeps apart, such as HLE with and without tools — and the model columns are renamed to the names used on this site. DeepSeek's table is also republished, unchanged, on the model page at ollama.com; that copy prints 64.0 for the V4.1 row of NL2Repo-Bench where the report prints 65.4, and this site follows the report.

huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/resolve/main/DeepSeek_V41_Tech_Report.pdf

Sources

Every page used for this benchmark, read on 17 October 2026.