AIEvals

Benchmarks

ExploitGym

Whether a model can turn a known vulnerability into a working exploit, reported as a Pass@1 rate alongside the other cyber-security evaluations.

SecurityWang et al. (2026)Vendor-reported onlyReported by Z.ai, DeepSeek

How it is run

DeepSeek lists it as one of three cyber-security benchmarks in its V4.1 evaluation and reports each model's Pass@1 under its own harness. No task counts or public leaderboard are published with those figures.

What it cannot tell you

No standalone page for this benchmark could be verified when this page was written, so the reference here is the report that publishes the numbers. That makes it the weakest kind of evidence on this site: one publisher, no reproduction, and nothing to check the harness against.

Why there is no independent table

No independent suite in this dataset runs ExploitGym. The numbers below are the ones model publishers report about their own models, which is a different kind of evidence: each publisher chooses its own harness and effort level, so a row is readable on its own and not against the other rows.

Read this with

Benchmarks in the same category, which usually disagree with this one on purpose.

  • CWE-bench v1Same category. Remediating software security vulnerabilities, measured as the share of tasks successfully completed.
  • CyberBench v1.1Same category. Whether autonomous agents can find and trigger real vulnerabilities in open-source projects by submitting a proof-of-concept input, and produce a source patch that fixes the bug without breaking behaviour.
  • SRE BenchSame category. Whether an agent can work out what a real-world binary does without its source code: reverse engineering across network protocols, firmware, games, file-format recovery and malware.
  • CyberGymSame category. Whether an agent can analyse real vulnerabilities in large, long-lived codebases — the work of a security researcher rather than a quiz about security.
  • SEC-bench ProSame category. Whether an agent can find security bugs in critical software — V8, Firefox and the Linux kernel — under a fixed time budget.
  • ExploitBenchSame category. How far up the exploitation ladder an agent climbs on a production JavaScript engine: reaching vulnerable code, triggering the bug, building an exploit primitive, and finally achieving arbitrary code execution, with sixteen capabilities measured per run.

As reported by each publisher

Kept separate per publisher, because the settings are not shared.

Z.ai

GLM-5.3: Frontier Coding with Emergent Cyber Capabilities — evaluation table · 14 August 2026

RankRowModelScore (tasks)
3ExploitGym (2h)GLM 5.3105
6ExploitGym (2h)GLM 5.229
5ExploitGym (2h)Kimi K336
7ExploitGym (2h)Qwen3.8 Max14
4ExploitGym (2h)Claude Opus 4.880
2ExploitGym (2h)Claude Fable 5181
1ExploitGym (2h)GPT-5.6 Sol216
3ExploitGym (6h)GLM 5.3130
6ExploitGym (6h)GLM 5.239
5ExploitGym (6h)Kimi K370
7ExploitGym (6h)Qwen3.8 Max26
4ExploitGym (6h)Claude Opus 4.8120
2ExploitGym (6h)Claude Fable 5247
1ExploitGym (6h)GPT-5.6 Sol293

Ranks compare reported scores in the same published row; ties share a rank.

ExploitGym (2h). Tasks completed inside a 2-hour budget, on a different scale from the percentage rows: Z.ai prints 105 / 130.

ExploitGym (6h). The same runs given a 6-hour budget; the increase over the 2-hour row is what more time buys.

The comparison table Z.ai publishes with GLM-5.3, read on the model card itself. Z.ai states that GLM-5.3 uses the same base model as GLM-5.2 and that every gain comes from post-training, so the GLM-5.2 column is the same architecture one training round earlier. Column names were restated in the form this site uses, and Z.ai labels its own Fable column "Fable 5 (w/ fallback)", which is recorded here as Claude Fable 5 and is the only column whose runs used a fallback model. Rows whose scale is not a percentage keep their own unit: ExploitGym is reported as tasks completed with a 2-hour and a 6-hour budget and is split into two rows so both numbers stay readable, and GDPval-AA v2 is a score rather than a percentage. The table is the only source for several benchmarks on this site, and nothing in it is an independent run.

z.ai/blog/glm-5.3

DeepSeek

DeepSeek-V4.1-Flash: comparison with closed and open-source models · 10 September 2026

RankModelScore
2Claude Opus 522.1%
1GPT-5.6 Sol33.7%
4GLM 5.315.0%
5DeepSeek V4 Pro5.4%
6DeepSeek V4 Flash1.8%
3DeepSeek V4.1 Flash15.3%

Ranks compare reported scores in the same published row; ties share a rank.

Table 3 of the V4.1 technical report, published with the model release. Every column is the vendor's own run at its maximum reasoning effort, each on that vendor's own harness, so this compares publication claims rather than one shared run. DeepSeek renamed nothing and disclosed no fallback rates; the report instead documents the opposite problem, agents searching for ways to game the harness, and says its visual-agent rows were run with a 512k-token context. The benchmark names are reproduced as DeepSeek wrote them — including rows this site keeps apart, such as HLE with and without tools — and the model columns are renamed to the names used on this site. DeepSeek's table is also republished, unchanged, on the model page at ollama.com; that copy prints 64.0 for the V4.1 row of NL2Repo-Bench where the report prints 65.4, and this site follows the report.

huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/resolve/main/DeepSeek_V41_Tech_Report.pdf

Sources

Every page used for this benchmark, read on 17 October 2026.