AIEvals

Benchmarks

CyberBench v1.1

Whether autonomous agents can find and trigger real vulnerabilities in open-source projects by submitting a proof-of-concept input, and produce a source patch that fixes the bug without breaking behaviour.

SecurityVals AI, building on CyberGym and ARVO43 models run by Vals AIUpdated 29 September 2026

How it is run

Two tracks — 60 PoC tasks and 56 patch tasks — built on OSS-Fuzz crash regressions reproduced from ARVO images, mostly memory-safety, sanitizer and undefined-behaviour failures in native code. Provider refusals count as failures, and fallback routing is identified so the published score can be compared with fallbacks counted as failures.

What it cannot tell you

The scope is memory-safety bugs in native code, not web, identity or network security. Fallbacks move this board more than any other on the site: Sonnet 5.5 scores 59.58% as published and 41.97% with fallback-assisted tasks counted as failures.

Leader
GPT-6 Sol77.98% · Vals AI's own list
Top-ten spread
5.00%Leader minus tenth place, or last place in smaller fields.
Median
69.41%of the 43 rows published
No published result
15of 46 models on this site

Independent leaderboard · Vals AI

Top 43 of 43 runs by Vals AI, one harness.

#ModelScore
1GPT-6 Sol77.98%
2Gemini 4 Argon77.86%
3GPT-5.6 Sol76.31%
4GPT-6 Luna76.25%
5MiMo-V2.6-Flash75.36%
6Kimi K375.24%
7Claude Opus 5 5 Range74.58%
8DeepSeek V4.1 Flash73.69%
9GPT-5.6 Luna73.63%
10Gpt Daybreak Blue72.98%
11DeepSeek V4 Pro72.86%
11MiMo V2.6 Pro72.86%
13Muse Spark 1.3 Max72.74%
14Deepseek V4 Flash 073172.08%
14GPT-5.6 Terra72.08%
14GLM 5.372.08%
17Grok 4.571.01%
18Claude Fable 5.170.42%
19Claude Sonnet 5 Range70.30%
20Muse Spark 1.269.52%
21Grok 4.769.46%
22Muse Spark 1 369.41%
23GLM 5.3 Flash67.86%
24Qwen3.8 27b67.80%
25Grok 4.667.68%
26Inkling66.79%
27Claude Opus 565.36%
27Hunyuan Hy4 Preview65.36%
29MiniMax M365.00%
30Inkling Small64.11%
31Claude Sonnet 561.91%
32Claude Sonnet 5.559.58%
33Gemini 3.5 Flash Lite59.11%
34Ling 3.0 Flash Af Rc358.33%
35Claude Opus 5.555.36%
36Mercury 2.545.48%
37Gemini 3.6 Flash45.36%
38Gemini 3.7 Flash43.75%
38Gemini 3.8 Flash43.75%
40GPT-6 Astra41.07%
41GPT-6.1 Sol39.29%
42Nemotron Lightning 3p5 30b A3b29.46%
43Qwen3.8 Max28.57%
Source: www.vals.ai/benchmarks/cyber

No published result tracked

15 of the 46 models on this site have no published number on this benchmark, so their cells in the home table stay blank rather than reading as zero.

Step 5 PreviewQwen3.8 2.4T A95BQwen3.8-Flash-NextQwen3.8 Max (0902)Claude Fable 5Claude Opus 4.8Claude Opus 4.7GPT-5.4GPT-5.5DeepSeek V4.1 Flash 1017GLM-LegalClaude Haiku 5.5BeamGemini 3.1 ProMistral Medium 3.5

Read this with

Benchmarks in the same category, which usually disagree with this one on purpose.

  • CWE-bench v1Same category. Remediating software security vulnerabilities, measured as the share of tasks successfully completed.
  • SRE BenchSame category. Whether an agent can work out what a real-world binary does without its source code: reverse engineering across network protocols, firmware, games, file-format recovery and malware.
  • CyberGymSame category. Whether an agent can analyse real vulnerabilities in large, long-lived codebases — the work of a security researcher rather than a quiz about security.
  • SEC-bench ProSame category. Whether an agent can find security bugs in critical software — V8, Firefox and the Linux kernel — under a fixed time budget.
  • ExploitGymSame category. Whether a model can turn a known vulnerability into a working exploit, reported as a Pass@1 rate alongside the other cyber-security evaluations.
  • ExploitBenchSame category. How far up the exploitation ladder an agent climbs on a production JavaScript engine: reaching vulnerable code, triggering the bug, building an exploit primitive, and finally achieving arbitrary code execution, with sixteen capabilities measured per run.

Sources

Every page used for this benchmark, read on 17 October 2026.