AIEvals

Benchmarks

SRE Bench

Whether an agent can work out what a real-world binary does without its source code: reverse engineering across network protocols, firmware, games, file-format recovery and malware.

SecurityVals AI16 models run by Vals AIUpdated 29 September 2026

How it is run

19 programs averaging over 16,915 lines of code were written from scratch as a clean-room benchmark — more than 5,000 expert hours — and paired with a 27K-line anti-analysis suite, giving 262 contamination-free instances and 1,572 deterministically verifiable tasks. Every agent gets the same professional toolkit inside its sandbox: Ghidra, radare2, GDB, angr, binutils and dynamic tracers, plus tshark and exploitation libraries.

What it cannot tell you

This beta has a small field and large differences between models. Every agent receives the same professional toolkit, so access to those tools alone does not explain the differences in successful analysis.

Leader
GPT-6 Astra56.87% · Vals AI's own list
Top-ten spread
52.67%Leader minus tenth place, or last place in smaller fields.
Median
8.39%of the 16 rows published
No published result
31of 46 models on this site

Independent leaderboard · Vals AI

Top 16 of 16 runs by Vals AI, one harness.

#ModelScore
1GPT-6 Astra56.87%
2GPT-6.1 Sol50.76%
3Gemini 4 Argon44.27%
4Claude Opus 5.533.59%
5GPT-5.6 Sol30.53%
6Claude Sonnet 5.530.15%
7Claude Fable 5.122.90%
8Claude Opus 512.21%
9Gemini 3.7 Flash4.58%
10MiMo-V2.6-Flash4.20%
11GPT-5.53.82%
12MiMo V2.6 Pro3.05%
13Hunyuan Hy4 Preview2.29%
14DeepSeek V4.1 Flash0.76%
14Grok 4.50.76%
16Glm 5.20.00%
Source: www.vals.ai/benchmarks/srebench

No published result tracked

31 of the 46 models on this site have no published number on this benchmark, so their cells in the home table stay blank rather than reading as zero.

Step 5 PreviewGPT-6 LunaGPT-5.6 TerraGLM 5.3 FlashQwen3.8 2.4T A95BQwen3.8-Flash-NextQwen3.8 Max (0902)Claude Fable 5Grok 4.6Claude Opus 4.8Claude Opus 4.7Muse Spark 1.2GPT-5.4DeepSeek V4.1 Flash 1017GLM-LegalClaude Haiku 5.5BeamClaude Sonnet 5GPT-6 SolGPT-5.6 LunaGemini 3.8 FlashGemini 3.1 ProGrok 4.7DeepSeek V4 ProKimi K3Qwen3.8 MaxGLM 5.3Muse Spark 1.3 MaxMiniMax M3InklingMistral Medium 3.5

Read this with

Benchmarks in the same category, which usually disagree with this one on purpose.

  • CWE-bench v1Same category. Remediating software security vulnerabilities, measured as the share of tasks successfully completed.
  • CyberBench v1.1Same category. Whether autonomous agents can find and trigger real vulnerabilities in open-source projects by submitting a proof-of-concept input, and produce a source patch that fixes the bug without breaking behaviour.
  • CyberGymSame category. Whether an agent can analyse real vulnerabilities in large, long-lived codebases — the work of a security researcher rather than a quiz about security.
  • SEC-bench ProSame category. Whether an agent can find security bugs in critical software — V8, Firefox and the Linux kernel — under a fixed time budget.
  • ExploitGymSame category. Whether a model can turn a known vulnerability into a working exploit, reported as a Pass@1 rate alongside the other cyber-security evaluations.
  • ExploitBenchSame category. How far up the exploitation ladder an agent climbs on a production JavaScript engine: reaching vulnerable code, triggering the bug, building an exploit primitive, and finally achieving arbitrary code execution, with sixteen capabilities measured per run.

Sources

Every page used for this benchmark, read on 17 October 2026.