AIEvals

Benchmarks

SEC-bench Pro

Whether an agent can find security bugs in critical software — V8, Firefox and the Linux kernel — under a fixed time budget.

SecuritySEC-bench team (open source)Vendor-reported onlyReported by DeepSeek

How it is run

A self-evolving, versioned benchmark: the snapshot used here is version 260505, with a 344-instance leaderboard across the three projects and a separate count of tasks completed within the timeout.

What it cannot tell you

Version-scoped, so a number belongs to one snapshot of a benchmark whose tasks keep changing; DeepSeek ran it with the Claude Code harness specifically for its session-compaction behaviour, which is part of the result rather than a neutral choice.

Why there is no independent table

No independent suite in this dataset runs SEC-bench Pro. The numbers below are the ones model publishers report about their own models, which is a different kind of evidence: each publisher chooses its own harness and effort level, so a row is readable on its own and not against the other rows.

Read this with

Benchmarks in the same category, which usually disagree with this one on purpose.

  • CWE-bench v1Same category. Remediating software security vulnerabilities, measured as the share of tasks successfully completed.
  • CyberBench v1.1Same category. Whether autonomous agents can find and trigger real vulnerabilities in open-source projects by submitting a proof-of-concept input, and produce a source patch that fixes the bug without breaking behaviour.
  • SRE BenchSame category. Whether an agent can work out what a real-world binary does without its source code: reverse engineering across network protocols, firmware, games, file-format recovery and malware.
  • CyberGymSame category. Whether an agent can analyse real vulnerabilities in large, long-lived codebases — the work of a security researcher rather than a quiz about security.
  • ExploitGymSame category. Whether a model can turn a known vulnerability into a working exploit, reported as a Pass@1 rate alongside the other cyber-security evaluations.
  • ExploitBenchSame category. How far up the exploitation ladder an agent climbs on a production JavaScript engine: reaching vulnerable code, triggering the bug, building an exploit primitive, and finally achieving arbitrary code execution, with sixteen capabilities measured per run.

As reported by each publisher

Kept separate per publisher, because the settings are not shared.

DeepSeek

DeepSeek-V4.1-Flash: comparison with closed and open-source models · 10 September 2026

RankModelScore
1GPT-5.6 Sol74.3%
3DeepSeek V4 Pro56.4%
4DeepSeek V4 Flash30.9%
2DeepSeek V4.1 Flash62.8%

Ranks compare reported scores in the same published row; ties share a rank.

Table 3 of the V4.1 technical report, published with the model release. Every column is the vendor's own run at its maximum reasoning effort, each on that vendor's own harness, so this compares publication claims rather than one shared run. DeepSeek renamed nothing and disclosed no fallback rates; the report instead documents the opposite problem, agents searching for ways to game the harness, and says its visual-agent rows were run with a 512k-token context. The benchmark names are reproduced as DeepSeek wrote them — including rows this site keeps apart, such as HLE with and without tools — and the model columns are renamed to the names used on this site. DeepSeek's table is also republished, unchanged, on the model page at ollama.com; that copy prints 64.0 for the V4.1 row of NL2Repo-Bench where the report prints 65.4, and this site follows the report.

huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/resolve/main/DeepSeek_V41_Tech_Report.pdf

Sources

Every page used for this benchmark, read on 17 October 2026.