AIEvals

Benchmarks

CWE-bench v1

Remediating software security vulnerabilities, measured as the share of tasks successfully completed.

SecurityCWE-bench authors, quoted by Google DeepMindVendor-reported onlyReported by Google DeepMind

How it is run

Google quotes the official CWE-bench v1 public leaderboard. Scores rank by pass@1, with pass@4 used to break ties.

What it cannot tell you

The table reports the pass@1 figure. Equal percentages alone do not establish an equal published rank because pass@4 can break ties; v0 is a separate version.

Why there is no independent table

No independent suite in this dataset runs CWE-bench v1. The numbers below are the ones model publishers report about their own models, which is a different kind of evidence: each publisher chooses its own harness and effort level, so a row is readable on its own and not against the other rows.

Read this with

Benchmarks in the same category, which usually disagree with this one on purpose.

  • CyberBench v1.1Same category. Whether autonomous agents can find and trigger real vulnerabilities in open-source projects by submitting a proof-of-concept input, and produce a source patch that fixes the bug without breaking behaviour.
  • SRE BenchSame category. Whether an agent can work out what a real-world binary does without its source code: reverse engineering across network protocols, firmware, games, file-format recovery and malware.
  • CyberGymSame category. Whether an agent can analyse real vulnerabilities in large, long-lived codebases — the work of a security researcher rather than a quiz about security.
  • SEC-bench ProSame category. Whether an agent can find security bugs in critical software — V8, Firefox and the Linux kernel — under a fixed time budget.
  • ExploitGymSame category. Whether a model can turn a known vulnerability into a working exploit, reported as a Pass@1 rate alongside the other cyber-security evaluations.
  • ExploitBenchSame category. How far up the exploitation ladder an agent climbs on a production JavaScript engine: reaching vulnerable code, triggering the bug, building an exploit primitive, and finally achieving arbitrary code execution, with sixteen capabilities measured per run.

As reported by each publisher

Kept separate per publisher, because the settings are not shared.

Google DeepMind

Gemini 4 Argon: model evaluation · October 2026

RankModelScore
1Gemini 4 Argon68.0%
1GPT-6 Astra68.0%
4Claude Fable 5.158.0%
3Claude Opus 5.567.0%

Ranks compare reported scores in the same published row; ties share a rank.

CWE-bench v1. Quoted from the official leaderboard; ranking uses pass@1 with pass@4 as the tie-breaker.

Page 5 of Google DeepMind’s model evaluation, read 1 October 2026. Gemini runs use the API at the highest thinking setting, generally pass@1 without majority voting or parallel test-time compute; smaller benchmarks average multiple trials. Other models use their highest available published reasoning results. Google quotes Vals AI for knowledge-work and Vibe Code results, Zapier’s private-set AutomationBench leaderboard, Proximal for FrontierSWE v2, Surge AI for RiemannBench and no-tools Chartography, and the CWE-bench v1 leaderboard. Argon’s DeepSWE uses mini-swe; Terminal-Bench 4.0 is self-computed for Argon and quoted from the author board for other models. PostTrainBench v1.1 uses OpenCode with one H100 and a 10-hour budget for all models. Terminal-Bench Science uses a 6x verifier timeout for Argon. LABBench 2 uses terminal, bioinformatics tools, Python, R and internet access for all models. GraphWalks uses BFS F1 on two separate subsets: 650 items up to 128k tokens and 200 items from 256k to 1M. Agent’s Last Exam uses ALE-Claw, a 5-hour window and binary pass rate, with safety filters enabled for Argon. OSWorld 2.0 uses offline partial reward, the best of three single-attempt runs for Argon, safety filters, 1080p, 500 steps, Gemini CUA with parallel batch tools and compaction, screenshot-only observation and the 08.08 patch; Claude’s combined online/offline results are omitted. LVBench runs without tools, using 1 FPS for Gemini, 800 frames for Astra, 300 for Fable and 600 for Opus. Missing cells remain blank. These are the report’s published figures, preserved separately from this site’s independent source snapshots.

storage.googleapis.com/deepmind-media/gemini/gemini_4_argon_model_evaluation.pdf

Sources

Every page used for this benchmark, read on 17 October 2026.