AIEvals

Benchmarks

FrontierCode v1.1

Whether a coding agent produces changes that are ready to merge into a real codebase, rather than a patch that merely passes a test.

Agentic codingReported by OpenAI and AnthropicVendor-reported onlyReported by Anthropic, Anthropic, OpenAI, Anthropic, Anthropic

How it is run

This page covers Main, the 100 hardest tasks. Extended contains all 150 tasks and has its own page. Anthropic’s Haiku 5.5 report quotes five Cognition runs per task: Haiku scores 46.4% at max and 45.8% at xhigh; Sonnet scores 46.2% at max and 52.1% at xhigh.

What it cannot tell you

Main and Extended are different task sets published side by side, so the higher Extended number is not a stronger result than a Main number from the same table. The effort setting moves this benchmark enough that a Max-effort figure can sit below the same model's Xhigh figure.

Why there is no independent table

No independent suite in this dataset runs FrontierCode v1.1. The numbers below are the ones model publishers report about their own models, which is a different kind of evidence: each publisher chooses its own harness and effort level, so a row is readable on its own and not against the other rows.

Read this with

Other versions of the same evaluation. Versions are not comparable: the task set changes between them.

As reported by each publisher

Kept separate per publisher, because the settings are not shared.

Anthropic

Claude Haiku 5.5: launch evaluation table · 7 October 2026

RankModelScore
2Claude Haiku 5.546.4%
3GPT-6 Luna42.4%
1Claude Sonnet 5.552.1%

Ranks compare reported scores in the same published row; ties share a rank.

FrontierCode v1.1 (Main). Haiku max; Sonnet xhigh, not its 46.2% max-effort run.

All eight results and four comparison columns from Anthropic’s announcement, read 8 October 2026. Haiku 5.5 uses max effort; Sonnet’s FrontierCode uses xhigh. HLE is split into no-tools and with-tools rows. OSWorld uses partial credit on the official 82-task offline subset, differing from earlier combined-subset reports. AA ran GDPval-AA and AA-Briefcase; these rounded quotes from Anthropic remain separate from the directly imported public AA measurements. GPT results retain the source and configuration described in the system card. Missing cells stay blank; the reported Haiku 4.5 Terminal-Bench score is zero. Each rank compares one published row only.

www.anthropic.com/claude-haiku-5-5

Anthropic

Claude Haiku 5.5: system card capability evaluations · 7 October 2026

RankRowModelScore
1FrontierCode v1.1 (Main, max)Claude Haiku 5.546.4%
2FrontierCode v1.1 (Main, max)Claude Sonnet 5.546.2%
3FrontierCode v1.1 (Main, max)GPT-6 Luna42.4%
2FrontierCode v1.1 (Main, xhigh)Claude Haiku 5.545.8%
1FrontierCode v1.1 (Main, xhigh)Claude Sonnet 5.552.1%

Ranks compare reported scores in the same published row; ties share a rank.

FrontierCode v1.1 (Main, max). §8.3: Cognition runs, mean of five attempts on the hardest 100 tasks. Claude Code / Codex.

FrontierCode v1.1 (Main, xhigh). Table 8.1.A: xhigh is separate from max; Sonnet scores higher at xhigh.

Section 8, pages 111–139 of the Haiku 5.5 system card, read 8 October 2026. Values come from exact prose, tables and labelled figures. Unless specified, Haiku uses adaptive thinking at max effort and five trials with evaluation-specific context up to 1M. Each row retains its subset, metric, effort and source notes; no score is assigned to an unspecified SWE-Bench Pro version. Terminal runs use safeguards without fallback and no internet egress. ProgramBench uses filtered tasks and hidden-test pass rate without a time limit. OSWorld partial and strict results use the official 82-task offline subset. CAD uses 0–1 voxel IoU. Healthcare raw and length-adjusted scores stay separate. Life-science results disable Haiku biology safeguards. Independently run figures quoted by Anthropic remain publisher reports here. Missing comparison values are not inferred from rounded charts.

www.anthropic.com/claude-haiku-5-5-system-card

OpenAI

GPT-6 Astra: evaluation table · 3 September 2026

Ranks compare reported scores in the same published row; ties share a rank.

OpenAI reports the maximum score at any effort level, in its own research environment or through its API. Claude and Gemini figures are taken by OpenAI from those vendors' published reports, so this table is a comparison between publication claims, not between runs.

openai.com/index/gpt-6-astra/

Anthropic

Claude Opus 5.5: evaluation table · 22 September 2026

RankModelScore
1Claude Opus 5.554.4%
3Claude Fable 5.150.3%
4Claude Opus 548.0%
2GPT-6 Astra53.3%
5GPT-5.6 Sol47.5%

Ranks compare reported scores in the same published row; ties share a rank.

Anthropic runs its own harness and states that margin size is a weakening guide to real differences at this level. Two rows rely on other publishers: AutomationBench figures come from Zapier, and the GPT figures are as reported by OpenAI. The Claude Opus 5.5 row was run with production safeguards enabled, so some security, biology and frontier-development tasks were completed by a fallback model.

www.anthropic.com/news/claude-opus-5-5

Anthropic

Claude Sonnet 5.5: evaluation table · 28 September 2026

RankModelScore
3Claude Sonnet 5.546.2%
4Claude Sonnet 542.4%
1Claude Opus 5.554.4%
2GPT-6 Sol49.3%

Ranks compare reported scores in the same published row; ties share a rank.

FrontierCode 1.1 (Main). Sonnet 5.5's 46.2% is its Max-effort run; the same model scores 52.1% at Xhigh, and Anthropic says its Max run leaned on a code-review skill whose extra edits the benchmark penalises as out of scope.

Anthropic publishes this table with Sonnet 5.5, the second model in its Claude 5.5 line, and states that Sonnet 5.5 at Max effort performs comparably to Opus 5.5 on several evaluations while Opus 5.5 remains stronger on open-ended work. The GPT-6 Sol column is quoted from OpenAI, not re-run, and Anthropic notes an OpenAI image-understanding bug that may not yet be reflected in third-party scores. Two rows come from other publishers: Artificial Analysis ran GDPval-AA and AA-Briefcase on a pre-release Sonnet 5.5 deployment that Anthropic later found had a structured-output bug, and Surge AI runs Chartography. Sonnet 5.5 launched with the cyber safeguards first built for Opus-class models, so some security work was completed by a fallback model.

www.anthropic.com/claude-sonnet-5-5

Sources

Every page used for this benchmark, read on 17 October 2026.