AIEvals

Benchmarks

SWE-bench Multilingual

Resolving issues in existing software repositories across several programming languages.

Repository codingSWE-bench, reported by Reflection AIVendor-reported onlyReported by Anthropic, Reflection AI

How it is run

Reflection AI reports Beam and Nemotron 3 Ultra on the multilingual suite. Other model cells are not reported.

What it cannot tell you

Multilingual tasks are separate from SWE-bench Verified and SWE-Bench Pro. The announcement does not specify its harness or evaluation configuration.

Why there is no independent table

No independent suite in this dataset runs SWE-bench Multilingual. The numbers below are the ones model publishers report about their own models, which is a different kind of evidence: each publisher chooses its own harness and effort level, so a row is readable on its own and not against the other rows.

Read this with

Other versions of the same evaluation. Versions are not comparable: the task set changes between them.

  • SWE-bench MultimodalSame family. Resolving software-engineering issues that include visual context such as screenshots and design mockups.
  • SWE-bench (Verified)Same family. Whether a model can write a patch that resolves a real GitHub issue, judged by running the project's own unit tests.
  • SWE-Bench Pro (version unspecified)Same category. Resolving long-horizon repository issues requiring large, multi-file code changes.
  • SWE-Bench Pro V2 FullSame category. Repository issue resolution on the full public V2 set. Models must edit an existing codebase to satisfy the issue’s verification tests.
  • SWE-Bench Pro V2 HARDSame category. Repository issue resolution on the HARD subset of the public V2 set. Models must edit an existing codebase to satisfy the issue’s verification tests.
  • SWE-Bench Pro v1Same category. Resolving software-engineering issues in existing repositories.

As reported by each publisher

Kept separate per publisher, because the settings are not shared.

Anthropic

Claude Haiku 5.5: system card capability evaluations · 7 October 2026

RankModelScore
2Claude Haiku 5.583.7%
3Claude Haiku 4.567.4%
1Claude Sonnet 5.590.3%

Ranks compare reported scores in the same published row; ties share a rank.

SWEBench Multilingual. §8.2: 300 problems across nine languages; five trials.

Section 8, pages 111–139 of the Haiku 5.5 system card, read 8 October 2026. Values come from exact prose, tables and labelled figures. Unless specified, Haiku uses adaptive thinking at max effort and five trials with evaluation-specific context up to 1M. Each row retains its subset, metric, effort and source notes; no score is assigned to an unspecified SWE-Bench Pro version. Terminal runs use safeguards without fallback and no internet egress. ProgramBench uses filtered tasks and hidden-test pass rate without a time limit. OSWorld partial and strict results use the official 82-task offline subset. CAD uses 0–1 voxel IoU. Healthcare raw and length-adjusted scores stay separate. Life-science results disable Haiku biology safeguards. Independently run figures quoted by Anthropic remain publisher reports here. Missing comparison values are not inferred from rounded charts.

www.anthropic.com/claude-haiku-5-5-system-card

Reflection AI

Introducing Beam: model capability comparisons · 5 October 2026

RankModelScore
1Beam78.0%
2Nemotron 3 Ultra67.7%

Ranks compare reported scores in the same published row; ties share a rank.

Reflection AI’s four capability tables, read 6 October 2026. All 21 rows and eight model columns are transcribed; NR means not reported and remains blank. The source’s Qwen 3.8 Max column uses this catalogue’s Qwen3.8 Max spelling. Beam results are publisher reports; Reflection AI cites Artificial Analysis and Datacurve for comparison-model evaluations. The announcement does not specify a common harness, effort or trial count for every row. Ranks are derived within each published row from reported values only, with ties sharing a rank; they do not establish an independent or overall model ranking. AutomationBench uses the public split; AA-Omniscience is an index on the public split only, calculated by Reflection AI. BrowseComp and DeepSearchQA use context management. AA-LCR has no stated version. These settings remain separate from other source fields.

reflection.ai/blog/introducing-beam

Sources

Every page used for this benchmark, read on 17 October 2026.