AIEvals

Benchmarks

SWE-bench Multimodal

Resolving software-engineering issues that include visual context such as screenshots and design mockups.

Repository codingReported by AnthropicVendor-reported onlyReported by Anthropic

How it is run

Section 8.2 reports Claude results averaged over five trials with adaptive thinking at max effort for Haiku 5.5.

What it cannot tell you

Visual issue context changes the task set; these results are separate from Verified, Pro and Multilingual SWE-bench runs.

Why there is no independent table

No independent suite in this dataset runs SWE-bench Multimodal. The numbers below are the ones model publishers report about their own models, which is a different kind of evidence: each publisher chooses its own harness and effort level, so a row is readable on its own and not against the other rows.

Read this with

Other versions of the same evaluation. Versions are not comparable: the task set changes between them.

  • SWE-bench MultilingualSame family. Resolving issues in existing software repositories across several programming languages.
  • SWE-bench (Verified)Same family. Whether a model can write a patch that resolves a real GitHub issue, judged by running the project's own unit tests.
  • SWE-Bench Pro (version unspecified)Same category. Resolving long-horizon repository issues requiring large, multi-file code changes.
  • SWE-Bench Pro V2 FullSame category. Repository issue resolution on the full public V2 set. Models must edit an existing codebase to satisfy the issue’s verification tests.
  • SWE-Bench Pro V2 HARDSame category. Repository issue resolution on the HARD subset of the public V2 set. Models must edit an existing codebase to satisfy the issue’s verification tests.
  • SWE-Bench Pro v1Same category. Resolving software-engineering issues in existing repositories.

As reported by each publisher

Kept separate per publisher, because the settings are not shared.

Anthropic

Claude Haiku 5.5: system card capability evaluations · 7 October 2026

RankModelScore
2Claude Haiku 5.530.7%
3Claude Haiku 4.519.8%
1Claude Sonnet 5.554.3%

Ranks compare reported scores in the same published row; ties share a rank.

SWE-bench Multimodal. §8.2: visual context in issue descriptions; five trials.

Section 8, pages 111–139 of the Haiku 5.5 system card, read 8 October 2026. Values come from exact prose, tables and labelled figures. Unless specified, Haiku uses adaptive thinking at max effort and five trials with evaluation-specific context up to 1M. Each row retains its subset, metric, effort and source notes; no score is assigned to an unspecified SWE-Bench Pro version. Terminal runs use safeguards without fallback and no internet egress. ProgramBench uses filtered tasks and hidden-test pass rate without a time limit. OSWorld partial and strict results use the official 82-task offline subset. CAD uses 0–1 voxel IoU. Healthcare raw and length-adjusted scores stay separate. Life-science results disable Haiku biology safeguards. Independently run figures quoted by Anthropic remain publisher reports here. Missing comparison values are not inferred from rounded charts.

www.anthropic.com/claude-haiku-5-5-system-card

Sources

Every page used for this benchmark, read on 17 October 2026.