AIEvals

Benchmarks

MILU

Culturally grounded knowledge comprehension in ten Indic languages and English, reported as mean accuracy across eleven languages.

Multilingual knowledgeReported by AnthropicVendor-reported onlyReported by Anthropic

How it is run

Section 8.12.2 averages five trials at max effort with safety classifiers enabled, no tools and no custom system prompts.

What it cannot tell you

Blocked or unanswered questions are excluded; the language mix differs from Global MMLU. Its aggregate cannot substitute for individual-language reliability.

Why there is no independent table

No independent suite in this dataset runs MILU. The numbers below are the ones model publishers report about their own models, which is a different kind of evidence: each publisher chooses its own harness and effort level, so a row is readable on its own and not against the other rows.

Read this with

Benchmarks in the same category, which usually disagree with this one on purpose.

  • Global MMLUSame category. Knowledge and reasoning accuracy across 42 languages, including both high-resource and low-resource languages.

As reported by each publisher

Kept separate per publisher, because the settings are not shared.

Anthropic

Claude Haiku 5.5: system card capability evaluations · 7 October 2026

RankModelScore
4Claude Haiku 5.587.6%
3Claude Sonnet 5.591.6%
1Claude Opus 5.593.1%
2Claude Fable 5.193.0%

Ranks compare reported scores in the same published row; ties share a rank.

MILU. §8.12.2: average across eleven languages, five trials, max, no tools/custom prompts. Blocked or unanswered questions excluded.

Section 8, pages 111–139 of the Haiku 5.5 system card, read 8 October 2026. Values come from exact prose, tables and labelled figures. Unless specified, Haiku uses adaptive thinking at max effort and five trials with evaluation-specific context up to 1M. Each row retains its subset, metric, effort and source notes; no score is assigned to an unspecified SWE-Bench Pro version. Terminal runs use safeguards without fallback and no internet egress. ProgramBench uses filtered tasks and hidden-test pass rate without a time limit. OSWorld partial and strict results use the official 82-task offline subset. CAD uses 0–1 voxel IoU. Healthcare raw and length-adjusted scores stay separate. Life-science results disable Haiku biology safeguards. Independently run figures quoted by Anthropic remain publisher reports here. Missing comparison values are not inferred from rounded charts.

www.anthropic.com/claude-haiku-5-5-system-card

Sources

Every page used for this benchmark, read on 17 October 2026.