AIEvals

Benchmarks

SWE Atlas Codebase QnA

Answering questions that require understanding a software codebase.

Codebase understandingScale AI / SEAL, reported by Reflection AIVendor-reported onlyReported by Reflection AI

How it is run

Reflection AI publishes Codebase QnA results for Beam, GLM 5.3 and Kimi K3. These are the announcement’s figures rather than an independently imported author board.

What it cannot tell you

Codebase QnA is distinct from SWE Atlas Refactoring and Test Writing. The checked source does not specify harness, effort or trial count.

Why there is no independent table

No independent suite in this dataset runs SWE Atlas Codebase QnA. The numbers below are the ones model publishers report about their own models, which is a different kind of evidence: each publisher chooses its own harness and effort level, so a row is readable on its own and not against the other rows.

Read this with

Nothing else on this site shares this benchmark's family or category, so these are other evaluations the same publisher reports.

  • τ³-BankingBanking support with policy retrieval and multi-step tool calls.
  • IFBenchGeneralization to 58 verifiable output constraints absent from conventional instruction tests.
  • SWE-Bench Pro V2 HARDRepository issue resolution on the HARD subset of the public V2 set. Models must edit an existing codebase to satisfy the issue’s verification tests.
  • SWE-Bench Pro v1Resolving software-engineering issues in existing repositories.

As reported by each publisher

Kept separate per publisher, because the settings are not shared.

Reflection AI

Introducing Beam: model capability comparisons · 5 October 2026

RankModelScore
3Beam34.6%
2GLM 5.361.0%
1Kimi K368.0%

Ranks compare reported scores in the same published row; ties share a rank.

Reflection AI’s four capability tables, read 6 October 2026. All 21 rows and eight model columns are transcribed; NR means not reported and remains blank. The source’s Qwen 3.8 Max column uses this catalogue’s Qwen3.8 Max spelling. Beam results are publisher reports; Reflection AI cites Artificial Analysis and Datacurve for comparison-model evaluations. The announcement does not specify a common harness, effort or trial count for every row. Ranks are derived within each published row from reported values only, with ties sharing a rank; they do not establish an independent or overall model ranking. AutomationBench uses the public split; AA-Omniscience is an index on the public split only, calculated by Reflection AI. BrowseComp and DeepSearchQA use context management. AA-LCR has no stated version. These settings remain separate from other source fields.

reflection.ai/blog/introducing-beam

Sources

Every page used for this benchmark, read on 17 October 2026.