ITBench-AA
Kubernetes incident diagnosis from alerts, traces and topology snapshots.
IT operationsIBM / Artificial Analysis42 models run by Artificial AnalysisUpdated 30 September 2026
How it is run
59 incidents (40 public, 19 private), three repeats, Stirrup harness. Score is average precision at full recall: false negatives score zero.
What it cannot tell you
This evaluates diagnosis from offline snapshots, not live incident remediation or the original IBM resolution-rate metric.
- Leader
- GPT-5.6 Sol56.21% · Artificial Analysis results, grouped here by release
- Top-ten spread
- 9.55%Leader minus tenth place, or last place in smaller fields.
- Median
- 38.27%of the 42 rows published
- No published result
- 28of 46 models on this site
Artificial Analysis results by release
Highest published effort per model release, grouped here from AA’s public results. Grouping ranks are derived here; source settings and model-publisher reports stay separate.
| # | Model | Score | Effort |
|---|---|---|---|
| 1 | GPT-5.6 Sol | 56.21% | max |
| 2 | Gemini 3.8 Flash | 52.54% | high |
| 3 | GLM 5.3 Flash | 51.24% | max |
| 4 | GPT-5.6 Terra | 51.04% | max |
| 5 | Claude Fable 5.1 | 49.53% | max |
| 6 | GPT-6 Sol | 49.44% | max |
| 7 | GPT-6 Astra | 48.59% | max |
| 8 | Kimi K3 | 47.69% | max |
| 9 | DeepSeek V4.1 Flash | 46.92% | max |
| 10 | Claude Opus 4.7 | 46.66% | max |
| 11 | GLM-5.3 | 46.08% | max |
| 12 | GPT-5.5 | 45.81% | xhigh |
| 13 | GLM-5.2 | 42.66% | max |
| 14 | Qwen3.7 Max | 42.47% | Not specified |
| 15 | Grok 4.7 | 42.08% | xhigh |
| 16 | Gemini 3.5 Flash | 40.35% | high |
| 17 | GPT-5.6 Luna | 40.32% | max |
| 18 | GLM-5.1 | 40.25% | Not specified |
| 18 | Qwen3.8 Max (0902) | 40.25% | Not specified |
| 20 | Claude Sonnet 4.6 | 39.80% | max |
| 21 | DeepSeek V4 Pro 0424 | 38.32% | max |
| 22 | MiMo-V2.5-Pro | 38.23% | Not specified |
| 22 | Claude Opus 5.5 | 38.23% | max |
| 24 | Gemma 4 31B | 37.29% | Not specified |
| 25 | Qwen3.5 27B | 35.50% | Not specified |
| 26 | GPT-5.4 mini | 35.22% | xhigh |
| 27 | Qwen3.5 397B A17B | 34.09% | Not specified |
| 28 | Muse Spark 1.3 | 33.24% | max |
| 29 | Grok 4.3 | 32.72% | high |
| 30 | DeepSeek V4 Flash 0420 | 31.52% | max |
| 31 | Kimi K2.6 | 31.19% | Not specified |
| 32 | Gemini 3.1 Pro Preview | 30.33% | Not specified |
| 33 | Step 3.7 Flash | 30.27% | high |
| 34 | Claude 4.5 Haiku | 27.31% | Not specified |
| 35 | MiniMax-M2.7 | 26.46% | Not specified |
| 36 | GPT-5.4 nano | 24.39% | xhigh |
| 37 | Gemma 4 26B A4B | 23.63% | Not specified |
| 38 | Qwen3.5 35B A3B | 21.52% | Not specified |
| 39 | Grok 4.1 Fast | 17.89% | Not specified |
| 40 | gpt-oss-120b | 5.65% | high |
| 41 | Nemotron 3 Super 120B A12B | 1.13% | Not specified |
| 42 | Llama 3.3 Instruct 70B | 0.56% | Not specified |
No published result tracked
28 of the 46 models on this site have no published number on this benchmark, so their cells in the home table stay blank rather than reading as zero.
Step 5 PreviewGPT-6 LunaQwen3.8 2.4T A95BQwen3.8-Flash-NextMiMo-V2.6-FlashClaude Fable 5Grok 4.6Claude Opus 4.8Gemini 3.7 FlashMuse Spark 1.2GPT-5.4Grok 4.5DeepSeek V4.1 Flash 1017GLM-LegalClaude Haiku 5.5BeamGemini 4 ArgonClaude Sonnet 5.5Claude Opus 5Claude Sonnet 5GPT-6.1 SolDeepSeek V4 ProQwen3.8 MaxMiniMax M3Hunyuan Hy4 PreviewMiMo V2.6 ProInklingMistral Medium 3.5
Sources
Every page used for this benchmark, read on 17 October 2026.
- Benchmark site or the page that publishes itartificialanalysis.ai/evaluations/itbench-aa
- Independent resultsartificialanalysis.ai/evaluations/itbench-aa