Benchmarksxdev-bench-hard (average success rate)
xdev-bench-hard (average success rate)
Mean success rate across runs on Moonshot’s internal vulnerability and exploitation stress test.
Test definition
- Author / evaluator
- Reported by Moonshot AI
- Version / family
- xdev-bench-hard (average success rate) / xdev-bench-hard (average success rate)
- Evidence available
- 6 results in 1 comparison groups
How the test is scored
The supplementary figure reports average success rate separately from pass@3, using Kimi Code, Claude Code or Codex according to model.
Comparison notes
Average success rate is not the share of problems solved at pass@3. The unreleased unfiltered checkpoint is only an ablation.
A different task split, tool policy, judge or reasoning effort can change the result. Choose one comparison group below.
Results by evaluator
Select a publication and metric to compare the recorded field.
selfxdev-bench-hard: full supplementary figure · xdev-bench-hard (average success rate)
- 77.8%
- 71.4%
- 64.5%
- 57.1%
- 34.9%
- 15.9%
Bars start at zero. 6 reported results in this comparison group; ties share a rank. Source ↗
All observations
Every numeric result, with its source and reported run conditions.
6 results
| Model | Result (%) | Source | Run |
|---|---|---|---|
| GPT-5.6 Sol | 77.8 | self | DetailsAverage success rate, separate from best-of-three problems solved. Harnesses: Codex for GPT, Kimi Code for Kimi/checkpoint/K3, Claude Code for DeepSeek/GLM. Input/output tokens per successful trajectory, in figure order: 93.5M/336K, 216.4M/499K, 162.2M/1.5M, 159.8M/546K, 381.1M/3.6M, 115.7M/475K. Token usage is conditional on successful trajectories and is not a capability ranking. All labelled figures transcribed from the supplementary image, read 24 October 2026. No hard timeout or output limit; random run subset. Pass@3 and average success rate are different metrics. Exact token-usage labels are retained in the row note; the ablation is not presented as a released model.
|
| Kimi checkpoint without cyber risk-reduction | 71.4 | self | DetailsAverage success rate, separate from best-of-three problems solved. Harnesses: Codex for GPT, Kimi Code for Kimi/checkpoint/K3, Claude Code for DeepSeek/GLM. Input/output tokens per successful trajectory, in figure order: 93.5M/336K, 216.4M/499K, 162.2M/1.5M, 159.8M/546K, 381.1M/3.6M, 115.7M/475K. Token usage is conditional on successful trajectories and is not a capability ranking. All labelled figures transcribed from the supplementary image, read 24 October 2026. No hard timeout or output limit; random run subset. Pass@3 and average success rate are different metrics. Exact token-usage labels are retained in the row note; the ablation is not presented as a released model.
|
| DeepSeek V4.1 Flash | 57.1 | self | DetailsAverage success rate, separate from best-of-three problems solved. Harnesses: Codex for GPT, Kimi Code for Kimi/checkpoint/K3, Claude Code for DeepSeek/GLM. Input/output tokens per successful trajectory, in figure order: 93.5M/336K, 216.4M/499K, 162.2M/1.5M, 159.8M/546K, 381.1M/3.6M, 115.7M/475K. Token usage is conditional on successful trajectories and is not a capability ranking. All labelled figures transcribed from the supplementary image, read 24 October 2026. No hard timeout or output limit; random run subset. Pass@3 and average success rate are different metrics. Exact token-usage labels are retained in the row note; the ablation is not presented as a released model.
|
| Kimi K3.1 Pro | 64.5 | self | DetailsAverage success rate, separate from best-of-three problems solved. Harnesses: Codex for GPT, Kimi Code for Kimi/checkpoint/K3, Claude Code for DeepSeek/GLM. Input/output tokens per successful trajectory, in figure order: 93.5M/336K, 216.4M/499K, 162.2M/1.5M, 159.8M/546K, 381.1M/3.6M, 115.7M/475K. Token usage is conditional on successful trajectories and is not a capability ranking. All labelled figures transcribed from the supplementary image, read 24 October 2026. No hard timeout or output limit; random run subset. Pass@3 and average success rate are different metrics. Exact token-usage labels are retained in the row note; the ablation is not presented as a released model.
|
| GLM 5.3 | 34.9 | self | DetailsAverage success rate, separate from best-of-three problems solved. Harnesses: Codex for GPT, Kimi Code for Kimi/checkpoint/K3, Claude Code for DeepSeek/GLM. Input/output tokens per successful trajectory, in figure order: 93.5M/336K, 216.4M/499K, 162.2M/1.5M, 159.8M/546K, 381.1M/3.6M, 115.7M/475K. Token usage is conditional on successful trajectories and is not a capability ranking. All labelled figures transcribed from the supplementary image, read 24 October 2026. No hard timeout or output limit; random run subset. Pass@3 and average success rate are different metrics. Exact token-usage labels are retained in the row note; the ablation is not presented as a released model.
|
| Kimi K3 | 15.9 | self | DetailsAverage success rate, separate from best-of-three problems solved. Harnesses: Codex for GPT, Kimi Code for Kimi/checkpoint/K3, Claude Code for DeepSeek/GLM. Input/output tokens per successful trajectory, in figure order: 93.5M/336K, 216.4M/499K, 162.2M/1.5M, 159.8M/546K, 381.1M/3.6M, 115.7M/475K. Token usage is conditional on successful trajectories and is not a capability ranking. All labelled figures transcribed from the supplementary image, read 24 October 2026. No hard timeout or output limit; random run subset. Pass@3 and average success rate are different metrics. Exact token-usage labels are retained in the row note; the ablation is not presented as a released model.
|
Reproduction
Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.
Execution documentation ↗