Benchmarksxdev-bench-hard (Internal)
xdev-bench-hard (Internal)
Expert-curated source-code vulnerability discovery and exploit development, measured by pass@3.
Test definition
- Author / evaluator
- Reported by Moonshot AI
- Version / family
- xdev-bench-hard (Internal) / xdev-bench-hard
- Evidence available
- 9 results in 2 comparison groups
How the test is scored
Internal xdev-bench-hard pass@3, random subset of runs with no hard timeout or output-token limit. Kimi uses Kimi Code; GLM/DeepSeek use Claude Code. The supplementary chart labels DeepSeek as the undated Flash model, while this main table labels 1017; both claims retain their own labels.
Comparison notes
Publisher-reported results, read 24 October 2026. Compare reported values only within the same source row; tool settings, subsets, versions, judges and metrics must not be combined.
A different task split, tool policy, judge or reasoning effort can change the result. Choose one comparison group below.
Results by evaluator
Select a publication and metric to compare the recorded field.
selfKimi K3.1 Pro: complete release evaluation · xdev-bench-hard (Internal)
- 85.7%
- 81.0%
- 57.1%
Bars start at zero. 3 reported results in this comparison group; ties share a rank. Source ↗
All observations
Every numeric result, with its source and reported run conditions.
9 results
| Model | Result (%) | Source | Run |
|---|---|---|---|
| Kimi K3.1 Pro | 81.0 | self | DetailsInternal xdev-bench-hard pass@3, random subset of runs with no hard timeout or output-token limit. Kimi uses Kimi Code; GLM/DeepSeek use Claude Code. The supplementary chart labels DeepSeek as the undated Flash model, while this main table labels 1017; both claims retain their own labels. Read 24 October 2026. All 47 source rows and six comparison columns are retained, with text-only HLE and both ZeroBench metrics split explicitly. Kimi uses max reasoning effort, temperature 1.0 and top-p 0.95 unless a row states otherwise; proprietary comparison columns are labelled max with fallback. Comparison figures include quoted external results and mixed harnesses. Missing cells remain null and do not enter ranks. Lower hallucination rates rank first; every other rank compares larger values within the same row. Terminal-Bench Science’s September DeepSeek score is attributed separately according to footnote 2. These publication claims do not populate independent boards. Highspeed has no independently identified results and inherits none of these scores.
|
| 81.0xdev-bench-hard (figure pass@3) | self | DetailsGPT-5.6 Sol is marked as a third-party evaluation. The checkpoint without risk reduction is an unreleased ablation, not a catalogue model. The figure labels DeepSeek as undated Flash; this differs from the main table’s 1017 label. All labelled figures transcribed from the supplementary image, read 24 October 2026. No hard timeout or output limit; random run subset. Pass@3 and average success rate are different metrics. Exact token-usage labels are retained in the row note; the ablation is not presented as a released model.
| |
| DeepSeek V4.1 Flash 1017 | 85.7 | self | DetailsInternal xdev-bench-hard pass@3, random subset of runs with no hard timeout or output-token limit. Kimi uses Kimi Code; GLM/DeepSeek use Claude Code. The supplementary chart labels DeepSeek as the undated Flash model, while this main table labels 1017; both claims retain their own labels. Read 24 October 2026. All 47 source rows and six comparison columns are retained, with text-only HLE and both ZeroBench metrics split explicitly. Kimi uses max reasoning effort, temperature 1.0 and top-p 0.95 unless a row states otherwise; proprietary comparison columns are labelled max with fallback. Comparison figures include quoted external results and mixed harnesses. Missing cells remain null and do not enter ranks. Lower hallucination rates rank first; every other rank compares larger values within the same row. Terminal-Bench Science’s September DeepSeek score is attributed separately according to footnote 2. These publication claims do not populate independent boards. Highspeed has no independently identified results and inherits none of these scores.
|
| GLM 5.3 | 57.1 | self | DetailsInternal xdev-bench-hard pass@3, random subset of runs with no hard timeout or output-token limit. Kimi uses Kimi Code; GLM/DeepSeek use Claude Code. The supplementary chart labels DeepSeek as the undated Flash model, while this main table labels 1017; both claims retain their own labels. Read 24 October 2026. All 47 source rows and six comparison columns are retained, with text-only HLE and both ZeroBench metrics split explicitly. Kimi uses max reasoning effort, temperature 1.0 and top-p 0.95 unless a row states otherwise; proprietary comparison columns are labelled max with fallback. Comparison figures include quoted external results and mixed harnesses. Missing cells remain null and do not enter ranks. Lower hallucination rates rank first; every other rank compares larger values within the same row. Terminal-Bench Science’s September DeepSeek score is attributed separately according to footnote 2. These publication claims do not populate independent boards. Highspeed has no independently identified results and inherits none of these scores.
|
| 57.1xdev-bench-hard (figure pass@3) | self | DetailsGPT-5.6 Sol is marked as a third-party evaluation. The checkpoint without risk reduction is an unreleased ablation, not a catalogue model. The figure labels DeepSeek as undated Flash; this differs from the main table’s 1017 label. All labelled figures transcribed from the supplementary image, read 24 October 2026. No hard timeout or output limit; random run subset. Pass@3 and average success rate are different metrics. Exact token-usage labels are retained in the row note; the ablation is not presented as a released model.
| |
| GPT-5.6 Sol | 100.0xdev-bench-hard (figure pass@3) | self | DetailsGPT-5.6 Sol is marked as a third-party evaluation. The checkpoint without risk reduction is an unreleased ablation, not a catalogue model. The figure labels DeepSeek as undated Flash; this differs from the main table’s 1017 label. All labelled figures transcribed from the supplementary image, read 24 October 2026. No hard timeout or output limit; random run subset. Pass@3 and average success rate are different metrics. Exact token-usage labels are retained in the row note; the ablation is not presented as a released model.
|
| Kimi checkpoint without cyber risk-reduction | 90.5xdev-bench-hard (figure pass@3) | self | DetailsGPT-5.6 Sol is marked as a third-party evaluation. The checkpoint without risk reduction is an unreleased ablation, not a catalogue model. The figure labels DeepSeek as undated Flash; this differs from the main table’s 1017 label. All labelled figures transcribed from the supplementary image, read 24 October 2026. No hard timeout or output limit; random run subset. Pass@3 and average success rate are different metrics. Exact token-usage labels are retained in the row note; the ablation is not presented as a released model.
|
| DeepSeek V4.1 Flash | 85.7xdev-bench-hard (figure pass@3) | self | DetailsGPT-5.6 Sol is marked as a third-party evaluation. The checkpoint without risk reduction is an unreleased ablation, not a catalogue model. The figure labels DeepSeek as undated Flash; this differs from the main table’s 1017 label. All labelled figures transcribed from the supplementary image, read 24 October 2026. No hard timeout or output limit; random run subset. Pass@3 and average success rate are different metrics. Exact token-usage labels are retained in the row note; the ablation is not presented as a released model.
|
| Kimi K3 | 33.3xdev-bench-hard (figure pass@3) | self | DetailsGPT-5.6 Sol is marked as a third-party evaluation. The checkpoint without risk reduction is an unreleased ablation, not a catalogue model. The figure labels DeepSeek as undated Flash; this differs from the main table’s 1017 label. All labelled figures transcribed from the supplementary image, read 24 October 2026. No hard timeout or output limit; random run subset. Pass@3 and average success rate are different metrics. Exact token-usage labels are retained in the row note; the ablation is not presented as a released model.
|
Reproduction
Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.
Execution documentation ↗