BenchmarksDeepSWE v1.1
Deep SWE v1.1
Original, long-horizon software-engineering tasks in real codebases, scored as pass@1 over attempted rollouts.
- Author
- Datacurve
- Evidence
- 74 results9 comparison groups
- Dataset tasks
- Not pinned
- Split
- No revision
- License
- Not recorded
This test is catalogued, but no pinned dataset, verified adapter or online runner is recorded here.
See the run plan ↗What it measures
What the test does
Datacurve publishes 70 configurations across 28 models on mini-swe-agent for v1.1, updated 22 September 2026. The suite has 113 tasks across 91 repositories and five languages. This table shows each model's highest published pass@1 configuration, with its effort, task coverage and 95% confidence interval. Context-window failures and agent timeouts count as failures; provider, verifier and network errors are excluded.
What it can and cannot show
This measures software engineering, not overall model capability. Effort settings and scored attempt counts vary, and nearby confidence intervals overlap. Author runs and model-publisher reports are separate comparisons. GPT-6.1 Sol has no result in this author snapshot; OpenAI's release score is shown separately below.
Scores are comparable only inside the same split, metric, tool policy, judge and reasoning effort. Choose one comparison group below.
Scores
Choose one evaluator’s field. The bars show its recorded results; open the full list for sources and run conditions.
datacurveDeepSWE v1.1 · Datacurve
- 74.12%
- 73.83%
- 73.65%
- 72.67%
- 69.91%
- 69.62%
- 68.96%
- 68.51%
- 67.48%
- 67.19%
- 67.04%
- 65.49%
Bars start at zero. 28 reported results in this comparison group; ties share a rank. Published effort settings may vary. Source ↗
All 28 recorded results and run conditions
28 results
| Rank / model | Result (%) | Source | Run |
|---|---|---|---|
| 1GPT-6 Astra | 74.12 | datacurve | DetailsBest published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.
|
| 2Gemini 3.8 Flash | 73.83 | datacurve | DetailsBest published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.
|
| 3Claude Opus 5 | 73.65 | datacurve | DetailsBest published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.
|
| 4GPT-5.6 Sol | 72.67 | datacurve | DetailsBest published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.
|
| 5Claude Fable 5 | 69.91 | datacurve | DetailsBest published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.
|
| 6GPT-5.6 Terra | 69.62 | datacurve | DetailsBest published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.
|
| 7GLM 5.3 | 68.96 | datacurve | DetailsBest published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.
|
| 8Kimi K3 | 68.51 | datacurve | DetailsBest published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.
|
| 9Grok 4.6 | 67.48 | datacurve | DetailsBest published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.
|
| 10GPT-5.6 Luna | 67.19 | datacurve | DetailsBest published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.
|
| 11GPT-5.5 | 67.04 | datacurve | DetailsBest published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.
|
| 12Gemini 3.7 Flash | 65.49 | datacurve | DetailsBest published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.
|
Can it run here?
What is known about running this test, and what is still unknown.
No dataset revision is pinned for this benchmark. Published results remain useful evidence, but this page does not claim that the evaluator’s tasks, judge or harness can be executed here.
- Coverage stage
- Catalogue only
- Recipe state
- Recipe not catalogued
Remaining work
- No immutable dataset revision has been verified for this benchmark.
- No adapter or scorer fixture is recorded.
- Online execution is not enabled.
Coverage is derived from this catalogue’s dataset registry, not inferred from a benchmark name. A dataset revision and an execution result are different evidence.
Reproduce it
Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.
Execution documentation ↗