BenchmarksDeepSWE v1.1

DeepSWE v1.1

Original, long-horizon software-engineering tasks in real codebases, scored as pass@1 over attempted rollouts.

Software engineeringCatalogue onlyRecipe not catalogued
Author
Datacurve
Evidence
74 results9 comparison groups
Dataset tasks
Not pinned
Split
No revision
License
Not recorded
Can I run it here?Not catalogued for execution

This test is catalogued, but no pinned dataset, verified adapter or online runner is recorded here.

See the run plan ↗

What it measures

What the test does

Datacurve publishes 70 configurations across 28 models on mini-swe-agent for v1.1, updated 22 September 2026. The suite has 113 tasks across 91 repositories and five languages. This table shows each model's highest published pass@1 configuration, with its effort, task coverage and 95% confidence interval. Context-window failures and agent timeouts count as failures; provider, verifier and network errors are excluded.

What it can and cannot show

This measures software engineering, not overall model capability. Effort settings and scored attempt counts vary, and nearby confidence intervals overlap. Author runs and model-publisher reports are separate comparisons. GPT-6.1 Sol has no result in this author snapshot; OpenAI's release score is shown separately below.

Scores are comparable only inside the same split, metric, tool policy, judge and reasoning effort. Choose one comparison group below.

Scores

Choose one evaluator’s field. The bars show its recorded results; open the full list for sources and run conditions.

datacurveDeepSWE v1.1 · Datacurve

datacurveDatacurve% · Higher is better
  1. 1GPT-6 Astra74.12%
  2. 2Gemini 3.8 Flash73.83%
  3. 3Claude Opus 573.65%
  4. 4GPT-5.6 Sol72.67%
  5. 5Claude Fable 569.91%
  6. 6GPT-5.6 Terra69.62%
  7. 7GLM 5.368.96%
  8. 8Kimi K368.51%
  9. 9Grok 4.667.48%
  10. 10GPT-5.6 Luna67.19%
  11. 11GPT-5.567.04%
  12. 12Gemini 3.7 Flash65.49%

Bars start at zero. 28 reported results in this comparison group; ties share a rank. Published effort settings may vary. Source ↗

All 28 recorded results and run conditions

28 results

Results (%)
Rank / modelResult (%)SourceRun
1GPT-6 Astra74.12datacurve
Details

Best published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.

Metric
DeepSWE v1.1
Read
2026-09-22
Effort
xhigh
Tasks / attempts
113 / 452
Confidence interval
71.25 – 76.98
Build a reproduction plan →
2Gemini 3.8 Flash73.83datacurve
Details

Best published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.

Metric
DeepSWE v1.1
Read
2026-09-22
Effort
high
Tasks / attempts
113 / 447
Confidence interval
72.41 – 75.24
Build a reproduction plan →
3Claude Opus 573.65datacurve
Details

Best published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.

Metric
DeepSWE v1.1
Read
2026-09-22
Effort
max
Tasks / attempts
113 / 444
Confidence interval
69.78 – 77.52
Build a reproduction plan →
4GPT-5.6 Sol72.67datacurve
Details

Best published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.

Metric
DeepSWE v1.1
Read
2026-09-22
Effort
max
Tasks / attempts
113 / 450
Confidence interval
69.84 – 75.5
Build a reproduction plan →
5Claude Fable 569.91datacurve
Details

Best published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.

Metric
DeepSWE v1.1
Read
2026-09-22
Effort
xhigh
Tasks / attempts
113 / 452
Confidence interval
66.67 – 73.16
Build a reproduction plan →
6GPT-5.6 Terra69.62datacurve
Details

Best published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.

Metric
DeepSWE v1.1
Read
2026-09-22
Effort
max
Tasks / attempts
113 / 451
Confidence interval
67.07 – 72.18
Build a reproduction plan →
7GLM 5.368.96datacurve
Details

Best published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.

Metric
DeepSWE v1.1
Read
2026-09-22
Effort
max
Tasks / attempts
113 / 451
Confidence interval
65.94 – 71.98
Build a reproduction plan →
8Kimi K368.51datacurve
Details

Best published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.

Metric
DeepSWE v1.1
Read
2026-09-22
Effort
max
Tasks / attempts
113 / 451
Confidence interval
63.98 – 73.05
Build a reproduction plan →
9Grok 4.667.48datacurve
Details

Best published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.

Metric
DeepSWE v1.1
Read
2026-09-22
Effort
medium
Tasks / attempts
113 / 452
Confidence interval
65.2 – 69.76
Build a reproduction plan →
10GPT-5.6 Luna67.19datacurve
Details

Best published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.

Metric
DeepSWE v1.1
Read
2026-09-22
Effort
max
Tasks / attempts
113 / 448
Confidence interval
63.2 – 71.18
Build a reproduction plan →
11GPT-5.567.04datacurve
Details

Best published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.

Metric
DeepSWE v1.1
Read
2026-09-22
Effort
xhigh
Tasks / attempts
113 / 452
Confidence interval
60.57 – 73.5
Build a reproduction plan →
12Gemini 3.7 Flash65.49datacurve
Details

Best published pass@1 configuration per model on mini-swe-agent. Effort and task coverage vary; model-publisher reports are shown separately.

Metric
DeepSWE v1.1
Read
2026-09-22
Effort
medium
Tasks / attempts
113 / 452
Confidence interval
62.4 – 68.57
Build a reproduction plan →

Can it run here?

What is known about running this test, and what is still unknown.

No dataset revision is pinned for this benchmark. Published results remain useful evidence, but this page does not claim that the evaluator’s tasks, judge or harness can be executed here.

Coverage stage
Catalogue only
Recipe state
Recipe not catalogued

Remaining work

  • No immutable dataset revision has been verified for this benchmark.
  • No adapter or scorer fixture is recorded.
  • Online execution is not enabled.

Coverage is derived from this catalogue’s dataset registry, not inferred from a benchmark name. A dataset revision and an execution result are different evidence.

Reproduce it

Recipe not catalogued

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

Execution documentation ↗
Build a run plan →