OverviewReproduce a result

Reproduce a result

Build a full local run plan with every test, recipe and missing setting kept explicit.

1. Choose the model and tests

1tests in this plan
1verified public harness
0need access or recipe work

Blocked tests stay in the plan. Public code alone does not reproduce an evaluator’s score.

2. Review the plan

SWE-bench (Verified)Public harness

Maintainer instructions checked 24 October 2026. This command grades existing patches in Docker. An agent must first generate predictions with the chosen model; grading alone does not rerun model inference. Pin the agent, issue subset and Docker images. This is not a reconstruction of Vals’ bash-only harness. Inference, judging or local compute may incur costs; configure provider limits first.

Pinned dataset: princeton-nlp/SWE-bench_Verified ↗ @ c104f840cc67 · test · Benchmark author · Open

  1. Read the linked maintainer guide and pin the harness commit.
  2. Configure the exact model adapter, dataset revision, split and task limit.
  3. Match prompts, tools, sampling, repetitions and judge to the intended run.
  4. Run generation and scoring stages; retain raw outputs, task outcomes and logs.
swebench eval verified -p <model-predictions.jsonl> --run-id <unique-run-id> -j 1
Author / evaluator instructions ↗

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

  • Immutable model or endpoint version
  • Adapter verification against the pinned dataset
  • Harness commit
  • Prompt and tool policy
  • Reasoning effort
  • Sampling and seed
  • Task limit and repetitions
  • Judge and aggregation
  • Provider spending limits
  • Licence review for hosted or redistributed use

The dataset revision and split are pinned in this job. Model version, harness, prompts, tools, judge and sampling remain unmatched, so a fresh run is not an exact reproduction claim.

Understand this test →