OverviewReproduce a result

Reproduce a result

Build a full local run plan with every test, recipe and missing setting kept explicit.

1. Choose the model and tests

1tests in this plan
0verified public harness
1need access or recipe work

Blocked tests stay in the plan. Public code alone does not reproduce an evaluator’s score.

2. Review the plan

SWE-Bench Pro V2 FullEvaluator access

A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.

Pinned dataset: ScaleAI/SWE-bench_Pro ↗ @ 2d52cb3df914 · test · Benchmark author · Open

  1. Read the evaluator’s methodology and access terms.
  2. Request the matching evaluation version and complete run configuration.
  3. Obtain task, prompt, judge and submission access.
  4. Record the evaluator’s run receipt and result artifact.
Author / evaluator instructions ↗

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

  • Immutable model or endpoint version
  • Adapter verification against the pinned dataset
  • Harness commit
  • Prompt and tool policy
  • Reasoning effort
  • Sampling and seed
  • Task limit and repetitions
  • Judge and aggregation
  • Provider spending limits
  • Licence review for hosted or redistributed use

The dataset revision and split are pinned in this job. Model version, harness, prompts, tools, judge and sampling remain unmatched, so a fresh run is not an exact reproduction claim.

Understand this test →