OverviewReproduce a result
Reproduce a result
Build a full local run plan with every test, recipe and missing setting kept explicit.
1. Choose the model and tests
1tests in this plan
0verified public harness
1need access or recipe work
Blocked tests stay in the plan. Public code alone does not reproduce an evaluator’s score.
2. Review the plan
SWE Atlas Codebase QnARecipe not catalogued
Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.
Pinned dataset: ArtificialAnalysis/SWE-Atlas-QnA-AA ↗ @ 1f28d51b6e19 · test · Evaluator adaptation · Open
- Open the benchmark’s author page.
- Confirm code and dataset availability, licence, task split and scoring.
- Record the exact harness commit, model, sampling and judge before adding a command.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Adapter verification against the pinned dataset
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
The dataset revision and split are pinned in this job. Model version, harness, prompts, tools, judge and sampling remain unmatched, so a fresh run is not an exact reproduction claim.
Understand this test →