OverviewReproduce a result
Reproduce a result
Build a full local run plan with every test, recipe and missing setting kept explicit.
1. Choose the model and tests
Blocked tests stay in the plan. Public code alone does not reproduce an evaluator’s score.
2. Review the plan
τ³-BankingEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →ITBench-AAEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →MLCR-AAEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →ARC-AGI-3Recipe not catalogued
Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Open the benchmark’s author page.
- Confirm code and dataset availability, licence, task split and scoring.
- Record the exact harness commit, model, sampling and judge before adding a command.
2 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →SWE-Bench Pro V2 FullEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
Pinned dataset: ScaleAI/SWE-bench_Pro ↗ @ 2d52cb3df914 · test · Benchmark author · Open
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Adapter verification against the pinned dataset
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
- Licence review for hosted or redistributed use
The dataset revision and split are pinned in this job. Model version, harness, prompts, tools, judge and sampling remain unmatched, so a fresh run is not an exact reproduction claim.
Understand this test →SWE-Bench Pro V2 HARDEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
Pinned dataset: ScaleAI/SWE-bench_Pro ↗ @ 2d52cb3df914 · test · Benchmark author · Open
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Adapter verification against the pinned dataset
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
- Licence review for hosted or redistributed use
The dataset revision and split are pinned in this job. Model version, harness, prompts, tools, judge and sampling remain unmatched, so a fresh run is not an exact reproduction claim.
Understand this test →Vals RSI IndexEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →Vibe Code Bench 1–100Evaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →CUA-benchEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →AutomationBench-AAEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →SciCodeEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
Pinned dataset: SciCode1/SciCode ↗ @ 4510f6a6aa27 · test · Benchmark author · Open
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Adapter verification against the pinned dataset
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
The dataset revision and split are pinned in this job. Model version, harness, prompts, tools, judge and sampling remain unmatched, so a fresh run is not an exact reproduction claim.
Understand this test →GDP.pdfEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →CritPtEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →AA-OmniscienceEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →AA-LCR v1.1Evaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →Terminal-Bench 4.0Public harness
Author-verified command structure, checked 24 October 2026. The dataset is pinned; the harness, provider, agent and trial count still require configuration. This template does not establish the Artificial Analysis or Moonshot run settings. GPU sandbox and inference charges are possible.
Pinned dataset: harborframework/terminal-bench ↗ @ 6637b03798cf · tasks · Benchmark author · Open
- Install Harbor using the author’s run guide.
- Configure a GPU-capable sandbox and its spending limit.
- Choose the matching agent and provider/model identifier.
- Pin the Harbor commit, task subset, reasoning and sampling settings.
- Run the selected tasks; retain job logs, outcomes and configuration.
harbor run -d terminal-bench/terminal-bench@4.0.0 -e <gpu-sandbox> -a <agent> -m <provider/model> -k <repetitions>Author / evaluator instructions ↗3 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Adapter verification against the pinned dataset
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
The dataset revision and split are pinned in this job. Model version, harness, prompts, tools, judge and sampling remain unmatched, so a fresh run is not an exact reproduction claim.
Understand this test →Terminal-Bench 2.1Evaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
2 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →SWE-bench (Verified)Public harness
Maintainer instructions checked 24 October 2026. This command grades existing patches in Docker. An agent must first generate predictions with the chosen model; grading alone does not rerun model inference. Pin the agent, issue subset and Docker images. This is not a reconstruction of Vals’ bash-only harness. Inference, judging or local compute may incur costs; configure provider limits first.
Pinned dataset: princeton-nlp/SWE-bench_Verified ↗ @ c104f840cc67 · test · Benchmark author · Open
- Read the linked maintainer guide and pin the harness commit.
- Configure the exact model adapter, dataset revision, split and task limit.
- Match prompts, tools, sampling, repetitions and judge to the intended run.
- Run generation and scoring stages; retain raw outputs, task outcomes and logs.
swebench eval verified -p <model-predictions.jsonl> --run-id <unique-run-id> -j 1Author / evaluator instructions ↗1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Adapter verification against the pinned dataset
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
- Licence review for hosted or redistributed use
The dataset revision and split are pinned in this job. Model version, harness, prompts, tools, judge and sampling remain unmatched, so a fresh run is not an exact reproduction claim.
Understand this test →LiveCodeBenchPublic harness
Maintainer instructions checked 24 October 2026. Register the model adapter and prompt style before inference. Fix the release/date window, full versus lite tests, sampling and checker timeout. Reuse generated outputs; do not silently accept release_latest. Inference, judging or local compute may incur costs; configure provider limits first.
Pinned dataset: livecodebench/code_generation_lite ↗ @ 0fe84c3912ea · test · Benchmark author · Open
- Read the linked maintainer guide and pin the harness commit.
- Configure the exact model adapter, dataset revision, split and task limit.
- Match prompts, tools, sampling, repetitions and judge to the intended run.
- Run generation and scoring stages; retain raw outputs, task outcomes and logs.
python -m lcb_runner.runner.main --model <registered-model> --scenario codegeneration --evaluate --release_version <pinned-release>Author / evaluator instructions ↗1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Adapter verification against the pinned dataset
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
- Licence review for hosted or redistributed use
The dataset revision and split are pinned in this job. Model version, harness, prompts, tools, judge and sampling remain unmatched, so a fresh run is not an exact reproduction claim.
Understand this test →GPQA DiamondPublic harness
Maintainer instructions checked 24 October 2026. Use the Diamond task, not a different GPQA subset. Confirm the task ID with lm-eval ls tasks and pin the dataset and harness. The public harness is not the Vals or AA pipeline. Inference, judging or local compute may incur costs; configure provider limits first.
Pinned dataset: Idavidrein/gpqa ↗ @ 83022cefff93 · gpqa_diamond · Benchmark author · Gated
- Read the linked maintainer guide and pin the harness commit.
- Configure the exact model adapter, dataset revision, split and task limit.
- Match prompts, tools, sampling, repetitions and judge to the intended run.
- Run generation and scoring stages; retain raw outputs, task outcomes and logs.
lm-eval run --model <backend> --model_args <provider-configuration> --tasks <verified-gpqa-diamond-task> --output_path results --log_samplesAuthor / evaluator instructions ↗3 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Adapter verification against the pinned dataset
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
- Dataset access approval for the running account
The dataset revision and split are pinned in this job. Model version, harness, prompts, tools, judge and sampling remain unmatched, so a fresh run is not an exact reproduction claim.
Understand this test →MMLU ProPublic harness
Maintainer instructions checked 24 October 2026. Confirm the MMLU-Pro task ID and few-shot/prompt policy for your pinned harness. These are not the original MMLU tasks or the evaluator’s private configuration. Inference, judging or local compute may incur costs; configure provider limits first.
Pinned dataset: TIGER-Lab/MMLU-Pro ↗ @ b189ec765aa7 · test · Benchmark author · Open
- Read the linked maintainer guide and pin the harness commit.
- Configure the exact model adapter, dataset revision, split and task limit.
- Match prompts, tools, sampling, repetitions and judge to the intended run.
- Run generation and scoring stages; retain raw outputs, task outcomes and logs.
lm-eval run --model <backend> --model_args <provider-configuration> --tasks <verified-mmlu-pro-task> --output_path results --log_samplesAuthor / evaluator instructions ↗1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Adapter verification against the pinned dataset
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
The dataset revision and split are pinned in this job. Model version, harness, prompts, tools, judge and sampling remain unmatched, so a fresh run is not an exact reproduction claim.
Understand this test →MMMU ProEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
Pinned dataset: MMMU/MMMU_Pro ↗ @ 0d7426df4ccb · vision · Benchmark author · Open
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
2 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Adapter verification against the pinned dataset
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
The dataset revision and split are pinned in this job. Model version, harness, prompts, tools, judge and sampling remain unmatched, so a fresh run is not an exact reproduction claim.
Understand this test →Vibe Code Bench v1.1Evaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →Terminal-Bench Science 0.1Evaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
Pinned dataset: harborframework/terminal-bench-science ↗ @ 3fa37d3e709a · test · Benchmark author · Open
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
2 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Adapter verification against the pinned dataset
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
The dataset revision and split are pinned in this job. Model version, harness, prompts, tools, judge and sampling remain unmatched, so a fresh run is not an exact reproduction claim.
Understand this test →ProgramBenchEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
Pinned dataset: programbench/ProgramBench-Tests ↗ @ 5954a1957b5e · test · Benchmark author · Open
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Adapter verification against the pinned dataset
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
The dataset revision and split are pinned in this job. Model version, harness, prompts, tools, judge and sampling remain unmatched, so a fresh run is not an exact reproduction claim.
Understand this test →ProofBench v1.1Evaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
Pinned dataset: lm-provers/ProofBench ↗ @ c9ef8ca58c17 · train · Benchmark author · Open
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Adapter verification against the pinned dataset
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
- Licence review for hosted or redistributed use
The dataset revision and split are pinned in this job. Model version, harness, prompts, tools, judge and sampling remain unmatched, so a fresh run is not an exact reproduction claim.
Understand this test →LegalBenchEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
Pinned dataset: nguha/legalbench ↗ @ daec8237410a · test · Benchmark author · Open
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Adapter verification against the pinned dataset
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
The dataset revision and split are pinned in this job. Model version, harness, prompts, tools, judge and sampling remain unmatched, so a fresh run is not an exact reproduction claim.
Understand this test →Harvey's Legal Agent BenchmarkEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →SkillsBenchEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
Pinned dataset: benchflow/skillsbench ↗ @ be2a6ce2cb1f · test · Benchmark author · Open
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Adapter verification against the pinned dataset
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
The dataset revision and split are pinned in this job. Model version, harness, prompts, tools, judge and sampling remain unmatched, so a fresh run is not an exact reproduction claim.
Understand this test →MysteryMechanismEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →Vals IndexEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →BioMysteryBenchEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
Pinned dataset: Anthropic/BioMysteryBench-full ↗ @ b5a889c47572 · test · Benchmark author · Gated
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Adapter verification against the pinned dataset
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
- Dataset access approval for the running account
The dataset revision and split are pinned in this job. Model version, harness, prompts, tools, judge and sampling remain unmatched, so a fresh run is not an exact reproduction claim.
Understand this test →Code MigrationEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →CyberBench v1.1Evaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →EMBEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →Finance Agent (v2)Evaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →IOIEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →Legal Research BenchEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →MedCodeEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →MedScribeEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →Public Benefits Bench v1.1Evaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →SAGEEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →Tax Agent BenchEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →DeepSWE v1.1Recipe not catalogued
Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Open the benchmark’s author page.
- Confirm code and dataset availability, licence, task split and scoring.
- Record the exact harness commit, model, sampling and judge before adding a command.
2 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →FrontierCode v1.1Evaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
2 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →GDPval-AA v2.1Evaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
Pinned dataset: openai/gdpval ↗ @ 11e7900cdcac · train · Benchmark author · Open
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
2 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Adapter verification against the pinned dataset
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
- Licence review for hosted or redistributed use
The dataset revision and split are pinned in this job. Model version, harness, prompts, tools, judge and sampling remain unmatched, so a fresh run is not an exact reproduction claim.
Understand this test →AA-Briefcase v1.1Evaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
2 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →ChartographyRecipe not catalogued
Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Open the benchmark’s author page.
- Confirm code and dataset availability, licence, task split and scoring.
- Record the exact harness commit, model, sampling and judge before adding a command.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →Humanity's Last ExamPublic harness
Maintainer instructions checked 24 October 2026. Run from hle_eval. The separate run_judge_results.py stage is required to obtain a score. Pin judge model, prompts, token budget and full versus text-only split. Public HLE and HLE-Rolling are different datasets; this recipe covers public HLE only. Inference, judging or local compute may incur costs; configure provider limits first.
Pinned dataset: cais/hle ↗ @ 5a81a4c7271a · test · Benchmark author · Gated
- Read the linked maintainer guide and pin the harness commit.
- Configure the exact model adapter, dataset revision, split and task limit.
- Match prompts, tools, sampling, repetitions and judge to the intended run.
- Run generation and scoring stages; retain raw outputs, task outcomes and logs.
python run_model_predictions.py --dataset cais/hle --model <provider-model> --max_completion_tokens <budget> --num_workers 1 --max_samples <task-limit>Author / evaluator instructions ↗1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Adapter verification against the pinned dataset
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
- Dataset access approval for the running account
The dataset revision and split are pinned in this job. Model version, harness, prompts, tools, judge and sampling remain unmatched, so a fresh run is not an exact reproduction claim.
Understand this test →Artificial Analysis Intelligence IndexEvaluator access
A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.
No dataset revision is pinned for this test. The plan keeps the missing dataset, harness and settings visible.
- Read the evaluator’s methodology and access terms.
- Request the matching evaluation version and complete run configuration.
- Obtain task, prompt, judge and submission access.
- Record the evaluator’s run receipt and result artifact.
1 source-specific reference results for this model; each remains in the downloaded plan.
Configuration still required
- Immutable model or endpoint version
- Dataset revision and split
- Harness commit
- Prompt and tool policy
- Reasoning effort
- Sampling and seed
- Task limit and repetitions
- Judge and aggregation
- Provider spending limits
A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.
Understand this test →