Reproduce a result

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

2 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

2 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

2 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

2 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

2 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.

1. Read the evaluator’s methodology and access terms.
2. Request the matching evaluation version and complete run configuration.
3. Obtain task, prompt, judge and submission access.
4. Record the evaluator’s run receipt and result artifact.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.

1. Read the evaluator’s methodology and access terms.
2. Request the matching evaluation version and complete run configuration.
3. Obtain task, prompt, judge and submission access.
4. Record the evaluator’s run receipt and result artifact.

2 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.

1. Read the evaluator’s methodology and access terms.
2. Request the matching evaluation version and complete run configuration.
3. Obtain task, prompt, judge and submission access.
4. Record the evaluator’s run receipt and result artifact.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.

1. Read the evaluator’s methodology and access terms.
2. Request the matching evaluation version and complete run configuration.
3. Obtain task, prompt, judge and submission access.
4. Record the evaluator’s run receipt and result artifact.

2 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.

1. Read the evaluator’s methodology and access terms.
2. Request the matching evaluation version and complete run configuration.
3. Obtain task, prompt, judge and submission access.
4. Record the evaluator’s run receipt and result artifact.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.

1. Read the evaluator’s methodology and access terms.
2. Request the matching evaluation version and complete run configuration.
3. Obtain task, prompt, judge and submission access.
4. Record the evaluator’s run receipt and result artifact.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.

1. Read the evaluator’s methodology and access terms.
2. Request the matching evaluation version and complete run configuration.
3. Obtain task, prompt, judge and submission access.
4. Record the evaluator’s run receipt and result artifact.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Author-verified command structure, checked 24 October 2026. The dataset is pinned; the harness, provider, agent and trial count still require configuration. This template does not establish the Artificial Analysis or Moonshot run settings. GPU sandbox and inference charges are possible.

1. Install Harbor using the author’s run guide.
2. Configure a GPU-capable sandbox and its spending limit.
3. Choose the matching agent and provider/model identifier.
4. Pin the Harbor commit, task subset, reasoning and sampling settings.
5. Run the selected tasks; retain job logs, outcomes and configuration.

harbor run -d terminal-bench/terminal-bench@4.0.0 -e <gpu-sandbox> -a <agent> -m <provider/model> -k <repetitions>

4 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Maintainer instructions checked 24 October 2026. Use the Diamond task, not a different GPQA subset. Confirm the task ID with lm-eval ls tasks and pin the dataset and harness. The public harness is not the Vals or AA pipeline. Inference, judging or local compute may incur costs; configure provider limits first.

1. Read the linked maintainer guide and pin the harness commit.
2. Configure the exact model adapter, dataset revision, split and task limit.
3. Match prompts, tools, sampling, repetitions and judge to the intended run.
4. Run generation and scoring stages; retain raw outputs, task outcomes and logs.

lm-eval run --model <backend> --model_args <provider-configuration> --tasks <verified-gpqa-diamond-task> --output_path results --log_samples

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.

1. Read the evaluator’s methodology and access terms.
2. Request the matching evaluation version and complete run configuration.
3. Obtain task, prompt, judge and submission access.
4. Record the evaluator’s run receipt and result artifact.

2 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.

1. Read the evaluator’s methodology and access terms.
2. Request the matching evaluation version and complete run configuration.
3. Obtain task, prompt, judge and submission access.
4. Record the evaluator’s run receipt and result artifact.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.

1. Read the evaluator’s methodology and access terms.
2. Request the matching evaluation version and complete run configuration.
3. Obtain task, prompt, judge and submission access.
4. Record the evaluator’s run receipt and result artifact.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.

1. Read the evaluator’s methodology and access terms.
2. Request the matching evaluation version and complete run configuration.
3. Obtain task, prompt, judge and submission access.
4. Record the evaluator’s run receipt and result artifact.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.

1. Read the evaluator’s methodology and access terms.
2. Request the matching evaluation version and complete run configuration.
3. Obtain task, prompt, judge and submission access.
4. Record the evaluator’s run receipt and result artifact.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.

1. Read the evaluator’s methodology and access terms.
2. Request the matching evaluation version and complete run configuration.
3. Obtain task, prompt, judge and submission access.
4. Record the evaluator’s run receipt and result artifact.

2 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.

1. Read the evaluator’s methodology and access terms.
2. Request the matching evaluation version and complete run configuration.
3. Obtain task, prompt, judge and submission access.
4. Record the evaluator’s run receipt and result artifact.

2 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Maintainer instructions checked 24 October 2026. Run from hle_eval. The separate run_judge_results.py stage is required to obtain a score. Pin judge model, prompts, token budget and full versus text-only split. Public HLE and HLE-Rolling are different datasets; this recipe covers public HLE only. Inference, judging or local compute may incur costs; configure provider limits first.

1. Read the linked maintainer guide and pin the harness commit.
2. Configure the exact model adapter, dataset revision, split and task limit.
3. Match prompts, tools, sampling, repetitions and judge to the intended run.
4. Run generation and scoring stages; retain raw outputs, task outcomes and logs.

python run_model_predictions.py --dataset cais/hle --model <provider-model> --max_completion_tokens <budget> --num_workers 1 --max_samples <task-limit>

2 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

A complete executable recipe is not catalogued here. An evaluator result is not proof that its private tasks or judging pipeline can be run locally.

1. Read the evaluator’s methodology and access terms.
2. Request the matching evaluation version and complete run configuration.
3. Obtain task, prompt, judge and submission access.
4. Record the evaluator’s run receipt and result artifact.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Public execution instructions have not been verified in this catalogue. This status does not claim that the benchmark is closed or impossible to run.

1. Open the benchmark’s author page.
2. Confirm code and dataset availability, licence, task split and scoring.
3. Record the exact harness commit, model, sampling and judge before adding a command.

1 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

• Immutable model or endpoint version
• Dataset revision and split
• Harness commit
• Prompt and tool policy
• Reasoning effort
• Sampling and seed
• Task limit and repetitions
• Judge and aggregation
• Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.