AIEvalsReproduce

OverviewReproduce a result

Reproduce a result

Create a local evaluation plan with reference results, recipes, and required settings.

1. Choose the evaluation scope

1tests in this plan
1verified public harness
0need access or recipe work

The plan preserves every selected test, including blocked ones. Public code alone is not enough to reproduce an evaluator’s score.

2. Inspect recipes and missing settings

Humanity's Last ExamPublic harness

Maintainer instructions checked 24 October 2026. Run from hle_eval. The separate run_judge_results.py stage is required to obtain a score. Pin judge model, prompts, token budget and full versus text-only split. Public HLE and HLE-Rolling are different datasets; this recipe covers public HLE only. Inference, judging or local compute may incur costs; configure provider limits first.

  1. Read the linked maintainer guide and pin the harness commit.
  2. Configure the exact model adapter, dataset revision, split and task limit.
  3. Match prompts, tools, sampling, repetitions and judge to the intended run.
  4. Run generation and scoring stages; retain raw outputs, task outcomes and logs.
python run_model_predictions.py --dataset cais/hle --model <provider-model> --max_completion_tokens <budget> --num_workers 1 --max_samples <task-limit>
Author / evaluator instructions ↗

2 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

  • Immutable model or endpoint version
  • Dataset revision and split
  • Harness commit
  • Prompt and tool policy
  • Reasoning effort
  • Sampling and seed
  • Task limit and repetitions
  • Judge and aggregation
  • Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Understand this test →