AIEvalsReproduce

OverviewReproduce a result

Reproduce a result

Create a local evaluation plan with reference results, recipes, and required settings.

1. Choose the evaluation scope

1tests in this plan
1verified public harness
0need access or recipe work

The plan preserves every selected test, including blocked ones. Public code alone is not enough to reproduce an evaluator’s score.

2. Inspect recipes and missing settings

Terminal-Bench 4.0Public harness

Author-verified command structure, checked 24 October 2026. The dataset is pinned; the harness, provider, agent and trial count still require configuration. This template does not establish the Artificial Analysis or Moonshot run settings. GPU sandbox and inference charges are possible.

  1. Install Harbor using the author’s run guide.
  2. Configure a GPU-capable sandbox and its spending limit.
  3. Choose the matching agent and provider/model identifier.
  4. Pin the Harbor commit, task subset, reasoning and sampling settings.
  5. Run the selected tasks; retain job logs, outcomes and configuration.
harbor run -d terminal-bench/terminal-bench@4.0.0 -e <gpu-sandbox> -a <agent> -m <provider/model> -k <repetitions>
Author / evaluator instructions ↗

4 source-specific reference results for this model; each remains in the downloaded plan.

Configuration still required

  • Immutable model or endpoint version
  • Dataset revision and split
  • Harness commit
  • Prompt and tool policy
  • Reasoning effort
  • Sampling and seed
  • Task limit and repetitions
  • Judge and aggregation
  • Provider spending limits

A fresh evaluation cannot reproduce the published score exactly until its model version, data, harness, prompts, tools, judge and sampling settings are matched. Unknown source settings remain unknown.

Understand this test →