OverviewEvaluation API and MCP

Evaluation API and MCP

Score a short, pinned sample from your own tooling. AI Runtime credentials stay on the server.

What this API does

The API exposes the hosted prefix runner. It reads the first N rows of a pinned test split and returns the exact correct count, the sample denominator, per-task outcomes and provider-reported cost. It is a quick signal for a model route or deployment, not a full benchmark score and not a published leaderboard result.

Reports are unpublished. They are visible only to the API key that created them and expire after the operator retention window. The browser runner and the API share the same execution-control ledger, so a key cannot escape the daily task, concurrency or monthly budget limits.

Benchmark metadata, targets and service status are public. Every call that spends provider budget needs a bearer key. Keys are provisioned by the operator; there is no self-service sign-up. Contact us if you need access.

BASE=https://aievals.app
KEY=your-api-key

curl -s "$BASE/api/v1/benchmarks" | jq .
curl -s "$BASE/api/v1/targets" | jq .

HTTP API

Send a key in Authorization: Bearer <key> or X-AIEVALS-API-KEY: <key>. Responses are JSON and marked no-store.

MethodPathPurpose
GET/apiService status, hosted availability, configured limits and API version.
GET/api/v1/benchmarksHosted benchmark ids, task counts, dataset pins and sample limits.
GET/api/v1/benchmarks/{id}One hosted benchmark.
GET/api/v1/targetsConfigured AI Runtime routes and deployments.
POST/api/v1/evalsScore a bounded prefix sample. Requires an API key.
GET/api/v1/evals/{run_id}Read a report owned by the same API key.
GET/api/v1/usageDaily quota, concurrency, monthly budget and recent runs. Requires a key.
POST/api/mcpMCP Streamable HTTP JSON-RPC endpoint.

CORS is enabled for browser clients. Unknown methods return a JSON error with the same response shape as the rest of the API.

Create a bounded evaluation

curl -s -X POST "$BASE/api/v1/evals" \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: demo-2026-11-11" \
  -d '{"benchmark_id":"aime-2025","limit":5}' | jq .

The body accepts benchmark_id, optional limit (1–20), optional offset (zero-based start row), optional target_key and optional idempotency_key. A valid Idempotency-Key header is used when the body omits the key. Retrying the same key returns the stored report instead of spending again.

A successful create returns 201 (or 200 for a replay) with the report, current usage and the request key. Read it later at /api/v1/evals/{run_id}.

curl -s "$BASE/api/v1/usage" \
  -H "Authorization: Bearer $KEY" | jq .

MCP tools

The endpoint is https://aievals.app/api/mcp. It speaks stateless Streamable HTTP with JSON responses (no server-sent events) and protocol version 2025-03-26. Discovery works without a key; create_eval, get_eval and get_usage require the same bearer key.

{
  "mcpServers": {
    "aievals": {
      "url": "https://aievals.app/api/mcp",
      "headers": { "Authorization": "Bearer your-api-key" }
    }
  }
}
ToolWhat it returns
list_benchmarksHosted benchmark ids, datasets and sample limits.
get_benchmarkOne benchmark and its pinned revision.
list_targetsConfigured AI Runtime routes and deployments.
create_evalA bounded report. Spends provider budget and requires a key.
get_evalA report by run id, owned by the same key.
get_usageQuota, budget and recent runs.
get_statusService version, availability and limits.
# initialize
curl -s -X POST "$BASE/api/mcp" -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","clientInfo":{"name":"curl","version":"1"}}}'

# list tools
curl -s -X POST "$BASE/api/mcp" -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/list"}'

# create an evaluation
curl -s -X POST "$BASE/api/mcp" -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"create_eval","arguments":{"benchmark_id":"mmlu_pro","limit":5}}}'

Quotas, cost and honesty

The operator sets the hosted daily task limit, concurrent-run limit, monthly budget and maximum cost per task. The API publishes the effective values at /api and /api/v1/usage. A reservation is created before the provider call; a failed validation is not charged.

A result is an exact score only for the sampled rows. It does not estimate the full split, establish a rank or update a leaderboard. Dataset pins, licences and disclosure travel with the benchmark metadata. If the pinned data cannot support hosted use, the benchmark is not listed as hosted.

The AI Runtime credential never leaves the server. The API never returns provider keys, hidden prompts or other tenants’ reports. The usage endpoint shows only the current key’s bucket.

Error codes

StatusCodeMeaning
401access-deniedThe bearer key is missing or invalid.
409idempotency-conflictThe same request key was claimed but has no readable report.
422limit, offset, targetA sample or target setting is outside the configured range.
429quota-daily, quota-concurrent, budget-exhaustedThe key has reached a spending or concurrency limit.
502provider-error, provider-timeoutAI Runtime did not return a usable answer.
503hosted-disabled, control-unavailable, api-not-configuredHosted scoring or the API key configuration is unavailable.

All failures use {"ok":false,"error":{"code":"…","message":"…"}}. JSON-RPC failures keep the standard error object and may use HTTP 401 for an unauthenticated tool call.