OverviewEvaluation API and MCP
Evaluation API and MCP
Score a short, pinned sample from your own tooling. AI Runtime credentials stay on the server.
What this API does
The API exposes the hosted prefix runner. It reads the first N rows of a pinned test split and returns the exact correct count, the sample denominator, per-task outcomes and provider-reported cost. It is a quick signal for a model route or deployment, not a full benchmark score and not a published leaderboard result.
Reports are unpublished. They are visible only to the API key that created them and expire after the operator retention window. The browser runner and the API share the same execution-control ledger, so a key cannot escape the daily task, concurrency or monthly budget limits.
Benchmark metadata, targets and service status are public. Every call that spends provider budget needs a bearer key. Keys are provisioned by the operator; there is no self-service sign-up. Contact us if you need access.
BASE=https://aievals.app KEY=your-api-key curl -s "$BASE/api/v1/benchmarks" | jq . curl -s "$BASE/api/v1/targets" | jq .
HTTP API
Send a key in Authorization: Bearer <key> or X-AIEVALS-API-KEY: <key>. Responses are JSON and marked no-store.
| Method | Path | Purpose |
|---|---|---|
| GET | /api | Service status, hosted availability, configured limits and API version. |
| GET | /api/v1/benchmarks | Hosted benchmark ids, task counts, dataset pins and sample limits. |
| GET | /api/v1/benchmarks/{id} | One hosted benchmark. |
| GET | /api/v1/targets | Configured AI Runtime routes and deployments. |
| POST | /api/v1/evals | Score a bounded prefix sample. Requires an API key. |
| GET | /api/v1/evals/{run_id} | Read a report owned by the same API key. |
| GET | /api/v1/usage | Daily quota, concurrency, monthly budget and recent runs. Requires a key. |
| POST | /api/mcp | MCP Streamable HTTP JSON-RPC endpoint. |
CORS is enabled for browser clients. Unknown methods return a JSON error with the same response shape as the rest of the API.
Create a bounded evaluation
curl -s -X POST "$BASE/api/v1/evals" \
-H "Authorization: Bearer $KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: demo-2026-11-11" \
-d '{"benchmark_id":"aime-2025","limit":5}' | jq .The body accepts benchmark_id, optional limit (1–20), optional offset (zero-based start row), optional target_key and optional idempotency_key. A valid Idempotency-Key header is used when the body omits the key. Retrying the same key returns the stored report instead of spending again.
A successful create returns 201 (or 200 for a replay) with the report, current usage and the request key. Read it later at /api/v1/evals/{run_id}.
curl -s "$BASE/api/v1/usage" \ -H "Authorization: Bearer $KEY" | jq .
MCP tools
The endpoint is https://aievals.app/api/mcp. It speaks stateless Streamable HTTP with JSON responses (no server-sent events) and protocol version 2025-03-26. Discovery works without a key; create_eval, get_eval and get_usage require the same bearer key.
{
"mcpServers": {
"aievals": {
"url": "https://aievals.app/api/mcp",
"headers": { "Authorization": "Bearer your-api-key" }
}
}
}| Tool | What it returns |
|---|---|
list_benchmarks | Hosted benchmark ids, datasets and sample limits. |
get_benchmark | One benchmark and its pinned revision. |
list_targets | Configured AI Runtime routes and deployments. |
create_eval | A bounded report. Spends provider budget and requires a key. |
get_eval | A report by run id, owned by the same key. |
get_usage | Quota, budget and recent runs. |
get_status | Service version, availability and limits. |
# initialize
curl -s -X POST "$BASE/api/mcp" -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","clientInfo":{"name":"curl","version":"1"}}}'
# list tools
curl -s -X POST "$BASE/api/mcp" -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":2,"method":"tools/list"}'
# create an evaluation
curl -s -X POST "$BASE/api/mcp" -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"create_eval","arguments":{"benchmark_id":"mmlu_pro","limit":5}}}'Quotas, cost and honesty
The operator sets the hosted daily task limit, concurrent-run limit, monthly budget and maximum cost per task. The API publishes the effective values at /api and /api/v1/usage. A reservation is created before the provider call; a failed validation is not charged.
A result is an exact score only for the sampled rows. It does not estimate the full split, establish a rank or update a leaderboard. Dataset pins, licences and disclosure travel with the benchmark metadata. If the pinned data cannot support hosted use, the benchmark is not listed as hosted.
The AI Runtime credential never leaves the server. The API never returns provider keys, hidden prompts or other tenants’ reports. The usage endpoint shows only the current key’s bucket.
Error codes
| Status | Code | Meaning |
|---|---|---|
| 401 | access-denied | The bearer key is missing or invalid. |
| 409 | idempotency-conflict | The same request key was claimed but has no readable report. |
| 422 | limit, offset, target | A sample or target setting is outside the configured range. |
| 429 | quota-daily, quota-concurrent, budget-exhausted | The key has reached a spending or concurrency limit. |
| 502 | provider-error, provider-timeout | AI Runtime did not return a usable answer. |
| 503 | hosted-disabled, control-unavailable, api-not-configured | Hosted scoring or the API key configuration is unavailable. |
All failures use {"ok":false,"error":{"code":"…","message":"…"}}. JSON-RPC failures keep the standard error object and may use HTTP 401 for an unauthenticated tool call.