Run an evaluation and read the result
Evaluations
An evaluation scores an agent against a dataset of questions and expected answers with the evaluators you name. The run is asynchronous: start it, poll its progress, then read the run.
Who can use it: anyone holding an API key with the Read evaluations scope, which on the current build gates starting, polling, reading and comparing runs; listing datasets needs the same scope, and uploading one needs Upload evaluation datasets. The tenant must be licensed for evaluations.
Recorded on .
- Create a key with Read evaluations
On the Developer Hub's API keys tab, select Create Key and pick the Eval & optimize preset. Copy the key; it is shown once.
- Start the run and poll it
Take a dataset's id from
GET /public/v1/datasets, start the run withPOST /public/v1/evaluations, and pollGET /public/v1/evaluations/<run-id>/progressuntil it completes. - Read the result
GET /public/v1/evaluations/<run-id>returns the overall score, a score per evaluator and every sample. The run shows on Studio's Evaluations page with the same scores.
Prerequisites
- An API key with the Read evaluations scope. The Eval & optimize preset includes it. See Authentication.
- The id of the agent to evaluate. A version id can be stored on the run, but on the current build the agent's current version is what answers.
- A dataset: a CSV, JSON or JSON Lines file, as UTF-8 text, with a
questioncolumn, and optionallyexpected_answerandcontext. On the current build the upload route cannot succeed through the public API, so upload the file in Studio's Evaluations area and take its id fromGET /public/v1/datasets; see Compare runs and manage datasets.
Start a run and read it
-
Upload the dataset, or pick an existing one from
GET /public/v1/datasets. Uploading a file whose content matches an existing dataset returns that dataset rather than creating another.curl -X POST "https://<your tenant host>/public/v1/datasets" \-H "Authorization: Bearer kapi_..." \-F "file=@golden-questions.jsonl" \-F "name=Golden questions" \-F "version=v1"The response is the dataset record with its
idandsample_count. -
Start the run. Name the agent, the dataset and the evaluators by id.
curl -X POST "https://<your tenant host>/public/v1/evaluations" \-H "Authorization: Bearer kapi_..." \-H "Content-Type: application/json" \-d '{"name": "Nightly check", "agent_id": "<agent-id>", "dataset_id": "<dataset-id>", "evaluator_ids": ["ev-llm-judge", "ev-faith"]}'The response is the run record; keep its
id. Itsstatusstarts aspending. -
Poll the progress.
GET /public/v1/evaluations/<run-id>/progressreturnsstatus(pending,running,completed,failedorcancelled),progressas a percentage,completed_samples,total_samplesandcurrent_scoresso far. Stop polling oncompleted,failedorcancelled. -
Read the run.
GET /public/v1/evaluations/<run-id>returns the run record withoverall_score,scoresper evaluator,score_counts, theverdictagainstpass_threshold, anevaluator_reportthat accounts for every evaluator you selected as scored or skipped,errorwhen the run failed,cost_usdandjudge_call_count, plussamples: one entry per dataset row withquestion,expected_answer,actual_answer,context_retrieved,scores,verdict,judge_reasoning,latency_msandtool_calls.import time, requestsBASE = "https://<your tenant host>/public/v1"HEADERS = {"Authorization": "Bearer kapi_..."}run = requests.post(f"{BASE}/evaluations", headers=HEADERS, json={"agent_id": "<agent-id>", "dataset_id": "<dataset-id>","evaluator_ids": ["ev-llm-judge", "ev-faith"],}).json()while True:p = requests.get(f"{BASE}/evaluations/{run['id']}/progress", headers=HEADERS).json()print(p["status"], f"{p['progress']}%", f"{p['completed_samples']}/{p['total_samples']}")if p["status"] in ("completed", "failed", "cancelled"):breaktime.sleep(10)result = requests.get(f"{BASE}/evaluations/{run['id']}", headers=HEADERS).json()print(result["overall_score"], result["verdict"], result["scores"])for s in result["samples"]:if s["verdict"] != "pass":print(s["question"], "->", s["scores"]) -
Compare two runs.
POST /public/v1/evaluations/comparewithrun_idsreturns the runs' summaries inrunsand, inmetrics_comparison, one score per run per evaluator, for example the run before and after a prompt change. The deltas are yours to compute; see Compare runs and manage datasets.
Choosing evaluators
| Id | What it measures |
|---|---|
ev-llm-judge | A model judges accuracy, relevance, completeness and helpfulness. The default when the list is empty. |
ev-faith, ev-relevancy, ev-context-prec, ev-context-recall | The RAGAS metrics: faithfulness to the retrieved context, answer relevancy, context precision and context recall. ragas selects all four. |
ev-tunable-rag | A judge scoring coverage, correctness and relevance with configurable weights. |
ev-trajectory | Whether the agent's tool use was logical and effective. |
ev-safety | Adversarial safety: jailbreaks, harmful content and policy violations. |
ev-latency, ev-latency-p95, ev-runtime, ev-llm-calls, ev-tokens, ev-tool-call-success | Measurements from the run rather than scores; the timing and count evaluators do not count towards overall_score or the verdict. |
ev-numeric-accuracy, ev-citation-accuracy, ev-kg-entity, ev-kg-relationship, ev-kg-attribute, ev-kg-temporal, ev-bilingual-alias, ev-test-verdict | Deterministic checks that read their expected values from the dataset row. |
The groups ragas, trajectory, tunable_rag, safety and custom expand to the evaluators above; custom is the LLM judge.
Verify
The run appears on the Evaluations page in Studio with the same scores, and the evaluator_report lists every evaluator you asked for as scored or skipped with a reason.
Troubleshooting
- 422
Unknown evaluator id(s): an id is not in the table above. - 422
Dataset not foundorDataset has no samples to evaluate.: checkdataset_id, and that the file had rows. - 422 on upload: the file was empty or not UTF-8 text.
statusisfailed: readerroron the run.judge_error_countabove zero means some samples went unscored.- 401 with
AUTH_ERRORalthough the key works elsewhere: the key lacks the Read evaluations scope, which the gateway checks while validating the key. Uploading a dataset needs the Upload evaluation datasets scope instead, and is refused with 403 without it.