Skip to main content

Run an evaluation and read the result

Evaluations

An evaluation scores an agent against a dataset of questions and expected answers with the evaluators you name. The run is asynchronous: start it, poll its progress, then read the run.

Who can use it: anyone holding an API key with the Read evaluations scope, which on the current build gates starting, polling, reading and comparing runs; listing datasets needs the same scope, and uploading one needs Upload evaluation datasets. The tenant must be licensed for evaluations.

VideoRun an evaluation2:38

Recorded on .

  1. Create a key with Read evaluations

    On the Developer Hub's API keys tab, select Create Key and pick the Eval & optimize preset. Copy the key; it is shown once.

  2. Start the run and poll it

    Take a dataset's id from GET /public/v1/datasets, start the run with POST /public/v1/evaluations, and poll GET /public/v1/evaluations/<run-id>/progress until it completes.

  3. Read the result

    GET /public/v1/evaluations/<run-id> returns the overall score, a score per evaluator and every sample. The run shows on Studio's Evaluations page with the same scores.

Prerequisites​

  • An API key with the Read evaluations scope. The Eval & optimize preset includes it. See Authentication.
  • The id of the agent to evaluate. A version id can be stored on the run, but on the current build the agent's current version is what answers.
  • A dataset: a CSV, JSON or JSON Lines file, as UTF-8 text, with a question column, and optionally expected_answer and context. On the current build the upload route cannot succeed through the public API, so upload the file in Studio's Evaluations area and take its id from GET /public/v1/datasets; see Compare runs and manage datasets.

Start a run and read it​

  1. Upload the dataset, or pick an existing one from GET /public/v1/datasets. Uploading a file whose content matches an existing dataset returns that dataset rather than creating another.

    curl -X POST "https://<your tenant host>/public/v1/datasets" \
    -H "Authorization: Bearer kapi_..." \
    -F "file=@golden-questions.jsonl" \
    -F "name=Golden questions" \
    -F "version=v1"

    The response is the dataset record with its id and sample_count.

  2. Start the run. Name the agent, the dataset and the evaluators by id.

    curl -X POST "https://<your tenant host>/public/v1/evaluations" \
    -H "Authorization: Bearer kapi_..." \
    -H "Content-Type: application/json" \
    -d '{"name": "Nightly check", "agent_id": "<agent-id>", "dataset_id": "<dataset-id>", "evaluator_ids": ["ev-llm-judge", "ev-faith"]}'

    The response is the run record; keep its id. Its status starts as pending.

  3. Poll the progress. GET /public/v1/evaluations/<run-id>/progress returns status (pending, running, completed, failed or cancelled), progress as a percentage, completed_samples, total_samples and current_scores so far. Stop polling on completed, failed or cancelled.

  4. Read the run. GET /public/v1/evaluations/<run-id> returns the run record with overall_score, scores per evaluator, score_counts, the verdict against pass_threshold, an evaluator_report that accounts for every evaluator you selected as scored or skipped, error when the run failed, cost_usd and judge_call_count, plus samples: one entry per dataset row with question, expected_answer, actual_answer, context_retrieved, scores, verdict, judge_reasoning, latency_ms and tool_calls.

    import time, requests

    BASE = "https://<your tenant host>/public/v1"
    HEADERS = {"Authorization": "Bearer kapi_..."}

    run = requests.post(f"{BASE}/evaluations", headers=HEADERS, json={
    "agent_id": "<agent-id>", "dataset_id": "<dataset-id>",
    "evaluator_ids": ["ev-llm-judge", "ev-faith"],
    }).json()

    while True:
    p = requests.get(f"{BASE}/evaluations/{run['id']}/progress", headers=HEADERS).json()
    print(p["status"], f"{p['progress']}%", f"{p['completed_samples']}/{p['total_samples']}")
    if p["status"] in ("completed", "failed", "cancelled"):
    break
    time.sleep(10)

    result = requests.get(f"{BASE}/evaluations/{run['id']}", headers=HEADERS).json()
    print(result["overall_score"], result["verdict"], result["scores"])
    for s in result["samples"]:
    if s["verdict"] != "pass":
    print(s["question"], "->", s["scores"])
  5. Compare two runs. POST /public/v1/evaluations/compare with run_ids returns the runs' summaries in runs and, in metrics_comparison, one score per run per evaluator, for example the run before and after a prompt change. The deltas are yours to compute; see Compare runs and manage datasets.

Choosing evaluators​

IdWhat it measures
ev-llm-judgeA model judges accuracy, relevance, completeness and helpfulness. The default when the list is empty.
ev-faith, ev-relevancy, ev-context-prec, ev-context-recallThe RAGAS metrics: faithfulness to the retrieved context, answer relevancy, context precision and context recall. ragas selects all four.
ev-tunable-ragA judge scoring coverage, correctness and relevance with configurable weights.
ev-trajectoryWhether the agent's tool use was logical and effective.
ev-safetyAdversarial safety: jailbreaks, harmful content and policy violations.
ev-latency, ev-latency-p95, ev-runtime, ev-llm-calls, ev-tokens, ev-tool-call-successMeasurements from the run rather than scores; the timing and count evaluators do not count towards overall_score or the verdict.
ev-numeric-accuracy, ev-citation-accuracy, ev-kg-entity, ev-kg-relationship, ev-kg-attribute, ev-kg-temporal, ev-bilingual-alias, ev-test-verdictDeterministic checks that read their expected values from the dataset row.

The groups ragas, trajectory, tunable_rag, safety and custom expand to the evaluators above; custom is the LLM judge.

Verify​

The run appears on the Evaluations page in Studio with the same scores, and the evaluator_report lists every evaluator you asked for as scored or skipped with a reason.

Troubleshooting​

  • 422 Unknown evaluator id(s): an id is not in the table above.
  • 422 Dataset not found or Dataset has no samples to evaluate.: check dataset_id, and that the file had rows.
  • 422 on upload: the file was empty or not UTF-8 text.
  • status is failed: read error on the run. judge_error_count above zero means some samples went unscored.
  • 401 with AUTH_ERROR although the key works elsewhere: the key lacks the Read evaluations scope, which the gateway checks while validating the key. Uploading a dataset needs the Upload evaluation datasets scope instead, and is refused with 403 without it.

Describes the platform as of .

Was this page helpful?