# Run an evaluation and read the result

Upload a dataset, start an evaluation run for an agent, poll its progress, read the scores, verdict and per-sample detail, and compare two runs.

An evaluation scores an agent against a dataset of questions and expected answers with the evaluators you name. The run is asynchronous: start it, poll its progress, then read the run.

**Who can use it:** anyone holding an API key with the **Read evaluations** scope, which on the current build gates starting, polling, reading and comparing runs; listing datasets needs the same scope, and uploading one needs **Upload evaluation datasets**. The tenant must be licensed for evaluations.

Video: Run an evaluation (2:38), https://katonicdocsmedia.blob.core.windows.net/developers/run-an-evaluation/ac98cc125229/index.html

  
**Step 1. Create a key with Read evaluations**

On the Developer Hub's **API keys** tab, select **Create Key** and pick the **Eval & optimize** preset. Copy the key; it is shown once.

  
**Step 2. Start the run and poll it**

Take a dataset's id from `GET /public/v1/datasets`, start the run with `POST /public/v1/evaluations`, and poll `GET /public/v1/evaluations/<run-id>/progress` until it completes.

  
**Step 3. Read the result**

`GET /public/v1/evaluations/<run-id>` returns the overall score, a score per evaluator and every sample. The run shows on Studio's **Evaluations** page with the same scores.

## Prerequisites

- An API key with the Read evaluations scope. The **Eval & optimize** preset includes it. See Authentication.
- The id of the agent to evaluate. A version id can be stored on the run, but on the current build the agent's current version is what answers.
- A dataset: a CSV, JSON or JSON Lines file, as UTF-8 text, with a `question` column, and optionally `expected_answer` and `context`. On the current build the upload route cannot succeed through the public API, so upload the file in Studio's Evaluations area and take its id from `GET /public/v1/datasets`; see [Compare runs and manage datasets](/developers/guides/evaluations-compare-and-datasets).

## Start a run and read it

1. **Upload the dataset**, or pick an existing one from `GET /public/v1/datasets`. Uploading a file whose content matches an existing dataset returns that dataset rather than creating another.

   ```bash
   curl -X POST "https://<your tenant host>/public/v1/datasets" \
     -H "Authorization: Bearer kapi_..." \
     -F "file=@golden-questions.jsonl" \
     -F "name=Golden questions" \
     -F "version=v1"
   ```

   The response is the dataset record with its `id` and `sample_count`.

2. **Start the run.** Name the agent, the dataset and the evaluators by id.

   ```bash
   curl -X POST "https://<your tenant host>/public/v1/evaluations" \
     -H "Authorization: Bearer kapi_..." \
     -H "Content-Type: application/json" \
     -d '{"name": "Nightly check", "agent_id": "<agent-id>", "dataset_id": "<dataset-id>", "evaluator_ids": ["ev-llm-judge", "ev-faith"]}'
   ```

   The response is the run record; keep its `id`. Its `status` starts as `pending`.

3. **Poll the progress.** `GET /public/v1/evaluations/<run-id>/progress` returns `status` (`pending`, `running`, `completed`, `failed` or `cancelled`), `progress` as a percentage, `completed_samples`, `total_samples` and `current_scores` so far. Stop polling on `completed`, `failed` or `cancelled`.

4. **Read the run.** `GET /public/v1/evaluations/<run-id>` returns the run record with `overall_score`, `scores` per evaluator, `score_counts`, the `verdict` against `pass_threshold`, an `evaluator_report` that accounts for every evaluator you selected as scored or skipped, `error` when the run failed, `cost_usd` and `judge_call_count`, plus `samples`: one entry per dataset row with `question`, `expected_answer`, `actual_answer`, `context_retrieved`, `scores`, `verdict`, `judge_reasoning`, `latency_ms` and `tool_calls`.

   ```python
   import time, requests

   BASE = "https://<your tenant host>/public/v1"
   HEADERS = {"Authorization": "Bearer kapi_..."}

   run = requests.post(f"{BASE}/evaluations", headers=HEADERS, json={
       "agent_id": "<agent-id>", "dataset_id": "<dataset-id>",
       "evaluator_ids": ["ev-llm-judge", "ev-faith"],
   }).json()

   while True:
       p = requests.get(f"{BASE}/evaluations/{run['id']}/progress", headers=HEADERS).json()
       print(p["status"], f"{p['progress']}%", f"{p['completed_samples']}/{p['total_samples']}")
       if p["status"] in ("completed", "failed", "cancelled"):
           break
       time.sleep(10)

   result = requests.get(f"{BASE}/evaluations/{run['id']}", headers=HEADERS).json()
   print(result["overall_score"], result["verdict"], result["scores"])
   for s in result["samples"]:
       if s["verdict"] != "pass":
           print(s["question"], "->", s["scores"])
   ```

5. **Compare two runs.** `POST /public/v1/evaluations/compare` with `run_ids` returns the runs' summaries in `runs` and, in `metrics_comparison`, one score per run per evaluator, for example the run before and after a prompt change. The deltas are yours to compute; see [Compare runs and manage datasets](/developers/guides/evaluations-compare-and-datasets).

## Choosing evaluators

| Id | What it measures |
| --- | --- |
| `ev-llm-judge` | A model judges accuracy, relevance, completeness and helpfulness. The default when the list is empty. |
| `ev-faith`, `ev-relevancy`, `ev-context-prec`, `ev-context-recall` | The RAGAS metrics: faithfulness to the retrieved context, answer relevancy, context precision and context recall. `ragas` selects all four. |
| `ev-tunable-rag` | A judge scoring coverage, correctness and relevance with configurable weights. |
| `ev-trajectory` | Whether the agent's tool use was logical and effective. |
| `ev-safety` | Adversarial safety: jailbreaks, harmful content and policy violations. |
| `ev-latency`, `ev-latency-p95`, `ev-runtime`, `ev-llm-calls`, `ev-tokens`, `ev-tool-call-success` | Measurements from the run rather than scores; the timing and count evaluators do not count towards `overall_score` or the verdict. |
| `ev-numeric-accuracy`, `ev-citation-accuracy`, `ev-kg-entity`, `ev-kg-relationship`, `ev-kg-attribute`, `ev-kg-temporal`, `ev-bilingual-alias`, `ev-test-verdict` | Deterministic checks that read their expected values from the dataset row. |

The groups `ragas`, `trajectory`, `tunable_rag`, `safety` and `custom` expand to the evaluators above; `custom` is the LLM judge.

## Verify

The run appears on the **Evaluations** page in Studio with the same scores, and the `evaluator_report` lists every evaluator you asked for as scored or skipped with a reason.

## Troubleshooting

- **422 `Unknown evaluator id(s)`**: an id is not in the table above.
- **422 `Dataset not found` or `Dataset has no samples to evaluate.`**: check `dataset_id`, and that the file had rows.
- **422 on upload**: the file was empty or not UTF-8 text.
- **`status` is `failed`**: read `error` on the run. `judge_error_count` above zero means some samples went unscored.
- **401 with `AUTH_ERROR` although the key works elsewhere**: the key lacks the Read evaluations scope, which the gateway checks while validating the key. Uploading a dataset needs the **Upload evaluation datasets** scope instead, and is refused with 403 without it.

## Related

- Evaluations
- [Receive and verify webhooks](/developers/guides/receive-and-verify-webhooks)
- [Chat from your application](/developers/guides/chat-from-your-application)
- [Evaluations in Studio, in the help centre](/agents/evaluations)

Source: https://docs.katonic.ai/developers/guides/run-an-evaluation
