# Compare runs and manage datasets

What the compare route returns for two or more runs, the dataset formats and columns the engine reads, and what a run's config and record do and do not carry.

[Run an evaluation and read the result](/developers/guides/run-an-evaluation) walks one run from dataset to verdict. This page covers what sits around it: the dataset routes and the file shapes they accept, the compare route and the little it returns, and the parts of a run's `config` and record that decide the verdict.

**Who can use it:** anyone holding an API key with the **Read evaluations** scope to read runs, compare them and list datasets, and the **Run evaluations and upload datasets** scope to start a run or upload a dataset. The tenant must be licensed for evaluations.

## Datasets

`GET /public/v1/datasets` returns `datasets[]`, newest first, each with `id`, `name`, `description`, `version`, `tags`, `sample_count`, `visibility`, `created_by`, `created_at` and `updated_at`. There are no filters, and a dataset's rows are not readable over the public API; `sample_count` is what you get.

The upload, once it works, takes a multipart form with `file`, `name` (the file's stem when empty), `description`, `version` (`v1`), `tags` (a JSON list, or a comma-separated string) and `visibility` (`workspace`, `shared` or `private`; anything else becomes `workspace`). The engine reads the file by its suffix:

| File | Shape |
| --- | --- |
| `.json` | A list of rows, or an object with a `samples` or `data` list |
| `.jsonl` | One row per line |
| anything else | CSV with a header row |

| Column | Role |
| --- | --- |
| `question` (or `input`) | Required. Rows without one are dropped; a file with none left is refused |
| `expected_answer` (or `expected`, `output`) | Optional |
| `context` | Optional; the passages the answer should be grounded in |
| any other column | Kept under `fields`, where the deterministic evaluators read their expected values |

A file whose bytes match a dataset the tenant already has returns that dataset with status 200; the name, tags and version you sent are ignored. Rows that contain long runs of `???` are reported in a `warnings` list on the response, a sign of a broken text extraction.

## Compare

`POST /public/v1/evaluations/compare` takes `run_ids`, one or more, and returns the runs' summaries side by side with their scores lined up:

```json
{"runs": [{"id": "run-1a2b3c4d", "name": "Before", "overall_score": 0.78, "scores": {"ev-llm-judge": 0.81, "ev-faith": 0.74}, "verdict": "fail"},
          {"id": "run-5e6f7a8b", "name": "After", "overall_score": 0.86, "scores": {"ev-llm-judge": 0.88, "ev-faith": 0.83}, "verdict": "pass"}],
 "metrics_comparison": {"ev-llm-judge": [0.81, 0.88], "ev-faith": [0.74, 0.83]}}
```

`metrics_comparison` maps each evaluator to one score per run, in the order of `runs`. There are no deltas, no winner and no per-sample rows; compute what you need from the two lists.

**cURL**

```bash
curl -X POST "https://<your tenant host>/public/v1/evaluations/compare" \
  -H "Authorization: Bearer kapi_..." \
  -H "Content-Type: application/json" \
  -d '{"run_ids": ["<run-before>", "<run-after>"]}'
```

**Python**

```python

BASE = "https://<your tenant host>/public/v1"
HEADERS = {"Authorization": "Bearer kapi_..."}

cmp = requests.post(f"{BASE}/evaluations/compare", headers=HEADERS,
                    json={"run_ids": ["<run-before>", "<run-after>"]}).json()
before, after = cmp["runs"][0], cmp["runs"][1]
print(f"{before['name']} -> {after['name']}: overall {before['overall_score']} -> {after['overall_score']}")
for evaluator, scores in cmp["metrics_comparison"].items():
    if scores[0] is not None and scores[1] is not None:
        print(f"  {evaluator}: {scores[1] - scores[0]:+.3f}")
```

**TypeScript**

```typescript
const BASE = "https://<your tenant host>/public/v1";
const headers = { Authorization: "Bearer kapi_...", "Content-Type": "application/json" };

const cmp = await (await fetch(`${BASE}/evaluations/compare`, {
  method: "POST", headers, body: JSON.stringify({ run_ids: ["<run-before>", "<run-after>"] }),
})).json();
const [before, after] = cmp.runs;
console.log(`${before.name} -> ${after.name}: overall ${before.overall_score} -> ${after.overall_score}`);
for (const [evaluator, scores] of Object.entries<number[]>(cmp.metrics_comparison)) {
  if (scores[0] != null && scores[1] != null) console.log(`  ${evaluator}: ${(scores[1] - scores[0]).toFixed(3)}`);
}
```

Ids the tenant does not own are skipped without an error; when none are left the route answers 404 with `No matching runs found`. Nothing checks that the runs share an agent or a dataset, so comparing unlike runs is on you.

## What config decides

| Key in `config` | Effect |
| --- | --- |
| `threshold` | The pass mark as a percentage; `80` by default, stored as `pass_threshold` 0.8. An unreadable value falls back to 80 |
| `thresholds` | A pass mark per evaluator; every one must be met as well |
| `judge_model` | The model the judge evaluators call; the platform's configured judge when empty |
| `concurrency` | Samples scored in parallel, 1 to 20, default 5 |

The verdict is `fail` when the run failed, `skipped` when the run measured only timings and counts (then `overall_score` is null), and otherwise `pass` when `overall_score` meets `pass_threshold` and every per-evaluator mark. `evaluator_report[]` accounts for each evaluator you asked for with `status` `scored` or `skipped` and a `reason`, and `category` `quality` or `telemetry` says whether it counted.

## Limits and behaviour

- `agent_version_id` is stored on the run and copied to annotations, but the agent's current version is what answers; a run cannot be pinned to an older version on the current build.
- Runs and datasets created through the public API are recorded as created by the platform's service identity, not the key's owner. The `mine` filter on `GET /public/v1/evaluations` therefore does not find your runs, and a `private` dataset made this way is visible to every key in the tenant and hidden from your own login.
- A monthly run allowance in the licence, when set, refuses `POST /public/v1/evaluations` with 403 and a body whose `code` is `capability_denied` and `reason` `limit_reached`, plus `X-License-*` headers. On the current build the count behind it is the platform's, not the tenant's.
- `GET /public/v1/evaluations` lists up to 200 runs (default 50) with `agent_id`, `agent_version_id`, `status`, `mine` and `scheduled` filters.
- Nothing caps a dataset's size or row count in the engine.

## Troubleshooting

- **422 on upload**: on the current build, the gateway issue above. Once fixed, the message names the problem: an empty file, text that is not UTF-8, a CSV without a header, a row with the wrong number of columns, or no usable rows.
- **The upload answered 200 with a dataset you did not name**: the bytes matched an existing dataset, which was returned instead.
- **404 `No matching runs found` on compare**: every id is from another tenant or mistyped.
- **403 with `limit_reached`**: the monthly run allowance is spent; the headers say the limit and the count.
- **401 `Invalid or expired API key` on compare although listing datasets works**: the key lacks the Read evaluations scope, which the gateway checks while validating the key.

## Related

- [Run an evaluation and read the result](/developers/guides/run-an-evaluation)
- Evaluations
- [Evaluations in the help centre](/agents/evaluations)

Source: https://docs.katonic.ai/developers/guides/evaluations-compare-and-datasets
