Compare runs and manage datasets
Evaluations
Run an evaluation and read the result walks one run from dataset to verdict. This page covers what sits around it: the dataset routes and the file shapes they accept, the compare route and the little it returns, and the parts of a run's config and record that decide the verdict.
Who can use it: anyone holding an API key with the Read evaluations scope to read runs, compare them and list datasets, and the Run evaluations and upload datasets scope to start a run or upload a dataset. The tenant must be licensed for evaluations.
Datasets
GET /public/v1/datasets returns datasets[], newest first, each with id, name, description, version, tags, sample_count, visibility, created_by, created_at and updated_at. There are no filters, and a dataset's rows are not readable over the public API; sample_count is what you get.
The upload, once it works, takes a multipart form with file, name (the file's stem when empty), description, version (v1), tags (a JSON list, or a comma-separated string) and visibility (workspace, shared or private; anything else becomes workspace). The engine reads the file by its suffix:
| File | Shape |
|---|---|
.json | A list of rows, or an object with a samples or data list |
.jsonl | One row per line |
| anything else | CSV with a header row |
| Column | Role |
|---|---|
question (or input) | Required. Rows without one are dropped; a file with none left is refused |
expected_answer (or expected, output) | Optional |
context | Optional; the passages the answer should be grounded in |
| any other column | Kept under fields, where the deterministic evaluators read their expected values |
A file whose bytes match a dataset the tenant already has returns that dataset with status 200; the name, tags and version you sent are ignored. Rows that contain long runs of ??? are reported in a warnings list on the response, a sign of a broken text extraction.
Compare
POST /public/v1/evaluations/compare takes run_ids, one or more, and returns the runs' summaries side by side with their scores lined up:
{"runs": [{"id": "run-1a2b3c4d", "name": "Before", "overall_score": 0.78, "scores": {"ev-llm-judge": 0.81, "ev-faith": 0.74}, "verdict": "fail"},
{"id": "run-5e6f7a8b", "name": "After", "overall_score": 0.86, "scores": {"ev-llm-judge": 0.88, "ev-faith": 0.83}, "verdict": "pass"}],
"metrics_comparison": {"ev-llm-judge": [0.81, 0.88], "ev-faith": [0.74, 0.83]}}
metrics_comparison maps each evaluator to one score per run, in the order of runs. There are no deltas, no winner and no per-sample rows; compute what you need from the two lists.
- cURL
- Python
- TypeScript
curl -X POST "https://<your tenant host>/public/v1/evaluations/compare" \
-H "Authorization: Bearer kapi_..." \
-H "Content-Type: application/json" \
-d '{"run_ids": ["<run-before>", "<run-after>"]}'
import requests
BASE = "https://<your tenant host>/public/v1"
HEADERS = {"Authorization": "Bearer kapi_..."}
cmp = requests.post(f"{BASE}/evaluations/compare", headers=HEADERS,
json={"run_ids": ["<run-before>", "<run-after>"]}).json()
before, after = cmp["runs"][0], cmp["runs"][1]
print(f"{before['name']} -> {after['name']}: overall {before['overall_score']} -> {after['overall_score']}")
for evaluator, scores in cmp["metrics_comparison"].items():
if scores[0] is not None and scores[1] is not None:
print(f" {evaluator}: {scores[1] - scores[0]:+.3f}")
const BASE = "https://<your tenant host>/public/v1";
const headers = { Authorization: "Bearer kapi_...", "Content-Type": "application/json" };
const cmp = await (await fetch(`${BASE}/evaluations/compare`, {
method: "POST", headers, body: JSON.stringify({ run_ids: ["<run-before>", "<run-after>"] }),
})).json();
const [before, after] = cmp.runs;
console.log(`${before.name} -> ${after.name}: overall ${before.overall_score} -> ${after.overall_score}`);
for (const [evaluator, scores] of Object.entries<number[]>(cmp.metrics_comparison)) {
if (scores[0] != null && scores[1] != null) console.log(` ${evaluator}: ${(scores[1] - scores[0]).toFixed(3)}`);
}
Ids the tenant does not own are skipped without an error; when none are left the route answers 404 with No matching runs found. Nothing checks that the runs share an agent or a dataset, so comparing unlike runs is on you.
What config decides
Key in config | Effect |
|---|---|
threshold | The pass mark as a percentage; 80 by default, stored as pass_threshold 0.8. An unreadable value falls back to 80 |
thresholds | A pass mark per evaluator; every one must be met as well |
judge_model | The model the judge evaluators call; the platform's configured judge when empty |
concurrency | Samples scored in parallel, 1 to 20, default 5 |
The verdict is fail when the run failed, skipped when the run measured only timings and counts (then overall_score is null), and otherwise pass when overall_score meets pass_threshold and every per-evaluator mark. evaluator_report[] accounts for each evaluator you asked for with status scored or skipped and a reason, and category quality or telemetry says whether it counted.
Limits and behaviour
agent_version_idis stored on the run and copied to annotations, but the agent's current version is what answers; a run cannot be pinned to an older version on the current build.- Runs and datasets created through the public API are recorded as created by the platform's service identity, not the key's owner. The
minefilter onGET /public/v1/evaluationstherefore does not find your runs, and aprivatedataset made this way is visible to every key in the tenant and hidden from your own login. - A monthly run allowance in the licence, when set, refuses
POST /public/v1/evaluationswith 403 and a body whosecodeiscapability_deniedandreasonlimit_reached, plusX-License-*headers. On the current build the count behind it is the platform's, not the tenant's. GET /public/v1/evaluationslists up to 200 runs (default 50) withagent_id,agent_version_id,status,mineandscheduledfilters.- Nothing caps a dataset's size or row count in the engine.
Troubleshooting
- 422 on upload: on the current build, the gateway issue above. Once fixed, the message names the problem: an empty file, text that is not UTF-8, a CSV without a header, a row with the wrong number of columns, or no usable rows.
- The upload answered 200 with a dataset you did not name: the bytes matched an existing dataset, which was returned instead.
- 404
No matching runs foundon compare: every id is from another tenant or mistyped. - 403 with
limit_reached: the monthly run allowance is spent; the headers say the limit and the count. - 401
Invalid or expired API keyon compare although listing datasets works: the key lacks the Read evaluations scope, which the gateway checks while validating the key.
The same task elsewhere