Skip to main content

Compare runs and manage datasets

Evaluations

Run an evaluation and read the result walks one run from dataset to verdict. This page covers what sits around it: the dataset routes and the file shapes they accept, the compare route and the little it returns, and the parts of a run's config and record that decide the verdict.

Who can use it: anyone holding an API key with the Read evaluations scope to read runs, compare them and list datasets, and the Run evaluations and upload datasets scope to start a run or upload a dataset. The tenant must be licensed for evaluations.

Datasets​

GET /public/v1/datasets returns datasets[], newest first, each with id, name, description, version, tags, sample_count, visibility, created_by, created_at and updated_at. There are no filters, and a dataset's rows are not readable over the public API; sample_count is what you get.

The upload, once it works, takes a multipart form with file, name (the file's stem when empty), description, version (v1), tags (a JSON list, or a comma-separated string) and visibility (workspace, shared or private; anything else becomes workspace). The engine reads the file by its suffix:

FileShape
.jsonA list of rows, or an object with a samples or data list
.jsonlOne row per line
anything elseCSV with a header row
ColumnRole
question (or input)Required. Rows without one are dropped; a file with none left is refused
expected_answer (or expected, output)Optional
contextOptional; the passages the answer should be grounded in
any other columnKept under fields, where the deterministic evaluators read their expected values

A file whose bytes match a dataset the tenant already has returns that dataset with status 200; the name, tags and version you sent are ignored. Rows that contain long runs of ??? are reported in a warnings list on the response, a sign of a broken text extraction.

Compare​

POST /public/v1/evaluations/compare takes run_ids, one or more, and returns the runs' summaries side by side with their scores lined up:

{"runs": [{"id": "run-1a2b3c4d", "name": "Before", "overall_score": 0.78, "scores": {"ev-llm-judge": 0.81, "ev-faith": 0.74}, "verdict": "fail"},
{"id": "run-5e6f7a8b", "name": "After", "overall_score": 0.86, "scores": {"ev-llm-judge": 0.88, "ev-faith": 0.83}, "verdict": "pass"}],
"metrics_comparison": {"ev-llm-judge": [0.81, 0.88], "ev-faith": [0.74, 0.83]}}

metrics_comparison maps each evaluator to one score per run, in the order of runs. There are no deltas, no winner and no per-sample rows; compute what you need from the two lists.

curl -X POST "https://<your tenant host>/public/v1/evaluations/compare" \
-H "Authorization: Bearer kapi_..." \
-H "Content-Type: application/json" \
-d '{"run_ids": ["<run-before>", "<run-after>"]}'

Ids the tenant does not own are skipped without an error; when none are left the route answers 404 with No matching runs found. Nothing checks that the runs share an agent or a dataset, so comparing unlike runs is on you.

What config decides​

Key in configEffect
thresholdThe pass mark as a percentage; 80 by default, stored as pass_threshold 0.8. An unreadable value falls back to 80
thresholdsA pass mark per evaluator; every one must be met as well
judge_modelThe model the judge evaluators call; the platform's configured judge when empty
concurrencySamples scored in parallel, 1 to 20, default 5

The verdict is fail when the run failed, skipped when the run measured only timings and counts (then overall_score is null), and otherwise pass when overall_score meets pass_threshold and every per-evaluator mark. evaluator_report[] accounts for each evaluator you asked for with status scored or skipped and a reason, and category quality or telemetry says whether it counted.

Limits and behaviour​

  • agent_version_id is stored on the run and copied to annotations, but the agent's current version is what answers; a run cannot be pinned to an older version on the current build.
  • Runs and datasets created through the public API are recorded as created by the platform's service identity, not the key's owner. The mine filter on GET /public/v1/evaluations therefore does not find your runs, and a private dataset made this way is visible to every key in the tenant and hidden from your own login.
  • A monthly run allowance in the licence, when set, refuses POST /public/v1/evaluations with 403 and a body whose code is capability_denied and reason limit_reached, plus X-License-* headers. On the current build the count behind it is the platform's, not the tenant's.
  • GET /public/v1/evaluations lists up to 200 runs (default 50) with agent_id, agent_version_id, status, mine and scheduled filters.
  • Nothing caps a dataset's size or row count in the engine.

Troubleshooting​

  • 422 on upload: on the current build, the gateway issue above. Once fixed, the message names the problem: an empty file, text that is not UTF-8, a CSV without a header, a row with the wrong number of columns, or no usable rows.
  • The upload answered 200 with a dataset you did not name: the bytes matched an existing dataset, which was returned instead.
  • 404 No matching runs found on compare: every id is from another tenant or mistyped.
  • 403 with limit_reached: the monthly run allowance is spent; the headers say the limit and the count.
  • 401 Invalid or expired API key on compare although listing datasets works: the key lacks the Read evaluations scope, which the gateway checks while validating the key.

The same task elsewhere

Describes the platform as of .

Was this page helpful?