# The evaluators

What each evaluator measures, which ones need a judge model, which need gold data in the dataset, and which are measurements that never pass or fail a run.

An evaluator scores one sample. A run's score is the mean of the quality evaluators across every sample. This page is the catalogue: what each one asks, and what it needs before it can answer.

**Who can use it:** the **Builder** role or higher, in a tenant licensed for evaluations.

## Three kinds

| Kind | How it scores | What it needs |
| --- | --- | --- |
| **Model-graded** | A judge model reads the answer and scores it | A judge model, and usually the expected answer |
| **Deterministic** | Arithmetic against gold data in the dataset | A gold column in the dataset's own fields |
| **Measurement** | Reads what the run did | Nothing; it observes the agent's own trace |

Measurement evaluators are excluded from the overall score and from pass or fail. A run that selects only measurements reports no overall score and no verdict, and says so rather than inventing a percentage.

## Model-graded

| Evaluator | What it asks |
| --- | --- |
| **LLM Judge** | Is the answer accurate, relevant, complete and helpful? |
| **Faithfulness** | Is every claim in the answer supported by the context the agent actually retrieved? |
| **Answer Relevancy** | Does the answer address the question? |
| **Context Precision** | How much of what was retrieved was relevant? |
| **Context Recall** | How much of the expected answer was covered by what was retrieved? |
| **Tunable RAG Judge** | Coverage, correctness and relevance, weighted as you configure |
| **Trajectory Eval** | Was the agent's use of its tools logical and effective? |
| **Safety / Red Team** | Jailbreaks, harmful content, policy violations |

**RAGAS** in the evaluation form is not one evaluator: it selects Faithfulness, Answer Relevancy, Context Precision and Context Recall together.

Faithfulness judges against the context the agent retrieved during the run, not the context column in your dataset. Where the agent made no retrieval call, it falls back to the dataset's context. That is what makes it the evaluator for an agent that answers from knowledge, and a poor choice for one that does not.

## Deterministic

These need gold data, which arrives in a dataset's own columns. Any column you do not map to question, expected answer, context or category is carried through to the row, and that is where these evaluators read from.

| Evaluator | What it checks | The gold it needs |
| --- | --- | --- |
| **Numeric Field Accuracy** | Numbers in the answer against expected values, exactly or within a tolerance | A numeric field |
| **Citation Accuracy** | That the expected document, period and field all appear in the answer | A citation field |
| **KG Entity Correctness**, **KG Relationship Correctness**, **KG Attribute Accuracy** and **KG Temporal Correctness** | Four measures of an answer against a gold knowledge graph, scored as F1 | A gold graph |
| **Bilingual Alias Correctness** | That the right accepted alias was used, in either language, and no confusable one | A bilingual field |
| **Tool-Call Success** | The share of tool calls that did not error | Nothing; a run with no tool calls scores 1 |
| **Test Verdict** | A coding task's own pass or fail marker | Nothing; a sample with no marker is left unscored |

A deterministic evaluator never fails the run over malformed gold. It leaves the sample unscored and says why, exactly as a judge that could not reach a verdict does.

## Measurement

**Avg LLM Latency**, **Latency p95**, **Avg Workflow Runtime**, **Avg LLM Calls** and **Avg Tokens per LLM Call** report what the run cost in time and calls. They carry a unit rather than a percentage, and a sample count rather than a pass and fail split.

## What "not scored" means

A run's **By evaluator** tab lists the evaluators you selected that produced no score, each with its reason: a dataset with no gold column for a deterministic evaluator, a judge that could not be reached, a sample the agent failed before scoring began. An evaluator that scored nothing is never shown as zero.

Where more than a quarter of judge calls fail, the run is marked degraded and says how many dispatches went unscored. It still reports a score, and the score covers only what was scored.

## Related

- [Run an evaluation](/agents/evaluations-run)
- [Read a run's results](/agents/evaluations-results)
- [Evaluation datasets](/agents/evaluations-datasets)

Source: https://docs.katonic.ai/agents/evaluations-evaluators
