Skip to main content

The evaluators

Evaluations

An evaluator scores one sample. A run's score is the mean of the quality evaluators across every sample. This page is the catalogue: what each one asks, and what it needs before it can answer.

Who can use it: the Builder role or higher, in a tenant licensed for evaluations.

Three kinds​

KindHow it scoresWhat it needs
Model-gradedA judge model reads the answer and scores itA judge model, and usually the expected answer
DeterministicArithmetic against gold data in the datasetA gold column in the dataset's own fields
MeasurementReads what the run didNothing; it observes the agent's own trace

Measurement evaluators are excluded from the overall score and from pass or fail. A run that selects only measurements reports no overall score and no verdict, and says so rather than inventing a percentage.

Model-graded​

EvaluatorWhat it asks
LLM JudgeIs the answer accurate, relevant, complete and helpful?
FaithfulnessIs every claim in the answer supported by the context the agent actually retrieved?
Answer RelevancyDoes the answer address the question?
Context PrecisionHow much of what was retrieved was relevant?
Context RecallHow much of the expected answer was covered by what was retrieved?
Tunable RAG JudgeCoverage, correctness and relevance, weighted as you configure
Trajectory EvalWas the agent's use of its tools logical and effective?
Safety / Red TeamJailbreaks, harmful content, policy violations

RAGAS in the evaluation form is not one evaluator: it selects Faithfulness, Answer Relevancy, Context Precision and Context Recall together.

Faithfulness judges against the context the agent retrieved during the run, not the context column in your dataset. Where the agent made no retrieval call, it falls back to the dataset's context. That is what makes it the evaluator for an agent that answers from knowledge, and a poor choice for one that does not.

Deterministic​

These need gold data, which arrives in a dataset's own columns. Any column you do not map to question, expected answer, context or category is carried through to the row, and that is where these evaluators read from.

EvaluatorWhat it checksThe gold it needs
Numeric Field AccuracyNumbers in the answer against expected values, exactly or within a toleranceA numeric field
Citation AccuracyThat the expected document, period and field all appear in the answerA citation field
KG Entity Correctness, KG Relationship Correctness, KG Attribute Accuracy and KG Temporal CorrectnessFour measures of an answer against a gold knowledge graph, scored as F1A gold graph
Bilingual Alias CorrectnessThat the right accepted alias was used, in either language, and no confusable oneA bilingual field
Tool-Call SuccessThe share of tool calls that did not errorNothing; a run with no tool calls scores 1
Test VerdictA coding task's own pass or fail markerNothing; a sample with no marker is left unscored

A deterministic evaluator never fails the run over malformed gold. It leaves the sample unscored and says why, exactly as a judge that could not reach a verdict does.

Measurement​

Avg LLM Latency, Latency p95, Avg Workflow Runtime, Avg LLM Calls and Avg Tokens per LLM Call report what the run cost in time and calls. They carry a unit rather than a percentage, and a sample count rather than a pass and fail split.

What "not scored" means​

A run's By evaluator tab lists the evaluators you selected that produced no score, each with its reason: a dataset with no gold column for a deterministic evaluator, a judge that could not be reached, a sample the agent failed before scoring began. An evaluator that scored nothing is never shown as zero.

Where more than a quarter of judge calls fail, the run is marked degraded and says how many dispatches went unscored. It still reports a score, and the score covers only what was scored.

Describes the platform as of .

Was this page helpful?