Evaluation metrics are the scores an evaluation gives an assistant's responses. Each metric looks at one quality, such as whether the answer addresses the question, whether it's consistent with the documents the assistant found, or whether it's safe. This page explains every metric you can choose when you evaluate an assistant, with formulas where they apply.How to choose metrics#
In the Metrics step of Start evaluation, metrics are grouped in three sections: Generation Quality (11 metrics), Retrieval Quality (5), and Ethics And Safety (10). Each card shows a short description, Search metrics... finds a metric by name, and Documentation opens this page.Generation Quality metrics score the answer itself. Some compare it with the expected response you wrote in the dataset (for example Semantic Similarity, Answer Accuracy, Exact Match); others need no expected response (Response Relevancy).
Retrieval Quality metrics score the documents the assistant retrieved before answering. They're most useful for assistants that answer from a corpus.
Ethics and Safety metrics check whether responses are harmful, biased, toxic, or otherwise unsafe.
1. Generation Quality Metrics#
1.1 Semantic Similarity#
Definition: Embedding-based similarity between the response and the reference answer. It evaluates only the semantic similarity between the generated answer and the ground truth.Semantic similarity compares meaning only; it doesn't check whether the facts are right. To check facts, add Factual Correctness or Answer Accuracy.
1.2 Response Relevancy#
Definition: Response Relevancy measures how well the assistant’s response directly addresses the user’s input or question. It ensures that the response is pertinent, avoids irrelevant details, and fulfills the user's informational need.Range:
0 to 1 — Higher values indicate stronger alignment between the response and the original query.A value close to 1 means the response is highly relevant to the input.
A lower value indicates that the response may include off-topic or incomplete info
Convert each sentence or response segment into a vector using an embedding model.
Compute the cosine similarity between the embeddings of the generated response and the user input.
Average the similarity scores across all segments.
Where:#
1.3 BLEU Score#
Definition: A widely used metric based on n-gram precision and brevity penalty, used to compare generated text with reference responses.0 (no match at all) to 1 (perfect match with the reference).Where:#
1.4 ROUGE Score#
Definition: Measures overlap between n-grams of the generated text and reference text. Includes precision, recall, and F1-score.ROUGE-N: Overlap of n-grams
ROUGE-L: Longest Common Subsequence
0 to 1
A score of 0 means no overlap between the generated text and the reference (poor quality), while a score of 1 means perfect overlap (ideal match)
1.5 Faithfulness#
Definition:
The Faithfulness metric measures how factually consistent a response is with the retrieved context. It ensures that the assistant does not hallucinate or fabricate information that is not grounded in the provided sources.Range:
0 to 1 — Higher scores indicate better consistency with the retrieved context.1.
Identify all the claims made in the response.
2.
For each claim, verify whether it is supported or inferable from the retrieved context.
3.
Compute the score using the formula below.
A score of 1 means all claims are backed by the retrieved context.
A lower score indicates that some claims are unsubstantiated or hallucinated.
1.6 Other generation quality metrics#
These metrics are also available when you start an evaluation:Factual Correctness: Breaks the response and the reference into atomic claims and scores precision, recall, and F1 over them.
Answer Accuracy: A lightweight LLM judge of how correct the response is compared with the reference (1 to 2 LLM calls).
Response Groundedness: A lightweight LLM judge of how well the response is grounded in the retrieved context.
Rubric Score: A 1 to 5 score of response quality against the reference, based on a rubric and an LLM judge.
Exact Match: 1.0 if the response exactly matches the reference, otherwise 0.0.
String Similarity: A deterministic string-similarity score between the response and the reference, with no LLM.
2. Retrieval Quality Metrics#
2.1 Context Recall#
Definition: Measures how much of the benchmark response (reference answer) can be generated using the retrieved data points.
2.2 Context-Entity Recall#
Definition: Evaluates how many common entities (keywords or concepts) are shared between the retrieved context and the reference response.
2.3 Context Precision#
Definition:
Measures how well the retriever ranks the relevant chunks higher than the irrelevant ones for a given query. In other words, it checks if the most useful chunks appear at the top of the retrieved list. = number of chunks retrieved (e.g., top-5 or top-10 results).
= relevance indicator at rank : if the chunk at position is relevant.
if the chunk at position is not relevant.
2.4 Noise Sensitivity#
Definition: Measures how frequently the assistant generates incorrect responses due to irrelevant or misleading context.Range: 0 (better) to 1 (worse)
2.5 Context Relevance#
Definition: A lightweight LLM judge of how relevant the retrieved context is to the question.
3. Ethics and Safety Metrics#
These are LLM-based critic metrics, evaluating the ethical quality of responses by asking specific safety-related questions and applying a majority vote mechanism.Workflow:#
2.
Make 3 independent LLM calls.
3.
Apply majority vote to determine the binary outcome (e.g., Harmful or Not Harmful).
Evaluated Categories:#
Harmful: Promotes or causes harm.
Malicious: Exploits or misleads for harmful purposes.
Bias: Reinforces stereotypes or unfair treatment.
Toxic: Uses abusive or aggressive language.
Hateful: Encourages discrimination or hate speech.
Sexual: Contains inappropriate or explicit sexual content.
Violent: Incites or glorifies violence.
Insensitive: Disrespectful to identities or situations.
Self-harm: Promotes suicidal or self-injurious behavior.
Manipulative: Deceptively influences user behavior.
Summary Table#
| Metric | Category | Measures | Score Range / Type |
|---|
| Semantic Similarity | Generation | Semantic Similarity | 0–1 |
| Response Relevancy | Generation | Relevance to Query | 0–1 |
| BLEU Score | Generation | n-gram Match + Brevity | 0–1 |
| ROUGE Score | Generation | Word Sequence Overlap | 0–1 |
| Faithfulness | Generation | Factual consistency with retrieved context | 0–1 |
| Factual Correctness | Generation | Claim-level precision, recall, and F1 | 0–1 |
| Answer Accuracy | Generation | LLM-judged correctness vs. reference | LLM judge |
| Response Groundedness | Generation | LLM-judged grounding in context | LLM judge |
| Rubric Score | Generation | Rubric-based quality | 1–5 |
| Exact Match | Generation | Exact match with reference | 0 or 1 |
| String Similarity | Generation | String similarity with reference | 0–1 |
| Context Recall | Retrieval | Data-Answer Overlap | 0–1 |
| Context-Entity Recall | Retrieval | Entity Overlap | 0–1 |
| Context Precision | Retrieval | Relevant Context Rankings | 0–1 |
| Noise Sensitivity | Retrieval | Errors Due to Noise | 0–1 (lower is better) |
| Context Relevance | Retrieval | LLM-judged relevance of context | LLM judge |
| Ethics and Safety | Safety | Binary Verdicts via LLM | Yes / No |
Use cases#
Policy assistant: Choose Answer Accuracy and Faithfulness to check that answers match the expected policy answers and stick to the documents.Search-heavy assistant: Add Context Precision and Context Recall to see whether poor answers come from retrieving the wrong documents.Public assistant: Add the Ethics and Safety metrics before opening an assistant to the whole organization. 