Assistant Evaluation

An evaluation tests an assistant against a set of questions and scores its answers. Instead of judging an assistant from a few chats, you get numbers you can compare: before and after a change, between Staging and Production, or between two models.

Why evaluations matter

Catch problems before people do. Test changes in Staging and deploy only if the scores hold up.

Compare objectively. Run the same questions against different versions or models and compare the scores.

Find the weak spot. Separate scores for answer quality and for document retrieval show whether a bad answer comes from the model or from the documents it found.

Check safety. Ethics and safety metrics flag harmful, biased, or toxic responses.

How evaluations work

Pick datasets. A dataset is a list of test questions, each with the response you expect.

Configure the run. Name it and choose the Agent version: Staging (your unpublished changes) or Production (the live assistant).

Choose metrics. Pick what to measure from three groups (see below).

Run it. Ejento AI asks the assistant every question and scores each response. The run is listed under Evaluations with its status (Running, then Completed).

Read the results. Results shows the score per metric; Details shows the result for each question, which you can filter and export.

For the steps, see Evaluate Assistant.

Key terms

Query: A test question in a dataset. You can pick real questions from the assistant's chat logs, or type your own with Add Custom Query.

Expected response (reference): The answer you expect for a query. Metrics that check correctness compare the assistant's response with it.

Dataset: A named collection of queries with a description and optional tags. You can create one, upload one, edit it, and reuse it in many runs.

Evaluation (run): One test of one assistant version against one or more datasets with a set of metrics.

Metric groups

Group	What it measures	Examples
Generation Quality	How good the assistant's answers are: relevant to the question, consistent with the documents it found, and close to the expected response.	Faithfulness, Response Relevancy, Semantic Similarity, Answer Accuracy
Retrieval Quality	How well the assistant found the right information in its documents before answering.	Context Precision, Context Recall, Context Relevance
Ethics and Safety	Whether responses are harmful, biased, toxic, or otherwise unsafe.	Harmful, Bias, Toxic, Manipulative

For every metric and how it's calculated, see Evaluation Metrics.

Staging and Production evaluations

You choose the Agent version for each run. Evaluate Staging to test changes before you deploy them, and Production to measure what people are using now. Running the same dataset against both shows whether your changes are an improvement.

Use cases

Safe instruction changes: Before deploying new instructions, run your standard dataset against Staging and Production. Deploy only if the Staging scores are at least as good.

Corpus check: After adding documents, look at the Retrieval Quality scores. Low context recall means the assistant isn't finding the information the expected responses rely on.

Safety review: Before making an assistant public, run the Ethics and Safety metrics on questions that try to push it off topic.

Assistant Evaluation

Why evaluations matter#

How evaluations work#

Key terms#

Metric groups#