Evaluate Assistant

Test how well an assistant answers by running it against a set of questions and scoring its responses. An evaluation shows, with numbers, whether a change to the assistant made it better or worse.

Open the Assistants section. If you're not sure how, see Add Assistant.

How an evaluation works

You choose one or more datasets: lists of test questions, each with the answer you expect.

Ejento AI sends every question to the assistant version you pick, Staging or Production.

It scores each response with the metrics you choose, for example how relevant the answer is, or how close it is to your expected answer.

You see an average score per metric, and the result for each question.

To learn more about evaluations, see Assistant Evaluation. For what each metric measures, see Evaluation Metrics.

Use cases

Check a change before deploying: Run the same dataset against Staging and Production. If Staging scores better, deploy it.

Compare models: Change the model in Staging, run the evaluation again, and compare the scores.

Watch answer quality over time: Build a dataset from real questions in the chat logs and rerun it regularly, so you notice when quality drops.

1. Open the evaluation page

Search for your assistant, click the three dots in its Actions column, then click Evaluate.

2. Start a new evaluation

If the assistant has never been evaluated, click Get Started under Start an Evaluation. Otherwise, click Start Evaluation at the top right of the Evaluations list.

The Start Evaluation window walks you through four steps: Datasets, Configuration, Metrics, and Review. The This run summary on the left shows your choices so far.

3. Create a dataset

Click Add, then Create Dataset (or Upload Dataset to upload one). Enter a Dataset name and, optionally, a Description and Tags. Then add questions in either of two ways:

Select from Chat Logs: check real questions people have asked this assistant.

Add Custom Query: type a Question and the Response you expect, then click Save. The expected response is what the answer-quality metrics compare against.

Click Save & Select. The message Dataset created successfully appears and the dataset is selected.

4. Select the datasets

Under Available Datasets, check the datasets to test against, then click Next. Each card shows the dataset's description and number of queries; use the pencil icon to edit a dataset or the trash icon to delete it.

5. Configure the run

Enter an Evaluation name and choose the Agent version to test:

Staging: your unpublished changes. Use it to test changes before you deploy them.

Production: the live assistant people use.

The Model is the assistant's own model. Expand Advanced to see the instructions that will be tested. Click Next.

6. Choose the metrics

Select the metrics to measure. They're grouped under Generation Quality (how good the answers are), Retrieval Quality (how well the assistant found the right documents), and Ethics and Safety (whether answers are harmful or biased). Each card describes its metric; use Search metrics... to find one. Click Next.

7. Review and start the evaluation

Check your choices, use Edit to change any section, then click Start evaluation. The message Evaluation started! appears, and the run is listed under Evaluations with the status Running and its progress.

8. View the results

When the status changes to Completed (click Refresh to update it), click the eye icon (View Result) in the run's Actions column. The Results tab shows the score for each metric; the Details tab shows the result for each question, with Filters and Export Results.

Evaluate Assistant

How an evaluation works#

Use cases#

1. Open the evaluation page#

2. Start a new evaluation#

3. Create a dataset#

4. Select the datasets#

5. Configure the run#

6. Choose the metrics#

7. Review and start the evaluation#

8. View the results#