Assistant Evaluation

Evaluation tests are a crucial part of ensuring that AI assistants provide accurate, relevant, and high-quality responses. In Ejento AI, the evaluation process involves testing the assistant's responses against benchmark queries to assess their effectiveness and quality.

Evaluation Environments

Ejento AI supports two fully independent environments for managing evaluations. Each environment provides the complete set of evaluation capabilities like create, update, delete, run, and is completely isolated from the other.

🟢 Production

Live assistant environment. Evaluations here reflect real-world performance and are used for monitoring and quality assurance.

🟡 Staging

Safe testing environment. Use Staging to design, iterate, and validate evaluations before promoting changes to Production.

🔒 Environment Isolation
Evaluations created in Staging are not visible in Production, and vice versa. This isolation is enforced automatically — no configuration is required. Use Staging freely for experimentation without any risk to Production metrics.

Key Components of Evaluation Tests

1. Queries

Queries are the input questions or prompts directed to the AI assistant. In the evaluation context, these queries serve as the basis for creating datasets. They represent the real-world questions that users might ask the assistant.

2. Datasets

Datasets are collections of queries selected for evaluation. When creating a dataset in Ejento AI, you can choose multiple queries that the assistant has previously handled. Each dataset is given a unique name and description to identify its purpose and content.

3. Evaluation Metrics

The core of the evaluation process lies in the metrics used to measure the assistant's performance. The key metrics include:

Answer Similarity: This metric assesses how closely the assistant's response matches the benchmark or correct answer. High similarity indicates accurate answers.

Answer Relevance: This measures how well the assistant's response directly addresses the user's query. It ensures that the response is pertinent to the question asked.

Faithfulness: Faithfulness checks whether the assistant's response accurately reflects the underlying source content or knowledge base. It prevents the generation of misleading or incorrect information.

Context Recall: This metric evaluates the assistant's ability to maintain context and continuity in its responses. It is crucial for multi-turn conversations where understanding the context of previous queries is necessary.

How to run evaluation tests

To run an end-to-end evaluation of an assistant, including creating datasets, executing tests, and reviewing metric results — follow the step by step tutorial: Evaluate an Assistant →

Importance of Evaluation Tests

Evaluation tests provide insights into the strengths and weaknesses of the AI assistant. By analyzing the results from various metrics, developers and users can:

Identify areas where the assistant performs well.

Detect issues where the responses may be lacking in accuracy, relevance, or context.

Make informed decisions on how to improve the assistant's performance through retraining or adjusting the underlying algorithms.

Conclusion

Evaluation tests are an essential tool in the development and maintenance of AI assistants in Ejento AI. By understanding and utilizing queries, datasets, and evaluation metrics, users can ensure that their AI assistants provide high-quality responses, improving user satisfaction and effectiveness.

Assistant Evaluation

Evaluation Environments#

🟢 Production#

🟡 Staging#

Key Components of Evaluation Tests#

1. Queries#

2. Datasets#

3. Evaluation Metrics#

How to run evaluation tests#

Importance of Evaluation Tests#

Conclusion#