Creating an evalsets

Overview

What are evalsets? See Evals

Complete these steps after finishing the Self-Service Installation Guide.

Before starting, make sure you have indexed all the documents:

Run the num_docs command to get document and chunk statistics.
Make sure the number of documents indexed aligns with your expectations and that the corresponding chunks were processed.

Create an Eval

Find your client's directory in serenity/clients/<your-client-name>
```
cd serenity/clients/<your-client-name>
```
Create eval folder if not exists
```
mkdir -p eval
```
Create a new .yaml eval file. You can use whatever name you prefer. By default, the default.yaml is run by eval command.
```
touch eval/default.yaml
```
Edit the eval file. See Eval Configuration for more details.

Here is an example of a simple eval file:

questions:
  tenant1:
    - question: What is the Snapweaver?
      instructions: the answer should mention that Snapweaver is a software product.
    - question: How to install Snapweaver?
      target_url: https://snapweaver.staging.serenitygpt.com/installation.html
  tenant2:
    - question: What is the last version of product?
      instructions: the answer should ask user to clarify the question.

Run and Analyze Evals

Run the eval command with --file <eval-file-name> argument to run the eval.
Examine the evaluation output following Interpreting Eval Results.

NOTE: Eval results are not deterministic! So, the results may vary slightly between runs even if no changes were made