# Evaluating Your CALM Assistant

This page covers how to evaluate your CALM assistant comprehensively using end-to-end (E2E) tests. You’ll learn why E2E tests are an essential component of your testing and quality strategy, the difference between qualitative and quantitative assessments, and best practices for writing and running your tests.

You can do 3 types of evaluation in Rasa:

- **Qualitative Evaluation with Inspector**  
  Tools like the Inspector allow you to explore real or test conversations step by step, seeing exactly how the LLM and flows behave.

- **Quantitative Evaluation with E2E Tests**  
  End-to-end tests automate entire conversation scenarios, ensuring that your assistant handles them exactly as you expect. E2E tests are best suited for scripted conversations and are an important guardrail to prevent regressions—especially as you iterate on your flows, patterns, or LLM prompts.

- **Simulation & Evaluation**  
  LLM-simulated conversations let you test whether your agent achieves high-level goals without scripting every turn. Use this in your local build loop alongside E2E tests in CI.

## What Is E2E Testing?
End-to-End testing in Rasa checks your assistant as a _whole_ system, from user message to final bot response or action. This includes:

- **Assertions**: Validate that certain events happen in the conversation as expected (e.g., a slot is set, a flow is started, a specific action is triggered, a response is grounded).
- **Custom Actions**: Run or stub out actions to test side effects (e.g., calling an API or setting multiple slots).
- **Generative Responses**: Use specialized assertions (like “generative_response_is_relevant”) to confirm that generative responses from the Response Rephraser or Enterprise Search are sufficiently on-topic or factually accurate.
- **Test Coverage**: Understand which flows or commands have been tested.

In short, E2E tests act like conversation “blueprints” that must pass unchanged, ensuring your assistant consistently handles the end-to-end user journey.

## How to Write E2E Tests
E2E test cases live in YAML files in your project’s `tests` directory. These can be subdivided for better organization (e.g., `tests/e2e_test_cases.yml`).

### 1. Step-Based Test Case Format:
Create a file (e.g., `tests/e2e_test_cases.yml`) where you include test cases:

Each `test_case`:  
- Has a `name` (e.g., `user books a restaurant`).  
- Contains a sequence of `steps` describing the interaction.

### 2. Writing Steps
**`user` step**  
Simulates the user’s message. May optionally include metadata to specify additional context (e.g., device info or session data).

**`bot` or `utter` step**  
Checks the expected textual response from the bot.  
- **`bot:`** matches the exact text of the bot’s last utterance.  
- **`utter:`** matches the domain-defined response name (like `utter_welcome`).

**Slot checks**  
- **`slot_was_set:`** confirms the assistant sets a slot. If you provide a value, it checks that it’s the correct value.  
- **`slot_was_not_set:`** confirms the assistant did _not_ set a particular slot or did _not_ set it to a specific value.

### 3. Using Assertions Instead of Steps
If you want more detailed checks, you can use _assertions_. For example:

```yaml
test_cases:
  - test_case: flight_booking
    steps:
      - user: "I want to book a flight"
        assertions:
          - flow_started: "flight_booking"
          - bot_uttered:
              utter_name: "utter_ask_destination"
      - user: "New York"
        assertions:
          - slot_was_set:
              - name: "destination"
                value: "New York"
```

Assertions allow you to check events like flows starting, or to confirm if a generative response is relevant/grounded, among others.

## How to Run Tests
Once you have written your E2E tests, you can run them via the CLI using the command: `rasa test e2e`

### Testing Custom Actions
By default, E2E tests call your action server in the background if your `endpoints.yml` is configured. Make sure the action server is running:

```bash

rasa run actions &
rasa test e2e
```

### Interpreting Results
- The CLI will print a summary of passing or failing test cases.
- Any mismatches appear in a diff-like format, showing the expected versus actual events.

## Test Results Analysis
When running test cases with assertions, the test runner provides an accuracy summary for each assertion type in that specific test run. The accuracy is calculated by dividing the number of successful assertions by the total of successful and failed assertions.

## To Sum it Up:
- **Use Test-Driven Development**: Begin writing E2E tests (or converting real conversations) early.
- **Automate**: Integrate E2E tests into your CI/CD pipeline to catch regressions.
- **Monitor**: Once in production, keep an eye on real user interactions for new edge cases or performance drifts, then add or update tests accordingly.