Evaluating Your CALM Assistant

This page covers how to evaluate your CALM assistant comprehensively using end-to-end (E2E) tests. You’ll learn why E2E tests are an essential component of your testing and quality strategy, the difference between qualitative and quantitative assessments, and best practices for writing and running your tests.

You can do 3 types of evaluation in Rasa:

• Qualitative Evaluation with Inspector
  Tools like the Inspector allow you to explore real or test conversations step by step, seeing exactly how the LLM and flows behave.

• Quantitative Evaluation with E2E Tests
  End-to-end tests automate entire conversation scenarios, ensuring that your assistant handles them exactly as you expect. E2E tests are best suited for scripted conversations and are an important guardrail to prevent regressions—especially as you iterate on your flows, patterns, or LLM prompts.

• Simulation & Evaluation
  LLM-simulated conversations let you test whether your agent achieves high-level goals without scripting every turn. Use this in your local build loop alongside E2E tests in CI.

What Is E2E Testing?

End-to-End testing in Rasa checks your assistant as a whole system, from user message to final bot response or action. This includes:

• Assertions: Validate that certain events happen in the conversation as expected (e.g., a slot is set, a flow is started, a specific action is triggered, a response is grounded).
• Custom Actions: Run or stub out actions to test side effects (e.g., calling an API or setting multiple slots).
• Generative Responses: Use specialized assertions (like “generative_response_is_relevant”) to confirm that generative responses from the Response Rephraser or Enterprise Search are sufficiently on-topic or factually accurate.
• Test Coverage: Understand which flows or commands have been tested.

In short, E2E tests act like conversation “blueprints” that must pass unchanged, ensuring your assistant consistently handles the end-to-end user journey.

How to Write E2E Tests

E2E test cases live in YAML files in your project’s tests directory. These can be subdivided for better organization (e.g., tests/e2e_test_cases.yml).

1. Step-Based Test Case Format:

Create a file (e.g., tests/e2e_test_cases.yml) where you include test cases:

Each test_case:

• Has a name (e.g., user books a restaurant).
• Contains a sequence of steps describing the interaction.

2. Writing Steps

user step
Simulates the user’s message. May optionally include metadata to specify additional context (e.g., device info or session data).

bot or utter step
Checks the expected textual response from the bot.

• bot: matches the exact text of the bot’s last utterance.
• utter: matches the domain-defined response name (like utter_welcome).

Slot checks

• slot_was_set: confirms the assistant sets a slot. If you provide a value, it checks that it’s the correct value.
• slot_was_not_set: confirms the assistant did not set a particular slot or did not set it to a specific value.

3. Using Assertions Instead of Steps

If you want more detailed checks, you can use assertions. For example:

test_cases:
  - test_case: flight_booking
    steps:
      - user: "I want to book a flight"
        assertions:
          - flow_started: "flight_booking"
          - bot_uttered:
              utter_name: "utter_ask_destination"
      - user: "New York"
        assertions:
          - slot_was_set:
              - name: "destination"
                value: "New York"

Assertions allow you to check events like flows starting, or to confirm if a generative response is relevant/grounded, among others.

How to Run Tests

Once you have written your E2E tests, you can run them via the CLI using the command: rasa test e2e

Testing Custom Actions

By default, E2E tests call your action server in the background if your endpoints.yml is configured. Make sure the action server is running:

rasa run actions &
rasa test e2e

Interpreting Results

• The CLI will print a summary of passing or failing test cases.
• Any mismatches appear in a diff-like format, showing the expected versus actual events.

Test Results Analysis

When running test cases with assertions, the test runner provides an accuracy summary for each assertion type in that specific test run. The accuracy is calculated by dividing the number of successful assertions by the total of successful and failed assertions.

To Sum it Up:

• Use Test-Driven Development: Begin writing E2E tests (or converting real conversations) early.
• Automate: Integrate E2E tests into your CI/CD pipeline to catch regressions.
• Monitor: Once in production, keep an eye on real user interactions for new edge cases or performance drifts, then add or update tests accordingly.