Test cases reference | Rasa Documentation

Why testing your assistant?

For more information E2E testing, see the E2E testing product documentation.

To write test cases, you need to create a YAML file inside the tests directory of your project. The name of the file should be e2e_test_cases.yml. You can also create a subdirectory inside the tests directory and place your test case YAML files there. These files will be automatically discovered and run by Rasa, however, you need to provide the path to the subdirectory as a positional argument to the rasa test e2e command.

Each input file must contain the test_cases key. The value of this key is a list of test cases. Each test case must include a name given to the test_case key and a list of test steps given to the steps key. A step can be either one of the following:

user: refers to a user's message.
bot: denotes a textual response generated by the bot.
utter: refers to a bot response name as defined in the domain file.
slot_was_set: indicates the successful setting of a slot, and depending on how it's defined:
- if a slot name is provided, e.g. my_slot: checks that a slot with the given name is set.
- if a key-value pair is provided, e.g. my_slot: value: checks that a slot with the given name is set with the expected value.
slot_was_not_set: specifies the failure to set a slot, and depending on the definition:
- if a slot name is provided, e.g. my_slot: checks that a slot with the given name is not set.
- if a key-value pair is provided, e.g. my_slot: value: checks that a slot with the given name is not set or that its value differs from the expected value.

The following example illustrates how user, bot, utter, and slot_was_set steps can be used in a test case:

tests/e2e_test_cases.yml

test_cases:
  - test_case: user books a restaurant # name of the test case must be provided and be unique
    steps:
      - user: I want to book a table for 4 at Xaigon for 8pm tonight
      - slot_was_set:
          - book_restaurant_name_of_restaurant: Xaigon
          - book_restaurant_number_of_people: "4"
          - book_restaurant_date: tonight
          - book_restaurant_time: 8pm
      - utter: utter_restaurant_available
      - utter: utter_ask_book_restaurant_reservation_name
      - user: Emil
      - slot_was_set:
          - book_restaurant_reservation_name: Emil
      - utter: utter_ask_book_restaurant_confirmation
      - user: yes
      - bot: Thank you for booking the table at Xaigon for 4 people at 8pm tonight.

In each of the steps after the user step, you can specify multiple expected events. Any additional events that are found are ignored and do not cause the test to fail.

Slots

Slots can be specified as a list of either string values (representing the slot name) or of slot name and slot value pairs. If the slot is specified as a key-value pair, the values are also compared. If the slot step contains only the slot name, it is asserted that the slot was either set (when using slot_was_set step) or not (when using slot_was_not_set step).

tests/e2e_test_cases.yml

- slot_was_set:
    - book_restaurant_name_of_restaurant: Xaigon
    - book_restaurant_number_of_people: "4"
    - book_restaurant_time
- slot_was_not_set:
    - book_restaurant_reservation_name
    - book_restaurant_reservation_date: "2023-11-11"

It is not required to specify all the slots events in the slot_was_set or slot_was_not_set steps, you can indicate only a subset of slots that you want to check. You can think of slot_was_not_set as an inverse of slot_was_set and when specifying a slot_was_not_set step it looks for the absence of the SlotSet event in the tracker store but only for that particular user step, not globally.

Fixtures for Pre-Filled Slots

Using fixtures is an optional feature that enables the pre-filling of slots, ensuring specific context before individual test cases are run. You can define fixtures in either:

Test case files — using the fixtures key at the top level of a test case YAML file (e.g. e2e_test_cases.yml).
Conftest files — using the same fixtures key in a conftest.yml or conftest.yaml file at the root of your test tree or in any subfolder.

Within a single file, each fixture name must be unique. Across files, the same fixture name can be used to override a fixture from a parent scope. Fixture names correspond to sets of slot key-value pairs. When a particular test case needs predefined slot values, you can reference the fixture name within the test case definition by adding it to the fixtures key.

Consider the following example, which includes a test case file with fixtures and two test cases that leverage these fixtures:

tests/fixture-tests.yml

fixtures:
  - premium:
      - membership_type: premium
      - logged_in: True
  - standard:
      - logged_in: False
      - membership_type: standard

test_cases:
  - test_case: "test_premium_booking"
    fixtures:
      - premium
    steps:
      - user: "Hi!"
      - bot: "Welcome back! How can I help you?"
      - user: "I want to book a trip."
      - utter: utter_ask_location
      - user: "I would like to travel to Lisbon."
      - slot_was_set:
          - location: "Lisbon"
      - utter: utter_ask_date
      - user: "I would like to travel on 22nd of June."
      - slot_was_set:
          - travel_date: "2023-06-22"
      - bot: "Great! I will book your trip to Lisbon on 22nd of June."
      - bot: "You saved 20% by being a premium member."

- test_case: "test_anonymous_booking"
    fixtures:
      - standard
    steps:
      - user: "Hi!"
      - bot: "Hey! How can I help you?"
      - user: "I want to book a trip."
      - utter: utter_ask_location
      - user: "I would like to travel to Paris."
      - slot_was_set:
          - location: "Paris"
      - utter: utter_ask_date
      - user: "I would like to travel on 2nd of April."
      - slot_was_set:
          - travel_date: "2023-04-02"
      - bot: "Great! I will book your trip to Paris on 2nd of April."
      - bot: "You can also choose to save 20% by becoming a premium member."

These slots in fixtures are set after the action_session_start action and before the first step is executed by the test runner.

Conftest-style fixture hierarchy

You can define fixtures in conftest files (conftest.yml or conftest.yaml) so that all e2e tests under a given path see them, without repeating definitions in every test file.

Fixtures defined in tests/conftest.yml are visible to all test files under tests/.