Testing Your Assistant

Overview

This document provides information on how to test your Rasa assistant, focusing on different aspects such as End-to-End testing, Evaluating an NLU and Core Model, and more.

End-to-End Testing

Rasa Open Source lets you test dialogues end-to-end by running through test conversations and making sure that both NLU and Core make correct predictions.
To do this, you need some stories in the end-to-end format, which includes both the NLU output and the original text. Here are some examples:

By default Rasa Open Source saves conversation tests to tests/conversation_tests.md. You can test your assistant against them by running:

$ rasa test

Note

Custom Actions are not executed as part of end-to-end tests. If your custom actions append any events to the tracker, this has to be reflected in your end-to-end tests (e.g. by adding slot events to your end-to-end story).

Make sure your model file in models is a combined core and nlu model. If it does not contain an NLU model, Core will use the default RegexInterpreter.

Evaluating an NLU Model

A standard technique in machine learning is to keep some data separate as a test set. You can split your NLU training data into train and test sets using:

rasa data split nlu

Using the command below, you can see how well your NLU model predicts the test cases:

rasa test nlu -u train_test_split/test_data.md --model models/nlu-20180323-145833.tar.gz

You can also estimate how well your model generalises using cross-validation by adding the flag --cross-validation:

rasa test nlu -u data/nlu.md --config config.yml --cross-validation

The full list of options for the script is:

usage: rasa test nlu [-h] [-v] [-vv] [--quiet] [-m MODEL] [-u NLU] [--out OUT]
                     [--successes] [--no-errors] [--histogram HISTOGRAM]
                     [--confmat CONFMAT] [-c CONFIG [CONFIG ...]]
                     [--cross-validation] [-f FOLDS] [-r RUNS]
                     [--percentages [PERCENTAGES ...]] [--no-plot]

Comparing NLU Pipelines

You can pass multiple pipeline configurations (or a folder containing them) to the CLI to run a comparative examination between the pipelines:

$ rasa test nlu --config pretrained_embeddings_spacy.yml supervised_embeddings.yml

Evaluating a Core Model

You can evaluate your trained model on a set of test stories using the evaluate script:

rasa test core --stories test_stories.md --out results

This will print the failed stories to results/failed_stories.md. We count any story as failed if at least one of the actions was predicted incorrectly.

Comparing Core Configurations

To choose a configuration for your core model, you want to measure how well Rasa Core will generalise to conversations which it hasn’t seen before. You can train models for different configurations and then evaluate them:

$ rasa train core -c config_1.yml config_2.yml \
  -d domain.yml -s stories_folder --out comparison_models --runs 3 \
  --percentages 0 5 25 50 70 95

Then evaluate the results:

$ rasa test core -m comparison_models --stories stories_folder --out comparison_results --evaluate-model-directory

Summary

This document covers essential instructions to help you evaluate and test your assistant effectively. Using the commands and practices outlined, you can ensure your Rasa model performs up to expectations.