Testing Your Assistant

These docs are for version 1.x of Rasa Open Source.

User Guide

NLU

Core

Conversation Design

API Reference

Migrate from (beta)

Reference

Testing Your Assistant

End-to-End Testing

Rasa Open Source lets you test dialogues end-to-end by running through test conversations and making sure that both NLU and Core make correct predictions.

To do this, you need some stories in the end-to-end format, which includes both the NLU output and the original text. Here are some examples:

By default Rasa Open Source saves conversation tests to tests/conversation_tests.md. You can test your assistant against them by running:

$ rasa test

Note: Custom Actions are not executed as part of end-to-end tests.

Evaluating an NLU Model

A standard technique in machine learning is to keep some data separate as a test set. You can split your NLU training data into train and test sets using:

rasa data split nlu

If you’ve done this, you can see how well your NLU model predicts the test cases using this command:

rasa test nlu -u train_test_split/test_data.md --model models/nlu-20180323-145833.tar.gz

Comparing NLU Pipelines

By passing multiple pipeline configurations (or a folder containing them) to the CLI, Rasa will run a comparative examination between the pipelines.

$ rasa test nlu --config pretrained_embeddings_spacy.yml supervised_embeddings.yml --nlu data/nlu.md --runs 3 --percentages 0 25 50 70 90

Intent Classification

The evaluation script will produce a report, confusion matrix, and confidence histogram for your model. The report logs precision, recall, and f1 measure for each intent and entity, as well as providing an overall average.

Comparing Core Configurations

To choose a configuration for your core model or to choose hyperparameters for a specific policy, you want to measure how well Rasa Core will generalize to conversations which it hasn’t seen before. Rasa Core has some scripts to help you choose and fine-tune your policy configuration.

$ rasa train core -c config_1.yml config_2.yml \
  -d domain.yml -s stories_folder --out comparison_models --runs 3 \
  --percentages 0 5 25 50 70 95

Once this script has finished, you can use the evaluate script in compare mode to evaluate the models you just trained:

$ rasa test core -m comparison_models --stories stories_folder --out comparison_results --evaluate-model-directory

Note: This training process can take a long time, so we’d suggest letting it run somewhere in the background where it can’t be interrupted.