Testing Your Assistant
Testing Your Assistant
Overview
Rasa Open Source lets you test dialogues end-to-end by running through test conversations and making sure that both NLU and Core make correct predictions. To do this, you need some stories in the end-to-end format, which includes both the NLU output and the original text. Here are some examples:
By default Rasa Open Source saves conversation tests to tests/conversation_tests.md. You can test your assistant against them by running:
$ rasa test
Note
Custom Actions are not executed as part of end-to-end tests. If your custom actions append any events to the tracker, this has to be reflected in your end-to-end tests (e.g. by adding slot events to your end-to-end story).
Make sure your model file in models is a combined core and nlu model. If it does not contain an NLU model, Core will use the default RegexInterpreter.
Evaluating an NLU Model
A standard technique in machine learning is to keep some data separate as a test set. You can split your NLU training data into train and test sets using:
rasa data split nlu
If you’ve done this, you can see how well your NLU model predicts the test cases using this command:
rasa test nlu -u train_test_split/test_data.md --model models/nlu-20180323-145833.tar.gz
If you don’t want to create a separate test set, you can still estimate how well your model generalises using cross-validation. To do this, add the flag --cross-validation:
rasa test nlu -u data/nlu.md --config config.yml --cross-validation
The full list of options for the script is:
usage: rasa test nlu [-h] [-v] [-vv] [--quiet] [-m MODEL] [-u NLU] [--out OUT]
[--successes] [--no-errors] [--histogram HISTOGRAM]
[--confmat CONFMAT] [-c CONFIG [CONFIG ...]]
[--cross-validation] [-f FOLDS] [-r RUNS]
[--p PERCENTAGES [PERCENTAGES ...]] [--no-plot]
optional arguments:
-h, --help show this help message and exit
-m MODEL, --model MODEL
Path to a trained Rasa model. If a directory is
specified, it will use the latest model in this
directory. (default: models)
-u NLU, --nlu NLU File or folder containing your NLU data. (default:
data)
--out OUT Output path for any files created during the
evaluation. (default: results)
--successes If set successful predictions (intent and entities)
will be written to a file. (default: False)
--no-errors If set incorrect predictions (intent and entities)
will NOT be written to a file. (default: False)
--histogram HISTOGRAM
Output path for the confidence histogram. (default:
hist.png)
--confmat CONFMAT Output path for the confusion matrix plot. (default:
confmat.png)
-c CONFIG [CONFIG ...], --config CONFIG [CONFIG ...]
Model configuration file. If a single file is passed
and cross validation mode is chosen, cross-validation
is performed, if multiple configs or a folder of
configs are passed, models will be trained and
compared directly. (default: None)
--no-plot Don't render evaluation plots (default: False)
Python Logging Options:
-v, --verbose Be verbose. Sets logging level to INFO. (default:
None)
-vv, --debug Print lots of debugging statements. Sets logging level
to DEBUG. (default: None)
--quiet Be quiet! Sets logging level to WARNING. (default:
None)
Cross Validation:
--cross-validation Switch on cross validation mode. Any provided model
will be ignored. (default: False)
-f FOLDS, --folds FOLDS
Number of cross validation folds (cross validation
only). (default: 5)
Comparison Mode:
-r RUNS, --runs RUNS Number of comparison runs to make. (default: 3)
-p PERCENTAGES [PERCENTAGES ...], --percentages PERCENTAGES [PERCENTAGES ...]
Percentages of training data to exclude during
comparison. (default: [0, 25, 50, 75])
Comparing NLU Pipelines
By passing multiple pipeline configurations (or a folder containing them) to the CLI, Rasa will run a comparative examination between the pipelines.
$ rasa test nlu --config pretrained_embeddings_spacy.yml supervised_embeddings.yml
--nlu data/nlu.md --runs 3 --percentages 0 25 50 70 90
Intent Classification
The evaluation script will produce a report, confusion matrix, and confidence histogram for your model.
The report logs precision, recall and f1 measure for each intent and entity, as well as providing an overall average.
You can save these reports as JSON files using the --report argument.
Entity Extraction
The CRFEntityExtractor is the only entity extractor which you train using your own data, and so is the only one that will be evaluated. If you use the spaCy or duckling pre-trained entity extractors, Rasa NLU will not include these in the evaluation.
Rasa NLU will report recall, precision, and f1 measure for each entity type that CRFEntityExtractor is trained to recognize.
Evaluating a Core Model
You can evaluate your trained model on a set of test stories by using the evaluate script:
rasa test core --stories test_stories.md --out results
This will print the failed stories to results/failed_stories.md. We count any story as failed if at least one of the actions was predicted incorrectly.
The full list of options for the script is:
usage: rasa test core [-h] [-v] [-vv] [--quiet] [-m MODEL [MODEL ...]]
[-s STORIES] [--max-stories MAX_STORIES] [--out OUT]
[--e2e] [--endpoints ENDPOINTS]
[--fail-on-prediction-errors] [--url URL]
[--evaluate-model-directory] [--no-plot]
Comparing Core Configurations
To choose a configuration for your core model, or to choose hyperparameters for a specific policy, you want to measure how well Rasa Core will generalise to conversations which it hasn’t seen before. Rasa Core has some scripts to help you choose and fine-tune your policy configuration. Once you are happy with it, you can then train your final configuration on your full data set.
Note
This training process can take a long time, so we’d suggest letting it run somewhere in the background where it can’t be interrupted.