Evaluating Models

Warning: This document is for an old version of Rasa.

Evaluating an NLU Model

A standard technique in machine learning is to keep some data separate as a test set. You can split your NLU training data into train and test sets using:

rasa data split nlu

If you’ve done this, you can see how well your NLU model predicts the test cases using this command:

rasa test nlu -u train_test_split/test_data.md --model models/nlu-20180323-145833.tar.gz

If you don’t want to create a separate test set, you can still estimate how well your model generalises using cross-validation. To do this, add the flag --cross-validation:

rasa test nlu -u data/nlu.md --config config.yml --cross-validation

The full list of options for the script is:

usage: rasa test nlu [-h] [-v] [-vv] [--quiet] [-m MODEL] [-u NLU] [--out OUT]
                     [--successes] [--no-errors] [--histogram HISTOGRAM]
                     [--confmat CONFMAT] [-c CONFIG [CONFIG ...]]
                     [--cross-validation] [-f FOLDS] [-r RUNS]
                     [-p PERCENTAGES [PERCENTAGES ...]] [--no-plot]

optional arguments:
  -h, --help            show this help message and exit
  -m MODEL, --model MODEL
                        Path to a trained Rasa model. If a directory is
                        specified, it will use the latest model in this
                        directory. (default: models)
  -u NLU, --nlu NLU     File or folder containing your NLU data. (default:
                        data)
  --out OUT             Output path for any files created during the
                        evaluation. (default: results)
  --successes           If set successful predictions (intent and entities)
                        will be written to a file. (default: False)
  --no-errors           If set incorrect predictions (intent and entities)
                        will NOT be written to a file. (default: False)
  --histogram HISTOGRAM
                        Output path for the confidence histogram. (default:
                        hist.png)
  --confmat CONFMAT     Output path for the confusion matrix plot. (default:
                        confmat.png)
  -c CONFIG [CONFIG ...], --config CONFIG [CONFIG ...]
                        Model configuration file. If a single file is passed
                        and cross-validation mode is chosen, cross-validation
                        is performed, if multiple configs or a folder of
                        configs are passed, models will be trained and
                        compared directly. (default: None)
  --no-plot             Don't render evaluation plots (default: False)

Python Logging Options:
  -v, --verbose         Be verbose. Sets logging level to INFO. (default:
                        None)
  -vv, --debug          Print lots of debugging statements. Sets logging level
                        to DEBUG. (default: None)
  --quiet               Be quiet! Sets logging level to WARNING. (default:
                        None)

Cross Validation:
  --cross-validation    Switch on cross validation mode. Any provided model
                        will be ignored. (default: False)
  -f FOLDS, --folds FOLDS
                        Number of cross validation folds (cross validation
                        only). (default: 5)

Comparison Mode:
  -r RUNS, --runs RUNS  Number of comparison runs to make. (default: 3)
  -p PERCENTAGES [PERCENTAGES ...], --percentages PERCENTAGES [PERCENTAGES ...]
                        Percentages of training data to exclude during
                        comparison. (default: [0, 25, 50, 75])

Comparing NLU Pipelines

By passing multiple pipeline configurations (or a folder containing them) to the CLI, Rasa will run a comparative examination between the pipelines.

$ rasa test nlu --config pretrained_embeddings_spacy.yml supervised_embeddings.yml
  --nlu data/nlu.md --runs 3 --percentages 0 25 50 70 90

The command in the example above will create a train/test split from your data, then train each pipeline multiple times with 0, 25, 50, 70 and 90% of your intent data excluded from the training set. The models are then evaluated on the test set and the f1-score for each exclusion percentage is recorded. This process runs three times (i.e. with 3 test sets in total) and then a graph is plotted using the means and standard deviations of the f1-scores.

Intent Classification

The evaluation script will produce a report, confusion matrix, and confidence histogram for your model.

The report logs precision, recall and f1 measure for each intent and entity, as well as providing an overall average. You can save these reports as JSON files using the --report argument.

The confusion matrix shows you which intents are mistaken for others; any samples which have been incorrectly predicted are logged and saved to a file called errors.json for easier debugging.

The histogram that the script produces allows you to visualise the confidence distribution for all predictions, with the volume of correct and incorrect predictions being displayed by blue and red bars respectively. Improving the quality of your training data will move the blue histogram bars to the right and the red histogram bars to the left of the plot.

Response Selection

The evaluation script will produce a combined report for all response selector models in your pipeline.

The report logs precision, recall and f1 measure for each response, as well as providing an overall average. You can save these reports as JSON files using the --report argument.

Entity Extraction

The CRFEntityExtractor is the only entity extractor which you train using your own data, and so is the only one that will be evaluated. If you use the spaCy or duckling pre-trained entity extractors, Rasa NLU will not include these in the evaluation.

Rasa NLU will report recall, precision, and f1 measure for each entity type that CRFEntityExtractor is trained to recognize.

Entity Scoring

To evaluate entity extraction we apply a simple tag-based approach. We don’t consider BILOU tags, but only the entity type tags on a per token basis. For location entity like “near Alexanderplatz” we expect the labels LOC LOC instead of the BILOU-based B-LOC L-LOC. Our approach is more lenient when it comes to evaluation, as it rewards partial extraction and does not punish the splitting of entities.

Evaluating a Core Model

You can evaluate your trained model on a set of test stories by using the evaluate script:

rasa test core --stories test_stories.md --out results

This will print the failed stories to results/failed_stories.md. We count any story as failed if at least one of the actions was predicted incorrectly. In addition, this will save a confusion matrix to a file called results/story_confmat.pdf. For each action in your domain, the confusion matrix shows how often the action was correctly predicted and how often an incorrect action was predicted instead.

The full list of options for the script is:

usage: rasa test core [-h] [-v] [-vv] [--quiet] [-m MODEL [MODEL ...]]
                      [-s STORIES] [--max-stories MAX_STORIES] [--out OUT]
                      [--e2e] [--endpoints ENDPOINTS]
                      [--fail-on-prediction-errors] [--url URL]
                      [--evaluate-model-directory] [--no-plot]

optional arguments:
  -h, --help            show this help message and exit
  -m MODEL [MODEL ...], --model MODEL [MODEL ...]
                        Path to a pre-trained model. If it is a 'tar.gz' file
                        that model file will be used. If it is a directory,
                        the latest model in that directory will be used
                        (exception: '--evaluate-model-directory' flag is set).
                        If multiple 'tar.gz' files are provided, all those
                        models will be compared. (default: [None])
  -s STORIES, --stories STORIES
                        File or folder containing your test stories. (default:
                        data)
  --max-stories MAX_STORIES
                        Maximum number of stories to test on. (default: None)
  --out OUT             Output path for any files created during the
                        evaluation. (default: results)
  --e2e, --end-to-end   Run an end-to-end evaluation for combined action and
                        intent prediction. Requires a story file in end-to-end
                        format. (default: False)
  --endpoints ENDPOINTS
                        Configuration file for the connectors as a yml file.
                        (default: None)
  --fail-on-prediction-errors
                        If a prediction error is encountered, an exception is
                        thrown. This can be used to validate stories during
                        tests, e.g. on travis. (default: False)
  --url URL             If supplied, downloads a story file from a URL and
                        trains on it. Fetches the data by sending a GET
                        request to the supplied URL. (default: None)
  --evaluate-model-directory
                        Should be set to evaluate models trained via 'rasa
                        train core --config <config-1> <config-2>'. All models
                        in the provided directory are evaluated and compared
                        against each other. (default: False)
  --no-plot             Don't render evaluation plots (default: False)

Python Logging Options:
  -v, --verbose         Be verbose. Sets logging level to INFO. (default:
                        None)
  -vv, --debug          Print lots of debugging statements. Sets logging level
                        to DEBUG. (default: None)
  --quiet               Be quiet! Sets logging level to WARNING. (default:
                        None)

Comparing Core Configurations

To choose a configuration for your core model, or to choose hyperparameters for a specific policy, you want to measure how well Rasa Core will generalise to conversations which it hasn’t seen before. Especially in the beginning of a project, you do not have a lot of real conversations to use to train your bot, so you don’t just want to throw some away to use as a test set.

Rasa Core has some scripts to help you choose and fine-tune your policy configuration. Once you are happy with it, you can then train your final configuration on your full data set. To do this, you first have to train models for your different configurations. Create two (or more) config files including the policies you want to compare, and then use the compare mode of the train script to train your models:

$ rasa train core -c config_1.yml config_2.yml \
  -d domain.yml -s stories_folder --out comparison_models --runs 3 \
  --percentages 0 5 25 50 70 95

For each policy configuration provided, Rasa Core will be trained multiple times with 0, 5, 25, 50, 70 and 95% of your training stories excluded from the training data. This is done for multiple runs to ensure consistent results.

Once this script has finished, you can use the evaluate script in compare mode to evaluate the models you just trained:

$ rasa test core -m comparison_models --stories stories_folder
--out comparison_results --evaluate-model-directory

This will evaluate each of the models on the provided stories (can be either training or test set) and plot some graphs to show you which policy performs best. By evaluating on the full set of stories, you can measure how well Rasa Core is predicting the held-out stories.

To compare single policies create config files containing only one policy each. If you’re not sure which policies to compare, we’d recommend trying out the EmbeddingPolicy and the KerasPolicy to see which one works better for you.

End-to-End Evaluation

Rasa lets you evaluate dialogues end-to-end, running through test conversations and making sure that both NLU and Core make correct predictions. To do this, you need some stories in the end-to-end format, which includes both the NLU output and the original text. Here is an example:

## end-to-end story 1
* greet: hello
   - utter_ask_howcanhelp
* inform: show me [chinese](cuisine) restaurants
   - utter_ask_location
* inform: in [Paris](location)
   - utter_ask_price

If you’ve saved end-to-end stories as a file called e2e_stories.md, you can evaluate your model against them by running:

$ rasa test --stories e2e_stories.md --e2e

Note

Make sure your model file in models is a combined core and nlu model. If it does not contain an NLU model, Core will use the default RegexInterpreter.