Evaluating Models

These docs are for version 1.x of Rasa Open Source.

Evaluating Models

Evaluating an NLU Model

A standard technique in machine learning is to keep some data separate as a test set. You can split your NLU training data into train and test sets using:

rasa data split nlu

If you’ve done this, you can see how well your NLU model predicts the test cases using this command:

rasa test nlu -u train_test_split/test_data.md --model models/nlu-20180323-145833.tar.gz

If you don’t want to create a separate test set, you can still estimate how well your model generalises using cross-validation. To do this, add the flag --cross-validation:

rasa test nlu -u data/nlu.md --config config.yml --cross-validation

The full list of options for the script is:

usage: rasa test nlu [-h] [-v] [-vv] [--quiet] [-m MODEL] [-u NLU] [--out OUT]
                     [--successes] [--no-errors] [--histogram HISTOGRAM]
                     [--confmat CONFMAT] [-c CONFIG [CONFIG ...]]
                     [--cross-validation] [-f FOLDS] [-r RUNS]
                     [-p PERCENTAGES [PERCENTAGES ...]] [--no-plot]

optional arguments:
  -h, --help            show this help message and exit
  -m MODEL, --model MODEL
                        Path to a trained Rasa model. If a directory is
                        specified, it will use the latest model in this
                        directory. (default: models)
  -u NLU, --nlu NLU     File or folder containing your NLU data. (default:
                        data)
  --out OUT             Output path for any files created during the
                        evaluation. (default: results)
  --successes           If set successful predictions (intent and entities)
                        will be written to a file. (default: False)
  --no-errors           If set incorrect predictions (intent and entities)
                        will NOT be written to a file. (default: False)
  --histogram HISTOGRAM
                        Output path for the confidence histogram. (default:
                        hist.png)
  --confmat CONFMAT     Output path for the confusion matrix plot. (default:
                        confmat.png)
  -c CONFIG [CONFIG ...], --config CONFIG [CONFIG ...]
                        Model configuration file. If a single file is passed
                        and cross validation mode is chosen, cross-validation
                        is performed, if multiple configs or a folder of
                        configs are passed, models will be trained and
                        compared directly. (default: None)
  --no-plot             Don't render evaluation plots (default: False)

Python Logging Options:
  -v, --verbose         Be verbose. Sets logging level to INFO. (default:
                        None)
  -vv, --debug          Print lots of debugging statements. Sets logging level
                        to DEBUG. (default: None)
  --quiet               Be quiet! Sets logging level to WARNING. (default:
                        None)

Comparing NLU Pipelines

By passing multiple pipeline configurations (or a folder containing them) to the CLI, Rasa will run a comparative examination between the pipelines.

$ rasa test nlu --config pretrained_embeddings_spacy.yml supervised_embeddings.yml
  --nlu data/nlu.md --runs 3 --percentages 0 25 50 70 90

The command in the example above will create a train/test split from your data, then train each pipeline multiple times with 0, 25, 50, 70 and 90% of your intent data excluded from the training set. The models are then evaluated on the test set and the f1-score for each exclusion percentage is recorded. This process runs three times (i.e. with 3 test sets in total) and then a graph is plotted using the means and standard deviations of the f1-scores.

Intent Classification

The evaluation script will produce a report, confusion matrix, and confidence histogram for your model.

The report logs precision, recall and f1 measure for each intent and entity, as well as providing an overall average. You can save these reports as JSON files using the --report argument.

The confusion matrix shows you which intents are mistaken for others; any samples which have been incorrectly predicted are logged and saved to a file called errors.json for easier debugging.

The histogram that the script produces allows you to visualise the confidence distribution for all predictions, with the volume of correct and incorrect predictions being displayed by blue and red bars respectively.

Response Selection

The evaluation script will produce a combined report for all response selector models in your pipeline.

Entity Extraction

The CRFEntityExtractor is the only entity extractor which you train using your own data, and so is the only one that will be evaluated.

Entity Scoring

To evaluate entity extraction we apply a simple tag-based approach. We don’t consider BILOU tags, but only the entity type tags on a per token basis.

Evaluating a Core Model

You can evaluate your trained model on a set of test stories by using the evaluate script:

rasa test core --stories test_stories.md --out results

This will print the failed stories to results/failed_stories.md.

Comparing Core Configurations

To choose a configuration for your core model, or to choose hyperparameters for a specific policy, you want to measure how well Rasa Core will generalise to conversations which it hasn’t seen before.

End-to-End Evaluation

Rasa lets you evaluate dialogues end-to-end, running through test conversations and making sure that both NLU and Core make correct predictions.

## end-to-end story 1
* greet: hello
   - utter_ask_howcanhelp
* inform: show me [chinese](cuisine) restaurants
   - utter_ask_location
* inform: in [Paris](location)
   - utter_ask_price

If you’ve saved end-to-end stories as a file called e2e_stories.md, you can evaluate your model against them by running:

$ rasa test --stories e2e_stories.md --e2e