# Evaluating Models

## [Evaluating an NLU Model](https://legacy-docs-v1.rasa.com/1.7.4/user-guide/evaluating-models/#id4) 
A standard technique in machine learning is to keep some data separate as a _test set_.
You can [split your NLU training data](https://legacy-docs-v1.rasa.com/1.7.4/user-guide/command-line-interface/#train-test-split)
into train and test sets using:

```
rasa data split nlu
```

If you’ve done this, you can see how well your NLU model predicts the test cases using this command:

```
rasa test nlu -u train_test_split/test_data.md --model models/nlu-20180323-145833.tar.gz
```

If you don’t want to create a separate test set, you can still estimate how well your model generalises using cross-validation. To do this, add the flag `--cross-validation`:

```
rasa test nlu -u data/nlu.md --config config.yml --cross-validation
```

The full list of options for the script is:

```
usage: rasa test nlu [-h] [-v] [-vv] [--quiet] [-m MODEL] [-u NLU] [--out OUT]
                     [--successes] [--no-errors] [--histogram HISTOGRAM]
                     [--confmat CONFMAT] [-c CONFIG [CONFIG ...]]
                     [--cross-validation] [-f FOLDS] [-r RUNS]
                     [-p PERCENTAGES [PERCENTAGES ...]] [--no-plot]

optional arguments:
  -h, --help            show this help message and exit
  -m MODEL, --model MODEL
                        Path to a trained Rasa model. If a directory is
                        specified, it will use the latest model in this
                        directory. (default: models)
  -u NLU, --nlu NLU     File or folder containing your NLU data. (default:
                        data)
  --out OUT             Output path for any files created during the
                        evaluation. (default: results)
  --successes           If set successful predictions (intent and entities)
                        will be written to a file. (default: False)
  --no-errors           If set incorrect predictions (intent and entities)
                        will NOT be written to a file. (default: False)
  --histogram HISTOGRAM
                        Output path for the confidence histogram. (default:
                        hist.png)
  --confmat CONFMAT     Output path for the confusion matrix plot. (default:
                        confmat.png)
  -c CONFIG [CONFIG ...], --config CONFIG [CONFIG ...]
                        Model configuration file. If a single file is passed
                        and cross validation mode is chosen, cross-validation
                        is performed, if multiple configs or a folder of
                        configs are passed, models will be trained and
                        compared directly. (default: None)
  --no-plot             Don't render evaluation plots (default: False)

Python Logging Options:
  -v, --verbose         Be verbose. Sets logging level to INFO. (default:
                        None)
  -vv, --debug          Print lots of debugging statements. Sets logging level
                        to DEBUG. (default: None)
  --quiet               Be quiet! Sets logging level to WARNING. (default:
                        None)

Cross Validation:
  --cross-validation    Switch on cross validation mode. Any provided model
                        will be ignored. (default: False)
  -f FOLDS, --folds FOLDS
                        Number of cross validation folds (cross validation
                        only). (default: 5)

Comparison Mode:
  -r RUNS, --runs RUNS  Number of comparison runs to make. (default: 3)
  -p PERCENTAGES [PERCENTAGES ...], --percentages PERCENTAGES [PERCENTAGES ...]
                        Percentages of training data to exclude during
                        comparison. (default: [0, 25, 50, 75])
```

### [Comparing NLU Pipelines](https://legacy-docs-v1.rasa.com/1.7.4/user-guide/evaluating-models/#id5) 
By passing multiple pipeline configurations (or a folder containing them) to the CLI, Rasa will run
a comparative examination between the pipelines.

```
$ rasa test nlu --config pretrained_embeddings_spacy.yml supervised_embeddings.yml
  --nlu data/nlu.md --runs 3 --percentages 0 25 50 70 90
```

The command in the example above will create a train/test split from your data,
then train each pipeline multiple times with 0, 25, 50, 70 and 90% of your intent data excluded from the training set.
The models are then evaluated on the test set and the f1-score for each exclusion percentage is recorded. This process
runs three times (i.e. with 3 test sets in total) and then a graph is plotted using the means and standard deviations of
the f1-scores.

### [Intent Classification](https://legacy-docs-v1.rasa.com/1.7.4/user-guide/evaluating-models/#id6) 
The evaluation script will produce a report, confusion matrix,
and confidence histogram for your model.

The report logs precision, recall and f1 measure for
each intent and entity, as well as providing an overall average.
You can save these reports as JSON files using the `--report` argument.

The confusion matrix shows you which
intents are mistaken for others; any samples which have been
incorrectly predicted are logged and saved to a file
called `errors.json` for easier debugging.

The histogram that the script produces allows you to visualise the
confidence distribution for all predictions,
with the volume of correct and incorrect predictions being displayed by
blue and red bars respectively.
Improving the quality of your training data will move the blue
histogram bars to the right and the red histogram bars
to the left of the plot.

### [Response Selection](https://legacy-docs-v1.rasa.com/1.7.4/user-guide/evaluating-models/#id7) 
The evaluation script will produce a combined report for all response selector models in your pipeline.

### [Entity Extraction](https://legacy-docs-v1.rasa.com/1.7.4/user-guide/evaluating-models/#id8) 
The `CRFEntityExtractor` is the only entity extractor which you train using your own data,
and so is the only one that will be evaluated. If you use the spaCy or duckling
pre-trained entity extractors, Rasa NLU will not include these in the evaluation.

### [Entity Scoring](https://legacy-docs-v1.rasa.com/1.7.4/user-guide/evaluating-models/#id9) 
To evaluate entity extraction we apply a simple tag-based approach. We don’t consider BILOU tags, but only the
entity type tags on a per token basis. For location entity like “near Alexanderplatz” we
expect the labels `LOC LOC` instead of the BILOU-based `B-LOC L-LOC`.

## [Evaluating a Core Model](https://legacy-docs-v1.rasa.com/1.7.4/user-guide/evaluating-models/#id10) 
You can evaluate your trained model on a set of test stories
by using the evaluate script:

```
rasa test core --stories test_stories.md --out results
```

This will print the failed stories to `results/failed_stories.md`.

### [Comparing Core Configurations](https://legacy-docs-v1.rasa.com/1.7.4/user-guide/evaluating-models/#id11) 
To choose a configuration for your core model, or to choose hyperparameters for a
specific policy, you want to measure how well Rasa Core will generalise
to conversations which it hasn’t seen before.

### [End-to-End Evaluation](https://legacy-docs-v1.rasa.com/1.7.4/user-guide/evaluating-models/#id12) 
Rasa lets you evaluate dialogues end-to-end, running through
test conversations and making sure that both NLU and Core make correct predictions.