# Evaluating Models

**Warning:** This document is for an old version of Rasa.

## Evaluating an NLU Model
A standard technique in machine learning is to keep some data separate as a *test set*.
You can [split your NLU training data](https://legacy-docs-v1.rasa.com/1.2.9/user-guide/command-line-interface/#train-test-split) into train and test sets using:

```bash
rasa data split nlu
```

If you’ve done this, you can see how well your NLU model predicts the test cases using this command:

```bash
rasa test nlu -u test_set.md --model models/nlu-20180323-145833.tar.gz
```

If you don’t want to create a separate test set, you can still estimate how well your model generalises using cross-validation. To do this, add the flag `--cross-validation`:

```bash
rasa test nlu -u data/nlu.md --config config.yml --cross-validation
```

The full list of options for the script is:

```bash
usage: rasa test nlu [-h] [-v] [-vv] [--quiet] [-m MODEL] [-u NLU] [--out OUT]
                     [--successes] [--no-errors] [--histogram HISTOGRAM]
                     [--confmat CONFMAT] [-c CONFIG [CONFIG ...]]
                     [--cross-validation] [-f FOLDS] [-r RUNS]
                     [-p PERCENTAGES [PERCENTAGES ...]] [--no-plot]

optional arguments:
  -h, --help            show this help message and exit
  -m MODEL, --model MODEL
                        Path to a trained Rasa model. If a directory is
                        specified, it will use the latest model in this
                        directory. (default: models)
  -u NLU, --nlu NLU     File or folder containing your NLU data. (default:
                        data)
  --out OUT             Output path for any files created during the
                        evaluation. (default: results)
  --successes           If set successful predictions (intent and entities)
                        will be written to a file. (default: False)
  --no-errors           If set incorrect predictions (intent and entities)
                        will NOT be written to a file. (default: False)
  --histogram HISTOGRAM
                        Output path for the confidence histogram. (default:
                        hist.png)
  --confmat CONFMAT     Output path for the confusion matrix plot. (default:
                        confmat.png)
  -c CONFIG [CONFIG ...], --config CONFIG [CONFIG ...]
                        Model configuration file. If a single file is passed
                        and cross validation mode is chosen, cross-validation
                        is performed, if multiple configs or a folder of
                        configs are passed, models will be trained and
                        compared directly. (default: None)
  --no-plot             Don't render evaluation plots (default: False)

Python Logging Options:
  -v, --verbose         Be verbose. Sets logging level to INFO. (default:
                        None)
  -vv, --debug          Print lots of debugging statements. Sets logging level
                        to DEBUG. (default: None)
  --quiet               Be quiet! Sets logging level to WARNING. (default:
                        None)

Cross Validation:
  --cross-validation    Switch on cross validation mode. Any provided model
                        will be ignored. (default: False)
  -f FOLDS, --folds FOLDS
                        Number of cross validation folds (cross validation
                        only). (default: 5)

Comparison Mode:
  -r RUNS, --runs RUNS  Number of comparison runs to make. (default: 3)
  -p PERCENTAGES [PERCENTAGES ...], --percentages PERCENTAGES [PERCENTAGES ...]
                        Percentages of training data to exclude during
                        comparison. (default: [0, 25, 50, 75])
```

### Comparing NLU Pipelines
By passing multiple pipeline configurations (or a folder containing them) to the CLI, Rasa will run a comparative examination between the pipelines.

```bash
$ rasa test nlu --config pretrained_embeddings_spacy.yml supervised_embeddings.yml
  --nlu data/nlu.md --runs 3 --percentages 0 25 50 70 90
```

### Intent Classification
The evaluation script will produce a report, confusion matrix, and confidence histogram for your model.

### Entity Extraction
The `CRFEntityExtractor` is the only entity extractor which you train using your own data, and so is the only one that will be evaluated.

### Evaluating a Core Model
You can evaluate your trained model on a set of test stories by using the evaluate script:

```bash
rasa test core --stories test_stories.md --out results
```

### Comparing Policies
To choose a specific policy configuration, or to choose hyperparameters for a specific policy, you want to measure how well Rasa Core will generalise to conversations which it hasn’t seen before.

### End-to-End Evaluation
Rasa lets you evaluate dialogues end-to-end, running through test conversations and making sure that both NLU and Core make correct predictions.