# Evaluating Models - [Evaluating an NLU Model](https://legacy-docs-v1.rasa.com/1.7.3/user-guide/evaluating-models/#evaluating-an-nlu-model) - [Comparing NLU Pipelines](https://legacy-docs-v1.rasa.com/1.7.3/user-guide/evaluating-models/#comparing-nlu-pipelines) - [Intent Classification](https://legacy-docs-v1.rasa.com/1.7.3/user-guide/evaluating-models/#intent-classification) - [Response Selection](https://legacy-docs-v1.rasa.com/1.7.3/user-guide/evaluating-models/#response-selection) - [Entity Extraction](https://legacy-docs-v1.rasa.com/1.7.3/user-guide/evaluating-models/#entity-extraction) - [Entity Scoring](https://legacy-docs-v1.rasa.com/1.7.3/user-guide/evaluating-models/#entity-scoring) - [Evaluating a Core Model](https://legacy-docs-v1.rasa.com/1.7.3/user-guide/evaluating-models/#evaluating-a-core-model) - [Comparing Core Configurations](https://legacy-docs-v1.rasa.com/1.7.3/user-guide/evaluating-models/#comparing-core-configurations) - [End-to-End Evaluation](https://legacy-docs-v1.rasa.com/1.7.3/user-guide/evaluating-models/#end-to-end-evaluation) ## Evaluating an NLU Model A standard technique in machine learning is to keep some data separate as a _test set_. You can [split your NLU training data](https://legacy-docs-v1.rasa.com/1.7.3/user-guide/command-line-interface/#train-test-split) into train and test sets using: ``` rasa data split nlu ``` If you’ve done this, you can see how well your NLU model predicts the test cases using this command: ``` rasa test nlu -u train_test_split/test_data.md --model models/nlu-20180323-145833.tar.gz ``` If you don’t want to create a separate test set, you can still estimate how well your model generalises using cross-validation. To do this, add the flag `--cross-validation`: ``` rasa test nlu -u data/nlu.md --config config.yml --cross-validation ``` The full list of options for the script is: ``` usage: rasa test nlu [-h] [-v] [-vv] [--quiet] [-m MODEL] [-u NLU] [--out OUT] [--successes] [--no-errors] [--histogram HISTOGRAM] [--confmat CONFMAT] [-c CONFIG [CONFIG ...]] [--cross-validation] [-f FOLDS] [-r RUNS] [-p PERCENTAGES [PERCENTAGES ...]] [--no-plot] optional arguments: -h, --help show this help message and exit -m MODEL, --model MODEL Path to a trained Rasa model. If a directory is specified, it will use the latest model in this directory. (default: models) -u NLU, --nlu NLU File or folder containing your NLU data. (default: data) --out OUT Output path for any files created during the evaluation. (default: results) --successes If set successful predictions (intent and entities) will be written to a file. (default: False) --no-errors If set incorrect predictions (intent and entities) will NOT be written to a file. (default: False) --histogram HISTOGRAM Output path for the confidence histogram. (default: hist.png) --confmat CONFMAT Output path for the confusion matrix plot. (default: confmat.png) -c CONFIG [CONFIG ...], --config CONFIG [CONFIG ...] Model configuration file. If a single file is passed and cross validation mode is chosen, cross-validation is performed, if multiple configs or a folder of configs are passed, models will be trained and compared directly. (default: None) --no-plot Don't render evaluation plots (default: False) Python Logging Options: -v, --verbose Be verbose. Sets logging level to INFO. (default: None) -vv, --debug Print lots of debugging statements. Sets logging level to DEBUG. (default: None) --quiet Be quiet! Sets logging level to WARNING. (default: None) Cross Validation: --cross-validation Switch on cross validation mode. Any provided model will be ignored. (default: False) -f FOLDS, --folds FOLDS Number of cross validation folds (cross validation only). (default: 5) Comparison Mode: -r RUNS, --runs RUNS Number of comparison runs to make. (default: 3) -p PERCENTAGES [PERCENTAGES ...], --percentages PERCENTAGES [PERCENTAGES ...] Percentages of training data to exclude during comparison. (default: [0, 25, 50, 75]) ``` ### Comparing NLU Pipelines By passing multiple pipeline configurations (or a folder containing them) to the CLI, Rasa will run a comparative examination between the pipelines. ``` $ rasa test nlu --config pretrained_embeddings_spacy.yml supervised_embeddings.yml --nlu data/nlu.md --runs 3 --percentages 0 25 50 70 90 ``` The command in the example above will create a train/test split from your data, then train each pipeline multiple times with 0, 25, 50, 70 and 90% of your intent data excluded from the training set. The models are then evaluated on the test set and the f1-score for each exclusion percentage is recorded. This process runs three times (i.e. with 3 test sets in total) and then a graph is plotted using the means and standard deviations of the f1-scores. ### Intent Classification The evaluation script will produce a report, confusion matrix, and confidence histogram for your model. The report logs precision, recall and f1 measure for each intent and entity, as well as providing an overall average. You can save these reports as JSON files using the `--report` argument. The confusion matrix shows you which intents are mistaken for others; any samples which have been incorrectly predicted are logged and saved to a file called `errors.json` for easier debugging. The histogram that the script produces allows you to visualise the confidence distribution for all predictions, with the volume of correct and incorrect predictions being displayed by blue and red bars respectively. Improving the quality of your training data will move the blue histogram bars to the right and the red histogram bars to the left of the plot. ### Entity Extraction The `CRFEntityExtractor` is the only entity extractor which you train using your own data, and so is the only one that will be evaluated. If you use the spaCy or duckling pre-trained entity extractors, Rasa NLU will not include these in the evaluation. Rasa NLU will report recall, precision, and f1 measure for each entity type that `CRFEntityExtractor` is trained to recognize. ### Entity Scoring To evaluate entity extraction we apply a simple tag-based approach. We don’t consider BILOU tags, but only the entity type tags on a per token basis. For location entity like “near Alexanderplatz” we expect the labels `LOC LOC` instead of the BILOU-based `B-LOC L-LOC`. Our approach is more lenient when it comes to evaluation, as it rewards partial extraction and does not punish the splitting of entities. ## Evaluating a Core Model You can evaluate your trained model on a set of test stories by using the evaluate script: ``` rasa test core --stories test_stories.md --out results ``` This will print the failed stories to `results/failed_stories.md`. We count any story as failed if at least one of the actions was predicted incorrectly. In addition, this will save a confusion matrix to a file called `results/story_confmat.pdf`. For each action in your domain, the confusion matrix shows how often the action was correctly predicted and how often an incorrect action was predicted instead. The full list of options for the script is: ``` usage: rasa test core [-h] [-v] [-vv] [--quiet] [-m MODEL [MODEL ...]] [-s STORIES] [--max-stories MAX_STORIES] [--out OUT] [--e2e] [--endpoints ENDPOINTS] [--fail-on-prediction-errors] [--url URL] [--evaluate-model-directory] [--no-plot] optional arguments: -h, --help show this help message and exit -m MODEL [MODEL ...], --model MODEL [MODEL ...] Path to a pre-trained model. If it is a 'tar.gz' file that model file will be used. If it is a directory, the latest model in that directory will be used (exception: '--evaluate-model-directory' flag is set). If multiple 'tar.gz' files are provided, all those models will be compared. (default: [None]) -s STORIES, --stories STORIES File or folder containing your test stories. (default: data) --max-stories MAX_STORIES Maximum number of stories to test on. (default: None) --out OUT Output path for any files created during the evaluation. (default: results) --e2e, --end-to-end Run an end-to-end evaluation for combined action and intent prediction. Requires a story file in end-to-end format. (default: False) --endpoints ENDPOINTS Configuration file for the connectors as a yml file. (default: None) --fail-on-prediction-errors If a prediction error is encountered, an exception is thrown. This can be used to validate stories during tests, e.g. on travis. (default: False) --url URL If supplied, downloads a story file from a URL and trains on it. Fetches the data by sending a GET request to the supplied URL. (default: None) --evaluate-model-directory Should be set to evaluate models trained via 'rasa train core --config '. All models in the provided directory are evaluated and compared against each other. (default: False) --no-plot Don't render evaluation plots (default: False) Python Logging Options: -v, --verbose Be verbose. Sets logging level to INFO. (default: None) -vv, --debug Print lots of debugging statements. Sets logging level to DEBUG. (default: None) --quiet Be quiet! Sets logging level to WARNING. (default: None) ```