Evaluating Models
Evaluating Models
Evaluating an NLU Model
A standard technique in machine learning is to keep some data separate as a test set. You can split your NLU training data into train and test sets using:
rasa data split nlu
If you’ve done this, you can see how well your NLU model predicts the test cases using this command:
rasa test nlu -u train_test_split/test_data.md --model models/nlu-20180323-145833.tar.gz
If you don’t want to create a separate test set, you can still estimate how well your model generalises using cross-validation. To do this, add the flag --cross-validation:
rasa test nlu -u data/nlu.md --config config.yml --cross-validation
The full list of options for the script is:
usage: rasa test nlu [-h] [-v] [-vv] [--quiet] [-m MODEL] [-u NLU] [--out OUT]
[--successes] [--no-errors] [--histogram HISTOGRAM]
[--confmat CONFMAT] [-c CONFIG [CONFIG ...]]
[--cross-validation] [-f FOLDS] [-r RUNS]
[-p PERCENTAGES [PERCENTAGES ...]] [--no-plot]
optional arguments:
-h, --help show this help message and exit
-m MODEL, --model MODEL
Path to a trained Rasa model. If a directory is
specified, it will use the latest model in this
directory. (default: models)
-u NLU, --nlu NLU File or folder containing your NLU data. (default:
data)
--out OUT Output path for any files created during the
evaluation. (default: results)
--successes If set successful predictions (intent and entities)
will be written to a file. (default: False)
--no-errors If set incorrect predictions (intent and entities)
will NOT be written to a file. (default: False)
--histogram HISTOGRAM
Output path for the confidence histogram. (default:
hist.png)
--confmat CONFMAT Output path for the confusion matrix plot. (default:
confmat.png)
-c CONFIG [CONFIG ...], --config CONFIG [CONFIG ...]
Model configuration file. If a single file is passed
and cross validation mode is chosen, cross-validation
is performed, if multiple configs or a folder of
configs are passed, models will be trained and
compared directly. (default: None)
--no-plot Don't render evaluation plots (default: False)
Python Logging Options:
-v, --verbose Be verbose. Sets logging level to INFO. (default:
None)
-vv, --debug Print lots of debugging statements. Sets logging level
to DEBUG. (default: None)
--quiet Be quiet! Sets logging level to WARNING. (default:
None)
Cross Validation:
--cross-validation Switch on cross validation mode. Any provided model
will be ignored. (default: False)
-f FOLDS, --folds FOLDS
Number of cross validation folds (cross validation
only). (default: 5)
Comparison Mode:
-r RUNS, --runs RUNS Number of comparison runs to make. (default: 3)
-p PERCENTAGES [PERCENTAGES ...], --percentages PERCENTAGES [PERCENTAGES ...]
Percentages of training data to exclude during
comparison. (default: [0, 25, 50, 75])
Comparing NLU Pipelines
By passing multiple pipeline configurations (or a folder containing them) to the CLI, Rasa will run a comparative examination between the pipelines.
$ rasa test nlu --config pretrained_embeddings_spacy.yml supervised_embeddings.yml
--nlu data/nlu.md --runs 3 --percentages 0 25 50 70 90
The command in the example above will create a train/test split from your data, then train each pipeline multiple times with 0, 25, 50, 70 and 90% of your intent data excluded from the training set. The models are then evaluated on the test set and the f1-score for each exclusion percentage is recorded. This process runs three times (i.e. with 3 test sets in total) and then a graph is plotted using the means and standard deviations of the f1-scores.
Intent Classification
The evaluation script will produce a report, confusion matrix, and confidence histogram for your model.
The report logs precision, recall and f1 measure for
each intent and entity, as well as providing an overall average.
You can save these reports as JSON files using the --report argument.
The confusion matrix shows you which
intents are mistaken for others; any samples which have been
incorrectly predicted are logged and saved to a file
called errors.json for easier debugging.
The histogram that the script produces allows you to visualise the confidence distribution for all predictions, with the volume of correct and incorrect predictions being displayed by blue and red bars respectively. Improving the quality of your training data will move the blue histogram bars to the right and the red histogram bars to the left of the plot.
Entity Extraction
The CRFEntityExtractor is the only entity extractor which you train using your own data,
and so is the only one that will be evaluated. If you use the spaCy or duckling
pre-trained entity extractors, Rasa NLU will not include these in the evaluation.
Rasa NLU will report recall, precision, and f1 measure for each entity type that
CRFEntityExtractor is trained to recognize.
Entity Scoring
To evaluate entity extraction we apply a simple tag-based approach. We don’t consider BILOU tags, but only the
entity type tags on a per token basis. For location entity like “near Alexanderplatz” we
expect the labels LOC LOC instead of the BILOU-based B-LOC L-LOC. Our approach is more lenient
when it comes to evaluation, as it rewards partial extraction and does not punish the splitting of entities.
Evaluating a Core Model
You can evaluate your trained model on a set of test stories by using the evaluate script:
rasa test core --stories test_stories.md --out results
This will print the failed stories to results/failed_stories.md.
We count any story as failed if at least one of the actions
was predicted incorrectly.
In addition, this will save a confusion matrix to a file called
results/story_confmat.pdf. For each action in your domain, the confusion
matrix shows how often the action was correctly predicted and how often an
incorrect action was predicted instead.
The full list of options for the script is:
usage: rasa test core [-h] [-v] [-vv] [--quiet] [-m MODEL [MODEL ...]]
[-s STORIES] [--max-stories MAX_STORIES] [--out OUT]
[--e2e] [--endpoints ENDPOINTS]
[--fail-on-prediction-errors] [--url URL]
[--evaluate-model-directory] [--no-plot]
optional arguments:
-h, --help show this help message and exit
-m MODEL [MODEL ...], --model MODEL [MODEL ...]
Path to a pre-trained model. If it is a 'tar.gz' file
that model file will be used. If it is a directory,
the latest model in that directory will be used
(exception: '--evaluate-model-directory' flag is set).
If multiple 'tar.gz' files are provided, all those
models will be compared. (default: [None])
-s STORIES, --stories STORIES
File or folder containing your test stories. (default:
data)
--max-stories MAX_STORIES
Maximum number of stories to test on. (default: None)
--out OUT Output path for any files created during the
evaluation. (default: results)
--e2e, --end-to-end Run an end-to-end evaluation for combined action and
intent prediction. Requires a story file in end-to-end
format. (default: False)
--endpoints ENDPOINTS
Configuration file for the connectors as a yml file.
(default: None)
--fail-on-prediction-errors
If a prediction error is encountered, an exception is
thrown. This can be used to validate stories during
tests, e.g. on travis. (default: False)
--url URL If supplied, downloads a story file from a URL and
trains on it. Fetches the data by sending a GET
request to the supplied URL. (default: None)
--evaluate-model-directory
Should be set to evaluate models trained via 'rasa
train core --config <config-1> <config-2>'. All models
in the provided directory are evaluated and compared
against each other. (default: False)
--no-plot Don't render evaluation plots (default: False)
Python Logging Options:
-v, --verbose Be verbose. Sets logging level to INFO. (default:
None)
-vv, --debug Print lots of debugging statements. Sets logging level
to DEBUG. (default: None)
--quiet Be quiet! Sets logging level to WARNING. (default:
None)