Testing Your Assistant
These docs are for version 1.x of Rasa Open Source.
User Guide
- Installation
- Tutorial: Rasa Basics
- Tutorial: Building Assistants
- Command Line Interface
- Architecture
- Messaging and Voice Channels
- Testing Your Assistant
- Setting up CI/CD
- Validate Data
- Configuring the HTTP API
- Deploying Your Rasa Assistant
- Cloud Storage
NLU
- About
- Using NLU Only
- Training Data Format
- Language Support
- Choosing a Pipeline
- Components
- Entity Extraction
Core
- About
- Stories
- Domains
- Responses
- Actions
- Reminders and External Events
- Policies
- Slots
- Forms
- Retrieval Actions
- Interactive Learning
- Fallback Actions
- Knowledge Base Actions
Conversation Design
API Reference
- Action Server
- HTTP API
- Jupyter Notebooks
- Agent
- Custom NLU Components
- Rasa SDK
- Events
- Tracker
- Tracker Stores
- Event Brokers
- Lock Stores
- Training Data Importers
- Featurization of Conversations
- TensorFlow Configuration
- Migration Guide
- Rasa Open Source Change Log
Migrate from (beta)
Reference
Versions
viewing: 1.10.2
Warning
This document is for an old version of Rasa. The latest version is 1.10.26.
Testing Your Assistant
End-to-End Testing
Rasa Open Source lets you test dialogues end-to-end by running through test conversations and making sure that both NLU and Core make correct predictions.
To do this, you need some stories in the end-to-end format, which includes both the NLU output and the original text. Here are some examples:
By default Rasa Open Source saves conversation tests to tests/conversation_tests.md. You can test your assistant against them by running:
$ rasa test
Note
Custom Actions are not executed as part of end-to-end tests. If your custom actions append any events to the tracker, this has to be reflected in your end-to-end tests (e.g. by adding slot events to your end-to-end story).
If you have any questions or problems, please share them with us in the dedicated testing section on our forum !
Evaluating an NLU Model
A standard technique in machine learning is to keep some data separate as a test set. You can split your NLU training data into train and test sets using:
rasa data split nlu
Comparing NLU Pipelines
By passing multiple pipeline configurations (or a folder containing them) to the CLI, Rasa will run a comparative examination between the pipelines.
$ rasa test nlu --config pretrained_embeddings_spacy.yml supervised_embeddings.yml
--nlu data/nlu.md --runs 3 --percentages 0 25 50 70 90
Intent Classification
The evaluation script will produce a report, confusion matrix, and confidence histogram for your model.
The report logs precision, recall and f1 measure for each intent and entity, as well as providing an overall average. You can save these reports as JSON files using the --report argument.
Entity Extraction
The CRFEntityExtractor is the only entity extractor which you train using your own data, and so is the only one that will be evaluated. If you use the spaCy or duckling pre-trained entity extractors, Rasa NLU will not include these in the evaluation.
Rasa NLU will report recall, precision, and f1 measure for each entity type that CRFEntityExtractor is trained to recognize.