Annotate NLU Examples

Rasa X version 0.33.0 or higher is only compatible with Rasa Open Source 2.x. If you are on Rasa Open Source 1.x, please check the compatibility matrix for a compatible version.

Improving your NLU model based on messages from real conversations is a crucial piece of building an assistant that can handle real users. Coming up with examples yourself can help you bootstrap at the beginning of a project, but when you’re going into production, less than 10% of your data should be synthetic.

In preparing for production, you should have already shared your bot with guest testers or other internal users. Looking through the messages from guest testers is key to building out your own examples to correctly respond to users who don’t know the inner workings of your assistant and what it can do.

Once your assistant is in production, annotating incoming messages can have an even bigger impact on your assistant’s success, as the messages are from actual end users. Annotating these users’ messages is a great way to understand your bot’s successes and failures and add data that reflects what real users say to your assistant.

Improving NLU performance using Rasa X #

Rasa X provides you with insights on how to improve your NLU training data. It looks at the results of a cross-validation, the messages in your NLU inbox, and other metrics to calculate these insights. Rasa X presents the insights found together with suggestions on how to address them in the insights screen:

The insights can be used to help you practice Conversation-Driven Development with Rasa X. You can use insights to:

Changing the Schedule #

Running Rasa X locally?

If you have installed Rasa X locally, Rasa X will run the insights calculation on startup instead of periodically. Please see this section for more information.

Rasa X uses the results of a cross-validation, among other things, to identify problems in your training data. Depending on the amount of NLU training data you have, cross-validation can take several hours. For a rough estimation of how long cross validation will take, you can take the average training time of your model and multiply it by the used fold for cross-validation. By default, the used fold is 4, which means Rasa X will train 4 models on 80 % of the training data and evaluate it against the remaining 20 %.

Performance Implications

Rasa X will use rasa-worker to run the cross-validation so that the computationally intense process will not impact production server performance. Note that ongoing cross-validations might still affect model trainings which were triggered via the Rasa X UI or API.

Run the insight calculation at night so that you have the latest results available the next day. By default, Rasa X will run the evaluation every day at 2am. If you want to deactivate intent insights or use a different time for the insight calculation, you can adjust the schedule in the insights screen:

You can disable the scheduled runs by setting schedule to null via this endpoint.

Insight Calculators #

Configuration #

The insight calculation uses the following calculators to calculate the insights. You can update the used calculators and their parameters via this endpoint.

For example, to set the MinimumExampleInsightCalculator to create insights for every intent which has less than 100 examples, send the following request:

curl --request PUT \
 --url <rasa x url>/api/insights/config \
 --header 'Authorization: Bearer <your access token>'\
 --header 'Content-Type: application/json'\
 --data '{
"schedule": null,
"cross_validation_folds": <number of cross-validation folds>,
"calculator_configuration": null
}'

To reset the calculator_configuration to the default settings, set calculator_configuration to null:

curl --request PUT \
 --url <rasa x url>/api/insights/config \
 --header 'Authorization: Bearer <your access token>'\
 --header 'Content-Type: application/json'\
 --data '{
"schedule": "0 2 * * *",
"cross_validation_folds": <number of cross-validation folds>,
"calculator_configuration": null
}'

Available Calculators #

MinimumExampleInsightCalculator #
ConfusionInsightCalculator #
ClassBiasInsightCalculator #
WrongAnnotationInsightCalculator #
NLUInboxConfidenceInsightCalculator #

Configuring the Cross-Validation Folds #

By default, Rasa X will use a 4-fold cross-validation to analyze the quality of your training data. You can configure a different threshold via this endpoint.

Accessing the Cross-Validation Results #

Rasa X uses the cross-validation results among other data points to calculate insights for your NLU data. If you want to access the raw cross-validation results with all details, you can use this endpoint to retrieve the result.

Intent Warnings and Suggestions #

The NLU insights screen also shows you warnings and suggestions of how your training data can be improved. The warnings come from insight calculators which means the data you see there depends on the configuration.

Downloading the NLU insights report #

You can download the JSON report containing the results of the latest NLU insights evaluation. This report can be used to compare performance over time and to see more detailed results.

Annotating in the NLU Inbox #

The NLU Inbox is a collection of all of the messages users have sent that aren’t already part of your training data. Whenever you get new messages, a badge in the sidebar will indicate that you have new data to process. Processing this inbox is the fastest way to improve your assistant’s NLU model.

Annotating User Messages #

As messages from users come in, they will appear in the NLU inbox alongside the predicted intent and confidence of the prediction. Hovering over any entry brings up the Mark Correct and Delete icons. If a message’s prediction is correct, you can save it to your training data directly by selecting Mark Correct.

If the predicted intent is incorrect, select a different intent from the drop-down menu. If necessary, you can create a new intent by scrolling to Create new intent in the drop-down. Save the new prediction with the Mark Correct button.

If for some reason you do not want a user’s message saved to your training data (for example, you may not want to save personally identifiable information (PII) into your dataset), use the Delete button to remove the entry from the inbox.

Annotating in Bulk #

Once you are comfortable annotating NLU data by correcting predictions and marking correct ones, you can speed up your annotation workflow by making use of some helpful features:

Keyboard Shortcuts #

The NLU Inbox is supported by keyboard shortcuts which help you save and delete messages with less clicking. To select the initial message, push the j key. You can then use the following keyboard shortcuts to interact with the NLU Inbox:

Bulk Actions #

You can also use the select boxes next to each message to bulk annotate messages. To do so, select multiple messages that you would like to mark correct or delete. A header will appear that shows you how many messages you have selected and has buttons for bulk actions. Click either of the purple buttons in the header bar to mark all of your selections as correct or delete them.

Annotating in Conversations #

While you’re reviewing conversations or fixing problems, you may notice an incorrect NLU prediction in a conversation. Instead of finding and fixing that prediction in the NLU Inbox, you can annotate it directly from the conversations page.

Select the incorrectly predicted message to bring up the annotation pane on the right-hand side. You’ll see a drop-down menu similar to the one in the NLU Inbox where the predicted intent is selected. Choose the correct intent from the drop-down and push the Save button to correct the intent.