Annotate NLU Examples

Rasa X version 0.33.0 or higher is only compatible with Rasa Open Source 2.x. If you are on Rasa Open Source 1.x, please check the compatibility matrix for a compatible version.

Improving your NLU model based on messages from real conversations is a crucial piece of building an assistant that can handle real users. Coming up with examples yourself can help you bootstrap at the beginning of a project, but when you’re going into production, less than 10% of your data should be synthetic.

In preparing for production, you should have already shared your bot with guest testers or other internal users. Looking through the messages from guest testers is key to building out your own examples to correctly respond to users who don’t know the inner workings of your assistant and what it can do.

Once your assistant is in production, annotating incoming messages can have an even bigger impact on your assistant’s success, as the messages are from actual end users. Annotating these users’ messages is a great way to understand your bot’s successes and failures and add data that reflects what real users say to your assistant.

Improving NLU performance using Rasa X

Rasa X provides you with insights on how to improve your NLU training data. It looks at the results of a cross-validation, the messages in your NLU inbox, and other metrics to calculate these insights. Rasa X presents the insights found together with suggestions on how to address them in the insights screen:

The insights can be used to help you practice Conversation-Driven Development with Rasa X. Use insights to:

• Decide how to best invest your time when annotating NLU messages
• Understand the impact of changes to your NLU training data

Changing the Schedule

Rasa X uses the results of a cross-validation, among other things, to identify problems in your training data. Depending on the amount of NLU training data you have, cross-validation can take several hours. For a rough estimation of how long cross-validation will take, you can take the average training time of your model and multiply it by the used fold for cross-validation. By default, the used fold is 4, which means Rasa X will train 4 models on 80 % of the training data and evaluate it against the remaining 20 %.

Performance Implications

Rasa X will use rasa-worker to run the cross-validation so that the computationally intense process will not impact production server performance. Note that ongoing cross-validations might still affect model trainings which were triggered via the Rasa X UI or API.

Run the insight calculation at night so that you have the latest results available the next day. By default, Rasa X will run the evaluation every day at 2am. If you want to deactivate intent insights or use a different time for the insight calculation, you can adjust the schedule in the insights screen:

You can disable the scheduled runs by setting schedule to null via this endpoint.

Insight Calculators

Configuration

The insight calculation uses the following calculators to calculate the insights. You can update the used calculators and their parameters via this endpoint.

For example, to set the MinimumExampleInsightCalculator to create insights for every intent which has less than 100 examples, send the following request:

curl --request PUT \
 --url <rasa x url>/api/insights/config \
 --header 'Authorization: Bearer <your access token>' \
 --header 'Content-Type: application/json' \
 --data '{
"schedule": "0 2 * * *",
"cross_validation_folds": <number of cross-validation folds>,
"calculator_configuration": {
"MinimumExampleInsightCalculator": {"required_number_of_examples_per_intent": 100},
"ConfusionInsightCalculator": {},
"ClassBiasInsightCalculator": {},
"WrongAnnotationInsightCalculator": {},
"NLUInboxConfidenceInsightCalculator": {}
}
}'

To reset the calculator_configuration to the default settings, set calculator_configuration to null:

curl --request PUT \
 --url <rasa x url>/api/insights/config \
 --header 'Authorization: Bearer <your access token>' \
 --header 'Content-Type: application/json' \
 --data '{
"schedule": "0 2 * * *",
"cross_validation_folds": <number of cross-validation folds>,
"calculator_configuration": null
}'

Note

Rasa X currently only calculates insights for intents and retrieval intents. Entities are not considered.

Available Calculators

MinimumExampleInsightCalculator

• Name: MinimumExampleInsightCalculator
• Description: Makes sure that each intent has at least the minimum required number of examples.
• Parameters:
  • required_number_of_examples_per_intent (default: 20): Each intent has to have at least this number of examples. Otherwise, a new insight is created.

ConfusionInsightCalculator

• Name: ConfusionInsightCalculator
• Description: Checks if an intent is confused often. This can for example happen if the examples of two intents are too similar.
• Parameters:
  • top_x_percent_threshold (default: 1.0): Only return new insights for intents with the top_x_percent_threshold percent confusions.
  • recall_threshold (default: 0.85): An insight is only created if the recall for the intent is lower than this threshold.

ClassBiasInsightCalculator

• Name: ClassBiasInsightCalculator
• Description: Checks for intents predicted disproportionately often. This happens when there is a bias for this intent.
• Parameters:
  • maximal_precision (default: 0.4): Only create insight if intent has precision lower than this given threshold.
  • minimum_recall (default: 0.8): Only create insight if recall for intent is greater than recall threshold.

WrongAnnotationInsightCalculator

• Name: WrongAnnotationInsightCalculator
• Description: Checks for intents which are diluted by wrongly annotated training data.
• Parameters:
  • confidence_threshold (default: 0.9): Only create insights if an intent is wrongly predicted with high confidence.
  • top_x_percent_threshold (default: 0.1): Only return new insights for intents with this percent of wrong annotations.

NLUInboxConfidenceInsightCalculator

• Name: NLUInboxConfidenceInsightCalculator
• Description: Searches the NLU Inbox for user message predictions with low confidence.
• Parameters:
  • lower_confidence_threshold (default: 0.3): Only look for messages with confidences greater than or equal to threshold.
  • upper_confidence_threshold (default: 0.8): Only look for messages with confidence scores smaller than or equal to threshold.
  • only_classifications_by_active_model (default: True): Only consider predictions from the currently active model.

Configuring the Cross-Validation Folds

By default, Rasa X will use a 4-fold cross-validation to analyze the quality of your training data. You can configure a different threshold via this endpoint.

Accessing the Cross-Validation Results

Rasa X uses the cross-validation results among other data points to calculate insights for your NLU data. To access the raw cross-validation results, you can use this endpoint to retrieve the result.

Intent Warnings and Suggestions

The NLU insights screen also shows warnings and suggestions on how your training data can be improved. The warnings come from insight calculators. The "Suggestions" column always contains a link to the specific intent on NLU Inbox screen. Use this link to annotate your training data, and then re-run the evaluation process to see the updated list of suggestions.

Downloading the NLU insights report

You can download the JSON report containing the results of the latest NLU insights evaluation. This report can be used to compare performance over time and see more detailed results. To get it, click on the "Download full evaluation results" link in the "Run evaluations" popup:

You can also download intent insights reports manually or automatically using the API.

Annotating in the NLU Inbox

The NLU Inbox is a collection of all of the messages users have sent that aren’t already part of your training data. Whenever you get new messages, a badge in the sidebar will indicate that you have new data to process. Processing this inbox is the fastest way to improve your assistant’s NLU model.

Annotating User Messages

As messages from users come in, they will appear in the NLU inbox alongside the predicted intent and confidence of the prediction. If the predicted intent is incorrect, select a different intent from the drop-down menu. Save the new prediction with the Mark Correct button.

Annotating in Bulk

Once you are comfortable annotating NLU data, you can speed up your annotation workflow by making use of some helpful features.