NLU Components | Rasa Documentation

NLU-based assistants

This section refers to building NLU-based assistants. If you are working with Conversational AI with Language Models (CALM), this content may not apply to you.

Installation Requirements

To use NLU components, you need to install the nlu dependency group:

pip install 'rasa-pro[nlu]'

For more information about dependency groups, see our Python Versions and Dependencies reference page.

Tokenizers ​

Tokenizers split text into tokens.
If you want to split intents into multiple labels, e.g. for predicting multiple intents or for modeling hierarchical intent structure, use the following flags with any tokenizer:

• intent_tokenization_flag indicates whether to tokenize intent labels or not. Set it to True, so that intent labels are tokenized.
• intent_split_symbol sets the delimiter string to split the intent labels, default is underscore (_).

WhitespaceTokenizer ​

• Short

Tokenizer using whitespaces as a separator

• Outputs

tokens for user messages, responses (if present), and intents (if specified)

• Requires

Nothing

• Description

Creates a token for every whitespace separated character sequence.
Any character not in: a-zA-Z0-9_#@& will be substituted with whitespace before splitting on whitespace if the character fulfills any of the following conditions:

• the character follows a whitespace: `