## NLU-based assistants

This section refers to building NLU-based assistants. If you are working with [Conversational AI with Language Models (CALM)](/content/docs/calm/index.html), this content may not apply to you.

### Installation Requirements

To use NLU components, you need to install the `nlu` dependency group:

```bash
pip install 'rasa-pro[nlu]'
```

For more information about dependency groups, see our [Python Versions and Dependencies](/content/docs/reference/python-versions-and-dependencies/index.html) reference page.

## Tokenizers  [​](/content/docs/reference/config/components/nlu-components/#tokenizers/index.html)

Tokenizers split text into tokens.  
If you want to split intents into multiple labels, e.g. for predicting multiple intents or for modeling hierarchical intent structure, use the following flags with any tokenizer:

- `intent_tokenization_flag` indicates whether to tokenize intent labels or not. Set it to `True`, so that intent labels are tokenized.
- `intent_split_symbol` sets the delimiter string to split the intent labels, default is underscore (`_`).

### WhitespaceTokenizer  [​](/content/docs/reference/config/components/nlu-components/#whitespacetokenizer/index.html)

- **Short**

Tokenizer using whitespaces as a separator

- **Outputs**

`tokens` for user messages, responses (if present), and intents (if specified)

- **Requires**

Nothing

- **Description**

Creates a token for every whitespace separated character sequence.  
Any character not in: `a-zA-Z0-9_#@&` will be substituted with whitespace before splitting on whitespace if the character fulfills any of the following conditions:

- the character follows a whitespace: `