Choosing a Pipeline

In Rasa Open Source, incoming messages are processed by a sequence of components. These components are executed one after another in a so-called processing pipeline defined in your config.yml. Choosing an NLU pipeline allows you to customize your model and finetune it on your dataset.

How to Choose a Pipeline

The Short Answer

If your training data is in English, a good starting point is the following pipeline:

language: "en"

pipeline:
  - name: ConveRTTokenizer
  - name: ConveRTFeaturizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: "char_wb"
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 100
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 100

If your training data is not in English, start with the following pipeline:

language: "fr"  # your two-letter language code

pipeline:
  - name: WhitespaceTokenizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: "char_wb"
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 100
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 100

A Longer Answer

We recommend using following pipeline, if your training data is in English:

language: "en"

The pipeline contains the [ConveRTFeaturizer](https://legacy-docs-v1.rasa.com/1.10.8/nlu/components/#convertfeaturizer) that provides pre-trained word embeddings of the user utterance.
Pre-trained word embeddings are helpful as they already encode some kind of linguistic knowledge.

### Choosing the Right Components

There are components for entity extraction, for intent classification, response selection,
pre-processing, and others. You can learn more about any specific component on the [Components](https://legacy-docs-v1.rasa.com/1.10.8/nlu/components/#components) page.
If you want to add your own component, for example to run a spell-check or to do sentiment analysis, check out [Custom NLU Components](https://legacy-docs-v1.rasa.com/1.10.8/api/custom-nlu-components/#custom-nlu-components).

## Tokenization

For tokenization of English input, we recommend the [ConveRTTokenizer](https://legacy-docs-v1.rasa.com/1.10.8/nlu/components/#converttokenizer).
You can process other whitespace-tokenized (words are separated by spaces) languages
with the [WhitespaceTokenizer](https://legacy-docs-v1.rasa.com/1.10.8/nlu/components/#whitespacetokenizer).

## Entity Recognition / Intent Classification / Response Selectors

Depending on your data you may want to only perform intent classification, entity recognition or response selection.
Or you might want to combine multiple of those tasks.
We support several components for each of the tasks. All of them are listed in [Components](https://legacy-docs-v1.rasa.com/1.10.8/nlu/components/#components).
We recommend using [DIETClassifier](https://legacy-docs-v1.rasa.com/1.10.8/nlu/components/#diet-classifier) for intent classification and entity recognition
and [ResponseSelector](https://legacy-docs-v1.rasa.com/1.10.8/nlu/components/#response-selector) for response selection.

## Comparing Pipelines

Rasa gives you the tools to compare the performance of multiple pipelines on your data directly.
See [Comparing NLU Pipelines](https://legacy-docs-v1.rasa.com/1.10.8/user-guide/testing-your-assistant/#comparing-nlu-pipelines) for more information.

## Handling Class Imbalance

Classification algorithms often do not perform well if there is a large class imbalance,
for example if you have a lot of training data for some intents and very little training data for others.
To mitigate this problem, you can use a `balanced` batching strategy.

## Component Lifecycle

Each component processes an input and/or creates an output. The order of the components is determined by
the order they are listed in the `config.yml`; the output of a component can be used by any other component that
comes after it in the pipeline. Some components only produce information used by other components
in the pipeline. Other components produce `output` attributes that are returned after the processing has finished.

## Pipeline Templates (deprecated)

Pipeline templates are deprecated as of Rasa 1.8. For more information about a deprecated pipeline template,
check [How to Choose a Pipeline](https://legacy-docs-v1.rasa.com/1.10.8/nlu/choosing-a-pipeline/#how-to-choose-a-pipeline).