Choosing a Pipeline

In Rasa Open Source, incoming messages are processed by a sequence of components. These components are executed one after another in a so-called processing pipeline defined in your config.yml. Choosing an NLU pipeline allows you to customize your model and finetune it on your dataset.

How to Choose a Pipeline

The Short Answer

If your training data is in English, a good starting point is the following pipeline:

language: "en"

pipeline:
  - name: ConveRTTokenizer
  - name: ConveRTFeaturizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: "char_wb"
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 100
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 100

If your training data is not in English, start with the following pipeline:

language: "fr"  # your two-letter language code

pipeline:
  - name: WhitespaceTokenizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: "char_wb"
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 100
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 100

A Longer Answer

We recommend using following pipeline, if your training data is in English:

language: "en"

The pipeline contains the [ConveRTFeaturizer](https://legacy-docs-v1.rasa.com/1.10.17/nlu/components/#convertfeaturizer) that provides pre-trained word embeddings of the user utterance.
Pre-trained word embeddings are helpful as they already encode some kind of linguistic knowledge.
For example, if you have a sentence like “I want to buy apples” in your training data, and Rasa is asked to predict
 the intent for “get pears”, your model already knows that the words “apples” and “pears” are very similar.
This is especially useful if you don’t have enough training data.

> It is helpful to know that the component lifecycle ensures each component processes an input and creates an output, which allows for logical component interaction within the pipeline.

#### Multi-Intent Classification

You can use Rasa Open Source components to split intents into multiple labels. For example, you can predict
multiple intents (`thank+goodbye`) or model hierarchical intent structure (`feedback+positive` being more similar
to `feedback+negative` than `chitchat`). To do this, you need to use the [DIETClassifier](https://legacy-docs-v1.rasa.com/1.10.17/nlu/components/#diet-classifier) in your pipeline.

### Comparing Pipelines

Rasa gives you the tools to compare the performance of multiple pipelines on your data directly.

### Handling Class Imbalance

Classification algorithms often do not perform well if there is a large class imbalance,
for example if you have a lot of training data for some intents and very little training data for others.
To mitigate this problem, you can use a `balanced` batching strategy.