# Policies

## Configuring Policies
The `rasa.core.policies.Policy` class decides which action to take at every step in the conversation.

There are different policies to choose from, and you can include multiple policies in a single [`rasa.core.agent.Agent`](https://legacy-docs-v1.rasa.com/1.9.7/api/agent/#rasa.core.agent.Agent).

**Note**  
Per default a maximum of 10 next actions can be predicted by the agent after every user message. To update this value you can set the environment variable `MAX_NUMBER_OF_PREDICTIONS` to the desired number of maximum predictions.

Your project’s `config.yml` file takes a `policies` key which you can use to customize the policies your assistant uses. In the example below, the last two lines show how to use a custom policy class and pass arguments to it.

```yaml
policies:
  - name: "KerasPolicy"
    featurizer:
    - name: MaxHistoryTrackerFeaturizer
      max_history: 5
      state_featurizer:
        - name: BinarySingleStateFeaturizer
  - name: "MemoizationPolicy"
    max_history: 5
  - name: "FallbackPolicy"
    nlu_threshold: 0.4
    core_threshold: 0.3
    fallback_action_name: "my_fallback_action"
  - name: "path.to.your.policy.class"
    arg1: "..."
```

### Max History
One important hyperparameter for Rasa Core policies is the `max_history`. This controls how much dialogue history the model looks at to decide which action to take next.

You can set the `max_history` by passing it to your policy’s `Featurizer` in the policy configuration yaml file.

**Note**  
Only the `MaxHistoryTrackerFeaturizer` uses a max history, whereas the `FullDialogueTrackerFeaturizer` always looks at the full conversation history. See [Featurization of Conversations](https://legacy-docs-v1.rasa.com/1.9.7/api/core-featurization/#featurization-conversations) for details.

As an example, let’s say you have an `out_of_scope` intent which describes off-topic user messages. If your bot sees this intent multiple times in a row, you might want to tell the user what you can help them with. So your story might look like this:

```yaml
* out_of_scope
   - utter_default
* out_of_scope
   - utter_default
* out_of_scope
   - utter_help_message
```

For Rasa Core to learn this pattern, the `max_history` has to be at least 4.

If you increase your `max_history`, your model will become bigger and training will take longer. If you have some information that should affect the dialogue very far into the future, you should store it as a slot. Slot information is always available for every featurizer.

### Data Augmentation
When you train a model, by default Rasa Core will create longer stories by randomly gluing together the ones in your stories files. This is because if you have stories like:

```yaml
# thanks
* thankyou
   - utter_youarewelcome

# bye
* goodbye
   - utter_goodbye
```

You actually want to teach your policy to **ignore** the dialogue history when it isn’t relevant and just respond with the same action no matter what happened before.

You can alter this behaviour with the `--augmentation` flag. Which allows you to set the `augmentation_factor`. The `augmentation_factor` determines how many augmented stories are subsampled during training. The augmented stories are subsampled before training since their number can quickly become very large, and we want to limit it. The number of sampled stories is `augmentation_factor` x10. By default augmentation is set to 20, resulting in a maximum of 200 augmented stories.

`--augmentation 0` disables all augmentation behavior. The memoization based policies are not affected by augmentation (independent of the `augmentation_factor`) and will automatically ignore all augmented stories.

## Action Selection
At every turn, each policy defined in your configuration will predict a next action with a certain confidence level. For more information about how each policy makes its decision, read into the policy’s description below. The bot’s next action is then decided by the policy that predicts with the highest confidence.

In the case that two policies predict with equal confidence (for example, the Memoization and Mapping Policies always predict with confidence of either 0 or 1), the priority of the policies is considered. Rasa policies have default priorities that are set to ensure the expected outcome in the case of a tie. They look like this, where higher numbers have higher priority:

> 5. `FormPolicy`  
> 4. `FallbackPolicy` and `TwoStageFallbackPolicy`  
> 3. `MemoizationPolicy` and `AugmentedMemoizationPolicy`  
> 2. `MappingPolicy`  
> 1. `TEDPolicy`, `EmbeddingPolicy`, `KerasPolicy`, and `SklearnPolicy`

This priority hierarchy ensures that, for example, if there is an intent with a mapped action, but the NLU confidence is not above the `nlu_threshold`, the bot will still fall back. In general, it is not recommended to have more than one policy per priority level, and some policies on the same priority level, such as the two fallback policies, strictly cannot be used in tandem.

If you create your own policy, use these priorities as a guide for figuring out the priority of your policy. If your policy is a machine learning policy, it should most likely have priority 1, the same as the Rasa machine learning policies.

**Warning**  
All policy priorities are configurable via the `priority:` parameter in the configuration, but we **do not recommend** changing them outside of specific cases such as custom policies. Doing so can lead to unexpected and undesired bot behavior.

## Keras Policy
The `KerasPolicy` uses a neural network implemented in [Keras](http://keras.io/) to select the next action. The default architecture is based on an LSTM, but you can override the `KerasPolicy.model_architecture` method to implement your own architecture.

```python
def model_architecture(self, input_shape: Tuple[int, int], output_shape: Tuple[int, Optional[int]]) -> tf.keras.models.Sequential:
    """Build a keras model and return a compiled model."""

from tensorflow.keras.models import Sequential
    from tensorflow.keras.layers import (
        Masking,
        LSTM,
        Dense,
        TimeDistributed,
        Activation,
    )

# Build Model
    model = Sequential()

# the shape of the y vector of the labels,
    # determines which output from rnn will be used
    # to calculate the loss
    if len(output_shape) == 1:
        # y is (num examples, num features) so
        # only the last output from the rnn is used to
        # calculate the loss
        model.add(Masking(mask_value=-1, input_shape=input_shape))
        model.add(LSTM(self.rnn_size, dropout=0.2))
        model.add(Dense(input_dim=self.rnn_size, units=output_shape[-1]))
    elif len(output_shape) == 2:
        # y is (num examples, max_dialogue_len, num features) so
        # all the outputs from the rnn are used to
        # calculate the loss, therefore a sequence is returned and
        # time distributed layer is used

# the first value in input_shape is max dialogue_len,
        # it is set to None, to allow dynamic_rnn creation
        # during prediction
        model.add(Masking(mask_value=-1, input_shape=(None, input_shape[1])))
        model.add(LSTM(self.rnn_size, return_sequences=True, dropout=0.2))
        model.add(TimeDistributed(Dense(units=output_shape[-1])))
    else:
        raise ValueError(
            "Cannot construct the model because"
            "length of output_shape = {} "
            "should be 1 or 2."
            "".format(len(output_shape))
        )

model.add(Activation("softmax"))

model.compile(
        loss="categorical_crossentropy", optimizer="rmsprop", metrics=["accuracy"]
    )

if common_utils.obtain_verbosity() > 0:
        model.summary()

return model
```

and the training is run here:

```python
def train(self, training_trackers: List[DialogueStateTracker], domain: Domain, **kwargs: Any) -> None:
    np.random.seed(self.random_seed)
    tf.random.set_seed(self.random_seed)

training_data = self.featurize_for_training(training_trackers, domain, **kwargs)
    # noinspection PyPep8Naming
    shuffled_X, shuffled_y = training_data.shuffled_X_y()

if self.model is None:
        self.model = self.model_architecture(
            shuffled_X.shape[1:], shuffled_y.shape[1:]
        )

logger.debug(
        f"Fitting model with {training_data.num_examples()} total samples and a "
        f"validation split of {self.validation_split}."
    )

# filter out kwargs that cannot be passed to fit
    self._train_params = self._get_valid_params(
        self.model.fit, **self._train_params
    )

self.model.fit(
        shuffled_X,
        shuffled_y,
        epochs=self.epochs,
        batch_size=self.batch_size,
        shuffle=False,
        verbose=common_utils.obtain_verbosity(),
        **self._train_params,
    )
    self.current_epoch = self.epochs

logger.debug("Done fitting Keras Policy model.")
```

You can implement the model of your choice by overriding these methods, or initialize `KerasPolicy` with pre-defined `keras model`.

In order to get reproducible training results for the same inputs you can set the `random_seed` attribute of the `KerasPolicy` to any integer.

## Embedding Policy
> **Warning**  
> `EmbeddingPolicy` was renamed to `TEDPolicy`. Please use [TED Policy](https://legacy-docs-v1.rasa.com/1.9.7/core/policies/#ted-policy) instead of `EmbeddingPolicy` in your policy configuration. The functionality of the policy stayed the same.

## TED Policy
The Transformer Embedding Dialogue (TED) Policy is described in [our paper](https://arxiv.org/abs/1910.00486).

This policy has a pre-defined architecture, which comprises the following steps:

> - concatenate user input (user intent and entities), previous system actions, slots and active forms for each time
> step into an input vector to pre-transformer embedding layer;
> - feed it to transformer;
> - apply a dense layer to the output of the transformer to get embeddings of a dialogue for each time step;
> - apply a dense layer to create embeddings for system actions for each time step;
> - calculate the similarity between the dialogue embedding and embedded system actions. This step is based on the [StarSpace](https://arxiv.org/abs/1709.03856) idea.

It is recommended to use `state_featurizer=LabelTokenizerSingleStateFeaturizer(...)` (see [Featurization of Conversations](https://legacy-docs-v1.rasa.com/1.9.7/api/core-featurization/#featurization-conversations) for details).

**Configuration:**

> Configuration parameters can be passed as parameters to the `TEDPolicy` within the configuration file.  
> If you want to adapt your model, start by modifying the following parameters:
> 
> - `epochs`:  
> This parameter sets the number of times the algorithm will see the training data (default: `1`).  
> One `epoch` is equals to one forward pass and one backward pass of all the training examples.  
> Sometimes the model needs more epochs to properly learn.  
> Sometimes more epochs don’t influence the performance.  
> The lower the number of epochs the faster the model is trained.
> 
> - `hidden_layers_sizes`:  
> This parameter allows you to define the number of feed forward layers and their output dimensions for dialogues and intents (default: `dialogue: [], label: []`).  
> Every entry in the list corresponds to a feed forward layer.  
> For example, if you set `dialogue: [256, 128]`, we will add two feed forward layers in front of the transformer. The vectors of the input tokens (coming from the dialogue) will be passed on to those layers. The first layer will have an output dimension of 256 and the second layer will have an output dimension of 128. If an empty list is used (default behaviour), no feed forward layer will be added.
> 
> - `number_of_transformer_layers`:  
> This parameter sets the number of transformer layers to use (default: `1`).  
> The number of transformer layers corresponds to the transformer blocks to use for the model.
> 
> - `transformer_size`:  
> This parameter sets the number of units in the transformer (default: `128`).  
> The vectors coming out of the transformers will have the given `transformer_size`.
> 
> - `weight_sparsity`:  
> This parameter defines the fraction of kernel weights that are set to 0 for all feed forward layers in the model (default: `0.8`). The value should be between 0 and 1. If you set `weight_sparsity` to 0, no kernel weights will be set to 0, the layer acts as a standard feed forward layer. You should not set `weight_sparsity` to 1 as this would result in all kernel weights being 0, i.e. the model is not able to learn.
> 
> **Warning**  
> Pass an appropriate number, for example 50, of `epochs` to the `TEDPolicy`, otherwise the policy will be trained only for `1` epoch.
> 
> **Warning**  
> Default `max_history` for this policy is `None` which means it’ll use the `FullDialogueTrackerFeaturizer`. We recommend to set `max_history` to some finite value in order to use `MaxHistoryTrackerFeaturizer` for **faster training**. See [Featurization of Conversations](https://legacy-docs-v1.rasa.com/1.9.7/api/core-featurization/#featurization-conversations) for details. We recommend to increase `batch_size` for `MaxHistoryTrackerFeaturizer` (e.g. "batch_size": [32, 64])

| Parameter | Default Value | Description |
|-----------|---------------|-------------|
| hidden_layers_sizes | dialogue: [] | Hidden layer sizes for layers before the embedding layers |
| transformer_size | 128 | Number of units in transformer. |
| number_of_transformer_layers | 1 | Number of transformer layers. |
| random_seed | None | Set random seed to any 'int' to get reproducible results. |