LLM Configuration for Rasa Pro ≥ 3.11 | Rasa Documentation

LLM Configuration for Rasa Pro 3.10 and below

For Rasa Pro versions 3.10 and below, refer to the LLM Configuration for <=3.10 page.

Overview

This page applies to the following components which use LLMs:

All the above components can be configured to change:

Starting with version Rasa Pro 3.10, CALM uses LiteLLM under the hood to integrate with different LLM providers. Hence, all LiteLLM's integrated providers are supported with CALM as well. We explicitly mention the settings required for the most frequently used ones in the sections below.

Declaring LLM deployments

LLM deployments are always declared in groups comprising of 1 or more deployments. The below sections explain how to declare these groups.

Model Groups

Model groups allow you to define multiple models under a single ID which can be accessed by any component. Model groups are defined in the endpoints.yml file under the model_groups key, separating model definitions from individual component configurations. For example:

   model_groups:
     - id: openai-direct  # Unique identifier for the model group
       models:
         ...

Defining a single model group

   model_groups:
     - id: openai-direct  # Unique identifier for the model group
       models:
         - provider: openai
           model: gpt-5.1-2025-11-13

Required Parameters

There are certain required parameters for each model group:

  1. provider - Unique identifier of the LLM provider to be used.
  2. model - Specifies the name of the model identifier available from the LLM provider's documentation, for e.g. gpt-5.1-2025-11-13.

Optional Parameters

Each model group also accepts inference time parameters like temperature, etc which are optional but can be useful in extracting the best performance out of the model being used. Please refer to the official LiteLLM documentation for a list of such parameters supported.

When configuring a particular provider, there are a few provider specific settings which are explained under each provider's individual sub-section below.

Reasoning models and reasoning_effort

For models that support it, reasoning_effort is an optional inference-time parameter that controls how much internal reasoning the model applies before returning an answer. Higher values typically increase latency and cost; lower values favor faster responses. Allowed values depend on the provider and model (for example none, minimal, low, and high for many OpenAI reasoning-capable models). See provider documentation for more details, e.g. OpenAI Developers: Reasoning models or Claude API Docs: Effort.

You can set reasoning_effort alongside other model settings in each entry under models in endpoints.yml, for example:

   model_groups:
     - id: openai-reasoning
       models:
         - provider: openai
           model: gpt-5-mini-2025-08-07
           reasoning_effort: "minimal"

By default, Rasa tries to apply the lowest appropriate reasoning_effort for the resolved model name (using provider metadata where available). To override this behavior, set reasoning_effort explicitly in your model configuration. Models that do not support this parameter do not receive a default.

Referencing environment variables in the model configuration

To reference environment variables in the model configuration, you can use the ${} syntax. For example:

   model_groups:
     - id: openai-direct
       models:
         - provider: openai
           model: gpt-5.1-2025-11-13
           api_key: ${MY_OPENAI_API_KEY}

In the above example, the api_key parameter references the environment variable MY_OPENAI_API_KEY.

LLM API health check

The model config and the connection to the LLM provider can be validated by setting the LLM_API_HEALTH_CHECK environment variable to true.

export LLM_API_HEALTH_CHECK=true

By default, the variable is set to False. When set to True, all LLM deployments defined will be checked for availability by making a test API request.

Using a model group in a component

Components using an LLM can be configured to use any of the declared model groups in the component's configuration. To use a model group, you can specify the model_group key under the llm key. For example:

   recipe: default.v1
   language: en
   pipeline:
   - name: CompactLLMCommandGenerator
     llm:
        model_group: openai-direct

Defining multiple model groups

   model_groups:
     - id: openai-gpt-5-1
       models:
         - provider: openai
           model: gpt-5.1-2025-11-13
     - id: openai-gpt-5-mini
       models:
         - provider: openai
           model: gpt-5-mini-2025-08-07

The examples above illustrate how to define a model groups consisting of a single model deployment. In order to handle a larger volume of conversations, it is recommended to include multiple model deployments within a model group. To do so you can add additional deployments to the models list as explained in the Multi-LLM routing page.

Using different model groups in different components

   recipe: default.v1
   pipeline:
   - name: LLMBasedRouter
     calm_entry:
       sticky: ...
     nlu_entry:
       sticky: ...
     non_sticky: ...
     llm:
        model_group: openai-gpt-5-mini
   - name: CompactLLMCommandGenerator
     llm:
        model_group: openai-gpt-5-1

Multiple components can rely on the same model group and a single component can use multiple models defined in a model group, via the LLM router.

Chat completion models

Default Provider

CALM is LLM agnostic and can be configured with different LLMs, but OpenAI is the default model provider. Majority of our experiments have been with models available on OpenAI or OpenAI Azure service. The performance of your assistant may vary when using other LLMs, but improvements can be made by tuning flow and collect step descriptions.

OpenAI

API Token

The API token authenticates your requests to the OpenAI API.

To configure the API token, follow these steps:

  1. If you haven't already, sign up for an account on the OpenAI platform.
  2. Navigate to the OpenAI Key Management page, and click on the "Create New Secret Key" button to initiate the process of obtaining <your-api-key>.
  3. The API key can be set in the model configuration or through an environment variable.

To set the API key in the config, you can use the api_key parameter in the model configuration:

   model_groups:
     - id: openai-direct
       models:
         - provider: openai
           model: gpt-5.1-2025-11-13
           api_key: ${MY_OPENAI_API_KEY}

The api_key parameter can be set in the model configuration for each model in the model_groups section of the endpoints.yml file. For security reasons, the value of the api_key must reference an environment variable, as demonstrated above. This approach ensures sensitive information is securely stored. Directly assigning the API key in the configuration file is not allowed, as it could potentially expose the key to unauthorized access.

To set the API key as an environment variable, you can use the following command in a terminal or command prompt:

export OPENAI_API_KEY=<your-api-key>

setx OPENAI_API_KEY <your-api-key>

This will apply to future cmd prompt window, so you will need to open a new one to use that variable. Replace <your-api-key> with the actual API key you obtained from the OpenAI platform.

Configuration

There are no additional OpenAI specific parameters to be configured. However, there could be model specific parameters like temperature that you might want to modify. Names for such parameters can found in OpenAI's API documentation and defined under llm key of the component's configuration. Please refer to LiteLLM's documentation to know the list of models supported from the OpenAI platform.

Model deprecations

OpenAI regularly publishes a deprecation schedule for its models. This schedule can be accessed in the documentation published by OpenAI.

Azure OpenAI Service

API Token

The API token authenticates your requests to the Azure OpenAI Service.

Set the API token as an environment variable. You can use the following command in a terminal or command prompt:

export AZURE_API_KEY=<your-api-key>

setx AZURE_API_KEY <your-api-key>

This will apply to future cmd prompt window, so you will need to open a new one to use that variable. Replace <your-api-key> with the actual API key you obtained from the Azure OpenAI Service platform.

Configuration

To access models provided by Azure OpenAI Service, there are a few additional parameters that need to be configured:

Model specific parameters like temperature can be defined as well. Refer to OpenAI Azure service's API documentation for information on available parameter names.

A complete example configuration of the CompactLLMCommandGenerator using Azure OpenAI Service would look like this:

    - name: CompactLLMCommandGenerator
      llm:
        model_group: my_azure_deployment

   model_groups:
     - id: my_azure_deployment
       models:
         - provider: azure
           deployment: ${AZURE_DEPLOYMENT_NAME}
           api_base: ${AZURE_API_BASE}
           api_version: ${AZURE_API_VERSION}
           api_key: ${MY_AZURE_API_KEY}
           timeout: 7

Amazon Bedrock

Requirements:

  1. Make sure you have rasa-pro>=3.11.x installed.
  2. Install boto3>=1.28.57.
  3. Make sure your AWS credentials are accessible via credentials, IAM role, or environment variables.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "bedrock:InvokeModel",
                "bedrock:InvokeModelWithResponseStream"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "bedrock:GetModelInvocationLoggingConfiguration"
            ],
            "Resource": "*"
        }
    ]
}

Configuration

Edit config.yaml to use an appropriate model_group from the endpoints.yaml

    - name: CompactLLMCommandGenerator
      llm:
        model_group: amazon_bedrock # Model group ID

   model_groups:
     - id: amazon_bedrock
       models:
         - provider: bedrock
           model: anthropic.claude-instant-v1

Configuring self-signed SSL certificates

In environments where a proxy performs TLS interception, Rasa may need to be configured to trust the certificates used by your proxy. By default, certificates are loaded from the OS certificate store. However, if your setup involves custom self-signed certificates, you can specify these by setting the RASA_CA_BUNDLE environment variable.