# LLM Configuration for Rasa Pro 3.10 and below For Rasa Pro versions `3.10` and below, refer to the [LLM Configuration for `<=3.10`](/content/docs/reference/config/components/llm-configuration/index.html) page. ## Overview This page applies to the following components which use LLMs: - [SearchReadyLLMCommandGenerator](/content/docs/reference/config/components/llm-command-generators/#searchreadyllmcommandgenerator/index.html) - [CompactLLMCommandGenerator](/content/docs/reference/config/components/llm-command-generators/#compactllmcommandgenerator/index.html) - [EnterpriseSearchPolicy](/content/docs/reference/config/policies/enterprise-search-policy/index.html) - [ContextualResponseRephraser](/content/docs/reference/primitives/contextual-response-rephraser/index.html) - [LLMBasedRouter](/content/docs/reference/config/components/coexistence-routers/#llmbasedrouter/index.html) - [SingleStepLLMCommandGenerator](/content/docs/reference/config/components/deprecated-components/#singlestepllmcommandgenerator/index.html) (deprecated) - [MultiStepLLMCommandGenerator](/content/docs/reference/config/components/deprecated-components/#multistepllmcommandgenerator/index.html) (deprecated) - [IntentlessPolicy](/content/docs/reference/config/policies/intentless-policy/index.html) (deprecated) All the above components can be configured to change: - the LLM provider - the model(s) to be used Starting with version Rasa Pro `3.10`, CALM uses [LiteLLM](https://litellm.vercel.app/) under the hood to integrate with different LLM providers. Hence, all [LiteLLM's integrated providers](https://litellm.vercel.app/docs/providers) are supported with CALM as well. We explicitly mention the settings required for the most frequently used ones in the sections below. ## Declaring LLM deployments LLM deployments are always declared in groups comprising of 1 or more deployments. The below sections explain how to declare these groups. ### Model Groups Model groups allow you to define multiple models under a single ID which can be accessed by any component. Model groups are defined in the `endpoints.yml` file under the `model_groups` key, separating model definitions from individual component configurations. For example: ```yaml model_groups: - id: openai-direct # Unique identifier for the model group models: ... ``` - The `id` key uniquely identifies the model group. - The `models` key lists all model deployments in that group. - Each model in the list includes a configuration, explained in the following sections. ### Defining a single model group ```yaml model_groups: - id: openai-direct # Unique identifier for the model group models: - provider: openai model: gpt-5.1-2025-11-13 ``` #### Required Parameters There are certain required parameters for each model group: 1. `provider` - Unique identifier of the LLM provider to be used. 2. `model` - Specifies the name of the model identifier available from the LLM provider's documentation, for e.g. `gpt-5.1-2025-11-13`. #### Optional Parameters Each model group also accepts inference time parameters like `temperature`, etc which are optional but can be useful in extracting the best performance out of the model being used. Please refer to the [official LiteLLM documentation](https://litellm.vercel.app/docs/completion/input) for a list of such parameters supported. When configuring a particular provider, there are a few provider specific settings which are explained under each [provider's individual sub-section below](/content/docs/reference/config/components/llm-configuration/index.html). #### Reasoning models and `reasoning_effort` For models that support it, `reasoning_effort` is an optional inference-time parameter that controls how much internal reasoning the model applies before returning an answer. Higher values typically increase latency and cost; lower values favor faster responses. Allowed values depend on the provider and model (for example `none`, `minimal`, `low`, and `high` for many OpenAI reasoning-capable models). See provider documentation for more details, e.g. [OpenAI Developers: Reasoning models](https://developers.openai.com/api/docs/guides/reasoning) or [Claude API Docs: Effort](https://platform.claude.com/docs/en/build-with-claude/effort). You can set `reasoning_effort` alongside other model settings in each entry under `models` in `endpoints.yml`, for example: ```yaml model_groups: - id: openai-reasoning models: - provider: openai model: gpt-5-mini-2025-08-07 reasoning_effort: "minimal" ``` By default, Rasa tries to apply the lowest appropriate `reasoning_effort` for the resolved model name (using provider metadata where available). To override this behavior, set `reasoning_effort` explicitly in your model configuration. Models that do not support this parameter do not receive a default. #### Referencing environment variables in the model configuration To reference environment variables in the model configuration, you can use the `${}` syntax. For example: ```yaml model_groups: - id: openai-direct models: - provider: openai model: gpt-5.1-2025-11-13 api_key: ${MY_OPENAI_API_KEY} ``` In the above example, the `api_key` parameter references the environment variable `MY_OPENAI_API_KEY`. #### LLM API health check The model config and the connection to the LLM provider can be validated by setting the `LLM_API_HEALTH_CHECK` environment variable to `true`. ```bash export LLM_API_HEALTH_CHECK=true ``` By default, the variable is set to `False`. When set to `True`, all LLM deployments defined will be checked for availability by making a test API request. ### Using a model group in a component Components using an LLM can be configured to use any of the declared model groups in the component's configuration. To use a model group, you can specify the `model_group` key under the `llm` key. For example: ```yaml recipe: default.v1 language: en pipeline: - name: CompactLLMCommandGenerator llm: model_group: openai-direct ``` ### Defining multiple model groups ```yaml model_groups: - id: openai-gpt-5-1 models: - provider: openai model: gpt-5.1-2025-11-13 - id: openai-gpt-5-mini models: - provider: openai model: gpt-5-mini-2025-08-07 ``` The examples above illustrate how to define a model groups consisting of a single model deployment. In order to handle a larger volume of conversations, it is recommended to include multiple model deployments within a model group. To do so you can add additional deployments to the `models` list as explained in the [Multi-LLM routing](/content/docs/reference/deployment/multi-llm-routing/index.html) page. ### Using different model groups in different components ```yaml recipe: default.v1 pipeline: - name: LLMBasedRouter calm_entry: sticky: ... nlu_entry: sticky: ... non_sticky: ... llm: model_group: openai-gpt-5-mini - name: CompactLLMCommandGenerator llm: model_group: openai-gpt-5-1 ``` Multiple components can rely on the same model group and a single component can use multiple models defined in a model group, via the [LLM router](/content/docs/reference/deployment/multi-llm-routing/index.html). ## Chat completion models Default Provider CALM is LLM agnostic and can be configured with different LLMs, but OpenAI is the default model provider. Majority of our experiments have been with models available on OpenAI or OpenAI Azure service. The performance of your assistant may vary when using other LLMs, but improvements can be made by tuning flow and collect step descriptions. ### OpenAI #### API Token The API token authenticates your requests to the OpenAI API. To configure the API token, follow these steps: 1. If you haven't already, sign up for an account on the OpenAI platform. 2. Navigate to the [OpenAI Key Management page](https://platform.openai.com/account/api-keys), and click on the "Create New Secret Key" button to initiate the process of obtaining ``. 3. The API key can be set in the model configuration or through an environment variable. To set the API key in the config, you can use the `api_key` parameter in the model configuration: ```yaml model_groups: - id: openai-direct models: - provider: openai model: gpt-5.1-2025-11-13 api_key: ${MY_OPENAI_API_KEY} ``` The `api_key` parameter can be set in the model configuration for each model in the `model_groups` section of the `endpoints.yml` file. For security reasons, the value of the `api_key` must reference an environment variable, as demonstrated above. This approach ensures sensitive information is securely stored. Directly assigning the API key in the configuration file is not allowed, as it could potentially expose the key to unauthorized access. To set the API key as an environment variable, you can use the following command in a terminal or command prompt: - Linux/MacOS ```shell export OPENAI_API_KEY= ``` - Windows ```shell setx OPENAI_API_KEY ``` This will apply to future cmd prompt window, so you will need to open a new one to use that variable. Replace `` with the actual API key you obtained from the OpenAI platform. #### Configuration There are no additional OpenAI specific parameters to be configured. However, there could be model specific parameters like `temperature` that you might want to modify. Names for such parameters can found in [OpenAI's API documentation](https://platform.openai.com/docs/api-reference/chat) and defined under `llm` key of the component's configuration. Please refer to [LiteLLM's documentation](https://litellm.vercel.app/docs/providers/openai#openai-chat-completion-models) to know the list of models supported from the OpenAI platform. #### Model deprecations OpenAI regularly publishes a deprecation schedule for its models. This schedule can be accessed in the [documentation published by OpenAI](https://platform.openai.com/docs/deprecations). ### Azure OpenAI Service #### API Token The API token authenticates your requests to the Azure OpenAI Service. Set the API token as an environment variable. You can use the following command in a terminal or command prompt: - Linux/MacOS ```shell export AZURE_API_KEY= ``` - Windows ```shell setx AZURE_API_KEY ``` This will apply to future cmd prompt window, so you will need to open a new one to use that variable. Replace `` with the actual API key you obtained from the Azure OpenAI Service platform. #### Configuration To access models provided by [Azure OpenAI Service](https://azure.microsoft.com/en-in/products/ai-services/openai-service), there are a few additional parameters that need to be configured: - `provider` - Set to `azure`. - `api_base` - The URL for your Azure OpenAI instance. An example might look like this: `https://my-azure.openai.azure.com/`. - `api_version` - The API version to use for this operation. This follows the YYYY-MM-DD format and the value should be enclosed in single or double quotes. - `deployment` - Name of the deployment on Azure. Model specific parameters like `temperature` can be defined as well. Refer to [OpenAI Azure service's API documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#request-body) for information on available parameter names. A complete example configuration of the `CompactLLMCommandGenerator` using Azure OpenAI Service would look like this: ```yaml - name: CompactLLMCommandGenerator llm: model_group: my_azure_deployment ``` ```yaml model_groups: - id: my_azure_deployment models: - provider: azure deployment: ${AZURE_DEPLOYMENT_NAME} api_base: ${AZURE_API_BASE} api_version: ${AZURE_API_VERSION} api_key: ${MY_AZURE_API_KEY} timeout: 7 ``` ### Amazon Bedrock #### Requirements: 1. Make sure you have `rasa-pro>=3.11.x` installed. 2. Install `boto3>=1.28.57`. 3. Make sure your AWS credentials are accessible via credentials, IAM role, or environment variables. - If you are using AWS credentials, ensure that the `~/.aws/credentials` file is set up with the correct access key and secret key. - If you are using an IAM role, ensure that the role has the necessary permissions to access Amazon Bedrock models and to have requested model access to the model of choice in AWS Bedrock. For example, you can use the following policy to allow access to all Bedrock models and to grant access to model invocation logging configuration, which is required during LLM client validation: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "bedrock:GetModelInvocationLoggingConfiguration" ], "Resource": "*" } ] } ``` #### Configuration Edit `config.yaml` to use an appropriate `model_group` from the `endpoints.yaml` ```yaml - name: CompactLLMCommandGenerator llm: model_group: amazon_bedrock # Model group ID ``` ```yaml model_groups: - id: amazon_bedrock models: - provider: bedrock model: anthropic.claude-instant-v1 ``` ## Configuring self-signed SSL certificates In environments where a proxy performs TLS interception, Rasa may need to be configured to trust the certificates used by your proxy. By default, certificates are loaded from the OS certificate store. However, if your setup involves custom self-signed certificates, you can specify these by setting the `RASA_CA_BUNDLE` environment variable.