Build your first agent in just a few minutes with [Rasa Copilot](https://hello.rasa.com/?utm_source=docs&utm_medium=referral&utm_campaign=docs_cta).

On this page

Distributed tracing tracks requests as they flow through a distributed
system (in this case: a Rasa assistant), sending data about the requests
to a **tracing backend** which collects all trace data and enables
inspecting it. Trace data helps you understand the flow of requests
through both the components of a single service (Rasa itself), and across
different distributed services, for example, your action server.

### Supported Tracing Backends/Collectors

To trace requests in Rasa, you can either use
[Jaeger](https://www.jaegertracing.io/) as a backend, or use
the [OTEL Collector (OpenTelemetry Collector)](https://opentelemetry.io/docs/collector/).
to collect traces and then send them to the backend of your choice.
See [Configuring a Tracing Backend or Collector](/content/docs/reference/integrations/tracing/#configuring-a-tracing-backend-or-collector/index.html)
for instructions.

### Rasa Channels

Trace context sent along with requests using the [W3C Trace Context Specification](https://www.w3.org/TR/trace-context/)
via the REST channel is used to continue tracing in Rasa.

#### Rasa Inspector

If you have enabled tracing in Rasa and are using the [Rasa Inspector](/content/docs/pro/testing/trying-assistant/index.html)
debugging tool to try your assistant, note that in addition to the expected tracing span for the `Agent.handle_message`
method call, the tracing backend will collect independent tracing spans for the `MessageProcessor.get_tracker`
method calls. This is expected behaviour because the Rasa Inspector tool uses the Rasa [HTTP API endpoints](/content/docs/reference/api/pro/http-api/index.html) to
retrieve the conversation tracker which is required by the Inspector interface.

### Action Server

The trace context from Rasa is sent along with requests to the custom action server
using the [W3C Trace Context Specification](https://www.w3.org/TR/trace-context/) and
then used to continue tracing the request through the custom action server.

Tracing is continued in the action server by instrumenting the webhook that receives custom actions.
See [Action server attributes](/content/docs/reference/integrations/tracing/#action-executor-attributes/index.html) for the attributes captured as part of the trace context.

See [traced events](/content/docs/reference/integrations/tracing/#traced-events/index.html) for
details on what attributes are made available as part of the trace context in Rasa.

## Questions Tracing Can Help Answer

Tracing can help troubleshoot issues in development and production, by answering questions such as:

- How does a user message request get processed across different components i.e. dialogue understanding components (NLU, `CommandGenerator`, `CommandProcessorComponent`), policies, and action server?
- Why has my Rasa assistant decided to execute a certain action?
- Why has my Rasa assistant been slow to respond?
- Why have my custom actions been slow to execute?
- What is my OpenAI prompt token usage?
- What is the performance of my Rasa assistant across different flows?
- What is the performance of my Rasa assistant across different LLM models?
- What is the performance of my Rasa assistant across different vector stores?

## Configuring a Tracing Backend or Collector

To configure a tracing backend or collector, add a `tracing` entry to your endpoints
i.e. in your `endpoints.yml` file, or in the relevant section of your Helm values in a deployment.

Enabling tracing activates Rasa's instrumentation layer. That layer also records [Observability Metrics](/content/docs/pro/improve/observability-metrics/index.html). If you configure a `metrics` block in your endpoints file, you must also enable tracing or no metric measurements will be collected.

### Jaeger

To configure a Jaeger tracing backend, specify the `type` as `jaeger`.

endpoints.yml

```yaml
tracing:
  type: jaeger
  host: localhost
  port: 4317
  service_name: rasa
  sync_export: ~
```

### OTEL Collector

Collectors are components that collect traces in a vendor-agnostic way and then forward them to various backends.
For example, the OpenTelemetry Collector (OTEL) can collect traces from multiple different components and instrumentation libraries, and then export them to multiple different backends e.g. jaeger.

To configure an OTEL Collector, specify the `type` as `otlp`.

endpoints.yml

```yaml
tracing:
  type: otlp
  endpoint: my-otlp-host:4317
  insecure: false
  service_name: rasa
  root_certificates: ./tests/unit/tracing/fixtures/ca.pem
```

## Traced Events

The Rasa service areas that are traceable cover the actions required to:

- **train a model** (i.e., the training of each graph component)
- **handle a message**

### Model Training

Tracing is enabled for model training by instrumenting Rasa [`GraphTrainer`](https://legacy-docs-oss.rasa.com/docs/rasa/reference/rasa/engine/training/graph_trainer/#graphtrainer-objects) and [`GraphNode`](https://legacy-docs-oss.rasa.com/docs/rasa/reference/rasa/engine/graph/#graphnode-objects) classes.

#### `GraphTrainer` Attributes

The following attributes can be inspected during training of `GraphTrainer`:

- `training_type`of model configuration:
  - "NLU"
  - "CORE"
  - "BOTH"
  - "END-TO-END"
- `language` of model configuration
- `recipe_name` used in the `config.yml` file
- `output_filename`: the location where the packaged model is saved
- `is_finetuning`: boolean argument, if `True` enables incremental training

#### `GraphNode` Attributes

The following attributes are captured during the training (as well as prediction during message handling) of every graph node:

- `node_name`
- `component_class`
- `fn_name`: method of component class that gets called

### Message Handling

The following Rasa classes are instrumented to enable tracing during message handling:

- `Agent`
- `MessageProcessor`
- [`TrackerStore`](/content/docs/reference/integrations/tracker-stores/index.html)
- [`LockStore`](/content/docs/reference/integrations/lock-stores/index.html)
- [`CompactLLMCommandGenerator`](/content/docs/reference/integrations/tracing/#compactllmcommandgenerator-attributes/index.html)
- [`SearchReadyLLMCommandGenerator`](/content/docs/reference/integrations/tracing/#searchreadycommandgenerator-attributes/index.html)
- [`NLUCommandAdapter`](/content/docs/reference/integrations/tracing/#nlucommandadapter-attributes/index.html)
- [`FlowPolicy`](/content/docs/reference/config/policies/flow-policy/index.html)
- [`EnterpriseSearchPolicy`](/content/docs/reference/integrations/tracing/#enterprisesearchpolicy-attributes/index.html)
- [`InformationRetrieval`](/content/docs/reference/integrations/tracing/#informationretrieval-attributes/index.html)
- [`EndpointConfig`](/content/docs/reference/integrations/tracing/#endpointconfig-attributes/index.html)

In addition, the following Python modules were instrumented to enable tracing during message handling:

- command processor module, i.e. utility functions leveraged by the `CommandProcessorComponent` to pre-process [predicted commands](/content/docs/reference/config/components/llm-command-generators/#command-reference/index.html)
- flow executor module, i.e. utility functions leveraged by `FlowPolicy` to advance flows

Namely, these operations are now traceable:
- receiving a message
- parsing the message
- predicting [commands](/content/docs/reference/config/components/llm-command-generators/#command-reference/index.html)
- [pre-processing commands](/content/docs/reference/integrations/tracing/#command-processor-module-attributes/index.html)
- predicting the next action
- running the action
- advancing [flows](/content/docs/reference/integrations/tracing/#flow-executor-module-attributes/index.html)
- searching documents in [vector stores](/content/docs/reference/config/policies/enterprise-search-policy/#vector-store/index.html) for enterprise search
- generating LLM answers by policies e.g. [`EnterpriseSearchPolicy`](/content/docs/reference/config/policies/enterprise-search-policy/index.html)
- tracing [prompt token usage](/content/docs/reference/integrations/tracing/#tracing-prompt-token-usage/index.html)
- retrieving and saving the tracker
- locking the conversation
- publishing to the event broker
- making requests to the action server or nlg server
- passing the trace context to the action server

## Tracing in the Action Server

API Requests are traced as they flow through the action server by instrumenting the webhook that receives custom actions and other classes involved in the execution of custom actions.

### Webhook Attributes

The following attributes are captured as part of the trace context of the webhook that receives custom actions;

- `http.method`: the http method used to make the request
- `http.route`: the endpoint of the request
- `next_action`: the name of the next action to be executed
- `version`: the rasa version used
- `sender_id`: the id of the conversation
- `message_id`: the unique message id