## Tracing

Distributed tracing tracks requests as they flow through a distributed system (in this case: a Rasa assistant), sending data about the requests to a **tracing backend** which collects all trace data and enables inspecting it. Trace data helps you understand the flow of requests through both the components of a single service (Rasa itself), and across different distributed services, for example, your action server.

### Supported Tracing Backends/Collectors

To trace requests in Rasa Pro, you can either use [Jaeger](https://www.jaegertracing.io/) as a backend, or use the [OTEL Collector (OpenTelemetry Collector)](https://opentelemetry.io/docs/collector/) to collect traces and then send them to the backend of your choice. See [Configuring a Tracing Backend or Collector](https://legacy-docs-oss.rasa.com/docs/rasa/next/monitoring/tracing/#configuring-a-tracing-backend-or-collector) for instructions.

### Enabling / Disabling

Tracing is automatically enabled in Rasa Pro by [configuring a supported tracing backend](https://legacy-docs-oss.rasa.com/docs/rasa/next/monitoring/tracing/#configuring-a-tracing-backend-or-collector). No further action is required to enable tracing.

You can disable tracing by leaving the `tracing:` configuration key empty in your endpoints file.

### Rasa Channels

Trace context sent along with requests using the [W3C Trace Context Specification](https://www.w3.org/TR/trace-context/) via the REST channel is used to continue tracing in Rasa Pro.

### Action Server

The trace context from Rasa Pro is sent along with requests to the custom action server using the [W3C Trace Context Specification](https://www.w3.org/TR/trace-context/) and then used to continue tracing the request through the custom action server.

Tracing is continued in the action server by instrumenting the webhook that receives custom actions. See [Action server attributes](https://legacy-docs-oss.rasa.com/docs/rasa/next/monitoring/tracing/#action-server-attributes) for the attributes captured as part of the trace context.

See [traced events](https://legacy-docs-oss.rasa.com/docs/rasa/next/monitoring/tracing/#traced-events) for details on what attributes are made available as part of the trace context in Rasa Pro.

## Configuring a Tracing Backend or Collector

To configure a tracing backend or collector, add a `tracing` entry to your endpoints i.e. in your `endpoints.yml` file, or in the relevant section of your Helm values in a deployment.

### Jaeger

To configure a Jaeger tracing backend, specify the `type` as `jaeger`.

```
tracing:
  type: jaeger
  host: localhost
  port:6831
  service_name: rasa
  sync_export:~
```

### OTEL Collector

Collectors are components that collect traces in a vendor-agnostic way and then forward them to various backends. For example, the OpenTelemetry Collector (OTEL) can collect traces from multiple different components and instrumentation libraries, and then export them to multiple different backends e.g. jaeger.

To configure an OTEL Collector, specify the `type` as `otlp`.

```
tracing:
  type: otlp
  endpoint: my-otlp-host:4318
  insecure:false
  service_name: rasa
  root_certificates: ./tests/unit/tracing/fixtures/ca.pem
```

## Traced Events

The Rasa service areas that are traceable cover the actions required to:

- **train a model** (i.e., the training of each [graph component](/content/docs/rasa/custom-graph-components#graph-components/index.html))
- **handle a message**

### Model Training

Tracing is enabled for model training by instrumenting Rasa [`GraphTrainer`](/content/docs/rasa/reference/rasa/engine/training/graph_trainer/#graphtrainer-objects/index.html) and [`GraphNode`](/content/docs/rasa/reference/rasa/engine/graph/#graphnode-objects/index.html) classes.

#### `GraphTrainer` Attributes

The following attributes can be inspected during training of `GraphTrainer`:

- `training_type` of model configuration:
  - "NLU"
  - "CORE"
  - "BOTH"
  - "END-TO-END"
- `language` of model configuration
- `recipe_name` used in the `config.yml` file
- `output_filename`: the location where the packaged model is saved
- `is_finetuning`: boolean argument, if `True` enables incremental training

#### `GraphNode` Attributes

The following attributes are captured during the training (as well as prediction during message handling) of every graph node:

- `node_name`
- `component_class`
- `fn_name`: method of component class that gets called

### Message Handling

The following Rasa classes are instrumented to enable tracing during message handling:

- [`Agent`](/content/docs/rasa/reference/rasa/core/agent/#agent-objects/index.html)
- [`MessageProcessor`](/content/docs/rasa/reference/rasa/core/processor/#messageprocessor-objects/index.html)
- [`TrackerStore`](/content/docs/rasa/tracker-stores/index.html)
- [`LockStore`](/content/docs/rasa/lock-stores/index.html)

Namely, these operations are now traceable:

- receiving a message
- parsing the message
- predicting the next action
- running the action
- retrieving and saving the tracker
- locking the conversation
- publishing to the event broker
- passing the trace context to the action server

#### `Agent` Attributes

Tracing the `Agent` instance handling a message captures the following attributes:

- `input_channel`: the name of the channel connector
- `sender_id`: the conversation id
- `model_id`: a unique identifier for the model
- `model_name`: the model name

#### `MessageProcessor` Attributes

The following `MessageProcessor` attributes are extracted during the tracing:

- `number_of_events`: number of events in tracker
- `action_name`: the name of the predicted and executed action
- `sender_id`: the conversation id of the `DialogueStateTracker` object
- `message_id`: the unique message id

The latter three attributes are also injected in the trace context that gets passed to the requests made to the custom action server.

#### `TrackerStore` & `LockStore` Attributes

Observable `TrackerStore` and `LockStore` attributes include:

- `number_of_streamed_events`: number of new events to stream
- `broker_class`: the `EventBroker` on which the new events are published
- `lock_store_class`: Name of lock store used to lock conversations while messages are actively processed

## Tracing in the Action Server

API Requests are traced as they flow through the action server by instrumenting the webhook that receives custom actions.

### Action server Attributes

The following attributes are captured as part of the trace context;

- `http.method`: the http method used to make the request
- `http.route`: the endpoint of the request
- `next_action`: the name of the next action to be executed
- `version`: the rasa version used
- `sender_id`: the id of the source of the message
- `message_id`: the unique message id

You can also continue tracing the request further along your custom action code by [creating spans](https://opentelemetry.io/docs/instrumentation/python/manual/#creating-spans) to track the execution of any desired object.

Enabling and disabling tracing in the action server is also done in the same way as described [above](https://legacy-docs-oss.rasa.com/docs/rasa/next/monitoring/tracing/#enabling--disabling). The same Tracing Backends/Collectors listed [above](https://legacy-docs-oss.rasa.com/docs/rasa/next/monitoring/tracing/#supported-tracing-backendscollectors) are also supported for the action server. See [Configuring a Tracing Backend or Collector](https://legacy-docs-oss.rasa.com/docs/rasa/next/monitoring/tracing/#configuring-a-tracing-backend-or-collector) for further instructions.