Telemetry Events

Telemetry events are only reported if telemetry is enabled. A detailed explanation on the reasoning behind collecting optional telemetry events can be found in our telemetry documentation.

CALM

Response Rephrased

backend Triggered when a response is rephrased.
Event properties:

• rephrase_all (boolean): True if the rephraser is setup to rephrase all responses.
• custom_prompt_template (string): A custom prompt template, if specified.
• llm_type (string): The type of LLM.
• llm_model (string): The model name of the LLM.

Intentless Policy Training Started

backend Triggered when a user trains the IntentlessPolicy.

Intentless Policy Training Completed

backend Triggered when the training of IntentlessPolicy completed.
Event properties:

• embeddings_type (string): The type of embeddings.
• embeddings_model (string): The model name for embeddings.
• llm_type (string): The type of LLM.
• llm_model (string): The model name of the LLM.

Intentless Policy Predicted

backend Triggered when the IntentlessPolicy makes a prediction.
Event properties:

• embeddings_type (string): The type of embeddings.
• embeddings_model (string): The model name for embeddings.
• llm_type (string): The type of LLM.
• llm_model (string): The model name of the LLM.
• score (number): The prediction score.

Enterprise Search Policy Training Started

backend Triggered when a user trains the EnterpriseSearchPolicy.

Enterprise Search Policy Training Completed

backend Triggered when the training of EnterpriseSearchPolicy completed.
Event properties:

• vector_store (string): The vector store.
• embeddings_type (string): The type of embeddings.
• embeddings_model (string): The model name for embeddings.
• llm_type (string): The type of LLM.
• llm_model (string): The model name of the LLM.
• citation_enabled (boolean): Whether source_citation is enabled or not.

Enterprise Search Policy Predicted

backend Triggered when the EnterpriseSearchPolicy makes a prediction.
Event properties:

PII Management in CALM Enabled

backend Triggered when PII management in CALM is enabled.
Event properties:

• num_total_rules (integer): Total number of anonymization rules defined.
• redact_count (integer): Number of redact rules defined.
• mask_count (integer): Number of mask rules defined.
• stream_pii (boolean): Whether streaming PII in un-anonymized events to event brokers is enabled or not.
• tracker_store_anonymization_enabled (boolean): Whether anonymization of eligible trackers in the tracker store is enabled or not.
• tracker_store_deletion_enabled (boolean): Whether deletion of eligible trackers in the tracker store is enabled or not.
• anonymization_cron_trigger (string): Cron trigger for anonymization of eligible tracker sessions in the tracker store.
• deletion_cron_trigger (string): Cron trigger for deletion of eligible tracker sessions in the tracker store.

End-to-End Testing

E2E Test Run Started

backend Triggered when end-to-end testing has been started.
Event properties:

• number_of_test_cases (integer): Number of test cases to be run.
• number_of_fixtures (integer): Number of fixtures defined globally.
• uses_fixtures (boolean): Indicates if any fixtures have been defined globally.
• uses_metadata (boolean): Indicates if any metadata has been defined globally.
• number_of_metadata (integer): Number of metadata defined globally.
• uses_assertions (boolean): Indicates if any assertions have been defined in test cases.
• flow_started_count (integer): Number of flow_started assertion type used in the test run.
• flow_completed_count (integer): Number of flow_completed assertion type used in the test run.
• flow_cancelled_count (integer): Number of flow_cancelled assertion type used in the test run.
• pattern_clarification_count (integer): Number of pattern_clarification assertion type used in the test run.
• action_executed_count (integer): Number of action_executed assertion type used in the test run.
• slot_was_set_count (integer): Number of slot_was_set assertion type used in the test run.
• slot_was_not_set_count (integer): Number of slot_was_not_set assertion type used in the test run.
• bot_uttered_count (integer): Number of bot_uttered assertion type used in the test run.
• generative_response_is_relevant_count (integer): Number of generative_response_is_relevant assertion type used in the test run.
• generative_response_is_grounded_count (integer): Number of generative_response_is_grounded assertion type used in the test run.

Model Training

Training Started

backend A training of a Rasa machine learning model got started. The event provides information on aggregated training data statistics.
Event properties:

• language (string): Language model is trained with, e.g. 'en'.
• training_id (string): Generated unique identifier for this training.
• type (string): Type of model trained, either 'nlu', 'core' or 'rasa'.
• pipeline (undefined): List of the pipeline configurations used for training.
• policies (undefined): List of the policy configurations used for training.
• train_schema (undefined): Training graph schema for graph recipe
• predict_schema (undefined): Predict graph schema for graph recipe
• num_intent_examples (integer): Number of NLU examples.
• num_entity_examples (integer): Number of entity examples.
• num_actions (integer): Number of actions defined in the domain.
• num_templates (integer): Number of templates or responses defined in the domain.
• num_conditional_response_variations (integer): Number of conditional response variations defined in the domain.
• num_slot_mappings (integer): Number of total slot mappings defined in the domain.
• num_custom_slot_mappings (integer): Number of custom slot mappings defined in the domain.
• num_conditional_slot_mappings (integer): Number of slot mappings with conditions attached defined in the domain.
• num_slots (integer): Number of slots defined in the domain.
• num_forms (integer): Number of forms defined in the domain.
• num_intents (integer): Number of intents defined in the domain.
• num_entities (integer): Number of entities defined in the domain.
• num_story_steps (integer): Number of story steps available.
• num_lookup_tables (integer): Number of different lookup tables.
• num_synonyms (integer): Total number of entity synonyms defined.
• num_regexes (integer): Total number of regexes defined.
• is_finetuning (boolean): True if a model is trained by finetuning an existing model.
• recipe (string): Recipe used in training the model, either 'default.v1' or 'graph.v1'.
• num_flows (integer): Number of flows.
• num_flows_with_nlu_trigger (integer): Number of flows that have an NLU trigger defined.
• num_flows_with_flow_guards (integer): Number of flows that have a flow guard condition.
• num_flows_with_not_startable_flow_guards (integer): Number of flows with the flow guard condition 'if: False'.
• num_collect_steps (integer): Number of collect steps in flows.
• num_collect_steps_with_separate_utter (integer): Number of collect steps which have a different utterance defined.
• num_collect_steps_with_rejections (integer): Number of collect steps with rejections included.
• num_collect_steps_with_not_reset_after_flow_ends (integer): Number of collect steps with 'reset_after_flow_ends' set to 'False'.
• num_set_slot_steps (integer): Number of set slot steps in flows.
• num_link_steps (integer): Number of link steps in flows.
• num_call_steps (integer): Number of call steps in flows.
• max_depth_of_if_construct (integer): Maximum depth of an if construct.
• num_shared_slots_between_flows (integer): Number of slots being shared across flows.
• llm_command_generator_model_name (string): The name of the model used in the 'LLMCommandGenerator'.
• llm_command_generator_custom_prompt_used (boolean): True, if a custom prompt was configured for the 'LLMCommandGenerator', False otherwise.
• multi_step_llm_command_generator_custom_handle_flows_prompt_used (boolean): True, if a custom prompt was configured for handling flows in the 'MultiStepLLMCommandGenerator', False otherwise.
• multi_step_llm_command_generator_custom_fill_slots_prompt_used (boolean): True, if a custom prompt was configured for filling slots in the 'MultiStepLLMCommandGenerator', False otherwise.
• flow_retrieval_enabled (boolean): True, if flow retrieval is configured for the 'LLMCommandGenerator', False otherwise.
• flow_retrieval (string): The name of the embedding model used by flow retrieval within 'LLMCommandGenerator'.
• agents (object): Agent configuration and usage information including MCP servers, agents, and their usage in flows.

Training Completed

backend The training of a Rasa machine learning model finished. The event provides information about the resulting model.
Event properties:

• training_id (string): Generated unique identifier for this training. Can be used to join with 'Training Started'.
• type (string): Type of model trained, either 'nlu', 'core' or 'rasa'.
• runtime (integer): The time in seconds it took to train the model.

Model Testing

Model Core Tested

backend Triggered when a Core model is getting tested.
Event properties:

• project (string,null): Fingerprint of the project the tested model got trained in.
• num_story_steps (integer): Number of story steps used for testing
• end_to_end (boolean): Indicates if tests are running in end-to-end mode, testing message handling and dialogue handling at the same time

Model NLU Tested

backend Triggered when an NLU model is getting tested.
Event properties:

• num_intent_examples (integer): Number of NLU examples.
• num_entity_examples (integer): Number of entity examples.
• num_lookup_tables (integer): Number of different lookup tables.
• num_synonyms (integer): Total number of entity synonyms defined.
• num_regexes (integer): Total number of regexes defined.

Model Serving

Interactive Learning Started

backend Triggered when an interactive learning session got started.
Event properties:

• skip_visualization (boolean): Whether the visualization of stories should be shown during the interactive learning session
• save_in_e2e (boolean): Whether the data should be stored in end-to-end format

Server Started

backend Triggered when a Rasa server gets started.
Event properties:

• input_channels (array): Names of the used input channels
• api_enabled (boolean): Indicator if the API is enabled or if only the input channel is running
• number_of_workers (integer): Amount of Sanic workers started as part of the server
• endpoints_nlg (string,null): Type of the used NLG endpoint
• endpoints_nlu (string,null): Type of the used NLU endpoint
• endpoints_action_server (string,null): Type of the used action server
• endpoints_model_server (string,null): Type of the used model server
• endpoints_tracker_store (string,null): Type of the used tracker store
• endpoints_lock_store (string,null): Type of the used lock store
• endpoints_event_broker (string,null): Type of the used event broker
• project (string,null): Hash of the deployed model the server is started with

Shell Started

backend Triggered when a shell session is started to talk to a trained bot.
Event properties:

• type (string): Type of the model, either 'nlu', 'core' or 'rasa'.

Markers Extraction

Markers Extraction Initiated

backend Triggered when marker extraction has been initiated.
Event properties:

• strategy (string): Strategy to use when selecting trackers to extract from.
• only_extract (boolean): Indicates if path to write out statistics hasn't been specified.
• seed (boolean): The seed to initialise the random number generator for use with the 'sample' strategy.
• count (integer,null): Number of trackers to extract from (for any strategy except 'all').

Markers Extracted

backend Triggered when markers have been extracted.
Event properties:

• trackers_count (integer): Number of processed trackers.

Markers Parsed

backend Triggered when markers have been successfully parsed.
Event properties:

• marker_count (integer): Number of parsed markers.
• max_depth (integer): Maximum depth of the parsed markers.
• branching_factor (integer): Maximum number of children of any of the parsed markers.

Markers Statistics Computed

backend Triggered when marker statistics have been computed.
Event properties:

• trackers_count (integer): Number of processed trackers.

Data Handling

Training Data Split

backend Triggered when training data gets split.
Event properties:

• fraction (number): Percentage of the data which goes into training data (the rest goes into the test set).
• type (string): Type of data, either 'nlu', 'core' or 'rasa'.

Training Data Validated

backend Triggered when training data gets validated.
Event properties:

• validation_success (boolean): whether the validation was successful

Training Data Converted

backend Triggered when training data gets converted.
Event properties:

• output_format (string): target format of the converter
• type (string): Type of data, either 'nlu', 'core', 'config' or 'nlg'.

Tracker Exported

backend Triggered when conversations get exported from a tracker store through an event broker.
Event properties:

• event_broker (string): Name of the used event broker
• tracker_store (string): Name of the used tracker store
• number_of_exported_events (integer): Number of events exported through the event broker

Story Visualization Started

backend Triggered when stories are getting visualized.

Rasa Pro Services

Analytics Started

backend Triggered when the Analytics pipeline is started.
Event properties:

• consumers (number): The number of Kafka consumers.

Assistant Session Started

backend Triggered when the Analytics pipeline detects a new session start.

Miscellaneous

Telemetry Disabled

backend Triggered when telemetry reporting gets disabled. Last event sent before disabling telemetry. This event is not sent, if the user never enabled telemetry reporting before deactivating it.

Project Created

backend Triggered when a project is created using rasa init.
Event properties:

• init_directory (string): Hash of the directory path the project is created in.