Overview

Markers are conditions that allow you to describe and mark points of interest in dialogues for evaluating your bot.

In Rasa, a dialogue is represented as a sequence of events, which include bot actions that were executed, intents that were detected, and slots that were set. Markers allow you to describe conditions over such events. When the conditions are met, the relevant events are marked for further analysis or inspection.

There are several downstream applications for Markers. For example, they can be used to define and measure your bot's Key Performance Indicators (KPIs), such as dialogue completion or task success. Take Carbon Bot for example, which helps users offset their carbon emissions from flying. For Carbon Bot, you can define dialogue completion as "all mandatory slots have been filled", and task success as "all mandatory slots have been filled and a carbon estimate has been successfully computed". Marking when these important events occur allows you to measure Carbon Bot's success rate.

Markers also allow you to diagnose your dialogues by surfacing important events for further inspection. For example, you might observe that Carbon Bot tends to successfully set the travel_departure and travel_destination slots, but fails to set the travel_flight_class slot. You can define a marker to quantify how often this behavior occurs and surface relevant dialogues for review as part of Conversation Driven Development (CDD).

Marker definitions are written in YAML in a marker configuration file. For example, here are the markers that define dialogue completion and task success for Carbon Bot:

marker_dialogue_completion:
  and:
    - slot_was_set: travel_departure
    - slot_was_set: travel_destination
    - slot_was_set: travel_flight_class
marker_task_success:
  description: "Measure task success where all required slots are set and the custom action was triggered"
  and:
    - slot_was_set: travel_departure
    - slot_was_set: travel_destination
    - slot_was_set: travel_flight_class
    - action: provide_carbon_estimate

And here is the marker for surfacing dialogues where all mandatory slots are set except travel_flight_class:

marker_dialogue_mandatory_slot_failure:
  and:
    - slot_was_set: travel_departure
    - slot_was_set: travel_destination
    - not:
        - slot_was_set: travel_flight_class

The next sections explain how to write marker definitions, how to apply them to your existing dialogues, and what the output format looks like.

Defining Markers

Markers should be defined in a marker configuration file written in YAML. Each marker should have a unique identifier, and consists of at least one event condition. Markers can also contain operators, which allow you to express more nuanced behavior or combine event conditions.

Consider the following marker definition:

marker_mood_expressed:
  description: "Mood expressed was either unhappy or great"
  or:
    - intent: mood_unhappy
    - intent: mood_great

The unique marker identifier is marker_mood_expressed. This marker definition contains one operator or, and two event conditions intent: mood_unhappy and intent: mood_great. This markers will be true at every point in the dialogue where the user expressed either a mood_unhappy or a mood_great. More precisely, the marker will be true for every event which is a UserUttered() with the intent equal to mood_unhappy or a mood_great.

Event Conditions

The following event condition labels are supported:

• action: the specified bot action was executed.
• intent: the specified user intent was detected.
• slot_was_set: the specified slot was set.

The negated forms of the labels are also supported:

• not_action: the event is not the specified bot action.
• not_intent: the event is not the specified user intent.
• slot_was_not_set: the specified slot has not been set.

Operators

The following operators are supported:

• and: all listed conditions applied.
• or: any of the listed conditions applied.
• not: the condition did not apply. This operator only accepts 1 condition.
• seq: the list of conditions applied in the specified order, with any number of events occurring in-between.
• at_least_once: the listed marker definitions occurred at least once. Only the first occurrence will be marked.
• never: the listed marker definitions never occurred.

Marker Configuration

Here is an example of a marker configuration file containing several marker definitions. The example is created for mood bot, with a new slot name to illustrate the use of the label slot_was_set:

marker_name_provided:
  description: "slot `name` was provided"
  slot_was_set: name
marker_mood_expressed:
  or:
    - intent: mood_unhappy
    - intent: mood_great
marker_cheer_up_failed:
  seq:
    - intent: mood_unhappy
    - action: utter_cheer_up
    - action: utter_did_that_help
    - intent: deny
marker_bot_not_challenged:
  description: "Example of a negated marker, it can be used to surface conversations without bot_challenge intent"
  never:
    - intent: bot_challenge
marker_cheer_up_attempted:
  at_least_once:
    - action: utter_cheer_up
marker_mood_expressed_and_name_not_provided:
  and:
    - or:
        - intent: mood_unhappy
        - intent: mood_great
    - not:
        - slot_was_set: name

Note the following:

• Each marker has a unique identifier (or name) such as marker_name_provided.
• Each marker can have an optional description key that can be used for documentation.
• A marker definition can contain a single condition, as shown in marker_name_provided.
• A marker definition can contain a single operator with a list of conditions, as shown in marker_mood_expressed, marker_cheer_up_failed, marker_bot_not_challenged, and marker_cheer_up_attempted.
• A marker definition can contain nested operators, as shown in marker_mood_expressed_and_name_not_provided.
• The values assigned to event conditions must be valid according to your bot's domain.yml file. For example, in marker_mood_expressed, the intents mood_unhappy and mood_unhappy are both intents listed in the mood bot's domain.yml file.

Extracting Markers

Rasa Pro supports real-time processing of markers. Markers are extracted from dialogues already stored in a tracker store. To learn how to store interactions with your bot in a tracker store, read the Tracker Store page.

Once you've created your marker definitions in the marker configuration file, and have stored some dialogues in your tracker store, you can apply your markers to your trackers by running the following command:

rasa evaluate markers all --config markers.yml extracted_markers.csv

This script will process the marker definitions you provide in the marker configuration file: markers.yml. The script will output the extracted markers in the specified output file: extracted_markers.csv. It will also produce two summary statistics files. The format of the output files are described in the next section.