Training Data Format

You are viewing documentation for our open source project which is maintained by the community. If you want to get started building assistants with Rasa, please check out our latest documentation here.

Overview

Rasa Open Source uses YAML as a unified and extendable way to manage all training data, including NLU data, stories, and rules.

You can split the training data over any number of YAML files, and each file can contain any combination of NLU data, stories, and rules. The training data parser determines the training data type using top-level keys.

The domain uses the same YAML format as the training data and can also be split across multiple files or combined in one file. The domain includes the definitions for responses and forms. See the documentation for the domain for information on how to format your domain file.

High-Level Structure

Each file can contain one or more keys with corresponding training data. One file can contain multiple keys, but each key can only appear once in a single file. The available keys are:

• version
• nlu
• stories
• rules

You should specify the version key in all YAML training data files. If you don't specify a version key in your training data file, Rasa will assume you are using the latest training data format specification supported by the version of Rasa Open Source you have installed. Training data files with a Rasa Open Source version greater than the version you have installed on your machine will be skipped. Currently, the latest training data format specification for Rasa 2.x is 2.0.

Example

Here's a short example which keeps all training data in a single file:

version: "2.0"

nlu:
  - intent: greet
    examples: |
      - Hey
      - Hi
      - hey there [Sara](name)
  - intent: faq/language
    examples: |
      - What language do you speak?
      - Do you only handle english?

stories:
  - story: greet and faq
    steps:
      - intent: greet
      - action: utter_greet
      - intent: faq
      - action: utter_faq

rules:
  - rule: Greet user
    steps:
      - intent: greet
      - action: utter_greet

To specify your test stories, you need to put them into a separate file:

# tests/test_stories.yml
stories:
  - story: greet and ask language
    steps:
      - user: |
          hey
        intent: greet
      - action: utter_greet
      - user: |
          what language do you speak
        intent: faq/language
      - action: utter_faq

NLU Training Data

NLU training data consists of example user utterances categorized by intent. Training examples can also include entities. Entities are structured pieces of information that can be extracted from a user's message. You can also add extra information such as regular expressions and lookup tables to your training data to help the model identify intents and entities correctly.

NLU training data is defined under the nlu key. Items that can be added under this key are:

• Training examples grouped by user intent e.g. optionally with annotated entities

nlu:
  - intent: check_balance
    examples: |
      - What's my [credit](account) balance?
      - What's the balance on my [credit card account]{"entity":"account","value":"credit"}

• Synonyms

nlu:
  - synonym: credit
    examples: |
      - credit card account
      - credit account

• Regular expressions

nlu:
  - regex: account_number
    examples: |
      - \d{10,12}

• Lookup tables

nlu:
  - lookup: banks
    examples: |
      - JPMC
      - Comerica
      - Bank of America

Entities

Entities are structured pieces of information that can be extracted from a user's message. Entities are annotated in training examples with the entity's name. In addition to the entity name, you can annotate an entity with synonyms, roles, or groups.

In training examples, entity annotation would look like this:

nlu:
  - intent: check_balance
    examples: |
      - how much do I have on my [savings](account) account
      - how much money is in my [checking]{"entity": "account"} account
      - What's the balance on my [credit card account]{"entity":"account","value":"credit"}

Synonyms

Synonyms normalize your training data by mapping an extracted entity to a value other than the literal text extracted. You can define synonyms using the format:

nlu:
  - synonym: credit
    examples: |
      - credit card account
      - credit account

Regular Expressions

You can use regular expressions to improve intent classification and entity extraction using the RegexFeaturizer and RegexEntityExtractor components.

nlu:
  - regex: account_number
    examples: |
      - \d{10,12}

Conversation Training Data

Stories and rules are both representations of conversations between a user and a conversational assistant. They are used to train the dialogue management model. Stories are used to train a machine learning model to identify patterns in conversations and generalize to unseen conversation paths. Rules describe small pieces of conversations that should always follow the same path and are used to train the RulePolicy.

Stories

Stories are composed of:

• story: The story's name. The name is arbitrary and not used in training; you can use it as a human-readable reference for the story.
• metadata: arbitrary and optional, not used in training, you can use it to store relevant information about the story like e.g. the author.
• a list of steps: The user messages and actions that make up the story.

For example:

stories:
  - story: Greet the user
    metadata:
      author: Somebody
      key: value
    steps:
      - intent: greet
      - action: utter_greet

User Messages

All user messages are specified with the intent: key and an optional entities: key. While writing stories, you do not have to deal with the specific contents of the messages that the users send. Instead, you can take advantage of the output from the NLU pipeline, which uses a combination of an intent and entities to refer to all possible messages the users can send with the same meaning.

User messages follow the format:

stories:
  - story: user message structure
    steps:
      - intent: intent_name  # Required
        entities:  # Optional
          - entity_name: entity_value
      - action: action_name

Rules

Rules are listed under the rules key and look similar to stories. A rule also has a steps key, which contains a list of the same steps as stories do. Rules can additionally contain the conversation_started and conditions keys. These are used to specify conditions under which the rule should apply.

A rule that has a condition looks like this:

rules:
  - rule: Only say `hey` when the user provided a name
    condition:
      - slot_was_set:
          - user_provided_name: true
    steps:
      - intent: greet
      - action: utter_greet

Test Stories

Test stories check if a message is classified correctly as well as the action predictions. Test stories use the same format as stories, except that user message steps can include a user to specify the actual text and entity annotations of the user message.

End-to-end Training

End-to-end training is an experimental feature. With end-to-end training, you do not have to deal with the specific intents of the messages that are extracted by the NLU pipeline. Instead, you can put the text of the user message directly in the stories, by using user key.

This summarizes the structured training data formats used in Rasa Open Source. For more detailed examples and explanations, please refer to the respective sections of the documentation.