# Training Data Format

## Data Formats
You can provide training data as Markdown or as JSON, as a single file or as a directory containing multiple files. Note that Markdown is usually easier to work with.

### Markdown Format
Markdown is the easiest Rasa NLU format for humans to read and write. Examples are listed using the unordered list syntax, e.g. minus `-`, asterisk `*`, or plus `+`. Examples are grouped by intent, and entities are annotated as Markdown links, e.g. `[<entity text>](<entity name>)`, or by using the following syntax `[<entity-text>]{"entity": "<entity name>"}`. Using the latter syntax, you can also assign synonyms, roles, or groups to an entity, e.g. `[<entity-text>]{"entity": "<entity name>", "role": "<role name>", "group": "<group name>", "value": "<entity synonym>"}`. The keywords `role`, `group`, and `value` are optional in this notation. To understand what the labels `role` and `group` are for, see section [Entities Roles and Groups](https://legacy-docs-v1.rasa.com/1.10.26/nlu/entity-extraction/#entities-roles-groups).

```
## intent:check_balance
- what is my balance <!-- no entity -->
- how much do I have on my [savings](source_account) <!-- entity "source_account" has value "savings" -->
- how much do I have on my [savings account]{"entity": "source_account", "value": "savings"} <!-- synonyms, method 1-->
- Could I pay in [yen](currency)?  <!-- entity matched by lookup table -->

## intent:greet
- hey
- hello

## synonym:savings   <!-- synonyms, method 2 -->
- pink pig

## regex:zipcode
- [0-9]{5}

## lookup:additional_currencies  <!-- specify lookup tables in an external file -->
path/to/currencies.txt
```

### JSON Format
The JSON format consists of a top-level object called `rasa_nlu_data`, with the keys `common_examples`, `entity_synonyms` and `regex_features`. The most important one is `common_examples`.

```
{
    "rasa_nlu_data": {
        "common_examples": [],
        "regex_features" : [],
        "lookup_tables"  : [],
        "entity_synonyms": []
    }
}
```

The `common_examples` are used to train your model. You should put all of your training examples in the `common_examples` array. Regex features are a tool to help the classifier detect entities or intents and improve the performance.

## Improving Intent Classification and Entity Recognition
### Common Examples
Common examples have three components: `text`, `intent` and `entities`. The first two are strings while the last one is an array.

> - The _text_ is the user message \[required\]
> - The _intent_ is the intent that should be associated with the text \[optional\]
> - The _entities_ are specific parts of the text which need to be identified \[optional\]

Entities are specified with a `start` and an `end` value, which together make a range to apply to the string, e.g. in the example below, with `text="show me chinese restaurants"`, then `text[8:15] == 'chinese'`. Entities can span multiple words, and in fact the `value` field does not have to correspond exactly to the substring in your example. That way you can map synonyms, or misspellings, to the same `value`.

```
## intent:restaurant_search
- show me [chinese](cuisine) restaurants
```

### Regular Expression Features
Regular expressions can be used to support the intent classification and entity extraction. For example, if your entity has a deterministic structure (like a zipcode or an email address), you can use a regular expression to ease detection of that entity. For the zipcode example it might look like this:

```
## regex:zipcode
- [0-9]{5}

## regex:greet
- hey[^\\s]*
```

The name doesn’t define the entity nor the intent, it is just a human readable description for you to remember what this regex is used for and is the title of the corresponding pattern feature.

### Lookup Tables
Lookup tables provide a convenient way to supply a list of entity examples. The supplied lookup table files must be in a newline-delimited format. For example, `data/test/lookup_tables/plates.txt` may contain:

```
tacos
beef
mapo tofu
burrito
lettuce wrap
```

And can be loaded and used as shown here:

```
## lookup:plates
data/test/lookup_tables/plates.txt

## intent:food_request
- I'd like beef [tacos](plates) and a [burrito](plates)
- How about some [mapo tofu](plates)
```

When lookup tables are supplied in training data, the contents are combined into a large, case-insensitive regex pattern that looks for exact matches in the training examples. These regexes match over multiple tokens, so `lettuce wrap` would match `get me a lettuce wrap ASAP` as `[0 0 0 1 1 0]`. These regexes are processed identically to the regular regex patterns directly specified in the training data.

### Normalizing Data
#### Entity Synonyms
If you define entities as having the same value they will be treated as synonyms. Here is an example of that:

```
## intent:search
- in the center of [NYC]{"entity": "city", "value": "New York City"}
- in the centre of [New York City](city)
```

As you can see, the entity `city` has the value `New York City` in both examples, even though the text in the first example states `NYC`. By defining the value attribute to be different from the value found in the text between start and end index of the entity, you can define a synonym. Whenever the same text will be found, the value will use the synonym instead of the actual text in the message.
To use the synonyms defined in your training data, you need to make sure the pipeline contains the `EntitySynonymMapper` component.

Alternatively, you can add an "entity_synonyms" array to define several synonyms to one entity value. Here is an example of that:

```
## synonym:New York City
- NYC
- nyc
- the big apple
```

Note

Please note that adding synonyms using the above format does not improve the model’s classification of those entities. **Entities must be properly classified before they can be replaced with the synonym value.**

👋 I can help you get started with Rasa and answer your technical questions.