Build your first agent in just a few minutes with [Rasa Copilot](https://hello.rasa.com/?utm_source=docs&utm_medium=referral&utm_campaign=docs_cta).

A scenario describes a single test case for your agent: who the simulated user is, what they want to accomplish, and what a successful conversation looks like. Each scenario is an individual YAML file in `eval/scenarios/`.

A scenario has three parts:

- **`simulation_context`** — a natural language briefing that tells the LLM how to play the user role across the whole conversation.
- **`setup`** — optional initial slot values injected before the conversation starts (e.g. pre-authenticated state).
- **`goals`** — what success looks like, expressed as deterministic `assertions` (checked against the tracker event history) and natural language `criteria` (scored by an LLM judge).

## Schema

eval/scenarios/order_delay.yml

```yaml
scenario:
  name: Agent handles delayed order gracefully

simulation_context: >
    You are an impatient customer who escalates quickly if not given a direct answer.
    You want to find out why the order is delayed and get a clear resolution.

setup:
    initial_slots:
      authenticated: true
      account_id: "12345"
      order_id: "ORD-9981"

goals:
    criteria:
      - Agent communicates the delay reason clearly and empathetically
      - Agent does not ask the user to repeat information already provided
    assertions:
      - flow_started:
          flow_ids: [check_order_status]
          operator: any
      - flow_completed:
          flow_id: check_order_status
      - flow_cancelled:
          flow_id: verify_identity
      - action_executed: action_check_order_status
      - slot_was_set:
          - name: order_id
            value: "ORD-9981"
      - slot_was_not_set:
          - name: transfer_amount
      - bot_uttered:
          utter_name: utter_order_delayed
      - bot_did_not_utter:
          utter_name: utter_dont_know
      - pattern_clarification_contains:
          flow_ids: [check_order_status, track_shipment]
          operator: any
      - generative_response_is_relevant:
          utter_source: utter_order_status_details
      - generative_response_is_grounded:
          utter_source: utter_order_status_details
      - sequencing:
          - flow_started: verify_identity
          - action_executed: action_verify_account
          - slot_was_set: authenticated
          - flow_completed: verify_identity
          - flow_started: check_order_status
          - flow_completed: check_order_status
```

## Field Reference

### `simulation_context`

A free-form natural language description of the simulated user and the conversation goal. The LLM uses this to generate consistent user turns across the whole conversation. You can include user behaviour (emotional state, interaction style), conversational intent, and contextual facts (account state, order details).

A single sentence works. The more specific you are, the more targeted the simulation.

### `setup.initial_slots`

Slot values injected into the conversation before the first user turn. Use this to pre-authenticate users, pre-fill context, or skip onboarding steps that are out of scope.

When `initial_slots` are set, the evaluator sends a `/session_start` message first so that any custom `action_session_start` logic runs against an empty tracker, then injects the slot values on top.

Slot names must exist in your domain with compatible types. `validate_scenario` checks this before any simulation runs.

### `goals.criteria`

A list of natural language statements describing required bot-side behavior. The LLM judge evaluates each criterion independently after the conversation ends and produces a pass/fail verdict with a written rationale. A run is marked failed if any criterion fails.

Use criteria for deal-breakers — non-negotiable system behavior such as correct step ordering, mandatory tool calls, flow adherence, or required phrasing. Do not use criteria for subjective qualities like tone or empathy, or for overall user goal achievement; those are captured by quality metrics and do not gate the pass/fail verdict.

If you omit `criteria`, the run still produces a full transcript and assertion results.

### `goals.assertions`

Deterministic, binary checks run against the conversation's full event history. These use the same assertion types as E2E tests, with the addition of the `sequencing` type:

| Assertion                | Checks                                      | YAML value  |
|--------------------------|---------------------------------------------|-------------|
| `flow_started`           | A specific flow was started                  | [`{flow_ids: [...], operator: any|all}`](/content/docs/reference/testing/e2e/assertions/#flow-started-assertion/index.html) |
| `flow_completed`         | A flow reached a completed state             | [`{flow_id: "...", flow_step_id: "..."}`](/content/docs/reference/testing/e2e/assertions/#flow-completed-assertion/index.html) (`flow_step_id` optional) |
| `flow_cancelled`         | A flow was cancelled                          | [`{flow_id: "...", flow_step_id: "..."}`](/content/docs/reference/testing/e2e/assertions/#flow-cancelled-assertion/index.html) (`flow_step_id` optional) |
| `action_executed`        | A specific action was triggered              | [plain string — action name](/content/docs/reference/testing/e2e/assertions/#action-executed-assertion/index.html) |
| `slot_was_set`           | A slot was set (optionally to a specific value) | [list of `{name: "...", value: ...}`](/content/docs/reference/testing/e2e/assertions/#slot-was-set-assertion/index.html) (`value` optional) |
| `slot_was_not_set`       | A slot was not set, or was not set to a specific value | [list of `{name: "...", value: ...}`](/content/docs/reference/testing/e2e/assertions/#slot-was-not-set-assertion/index.html) (`value` optional) |
| `bot_uttered`            | The agent produced a response matching text, buttons, or a response name | [`{utter_name: "..."}` or `{text_matches: "..."}` or `{buttons: [...]}`](/content/docs/reference/testing/e2e/assertions/#bot-uttered-assertion/index.html) |
| `bot_did_not_utter`      | The agent did not produce a response matching a pattern | [same fields as `bot_uttered`](/content/docs/reference/testing/e2e/assertions/#bot-did-not-utter-assertion/index.html) |
| `pattern_clarification_contains` | The clarification suggestions included the expected flow IDs | [`{flow_ids: [...], operator: any|all}`](/content/docs/reference/testing/e2e/assertions/#pattern-clarification-contains-assertion/index.html) |
| `generative_response_is_relevant` | A generative response is contextually relevant to the user's message (LLM-scored) | [`{utter_source: "..."}`](/content/docs/reference/testing/e2e/assertions/#generative_response_is_relevant-assertion/index.html) — `threshold` and `utter_name` optional |
| `generative_response_is_grounded` | A generative response is factually accurate against its source material (LLM-scored) | [`{utter_source: "..."}`](/content/docs/reference/testing/e2e/assertions/#generative_response_is_grounded-assertion/index.html) — `threshold`, `utter_name`, `ground_truth` optional |
| `sequencing`             | A sequence of events occurred in the specified order | list of single-key step dicts |

#### `sequencing`

This takes a list of step matchers. Each step must match an event that appears strictly after the previous match in the conversation. All values are plain strings.