Simulating and Evaluating Your Agent | Rasa Documentation

Introduction

Simulation and evaluation gives you a way to test your agent's behavior before shipping, without manually replaying conversations every time you make a change. You write a scenario in YAML — or let the agent generate one from your flow definition — describing who the simulated user is, what they're trying to accomplish, and what success looks like. Rasa drives full multi-turn conversations automatically, then evaluates whether your agent met those goals.

This covers the full assertion set supported by E2E testing and adds LLM-as-judge scoring on top. Where evaluations show their real strength is with agents that have some degree of autonomy: E2E tests require you to specify every step of the conversation upfront, which works well for fully deterministic flows but becomes impractical as soon as the agent can make independent decisions — choosing between paths, generating responses, or invoking subagents.

Prerequisites

Before running evaluations, make sure you have the following in place:

rasa run --inspect

Quick Start

Once the skill is installed and your server is running, type a prompt into your AI coding assistant (such as GitHub Copilot, Cursor, or any MCP-compatible agent). You can generate a single scenario:

Simulate a happy path scenario for the add_contact skill.

Or generate multiple scenarios at once:

Generate and run scenarios for the billing skill: a happy path, a case where the user provides incorrect details, and a cancellation mid-flow.

The AI coding assistant will:

  1. Check whether eval/conftest.yml exists in your project. If it doesn't, it creates a starter template and pauses for you to fill in your LLM provider details.
  2. Read your flow definition to understand available slots, actions, and branching logic.
  3. Write one or more scenario YAMLs to eval/scenarios/.
  4. Call validate_scenario to check each scenario file for syntax and domain errors before running.
  5. Call evaluate_agent to run the simulation and evaluation and return a pass/fail summary with a link to each result file.

Writing Scenarios

Scenarios live in eval/scenarios/ as individual YAML files. Each scenario describes who the simulated user is, how the conversation should unfold, and what success looks like.

Generating Scenarios with the Agent

Instead of writing scenarios by hand, ask the agent to generate them. The skill reads your flow definition and applies evaluation best practices to produce grounded, useful scenarios.

Minimal — generates a first batch from the flow definition alone:

Generate scenarios for the add_contact skill.

With direction — specify the types of scenarios you want:

Generate scenarios for the billing skill: a happy path, a multi-turn conversation,
and a case where the user provides incorrect details.

Detailed — provide specific goals, assertions, or personas:

Generate a scenario where an impatient customer asks about a delayed order (ORD-9981).
Assert that the check_order_status flow completes and that the agent never asks for
information the user already provided.

Scenario generation and simulation are separate steps. After the agent writes YAMLs to eval/scenarios/, you can edit them manually before running. Use this to refine criteria, adjust assertions, or add edge cases.

Full Scenario Reference

Sample YAML Structure

eval/scenarios/order_delay.yaml

scenario:
  name: Agent handles delayed order gracefully

simulation_context: >
    You are an impatient customer who escalates quickly if not given a direct answer.
    You want to find out why the order is delayed and get a clear resolution.

setup:
    initial_slots:
      authenticated: true
      account_id: "12345"
      order_id: "ORD-9981"

goals:
    criteria:
      - Agent communicates the delay reason clearly and empathetically
      - Agent offers an alternative resolution when human handoff fails
      - Agent does not ask the user to repeat information already provided
    assertions:
      - flow_started: check_order_status
      - slot_was_set:
          name: order_id
          value: "ORD-9981"
      - flow_completed: check_order_status
      - bot_did_not_utter: "utter_dont_know"
      - sequencing:
          flow_started: verify_identity
          flow_completed: cancel_account

Configuring Evaluations

The eval/conftest.yml file controls which LLM models power the simulation and evaluation steps. Create it at the root of your project:

eval/conftest.yml

simulation:
  llm:
    provider: openai
    model: gpt-5.1

evaluation:
  llm:
    provider: openai
    model: gpt-5.1

The simulation.llm model drives the simulated user turns. The evaluation.llm model acts as the judge, scoring each conversation against your quality criteria and computing quality metrics. Configure them independently — a smaller, cheaper model works well for simulation while a more capable model is recommended for judging.

Running Evaluations

Use the evaluate_agent MCP tool by asking your IDE agent:

Run the order_delay scenario 3 times.

Run all scenarios in eval/scenarios/.

You control how many times each scenario is simulated. Running a scenario multiple times (N=3 or more) accounts for LLM non-determinism and gives a more reliable signal — a scenario that passes 3/3 is more meaningful than one that passes 1/1.

Reading Results

Results are written to eval/results/<timestamp>/ and preserved across runs:

eval/
  conftest.yml
  scenarios/
    order_delay.yaml
  results/
    2026-05-28_10-22-00/
      summary.txt
      order_delay/
        run_1.txt
        run_2.txt
        run_3.txt

Each run_N.txt contains the full evaluation output for one simulated conversation:

scenario: order_delay
run: 2
timestamp: 2026-05-28T10:22:11Z
conversation_id: sim-1f6de497-8e1a-45b6-8ad4-e8d68a229c50
overall_result: FAIL

--- Quality Criteria Results ---
[PASS] Agent communicates the delay reason clearly and empathetically
  rationale: Agent acknowledged the weather delay and expressed understanding before offering alternatives.

[FAIL] Agent does not ask the user to repeat information already provided
  rationale: Agent re-requested the order ID on turn 4, which the user had already provided on turn 1.

--- Quality Metrics ---
bot_quality: 3.5/5
  helpfulness: 4/5 - The agent provided a clear explanation and offered an alternative resolution.
  task_completion: 3/5 - The user's goal was partially met — the delay reason was communicated but the resolution step failed.
[... additional metrics ...]
summary: The agent handled the core inquiry well but introduced unnecessary friction by asking for
         information the user had already provided.

--- Assertion Results ---
[PASS] flow_started(check_order_status)
[PASS] slot_was_set(order_id)
[PASS] flow_completed: check_order_status
[FAIL] bot_did_not_utter: Bot uttered a forbidden utterance 'utter_dont_know'.

--- Raw Transcript ---
[1] user: Hey, where's my order ORD-9981?
[2] agent: Let me check that for you. Your order is delayed due to weather — estimated arrival in 3 days.
[3] user: That's not good enough. Can I speak to someone?
[4] agent: I understand your frustration. Could you share your order ID again?

--- Inspector URL ---
http://localhost:5005/webhooks/inspector/inspect.html?sender=sim-f3a1b2c4

Evaluations vs. E2E Tests

Both tools test your agent, but they serve different purposes. Use whichever fits the flow you're testing — or both in parallel.

E2E Tests Evaluations
Best for Deterministic, scripted flows with a fixed expected path Autonomous LLM-driven agents where the path is hard to fully pre-define
When to use CI pipelines, regression checks on controlled business logic Build loop — testing whether your agent achieves its goals while iterating
Pass/fail signal Binary per step LLM judge scores + deterministic assertions combined
Suitable for CI? Yes No

Evaluations are not yet suitable for blocking CI pipelines. Keep your E2E tests in CI for now and use evaluations in your local build loop.