## Introduction Simulation and evaluation gives you a way to test your agent's behavior before shipping, without manually replaying conversations every time you make a change. You write a scenario in YAML — or let the agent generate one from your flow definition — describing who the simulated user is, what they're trying to accomplish, and what success looks like. Rasa drives full multi-turn conversations automatically, then evaluates whether your agent met those goals. This covers the full assertion set supported by [E2E testing](/content/docs/pro/testing/evaluating-assistant/index.html) and adds LLM-as-judge scoring on top. Where evaluations show their real strength is with **agents that have some degree of autonomy**: E2E tests require you to specify every step of the conversation upfront, which works well for fully deterministic flows but becomes impractical as soon as the agent can make independent decisions — choosing between paths, generating responses, or invoking subagents. ## Prerequisites Before running evaluations, make sure you have the following in place: - Rasa Pro installed with the `rasa tools` component enabled. - The `rasa-simulating-conversations` agent skill installed. Run `rasa tools init` in your project root to install it alongside other skills — no extra steps needed. - To use a natural language interface for generating scenarios and running evaluations, you need an IDE agent (such as GitHub Copilot, Cursor, or any MCP-compatible AI agent) connected to the `rasa tools run` FastMCP server. For more details, refer to the documentation on the [Rasa MCP Tools](/content/docs/pro/installation/rasa-mcp-tools/index.html) - This skill has been most extensively tested with Claude Code (Sonnet 4.6) and Cursor with Composer 2.5. Either is a good starting point if you are unsure which to use. - The `rest` and `inspector` channel configured in your `credentials.yml`. - If you want to inspect simulation results in the [Inspector](/content/docs/pro/testing/trying-assistant/index.html), start your server with the `--inspect` flag: ```bash rasa run --inspect ``` ## Quick Start Once the skill is installed and your server is running, type a prompt into your AI coding assistant (such as GitHub Copilot, Cursor, or any MCP-compatible agent). You can generate a single scenario: ```text Simulate a happy path scenario for the add_contact skill. ``` Or generate multiple scenarios at once: ```text Generate and run scenarios for the billing skill: a happy path, a case where the user provides incorrect details, and a cancellation mid-flow. ``` The AI coding assistant will: 1. Check whether `eval/conftest.yml` exists in your project. If it doesn't, it creates a starter template and pauses for you to fill in your LLM provider details. 2. Read your flow definition to understand available slots, actions, and branching logic. 3. Write one or more scenario YAMLs to `eval/scenarios/`. 4. Call `validate_scenario` to check each scenario file for syntax and domain errors before running. 5. Call `evaluate_agent` to run the simulation and evaluation and return a pass/fail summary with a link to each result file. ## Writing Scenarios Scenarios live in `eval/scenarios/` as individual YAML files. Each scenario describes who the simulated user is, how the conversation should unfold, and what success looks like. ### Generating Scenarios with the Agent Instead of writing scenarios by hand, ask the agent to generate them. The skill reads your flow definition and applies evaluation best practices to produce grounded, useful scenarios. **Minimal** — generates a first batch from the flow definition alone: ```text Generate scenarios for the add_contact skill. ``` **With direction** — specify the types of scenarios you want: ```text Generate scenarios for the billing skill: a happy path, a multi-turn conversation, and a case where the user provides incorrect details. ``` **Detailed** — provide specific goals, assertions, or personas: ```text Generate a scenario where an impatient customer asks about a delayed order (ORD-9981). Assert that the check_order_status flow completes and that the agent never asks for information the user already provided. ``` Scenario generation and simulation are separate steps. After the agent writes YAMLs to `eval/scenarios/`, you can edit them manually before running. Use this to refine criteria, adjust assertions, or add edge cases. ### Full Scenario Reference #### Sample YAML Structure eval/scenarios/order_delay.yaml ```yaml scenario: name: Agent handles delayed order gracefully simulation_context: > You are an impatient customer who escalates quickly if not given a direct answer. You want to find out why the order is delayed and get a clear resolution. setup: initial_slots: authenticated: true account_id: "12345" order_id: "ORD-9981" goals: criteria: - Agent communicates the delay reason clearly and empathetically - Agent offers an alternative resolution when human handoff fails - Agent does not ask the user to repeat information already provided assertions: - flow_started: check_order_status - slot_was_set: name: order_id value: "ORD-9981" - flow_completed: check_order_status - bot_did_not_utter: "utter_dont_know" - sequencing: flow_started: verify_identity flow_completed: cancel_account ``` ## Configuring Evaluations The `eval/conftest.yml` file controls which LLM models power the simulation and evaluation steps. Create it at the root of your project: eval/conftest.yml ```yaml simulation: llm: provider: openai model: gpt-5.1 evaluation: llm: provider: openai model: gpt-5.1 ``` The `simulation.llm` model drives the simulated user turns. The `evaluation.llm` model acts as the judge, scoring each conversation against your quality criteria and computing quality metrics. Configure them independently — a smaller, cheaper model works well for simulation while a more capable model is recommended for judging. ## Running Evaluations Use the `evaluate_agent` MCP tool by asking your IDE agent: ```text Run the order_delay scenario 3 times. ``` ```text Run all scenarios in eval/scenarios/. ``` You control how many times each scenario is simulated. Running a scenario multiple times (N=3 or more) accounts for LLM non-determinism and gives a more reliable signal — a scenario that passes 3/3 is more meaningful than one that passes 1/1. ## Reading Results Results are written to `eval/results//` and preserved across runs: ```text eval/ conftest.yml scenarios/ order_delay.yaml results/ 2026-05-28_10-22-00/ summary.txt order_delay/ run_1.txt run_2.txt run_3.txt ``` Each `run_N.txt` contains the full evaluation output for one simulated conversation: ```text scenario: order_delay run: 2 timestamp: 2026-05-28T10:22:11Z conversation_id: sim-1f6de497-8e1a-45b6-8ad4-e8d68a229c50 overall_result: FAIL --- Quality Criteria Results --- [PASS] Agent communicates the delay reason clearly and empathetically rationale: Agent acknowledged the weather delay and expressed understanding before offering alternatives. [FAIL] Agent does not ask the user to repeat information already provided rationale: Agent re-requested the order ID on turn 4, which the user had already provided on turn 1. --- Quality Metrics --- bot_quality: 3.5/5 helpfulness: 4/5 - The agent provided a clear explanation and offered an alternative resolution. task_completion: 3/5 - The user's goal was partially met — the delay reason was communicated but the resolution step failed. [... additional metrics ...] summary: The agent handled the core inquiry well but introduced unnecessary friction by asking for information the user had already provided. --- Assertion Results --- [PASS] flow_started(check_order_status) [PASS] slot_was_set(order_id) [PASS] flow_completed: check_order_status [FAIL] bot_did_not_utter: Bot uttered a forbidden utterance 'utter_dont_know'. --- Raw Transcript --- [1] user: Hey, where's my order ORD-9981? [2] agent: Let me check that for you. Your order is delayed due to weather — estimated arrival in 3 days. [3] user: That's not good enough. Can I speak to someone? [4] agent: I understand your frustration. Could you share your order ID again? --- Inspector URL --- http://localhost:5005/webhooks/inspector/inspect.html?sender=sim-f3a1b2c4 ``` ## Evaluations vs. E2E Tests Both tools test your agent, but they serve different purposes. Use whichever fits the flow you're testing — or both in parallel. | | E2E Tests | Evaluations | | --- | --- | --- | | **Best for** | Deterministic, scripted flows with a fixed expected path | Autonomous LLM-driven agents where the path is hard to fully pre-define | | **When to use** | CI pipelines, regression checks on controlled business logic | Build loop — testing whether your agent achieves its goals while iterating | | **Pass/fail signal** | Binary per step | LLM judge scores + deterministic assertions combined | | **Suitable for CI?** | Yes | No | Evaluations are not yet suitable for blocking CI pipelines. Keep your E2E tests in CI for now and use evaluations in your local build loop.