Skip to content
RasaGet Developer Edition

tutorial

Chapter 0 of 5

Measure Your Agent: Simulation, Assertions, and LLM Judges

by Rod Rivera Published

You changed a prompt and the demo went fine. Is the agent better? Build a simulation-based evaluation suite that answers with evidence — an LLM plays your customer, assertions stay the ground truth, and a judge scores what facts cannot reach.

You changed a prompt. You ran the demo conversation. It went fine.

Is the agent better?

You genuinely do not know. The demo went fine last time too, before the bug report. And the change you made touched a model whose output varies between identical runs — so even “it went fine twice” is weaker evidence than it feels like.

This tutorial builds the thing that replaces that feeling with evidence: a simulation-based evaluation suite. Not a metaphorical one — the companion pattern is a runnable project you will clone, train, and evaluate.

Three questions, one framework

The central idea — and the thing most teams get wrong — is that “is the agent good?” is actually three different questions:

QuestionAnswered byCharacter
Did the agent do the right thing?assertions — facts about the trackerExact, repeatable, no judge noise
Did it handle the user well?criteria — scored by an LLM judgeTolerates paraphrase; costs variance and money
Does it survive a real, unscripted user?the LLM user-simulatorImprovises the customer’s side from stage directions

On Rasa Mantle all three live in one framework: you author scenarios, a simulator LLM plays the customer, and the finished transcript is scored twice — deterministically against tracker events, and by a judge against your natural-language criteria.

Why a simulator at all? Because Mantle skills are not scripts. A skill.md gives the model intent and constraints, not a dialogue path — so a test file of hard-coded user turns exercises one path through a system whose defining property is that it improvises paths. The simulator meets the agent on its own terms. The rule that keeps the whole thing honest underneath:

Assert everything you can express as a fact about the conversation. Reach for the judge only where there is no fact to assert.

Teams that invert this end up with a slow, expensive, flaky suite that measures their judge’s mood as much as their agent’s behaviour.

The two paths do not read the same thing, and that is the detail worth carrying into Chapter 2:

One scenario file feeds three things. Its simulation_context drives a simulator LLM that plays the customer turn by turn against the trained agent, and that loop produces two separate artefacts: the conversation tracker, a structural record, and the transcript, what was said. Assertions are checked against the tracker with no LLM involved, so they are exact and repeatable. Criteria are scored by a judge LLM reading the transcript, which tolerates paraphrase but costs variance and money. A run passes only when every assertion and every criterion passes. Every run bills three models: the agent's own from integrations.yml, and the simulator and judge from eval/conftest.yml.

The agent is a fixture, on purpose

The companion project ships a two-skill retail-banking assistant with hard-coded data. It is deliberately boring, and the boringness is a design decision worth stealing: a failing scenario should always mean the agent changed, never that the data moved. Wire an eval suite to a live database and every flaky run becomes a debugging session about the data. The harness is the subject here; the agent is the lab rat.

The chapters

#ChapterThe question it teaches you to answer
1Set up the labDoes it run on my machine, and what does a passing suite look like?
2AssertionsDid the agent do the right thing — exactly, repeatably, judge-free?
3The simulatorDoes the agent survive a customer who never read your script?
4The judgeWas the answer grounded and relevant — and what do those words mean?
5What your eval cannot tell youWhen is a green run evidence, and when is it noise?

Chapter 5 is short and it is the one to not skip. An eval that oversells itself is worse than no eval, because it launders a guess into a number.

Before you start

  • Python 3.11 or 3.12, uv, and git.
  • A free Rasa Pro Developer Edition licence and an OpenAI API key. You need both from Chapter 1 — evaluating an agent requires running one.
  • A coding agent that can drive an MCP server (Claude Code, Cursor, or Copilot) — scenarios are run through Rasa’s MCP server, per the simulation docs.
  • A cost note up front, and it is blunter than it used to be: every simulated run bills three LLMs — the agent’s own model, the simulator, and the judge. It is cents per run, not dollars — but knowing that no part of this instrument is free is half of what Chapter 5 teaches. You iterate with small run counts and save the big batches for decisions.

Everything here was verified against rasa-pro==3.20.0.dev6, and — a convention this series inherits from the companion pattern — every claim about engine behaviour cites the engine source file it was read from. When you wonder “but how does it actually score that?”, the file and line are right there.