Clone the companion repository and move into the pattern:
git clone https://github.com/RasaHQ/rasa-community-resources.git
cd rasa-community-resources/patterns/evaluation-harnessCredentials and training
cp .env.example .env # fill in RASA_LICENSE and OPENAI_API_KEY
uv sync --prerelease=allow
uv run rasa trainWhile that trains, meet the lab rat. The agent is a retail-banking assistant with exactly two skills:
check_balance— a task skill: disambiguate which account, call a tool, read back a number. Its tool returns hard-coded balances (skills/check_balance/tools.py): the Everyday Checking account contains $1,284.53 today, tomorrow, and forever, and Rainy Day Savings holds $9,140.00.faq_support— a knowledge skill: answer questions about overdraft fees and card replacement from hard-coded policy text (skills/faq_support/tools.py).
Hard-coded is the point. When a scenario fails against this agent, there is exactly one explanation: the agent’s behaviour changed. No stale database, no flaky API, no timezone. Every eval suite you build for a real agent will have to fight for that property; the fixture gets it for free, which makes it the right place to learn what the instruments measure.
The suite you just cloned
Everything lives under eval/:
eval/
├── conftest.yml # the two evaluation LLMs: simulator + judge
└── scenarios/ # one behaviour per file, seven in alleval/conftest.yml configures the user simulator (which generates the
customer’s turns) and the judge (which scores the transcript)
independently — different jobs, separately swappable. The agent’s own model
lives in integrations.yml; changing that is an experiment, changing the
judge changes what “pass” means.
The seven scenarios are the standard Mantle behaviour checklist — happy path, out-of-order input, disambiguation, correction, digression-and-resume, refusal instead of fabrication, and a negative case where the right behaviour is to start nothing at all.
Run one scenario
Scenarios run through Rasa’s MCP server, driven from your coding agent (per-editor setup in the simulation docs):
uv run rasa tools run --mode stdio # loads this project's .envThen ask your coding agent, in natural language:
Run the balance_named_up_front scenario in eval/scenarios/.
Three things happen: the simulator reads the scenario’s
simulation_context and plays the customer, turn by turn, against your
trained agent; the finished transcript is checked against the scenario’s
assertions (deterministic, tracker-level); and the judge scores its
criteria (natural language). Results land under
eval/results/<timestamp>/ — a run_N.txt per run with the transcript,
every assertion’s PASS/FAIL, every criterion’s score with the judge’s
rationale, and an Inspector URL to replay the conversation, plus a
summary.txt across scenarios.
A run passes only when every assertion and every criterion passes. Quality metrics (helpfulness, task completion and friends) are recorded but do not gate.
Two observations before the dissection
One run is three LLMs’ worth of tokens. The agent, the simulator, and the judge all bill on every run. There is no free tier of this instrument — which is why you iterate on one scenario with a run count of 1, and save “run everything five times” for decisions that deserve it.
The same run produces two kinds of evidence. The assertion results are exact and repeatable-in-principle; the criteria scores are an LLM’s opinion with variance attached. Keeping straight which kind you are reading is most of the skill, and the next three chapters take them one at a time.
Check your understanding
- Why are hard-coded balances a feature of an eval fixture rather than laziness?
- Which three models does a single scenario run pay for, and which one’s swap would change what “pass” means?
- Where do you look first when a run fails: the transcript, the assertion results, or the criteria rationale — and why?
Got a pass report on your screen? Good. Now let’s find out what, precisely, you just proved.