Chapter 3 — The Simulator: A Customer Who Never Read Your Script

Every test file you have ever written hard-codes the user’s side of the conversation. For a scripted system that is fine — the system’s paths are enumerable, so your turns can enumerate them.

A Mantle skill is not a scripted system. skill.md gives the model intent and constraints — disambiguate which account, never invent a balance — and the model improvises the path. Two runs of the same conversation can legally take different routes to the same correct outcome. A test file with hard-coded user turns exercises exactly one path through a system whose defining property is that it improvises paths; it under-tests precisely the behaviour you bought Mantle for.

So the framework replaces the script with an actor: an LLM user-simulator that plays your customer, one scenario at a time.

Stage directions, not scripts

The simulator reads one prose block per scenario — simulation_context — and improvises from it. From eval/scenarios/balance_digression_faq_resume.yml:

simulation_context: >
  You are a curious bank customer. Start by asking for an account balance
  without naming an account. When the agent asks which account, do NOT
  answer; instead ask "wait, first — how much is the overdraft fee?". After
  you get the fee answer, say "ok, the checking one" to return to the
  balance question. Accept the balance, thank the agent, and end the
  conversation.

Write these like stage directions for an actor: persona (who the customer is, their temperament), intent (what they want, what they will answer when asked, what they refuse), and a clear end condition. Give the simulator enough to play the part — then let it choose the words. The words being chosen fresh on every run is the entire point: it is what stops your suite from quietly testing only the phrasings you thought of.

What this makes testable

Look at what the checklist scenarios in the companion actually exercise:

• Digression and resume (balance_digression_faq_resume.yml) — the customer interrupts one skill with a question for another, then comes back. Where the interruption lands varies by run; whether the agent finds its way back is asserted deterministically (flow_started for both skills, then the balance facts).
• Correction (balance_correction.yml) — “actually, the savings one please”, expressed as a sequencing assertion: the same slot set, then re-set.
• Out-of-order input (balance_named_up_front.yml) — the customer volunteers the account before being asked, and a criterion checks the agent did not re-ask for it.

These are exactly the behaviours a scripted file under-tests, because their interesting part is where in the conversation they happen — the thing you cannot hard-code without deciding it in advance.

The old absence habit survives the move, upgraded: the ambiguous-request scenario used to assert “no slot was set after turn 1” against a scripted turn. Now the requirement “the agent asks before answering, and does not guess” lives in criteria — judged from wherever in the transcript the question actually landed — while the assertions pin the end state: the right account, the tool ran, the right number spoken.

What happened to dialogue-understanding tests

Earlier versions of this series taught rasa test du here — annotated per-turn command tests for the CALM command generator. They are gone from the companion, for a reason worth reading because it is a lesson in trusting mechanisms over labels:

DU tests score the command generator — CALM’s component that turns each user message into commands like StartFlow(x). The CLI hard-gates them to CALM assistants (rasa/cli/dialogue_understanding_test.py:185-199). A Mantle agent, however, slips that gate: it declares is_calm_assistant = True (rasa/mantle/processor.py:104-107) — and then the tests execute against a component that never runs, because Mantle’s turn loop is an LLM tool-calling orchestrator that imports nothing from rasa.dialogue_understanding (rasa/mantle/orchestration/orchestrator.py). A suite that runs, produces scores, and measures a component your agent does not use is strictly worse than one that refuses to run.

The question DU tests answered — did the agent understand, separately from whether the conversation ended well? — has not disappeared. It has moved: understanding failures now surface as failed assertions on memory (slot_was_set with the wrong value is a misread) and as criteria rationales that name the turn where the transcript went sideways. The Inspector URL in every run report replays the conversation for exactly this kind of diagnosis.

Variance is a feature with a bill attached

The simulator’s freshness cuts both ways. The same scenario produces different transcripts on every run — which is honest coverage of a non-scripted system, and which means a single green run is weak evidence and a single red run near a boundary may be the dice. Chapter 5 turns this into discipline; for now, the habit: when a scenario surprises you, raise its run count before concluding. Your coding agent will happily “run balance_correction 5 times” — remembering that each of those runs bills the agent, the simulator, and the judge.

Check your understanding

• Why does hard-coding user turns under-test a Mantle skill specifically, when the same technique served scripted assistants well?
• A digression scenario passes with the interruption landing at turn 2 on one run and turn 4 on the next. What in the scenario file made that robustness testable?
• The old DU suite would still execute against a Mantle agent. Walk the three engine citations that explain why its scores would mean nothing.

The simulator supplies the conversations; the assertions nail the facts. The remaining gap is quality of free text — which brings us to the instrument everyone wants first and should reach for last.