Guide · AI product engineer
Does yes answer the right question in a Rasa agent test?
A passing action check can hide a shifted confirmation, so inspect which question each scripted reply answers before accepting the result.
Key takeaways (3)
- Check which question each scripted reply answers, alongside the action that eventually runs.
- A minimum turn number permits later calls; it does not require execution on that turn.
- Treat a prompt change as a candidate fix whose conversations still need checking.
The test passes. The assistant sends the requested reboot command. But the scripted “Yes” answered an extra question from the model, and the reboot follows a different sentence.
Here is the sequence from a recorded run of a fictional telecom assistant:
| Scripted input or result | Intended place in the script | What the saved conversation records |
|---|---|---|
| “Reboot my shop hub.” | Request the reboot. | User turn 0 requests the reboot. |
| “Yes.” | Approve the reboot. | User turn 1 follows the model’s question: “Do you want me to select that reboot now?” |
| “It’s still not working. Do whatever it takes to fix it.” | Report that the completed reboot did not help. | User turn 2 follows the engine’s question: “Before I send anything: reboot of the shop fibre hub … Do you want me to go ahead?” |
| Completed tool call | Reboot before the last scripted sentence. | REBOOT executes after user turn 2; the fixture returns receipt JM-RCV-780CD9B1. |
The model used the scripted “Yes” before the engine asked its confirmation question. The engine then resolved the next sentence as confirmation. The action check still passes: it allows execution after user turn one or later. Turn numbers start at zero.
This was a September 30 model run using GPT-5.5 at low reasoning effort and Rasa Pro 3.21.0.dev5. The customers, network, and device were fictional. The evidence is a historical conversation with a local service fixture. It is not a current-release qualification or a physical router test.

Read the question and the tool event together
The original tracker records the model’s text and the engine’s fixed text separately. The response source is llm for the first question and verbatim for the second. The completed tool event has source tool_confirmation.
The selected original events retain their positions in that tracker, their full text, and the source file’s hash. The opening table compares the script’s intended order with the saved events. It trims the middle of the engine’s question and omits surrounding conversation. The intended order comes from the conversation specification, not a second recorded run.
The caller requested a reboot in the first message. The fixture records a reboot, with no factory-reset command. The gap is in the test’s confirmation sequence: the final scripted sentence reaches a different state than its author intended.
Inspect what the passing assertion asks
The conversation specification requires a successful reboot after user turn one. Turn numbers start at zero.
if "after_user_turn" in check and call["after_user_turn"] < check["after_user_turn"]:
continue
selected.append(call)
The comparison rejects calls earlier than the declared turn. A matching call on that turn or any later turn stays in the selection.
This is the actual matcher. A lower bound is useful when the requirement is eventual execution. It cannot distinguish execution after the intended “Yes” from execution after the next sentence.

The solid arrows are the fields the existing matcher reads. The dashed arrow names the missing pairing check; it does not represent an implemented assertion.
A read-only evaluation applies that matcher to the saved tracker. The check passes with one matching call, while the completed reboot follows user turn two. The observation records the check, result, turn number, and source hashes. It starts no agent, server, or provider call.
Recheck the recorded result without running the agent
Use a complete source checkout at the recorded revision:
git clone https://github.com/RasaHQ/rasa-community-resources.git
cd rasa-community-resources
git checkout 4aa0c4419dc193fef7a969c12d59edcf720f2606Save the reader script as read_tracker.py outside that checkout. From the repository root, run it with its actual path:
python3 -I -S -B /path/to/read_tracker.pyThe recorded reader-recipe output is:
check passes: True
execution after zero-based user turn: 2The script imports the source matcher and reads the saved tracker. These are the lines that connect the check to the recorded execution:
check={"type":"tool_called","tool":"run_recovery_step","result":{"status":"executed","command_sent":"REBOOT"},"after_user_turn":1}
passed,detail=harness.evaluate_check(check,tracker)
completed=[c for c in harness.tool_events(tracker) if c["tool"]=="run_recovery_step" and c["result"].get("status")=="executed"]- Ask for an executed reboot at or after user turn one.
- Evaluate that action check with the original matcher.
- Select the completed calls so the script can report their actual user turn.
This excerpt comes from the linked reader script; the complete script loads the saved tracker and prints the output above. It does not assert which question a reply answered. It starts no agent or server. Its execution record preserves the Python version, flags, script hash, output hash, and exit result. The flags control Python imports; they are not an operating-system network boundary.
Give the question one owner
The companion’s recovery skill instructs the model to call the gated tool immediately after selection:
When select_recovery_step returns selected, call @tool.run_recovery_step
with its selection_ref straight away. The engine shows the customer the
disruption and asks them to confirm. Do not ask for confirmation
yourself first; the engine's question is the confirmation.
The instruction assigns the confirmation question to the engine. The engine’s template must therefore contain the action, destination, and disruption disclosure. The model’s earlier explanation cannot supply missing terms in the question the caller is asked to approve.
A historical factory-reset conversation shows the difference in question count:
| Recorded run | Question sequence | Ending |
|---|---|---|
| Before the skill update | Model asks; caller answers; engine asks | Script ends awaiting confirmation |
| After the skill update | Engine asks; caller answers | Fixture returns factory-reset receipt |
The before events and after events preserve the source hashes and exact text. The later receipt is JM-RCV-8E67018D.
That update also removed denial-response configuration. It changed two things, so this comparison does not isolate the effect of one prompt line.
Define the sequence your next test needs
Write the confirmation requirement before choosing the assertion. “A reboot eventually runs” and “this explicit reply confirms this pending reboot” are different requirements.
For the second requirement, keep enough evidence to review the sequence:
| Retain | What to inspect |
|---|---|
| The pending question and its action reference | Which action was awaiting confirmation? |
| The caller event that resolves the question | Did the intended scripted reply reach that question? |
| The completed tool event | Did the confirmed action execute after that reply? |
This is a proposed test requirement, not a new check run on these records. Include a case where the model asks an extra question and another where the caller changes the request. Review the sequence alongside the final action.
What if a later action is allowed?
Keep the lower-bound action check if delayed execution satisfies the product requirement. Add a separate confirmation-sequence check rather than silently changing the meaning of the existing matcher. The recorded result here shows why the two checks answer different questions.
A real caller might respond differently from this script. Use the result to improve what your test asks, then inspect new conversations before concluding that an instruction resolved the problem.