Make an agent release decision with evidence
by Rasa team
A release meeting should end with a decision, an owner, and a reason somebody else can inspect. If the only evidence is a successful demo, the team has learned that one conversation worked.
This guide helps an AI product manager turn evaluation results into a release decision without writing test code. Bring the engineer responsible for the release and the domain owner who understands the cost of an incorrect outcome.
Read the results before the percentage
Here is a fictional review exercise, not a Rasa benchmark. A travel assistant completes 45 of 50 eligible itinerary tasks. Five fail: two because the booking service times out, two because the answer cites the wrong policy, and one because the customer identity is not established.
The completion rate is 45 / 50 = 90%. That number alone cannot justify a release. The identity failure and the policy errors need different decisions from a timeout with a clear handoff.
| Evidence | Question for the release owner | Possible decision |
|---|---|---|
| Correct task completion | Was “correct” defined before reviewing outputs? | Re-score if the definition changed |
| Wrong policy answers | Can the affected slice be isolated? | Exclude that slice until corrected |
| Missing customer identity | Could private information be exposed? | Block the affected action and investigate |
| Service timeouts | Did the customer receive a recoverable next step? | Permit a limited pilot only with a working fallback |
| Human handoffs | Did an actual receiving queue accept them? | Fix delivery before calling handoff successful |
These decisions are examples. Your domain owner must set the release criteria for your context, before examining the candidate’s results. Moving a threshold after a failure is a new product decision and belongs in the record.
Separate three kinds of evidence
Task evidence shows whether the intended outcome happened. Control evidence shows that forbidden behavior was prevented. Operational evidence shows what happens when a dependency or a human receiving team is unavailable.
A good answer does not prove that an action was authorized. A log saying “handoff sent” does not prove that a support queue received it. Ask for the resulting state: the tool result, receiving ticket, or blocked operation.
Rasa’s simulation evaluation documentation describes simulation-based evaluation. The community evaluation harness tutorial provides an implementation path. Automated checks and model judgments answer different questions; a model’s positive opinion should not override a failed factual assertion.
Write the release record
Use this template in the meeting:
- Candidate version and evaluation dataset version.
- Intended users and included workflow slices.
- Pre-agreed criteria, raw counts, and failed examples.
- Explicit exclusions and unresolved uncertainty.
- Decision: hold, limited pilot, or expand a current rollout.
- Person who can pause the release and the signal that triggers it.
- Date and evidence required for the next decision.
For the fictional exercise above, a defensible record might hold the identity-sensitive workflow while the team fixes authorization. It would not relabel the failed identity case “out of scope” unless the deployed product also makes that workflow unavailable.
Keep the pilot informative
Choose a limited audience whose outcomes the team can review, and ensure the receiving support team can handle fallbacks. Agree who checks the first results and who can disable the agent. The operations guide turns those obligations into a rollout checklist.
When live requests differ from the evaluation set, preserve the surprising cases with appropriate data handling and add them to the next review. Do not claim the original 90% predicts live performance: this example is fictional, the sample is small, and the eligible task mix may differ.
Before your next release meeting, fill the template with one real candidate. If any required evidence is missing, record “unknown” and assign the smallest experiment that could resolve it. The NIST AI RMF Core provides the broader context for documenting measurement and managing identified risks.