Skip to content
RasaGet Developer Edition

tutorial

Chapter 4 of 5

Chapter 4 — The Judge: Grounded Is Not Relevant

by Rod Rivera Published

LLM-as-a-judge scoring, demystified to the mechanism: groundedness is a ratio of statements, relevance is cosine similarity of invented questions — and each has a blind spot the other covers. Plus the trap where a judge silently has nothing to score.

Some questions have no fact to assert. “Is this answer actually supported by our overdraft policy?” is not a tracker fact. For free text, the framework sends the answer to a second LLM — the judge — and compares a returned score to a threshold.

The judge holds two jobs in a scenario. It scores your criteria — the natural-language goals like “the agent asks exactly one clarifying question” — against the whole transcript, with a written rationale per criterion. And it powers the two generative assertions, which score a specific answer against a specific standard. The criteria are where you express judged intent; the generative assertions are precision tools, and here is the sentence this chapter exists for: they are not variations of one idea. They are computed by completely different machinery, they fail differently, and confusing them is the most common way to get a meaningless eval.

(The machinery, for the record, is the same code the classic e2e tests used — the scenario engine imports rasa.e2e_test.assertions directly (rasa/builder/copilot/mcp_server/tools/assertion_engine.py:9), so every citation below reads from the engine you have installed.)

Groundedness: a ratio of statements

From eval/scenarios/faq_grounded_answer.yml:

assertions:
  - generative_response_is_grounded:
      threshold: 0.8
      ground_truth: >-
        A replacement debit card is issued free of charge once per
        calendar year. Additional replacements cost $12 each. Standard
        delivery takes five to seven business days.

Mechanism: the judge splits the answer into atomic statements and marks each one supported or unsupported against your ground truth (rasa/e2e_test/llm_judge_prompts/groundedness_prompt_template.jinja2:7-12). Then the score is plain arithmetic — no embeddings involved (rasa/e2e_test/utils/generative_assertions.py:156-166):

score = supported statements / total statements

This is the metric for “did the agent make something up?” — and the companion’s most valuable scenario turns it inside out: faq_unknown_topic_refused.yml asks about a policy the source does not contain, with a ground_truth that describes the refusal. An agent that fills the gap from general banking knowledge produces fluent, helpful, unsupported statements — and scores badly against a truth that says “I don’t have that.”

The denominator insight — the detail that changes how you read every threshold: the judge chooses how many statements the answer splits into, not you. A terse two-statement answer can only score 0, 0.5, or 1.0. Against that answer, threshold: 0.8 does not mean “80% good” — it means “both statements must be supported”. Read your thresholds as fractions of a small integer, not as percentages.

Relevance: cosine similarity of invented questions

assertions:
  - generative_response_is_relevant:
      threshold: 0.7

Completely different mechanism. The judge is shown only the answer — not the ground truth, not your policy — and invents three questions that answer would address (answer_relevance_prompt_template.jinja2:1-9; the count is num_variations = 3 at assertions.py:1595). Those invented questions and the user’s real question are embedded, and the score is their mean cosine similarity (generative_assertions.py:123-153).

Sit with the consequence, because it is the most important sentence in this chapter:

Relevance never sees your ground truth. A confidently wrong answer to exactly the right question scores high.

Relevance detects evasion and topic drift — an agent that answers the question it wished you’d asked. It cannot detect fabrication. Which is why faq_grounded_answer.yml asserts both metrics on the same turn: groundedness catches the confident lie, relevance catches the accurate dodge. Neither is sufficient alone.

(Side effect of the mechanism: relevance needs an embedding model as well as a judge — default OpenAI text-embedding-3-small, generative_assertions.py:28-31. Groundedness never embeds anything.)

Pin your judge

The judge — and, separately, the simulator — is configured in eval/conftest.yml:

simulation:
  llm: # drives the user simulator
    provider: openai
    model: gpt-5.1
evaluation:
  llm: # the judge: criteria scores + quality metrics
    provider: openai
    model: gpt-5.1

The two are independent on purpose, and the asymmetry matters: you can cheapen the simulator without touching what your results mean, but the judge model is an input to your scores. Swap it and yesterday’s 0.85 and today’s 0.79 were produced by different judges — nothing will tell you that unless the model name is pinned in a reviewed file. This file is that file.

The trap: a judge with nothing to score

This one trips everyone, and the error message doesn’t help. Generative assertions do not look at every bot message. With no utter_source: given, they consider only messages whose source metadata is one of:

EnterpriseSearchPolicy · ContextualResponseRephraser · IntentlessPolicy

(rasa/e2e_test/utils/generative_assertions.py:34-38.) A plain templated response is invisible to them. Responses become eligible when they are generated or passed through the rephraser — and the rephraser only touches responses that explicitly set rephrase: true (rasa/core/nlg/contextual_response_rephraser.py:122-124). Otherwise a response’s source is simply its action name — which you can target, by naming it in utter_source:.

So: if a generative assertion behaves as though it never ran, check the source of the message you think it is judging. Before the threshold, before the prompt, before blaming the judge.

Thresholds are not portable

A last habit before the honesty chapter: the thresholds in the companion are illustrative, not engine defaults, and any threshold you tune is tuned against three things at once — a specific judge model, a specific embedding model, and a specific set of ground-truth strings. Change any of the three and the old threshold is a number from a different experiment. (The engine’s own fallback, for the record, is DEFAULT_THRESHOLD = 0.5, assertions.py:70 — treat that as a placeholder, not a recommendation.)

Check your understanding

  • Your FAQ agent starts answering overdraft questions with confident, fluent, wrong numbers. Which metric catches it, and why does the other one score it high?
  • Why does the refusal scenario’s ground_truth describe what the agent should say it cannot do, rather than containing the right answer?
  • Why does changing your embedding model invalidate relevance thresholds but leave groundedness scores untouched?

You now command the whole framework. The last chapter is about the hardest part: knowing what your green checkmarks are actually worth.