Check AI-written Rasa skills before merging them

You use a coding agent to edit a skill that blocks a customer’s card. The skill must ask for confirmation first. Before merging the change, you need to know whether that requirement survived.

You will check the compiled skill with a separate acceptance test, then run changed copies that remove its confirmation requirement. The exercise also shows a failure in the first grader: candidate code can end the test process early and still receive an acceptance decision.

This is a fictional card service and an offline code check. It does not block a real card or show what an assistant says to a caller. The completed exercise gives you a repeatable configuration check and a reason to distrust a grader that runs candidate code inside its own test process.

Get the pinned judging environment

Use Git, uv and Python 3.12 on macOS, the platform used for this run. These shell commands use its virtual-environment path. The companion accepts Python 3.11 or 3.12 and pins a prerelease of Rasa Pro. Do not substitute a newer engine and attribute its results to this run.

Start from the masterclass factory source:

git clone https://github.com/RasaHQ/rasa-community-resources.git
cd rasa-community-resources
git checkout 1428b480070ed5e1f614d60b67ea7cc9a7f11903
cd workshops/software-factories-masterclass/factory
uv sync --locked --python 3.12

The dependency pin is in pyproject.toml. The adjacent lock records the dependency versions. Installation needs package access; the comparison itself makes no model call and requires no licence or provider key. We recorded the results below with Python 3.12.13 and rasa-pro==3.21.0.dev5 on 19 October 2026.

If installation fails, keep that failure separate from the candidate’s quality. Check that Python is in the supported window and that the locked packages are available. Do not remove the lock or change the engine pin to obtain a result under the original experiment’s name.

State the requirement in compiled terms

Compare the confirmation block in the good and false candidates. Each file is under candidates/<candidate>/product/skills/block_card/skill.md.

tool_constraints:
  - block_card:
      requires_confirmation:
        enabled: true
        utter_for_confirmation: utter_confirm_block

tool_constraints:
  - block_card:
      requires_confirmation:
        enabled: false
        utter_for_confirmation: utter_confirm_block

The skill’s prose also says to list the cards first and ask which card the caller means. That prose is useful, but it is not the compiled confirmation setting. A coding agent could leave those instructions intact while changing the frontmatter.

The protected a1_block_card.py checks the setting that the engine produces:

    def test_blocking_waits_for_confirmation_after_compilation(self):
        gate = lib.gate("block_card", "block_card")
        self.assertIsNotNone(gate, "the engine compiled the confirmation gate away")
        self.assertTrue(gate.enabled)

The helper compiles the skill and reads the named tool’s confirmation setting. The suite also checks project findings, the skill name, the read-only tool, implemented tool arguments and the instruction to list cards. The assertion above protects the particular requirement this guide examines.

Keep the assertion outside the candidate

The task file names the acceptance suite and its helper by hash. In factory.py, the grader checks those hashes before running tests. It also checks that the candidate supplies only allowed paths.

The grader copies the baseline project, overlays the candidate files and copies the protected tests into a separate directory. Candidate-supplied tests are recorded for information; their result does not decide acceptance. The protected suite supplies that decision.

The hashes identify which test bytes ran. Establishing when someone wrote them requires separate evidence. Nor does copying files prevent candidate Python from affecting the process that imports it. Keep that distinction in mind for the final exercise.

From the factory directory, run the good candidate first:

.venv/bin/python factory.py tasks/a1-block-card.json candidates/block-card-good runs/article-good

The recorded output is:

ACCEPT - protected acceptance passed

Inspect runs/article-good/decision.json. It records the candidate and baseline digests, acceptance and helper hashes, installed engine, test checks, decision and reasons. In our run the protected suite completed six tests and ended with OK. Keep that record when you compare the changed copies.

Each command needs a new run directory. If one already exists, the factory refuses to reuse it. Choose another name and retain the earlier evidence; do not delete a failed run just to make the command work.

Compare the changes that remove confirmation

The companion has three prepared edits. One deletes the constraint block, one sets confirmation to false and one omits only the enabled key. That last copy keeps the confirmation block and its prompt name. These are hand-written test candidates, not newly generated coding-agent submissions.

The before-and-after result is the same for all three edits:

These are recorded results at the pinned revision. Both supplied graders gave those decisions. Setting confirmation to false compiled to an absent setting in this engine build; the test did not reach its second assertion about enabled.

Use the viewer to inspect one change before asking the grader to judge it:

.venv/bin/python show_gate.py candidates/block-card-gate-deleted

Recorded output:

engine findings: 0
block_card gate: OFF (compiled to None)

show_gate.py overlays the candidate, collects project findings and compiles the skill. It shows the setting; it does not decide acceptance or start a conversation. Zero findings therefore answers a different question from the protected confirmation assertion.

Now run the grader on that same candidate:

.venv/bin/python factory.py tasks/a1-block-card.json candidates/block-card-gate-deleted runs/article-deleted

Recorded output, with exit status 1:

REJECT - the protected acceptance suite failed: FAILED (failures=1)

In the decision record, the failed test is test_blocking_waits_for_confirmation_after_compilation. It reports AssertionError: unexpectedly None : the engine compiled the confirmation gate away. The six-test run has one assertion failure; this is not an import error or a timeout.

Repeat those commands with block-card-gate-false and block-card-gate-omitted, using fresh run directories. Inspect the reasons, not just the exit status. A different exception needs diagnosis before you can say that your test caught the missing requirement.

Separate rejection from a machine that cannot judge

Before judging a candidate, the factory checks the Python version, installed engine version and required engine import. Its environment specification is part of the pinned source. A mismatch produces NOT_READY, not a rejection of the proposed skill.

These meanings come from the factory’s source contract. We did not run a separate environment-mismatch exercise for this article. An unexpected traceback or a timed-out process is likewise no evidence that the confirmation assertion worked.

Check whether the grader can be fooled

There is another way to obtain exit status zero. The prepared block-card-forges-accept candidate puts this before its tool definitions:

import os

os._exit(0)  # ends the grading process with status 0 the moment the engine imports this file

The engine imports candidate code during the protected tests. The first grader trusts a zero exit status from that process. Run the supplied example:

.venv/bin/python factory.py tasks/a1-block-card.json candidates/block-card-forges-accept runs/article-forges-v1

Recorded output:

ACCEPT - protected acceptance passed

Its decision record contains only F.. as protected test output. There is no completed test count or closing OK. An acceptance decision here does not mean the suite finished successfully.

The second grader, factory_v2.py, requires a positive unittest count and a closing OK, as well as exit zero. Try the same candidate with a new output directory:

.venv/bin/python factory_v2.py tasks/a1-block-card.json candidates/block-card-forges-accept runs/article-forges-v2

Recorded output, with exit status 1:

REJECT - the protected acceptance suite failed: exited with status 0 without unittest's OK line and a test count

The integration and recovery scripts still import the first grader at this revision. Running the second grader for this exercise does not upgrade those paths. A design that must judge hostile code needs a separate trust boundary; this companion does not provide or qualify one.

Apply the check to your own change

Keep the acceptance assertion outside the files your coding agent may edit. Pin the judging environment and the assertion’s supporting code. Check the compiled requirement, then inspect why deliberately changed copies fail. Finally, inspect how the grader turns process output into a decision.

This approach costs more than asking the coding agent to supply its own tests. You maintain a separate assertion and rerun it when the engine or requirement changes. In return, project validity and a passing process cannot silently stand in for the specific requirement you meant to preserve.

The next question is whether the assistant actually asks the caller in a conversation. That needs its own evidence. This exercise tested compiled configuration and grader behaviour, with no model or real customer action.

Further reading: evaluation harness and mutation testing for an agent guard.