Guide · Platform / operations engineer
Agent build selection checks in Rasa
Trace a repository check from the selected example to its test command before accepting a green result.
Key takeaways (3)
- Match the tested directory to the example being added, rather than relying on a job name.
- Distinguish an explicitly empty selection from an incomplete example that must fail.
- Keep container readiness, offline tests, and credentialed runtime qualification as separate release evidence.
You add an agent example to a repository. Its container check is green, and a banking test job is green too. Before accepting the change, check which directory that second job tested.
In the Rasa examples repository, the container workflow could build a new example while a separate test matrix still named older banking examples. The two checks answered different questions. Putting their green statuses together did not establish that the new example’s own tests ran.
This guide is for the operator deciding whether those checks cover a release candidate. You will inspect the selected target, the command, and the result. Its runnable exercise checks the selector offline; it does not build a container or call an agent.
The local comparison checks selection only. An empty list makes the workflow skip the example-test job.
Trace both workflows to their targets
A recorded Premier job was green at revision 5a0af21b. Its name was examples/fiserv-premier. Its steps built the container and reported READY; none ran the example’s make test command. The saved job metadata preserves the full revision and step list.
At that same revision, the separate banking test matrix named only jack-henry-banking and fiserv-banking. Premier was absent. This comparison inspects the matrix’s selectable names. Execution of that separate workflow on Premier’s change remains unverified.
The earlier Codespaces workflow builds each selected resource configuration. It runs the configuration’s creation check and looks for a READY line. In that continuous integration (CI) job, required-secret checks are deliberately skipped:
env:
READY_SKIP_SECRETS: '1'The corresponding readiness helper runs the project’s installation and validation check by default. Its success is useful environment evidence. Check the learner’s required credentials separately before a real conversation.
Meanwhile, the earlier banking test workflow contained this matrix:
matrix:
project: [jack-henry-banking, fiserv-banking]Those names differ from the new direct examples. They are shown here to explain the historical routing error, not as examples to restore or run.
| Evidence | What it establishes | What remains to check |
|---|---|---|
Container starts and reports READY | That selected configuration passed its setup check | Whether this example’s tests ran; whether required runtime secrets were checked |
Old matrix runs make test | Tests for the directories named by that matrix | Whether the new example is one of those directories |
Repaired job selects an example and runs its own make test | The selected example’s command ran in its selected environment | Actual assertions, failures, skips, and any separate live qualification |
Read the commands before interpreting the status. A test result belongs to the target that produced it.
Empty and incomplete need different outcomes
The repaired selector knows the direct Symitar, SilverLake, and Premier example names. It selects the ones actually present on the checked-out revision.
For each present example, it requires a Makefile, a dependency manifest, a lock, and the corresponding devcontainer configuration. A missing file raises Incomplete example instead of quietly dropping that target.
An empty checkout is different. The selector writes an explicit empty list, and the workflow skips the example-test job. That is a successful selection check with no example tests run. It must be recorded that way.
Here is the dangerous result when the selector itself knows only the old names. A disposable fixture contains examples/fiserv-premier but none of its required files. The output comparison, produced by this small probe, records both outcomes:
| Same incomplete Premier fixture | Actual selector result | Workflow consequence |
|---|---|---|
| Current target list | Incomplete example: examples/fiserv-premier/Makefile | Selection fails visibly |
| Declared old-target mutation | [] | Selection returns successfully; the example-test job’s empty-list condition skips it |
The second row is a local mutation, not an observed GitHub run. It exposes why successful empty selection needs a target-coverage control: a forgotten example looks exactly like legitimate absence.
The three selector controls use temporary directories. They check empty selection, complete new examples with old names ignored, and a present but incomplete example. Their fixture files contain placeholder text: these tests check selection and file presence, not the validity of a real lock or container configuration.
Route the missing input to the example owner
Refuse the incomplete target. Ask the example owner to supply its real manifest, resolved lock, Makefile, and configuration. Then let the selected environment run the actual tests. Placeholder files are sufficient for a selector fixture, not for a release candidate.
Check the executed command too
The repaired workflow uses the selected name for both its devcontainer configuration and its working directory. This excerpt is the command inside that container:
set -eu
unset RASA_LICENSE RASA_PRO_LICENSE OPENAI_API_KEY ANTHROPIC_API_KEY DEEPGRAM_API_KEY GEMINI_API_KEY
cd "$(git rev-parse --show-toplevel)/examples/${{ matrix.project }}"
make testThe job also sets inheritEnv: false. Its listed license and provider keys are unset before the test command. Keep those offline controls separate from a later credentialed conversation.
When a selected job fails, open that job’s output and retain the source revision, project name, command, and first failing assertion. Send the failure to the example owner. When it passes, inspect skipped tests too: a vendor-input test skipped because its definition file is absent is not a passing vendor-contract check.
Prove the selector notices the wrong targets
Use Python 3.12 and the pinned source:
git clone https://github.com/RasaHQ/rasa-community-resources.git
cd rasa-community-resources
git checkout 0285d2b86cc3d1fde13881ad68c5288429bce8e4- Any system
python3.12 -I -B -m unittest discover -s scripts -p test_banking_contract_scope.py -v
The recorded offline run on November 12, 2026 ended with:
----------------------------------------------------------------------
Ran 3 tests in 0.009s
OKIn a separate disposable copy, a declared mutation replaced only the selector’s target tuple with the old matrix names. The same controls then failed to recognize complete new examples and failed to reject an incomplete one. The mutation output ended with:
----------------------------------------------------------------------
Ran 3 tests in 0.009s
FAILED (failures=2)That mutation tests the controls against the historical target mismatch. It is not a replay of an old GitHub job or a result from a running agent. No production source was changed.