Evaluation / data specialist · Banking & fintech (external)
Help a customer block the right bank card
Ask which card the customer means when two cards have the same final digits.
Key takeaways (3)
- Two cards can share the same ending even when they belong to the verified caller.
- Resolve the spoken description to one service reference before asking for confirmation.
- Test every different selected and requested card pair before connecting the write tool.
A customer reports a lost card and gives its final four digits. Two cards on their account share that ending. Blocking either one without another question could stop the wrong card.
Use this fictional banking example to check how the assistant presents the two candidates and waits for a clear choice. The tests inspect the selected card and confirm that no card is blocked while the choice is unclear.
{"case": "banking-block-card", "label": "shared-ending", "output": {"candidates": ["debit card ending 5 5 0 2 on your joint savings account", "credit card ending 5 5 0 2 on your travel credit account"], "card_last_four": "5502", "effects": 0, "next_step": "More than one card could match, or the card to block is not the one the caller confirmed. Read the candidates by kind and account and ask which one. Do not block anything yet.", "reason": "ambiguous_card_selection", "status": "blocked"}}
This real offline result contains two owned cards with the same ending. Verifying the caller again or spacing the spoken digits cannot choose between them. Blocking either card now would guess which resource the caller meant.
Use the ending as a lookup alias. Resolve it to one owned service reference, then bind the block to that selection. The exercise below tries every different pair of owned references and removes one comparison to show what it protects.
Reproduce the shared ending
git clone https://github.com/RasaHQ/rasa-community-resources.git
cd rasa-community-resources
git checkout 4aa0c4419dc193fef7a969c12d59edcf720f2606
cd examples/mantle-voice-banking-block-card-gpt
Keep the full clone so the neighbouring case contract is available. Run all commands below from this selected example directory. The inline exercise uses only Python’s standard library and calls no provider or Rasa engine.
- Any system
python3 -B -m unittest discover -s tests -v
The pinned guard tests include
test_shared_ending_is_ambiguous_until_the_kind_is_given and
test_block_of_a_card_other_than_the_confirmed_one_is_refused.
Start by tracing selection before looking at the write.
In lib/northgate.py, selection filters only cards owned by the supplied customer. It then matches the normalised ending and any kind the caller supplied. Zero matches and multiple matches are refused; only one match returns a service reference.
Resolve the kind to an opaque reference
The recorded selection results differ only by kind:
| Caller supplies | Owned matches | Result | Write may use |
|---|---|---|---|
| Ending 5502 | Debit and credit | Ambiguous; ask which | None |
| Spaced ending 5 5 0 2, debit | Joint savings debit | Selected | NB-CARD-0103 |
| Spaced ending 5 5 0 2, credit | Travel credit | Selected | NB-CARD-0104 |
| Select NB-CARD-0101, request NB-CARD-0102 | Both owned, different | Blocked; zero effects | None |
| Reselect NB-CARD-0102, request NB-CARD-0102 | Same selected reference | Succeeded in fixture | NB-CARD-0102 |
Card kind resolves this fixture collision. In your adapter, ask for another supported distinction if kind still leaves several matches. Do not present an ambiguous candidate label as a selected resource.
The Rasa selection wrapper writes the selected reference and its display label to skill memory. A failed or ambiguous new selection clears those fields, so it cannot leave the previous selection in place:
result = ng.select_card(_service(), _get(context, CUSTOMER_KEY), card_last_four, card_kind)
if context is not None:
# A new selection always replaces the old one, so a correction can
# never leave the previous card confirmed.
context.memory.set("selected_card_ref", result.get("card_ref", ""))
context.memory.set("selected_card_label", result.get("card_label", ""))
return ToolResult(llm_response=result)
The block wrapper uses the verified customer and selected reference from memory, with the requested reference copied from the selection result:
result = ng.block_card(
_service(),
_get(context, CUSTOMER_KEY),
_get(context, "selected_card_ref"),
card_ref,
request_id=_conversation_id(),
)
return ToolResult(llm_response=result)
Caller words reach selection; the resolved resource reaches the write. Authentication establishes ownership. The comparison below establishes whether this write targets the selected owned card. Those are separate facts.
Test every different owned pair
Paste the following complete exercise from the selected example directory. It includes the recorded collision check, all different reference pairs and a deliberately broken version compiled in memory. It edits no source file.
python3 -B - <<'PY'
import json, types
from pathlib import Path
def load(project, filename, mutation=None):
path = Path.cwd() / 'lib' / filename
source = path.read_text()
if mutation:
old, new = mutation
assert source.count(old) == 1, 'mutation anchor changed'
source = source.replace(old, new)
m = types.ModuleType(project.replace('-', '_') + ('_mutant' if mutation else ''))
m.__file__ = str(path.resolve())
exec(compile(source, str(path), 'exec'), m.__dict__)
return m
def emit(case, label, value):
print(json.dumps({'case': case, 'label': label, 'output': value}, sort_keys=True))
# The same last four can name two owned cards. Kind disambiguates the alias.
ng = load('mantle-voice-banking-block-card-gpt', 'northgate.py')
svc = ng.CardService()
ambiguous = ng.select_card(svc, 'NB-CUST-7310', '5502')
emit('banking-block-card', 'shared-ending', ambiguous)
assert ambiguous['status'] == 'blocked' and len(ambiguous['candidates']) == 2
for kind in ('debit', 'credit'):
got = ng.select_card(svc, 'NB-CUST-7310', '5 5 0 2', kind)
emit('banking-block-card', 'resolve-' + kind, got)
assert got['status'] == 'selected'
refs = list(svc.cards_of('NB-CUST-7310'))
count = 0
for selected in refs:
for requested in refs:
if selected == requested:
continue
isolated = ng.CardService()
result = ng.block_card(isolated, 'NB-CUST-7310', selected, requested, 'all-pairs')
assert result['effects'] == 0 and result['status'] == 'blocked'
assert all(c['state'] == 'active' for c in isolated.cards.values())
count += 1
emit('banking-block-card', 'all-distinct-owned-ref-pairs', {'attempts': count, 'refused': count, 'effects': 0})
# A correction points at a different owned card: ownership alone must not suffice.
for mutant in (False, True):
ng = load('mantle-voice-banking-block-card-gpt', 'northgate.py',
('bool(selected_card_ref) and ref == selected_card_ref', 'bool(selected_card_ref)') if mutant else None)
svc = ng.CardService()
selected = ng.select_card(svc, 'NB-CUST-7310', '4417')['card_ref']
before = {r: c['state'] for r, c in svc.cards.items()}
wrong = ng.block_card(svc, 'NB-CUST-7310', selected, 'NB-CARD-0102', 'correction-proof')
changed = [r for r in before if before[r] != svc.cards[r]['state']]
emit('banking-block-card', 'deliberate-equality-removal' if mutant else 'shipped-correction-refusal',
{'selected': selected, 'requested': 'NB-CARD-0102', 'status': wrong['status'], 'reason': wrong['reason'], 'effects': wrong['effects'], 'changed_cards': changed})
assert (wrong['status'], changed) == (('succeeded', ['NB-CARD-0102']) if mutant else ('blocked', []))
if not mutant:
repaired = ng.select_card(svc, 'NB-CUST-7310', '8023')['card_ref']
done = ng.block_card(svc, 'NB-CUST-7310', repaired, repaired, 'correction-proof')
emit('banking-block-card', 'reselect-then-confirm-integration-precondition', {'status': done['status'], 'card_ref': done['card_ref'], 'effects': done['effects']})
PY
Actual recorded output from these exercise cases, 8 October 2026:
{"case": "banking-block-card", "label": "shared-ending", "output": {"candidates": ["debit card ending 5 5 0 2 on your joint savings account", "credit card ending 5 5 0 2 on your travel credit account"], "card_last_four": "5502", "effects": 0, "next_step": "More than one card could match, or the card to block is not the one the caller confirmed. Read the candidates by kind and account and ask which one. Do not block anything yet.", "reason": "ambiguous_card_selection", "status": "blocked"}}
{"case": "banking-block-card", "label": "resolve-debit", "output": {"card_label": "debit card ending 5 5 0 2 on your joint savings account", "card_ref": "NB-CARD-0103", "next_step": "Call block_card with this card_ref. The engine asks the caller to confirm this card before anything changes.", "state": "active", "status": "selected"}}
{"case": "banking-block-card", "label": "resolve-credit", "output": {"card_label": "credit card ending 5 5 0 2 on your travel credit account", "card_ref": "NB-CARD-0104", "next_step": "Call block_card with this card_ref. The engine asks the caller to confirm this card before anything changes.", "state": "active", "status": "selected"}}
{"case": "banking-block-card", "label": "all-distinct-owned-ref-pairs", "output": {"attempts": 20, "effects": 0, "refused": 20}}
{"case": "banking-block-card", "label": "shipped-correction-refusal", "output": {"changed_cards": [], "effects": 0, "reason": "ambiguous_card_selection", "requested": "NB-CARD-0102", "selected": "NB-CARD-0101", "status": "blocked"}}
{"case": "banking-block-card", "label": "reselect-then-confirm-integration-precondition", "output": {"card_ref": "NB-CARD-0102", "effects": 1, "status": "succeeded"}}
{"case": "banking-block-card", "label": "deliberate-equality-removal", "output": {"changed_cards": ["NB-CARD-0102"], "effects": 1, "reason": "verified_fixture_receipt", "requested": "NB-CARD-0102", "selected": "NB-CARD-0101", "status": "succeeded"}}
There are five owned references in the fixture. Trying each selected reference against every different requested reference produces 20 attempts. Each uses a fresh service, checks zero effects and checks every card remains active. A test that asks only whether the requested card is owned misses this class of error entirely.
The pure-Python loop does not ask the engine for confirmation. Its reselect case establishes the matching-reference precondition for that later integration check. It is not evidence that a spoken correction was understood.
Remove only the selection comparison
The selection comparison is in the pinned write function:
facts = {
"selected_card_owned": bool(customer_id) and card is not None and card["customer_id"] == customer_id,
"single_card_confirmed": bool(selected_card_ref) and ref == selected_card_ref,
}
The deliberate mutation removes only and ref == selected_card_ref.
Ownership still passes; some selection still exists. The wrong owned card
then blocks successfully. The final mutation output names the changed card,
so a misleading success status cannot hide which resource changed.
Keep the comparison inside the function that calls your card adapter. Its cost is maintaining a selected service reference separately from the caller’s description and testing that binding at execution. That work remains necessary when the display alias looks familiar.
Reselect before confirming the correction
The pinned skill.md puts the confirmation gate on the write tool:
tool_constraints:
- block_card:
requires: session.block_card.selected_card_ref
requires_confirmation:
enabled: true
utter_for_confirmation: utter_confirm_block_card
utter_on_user_denial: utter_block_cancelled
The skill directs a correction back through selection. Decline the old confirmation, resolve the corrected description and let the engine ask about the new selected card. Changing only the requested tool argument leaves the old memory binding behind and triggers the refusal shown above.

Selection, confirmation and execution are separate stages. The script isolates the last stage; the configured skill shows how the project reaches it.
Show what to inspect after a corrected card choice
Compare the returned service reference, memory selection and requested write reference. They must identify the same card. Inspect refusal effects and the service snapshot as well as status. Then test the dialogue gate separately: the question must describe the corrected card, not the previous selection.
A separate offline check on 8 October 2026 used the pinned Rasa SDK and ran the full suite without skips. Its actual summary was:
Ran 15 tests in 10.369s
OK
Project validation also printed:
validate_project: ok
To repeat that full check after installing the selected project:
uv run --locked python -m unittest discover -s tests -v
make validate
This validates tool imports and the project shape. It does not train a model,
call a provider or establish spoken confirmation behaviour. Without Rasa
installed, the earlier command can skip the SDK-dependent tool tests; read the
summary rather than treating any OK as a full-suite result.
Try a small microphone acceptance check
The pinned README supplies the live setup. With Python 3.12 and uv, install only this example:
uv sync --locked --python 3.12
make env
Fill the local .env with RASA_LICENSE, OPENAI_API_KEY and
DEEPGRAM_API_KEY. The project pins Rasa Pro 3.21.0.dev5.
Its integrations.yml selects gpt-5.5-2026-04-23,
Deepgram flux-general-en input and aura-2-andromeda-en output.
make validate
make train
make inspect
Allow microphone access. This optional path uses billed APIs and was not run
for the receipts above. Try the proposed caller line “I am Nadia Okafor,
born fourteen March nineteen eighty-six. Block my card ending five five zero
two.” The next choice should distinguish debit from credit. Say “The credit
card” and inspect that the selected reference is NB-CARD-0104 before agreeing
to the confirmation. In another short check, correct that choice at confirmation
and inspect the new selection and question before any write.
This is a manual acceptance check, not a voice reliability rate. The fixture stores cards in process memory. A real service needs durable, atomic handling and its own authenticated ownership checks; these results do not establish safety across restarts or concurrent requests.