Why TTS failover comes back after a voice agent restart

Your voice agent keeps logging that it switched to a backup text-to-speech (TTS) vendor. Something looks stuck, so you restart the voice service. The alerts stop. Then the next call comes in, and the same alert fires again.

We reproduced this offline with stub vendors and no Rasa server. A script builds a fresh copy of the companion’s voice router per call, as Rasa does with its speech engines. It runs two calls in one Python process, then a new process stands in for the restart. The first vendor, Rime, answers every request with HTTP 402: out of credits.

This is the output at companion commit 4aa0c44 on Rasa Pro 3.21.0.dev5, trimmed: the version and start-up lines and each call’s second sentence are cut.

$ python restart_probe.py first   # new process
-- call 1 (first process)
2026-10-03 02:47:33 [warning  ] voicerouter.tts.failing_over   attempt=1 from_provider=rime verdict='quota (HTTP 402) — skipping briefly' voice_changes=True
2026-10-03 02:47:33 [info     ] voicerouter.tts.failover       from_provider=rime reason='served after failover' to_provider=deepgram
   sentence 1 spoken by deepgram
-- call 2 (same process, next call)
   sentence 1 spoken by deepgram
$ python restart_probe.py new   # new process
-- call 3 (new process: the restart)
2026-10-03 02:47:34 [warning  ] voicerouter.tts.failing_over   attempt=1 from_provider=rime verdict='quota (HTTP 402) — skipping briefly' voice_changes=True
2026-10-03 02:47:34 [info     ] voicerouter.tts.failover       from_provider=rime reason='served after failover' to_provider=deepgram
   sentence 1 spoken by deepgram

Call 2 went straight to the backup, because the router remembered that Rime was out of credits. The restart wiped that memory. Call 3 tried Rime again, failed again and switched voice again. Nothing about Rime’s account changed.

Each restart cost one failed request to the empty account and a new voice on the next call’s first sentence. The stubs answer instantly, so the caller’s extra wait was not measured.

The fix is to read the failure type in the alert and act on that type. Restart only after the vendor is fixed. That matters if your on-call team restarts on every failover alert, and it matters more because of two outages that never log a failover at all:

• Every TTS vendor fails. The router logs a failed attempt per vendor and no failover. We drove Rasa 3.21.0.dev5’s streaming helper, which Rasa uses for model replies unless a response or tool-call hook is registered. The reply text was still produced and no audio reached the caller’s connection.
• The speech-to-text connection drops mid-call. In the Rasa versions we read, the engine logs a warning and ends the stream as if the call were over. The router counts no failure, so its metrics stay silent.

For text-to-speech, the failover line fires only on recovery, so it stays silent when nothing recovers. Both cases are shown below, with the log lines to alert on instead.

"""Does a restart make the router try an out-of-credits vendor again?

Offline. Builds the companion's real router from the example's
integrations.yml over the drill's stub engines (no network, no keys).
Here the first vendor returns HTTP 402 on every attempt: the account is
empty and nobody tops it up. No Rasa server runs: each "call" is a fresh
router, as Rasa builds new engines for every call. Run it twice: once for
two calls in one Python process, then again as a new process, which stands
in for a restart or redeploy.
"""
import asyncio, sys
from importlib.metadata import version
from pathlib import Path

sys.path.insert(0, str(Path(sys.argv[2]) / "examples/mantle-voice-routed-skills/scripts"))
import drill_failover as drill
import voicerouter

def always_402(turn, attempt):
    return drill.http_error(402, "Payment Required")

async def one_call(cfg, name):
    router = drill.build_router(cfg, always_402)  # Rasa builds new engines per call
    print(f"-- {name}")
    for turn, line in enumerate(drill.SCRIPT[:2], start=1):
        for p in router._providers:
            p.engine.turn, p.engine.attempt = turn, 0
        async for _ in router.synthesize(line):
            pass
        print(f"   sentence {turn} spoken by {router.active_provider}")

async def main(phase):
    print(f"rasa-pro {version('rasa-pro')}; voicerouter from {Path(voicerouter.__file__).parent.relative_to(Path(sys.argv[2]))}")
    cfg, _ = drill.load_tts_config(None)
    if phase == "first":
        await one_call(cfg, "call 1 (first process)")
        await one_call(cfg, "call 2 (same process, next call)")
    else:
        await one_call(cfg, "call 3 (new process: the restart)")

asyncio.run(main(sys.argv[1]))

Why the alert comes back after a restart

In the Rasa source we read (3.20.0rc1 and 3.21.0.dev5), a voice channel is built with one TTS engine. When it speaks a whole message, Rasa does not try a second vendor. In the example, that one engine is the router: a wrapper that holds a list of real vendors and tries them in order.

This excerpt of the example’s integrations.yml drops comments and voice settings and is shown without its source indentation:

tts:
  name: voicerouter.RoutedTTS
  policy:
    cooldown_seconds: 60
    health_scope: process
    same_provider_retries: 1
    retry_backoff_ms: 250
  providers:
    - name: rime
    - name: deepgram
    - name: voicerouter.providers.openai.OpenAITTS
      label: openai

In both Rasa versions, the voice channel builds a fresh pair of speech engines for every call. So the router keeps each vendor’s health in one dictionary that lives as long as the Python process. This is an excerpt of the router’s voicerouter/health.py:

# Deliberately in-process only. A shared store (Redis) would extend this across
# workers and is the obvious next step, but it introduces a dependency and a
# failure mode of its own, and process scope already removes the large majority
# of the waste.

_SHARED: dict[str, "HealthRegistry"] = {}

A restart or redeploy starts a new process, so that dictionary starts empty. The next call’s first sentence goes to Rime. For the router’s health record, that is the whole effect.

A restart also starts a fresh process environment, and in the Rasa 3.20.0rc1 source rasa run reads integrations.yml only at start-up. Both matter for fixes below.

The router follows the circuit breaker pattern: stop calling a failing vendor, then try it again later. The drawing shows one vendor’s states:

Solid arrows are the router’s own decisions. The dashed arrows are the restart, and it takes them whether or not anyone fixed the vendor.

A parked vendor stays on the list below the healthy ones. A switched-off vendor is dropped from it, and a later success does not clear it.

Read the failure type, not the failover

The router sorts every vendor error into a type, from the HTTP status, an AWS error code or the message text. The type decides how long the vendor is parked. This summary comes from the router’s voicerouter/failures.py, the same at both commits:

“What the vendor asks” is the Retry-After header on the error, when the vendor sends one. Anything the router cannot sort is unknown, and it gets the 60-second park from the example’s policy.

For a whole sentence, the router retries the last two types once on the same vendor, if the wait is a second or less. For a streamed reply, the router at 4aa0c44 has no such retry: any failure moves straight to the next vendor.

These are the default park times, an excerpt from voicerouter/failures.py:

DEFAULT_COOLDOWNS: dict[FailureKind, float] = {
    FailureKind.AUTH: 0.0,         # permanent; cooldown unused
    FailureKind.CONFIG: 0.0,       # permanent; cooldown unused
    FailureKind.QUOTA: 900.0,      # 15 minutes
    FailureKind.RATE_LIMIT: 20.0,  # overridden by Retry-After when present
    FailureKind.TRANSIENT: 15.0,
}

Where to read the type

The log line you get depends on how the vendor was asked to speak:

• Whole sentence. voicerouter.tts.failing_over carries the type in its verdict field.
• Streamed reply, 4aa0c44 only. voicerouter.tts.start_response_failed carries the type when a vendor will not start a stream. voicerouter.tts.streamed_response_failed carries it when a stream breaks after it started.
• Either way, voicerouter.tts.failover names the vendor that took over. It is logged only when a later vendor succeeds.

In the 4aa0c44 source, both ways record the failure type on the router’s OpenTelemetry counter, voicerouter.tts.failure, as its failure_kind attribute. So one alert on failure_kind covers both, if your tracing setup exports the counter.

When every vendor fails at once

We tried this offline at 4aa0c44 on Rasa Pro 3.21.0.dev5, with every stub vendor answering HTTP 402. The script drove Rasa’s own browser-audio voice channel, with a fake caller connection and a stub model. The whole-message case ran second, in the same process, with every vendor already parked.

These are its outputs, trimmed. Cut: start-up, debug and traceback lines, the version line, the two channel.supports_* lines and the == path headers.

acompletion_with_streaming returned normally; text: ['Your balance is two thousand pounds.']
2026-10-03 02:45:18 [warning  ] voicerouter.tts.start_response_failed error="402, message='Payment Required', url='https://vendor.example/v1'" provider=rime verdict='quota (HTTP 402) — skipping briefly'
2026-10-03 02:45:18 [warning  ] voicerouter.tts.start_response_failed error="402, message='Payment Required', url='https://vendor.example/v1'" provider=deepgram verdict='quota (HTTP 402) — skipping briefly'
2026-10-03 02:45:18 [warning  ] voicerouter.tts.start_response_failed error="402, message='Payment Required', url='https://vendor.example/v1'" provider=openai verdict='quota (HTTP 402) — skipping briefly'
2026-10-03 02:45:18 [error    ] output_channel.response_delivery_failed channel=BrowserAudioOutputChannel error_type=TTSError
websocket messages after path 1: 0
router failures by type after path 1: {'rime': {'quota': 1}, 'deepgram': {'quota': 1}, 'openai': {'quota': 1}}

2026-10-03 02:45:18 [warning  ] voicerouter.tts.failing_over   attempt=1 from_provider=rime verdict='quota (HTTP 402) — skipping briefly' voice_changes=True
2026-10-03 02:45:18 [warning  ] voicerouter.tts.failing_over   attempt=2 from_provider=deepgram verdict='quota (HTTP 402) — skipping briefly' voice_changes=True
2026-10-03 02:45:18 [warning  ] voicerouter.tts.failing_over   attempt=3 from_provider=openai verdict='quota (HTTP 402) — skipping briefly' voice_changes=True
send_text_message raised TTSError voicerouter: no TTS provider available — rime: quota, retry in 900s; deepgram: quota, retry in 900s; openai: quota, retr
websocket messages during path 2: 1
    not audio: {"marker":"f52eb928e9c14f3592d9060a144952fe"}
router failures by type after path 2: {'rime': {'quota': 2}, 'deepgram': {'quota': 2}, 'openai': {'quota': 2}}

Streamed, the reply text came back normally and nothing reached the caller; as a whole message, the call to speak raised an error. We did not run a Mantle turn, so what happens next on a real call was not tested.

In neither case did voicerouter.tts.failover fire, while the failure counts in both tabs show quota for every vendor. So page platform on-call on failure_kind across all vendors, on output_channel.response_delivery_failed and on any error that reads no TTS provider available.

"""What does Rasa's voice channel do when every TTS vendor fails?

Offline. The companion router, built from the example's integrations.yml,
over stub vendors that all answer HTTP 402, both when asked for a whole
sentence and when asked to open a streamed response. The channel is Rasa's
browser-audio voice output channel with a fake websocket that records what
it is sent. The model is a stub that streams two text chunks. No Rasa
server, no network, no keys, no model call.

Path 1 (streamed): Rasa's own streaming helper, acompletion_with_streaming,
which Mantle uses when the channel supports streaming.
Path 2 (whole message): the channel's send_text_message, which Mantle's
TurnContext.send uses.
"""
import asyncio, sys
from importlib.metadata import version
from pathlib import Path

sys.path.insert(0, str(Path(sys.argv[1]) / "examples/mantle-voice-routed-skills/scripts"))
import drill_failover as drill
from voicerouter.base import BuiltProvider, ProviderSpec, RouterPolicy
from voicerouter.routed_tts import RoutedTTS
from voicerouter.utterance import UtterancePolicy
from rasa.core.channels.voice_stream.audio_bytes import L16_16KHZ  # noqa
from rasa.core.channels.voice_stream.browser_audio import BrowserAudioOutputChannel
from rasa.core.channels.voice_stream.tts.tts_cache import TTSCache
from rasa.core.channels.voice_stream.tts.tts_engine import StreamState
from rasa.shared.providers.llm.llm_response import LLMResponse
from rasa.shared.utils.llm import acompletion_with_streaming
from rasa.core.channels.voice_stream.call_state import CallState, _call_state


class EmptyAccount:
    """A vendor whose account has no credits: every request is HTTP 402."""
    streaming_input = True
    current_language_config = {}

    def __init__(self):
        self.stream_state = StreamState.NO_STREAMING
        self.stop_streaming_output_audio_chunks = False

    async def connect(self): pass
    async def close_connection(self): pass

    async def synthesize(self, text, config=None):
        raise drill.http_error(402, "Payment Required")
        yield b""

    async def start_response(self, streaming_config=None):
        raise drill.http_error(402, "Payment Required")


class FakeWebsocket:
    def __init__(self): self.sent = []
    async def send(self, message): self.sent.append(message)


class StubModel:
    async def acompletion_stream(self, messages, **kwargs):
        for text in ["Your balance is ", "two thousand pounds."]:
            yield LLMResponse(id="stub", choices=[text], created=0, model="stub")


def build_router():
    cfg, _ = drill.load_tts_config(None)
    policy_raw = dict(cfg.get("policy") or {})
    utterance = UtterancePolicy.from_dict(policy_raw.pop("utterance_classes", None))
    policy = RouterPolicy.from_dict(policy_raw)
    specs = [ProviderSpec.from_dict(raw, i) for i, raw in enumerate(cfg["providers"])]
    return RoutedTTS([BuiltProvider(s, EmptyAccount()) for s in specs], policy, [],
                     utterance_policy=utterance)


async def main():
    print(f"rasa-pro {version('rasa-pro')}")
    _call_state.set(CallState())  # what Rasa sets at the start of each call
    router = build_router()
    ws = FakeWebsocket()
    channel = BrowserAudioOutputChannel(ws, router, TTSCache(max_size=10), L16_16KHZ)
    print("channel.supports_streaming:", channel.supports_streaming)
    print("channel.supports_response_writer:", channel.supports_response_writer)

    print("== path 1: streamed model reply")
    try:
        reply = await acompletion_with_streaming(StubModel(), "hi", channel, "caller-1")
        print("acompletion_with_streaming returned normally; text:", reply.choices)
    except Exception as e:
        print("acompletion_with_streaming raised", type(e).__name__, str(e)[:120])
    await channel.wait_for_response_deliveries()
    print("websocket messages after path 1:", len(ws.sent))
    print("router failures by type after path 1:", {m["provider"]: m["failures"] for m in router.metrics_snapshot()})

    print("== path 2: whole message (send_text_message)")
    before = len(ws.sent)
    try:
        await channel.send_text_message("caller-1", "Sorry, something went wrong.")
        print("send_text_message returned normally")
    except Exception as e:
        print("send_text_message raised", type(e).__name__, str(e)[:120])
    print("websocket messages during path 2:", len(ws.sent) - before)
    for m in ws.sent[before:]:
        print("   ", "audio" if '"audio"' in m else "not audio:", m[:90])

    print("router failures by type after path 2:", {m["provider"]: m["failures"] for m in router.metrics_snapshot()})

asyncio.run(main())

The speech-to-text blind spot

In the Rasa source we read (3.20.0rc1 and 3.21.0.dev5), the base speech-to-text engine catches a failed read itself. It logs a warning and ends the stream normally, and Deepgram’s engine uses that base method. The router treats a normal end as the end of the call, so it records no failure.

We showed this with a fake connection that fails on the first read, at 4aa0c44 on Rasa Pro 3.21.0.dev5. This is the output, trimmed: the asr.engine.initialized, engine class and voicerouter.asr.ready lines are cut, and so is a second case where the connection was never opened.

DeepgramASR overrides stream_asr_events: False
2026-10-03 02:28:50 [warning  ] Error while streaming ASR events: socket dropped mid-call (probe)
A: socket drops during a read -> RoutedASR.stream_asr_events ended without raising
A: socket drops during a read -> deepgram health: failures = 0 | active provider: deepgram

So for speech-to-text, page platform on-call on repeated Rasa warnings Error while streaming ASR events. The router’s metrics stay silent in this case. What the caller hears after that was not tested.

What to do for each failure type

When a failover alert fires, work through it in order.

How to check your runbook works

Rehearse the quota case before you need it. In a rasa-community-resources checkout, go to the examples/mantle-voice-routed-skills folder and install the example:

This runs uv’s uv sync --prerelease=allow into the example’s .venv folder. Then run the drill:

The drill needs no keys. It builds the real router from integrations.yml over stub vendors, fails the first vendor from the third sentence and speaks whole sentences only.

This is the output of the script make drill calls, scripts/drill_failover.py --scenario credits, at 4aa0c44. It is trimmed: the bold codes and the closing three-line summary are removed. Our run at 69e27b6 printed the same lines.

Routing rules read from examples/mantle-voice-routed-skills/integrations.yml
Vendor calls are stubbed — the routing decisions below are the real ones.
2026-10-03 02:28:28 [info     ] voicerouter.tts.ready          providers=['rime', 'deepgram', 'openai'] skipped=[]

credits — Primary runs out of credits (HTTP 402) from turn 3
  chain: rime > deepgram > openai

  1. [rime            ] Hi, you're through to Northwind. This is Vela.
  2. [rime            ] One moment.
2026-10-03 02:28:28 [warning  ] voicerouter.tts.failing_over   attempt=1 from_provider=rime verdict='quota (HTTP 402) — skipping briefly' voice_changes=True
2026-10-03 02:28:28 [info     ] voicerouter.tts.failover       from_provider=rime reason='served after failover' to_provider=deepgram
  3. [deepgram        ] Your current balance is two thousand four hundred fifty dollars.   <- rime -> deepgram, failover
  4. [deepgram        ] Transferring four hundred pounds to Sam Rivera - shall I go ahead?
  5. [deepgram        ] Okay, got it.
  6. [deepgram        ] That's done. Is there anything else?

  after the call, rime: open, 1 failure(s), last was quota, retried in 900.0s

Check that your alert routes the quota in the warning line to the account owner, not to a restart. The last line shows the real park: 900 seconds.

In patterns/voice-vendor-router, make test runs the router’s offline unit tests, six of which we reran at the pin. They check that health outlives a call and that a switched-off vendor stays off.

Trade-offs

Process memory, not shared memory. The health record covers one process and is not shared across workers. This guide makes no claim about several.

A reset function exists, but nothing calls it. In the companion, only the tests and the drill call reset_shared_registries(), and no endpoint exposes it. So in practice the reset is a restart, after the fix.

Per-call health. With health_scope: call, a restart changes nothing, but every call tries the failed vendor first and pays the failover on its opening sentence.

Limits

• Everything here is an offline test, a stubbed probe or drill, or a source reading. None of it is a live call, so it shows decisions, not what a caller hears.
• The 402 is synthetic. Nothing here says how any vendor fails in production.
• The router is the companion’s code, not part of Rasa. If you wrote your own, check where it keeps vendor health before you trust a restart.