Someone asked me recently about building a voice assistant that understands and speaks Nigerian Pidgin. Speech to text in, text to speech out, the usual stack. Good project. It's also a clean illustration of why multilingual voice AI is harder than it looks: the demo works on day three, then falls apart the first time a real person calls it.
I have shipped voice twice, in very different shapes. AmtHeld answers citizen calls for German public agencies in more than 25 languages, around the clock. Intavue runs a real-time voice interviewer that reads your CV and asks live follow-ups. Neither was hard because of the model. They were hard because of everything the language column in a vendor's docs doesn't tell you.
So here's what actually breaks when you build multilingual voice AI, and what I'd do differently starting a Pidgin agent tomorrow.
The benchmark number is not your number
Every speech-to-text vendor publishes a word error rate. It's almost always measured on read speech, clean microphone, speakers whose accent matches the training data. Your users are on a phone, outside, with a generator running.
That gap is enormous. Leaders report WER under 5% on standard English sets. The Edinburgh International Accents of English Corpus put the best model tested at 19.7% average across forty-plus accents versus 2.7% on US English read speech, and the sharpest drops were on Nigerian and Jamaican English. For indigenous African languages, the AfriSpeech-MultiBench work found models that advertise support producing WERs above 100%, meaning the transcript contains more errors than it does words.
Telephony makes it worse before your model even sees the audio. Phone calls arrive at 8 kHz through a lossy codec. Models trained on studio recordings lose accuracy the moment you put a PSTN trunk in front of them, and that penalty lands hardest on exactly the languages where training data was thin to begin with.
Treat the vendor's number as a ceiling you'll never actually reach.
Why multilingual voice AI breaks in ways English never did
The thing nobody budgets for is code-switching, which now has its own in-the-wild benchmark for African speech because it wrecks so many pipelines. A Pidgin speaker doesn't speak Pidgin. They speak Pidgin, English, and often Yoruba, Igbo or Hausa inside a single sentence, switching mid-clause without pausing. A German caller does the same thing with English loanwords. Most ASR pipelines decode into one language per utterance, so the moment the speaker switches, the decoder is committed to the wrong one.
Runtime language detection is the usual patch, and it degrades to roughly 70-80% accuracy precisely during code-switched speech. One turn in four or five gets routed into the wrong decoder, and the caller experiences that as the agent going deaf for a sentence.
Detecting language per turn is the wrong abstraction
The instinct is to run language ID on every turn and route accordingly. Don't. Per-turn detection flaps: the caller says a sentence that is 60% English, gets routed to the English decoder, then the next sentence is Pidgin-heavy and routes back. Every switch costs you accuracy at the boundary, and the switching itself becomes a failure mode.
What worked better for me: pin the language at session level, decided once from the first two or three turns plus whatever you already know about the caller, and let the ASR decode multilingual underneath instead of swapping decoders. Then treat a sustained run of low-confidence turns as the signal to re-evaluate, rather than one bad turn. You want hysteresis here, the same way you'd build a circuit breaker instead of reacting to every single failed request.
If the model you are using genuinely decodes multiple languages in one pass, use that and skip the routing layer entirely. Fewer moving parts beats clever orchestration.
Your TTS voice list is shorter than your STT language list
This catches people out late. A vendor may transcribe a language and have nothing to speak it back with. Nigerian Pidgin is a good example: there's no Pidgin voice waiting for you.
The workaround is usable. Pidgin orthography is close enough to English that a Nigerian-accented English voice reading Pidgin text lands somewhere between acceptable and good. But you inherit a normalization problem that's entirely yours. Numbers, dates, currency, phone numbers and addresses all get expanded by rules that assume English, and those rules are wrong. A naira amount read out by an English normalizer is not what a Lagos caller expects to hear. Every language you add is a small text-normalization project of its own, and it's the line item that always gets left out of the estimate.
Budget for it, or your agent will be understood perfectly and still sound foreign.
You cannot evaluate what nobody on the team can read
This is the part that decides whether the project ships. If nobody on your team speaks the target language, you have no idea whether the agent works. Your dashboard is green, latency is fine, and the transcripts might be gibberish.
You need native speakers in the loop before launch, not after the complaints. Concretely: collect 200 to 300 real calls, have native speakers correct the transcripts, and hold that as a fixed evaluation set you re-run on every prompt change and every model swap. That set is the only honest measure you'll have. I wrote more about building these harnesses in how to test AI features when the output changes every time, and the discipline is the same here. The difference is that you're buying the ground truth instead of writing it.
Score two things separately. Did the agent hear the caller, and did the agent do the right thing? Conflating them means a transcription regression looks like a reasoning regression, and you'll spend a week tuning the wrong prompt.
What I would actually build for a Pidgin agent
If this landed on my desk Monday, the order would be:
- Collect real audio first. Fifty phone calls from actual target users before writing any pipeline code. Not colleagues in a quiet room.
- Benchmark three ASR providers against that audio, not against their marketing. Expect the ranking to come out different from every public leaderboard.
- Ship English plus Pidgin only. Resist the urge to launch six languages. Each one multiplies the evaluation surface.
- Pin language per session, multilingual decode underneath, confidence-based re-evaluation with hysteresis.
- Write the number and currency normalizer by hand. It's a day of work, and it's the difference between sounding local and sounding imported.
- Put a human escape hatch on every call. When confidence stays low for three turns, transfer. Callers forgive a handoff. They don't forgive being trapped.
Everything else, including barge-in handling, streaming and the latency budget, is the same work as any voice agent, and I covered that in what it actually takes to ship a voice AI agent in production.
The honest version
Multilingual voice AI is not a model selection problem. The models are good enough, and they'll keep getting better without your help. What decides whether the thing works is boring: real audio collected early, native speakers checking the output, per-language normalization, and a graceful way to give up.
Most teams skip all four, ship on the vendor's WER number, and find out from their users. That's a slow and expensive way to learn something you could have known in week one.
If you're scoping a voice project in a language your team doesn't speak, I'm happy to look at it with you. That first conversation usually saves more than it costs.