A voice AI agent demo is the easiest thing in the world to fake and one of the hardest things to actually ship. You wire up a speech-to-text API, pipe the transcript into an LLM, send the reply to a text-to-speech service, and record a clean 30-second clip where nobody interrupts, nobody has an accent, and the network never hiccups. It looks like magic. Then a real user talks over it, and the whole illusion collapses.
I built Intavue — an AI interview-prep platform with a real-time voice interviewer that reads your resume and asks live follow-ups — solo, end to end. The voice pipeline was, by a wide margin, the hardest part of the product. Not the LLM. Not the UI. The voice. Here is what I learned about what production actually demands, so you can budget for it before you write a line of code.
The latency budget is the whole game
Natural conversation has a rhythm. When you finish a sentence and the other person doesn't respond within roughly 500 milliseconds, your brain registers it as a pause — hesitation, confusion, a bad line. Blow past a second and it stops feeling like a conversation and starts feeling like you're leaving voicemails at each other.
That number is your entire engineering budget, and you have to split it across a chain of services. A typical stitched pipeline spends 100–300ms on speech-to-text, 350–1,000ms on LLM inference, 90–200ms on text-to-speech, and another 50–200ms bouncing across networks between vendors. Add that up honestly and you land somewhere between 600ms and 1.7 seconds — right at the edge of "natural" and often past it.
Every architectural decision after this point exists to claw back milliseconds. If you internalize one thing, make it this: you are not building a chatbot that happens to talk. You are building a real-time system with a hard latency deadline, and the AI is one component inside it.
Stop thinking in requests. Start thinking in streams
The instinct from web development is request/response: send the whole input, wait, get the whole output. That instinct is exactly what kills voice agents. If you wait for the user to finish speaking, then wait for the full transcript, then wait for the complete LLM response, then wait for all the audio to synthesize, you have stacked four sequential waits on top of each other and your latency budget is gone.
The fix is to overlap everything. Speech-to-text transcribes in chunks as the user talks. The LLM starts generating as soon as it has the first few words of a complete thought. Text-to-speech begins synthesizing the opening of the reply before the model has finished the rest of it. Done right, the agent is already speaking the first half of its answer while it's still composing the second half. That parallelism — not a faster model — is what makes the difference between "acceptable" and "uncanny." The AssemblyAI voice stack breakdown is a good map of the moving parts if you want the full topology.
This is also why voice is genuinely harder than adding a text feature to an app. If you're used to the request/response world, I wrote about that mental model in how to add AI to your existing web app without rewriting it — voice takes every assumption in that post and puts a stopwatch on it.
Interruptions are the feature, not an edge case
Here is the thing the polished demos never show you: a real person interrupts. They cut in with "no, wait" or "actually—" and a human speaker stops instantly. If your agent plows through its scripted paragraph while the user is trying to talk, it feels broken, rude, and robotic in the exact way people hate.
So you have to build interruption handling as a first-class part of the loop. That means continuously listening even while the agent is speaking, detecting when the user has started talking, killing the in-flight audio immediately, discarding the response the model was mid-way through generating, and pivoting to the new input — all in a few hundred milliseconds. On Intavue, an interviewer that couldn't be interrupted mid-question felt useless; the entire point is a candidate reacting in real time. Getting barge-in to feel natural took more iteration than any other single feature in the voice layer.
Where the cost of a voice AI agent actually goes
Founders always ask about the model cost first, and it's almost always the wrong thing to worry about. Ongoing inference for speech-to-text, the LLM, and text-to-speech runs somewhere around $0.008–$0.06 per minute of conversation. At 10,000 minutes a month that's roughly $80–$600 — real, but rarely the thing that sinks a project.
The cost that actually hurts is engineering time. The gap between a working demo and a production-hardened agent — streaming everything, handling interruptions, dealing with packet loss and degraded audio, recovering when one vendor in the chain times out — is where the weeks go. Industry build estimates put a basic single-purpose voice agent at $15,000–$35,000 and a real multi-intent production system at $35,000–$80,000, and that spread is almost entirely about how much of that hardening you're willing to do. The API bill is a rounding error next to it.
The part nobody demos: failure
Every service in your chain will fail eventually. The speech-to-text vendor has a bad minute. The LLM provider rate-limits you. A packet drops and the audio arrives garbled. In a text app you show a spinner and retry. In a voice conversation there is no spinner — there's just dead air, and dead air on a live call reads as "this thing is broken."
Production voice means designing for graceful degradation: timeouts short enough that you fall back before the silence becomes obvious, a filler response to buy a beat when the model is slow, and a way to recover the conversation state when a component drops. This is unglamorous work that never makes it into a launch video, and it's most of what separates a prototype from something you'd put in front of paying users. It's the same discipline I lean on everywhere in production AI — the kind of "assume it will break" thinking I applied to retrieval in why your RAG pipeline is probably broken.
Build vs. buy your voice AI agent — the honest version
If you need a voice agent live in under six months and nobody on your team has done real-time media, streaming inference, and production voice testing before, use a managed platform or hire someone who has done it. There's no shame in it, and the orchestration tooling in 2026 is genuinely good.
Build custom when the voice experience is the product — when a generic agent won't do and the differentiation lives in how the conversation feels. That was true for Intavue: the interview only works if the follow-ups land in real time and the candidate can push back mid-answer. If your voice layer is a commodity, buy it. If it's the reason people would pay you, own it.
What I'd tell you before you write a line of code
Decide your latency target first, because it dictates your entire architecture. Design the whole pipeline as overlapping streams, never sequential requests. Treat interruptions and failure as core features, not polish you'll add later. And be honest about whether the voice experience is your actual differentiator or just a feature you think sounds cool — because production voice is a real commitment, and it's worth making only when the conversation itself is the thing people are paying for.
If you're weighing a build like this and want a second opinion from someone who's shipped one end to end, that's exactly the kind of problem I like to talk through.