How do AI voice agents work, and what can you build with one?

What a voice agent actually is

A voice agent is software that holds a spoken conversation with a human over a phone call, in-browser audio, or an embedded device. The agent listens to the caller, decides what to say back, either performs an action through a tool call or hands off to a person, and may keep context across turns.

Under the hood, most production voice agents stitch together three pieces: a speech-to-text model that turns the caller audio into text, a reasoning model that decides what to do next, and a text-to-speech model that streams the response back. The newer alternative is a fully speech-to-speech model that handles audio in and audio out without a separate transcription step. The OpenAI realtime API, Google Gemini Live, and Amazon Nova Sonic are examples of the latter; Vapi, Retell, and Bland bundle the former behind a hosted product.

How the pieces fit together

The voice stack is a turn-taking problem. The agent has to detect when the caller is done, stream a response with low latency, and usually interrupt itself when the caller talks over it. Most platforms ship a Voice Activity Detection (VAD) layer and a transport layer (WebRTC, Twilio Media Streams, SIP) that delivers audio packets in real time.

On top of that, the agent has a reasoning loop. It receives the caller text, often with a system prompt and a tool list, picks an action, and either speaks, calls a tool, or defers to a human. Low latency is the difference between a usable conversation and a frustrating one: the difference between an agent that finishes a sentence in 600 ms and one that takes 2.5 s is the difference between a real call and a held breath.

• Speech-to-text + LLM + text-to-speech: easier to swap models, more transcription steps, latency overhead is roughly 600-1500 ms.
• Speech-to-speech model: lower latency (often 300-600 ms), fewer intermediate artifacts, harder to debug because the model internal reasoning is hidden.
• Turn-taking and interruption: the agent must decide when to stop talking when the caller speaks; this is often the user-visible quality ceiling.

What people actually build with them

The patterns that work are the ones where the workflow is conversational, latency-sensitive, and full of routine paths. Intake calls for medical and dental practices, scheduling for service businesses, lead qualification for real estate and home services, post-call surveys, debt-collection outreach within tight regulatory bounds, and tier-one customer support filtering are the workloads that show up across the same-day HN posts and the Google Suggest vertical list.

The patterns that fail are the ones where the agent is asked to do something it cannot verify. Voice agents cannot reliably authenticate a caller by voice alone, cannot read a complex contract, and cannot replace a human in a regulated disclosure. Treating the agent as a routing layer, not an authority, keeps the call honest.

The settings where risks are inherent

A voice agent with a phone number and a function-call interface is closer to a customer-service rep than to a chatbot. That is also where the risk compounds. The agent can take actions in real time, on a real call, with a real human on the other end - and that human can be manipulated.

The risks worth budgeting for are not exotic. Caller-side prompt injection (someone on the call trying to get the agent to perform an unintended action), account takeover via voice-only authentication, accidental disclosure of sensitive data through the agent tool calls, and silent billing when the agent is cheaper than a human but still racking up per-minute model charges. A human-in-the-loop boundary at the highest-risk actions - refunds, account changes, regulated disclosures - is the simplest control.

• Identity: never let the agent treat voice verification as proof of identity. Use a separate factor or hand off to a human.
• Tool calls: keep the agent action surface narrow. Reading an account is fine; closing an account is not.
• Prompt injection: assume the caller is adversarial. Treat the caller speech as untrusted input.
• Cost: meters run in real time. Cap call duration, set per-call budgets, and alert on runaway sessions.

A useful next action

Pick one workflow that is conversational, latency-sensitive, and dominated by routine paths. Build a narrow prototype with one platform and one tool, in a sandboxed account, with a human in the loop before any irreversible action. Measure the latency, the resolution rate, the per-call cost, and the failure modes the prototype produces. Only after the prototype behaves the way you expect on real-looking calls should you turn it loose on a small slice of real traffic.