Why voice is harder than chat
Text chat hides latency. Voice exposes every millisecond — awkward pauses, talk-over, and robotic cadence break trust immediately.
A production voice agent stitches together four layers:
- Speech-to-text (STT) — streaming transcription with barge-in support
- LLM reasoning — intent, tool calls, policy checks
- Text-to-speech (TTS) — low-latency, natural prosody
- Telephony / WebRTC — phone numbers, SIP, or in-browser audio
Get one layer wrong and the whole experience feels broken, even if the model is smart.
Platform landscape (2025–2026)
Deepgram — unified voice agent API
Deepgram's Voice Agent API bundles STT, orchestration, and TTS in one WebSocket pipeline. It supports bring-your-own LLM and emphasizes enterprise deployment (HIPAA, VPC, self-hosted options). Strong choice when you want one vendor for the speech stack.
ElevenLabs — Conversational AI 2.0
ElevenLabs leads on voice quality and turn-taking. v2 adds integrated RAG, automatic language detection, multi-character personas, and full inbound/outbound telephony with SIP trunking — useful for branded customer-facing agents.
Vapi — orchestration for developers
Vapi sits in the middle: telephony, interruption handling, tool calling, and state management — with pluggable STT/TTS/LLM providers. Deepgram models are commonly configured as the speech layer inside Vapi deployments, which is a pattern I use when speed-to-ship matters more than building audio plumbing from scratch.
Twilio — the phone network
Twilio remains the default for PSTN numbers, Media Streams, and SMS fallbacks. Most voice-agent platforms integrate Twilio for real phone calls rather than browser-only demos.
Caller → Twilio / WebRTC
→ Vapi (or custom orchestrator)
→ Deepgram STT (streaming)
→ LLM + tools (OpenAI / Claude)
→ ElevenLabs or Deepgram TTS
→ Caller hears response (<800ms target)
Keep business logic outside the audio loop: lookup CRM records, book appointments, and escalate to humans through normal APIs the LLM calls as tools.
- Barge-in — user can interrupt; agent stops speaking
- Fallback paths — "I didn't catch that" + DTMF or SMS backup
- PII handling — redact or avoid logging raw audio/transcripts where regulated
- Latency budget — measure end-to-end; optimize the slowest hop first (often TTS or LLM)
- Evaluation — scripted call scenarios with expected tool calls and phrases
Voice fits products that are hands-busy or high-trust: booking lines, interview prep (Optivus), proctoring support, and IELTS practice flows. The same agent brain can power web voice widgets and phone lines if orchestration is separated from transport.
Voice agents are converging fast — fewer teams will stitch STT, LLM, and TTS manually. Whether you use an all-in-one API or Vapi + Deepgram + ElevenLabs, success still comes down to tight loops, clear tools, and ruthless latency testing.
Build the conversation like a product, not a demo script.