Overview

A voice AI agent that answers and places real phone calls for a service business. It listens in real time, works out who is calling (a customer, a teammate, or an unknown caller), routes the conversation through a LangGraph state machine, takes action in Google Calendar, and follows up by SMS or WhatsApp once the call ends. This was a four-engineer team project. My part: the outbound customer-verification workflow, the teammate delay-notification workflow, graceful call termination with SMS confirmation, call-outcome tracking, and the pre-generated audio fillers that hide slow tool calls from the caller.

The problem

Appointment-based businesses handle the same calls again and again: customers rescheduling, teammates reporting they are running late, and verification calls before a visit. Each follows a predictable script, but a person still has to look up the calendar, make the change, and send a confirmation. The goal was to let an agent handle these calls end to end and hand off only what it can't handle.

How it works

1. Call in or out: Twilio streams call audio over a WebSocket (Media Streams). Outbound calls are placed by the same service for verification and delay notices.
2. Listen: voice-activity detection and Deepgram streaming speech-to-text turn audio into transcripts, with interruption handling so callers can talk over the agent.
3. Identify the caller: the phone number is matched against a phonebook to decide the path: customer, teammate, or potential client.
4. Route with LangGraph: a StateGraph with conditional edges sends the call to the right node: greeting, customer intent (classified with GPT-4o-mini), teammate intent, outbound customer verification, or potential client.
5. Act with tools: LangGraph tool calls fetch, shift, or cancel appointments in Google Calendar, and only after the caller confirms.
6. Speak: Azure text-to-speech replies. Pre-generated filler phrases play while slow operations run.
7. Close the loop: the call ends gracefully, an SMS or WhatsApp confirmation goes out, and the outcome is written to MongoDB.

What I built

• Outbound customer verification: the agent calls the customer, confirms identity and appointment details, and records the result. This touched the call graph, the outbound WebSocket service, the media-stream model, and a new CustomerVerificationWorkflow (about 730 lines added in the main change).
• Teammate delay workflow: a teammate calls in to say they are late. The agent finds the affected appointments, updates the calendar, and notifies the customer.
• Graceful termination and confirmations: the agent detects when a conversation is finished, ends the call cleanly, then sends an SMS confirmation.
• Call-outcome tracking: every call's result is stored in MongoDB for follow-up and reporting.
• Latency-masking fillers: we measured the slow steps. A Google Calendar update took about 2.19 s and a calendar fetch about 617 ms. I added natural filler audio ("Let me update your calendar with the new time") so the caller never hears dead air.

Engineering decisions

• LangGraph over a single prompt: phone calls are multi-turn and stateful. Explicit nodes and conditional edges made each path testable and stopped the model from wandering between intents.
• Classify first, then act: a cheap intent classification (GPT-4o-mini) picks the workflow before any tool runs, which keeps latency and cost down.
• Confirmation before side effects: calendar changes only happen after the caller confirms, because an AI should not silently move someone's appointment.
• Fillers instead of faster APIs: we couldn't make Google Calendar faster, but we could stop the caller from noticing the wait.

Tech stack

Node.js · LangGraph · LangChain · OpenAI (GPT-4o-mini) · Twilio Voice, Media Streams, and SMS · WhatsApp via Twilio · Deepgram STT · Azure Speech TTS · Google Calendar API · MongoDB · WebSockets · React dashboard

Related work

Before this project I built a real-time phone call translator alone. It streams Twilio calls through Deepgram, translates with OpenAI and speaks the result through ElevenLabs. That is the same audio pipeline this agent is built on.

LangGraph

Twilio

Deepgram

OpenAI

Node.js

MongoDB

View project