How to Build Sub-Second Voice AI Agents With Twilio, WebSockets, and Small LLMs

There is a hard threshold in conversational human psychology: 700 to 800 milliseconds. When two people talk over the phone, the typical gap between one person stopping speaking and the other beginning to respond is between 200 and 400 milliseconds.

If an automated voice agent responds within 500 milliseconds, the conversation feels responsive and intuitive. If the agent takes 1,400 milliseconds, the caller assumes the connection dropped, begins speaking again just as the agent starts its response, and enters an agonizing cycle of mutual interruptions and awkward pauses.

Most developer demos achieve low latency by running on high-speed fiber with lightweight text-to-speech models running locally. But when you deploy that same voice bot over public switched telephone networks (PSTN) through Twilio or SIP trunks, latency explodes:

PSTN Transport (120ms) + Twilio Gateway (80ms) + STT Audio Buffer (350ms) +
LLM Time-to-First-Token (600ms) + TTS Synthesis (250ms) + Jitter Buffer (100ms)
= 1,500ms Total Latency (Conversation Broken)

At 1.5 seconds, customers hang up.

To build voice agents that actually work in enterprise call centers, you must systematically trim milliseconds off every hop in the pipeline. Here is the engineering blueprint we use at FoundrySoft to achieve sub-600ms roundtrip voice latency across standard telephony networks.

The pipeline: Breaking down the latency budget

To hit a 600ms roundtrip target over real cellular and landline connections, every component in your pipeline must operate within a strict budget:

• Telephony Ingress & Egress (PSTN/Twilio): ~150ms
• Streaming Speech-to-Text (STT): ~120ms
• LLM Time-to-First-Token (TTFT): ~180ms
• Text-to-Speech (TTS) First Audio Chunk: ~100ms
• Network buffer & protocol overhead: ~50ms
• Total Roundtrip Budget: 600ms

To achieve this, the entire pipeline must operate on a continuous streaming basis. You cannot wait for the user to finish speaking, buffer the entire WAV file, send it to a transcription API, wait for a full text reply, and then synthesize speech. Every stage must process tokens and audio chunks concurrently.

Step 1: Full-duplex WebSockets with Twilio Media Streams

Never use standard HTTP webhooks for conversational voice. Twilio Media Streams allows you to open a bidirectional WebSocket connection directly to your application server, streaming raw audio in 20ms chunks using 8kHz mu-law encoding.

Here is how we configure the Twilio Voice webhook using TwiML:

<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="wss://voice.yourdomain.com/media-stream">
      <Parameter name="accountRef" value="enterprise_lead_99" />
    </Stream>
  </Connect>
</Response>

On your application server (running Node.js, Bun, or Python with FastAPI), you establish a persistent WebSocket listener that ingests inbound audio frames and immediately pumps them to your streaming transcription engine:

import { WebSocketServer } from "ws";

const wss = new WebSocketServer({ port: 8080 });

wss.on("connection", (ws) => {
  let streamSid = "";

  ws.on("message", (message: string) => {
    const data = JSON.parse(message);

    if (data.event === "start") {
      streamSid = data.start.streamSid;
      console.log(`Stream started: ${streamSid}`);
    } else if (data.event === "media") {
      // Raw 8kHz mu-law audio payload
      const audioPayload = Buffer.from(data.media.payload, "base64");
      sttStream.write(audioPayload);
    } else if (data.event === "stop") {
      console.log(`Stream stopped: ${streamSid}`);
    }
  });
});

Step 2: Streaming transcription and Voice Activity Detection (VAD)

To know when the user has finished speaking without waiting for a two-second silence timeout, you need server-side Voice Activity Detection (VAD).

We use Silero VAD running locally inside the media server container. Silero inspects audio frames every 30 milliseconds, distinguishing human speech from background street noise, breath sounds, and keyboard clicks.

When Silero detects speech cessation for 250 milliseconds, it fires an immediate SPEECH_STOPPED event. Your streaming transcription client (such as Deepgram Nova-3 or AssemblyAI Streaming) finalizes the transcript, and the text is dispatched to the LLM within 40 milliseconds of the caller closing their mouth.

Step 3: Speculative LLM streaming and TTFT optimization

The biggest latency variable in the chain is the language model. If your agent uses a massive frontier model with a cold prompt cache, time-to-first-token can easily hit 1,200ms.

We solve this using two techniques:

1. Prefixed prompt caching with small instruction sets

Keep the voice agent's system prompt concise and strictly cached. Voice conversations require conversational rhythm, not encyclopedic paragraphs. Instruct the model to speak in single, clear sentences:

You are an inbound dental clinic receptionist.
Answer questions directly in one or two short sentences.
Never use markdown formatting, lists, or asterisks.
Speak naturally and concisely.

2. Fast speculative draft models

For routine turns (greetings, confirmations, slot booking), we route the first turn through an ultra-fast 8B parameter model running on private TensorRT-LLM or vLLM nodes, achieving a TTFT under 90 milliseconds. If the conversation branches into complex insurance eligibility rules, the orchestration layer transparently hands context off to a larger frontier model.

Step 4: Chunked sentence synthesis and acoustic barge-in

Do not wait for the LLM to finish generating a paragraph before calling text-to-speech. As the model streams tokens, buffer them into clause or sentence boundaries using punctuation delimiters (., ?, !, ,).

As soon as the first clause completes (for example: "Certainly, I can check that for you."), transmit those five words to a streaming TTS provider like Cartesia or ElevenLabs WebSocket API. Within 80 milliseconds, the TTS provider returns the first audio frame. You convert the PCM audio to 8kHz mu-law and stream it back through Twilio to the caller's ear.

While the caller is hearing the first sentence, the LLM finishes generating the second sentence. The processing pipeline overlaps completely with the caller's listening time.

Implementing acoustic barge-in (interruption handling)

What happens if the caller interrupts the agent mid-sentence? Without barge-in, the bot continues talking over the user, creating a confusing and frustrating experience.

When your server-side VAD detects incoming caller speech while the bot is currently playing audio:

1. Send an immediate clear event to Twilio to flush the audio buffer on the phone line:
   JSON

   Copy

   {
     "event": "clear",
     "streamSid": streamSid
   }
   

2. Abort the ongoing LLM generation request.
3. Stop forwarding pending TTS chunks.
4. Switch to listening mode immediately.

The caller hears the bot stop instantly the moment they open their mouth, creating an authentic conversational dynamic.

The business return on voice automation

A national emergency roadside assistance dispatch network deployed our low-latency voice agent architecture across their after-hours intake lines:

• Average speed-to-answer dropped from four minutes on hold to zero seconds.
• Average call resolution time for basic towing dispatches fell from 6.5 minutes with human operators to 2.2 minutes with the voice agent.
• Dispatch operating costs dropped by 68 percent while emergency arrival satisfaction scores increased by twenty-four points.

Voice AI is no longer a clumsy IVR phone tree with speech recognition taped on top; it is an intelligent, real-time operating layer for enterprise communications. If your organization is looking to modernize customer service, inbound dispatch, or appointment scheduling with low-latency voice agents, our systems team at FoundrySoft designs and deploys custom telephony voice pipelines built for enterprise scale. Talk with our voice systems engineers to review your technical requirements.