Building a Real-Time Voice Agent with Vercel AI SDK and LiveKit

Text-based chatbots are a solved problem. The real challenge in 2026 is voice.

When humans speak to each other, the interaction is highly dynamic. We interrupt each other, we use backchanneling ("uh-huh", "yeah"), and we expect responses within milliseconds. If you try to build a voice agent by chaining together REST APIs (Speech-to-Text -> LLM -> Text-to-Speech), you will inevitably end up with 2 to 3 seconds of latency. In a voice conversation, a 3-second delay feels like an eternity.

To build a production-grade voice agent, you must abandon HTTP polling and embrace WebRTC. In this massive 2,000+ word tutorial, we will build an ultra-low-latency voice agent using the Vercel AI SDK to handle the LLM orchestration and LiveKit to handle the real-time WebRTC audio streaming.

The Latency Problem in Voice

Let's dissect why naive voice agents fail.

If you use a traditional HTTP stack, the user speaks into their browser. The browser records a .wav file, waits for the user to stop speaking, and uploads the file to a server. The server sends it to OpenAI Whisper. Whisper takes 500ms to transcribe it to text. The server sends the text to GPT-4. GPT-4 takes 800ms to generate a response. The server sends the response to ElevenLabs. ElevenLabs takes 400ms to generate the audio. Finally, the audio is sent back to the browser.

You have lost at least 2.5 seconds, and the user has already assumed the bot is broken and said "Hello? Are you there?"

Why WebRTC & LiveKit?

WebRTC establishes a persistent, peer-to-peer UDP connection between the client browser and the server. Audio is streamed in tiny, continuous chunks (usually 20ms each).

LiveKit is an open-source WebRTC infrastructure platform. By routing our audio through LiveKit, we achieve two things:

1. Continuous Streaming: The server receives audio bytes the exact millisecond the user speaks them.
2. Interruption Handling: If the AI is speaking and the user interrupts, we can instantly drop the WebRTC audio track, stopping the AI mid-sentence.

The Vercel AI SDK fits perfectly into this architecture because it treats LLM outputs as native streams. We can pipe the Vercel AI SDK's text stream directly into our TTS engine, and pipe the TTS audio chunks directly into LiveKit.

Step 1: Backend Setup and Architecture

We will build this using a Next.js App Router backend.

npx create-next-app@latest voice-agent
cd voice-agent
npm install ai @ai-sdk/openai livekit-server-sdk @livekit/components-react @livekit/components-styles

You will need a LiveKit Cloud account (or a local self-hosted instance) and an OpenAI API key. Create your .env.local file:

OPENAI_API_KEY=sk-your-key
LIVEKIT_API_KEY=your-livekit-api-key
LIVEKIT_API_SECRET=your-livekit-secret
NEXT_PUBLIC_LIVEKIT_URL=wss://your-instance.livekit.cloud

Step 2: Generating the LiveKit Token

To allow a browser to connect to a WebRTC room, your secure backend must generate an access token. We will create a Next.js API route for this.

Create app/api/token/route.ts:

import { AccessToken } from 'livekit-server-sdk';
import { NextResponse } from 'next/server';

export async function GET(request: Request) {
  const roomName = 'voice-agent-room';
  const participantName = `user-${Math.floor(Math.random() * 10000)}`;

  const at = new AccessToken(
    process.env.LIVEKIT_API_KEY,
    process.env.LIVEKIT_API_SECRET,
    {
      identity: participantName,
    }
  );

  at.addGrant({ roomJoin: true, room: roomName });

  return NextResponse.json({ token: await at.toJwt() });
}

Step 3: The Frontend React Interface

We need a UI where the user can click "Connect" and start speaking. We will use LiveKit's React components, which abstract away the complex WebRTC connection logic.

Create app/page.tsx:

"use client";

import { useEffect, useState } from 'react';
import { LiveKitRoom, RoomAudioRenderer, BarVisualizer, useVoiceAssistant } from '@livekit/components-react';
import '@livekit/components-styles';

export default function VoiceAgentPage() {
  const [token, setToken] = useState("");

  useEffect(() => {
    // Fetch the token we generated in Step 2
    fetch('/api/token')
      .then((res) => res.json())
      .then((data) => setToken(data.token));
  }, []);

  if (!token) return <div className="p-10 flex justify-center text-zinc-400">Loading secure connection...</div>;

  return (
    <div className="min-h-screen bg-[#09090b] flex flex-col items-center justify-center font-sans text-zinc-100">
      <div className="max-w-md w-full p-8 border border-zinc-800 rounded-3xl bg-zinc-900/50 shadow-2xl flex flex-col items-center">
        <h1 className="text-2xl font-bold mb-8 tracking-tight">AI Voice Agent</h1>
        
        <LiveKitRoom
          video={false}
          audio={true}
          token={token}
          serverUrl={process.env.NEXT_PUBLIC_LIVEKIT_URL}
          connect={true}
        >
          <RoomAudioRenderer />
          <AgentVisualizer />
        </LiveKitRoom>
      </div>
    </div>
  );
}

// A custom component to show when the AI is speaking
function AgentVisualizer() {
  const { state, audioTrack } = useVoiceAssistant();
  
  return (
    <div className="flex flex-col items-center gap-6">
      <div className={`text-sm uppercase tracking-widest font-semibold ${
        state === 'speaking' ? 'text-emerald-400' : 'text-zinc-500'
      }`}>
        {state === 'speaking' ? 'Agent is speaking...' : 'Agent is listening...'}
      </div>
      
      {audioTrack && (
        <div className="h-24 w-full max-w-[200px] flex items-center justify-center">
          <BarVisualizer state={state} barCount={5} trackRef={audioTrack} />
        </div>
      )}
    </div>
  );
}

This frontend connects the user's microphone to the LiveKit room and automatically plays any audio that the AI pushes back into the room.

Step 4: The Node.js Agent Worker

Now for the complex part. We need a backend worker that listens to the LiveKit room. When it detects speech, it runs the Vercel AI SDK, converts the text to speech, and pushes the audio back.

Normally, you would run this in a separate Node.js process (not a Next.js serverless route, because WebRTC requires persistent long-lived connections). Let's write the worker script: agent-worker.ts.

import { Room, RoomEvent, RemoteParticipant, RemoteTrackPublication, RemoteAudioTrack } from 'livekit-server-sdk';
import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';

// Connect to the LiveKit room as an administrative participant
const room = new Room(
  process.env.NEXT_PUBLIC_LIVEKIT_URL!,
  process.env.LIVEKIT_API_KEY!,
  process.env.LIVEKIT_API_SECRET!
);

async function startAgent() {
  await room.connect('wss://your-instance.livekit.cloud', process.env.LIVEKIT_TOKEN!);
  console.log("Agent connected to WebRTC room.");

  // Listen for users turning on their microphones
  room.on(RoomEvent.TrackSubscribed, (track, publication, participant) => {
    if (track.kind === 'audio') {
      console.log(`Subscribed to audio from ${participant.identity}`);
      handleUserAudio(track as RemoteAudioTrack, participant);
    }
  });
}

// This function processes the incoming audio track
function handleUserAudio(track: RemoteAudioTrack, participant: RemoteParticipant) {
  // In a real production app, you would pipe this track into a VAD (Voice Activity Detector)
  // like Silero VAD to detect when the user STOPS speaking.
  // For this tutorial, we will mock the VAD event.
  
  const vad = new MockVAD(track);
  
  vad.on('speech_ended', async (transcribedText) => {
    console.log(`User said: ${transcribedText}`);
    
    // We use Vercel AI SDK to stream the response
    const result = await streamText({
      model: openai('gpt-4o'),
      messages: [{ role: 'user', content: transcribedText }],
    });

    // Pipe the text stream into our TTS engine, and pipe the audio into the room
    streamToRoom(result.textStream);
  });
}

Step 5: Streaming from Vercel AI SDK to Audio

The magic of the Vercel AI SDK is the textStream. As GPT-4 generates tokens, we do not wait for the sentence to finish. We immediately push chunks of text into a fast TTS provider (like Deepgram or ElevenLabs API).

async function streamToRoom(textStream: AsyncIterable<string>) {
  let sentenceBuffer = "";

  for await (const textChunk of textStream) {
    sentenceBuffer += textChunk;
    
    // Once we have a complete phrase or sentence, synthesize it
    if (sentenceBuffer.match(/[.!?]/)) {
      const audioBytes = await synthesizeSpeech(sentenceBuffer);
      sentenceBuffer = ""; // Reset buffer
      
      // Push the audio bytes to the LiveKit WebRTC track
      await pushAudioToLiveKit(audioBytes);
    }
  }
}

Handling Interruptions

What makes this architecture superior to HTTP is interruption handling.

If the AI is currently playing an audio track into the LiveKit room (via pushAudioToLiveKit), and our VAD detects that the human has started speaking again, we can instantly call track.stop(). The WebRTC stream terminates immediately, cutting the AI off mid-sentence. We then cancel the Vercel AI SDK streamText execution, saving API costs and creating a natural, conversational dynamic.

Conclusion

Building real-time voice agents requires a massive shift in how you think about data flow. You must move from request/response paradigms to continuous streaming pipelines.

By combining the standardized, ergonomic streaming APIs of the Vercel AI SDK with the robust WebRTC infrastructure of LiveKit, you bypass the historic complexity of voice engineering. You are no longer managing byte arrays and network latency; you are orchestrating streams of intelligence.

Frequently Asked Questions (FAQ)

What is the difference between REST and WebRTC for Voice AI? REST requires recording full audio files, uploading them, waiting for processing, and downloading the result (causing 2-5 seconds of latency). WebRTC streams audio chunks in real-time (UDP), reducing latency to under 500ms and allowing instant interruptions.

How do I handle user interruptions (barge-in) with Vercel AI SDK? When the WebRTC Voice Activity Detector (VAD) detects the user speaking, you immediately invoke track.stop() on the LiveKit frontend and use an AbortController to cancel the Vercel AI SDK streamText execution on the backend.

Can I use ElevenLabs with Vercel AI SDK? Yes. You can pipe the textStream from the Vercel AI SDK directly into ElevenLabs' WebSockets API to generate audio chunks as the LLM thinks.