Building a Realtime Voice Support Agent with Vercel AI SDK v7

Every voice agent I have shipped before this year had the same architecture problem: the moment you open a WebSocket to a realtime model, you are writing against that one provider's event format. Session events, audio chunks, tool call frames, turn detection, all of it proprietary. Switching providers meant rewriting the client.

Vercel AI SDK v7 takes a swing at this with provider-agnostic realtime support. The session runs in the browser, connects directly to the provider over WebSockets, and your React code talks to one hook: experimental_useRealtime. I rebuilt a use case I know well, a customer support agent that answers "where is my order?" over voice, to see how much of the old pain is actually gone.

The token dance

The browser never sees your API key. Your server mints a short-lived token, and the client uses that to open the socket. The server side is a few lines:

// app/api/realtime/setup/route.ts
import { openai } from '@ai-sdk/openai';

export async function POST() {
  const token = await openai.experimental_realtime.getToken({
    model: 'gpt-realtime',
  });
  return Response.json(token);
}

This endpoint is effectively a key-vending machine, so put your normal auth and rate limiting in front of it. Anyone who can hit it can open billable voice sessions.

The client is one hook

On the client, experimental_useRealtime manages the WebSocket, microphone capture, and audio playback. You configure the session where you create it:

'use client';
import { experimental_useRealtime } from '@ai-sdk/react';
import { openai } from '@ai-sdk/openai';

const realtime = experimental_useRealtime({
  model: openai.experimental_realtime('gpt-realtime'),
  api: { token: '/api/realtime/setup' },
  sessionConfig: {
    instructions:
      'You are a support agent for an online store. Look up orders before answering. Be brief.',
    voice: 'alloy',
    inputAudioTranscription: {},
    turnDetection: { type: 'server-vad' },
  },
  onToolCall: async ({ toolCall }) => {
    if (toolCall.toolName === 'getOrderStatus') {
      const res = await fetch('/api/orders/status', {
        method: 'POST',
        body: JSON.stringify(toolCall.args),
      });
      return res.json();
    }
  },
});

Calling realtime.connect() starts the session; realtime.messages gives you a normal UIMessage[] array, so the transcript renders with the exact same components I already had for text chat. That detail alone saved me an afternoon.

Tool calls run in the client

This was the architectural surprise: realtime tool execution is client-driven. The provider sends the tool call over the socket, your onToolCall handler runs it, and the result flows back into the conversation. There is no server-side agent loop.

For my order lookup that is the right shape. The browser calls my own authenticated API route, the route talks to the database, and the model gets back a small JSON payload while the user hears "let me check that for you." For anything sensitive, the pattern is the same as regular client code: the tool handler hits your API, and your API enforces who can see what. If you need to feed a result back manually, realtime.addToolOutput(toolCallId, result) does it.

The transcription config matters more than it looks, too. With inputAudioTranscription on, you get text of what the caller actually said, which is the difference between debuggable sessions and listening to recordings.

The provider-agnostic part is real

The reason this is a v7 story and not just an OpenAI story: swap the model line and the rest of the code stands still. Google's Gemini Live and xAI's Grok voice models plug into the same hook, and the AI Gateway normalizes all of them behind one interface:

// server: mint the token through the gateway instead
const token = await gateway.experimental_realtime.getToken({
  model: 'openai/gpt-realtime-2',
});

Change the model string, redeploy, and you are on a different provider. My session config and tool handlers did not change. That is the thing I could not do last year without a rewrite.

Why this matters

The demo took an evening: press a button, ask where order 4271 is, hear the agent check the database and answer. Voice quality and latency are the provider's problem now, and choosing a provider is a config change instead of an architecture decision.

It is experimental and the API will move. But if you have been putting off a voice interface because the last attempt welded you to one vendor's event soup, v7 is the first version where I would start that project again.