Grok Voice API Tutorial: How to Build a Realtime AI Voice Agent for Customer Support

Every AI voice agent I built before this year was a plumbing project: a speech-to-text stream feeding an LLM feeding a text-to-speech stream, with three vendors, three latency budgets, and a mixing desk of failure modes where they meet. The model never heard the customer sigh. It read a transcript of the sigh.

xAI's Grok Voice Agent API, which got a major upgrade this month alongside the Voice Agent Builder beta, takes the other architecture: one speech-to-speech model, one WebSocket, audio in and audio out. I rebuilt a client's customer support voice agent on it to see whether the single-path promise holds up. It does, and the interesting work moves from plumbing to conversation design. This tutorial walks through the build.

Connecting to the Grok Voice Agent API

Everything happens over a WebSocket at wss://api.x.ai/v1/realtime, with the model selected by query parameter. Use grok-voice-latest unless you have a reason to pin a version. Server-side you authenticate with a Bearer key; for browser or mobile clients, mint an ephemeral token instead of shipping your API key to the client.

The first event you send is a session.update that configures the whole agent:

import asyncio, json, websockets

async def run():
    async with websockets.connect(
        "wss://api.x.ai/v1/realtime?model=grok-voice-latest",
        additional_headers={"Authorization": f"Bearer {XAI_API_KEY}"},
    ) as ws:
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "voice": "ara",
                "instructions": SUPPORT_AGENT_PROMPT,
                "turn_detection": {
                    "type": "server_vad",
                    "silence_duration_ms": 600,
                },
                "audio": {
                    "input": {"format": {"type": "audio/pcm", "rate": 24000}},
                    "output": {"format": {"type": "audio/pcm", "rate": 24000}},
                },
                "tools": [ORDER_LOOKUP_TOOL],
            },
        }))
        await asyncio.gather(pump_mic(ws), play_responses(ws))

With server_vad on, turn-taking becomes the server's problem. You stream microphone audio with input_audio_buffer.append and the API decides when the caller has finished speaking. The silence_duration_ms knob matters more than anything else in the config; 600ms feels responsive for support calls, while the default is tuned more conservatively. A manual mode exists too (turn_detection: null plus explicit commit events) for push-to-talk interfaces.

Function calling in a voice agent

Our support agent looks up orders mid-conversation. Tools are declared in the session config with JSON schemas, the same mental model as any chat API, plus built-ins for web_search, x_search, file_search, and remote MCP servers. When the model calls your function, you get a response.function_call_arguments.done event. You execute, return the output as a conversation item, and ask the model to resume:

async def handle_event(ws, event):
    if event["type"] == "response.function_call_arguments.done":
        result = await lookup_order(json.loads(event["arguments"]))
        await ws.send(json.dumps({
            "type": "conversation.item.create",
            "item": {
                "type": "function_call_output",
                "call_id": event["call_id"],
                "output": json.dumps(result),
            },
        }))
        await ws.send(json.dumps({"type": "response.create"}))

Two details we learned the hard way. When the model fires parallel tool calls, resolve all of them before sending a single response.create; sending one per result makes the model start talking with half its facts. And wait for the current audio playback to finish before response.create, or the agent talks over itself. In a voice UI these aren't style points. They're the difference between "assistant" and "malfunction."

force_message: deterministic speech for compliance scripts

My favorite feature in the whole API is the least glamorous one. Support lines have scripted utterances that legal requires verbatim: recording disclosures, identity-verification language. Asking an LLM to say something exactly is famously a coin flip. The force_message item type bypasses the model entirely and pushes scripted text straight to speech synthesis:

await ws.send(json.dumps({
    "type": "conversation.item.create",
    "item": {
        "type": "force_message",
        "role": "assistant",
        "interruptible": False,
        "content": [{"type": "output_text", "text":
            "This call may be recorded for quality and training purposes."}],
    },
}))

It completes as a full turn on its own, so don't follow it with response.create. Setting interruptible: False means the caller can't barge past the disclosure. Per-response instruction overrides give you the same targeted control in the other direction: a response.create can carry one-turn instructions, like answering a single turn in Spanish. You get deterministic control exactly where you need it and model judgment everywhere else.

Session resumption for dropped connections

WebSockets drop, and by default the conversation history drops with them, which for a phone-length call is disqualifying. Session resumption fixes it. Set resumption: {"enabled": true} in your session config, capture the conversation.id from the server's conversation.created event, and on reconnect append ?conversation_id={id} to the URL. Cached turns replay as conversation.item.created events and the agent picks up where it left off. History survives 30 minutes of inactivity, enough for any hold music ever composed. You must opt in on both the original session and the resuming one. We learned that by not doing it.

Grok Voice API pricing and where this fits

Grok Voice runs at $0.05 per minute of agent audio. A support call averaging six minutes costs thirty cents in voice-model spend. Compare that with the fully loaded cost of a human answering the same tier-one "where is my order" call and the math ends quickly, which is exactly why the design bar goes up rather than down. A bad voice agent at scale is cheap frustration, industrially produced.

The single-path architecture is what makes the quality ceiling reachable. The model hears hesitation, interruption, and tone, all things a transcript deletes, and its responses come back with natural prosody that the stitched three-API stack never managed. If you tried the pipeline approach in 2024 and shelved it, the thing you shelved no longer exists. We have a phone number on this prototype now; that build is its own post.