How to Connect an AI Voice Agent to a Real Phone Number: Grok Voice SIP Setup and Cost Breakdown

The most requested AI project in our inbox this year isn't a chatbot. It's a phone number. Specifically: "our support line closes at 6 p.m. and our customers are still calling — can an AI answer the after-hours calls?" It's the perfect first voice-agent deployment, because the alternative isn't a human doing a better job. The alternative is voicemail.

Last week's post covered building a realtime agent on the Grok Voice Agent API over WebSocket. This one covers what makes it an AI phone answering service: SIP integration, telephony audio, and the operational details that only show up when calls arrive from the public switched telephone network instead of a laptop microphone.

Native G.711 support, no transcoding layer

Phone audio is G.711: μ-law in North America and Japan, A-law most other places, 8kHz, a codec standardized in 1972. Every voice AI stack I deployed before this one had a transcoding hop. G.711 in from the carrier, resample and convert for the model, convert back out. Extra latency, an extra service, an extra thing that breaks at 2 a.m.

Grok Voice accepts audio/pcmu and audio/pcma natively in the session config, so the 8kHz telephone audio goes straight into the model:

"audio": {
    "input":  {"format": {"type": "audio/pcmu"}},   # G.711 mu-law, 8kHz
    "output": {"format": {"type": "audio/pcmu"}},
},

xAI's SIP integration routes PSTN, contact-center, and PBX calls directly into a speech-to-speech session. A provisioned phone number bills at $0.01 per minute of telephony on top of the $0.05 per minute of agent audio. If you're already on Twilio or a contact-center platform, the media-stream bridge pattern works too, and xAI ships a Twilio telephony reference app. For a greenfield after-hours line, direct SIP means one less vendor in the call path.

Tuning voice activity detection for phone calls

The VAD settings that feel great on a headset are wrong for telephony, and this is where the prototype-to-production gap lives. Three adjustments that mattered on real calls:

Raise the silence duration. People on phones pause differently. They check the order number on another screen, or talk to someone in the room. We moved silence_duration_ms from 600 to about 900 for the after-hours line. Slightly slower turn-taking beats interrupting a customer mid-thought.

Raise the VAD threshold for noisy callers. Phone calls come from cars, kitchens, and sidewalks. The default turn_detection.threshold of 0.85 held up better than I expected, but for lines with known-noisy callers, pushing toward 0.9 stopped the agent from responding to a background television.

Turn on the idle timeout. turn_detection.idle_timeout_ms re-engages the model when the caller goes quiet after a response. On the web this is optional. On a phone it's essential, because silence on a call is ambiguous: did they hang up, or are they hunting for their invoice? A gentle "take your time, I'm still here" after about 8 seconds measurably reduced hang-ups in our call reviews.

Scripting the compliance opening

Every recorded support line has a disclosure requirement, and it has to be verbatim. The force_message item type synthesizes scripted text directly, with no model involvement:

{"type": "conversation.item.create",
 "item": {"type": "force_message", "role": "assistant",
          "interruptible": False,
          "content": [{"type": "output_text",
                       "text": "You've reached Acme support after hours. "
                               "This call is recorded for quality purposes."}]}}

We fire that the moment the SIP leg connects, before the model says a word. From the same toolbox, the replace map fixes pronunciations the speech synthesis gets wrong, mostly client brand names, without touching transcripts. Mapping "Skcript" to "Skript" saved us a very awkward demo.

Scope the agent before you scale it

An after-hours AI answering service doesn't need to resolve everything. Its job is to resolve the easy majority and capture the rest well. Our tool set for the line is deliberately small: order lookup, ticket creation, and a schedule_callback function that books a slot on the morning team's calendar. The system prompt is explicit that when confidence is low or the caller is frustrated, the right move is "I'll have a specialist call you at 9 a.m., and I've already created the ticket with everything you told me." Not another attempt.

The transcripts make this reviewable. Every turn comes back as text alongside the audio events, so the morning team starts the day with a queue of structured tickets plus readable transcripts, instead of fourteen voicemails of variable audibility.

AI phone agent cost versus an answering service

Our pilot line handles about 220 after-hours calls a month, averaging 5 minutes. The complete voice bill:

• Agent audio: 1,100 min × $0.05 = $55
• Telephony: 1,100 min × $0.01 = $11

Sixty-six dollars a month, plus API calls to our own backend. A human answering service quoted the same client roughly $1.10 per minute with no system access — over $1,200 a month to take messages. The agent resolves about 70% of calls outright (order status, returns initiation, store hours) and books callbacks for the rest.

The honest caveat: that 70% comes from scope discipline, not model magic. We launched with three tools and a narrow promise, measured for a month, then expanded. The teams that fail at AI voice agents launch with "handle our support calls" as the spec. Phones are an unforgiving medium. There's no UI to hide behind, and the caller decides within about ten seconds whether they're talking to something competent. Start with the after-hours line, where every alternative is worse, and let the transcripts tell you when you're ready for daytime.