How to Build a Multilingual AI Voice Agent with Grok Voice: Hindi, Spanish, and 20+ Languages

The dirty secret of most "multilingual" support deployments is that they're N monolingual deployments in a trench coat. An IVR menu asks the caller to press 2 for Spanish, routes to a separate agent config with a separate prompt and a separate voice, and God help the caller who switches languages mid-sentence because their billing question uses English product names.

This month xAI shipped 21 new multilingual voices for Grok Voice, on top of the original five, available across the realtime Voice Agent API, the Text-to-Speech API, and the Voice Agent Builder, plus naturalness upgrades to the originals. Combined with the model's native language auto-detection, the trench coat can finally come off. We rebuilt a client's support line as one multilingual AI voice agent that takes English, Hindi, and Spanish calls on the same number. Here's the build.

Language auto-detection does the routing

The core capability requires no configuration at all. Grok Voice detects the input language and responds naturally in the same language, mid-conversation, with no menu and no session restart. A caller who starts in English and drops into Hindi gets Hindi back. The supported list covers 20+ languages with native-quality accents, including the regional variants that matter operationally: pt-BR versus pt-PT, es-MX versus es-ES, three Arabic variants.

What you configure are the edges where auto-detection needs help:

"audio": {
    "input": {
        "format": {"type": "audio/pcm", "rate": 24000},
        "transcription": {
            "language_hint": "hi",
            "keyterms": ["Foundrysoft", "UPI mandate", "SKU-4417", "Navi Mumbai"],
        },
    },
},

language_hint is a bias, not a lock. It tells transcription which language to expect, which matters enormously for short utterances; "haan" and "hmm" are indistinguishable without a prior. If your telephony provider gives you the caller's country code, map it to a hint at session start. The model still follows the caller wherever they actually go.

keyterms (up to 100 of them) fixes the other classic failure: domain words that transcription mangles. Product names, SKU formats, and, critically for Indian deployments, the English brand and payment terms embedded in Hindi sentences. Code-switched speech is normal speech for most of the world. Seeding the vocabulary is the difference between an agent that understands "mera UPI mandate cancel karna hai" the first time and one that asks the caller to repeat themselves.

Put your language policy in the prompt

Auto-detection handles which language. Your instructions handle how the business speaks in it. Our session prompt carries a short language-policy section per supported language: greeting register (formal "aap" in Hindi, "usted" for Mexican Spanish), whether product names stay in English (they do), and what happens for unsupported languages (apologize in English, offer a callback with a human speaker).

For scripted moments, per-response instruction overrides are cleaner than prompt gymnastics. A response.create can carry one-turn instructions such as "deliver the closing summary in the caller's language, but read the ticket number in English digits." Ticket numbers read aloud as Hindi numerals produced our funniest QA clip and our clearest lesson.

Pronunciation maps, one per market

The replace config maps written phrases to spoken substitutions. It affects audio only, leaves transcripts untouched, matches case-insensitively, and resolves overlaps by longest match:

"replace": {
    "Foundrysoft": "Foundry-soft",
    "UPI": "U P I",
    "Sr.": "Senior",
},

We maintain one replace map per market, merged into the session config at connect time, because the English voice needs different fixes than the Hindi one. An abbreviation that reads fine in one language becomes soup in another. These maps are the least glamorous artifact in the whole deployment, and they're also what clients notice first. Nothing torpedoes trust faster than a robot mispronouncing the company that deployed it.

Choosing from 26 voices, or bringing your own

With 26 voices in the roster, selection is now a real design decision. Two lessons from our listening sessions. Test voices with your actual callers' languages rather than English demos, because naturalness varies by language, and the July upgrades moved several voices from acceptable to genuinely good in non-English speech. And favor consistency over per-language optimization: one voice across all languages on a single line sounds like one agent who happens to be multilingual, while switching voices per language sounds like being transferred, which puts the trench coat right back on.

For brands that want an owned sound, the Custom Voices API returns a voice_id that drops into the same voice parameter, across both the realtime API and TTS. One client uses this to match their existing IVR brand voice, a continuity their callers never consciously notice. That's the point.

What multilingual support actually cost

The client's old setup served English plus "please hold for a Hindi specialist." The rebuilt line served three languages from day one at the same $0.05 per minute of agent audio. The metric that moved was abandonment: Hindi-preferring callers who used to hang up during the English menu now just talk. Adding a language went from "hire for it" to "add a language-policy paragraph, extend the keyterms list, run QA calls with native speakers."

That last clause deserves the closing emphasis. The API makes multilingual voice support cheap. It doesn't make it automatic. Native-speaker QA caught register issues, a mistranslated refund term, and the ticket-number incident, none of which an English-speaking engineering team would have shipped knowingly and all of which we'd have shipped blind. Budget for the QA calls. They're still about 95% cheaper than running three deployments in a trench coat.