---
title: "Building a Sub-Second Voice Agent with Gemini Omni Flash"
description: "Voice agents die if they have latency. I used Gemini Omni Flash to build a real-time conversational agent that responds in under 500 milliseconds."
image: "https://foundrysoft.co/api/og?type=article&title=Building+a+Sub-Second+Voice+Agent+with+Gemini+Omni+Flash&cat=Tutorial+%2F%2F+AI+Agents&rt=15+min+read&au=Varun+Raj+Manoharan&dt=2026-07-10"
url: "https://foundrysoft.co/blog/gemini-omni-flash-realtime-voice-agent"
---

Tutorial // AI Agents 2026-07-10 15 min read

# Building a Sub-Second Voice Agent with Gemini Omni Flash

Voice agents die if they have latency. I used Gemini Omni Flash to build a real-time conversational agent that responds in under 500 milliseconds.

![Varun Raj Manoharan](https://foundrysoft.co/images/about/founder.webp)

Varun Raj Manoharan

Gemini Omni Flash Voice AI Agents Python

Building a text-based chatbot is forgiving. If a model takes three seconds to type out a response, users don't mind. They are used to watching the typing indicator. Voice is entirely different. If you pause for three seconds during a phone call, the human on the other end assumes the call dropped and starts saying "Hello?"

To build a voice agent, you need raw speed. I tested the new Gemini Omni Flash model specifically because it was designed for ultra-low-latency multimodal tasks. This tutorial covers how to build a voice agent step-by-step that bypasses traditional transcription layers and responds in under 500 milliseconds.

### The Latency Problem with Traditional Pipelines

If you build a voice agent using the standard stack, your pipeline looks like this:

1.  **Speech-to-Text (STT):** User speaks -> Stream audio to Whisper/Deepgram -> Wait for text transcription (200ms - 500ms).
2.  **LLM Inference:** Send text to GPT-4/Claude -> Wait for the first token of the response (500ms - 1000ms).
3.  **Text-to-Speech (TTS):** Stream the LLM text to ElevenLabs -> Wait for the first audio byte (200ms - 400ms).

Best case scenario, you are looking at 1 to 2 seconds of silence before the bot replies. Gemini Omni Flash changes this architecture. It natively understands audio inputs. You don't need the STT layer. You just stream the raw microphone bytes directly into the model's context window.

Here is how to build it, step-by-step.

### Step 1: Install Dependencies

To follow along, you need Python 3.10+, a Google Gemini API Key, and a microphone attached to your machine. First, install the required libraries for interacting with the Google API and capturing local audio.

Shell

Copy

```bash
pip install google-genai pyaudio asyncio
```

### Step 2: Configure the Microphone

We need to capture raw audio from the microphone. We will use `pyaudio` to handle the hardware interface. We configure it to capture 16kHz, 16-bit PCM audio, which is the standard format expected by most voice models.

Python

Copy

```python
import pyaudio

# Audio configuration constants
FORMAT = pyaudio.paInt16
CHANNELS = 1
RATE = 16000
CHUNK = 1024

def setup_microphone():
    """Initializes the PyAudio stream for microphone input."""
    audio = pyaudio.PyAudio()
    stream = audio.open(
        format=FORMAT,
        channels=CHANNELS,
        rate=RATE,
        input=True,
        frames_per_buffer=CHUNK
    )
    return stream
```

### Step 3: Capture the Audio Stream

In a production application, you would implement Voice Activity Detection (VAD) to only send audio when the user is speaking. For this tutorial, we will capture a fixed 3-second block of audio.

Python

Copy

```python
def capture_audio(stream, seconds=3):
    """Reads raw audio chunks from the microphone for a set duration."""
    print(f"Listening for {seconds} seconds...")
    frames = []

    # Calculate how many chunks we need to read to hit our target duration
    total_chunks = int(RATE / CHUNK * seconds)

    for _ in range(0, total_chunks):
        data = stream.read(CHUNK)
        frames.append(data)

    # Combine the chunks into a single byte string
    return b''.join(frames)
```

### Step 4: Stream to Gemini Omni Flash

Here is where the magic happens. We take the raw bytes from the microphone and hand them directly to the `gemini-omni-flash` model. We bypass transcription entirely.

Python

Copy

```python
import google.genai as genai
import asyncio

# Initialize the ultra-fast Omni Flash model client
client = genai.Client(api_key="YOUR_GEMINI_API_KEY")
MODEL_ID = 'gemini-omni-flash'

async def ask_gemini(audio_data):
    """Sends the raw audio payload to Gemini and awaits a text response."""
    print("Sending raw audio to Omni Flash...")

    response = await client.models.generate_content_async(
        model=MODEL_ID,
        contents=[
            "You are a helpful voice assistant. Keep your response under two sentences. Reply naturally to the audio provided.",
            {"mime_type": "audio/wav", "data": audio_data}
        ],
        config={"temperature": 0.3}
    )

    print(f"Agent Reply: {response.text}")
    return response.text
```

### Step 5: Put It Together and Connect TTS

Now we wire the pieces together. When you run this, notice how quickly the text comes back. To complete the voice loop, you would stream this resulting text back into a fast TTS engine.

Python

Copy

```python
async def main():
    # 1. Setup Mic
    stream = setup_microphone()

    # 2. Record
    audio_data = capture_audio(stream)

    # 3. Get LLM response instantly
    text_reply = await ask_gemini(audio_data)

    # 4. Stream to Text-to-Speech (Pseudo-code integration)
    # tts_socket = await connect_elevenlabs()
    # await tts_socket.send(text_reply)
    # ... play response through speakers

if __name__ == "__main__":
    asyncio.run(main())
```

### Why This Architecture Wins

When you run this code, the speed is jarring. Because Omni Flash does not have to wait for an external STT model to process a temporary `.wav` file, the Time-to-First-Token (TTFT) drops drastically. The model starts generating the text response almost exactly when the audio stream ends.

By avoiding transcription, you also preserve the acoustic features of the user's voice. The model can hear tone, emotion, and background noise. If the user sounds angry, the model knows it immediately. If you are building an AI receptionist or a real-time language tutor, drop the transcription layer. Let the model listen.

#### Related reading

[Why Trusting AI Generated Code Is the Wrong Goal

Trusting AI generated code was never the right goal, and the 4 percent of developers who say they fully trust it prove nothing is broken: the fix is an AI code review process that makes verification cheap instead of asking how much to trust the output.

AI Code Review Developer Trust Code Quality

](https://foundrysoft.co/blog/developers-dont-trust-ai-generated-code)[Five AI Agents, One Bug: When Missing Data Looks Like a Clean Result

We built and shipped five open-source vertical AI agents. Every single one had the same class of defect: absent or unreadable input rendered as a confident, clean answer. Here is what that bug looks like, why tests miss it, and what actually catches it.

AI Agents Testing Open Source

](https://foundrysoft.co/blog/five-ai-agents-one-bug-missing-data-clean-result)[Best Open Weight LLMs for Agents in 2026

A practical look at the best open weight LLMs for agents in 2026, organized by which constraint, cost, latency, or data residency, should actually decide the pick.

Open Weight LLMs AI Agents LLM Comparison

](https://foundrysoft.co/blog/best-open-weight-llms-agents-2026)

#### Next Article

[

Designing a Multi-Agent Swarm with Vercel AI SDK for Complex Workflows

](https://foundrysoft.co/blog/vercel-ai-sdk-multi-agent-swarm)

Available for new projects

## Let's build something great.

Have a project in mind? We are an elite software and AI development studio ready to bring your ideas to production. Let's talk about your roadmap.

Start a Project [See our work](https://foundrysoft.co/work)

```json
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "FoundrySoft",
  "url": "https://foundrysoft.co",
  "logo": "https://foundrysoft.co/logo.svg",
  "description": "FoundrySoft builds production-grade software and AI systems for US companies, from an India-based team of senior engineers.",
  "sameAs": [
    "https://github.com/foundrysofthq",
    "https://www.linkedin.com/company/foundrysoft",
    "https://www.instagram.com/foundrysoft/"
  ]
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "WebSite",
  "name": "FoundrySoft",
  "url": "https://foundrysoft.co"
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "TechArticle",
  "headline": "Building a Sub-Second Voice Agent with Gemini Omni Flash",
  "description": "Voice agents die if they have latency. I used Gemini Omni Flash to build a real-time conversational agent that responds in under 500 milliseconds.",
  "url": "https://foundrysoft.co/blog/gemini-omni-flash-realtime-voice-agent",
  "mainEntityOfPage": "https://foundrysoft.co/blog/gemini-omni-flash-realtime-voice-agent",
  "image": [
    "https://foundrysoft.co/images/blog/gemini-omni-flash-voice.jpg"
  ],
  "datePublished": "2026-07-10",
  "dateModified": "2026-07-10",
  "keywords": "Gemini, Omni Flash, Voice, AI Agents, Python",
  "author": {
    "@type": "Person",
    "name": "Varun Raj Manoharan"
  },
  "publisher": {
    "@type": "Organization",
    "name": "FoundrySoft",
    "logo": {
      "@type": "ImageObject",
      "url": "https://foundrysoft.co/logo.svg"
    }
  }
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "BreadcrumbList",
  "itemListElement": [
    {
      "@type": "ListItem",
      "position": 1,
      "name": "Home",
      "item": "https://foundrysoft.co/"
    },
    {
      "@type": "ListItem",
      "position": 2,
      "name": "Blog",
      "item": "https://foundrysoft.co/blog"
    },
    {
      "@type": "ListItem",
      "position": 3,
      "name": "Building a Sub-Second Voice Agent with Gemini Omni Flash",
      "item": "https://foundrysoft.co/blog/gemini-omni-flash-realtime-voice-agent"
    }
  ]
}
```