Extracting Structured JSON from Video Feeds with Gemini Omni Flash

Extracting data from documents is a solved problem. Extracting data from video feeds is a nightmare.

Historically, if you wanted to analyze a factory assembly line, you had to run a heavy script to slice the video into individual frames, run an Object Detection model on every single frame, and then write custom heuristics to stitch the chaotic output back together.

Gemini Omni Flash handles video natively. You don't slice frames; you just hand the model the file. This step-by-step tutorial demonstrates how to watch a recording of a factory assembly line and output a strictly typed, structured JSON log of what happened.

Step 1: Install the Stack

We are going to use TypeScript, the Vercel AI SDK, and Zod. The Vercel AI SDK provides a unified interface for calling models, and Zod allows us to define the exact shape of the JSON data we want back.

npm install ai @ai-sdk/google zod

Step 2: Define the Strict Zod Schema

If you ask an LLM to "analyze this video," it will write you a paragraph of prose. That is useless if you want to store the results in a database. You have to force the model to adhere to a strict schema.

Notice that we include a reasoning field. It is critical to force the model to explain its thought process before it outputs the final classification. This prevents hallucinations.

import { z } from 'zod';

export const DefectLogSchema = z.object({
  analysisSummary: z.string()
    .describe("A brief summary of what happened in the video clip."),
  defectsFound: z.number()
    .describe("Total number of defective items identified."),
  logs: z.array(z.object({
    timestampSeconds: z.number()
      .describe("The exact second the defect appeared on screen."),
    reasoning: z.string()
      .describe("Why you classified this as a defect based on visual evidence."),
    defectType: z.enum(['scratch', 'misaligned', 'missing_part', 'other']),
  })),
});

Step 3: Load the Video File

Next, we need to load the video file into memory. We will read a local .mp4 file into a Node.js Buffer so we can pass it directly into the SDK's payload.

import fs from 'fs/promises';

async function loadVideoBuffer(filepath: string): Promise<Buffer> {
  console.log(`Reading video file from ${filepath}...`);
  return await fs.readFile(filepath);
}

Step 4: Run the Structured Generation

Now we pass the video buffer and the Zod schema directly to the generateObject function in the AI SDK. The Google provider automatically handles chunking and uploading the multimodal payload to Gemini's servers.

import { generateObject } from 'ai';
import { google } from '@ai-sdk/google';

async function extractVideoData(videoBuffer: Buffer) {
  console.log("Analyzing with Gemini Omni Flash...");
  
  const result = await generateObject({
    model: google('gemini-omni-flash'),
    schema: DefectLogSchema, // The Zod schema we defined in Step 2
    messages: [
      {
        role: 'user',
        content: [
          { 
            type: 'text', 
            text: `You are an expert QA inspector monitoring a factory conveyor belt. 
                   Analyze this video feed. Log every time a defective item is pulled from the line.
                   Pay close attention to the visual condition of the metal parts.` 
          },
          {
            type: 'file',
            data: videoBuffer,
            mimeType: 'video/mp4',
          },
        ],
      },
    ],
  });

  return result.object;
}

Step 5: Execute and Parse Results

Finally, we tie it all together. The SDK automatically parses the returned JSON string and validates it against our Zod schema.

async function main() {
  try {
    const buffer = await loadVideoBuffer('./factory-feed.mp4');
    const data = await extractVideoData(buffer);
    
    console.log("Analysis Complete. Strongly-typed JSON output:");
    console.log(JSON.stringify(data, null, 2));
    
    // You can now confidently insert `data.logs` into PostgreSQL
  } catch (error) {
    console.error("Failed to analyze video:", error);
  }
}

main();

Why Omni Flash Changes the Economics

When I ran this against a two-minute video, Omni Flash processed it in about four seconds. The output matched our exact Zod specification, catching 3 defects and accurately identifying them as scratch or misaligned.

In the past, solving this specific problem meant hiring a Machine Learning Operations (MLOps) team to train a custom YOLO model and deploy it to expensive GPU infrastructure. Omni Flash changes the economics of video analysis. You no longer need custom models for simple counting and extraction tasks. You pass the buffer, define the schema, and let the model do the work.