--- title: "Extracting Structured JSON from Video Feeds with Gemini Omni Flash" description: "Processing video usually requires massive compute overhead. I used Gemini Omni Flash to extract structured JSON data directly from raw video feeds in real time." image: "https://foundrysoft.co/api/og?type=article&title=Extracting+Structured+JSON+from+Video+Feeds+with+Gemini+Omni+Flash&cat=Tutorial+%2F%2F+Data+Extraction&rt=14+min+read&au=Varun+Raj+Manoharan&dt=2026-07-09" url: "https://foundrysoft.co/blog/gemini-omni-flash-vision-data-extraction" --- Tutorial // Data Extraction 2026-07-09 14 min read # Extracting Structured JSON from Video Feeds with Gemini Omni Flash Processing video usually requires massive compute overhead. I used Gemini Omni Flash to extract structured JSON data directly from raw video feeds in real time. ![Varun Raj Manoharan](https://foundrysoft.co/images/about/founder.webp) Varun Raj Manoharan Gemini Omni Flash Video JSON Multimodal Extracting data from documents is a solved problem. Extracting data from video feeds is a nightmare. Historically, if you wanted to analyze a factory assembly line, you had to run a heavy script to slice the video into individual frames, run an Object Detection model on every single frame, and then write custom heuristics to stitch the chaotic output back together. Gemini Omni Flash handles video natively. You don't slice frames; you just hand the model the file. This step-by-step tutorial demonstrates how to watch a recording of a factory assembly line and output a strictly typed, structured JSON log of what happened. ### Step 1: Install the Stack We are going to use TypeScript, the Vercel AI SDK, and Zod. The Vercel AI SDK provides a unified interface for calling models, and Zod allows us to define the exact shape of the JSON data we want back. Shell Copy ```bash npm install ai @ai-sdk/google zod ``` ### Step 2: Define the Strict Zod Schema If you ask an LLM to "analyze this video," it will write you a paragraph of prose. That is useless if you want to store the results in a database. You have to force the model to adhere to a strict schema. Notice that we include a `reasoning` field. It is critical to force the model to explain its thought process _before_ it outputs the final classification. This prevents hallucinations. TypeScript Copy ```typescript import { z } from 'zod'; export const DefectLogSchema = z.object({ analysisSummary: z.string() .describe("A brief summary of what happened in the video clip."), defectsFound: z.number() .describe("Total number of defective items identified."), logs: z.array(z.object({ timestampSeconds: z.number() .describe("The exact second the defect appeared on screen."), reasoning: z.string() .describe("Why you classified this as a defect based on visual evidence."), defectType: z.enum(['scratch', 'misaligned', 'missing_part', 'other']), })), }); ``` ### Step 3: Load the Video File Next, we need to load the video file into memory. We will read a local `.mp4` file into a Node.js Buffer so we can pass it directly into the SDK's payload. TypeScript Copy ```typescript import fs from 'fs/promises'; async function loadVideoBuffer(filepath: string): Promise { console.log(`Reading video file from ${filepath}...`); return await fs.readFile(filepath); } ``` ### Step 4: Run the Structured Generation Now we pass the video buffer and the Zod schema directly to the `generateObject` function in the AI SDK. The Google provider automatically handles chunking and uploading the multimodal payload to Gemini's servers. TypeScript Copy ```typescript import { generateObject } from 'ai'; import { google } from '@ai-sdk/google'; async function extractVideoData(videoBuffer: Buffer) { console.log("Analyzing with Gemini Omni Flash..."); const result = await generateObject({ model: google('gemini-omni-flash'), schema: DefectLogSchema, // The Zod schema we defined in Step 2 messages: [ { role: 'user', content: [ { type: 'text', text: `You are an expert QA inspector monitoring a factory conveyor belt. Analyze this video feed. Log every time a defective item is pulled from the line. Pay close attention to the visual condition of the metal parts.` }, { type: 'file', data: videoBuffer, mimeType: 'video/mp4', }, ], }, ], }); return result.object; } ``` ### Step 5: Execute and Parse Results Finally, we tie it all together. The SDK automatically parses the returned JSON string and validates it against our Zod schema. TypeScript Copy ```typescript async function main() { try { const buffer = await loadVideoBuffer('./factory-feed.mp4'); const data = await extractVideoData(buffer); console.log("Analysis Complete. Strongly-typed JSON output:"); console.log(JSON.stringify(data, null, 2)); // You can now confidently insert `data.logs` into PostgreSQL } catch (error) { console.error("Failed to analyze video:", error); } } main(); ``` ### Why Omni Flash Changes the Economics When I ran this against a two-minute video, Omni Flash processed it in about four seconds. The output matched our exact Zod specification, catching 3 defects and accurately identifying them as `scratch` or `misaligned`. In the past, solving this specific problem meant hiring a Machine Learning Operations (MLOps) team to train a custom YOLO model and deploy it to expensive GPU infrastructure. Omni Flash changes the economics of video analysis. You no longer need custom models for simple counting and extraction tasks. You pass the buffer, define the schema, and let the model do the work. #### Related reading [Building a Sub-Second Voice Agent with Gemini Omni Flash Voice agents die if they have latency. I used Gemini Omni Flash to build a real-time conversational agent that responds in under 500 milliseconds. Gemini Omni Flash Voice ](https://foundrysoft.co/blog/gemini-omni-flash-realtime-voice-agent)[Deploying Gemini Omni Flash at the Edge for Zero-Latency RAG RAG pipelines are slow because of server round-trips. I moved a Gemini Omni Flash workflow entirely to Edge Functions to kill the cold start. Gemini Omni Flash Edge ](https://foundrysoft.co/blog/gemini-omni-flash-edge-deployment)[Processing Video Files with Vercel AI SDK v7 Multimodal Support Language models can finally see more than just text. I tested the new multimodal capabilities in Vercel AI SDK v7 to analyze raw video files. Vercel AI SDK v7 ](https://foundrysoft.co/blog/vercel-ai-sdk-v7-multimodal-inputs) #### Next Article [ Building a Persistent Data Analyst Agent with Vercel AI SDK and E2B ](https://foundrysoft.co/blog/vercel-ai-sdk-e2b-data-analyst) Available for new projects ## Let's build something great. Have a project in mind? We are an elite software and AI development studio ready to bring your ideas to production. Let's talk about your roadmap. Start a Project [See our work](https://foundrysoft.co/work) ```json { "@context": "https://schema.org", "@type": "Organization", "name": "FoundrySoft", "url": "https://foundrysoft.co", "logo": "https://foundrysoft.co/logo.svg", "description": "FoundrySoft builds production-grade software and AI systems for US companies, from an India-based team of senior engineers.", "sameAs": [ "https://github.com/foundrysofthq", "https://www.linkedin.com/company/foundrysoft", "https://www.instagram.com/foundrysoft/" ] } ``` ```json { "@context": "https://schema.org", "@type": "WebSite", "name": "FoundrySoft", "url": "https://foundrysoft.co" } ``` ```json { "@context": "https://schema.org", "@type": "TechArticle", "headline": "Extracting Structured JSON from Video Feeds with Gemini Omni Flash", "description": "Processing video usually requires massive compute overhead. I used Gemini Omni Flash to extract structured JSON data directly from raw video feeds in real time.", "url": "https://foundrysoft.co/blog/gemini-omni-flash-vision-data-extraction", "mainEntityOfPage": "https://foundrysoft.co/blog/gemini-omni-flash-vision-data-extraction", "image": [ "https://foundrysoft.co/images/blog/gemini-omni-flash-video.jpg" ], "datePublished": "2026-07-09", "dateModified": "2026-07-09", "keywords": "Gemini, Omni Flash, Video, JSON, Multimodal", "author": { "@type": "Person", "name": "Varun Raj Manoharan" }, "publisher": { "@type": "Organization", "name": "FoundrySoft", "logo": { "@type": "ImageObject", "url": "https://foundrysoft.co/logo.svg" } } } ``` ```json { "@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://foundrysoft.co/" }, { "@type": "ListItem", "position": 2, "name": "Blog", "item": "https://foundrysoft.co/blog" }, { "@type": "ListItem", "position": 3, "name": "Extracting Structured JSON from Video Feeds with Gemini Omni Flash", "item": "https://foundrysoft.co/blog/gemini-omni-flash-vision-data-extraction" } ] } ```