Processing Video Files with Vercel AI SDK v7 Multimodal Support

We spent the last two years feeding text strings into language models. Now the models can actually see what we see. Vercel AI SDK v7 pushes hard into multimodal territory, letting you pass images, audio, and even video files directly into your prompts.

I wanted to see how well it handles moving pictures, so I built a small script to ingest a short video clip and spit out a summary.

Dropping in the Video

You do not have to write custom chunking logic or extract frames manually anymore. The SDK abstracts the heavy lifting. You just pass the file as a part of the messages array.

import { generateText } from 'ai';
import { google } from '@ai-sdk/google';
import fs from 'fs/promises';

async function analyzeVideo() {
  const videoBuffer = await fs.readFile('./meeting-recording.mp4');

  const { text } = await generateText({
    model: google('gemini-3-pro'),
    messages: [
      {
        role: 'user',
        content: [
          { type: 'text', text: 'Watch this video and tell me what the three main action items were.' },
          {
            type: 'file',
            data: videoBuffer,
            mimeType: 'video/mp4',
          },
        ],
      },
    ],
  });

  console.log(text);
}

analyzeVideo();

The Results

I fed it a three-minute recording of a standup meeting. It chewed on it for about ten seconds and correctly pulled out the action items, even noting that one developer pointed at a specific line of code on the shared screen.

There is a catch, though. The models still struggle with temporal reasoning—understanding when something happened in a ten-minute video is harder than just recognizing what happened. You also have to watch your token limits. A long high-definition video will eat through your context window faster than you can blink.

If you are just summarizing short clips or pulling data from screen recordings, the v7 API makes it trivial. Just keep an eye on the billing dashboard.