Building a Browser-Controlling "Computer Use" Agent with Vercel AI SDK

The most highly requested feature for any AI application is simple: "Can the AI just do it for me?"

Users don't want an AI to write them an email explaining how to book a flight on Expedia; they want the AI to go to Expedia, fill out the forms, and book the flight. This is known as "Computer Use" or Browser Automation. It is the holy grail of agentic workflows.

In this deep-dive tutorial, we are going to build a "Computer Use" agent. We will combine the tool-calling orchestration of the Vercel AI SDK with the headless browser automation of Playwright. By the end, our agent will be able to navigate to any URL, read the screen, find buttons, and click them autonomously.

The Challenge of DOM Understanding

The hardest part of building a browser agent is not the automation; it is translating the visual web page into something the LLM can understand.

A modern web page DOM is massive. If you pass the raw HTML of Amazon.com into an LLM's context window, you will exhaust its token limit instantly, and the model will hallucinate because it cannot distinguish a visual div from an actionable <button>.

To solve this, we must simplify the DOM. We will use Playwright to inject a script that strips out styling and non-interactive elements, leaving only a clean, semantic map of actionable nodes (links, buttons, inputs) with unique ID numbers.

Step 1: Setting up the Playwright Environment

First, let's set up a Node.js project.

mkdir browser-agent
cd browser-agent
npm init -y
npm install ai @ai-sdk/openai playwright
npm install -D typescript @types/node tsx

Create an agent.ts file and initialize a Playwright browser instance.

import { chromium, Browser, Page } from 'playwright';
import { openai } from '@ai-sdk/openai';
import { generateText, tool } from 'ai';
import { z } from 'zod';

// Global state for our browser
let browser: Browser;
let page: Page;

async function initBrowser() {
  // We run in non-headless mode so you can watch the AI control the mouse!
  browser = await chromium.launch({ headless: false });
  const context = await browser.newContext();
  page = await context.newPage();
}

Step 2: The DOM Simplification Tool

We need a function that reads the current page and returns a simplified map of actionable elements. We will assign an "Element ID" to every button and link.

async function getSimplifiedDOM(page: Page) {
  // Inject a script into the browser to map actionable elements
  const domState = await page.evaluate(() => {
    let elementId = 0;
    const elementsMap: Record<number, string> = {};
    const actionableElements = document.querySelectorAll('button, a, input, select, textarea, [role="button"]');
    
    actionableElements.forEach((el) => {
      // Skip hidden elements
      const rect = el.getBoundingClientRect();
      if (rect.width === 0 || rect.height === 0) return;
      
      const id = elementId++;
      // Store a custom attribute on the DOM node so we can find it later
      el.setAttribute('data-ai-id', id.toString());
      
      // Clean up text content
      const text = (el.textContent || (el as HTMLInputElement).value || '').trim().replace(/\s+/g, ' ');
      const tag = el.tagName.toLowerCase();
      
      elementsMap[id] = `<${tag} id=${id}>${text}</${tag}>`;
    });
    
    return elementsMap;
  });

  return domState;
}

This drastically reduces token usage. Instead of 100,000 lines of HTML, the LLM receives: { 0: "<a id=0>Login</a>", 1: "<input id=1></input>", 2: "<button id=2>Submit</button>" }

Step 3: Defining the AI Tools

Now we define the tools using the Vercel AI SDK. We will give the agent three fundamental abilities: navigate, click, and type.

const browserTools = {
  navigate: tool({
    description: "Navigate the browser to a specific URL.",
    parameters: z.object({ url: z.string().url() }),
    execute: async ({ url }) => {
      console.log(`[Agent] Navigating to ${url}...`);
      await page.goto(url, { waitUntil: 'networkidle' });
      const dom = await getSimplifiedDOM(page);
      return `Navigated. Current actionable elements:\n${JSON.stringify(dom, null, 2)}`;
    }
  }),
  
  click: tool({
    description: "Click an element on the page using its numeric ID.",
    parameters: z.object({ id: z.number() }),
    execute: async ({ id }) => {
      console.log(`[Agent] Clicking element ID ${id}...`);
      
      // Tell Playwright to click the element with our custom data attribute
      await page.locator(`[data-ai-id="${id}"]`).click();
      await page.waitForLoadState('networkidle');
      
      const dom = await getSimplifiedDOM(page);
      return `Clicked. Current actionable elements:\n${JSON.stringify(dom, null, 2)}`;
    }
  }),

  type: tool({
    description: "Type text into an input element using its numeric ID.",
    parameters: z.object({ id: z.number(), text: z.string() }),
    execute: async ({ id, text }) => {
      console.log(`[Agent] Typing "${text}" into element ID ${id}...`);
      
      await page.locator(`[data-ai-id="${id}"]`).fill(text);
      return `Successfully typed "${text}".`;
    }
  })
};

Notice that every time the agent navigates or clicks, the tool returns the new simplified DOM. This is crucial. Web apps are dynamic. Clicking a button might open a modal with entirely new buttons. The LLM must receive the updated state.

Step 4: The Agent Loop

With the tools defined, we unleash the generateText function. We use maxSteps so the agent can string multiple actions together autonomously (e.g., Navigate -> Find Input -> Type -> Find Button -> Click).

async function runAgent(goal: string) {
  console.log(`Goal: ${goal}`);
  
  const result = await generateText({
    model: openai('gpt-4o'), // Use a strong reasoning model
    tools: browserTools,
    maxSteps: 15, // Allow up to 15 sequential browser actions
    messages: [
      {
        role: 'system',
        content: `You are an autonomous web browser agent. 
        Your goal is to complete the user's task by navigating URLs, clicking elements, and typing text.
        Always inspect the returned element IDs before clicking. Never guess an ID.`
      },
      { role: 'user', content: goal }
    ]
  });

  console.log(`\nFinal Result:\n${result.text}`);
}

Step 5: Execution

Let's test our agent with a complex, multi-step goal.

async function main() {
  await initBrowser();
  
  await runAgent(
    "Go to news.ycombinator.com, find the search bar at the bottom, type 'Vercel AI SDK', and click the submit/search button."
  );
  
  await browser.close();
}

main();

When you run this script using npx tsx agent.ts, a Chromium browser will pop open. You will watch as the agent autonomously types the Hacker News URL, evaluates the DOM, identifies the search input ID, types the query, identifies the search button ID, and clicks it.

Hardening the System (Production Considerations)

If you plan to deploy this to production, you must handle edge cases:

1. Multimodality: Modern models like gpt-4o have vision. Instead of just passing a text DOM, you can take a screenshot with Playwright (await page.screenshot()) and pass the base64 image to the Vercel AI SDK. This helps the model understand complex layouts (like maps or canvas elements) that the DOM parser misses.
2. Infinite Loops: If a click fails or a modal traps the agent, it might loop infinitely trying to click the same ID. Implement logic in your tool execution to track duplicate actions and explicitly return an error to the LLM (e.g., "Error: Element not visible, try something else").
3. CAPTCHAs: Agents are easily blocked by Cloudflare and CAPTCHAs. Use a managed browser service like Browserbase or Steel.dev that handles proxy rotation and CAPTCHA solving under the hood.

Computer Use agents represent a massive leap in utility. By giving the Vercel AI SDK a Playwright instance, you transform an LLM from an oracle that tells you things into a worker that does things.

Frequently Asked Questions (FAQ)

Why not pass the raw HTML page source to the LLM? Raw HTML is massive and full of irrelevant styling (CSS) and tracking scripts. Passing it to an LLM exhausts token limits quickly and causes hallucinations. You must simplify the DOM to only actionable elements.

How do AI agents solve CAPTCHAs? Standard headless Playwright instances will be blocked by Cloudflare and CAPTCHAs. To build production computer-use agents, use managed browser environments like Browserbase or Steel.dev which handle proxies and evasions.

Does Vercel AI SDK support Vision for Browser Agents? Yes. You can capture a screenshot using Playwright (page.screenshot()) and pass the base64 image along with the text prompt to multimodal models like gpt-4o or claude-3-5-sonnet via the Vercel AI SDK.