Building a Resilient Web Scraping Agent with Eve.dev and Puppeteer

Data extraction is one of the most common use cases for Large Language Models. However, when you combine LLMs with headless browsers (like Puppeteer or Playwright) to scrape complex websites, things break. Browsers crash. Websites enforce rate limits. Long LLM generations cause API timeouts.

If you use a standard stateless script, a crash at page 99 of 100 means you lose everything.

This is where Eve.dev changes the game. By leveraging its durable execution model, we can build a Web Scraping Agent that saves its progress automatically. If the process dies, Eve.dev restarts it exactly where it left off.

Designing the Scraping Agent

In our Eve.dev workspace, we'll create a scraper agent. Let's start with the filesystem structure:

/eve
  /agents
    /scraper
      instructions.md
      /tools
        navigate_to_page.ts
        extract_data.ts

Step 1: The Core Instructions

In /eve/agents/scraper/instructions.md, we define the agent's behavior:

# AI Web Scraper

You are an expert data extraction agent. Your goal is to navigate e-commerce websites and extract product details (name, price, stock status).

## Workflow Rules
1. Use `navigate_to_page` to load the URL.
2. If the page is rate-limited or blocks you, do not panic. The system will handle the pause.
3. Use `extract_data` to parse the HTML and return structured JSON.
4. Process one product category at a time to maintain stability.

Step 2: Creating the Navigation Tool

We need a tool to control our headless browser. Here, we create /eve/agents/scraper/tools/navigate_to_page.ts. We will use Puppeteer to load the page and extract the raw text or HTML.

import { tool } from '@eve/core';
import { z } from 'zod';
import puppeteer from 'puppeteer';

export default tool({
  name: 'navigate_to_page',
  description: 'Navigates to a URL and returns the page text content.',
  schema: z.object({
    url: z.string().url()
  }),
  execute: async ({ url }) => {
    console.log(`[Scraper] Navigating to ${url}...`);
    
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    
    try {
      await page.goto(url, { waitUntil: 'networkidle2' });
      // Extract visible text for the LLM to process
      const content = await page.evaluate(() => document.body.innerText);
      return { success: true, content: content.substring(0, 10000) }; // Limit context
    } catch (error) {
      return { success: false, error: error.message };
    } finally {
      await browser.close();
    }
  }
});

Step 3: Structured Data Extraction

Once the agent has the page text, it needs to structure it. This is where LLMs excel. We define an extract_data tool that forces the agent to output strict JSON.

import { tool } from '@eve/core';
import { z } from 'zod';

export default tool({
  name: 'extract_data',
  description: 'Save extracted product information into the database.',
  schema: z.object({
    products: z.array(z.object({
      name: z.string(),
      price: z.number(),
      inStock: z.boolean()
    }))
  }),
  execute: async ({ products }) => {
    console.log(`[Scraper] Saving ${products.length} products to database.`);
    // Insert into Postgres/MongoDB here
    return { success: true, savedCount: products.length };
  }
});

The Magic of Eve.dev Durability

Here is why this architecture is superior to a standard Python or Node.js script.

Imagine the agent scrapes 5 pages successfully. On the 6th page, the Node process runs out of memory and crashes.

In a traditional setup, you have to write complex database logic to track which URLs were scraped and which weren't, then write a recovery script.

With Eve.dev, the agent's memory (chat history and state) is continuously checkpointed. When you restart your server and call the agent with the same ID, Eve.dev looks at the state and says: "Ah, we were on step 6. The last tool call was navigate_to_page. Let's resume from there."

import { Eve } from '@eve/core';

const eve = new Eve({ workspace: './eve' });

// Running this again after a crash will resume exactly where it failed!
await eve.getAgent('scraper').then(agent => 
  agent.run({
    id: "scrape_job_104", // Workflow ID enables recovery
    input: "Scrape the laptops category on shop.example.com"
  })
);

Conclusion

Web scraping is inherently unstable. By delegating the orchestration, state management, and crash recovery to Eve.dev, you can focus purely on the extraction logic. You get the reasoning capabilities of LLMs combined with the unkillable durability of modern workflow engines.

FAQ

Why not just use LangChain for web scraping? While LangChain can extract data, it executes in memory. If the script crashes during a long scraping run, all progress is lost. Eve.dev persists the state to a database automatically.

How does Eve.dev handle API rate limits? Eve.dev allows tools to return a suspend command. If you hit an API limit, your tool can suspend the agent for 15 minutes, freeing up server resources, and Eve.dev will wake it up automatically.