---
title: "Building a Resilient Web Scraping Agent with Eve.dev and Puppeteer"
description: "Learn how to build a durable AI web scraping agent that extracts structured data, survives browser crashes, and handles rate limiting automatically using Eve.dev."
image: "https://foundrysoft.co/api/og?type=article&title=Building+a+Resilient+Web+Scraping+Agent+with+Eve.dev+and+Puppeteer&cat=Tutorial+%2F%2F+Data&rt=16+min+read&au=Varun+Raj+Manoharan&dt=2026-07-15"
url: "https://foundrysoft.co/blog/eve-dev-resilient-web-scraping"
---

Tutorial // Data 2026-07-15 16 min read

# Building a Resilient Web Scraping Agent with Eve.dev and Puppeteer

Learn how to build a durable AI web scraping agent that extracts structured data, survives browser crashes, and handles rate limiting automatically using Eve.dev.

![Varun Raj Manoharan](https://foundrysoft.co/images/about/founder.webp)

Varun Raj Manoharan

Eve.dev Web Scraping Puppeteer Data Extraction AI Agents

## Summary

**TL;DR:** Scraping the web with AI often fails due to timeouts and browser crashes. This tutorial explains how to use Eve.dev's durable execution to build a resilient data extraction agent that never loses state.

Data extraction is one of the most common use cases for Large Language Models. However, when you combine LLMs with headless browsers (like Puppeteer or Playwright) to scrape complex websites, things break. Browsers crash. Websites enforce rate limits. Long LLM generations cause API timeouts.

If you use a standard stateless script, a crash at page 99 of 100 means you lose everything.

This is where **Eve.dev** changes the game. By leveraging its durable execution model, we can build a Web Scraping Agent that saves its progress automatically. If the process dies, Eve.dev restarts it exactly where it left off.

### Designing the Scraping Agent

In our Eve.dev workspace, we'll create a `scraper` agent. Let's start with the filesystem structure:

TEXT

Copy

```text
/eve
  /agents
    /scraper
      instructions.md
      /tools
        navigate_to_page.ts
        extract_data.ts
```

### Step 1: The Core Instructions

In `/eve/agents/scraper/instructions.md`, we define the agent's behavior:

MARKDOWN

Copy

```markdown
# AI Web Scraper

You are an expert data extraction agent. Your goal is to navigate e-commerce websites and extract product details (name, price, stock status).

## Workflow Rules
1. Use `navigate_to_page` to load the URL.
2. If the page is rate-limited or blocks you, do not panic. The system will handle the pause.
3. Use `extract_data` to parse the HTML and return structured JSON.
4. Process one product category at a time to maintain stability.
```

### Step 2: Creating the Navigation Tool

We need a tool to control our headless browser. Here, we create `/eve/agents/scraper/tools/navigate_to_page.ts`. We will use Puppeteer to load the page and extract the raw text or HTML.

TypeScript

Copy

```typescript
import { tool } from '@eve/core';
import { z } from 'zod';
import puppeteer from 'puppeteer';

export default tool({
  name: 'navigate_to_page',
  description: 'Navigates to a URL and returns the page text content.',
  schema: z.object({
    url: z.string().url()
  }),
  execute: async ({ url }) => {
    console.log(`[Scraper] Navigating to ${url}...`);

    const browser = await puppeteer.launch();
    const page = await browser.newPage();

    try {
      await page.goto(url, { waitUntil: 'networkidle2' });
      // Extract visible text for the LLM to process
      const content = await page.evaluate(() => document.body.innerText);
      return { success: true, content: content.substring(0, 10000) }; // Limit context
    } catch (error) {
      return { success: false, error: error.message };
    } finally {
      await browser.close();
    }
  }
});
```

### Step 3: Structured Data Extraction

Once the agent has the page text, it needs to structure it. This is where LLMs excel. We define an `extract_data` tool that forces the agent to output strict JSON.

TypeScript

Copy

```typescript
import { tool } from '@eve/core';
import { z } from 'zod';

export default tool({
  name: 'extract_data',
  description: 'Save extracted product information into the database.',
  schema: z.object({
    products: z.array(z.object({
      name: z.string(),
      price: z.number(),
      inStock: z.boolean()
    }))
  }),
  execute: async ({ products }) => {
    console.log(`[Scraper] Saving ${products.length} products to database.`);
    // Insert into Postgres/MongoDB here
    return { success: true, savedCount: products.length };
  }
});
```

### The Magic of Eve.dev Durability

Here is why this architecture is superior to a standard Python or Node.js script.

Imagine the agent scrapes 5 pages successfully. On the 6th page, the Node process runs out of memory and crashes.

In a traditional setup, you have to write complex database logic to track which URLs were scraped and which weren't, then write a recovery script.

With Eve.dev, the agent's memory (chat history and state) is continuously checkpointed. When you restart your server and call the agent with the same ID, Eve.dev looks at the state and says: "Ah, we were on step 6. The last tool call was `navigate_to_page`. Let's resume from there."

TypeScript

Copy

```typescript
import { Eve } from '@eve/core';

const eve = new Eve({ workspace: './eve' });

// Running this again after a crash will resume exactly where it failed!
await eve.getAgent('scraper').then(agent =>
  agent.run({
    id: "scrape_job_104", // Workflow ID enables recovery
    input: "Scrape the laptops category on shop.example.com"
  })
);
```

### Conclusion

Web scraping is inherently unstable. By delegating the orchestration, state management, and crash recovery to Eve.dev, you can focus purely on the extraction logic. You get the reasoning capabilities of LLMs combined with the unkillable durability of modern workflow engines.

### FAQ

**Why not just use LangChain for web scraping?** While LangChain can extract data, it executes in memory. If the script crashes during a long scraping run, all progress is lost. Eve.dev persists the state to a database automatically.

**How does Eve.dev handle API rate limits?** Eve.dev allows tools to return a `suspend` command. If you hit an API limit, your tool can suspend the agent for 15 minutes, freeing up server resources, and Eve.dev will wake it up automatically.

#### Related reading

[Why Trusting AI Generated Code Is the Wrong Goal

Trusting AI generated code was never the right goal, and the 4 percent of developers who say they fully trust it prove nothing is broken: the fix is an AI code review process that makes verification cheap instead of asking how much to trust the output.

AI Code Review Developer Trust Code Quality

](https://foundrysoft.co/blog/developers-dont-trust-ai-generated-code)[Five AI Agents, One Bug: When Missing Data Looks Like a Clean Result

We built and shipped five open-source vertical AI agents. Every single one had the same class of defect: absent or unreadable input rendered as a confident, clean answer. Here is what that bug looks like, why tests miss it, and what actually catches it.

AI Agents Testing Open Source

](https://foundrysoft.co/blog/five-ai-agents-one-bug-missing-data-clean-result)[Best Open Weight LLMs for Agents in 2026

A practical look at the best open weight LLMs for agents in 2026, organized by which constraint, cost, latency, or data residency, should actually decide the pick.

Open Weight LLMs AI Agents LLM Comparison

](https://foundrysoft.co/blog/best-open-weight-llms-agents-2026)

#### Next Article

[

Create an AI Social Media Manager Agent with Eve.dev

](https://foundrysoft.co/blog/eve-dev-social-media-manager-agent)

Available for new projects

## Let's build something great.

Have a project in mind? We are an elite software and AI development studio ready to bring your ideas to production. Let's talk about your roadmap.

Start a Project [See our work](https://foundrysoft.co/work)

```json
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "FoundrySoft",
  "url": "https://foundrysoft.co",
  "logo": "https://foundrysoft.co/logo.svg",
  "description": "FoundrySoft builds production-grade software and AI systems for US companies, from an India-based team of senior engineers.",
  "sameAs": [
    "https://github.com/foundrysofthq",
    "https://www.linkedin.com/company/foundrysoft",
    "https://www.instagram.com/foundrysoft/"
  ]
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "WebSite",
  "name": "FoundrySoft",
  "url": "https://foundrysoft.co"
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "TechArticle",
  "headline": "Building a Resilient Web Scraping Agent with Eve.dev and Puppeteer",
  "description": "Learn how to build a durable AI web scraping agent that extracts structured data, survives browser crashes, and handles rate limiting automatically using Eve.dev.",
  "url": "https://foundrysoft.co/blog/eve-dev-resilient-web-scraping",
  "mainEntityOfPage": "https://foundrysoft.co/blog/eve-dev-resilient-web-scraping",
  "image": [
    "https://foundrysoft.co/images/blog/multi-agent-swarm.jpg"
  ],
  "datePublished": "2026-07-15",
  "dateModified": "2026-07-15",
  "keywords": "Eve.dev, Web Scraping, Puppeteer, Data Extraction, AI Agents",
  "author": {
    "@type": "Person",
    "name": "Varun Raj Manoharan"
  },
  "publisher": {
    "@type": "Organization",
    "name": "FoundrySoft",
    "logo": {
      "@type": "ImageObject",
      "url": "https://foundrysoft.co/logo.svg"
    }
  }
}
```

```json
{
  "@context": "https://schema.org",
  "@type": "BreadcrumbList",
  "itemListElement": [
    {
      "@type": "ListItem",
      "position": 1,
      "name": "Home",
      "item": "https://foundrysoft.co/"
    },
    {
      "@type": "ListItem",
      "position": 2,
      "name": "Blog",
      "item": "https://foundrysoft.co/blog"
    },
    {
      "@type": "ListItem",
      "position": 3,
      "name": "Building a Resilient Web Scraping Agent with Eve.dev and Puppeteer",
      "item": "https://foundrysoft.co/blog/eve-dev-resilient-web-scraping"
    }
  ]
}
```