Automating Legacy Enterprise Software With Browser Agents and Computer Use

Ask any Chief Operating Officer at an established enterprise where their team spends the most manual hours, and they will rarely say modern SaaS tools like Slack or Linear. They will point you toward an on-premises ERP installed in 2004, a custom AS400 billing emulator, or a state government compliance portal that requires five separate forms to register an employee.

These systems run the actual business. They hold thirty years of supply chain records, process payroll, and clear cross-border freight shipments. But they have no REST APIs, no webhooks, and no documentation. For the past decade, the only way to automate them was brittle Robotic Process Automation (RPA) scripts that broke whenever a screen resolution changed or a pop-up window appeared.

The arrival of multimodal vision models and computer-use agent frameworks changes this calculation. Instead of waiting for a multi-year, eight-figure ERP migration that might get canceled halfway through, you can build autonomous browser agents that read the screen, interact with UI elements, and complete complex multi-step data entry tasks with human-level adaptability.

Here is the exact technical blueprint we use at FoundrySoft to build resilient, production-grade browser agents for legacy enterprise automation.

The dual-engine architecture: Accessibility tree meets vision

The biggest mistake teams make when building browser agents is relying entirely on screenshots and coordinate clicking. Sending a 1080p screenshot to Claude or GPT-5 for every single mouse movement is slow, expensive, and prone to calibration drift on nested iframe components.

We use a hybrid approach that prioritizes deterministic DOM elements and uses vision only when the underlying document model is opaque.

Incoming Work Ticket (JSON)
        │
        ▼
┌───────────────────────────────────────────────┐
│ 1. Parse Accessibility Tree & DOM Snapshot    │
│    - Extract interactive inputs, buttons, ids │
└───────────────────────┬───────────────────────┘
                        │
       Is element standard HTML?
       ├──────── Yes ───────► [Direct Playwright Locator: Click / Type]
       │
       └──────── No (Canvas / Flash / Obfuscated)
                 │
                 ▼
┌───────────────────────────────────────────────┐
│ 2. Visual Grounding with Screen Coordinate ML │
│    - Bounding box inference & click dispatch  │
└───────────────────────────────────────────────┘

Stage 1: Accessibility tree parsing with Playwright

Modern web browsers generate an accessibility tree (a11y) designed for screen readers. This tree strips away styling, CSS animations, and layout clutter, leaving an annotated representation of interactive elements: buttons, text inputs, checkboxes, and modal dialogs.

When the agent opens a legacy portal, our runner grabs the a11y tree and assigns sequential numeric labels to interactive nodes:

import { chromium, Page } from "playwright";

export async function captureInteractiveElements(page: Page) {
  const snapshot = await page.accessibility.snapshot({ interestingOnly: true });
  return snapshot;
}

If the legacy form contains standard input tags, the LLM receives a compact text representation:

[14] Textbox: "Policy Number" (Required)
[15] Dropdown: "Claim Type" (Selected: "Auto Collision")
[16] Textbox: "Estimated Damage ($)"
[17] Button: "Submit Claim"

The model emits an action like fill(14, "POL-99214"). This executes directly via Playwright selectors in twenty milliseconds, bypassing expensive vision tokens entirely.

Stage 2: Multimodal visual grounding as fallback

When the legacy application uses HTML5 Canvas, Java applets, or nested iframes with obscured DOM attributes, the accessibility tree fails. At this point, the runner captures a viewport screenshot, overlays a subtle coordinate grid, and passes the image to the vision model.

The model calculates the center coordinates of the target UI control and returns a click command: click_at(x: 482, y: 310).

By combining both layers, our browser agents execute eighty percent of typical form interactions through instantaneous DOM actions, reserving visual coordinate computation for the complex custom UI widgets.

Preventing session leakage in isolated cloud sandboxes

Browser agents cannot run on shared developer workstations. If an agent operates a portal that contains HIPAA or financial data, credentials and session cookies must be isolated.

We run every browser session inside an ephemeral Docker container orchestrated on Kubernetes:

• Isolated virtual displays: Each container runs a headless X11 virtual framebuffer (Xvfb) paired with an isolated Chromium process.
• Single-use credential injection: Passwords and multi-factor session cookies are injected into the browser context at startup via secure memory volumes and wiped immediately after execution terminates.
• Network egress filtering: The container can only establish outbound TCP connections to the specific hostnames required by the target legacy application. Even if an agent encountered malicious prompt injection on a public webpage, it cannot transmit exfiltrated data to unknown IP addresses.

Handling edge cases: pop-ups, timeouts, and validation errors

Legacy enterprise portals are notorious for unpredictable behavior: session timeouts after five minutes of inactivity, unannounced maintenance modals, and validation errors rendered as plain red text instead of structured error responses.

A robust browser agent requires self-healing error loops:

1. State assertions before state mutations

Before clicking a "Finalize Payment" or "Delete Record" button, the agent inspects the page state to confirm that all preceding fields hold valid data. If a field shows a red validation border, the agent halts, extracts the error text, and corrects the input before proceeding.

2. Idempotency tokens on form submissions

If a network glitch interrupts a page reload after an agent clicks submit, the agent does not blindly click submit again. Instead, it navigates back to the search index and queries the record ID to verify whether the initial submission processed. This prevents duplicate invoice creations or repeated wire transfers.

3. Automatic session revival

When an ERP displays a session expired modal, our wrapper intercepts the dialog, re-authenticates using the secure identity vault, navigates back to the draft state, and resumes the work ticket without requiring human intervention.

The business case: RPA replacement and manual work reduction

A global freight logistics provider approached us to automate customs clearance filings across twelve regional port authorities. Their operations team spent an average of twenty-two minutes per shipment manually re-entering data from bills of lading into outdated port authority portals.

Within six weeks, we deployed a fleet of containerized browser agents:

• Average processing time per filing dropped from 22 minutes to 90 seconds.
• Human exception handling was reduced to less than 4 percent of anomalous shipments.
• The company avoided a projected $3.2 million internal IT overhaul to build custom EDI integrations that port authorities were unlikely to support anyway.

If your organization has high-value business processes trapped behind legacy web interfaces, green-screen emulators, or portal workflows with zero API support, you do not need to wait three years for an enterprise system replacement. Our engineering group at FoundrySoft designs and operates resilient browser automation agents that interface with your existing software on day one. Schedule a technical discovery session with our agent architects to evaluate your target workflows.