The Evolution from Web Scraping to Intelligent Browsing

Web scraping started with curl and regular expressions. You would fetch raw HTML, write a regex to pull out prices or product names, and hope the site did not change its markup. It was fragile, limited, and surprisingly effective for simple tasks.

That was the early 2000s. The tools have changed a lot since then.

The Regex and XPath Era

The first generation of scraping tools worked with static HTML. Libraries like Beautiful Soup in Python made it easier to parse HTML documents and extract data using CSS selectors or XPath expressions. You could write a script that pulled every <td> from a table or extracted all links matching a certain pattern.

This approach had a fundamental limitation: it only worked on server-rendered HTML. As websites moved toward client-side rendering with JavaScript frameworks, the raw HTML returned by a simple HTTP request often contained nothing useful, just a <div id="root"></div> and a pile of script tags.

The Headless Browser Phase

The response was headless browsers. Tools like PhantomJS (now discontinued) and later Puppeteer gave scrapers a real browser engine that executed JavaScript and rendered pages fully before extraction. You could wait for React to finish rendering, for API calls to complete, and for dynamic content to appear.

Puppeteer, released by Google in 2017, became the standard. It controlled a headless Chrome instance and let you script interactions: click this button, wait for that element, scroll down, take a screenshot. Playwright followed in 2020, adding cross-browser support and better handling of modern web features.

This was a significant improvement, but the scripts were still brittle. Change a class name, move a button, add a confirmation dialog, and the scraper would break. Every target site required custom code, and maintaining that code was a constant effort.

The Structural Extraction Layer

A middle generation of tools tried to add intelligence without full AI. Libraries like Scrapy added middleware systems and pipeline concepts. Services like Diffbot used computer vision and heuristics to automatically identify article content, product details, and forum posts without site-specific rules.

These tools worked surprisingly well for common page types. Diffbot could extract an article's title, author, date, and body text from most news sites without any configuration. But they struggled with unusual layouts, interactive content, and anything that did not match their pre-built models.

The Agent Generation

The current generation is fundamentally different. Instead of writing rules for extraction, you give an AI agent a goal. "Find the shipping cost for this product to Manchester." The agent loads the page, reads it, figures out where to find shipping information, and may need to interact with the page to get there, perhaps entering a postcode or selecting a delivery option.

Browser Use, Stagehand, and similar frameworks represent this shift. They combine browser automation (Playwright under the hood) with language models that can reason about page content and decide what to do next.

The key difference is adaptability. A traditional scraper breaks when the site changes. An agent can often handle layout changes, renamed buttons, and restructured pages because it understands the intent behind each action rather than following a rigid script.

What Actually Changed

Three things made intelligent browsing possible.

Language models got good enough to understand web pages. When you feed a DOM tree or accessibility tree to a model like GPT-4 or Claude, it can identify form fields, navigation elements, and content areas with reasonable accuracy. It understands that a field labelled "Postcode" expects a postal code, not because it was programmed to, but because it has seen enough web pages to recognise the pattern.

Vision models reached practical accuracy. Multi-modal agents can look at a screenshot and identify buttons, text fields, dropdown menus, and their spatial relationships. This matters because some information is only conveyed visually, through layout, colour, or iconography that does not appear in the DOM.

Browser automation frameworks matured. Playwright provides reliable, fast browser control with good APIs for interacting with modern web features like shadow DOM, iframes, and service workers. Without solid browser automation, the intelligence layer would have nothing to act on.

The Trade-offs

Intelligent browsing is not strictly better than traditional scraping. It is slower, because each action requires a round trip to a language model. It is more expensive, because every page observation costs API tokens. And it is less deterministic, because the model might take different paths through the same site on different runs.

For bulk data extraction where you need to scrape ten thousand product pages with the same structure, a well-written Playwright script will beat an AI agent on speed, cost, and reliability every time.

Where agents win is on tasks that require understanding and adaptation. Filling out a multi-step form where the fields change based on previous answers. Comparing prices across sites with completely different layouts. Completing a booking that involves choosing between options and handling edge cases.

What This Means for Website Owners

The shift from scraping to intelligent browsing changes what "being accessible to automated tools" means. It is no longer about having clean HTML that is easy to parse with XPath. It is about having a site that an AI agent can understand and interact with naturally.

Semantic markup, clear labels, logical page structure, and predictable behaviour, including proper Schema.org markup. These are the same things that make websites accessible to screen readers, and it is no coincidence. AI agents and screen readers both need to understand a page without seeing it the way a sighted human does.

The sites that invested in accessibility standards years ago are now, almost accidentally, the ones that work best with AI agents.