Dynamic Content Loading and the Timing Problem for Agents

Open any modern web page and watch the network tab. The HTML arrives first. Then CSS files. Then JavaScript bundles. Then API calls fire, fetching product data, user reviews, recommendations, pricing information. Each response triggers more rendering. The page assembles itself in stages over several seconds.

Humans barely notice. The page "feels" ready within a second or two, even if content is still loading in the background. AI agents face a harder question: when is this page actually done?

The "Ready" Problem

Browsers fire several events during page load. DOMContentLoaded fires when the HTML is parsed. The load event fires when all resources (images, scripts, stylesheets) are loaded. Neither event means the page content is complete.

On a typical e-commerce product page, especially single-page applications that rely heavily on client-side code, the load event fires in about 2.3 seconds (median from our testing). But product reviews don't appear until 4.1 seconds. Recommended products load at 5.8 seconds. Dynamic pricing (based on user location or A/B test) might not appear until 3.5 seconds. An agent that reads the page at the load event misses reviews and recommendations entirely.

The Waterfall of API Calls

Modern pages often have chained data dependencies. The product page loads, which triggers a call to fetch pricing, which triggers a call to check inventory at the nearest warehouse, which triggers a call to calculate delivery estimates. Each step depends on the previous one completing.

We profiled 75 product pages and found an average of 12 API calls per page, with the last one completing 6.2 seconds after the initial HTML load. The longest we recorded was 14.7 seconds on a travel booking site that was pulling real-time flight availability from multiple airline APIs.

An agent has two bad options: wait a fixed amount of time (wasting seconds on fast pages, missing data on slow ones) or try to detect when all API calls have completed (complex and unreliable).

MutationObserver: Watching the DOM Change

Sophisticated browser-using agents use the browser's MutationObserver API to watch for DOM changes. The logic: if the DOM hasn't changed for a certain period (say, two seconds), the page is probably done loading.

This works reasonably well but has failure modes. Some pages have continuous DOM mutations from animations, carousels, live clocks, or stock tickers. The DOM never truly "settles," so the agent either waits forever or picks an arbitrary timeout. Other pages load content in long-delayed batches; two seconds of quiet doesn't mean everything is loaded if the next batch fires at second three.

We've seen agents get tricked by chat widgets. A customer service chat bubble that animates on a loop creates constant DOM mutations. The agent never sees the page as "settled" and eventually hits its maximum timeout, having wasted time watching a bouncing chat icon.

Real Consequences

The timing problem creates inconsistent results. Run the same agent on the same page ten times and you might get different data each time, depending on network latency, server response times, and exactly when the agent decides to read the page.

In our shopping agent benchmarks, timing-related data inconsistency accounted for 23% of all agent errors. The agent found a product but missed the price. Or got the price but missed the "out of stock" badge that loaded two seconds later. Or read a placeholder price of £0.00 before the real price rendered.

Signals That Help Agents

Loading state indicators. A consistent pattern for indicating that content is still loading gives agents something to watch for. If every loading component uses a data-loading attribute or a shared CSS class like is-loading, agents can check whether any loading indicators are still active before reading the page.

Stable content containers. Pre-rendering empty containers with fixed dimensions (skeleton screens) that get filled with data is better than injecting new elements. The agent can see that a container exists and wait for its content to change from placeholder to real data.

HTTP headers for load timing. A custom response header like X-Expected-Load-Time: 5000 tells agents roughly how long to wait for all content. This is uncommon today, but it's a simple addition that would help enormously.

Structured data in initial HTML. If product data, pricing, and availability are included as JSON-LD structured data in the initial HTML response, agents can read that data immediately without waiting for visual rendering. The browser can still load the pretty version asynchronously.

The Standard We Need

There's no web standard for "this page is fully loaded and all content is ready to read." document.readyState comes close but doesn't account for asynchronous content. A standardised signal, something like a PageContentReady event that site developers fire when all meaningful content is loaded, would solve this for agents and for performance monitoring tools alike. Until that exists, agents are left guessing. And guessing means getting it wrong some of the time.