Insurance Comparison Through AI Agents: What Works and What Doesn't
Insurance comparison is one of the tasks people most frequently delegate to AI agents. The reasoning is obvious: comparing policies is tedious, the details matter, and humans are bad at holding many variables in their head simultaneously. Agents are good at exactly this kind of structured comparison.
Comparison sites handling car, home, and travel insurance sit across a wide spectrum of agent readiness. The differences come down to how much useful data agents can actually extract. It is worth walking through three patterns, from worst to best.
The Obstructive Pattern
Some sites present comparison results as a styled table where each row is a policy. So far, reasonable. But the table is built entirely with divs and CSS Grid. There are no table, tr, or td elements anywhere. Column headers are background images. The premium amounts are rendered using a custom web font where the characters map to different glyphs, presumably as an anti-scraping measure. So the HTML might contain "£abc.de" which renders visually as "£312.45" through font substitution.
This approach defeats every agent. Not one can extract a single accurate premium figure.
The quote journey is often just as problematic: a single-page application with many steps, no URL changes, and a short session timeout. If an agent pauses to process data between steps, the session expires and the entire quote restarts. Agents rarely complete the full journey.
The Partway Pattern
A clear improvement uses real HTML tables with proper th and td elements. Premiums are plain text. Policy names link to detail pages. Each result includes the insurer name, annual premium, excess amount, and a brief feature summary.
Where this pattern falls down is in the detail. When an agent needs to compare specific coverage items, like whether legal expenses cover is included or what the personal belongings limit is, this information often sits behind modal popups triggered by JavaScript event handlers. The modals load content dynamically and are not accessible via direct URLs. An agent can see a handful of headline figures per policy but cannot access the many detail fields that make the difference between policies.
The quote form may be well-structured with labelled inputs and visible validation messages, so agents can often complete the initial quote. But non-standard components, such as a custom autocomplete for vehicle modification questions in a car insurance flow, still trip up most agents.
The Agent-Ready Pattern
The strongest pattern takes a different approach to comparison results entirely. Rather than a table, each policy is presented as a standalone card with all key information visible. The card uses semantic HTML with microdata markup. Premium, excess, coverage limits, and included features are all present as text in the initial HTML response, with no JavaScript required to reveal them.
Each policy card links to a full detail page with a permanent URL. The detail page contains a structured comparison table (actual HTML table elements) covering every coverage item, limit, and exclusion. An agent can visit this page directly and extract the complete policy specification.
The quote form uses standard HTML inputs throughout. The vehicle modification section uses a simple text input with server-side validation rather than a complex autocomplete widget. Agents complete the full quote journey reliably.
The Business Case
When a comparison site is genuinely agent-ready, a high share of visitors arriving via AI agent referrals go on to complete a quote request, typically a better rate than organic search or paid advertising. The reason is straightforward: agent-referred visitors have already been matched to appropriate products, so they arrive knowing what they want.
Agent-referred visitors also tend to compare fewer policies before purchasing. They spend less time on the site but convert at a higher rate. This is efficient matching at work.
What Insurance Sites Should Fix First
If you run an insurance comparison site, three things matter most for agent compatibility.
First, comparison results must be machine-readable. Every premium, excess, and coverage figure should appear as plain text in well-structured HTML. Font-based obfuscation or image-rendered text will block every agent.
Second, policy details cannot hide behind modals or JavaScript interactions. If an agent cannot link directly to the full policy specification, it cannot do proper comparison. Give every policy detail page a permanent, accessible URL.
Third, quote forms need to use standard HTML form elements. Every custom widget, non-standard autocomplete, and custom date picker is a potential failure point. The more standard your form controls, the more agents will complete the journey.
The anti-scraping concern is real, but font substitution and CAPTCHAs hurt legitimate agent traffic more than they deter determined scrapers. Rate limiting and API-first design are more effective and less destructive to the agent experience.