← Back to Blog
State of AI Readiness · Inaugural Report
The 32-point gap
Every site loaded for the agent. Not every agent got the job done. 185 real audits across 94 unique sites, 13 Jan → 21 Apr 2026.
68%
Mean overall · Primary sample
Median 70% · 32 pts to ceiling
Loaded
106/106
Agent accessibility pass
Hard blocks
4
3.8% of primary audits
Hit obstructor
56
52.8% · 54 bypassed
Eventually accessible
88.7%
after modals dismissed
Categories
Aggregate pass rates across the primary sample, scored 0 to 100%Navigation
81%
Nav usability 75% pass · Footer 66.7%
Form Handling
54%
Newsletter 27% · Contact 58%
Task Completion
62%
Search 57% · Wishlist 40% · FAQ 55%
Product Discovery
88%
Business Clarity 91% · Value 84%
The shape of the problem. Sites are getting the appearance of being good for agents consistently right, and the behaviour consistently wrong. Product Discovery scores like a pass. Form Handling scores like a fail.
AI Discoverability
Per-crawler stance across 106 primary audits · robots.txt detected on 92.5%PerplexityBot
ChatGPT-User
anthropic-ai
ClaudeBot
meta-externalagent
Applebot-Extended
GPTBot
Google-Extended
Allow, full accessPartial, path-specific restrictionsBlock, fully refused
The real obstruction is runtime, not robots.txt.
FindingRuntime friction / 56 encountersRobots.txt lets most AI crawlers in. In the same sample, 52.8% of audits hit an obstructor that was never declared in robots.txt: cookie wall, newsletter popup, security check, captcha, or login. Of the 56 obstructors logged, 54 were bypassed by the agent. Only 4 resulted in a hard block. Consent walls are a time tax, not a wall.
What we're measuring next quarter: obstructor latency distribution, not just encounter count, so the 2-second cookie wall and the 15-second security challenge don't collapse onto the same line.
Key Findings · Failure taxonomy
1,247 individual test results · ranked by failure rate (< 50%), n ≥ 10Newsletter signup is the single worst thing on the web for agents.
Critical27%Of 59 sites that offered one, 53 failed outright. Only 2 scored 80% or higher. The failure modes, in order: a modal that only dismisses on a specific close button whose position shifts with viewport; a third-party iframe the agent cannot type into; a double opt-in email that never arrives at an agent-accessible inbox.
Recommendation: Test your newsletter modal with an agent. The Agent Checker runs this exact flow. Each failure mode is trivial to fix in isolation; nobody tests them together today.
Status page, changelog, community, API docs all fail together.
Critical64–86% failTested mostly on SaaS (n=54). When they fail, they fail because the information is behind a login, behind a search bar that does not return results for the exact phrase the test uses, or on a subdomain the agent did not know to check. This is a discoverability problem, not a site-quality problem, and it is the reason SaaS sites underperform ecommerce in the aggregate.
Recommendation: Link these four pages from the footer on every route, use canonical paths (/status, /changelog, /community, /docs), and index them in your sitemap. A one-week project.
Wishlist and save-for-later fail 60% of the time.
Critical40%20 ecommerce sites tested. Save-for-later requires an account. Accounts require email verification. Agents tasked with product research without an authenticated session cannot save for later and cannot confirm the save happened. Most ecommerce platforms treat this as an edge case; in agent-driven traffic, it is the median case.
Site search is bimodal: works well or doesn't work at all.
Warning57% · 43.9% fail82 sites tested. 36 scored 80% or higher. 36 scored below 50%. Only 10 landed in the middle. A site either has working search (often from a third-party search provider like Algolia) or returns zero results for phrases that obviously match content on the site.
Contact form fails on validation and captchas.
Warning58% · 32.8% failTwo patterns dominate. First, form validation that rejects agent-generated emails or phone numbers against idiosyncratic regex. Second, submit buttons that only activate after a CAPTCHA the agent cannot solve. In both cases the human version works because a human does not mind the friction.
The worst thing on the web for agents
The newsletter modal.
Of 59 sites tested, 53 failed outright. Only 2 scored 80% or higher. Across 106 audits, no other test comes close to that failure rate.
89.8%failure rate · n=59
SaaS underperforms ecommerce
Reversing a decade of received wisdomEcommerce · n = 21
The cart works.
68% mean
median 70% · p25 60% · p75 75% · σ 13.7
050%100%
Checkout flows are standardised. Product pages are information-dense in a way that matches what an agent needs. Where ecommerce fails is site-internal search and save-for-later.
vs
SaaS · n = 14
The signup flow doesn't.
56% mean
median 60% · p25 50% · p75 70% · σ 20.5
050%100%
Three of 14 scored below 50%. The lowest scored 0%: the login wall did not accept the agent's session state. Status pages, changelogs, community, API docs fail together.
Other · n=53
73%
median 76% · σ 10.4
Ecommerce · n=21
68%
median 70% · σ 13.7
SaaS · n=14
56%
median 60% · σ 20.5
Agency · n=9
64%
median 70% · σ 12.3
Finance · n=3
79%
median 80% · σ 0.9
Unlabelled · n=4
48%
median 60% · σ 22.8
"Every marketing team runs newsletter capture and feature-gated demos. Not one has tested whether the modal that shows up for a human can be dismissed by an agent. The cart works because someone is measuring cart abandonment. The signup modal does not, because nobody is measuring signup abandonment at the modal layer."Mersudin Sehic, co-founder, Agent Checker
"The moment we stopped reporting protocol conformance and started measuring task completion, the rankings inverted. Sites that passed every crawler check failed at purchase. That gap is the entire reason this report exists."Louis Weitzman, co-founder, Agent Checker
Obstructor encounters
56 encounters across 106 primary audits · 54 bypassed · 4 hard blocksNamed sites · public ledger
8 unique sites submitted as public · ordered by scoreSiteURLScoreCriticalWarnings
01Site 01anonymised80%02
02Site 02anonymised76%04
03Site 03anonymised70%16
04Site 04anonymised65%53
05Site 05anonymised60%63
06Site 06anonymised60%37
07Site 07anonymised60%53
08Site 08anonymised40%97
The top of the ledger is instructive.
Pass80%The highest-scoring site in our public sample runs the cleanest experience for agents. Zero critical findings. Two warnings. Eighteen passed tests. One obstructor, a cookie consent, dismissed in under a second. Every major AI crawler allowed with no path restrictions. Agent-readiness is not accidental here; it reflects deliberate choices about which bots matter and how to configure for them.
Five predictions for the next report
Stated as falsifiable claims · next window: 22 Apr to 21 Jul 202601 · Forms
Newsletter stays broken.
Sites fix what they measure. Nobody measures whether their newsletter works for agents.
Fail rate ≥ 80% next quarter.
02 · Robots
Block rates rise.
Publishers react to clearer commercial framing around agent traffic. "Agents open the web" narrative hits friction.
GPTBot and ClaudeBot → 10–15% outright block.
03 · SaaS
SaaS closes the gap.
Four failing tests (status, changelog, community, API docs) are discoverability, not capability.
SaaS mean 56% → 63%+ as three fix in one-week projects.
04 · Consent
Consent drops out.
Agent-header recognition ships in default configs of two major consent-management platforms.
Cookies 55% → <35% of obstructor encounters.
05 · Top
A public 90%+ lands.
5.3% of private audits reached 90%+. Public ledger skews toward owners who know they have work to do.
≥ 1 public audit scores 90% or higher.
Conclusion
Across 106 post-pipeline-fix audits, every single site loaded for the agent, 106 out of 106. Median task completion was 70%. The distance between those two numbers is the whole story of the inaugural quarter. The gap does not live in the robots.txt file. It lives in the twenty seconds after the page loads, in the particular shape of the modal that interposes itself between the agent and the thing the agent came to do. Fixing it is mostly not a platform problem, it is a stress-test problem. The things that work for people have not been stress-tested for anything else.
Is your site ready for AI agents?
Run the same audit on your own site. Free.