Content Security Policies and Their Effect on Agent Browsing

Content Security Policy headers tell browsers which resources are allowed to load on a page. They are a solid defence against cross-site scripting and injection attacks. But when AI agents visit your site using headless browsers, strict CSP configurations can prevent pages from rendering fully, leaving agents with incomplete or broken content.

How CSP works in brief

A CSP header specifies allowed sources for scripts, styles, images, fonts, and other resources. Here is a typical policy:

Content-Security-Policy: default-src 'self'; script-src 'self' https://cdn.example.com; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:;

This tells the browser: only load scripts from the same origin or cdn.example.com, allow inline styles, and permit images from any HTTPS source or data URIs. Anything not covered by a specific directive falls back to default-src 'self'.

Where CSP causes problems for agents

AI agents that use headless browsers (Playwright, Puppeteer, or similar tools) render pages just like a regular browser. The CSP rules apply in full. Several common configurations cause issues:

Blocking inline scripts. If your CSP does not include 'unsafe-inline' or nonce-based script allowlists, inline <script> tags will not execute. Many sites rely on inline scripts for initialisation: setting up analytics, loading dynamic content, or hydrating server-rendered components. When these fail, the agent sees a half-rendered page.

Third-party resource restrictions. A strict default-src 'self' policy blocks all third-party resources unless explicitly allowed. If your page loads fonts from Google Fonts, scripts from a CDN, or images from a media server, the CSP must list each origin. Missing any of them means the resource does not load, and the page may break in ways that are invisible during normal testing but obvious to an agent.

Web Workers and Service Workers. Some single-page applications use workers for background processing. If your CSP's worker-src directive is too restrictive, workers fail to initialise, and features that depend on them simply stop working.

The agent-specific angle

Agents do not just render pages visually. They also inspect the DOM, read text content, and interact with elements. When CSP blocks a script that handles dynamic content loading, the agent may see a loading spinner that never resolves, or an empty container where product information should appear.

The challenge is that CSP violations are silent from the agent's perspective. The browser blocks the resource and logs a console error, but the page does not show an error message. The agent sees what looks like a normal page with missing content, and it may not know why.

Practical CSP configurations for agent compatibility

You do not need to weaken your CSP to accommodate agents. Here are approaches that maintain security while keeping pages functional:

Use nonces instead of unsafe-inline

Content-Security-Policy: script-src 'self' 'nonce-abc123';

<script nonce="abc123">
  // This runs because the nonce matches
  initApp();
</script>

Nonces let you allow specific inline scripts without opening the door to all inline execution. Generate a unique nonce per request.

Whitelist known CDN origins

Content-Security-Policy: script-src 'self' https://cdn.jsdelivr.net https://unpkg.com; style-src 'self' https://fonts.googleapis.com; font-src https://fonts.gstatic.com;

List every external origin your page needs. This is tedious but prevents the situation where a resource silently fails to load.

Use Content-Security-Policy-Report-Only for testing

Before deploying a new CSP, use the report-only variant:

Content-Security-Policy-Report-Only: default-src 'self'; report-uri /csp-reports;

This logs violations without blocking resources. Review the reports to find resources that would be blocked, then adjust your policy before enforcing it.

Testing with agent-like conditions

To see how your CSP affects agents, test your pages in a headless browser with the same constraints an agent faces. Launch Playwright or Puppeteer, load the page, and check the console for CSP violation messages. If critical scripts are blocked, the page that your agent sees is not the page your human visitors see.

You should also verify that your HTML structure remains accessible even when some scripts fail. A well-built page degrades gracefully: the content is present in the initial HTML, and JavaScript adds interactivity rather than content. This is essentially progressive enhancement, and it is the best insurance against CSP-related rendering issues.