HTTP status codes and how they affect agent browsing

HTTP status codes are a conversation between your server and the client. When that client is an AI agent, the conversation matters more than usual. Understanding how browser agents work helps explain why: agents make decisions based on status codes, and a wrong code can send an agent into a loop, cause it to give up on a valid page, or waste its budget on retries.

The codes agents care about most

Agents do not treat all status codes equally. Some trigger specific behaviours.

200 OK: The happy path. The agent got the page it asked for. No special behaviour needed.

301 Moved Permanently: The agent follows the redirect and updates its internal map. If it visits this URL again, a well-built agent will go directly to the new location. Use 301 when a page has permanently moved.

302 Found (Temporary Redirect): The agent follows the redirect but keeps the original URL in its map. This is the right code for temporary situations like maintenance or A/B testing. Agents will check the original URL again next time.

403 Forbidden: The agent knows it is not allowed to access this page. It will not retry. It will report the page as inaccessible. This is the right response for pages that require authentication the agent does not have.

404 Not Found: The agent treats this as a dead end. If it followed a link to get here, it will likely report a broken link. Use 404 for genuinely missing pages, not for access control.

429 Too Many Requests: This is critical for agents. A 429 tells the agent to slow down. Well-built agents will look for a Retry-After header and wait before trying again.

500 Internal Server Error: The agent may retry once or twice. If it keeps getting 500s, it will give up and report the site as unavailable. This becomes especially problematic alongside dynamic content loading issues where timing compounds the failures.

The redirect chain problem

One redirect is fine. Multiple chained redirects cause real problems for agents.

Request: GET /old-page
Response: 301 → /new-page
Request: GET /new-page
Response: 301 → /newer-page
Request: GET /newer-page
Response: 302 → /newest-page
Request: GET /newest-page
Response: 200 OK

Three redirects before reaching content. Each one costs time and may count against the agent's operation budget. Most agents have a redirect limit (typically 5-10), and exceeding it causes a hard failure.

Audit your redirect chains. Tools like curl -I -L will show the full chain:

curl -I -L https://example.com/old-page

Fix chains longer than one hop. Each page should redirect directly to the final destination.

Soft 404s are invisible traps

A soft 404 is when your server returns a 200 status code for a page that does not exist. Instead of a proper 404, the server shows a "page not found" message with a 200 status.

Agents take the 200 at face value. They think the page loaded successfully and try to extract content from it. The result might be an agent telling a user "the product page says Page Not Found" instead of recognising that the product does not exist.

# Bad: soft 404
GET /products/deleted-widget HTTP/1.1
HTTP/1.1 200 OK
Content-Type: text/html

<h1>Page Not Found</h1>

# Good: proper 404
GET /products/deleted-widget HTTP/1.1
HTTP/1.1 404 Not Found
Content-Type: text/html

<h1>Page Not Found</h1>

The HTML can be identical. The status code is what matters.

Rate limiting for agents

Agents make more requests per session than typical users. A human might load 5-10 pages during a visit. An agent completing a task might load 20-50. Your rate limiting needs to account for this.

If you return 429 responses, always include the Retry-After header:

HTTP/1.1 429 Too Many Requests
Retry-After: 30
Content-Type: application/json

{"error": "Rate limit exceeded. Please retry after 30 seconds."}

The header value is in seconds. Without it, agents have to guess how long to wait. Some will retry immediately, making the problem worse. Others will give up entirely.

Consider creating higher rate limits for authenticated agents. If a user has sent their personal agent to complete a task, that agent should have a reasonable request budget.

Authentication redirects

Many sites redirect unauthenticated requests to a login page. This creates confusion for agents:

GET /account/orders HTTP/1.1
HTTP/1.1 302 Found
Location: /login?redirect=/account/orders

The agent follows the redirect, sees a login form, and may not know whether to attempt login or report that the page requires authentication. A better approach:

GET /account/orders HTTP/1.1
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer
Content-Type: application/json

{"error": "Authentication required", "login_url": "/login"}

The 401 status code clearly communicates that authentication is needed. The agent can decide whether to attempt login or inform the user.

Checklist for getting status codes right

1. Return 404 (not 200) for missing pages
2. Return 401 for pages requiring authentication, not 302 to a login page
3. Keep redirect chains to a single hop
4. Include Retry-After with every 429 response
5. Use 301 for permanent moves, 302 for temporary ones
6. Return 503 with Retry-After during planned maintenance
7. Test your error pages with curl -I to verify the actual status codes

Status codes are simple in concept but easy to get wrong. For agents, every code is a signal that drives their next action. Get the signals right, and agents will work with your site smoothly. Get them wrong, and you will see agents stuck in redirect loops, retrying failed requests, or treating error pages as content.