Skip to content
Agent Checker
Back to Blog
TechnicalAPI DesignAI Agents

Content Negotiation: Serving Machine-Readable Formats to Agents

Agent Checker4 min read

Content negotiation is an HTTP mechanism where the client tells the server what format it wants, and the server responds accordingly. A browser sends Accept: text/html and gets a web page. An agent could send Accept: application/json and get structured data from the exact same URL. This is not new technology; it has been part of HTTP since 1.1. But very few sites use it, and agents would benefit enormously if more did.

How content negotiation works

The client includes an Accept header in its request, listing the formats it can handle in order of preference:

GET /products/wireless-keyboard HTTP/1.1
Accept: application/json, text/html;q=0.9

This says: "I prefer JSON, but I can accept HTML if JSON is not available." The q value (quality factor) indicates preference, with 1.0 being the default highest preference.

The server checks the Accept header and returns the appropriate format:

HTTP/1.1 200 OK
Content-Type: application/json
Vary: Accept

{
  "name": "Wireless Keyboard",
  "price": 49.99,
  "currency": "GBP",
  "availability": "in_stock",
  "sku": "WK-2045"
}

Or, for a browser:

HTTP/1.1 200 OK
Content-Type: text/html
Vary: Accept

<!DOCTYPE html>
<html>...the full web page...</html>

The Vary: Accept header is important: it tells caches that the response differs based on the Accept header, preventing a cached JSON response from being served to a browser or vice versa.

Why this matters for agents

Agents that parse HTML have to deal with navigation, sidebars, footers, cookie banners, and all the other elements that surround the actual content. Extracting a product price from a fully rendered web page requires either DOM parsing or visual interpretation. Getting the same data as clean JSON eliminates all of that work.

Content negotiation gives agents a direct path to structured data without requiring a separate API endpoint. The URL stays the same, which means links, bookmarks, and references all work regardless of the client type.

Content negotiation vs JSON-LD

There is an important distinction between content negotiation and embedding JSON-LD in your HTML. With JSON-LD, the agent still receives the full HTML page and has to find and parse the JSON-LD script block within it. With content negotiation, the agent receives only the structured data, nothing else.

Both approaches have their place:

| Approach | Pros | Cons | |----------|------|------| | JSON-LD in HTML | Works for all agents; no server changes needed for content type routing | Agent still downloads full HTML; data mixed with presentation | | Content negotiation | Clean, minimal response; agent gets only what it needs | Requires server-side routing logic; caching is more complex |

For most sites, JSON-LD is the easier starting point. Content negotiation is the next step for sites that serve high volumes of agent traffic or want to provide a true API-like experience without maintaining separate API routes.

Implementing content negotiation

Here is a basic example in Express.js:

app.get('/products/:slug', (req, res) => {
  const product = getProduct(req.params.slug);

  res.format({
    'application/json': () => {
      res.json({
        name: product.name,
        price: product.price,
        currency: product.currency,
        availability: product.availability
      });
    },
    'text/html': () => {
      res.render('product', { product });
    },
    default: () => {
      res.render('product', { product });
    }
  });
});

The res.format() method checks the Accept header and calls the matching handler. The default case handles clients that do not send a recognised Accept header.

Practical considerations

Always include the Vary: Accept header. Without it, CDNs and reverse proxies may cache the JSON response and serve it to browsers, or cache the HTML and serve it to agents.

Support application/ld+json as an Accept type. Some agents specifically request JSON-LD. Responding with a JSON-LD document (including @context and @type) gives them structured data in a format they already know how to parse.

Do not break existing URLs. Content negotiation should never change the behaviour for clients that do not send specific Accept headers. Browsers and headless browser agents should continue to get HTML as before.

Document the available formats. If your pages support content negotiation, mention it in your API documentation or a Link header pointing to the alternative representations:

Link: </products/wireless-keyboard>; rel="alternate"; type="application/json"

This tells agents that a JSON version exists at the same URL.

Content negotiation is one of the most underused features of HTTP. For sites that serve both human visitors and agent traffic, it offers a clean way to satisfy both audiences from a single URL structure, no separate API required.