Designing Docs for AI Agents
Jamdesk Team
Last week, Ben Swerdlow at Freestyle published Designing APIs for Agents, arguing that the API design rules we spent twenty years learning are backwards for AI agents. Agents read everything, so explicitness beats convenience, and precise errors beat forgiving defaults.
He's right, and the same inversion has already hit documentation.
Most docs teams are just coming to realize this.
We run a docs platform, so we know a few things about docs. Here's what's we've learned: write everything down, make every page stand alone, serve markdown instead of chrome, and treat your error messages as first-class content.
Key TakeawaysIn July 2025, Cloudflare measured Anthropic crawling roughly 38,000 pages for every visitor it referred back. Docs are being read at machine scale, by readers who never see your site.The same docs page costs 31x more bytes as HTML than as markdown. Chrome is a token tax on every agent that reads you.Progressive disclosure, the core of human docs design, actively hurts agents.You don't need separate docs. You need more entry points into the same content:llms.txt, per-page.mdendpoints, and an MCP server.
Your Reader are Now AI Agents
Documentation traffic is shifting from people to machines, fast. From May 2024 to May 2025, AI and search crawler traffic grew 18%, with OpenAI's GPTBot alone up 305% (Cloudflare, 2025).
Need more convincing? In July 2025, Anthropic's crawlers fetched about 38,000 pages for every one human visitor they referred back; OpenAI's ratio was around 1,100 to one (Cloudflare, 2025).
Meanwhile the humans moved to asking chat/AI agents instead of their own research. By 2025, Stack Overflow was receiving about as few questions per month as it did in 2009, the year it launched (The Pragmatic Engineer, 2025). Did the developers stop having questions? Nope, they started asking assistants, and the assistants ask your docs.
So the question isn't whether to design docs for agents. Agents are already your highest-volume audience.
The question is whether they can use what they find.
The Playbook We All Learned
Good docs for humans follow certain rules, which can be summarized as "Don't make me think". For example, start with a quickstart guide, everyone loves help videos, and hide the complex flags until the reader needs them. You also want to have everything well organized in a side nav so I can quickly find what I'm looking for.
Every one of those rules exists because human attention is scarce. Your users are busy, get bored in minutes, and leave the moment they feel lost (or assume this isn't what they are looking for). Traditionally, yours docs serve two purposes: technical/informational data and marketing. The former is self-evident, but marketing you might have not considered.
We wrote a whole post on writing documentation developers actually read, and if you compress it to one idea, it's attention management: respect the reader's time, surface the likely next step, get out of the way.
Design for the reader's scarcest resource, in other words. For humans, that's attention.
Agents Break Every Assumption
An agent's scarce resource is context tokens: what it can afford to fetch and hold while answering. Attention never enters into it.
When agents arrive at your site, they aren't navigating, click though your sidebar or typing in your search box, they are fetching. An agent lands on one URL, cold, and either finds the answer there or fetches again.
Agents read everything you give them, instantly. Swerdlow's phrase for APIs was "agents can read our entire docs in one sitting." Hiding complexity from them doesn't reduce confusion as it does for people.
And agents arrive opinionated. Their training data contains your 2025 docs, your deprecated auth flow, your old parameter names. Live documentation is the correction mechanism for those stale priors. If the current truth isn't written down and fetchable, the agent uses the old version.
One caution before you swing the other way: reading everything isn't free. In July 2025, Chroma tested 18 frontier models and found every single one degraded as input length grew, a failure mode they named context rot (Chroma Research, 2025), so don't make one giant page of all your docs.
Use Your Words
Give agents the words without the website by proving them with markdown (md file). We measured our own quickstart both ways, and you can rerun this today:
$ curl -sLo /dev/null -w "%{size_download} bytes\n" https://jamdesk.com/docs/quickstart
225527 bytes
$ curl -sLo /dev/null -w "%{size_download} bytes\n" https://jamdesk.com/docs/quickstart.md
7246 bytes
That is 31x fewer bytes, and another example is Vercel ran the same test on their own pages in 2026 and measured 500 KB of HTML against 3 KB of markdown.
All that extra load (218 K) are scripts, styles, nav, and hydration payload - what makes the web so pretty. But agents don't need or want this, since they will spend tokens downloading it all and then throw it away.
Measured 2026-07-16. Both URLs are live if you want to check our math.
And to really hammer the point home, we scored seven docs platforms for AI-friendliness and one served 1,049,919 bytes of HTML containing 1,279 bytes of text because the content only existed after JavaScript ran. To a crawler doing plain HTTP, those docs are a loading spinner.
The fix is a .md markdown mirror of every page. You create the same URL plus .md, plain markdown. Stripe ships this across their entire docs site. So do we, on every Jamdesk site, automatically.
Server-render your HTML too. But give agents the door that doesn't cost 31x.
Be Explicit
Swerdlow's harshest API rule is "defaults are bad" and the magic that helps humans makes agents guess. The docs equivalent is implicit knowledge is bad, explicit knowledge is good.
What do I mean by this? Remember, the AI agent might only be fetching a single page or even just a section of it. So you don't want to make the AI have to guess, assume, or have to figure out contradictions.
For example:
- No relative references. "As mentioned above" and "see the previous section" are dead weight to an agent who fetched one page. Link to the actual page instead. Also, this is a good SEO practice.
- Full context per page. Restate which product, which version, which API. The agent may have fetched nothing else.
- One name per concept. If the same thing is a "project" on one page, a "site" on another, and a "workspace" in the dashboard, an agent treats them as three things. Swerdlow calls this staying in distribution. Tech writers call it terminology discipline and consider it a best practice.
- And date things. "As of the 2026-07-28 protocol revision..." lets an agent weigh your page against what it half-remembers from training.
Errors Are Documentation
An agent debugs by searching fop the error. If it hits a failure, the agent does a search with the error message as a query string. Pretty sweet if your docs are the result!
A good idea is to document the error path with the same priority as your tutorials. A page per error, or a catalog, with the verbatim message as the heading:
## Error: `DNS_VERIFICATION_FAILED`
**What it means:** We couldn't find the TXT record proving you own this domain.
**Why it happens:** The record hasn't propagated (wait up to 48h), or it was
added to the wrong zone (check the apex, not the subdomain).
**Fix:** `dig TXT _jamdesk-verify.yourdomain.com` should return your token.
If it doesn't, re-add the record from Settings → Domains → Verify.
As Swerdlow put it for APIs: errors are one of the best surfaces an agent has for finding the happy path. Your docs are where that surface lives.
Examples are Important
For human readers, a code sample is a teaching aid. For agents, it's a template that gets instantiated verbatim, at scale. Agents pattern-match against examples more strongly than against prose, which means a subtly wrong sample has a large impact.
Some good guidelines on examples for AI agents:
- Runnable as printed. Real endpoints, real field names, no
<your-thing-here>inside strings that look copy-pasteable. - Complete. Include the imports, the auth header, the response shape. An agent won't infer the missing 20% the way a senior dev would; it will invent it.
- Current. A deprecated example is worse than no example, because it's evidence for the stale behavior the agent already believes.
The Discovery Stack: llms.txt, .md, MCP
Everything above assumes the agent found the right page. That's the job of three pieces of infrastructure, one of which is genuinely controversial.
Three entry points into the same content.
llms.txt is a markdown index at your site root: what exists, where, one fetch. By May 2026, 36,120 sites published one, up 8.8x in a year (PPC Land, 2026).
But do the AI agents read the llms.txt file? In June 2026, Ahrefs measured 97% of those files receiving zero requests in a month. Google's John Mueller had compared the file to the keywords meta tag a year earlier (2025), and defenders have argued with him since.
Our take: Mueller is right about background crawlers and wrong about what else matters. Agentic fetchers (a Claude Code session, a Cursor index, anything told to go read the docs) use it as a map when it exists, because it's the cheapest possible answer to "what pages should I fetch?"
Stripe ships one with instructions for LLM agents embedded in it. When it's auto-generated from your nav, like ours, the cost is zero. Zero-cost insurance doesn't need a consensus.
Per-page .md endpoints. As discussed above (oops, we shouldn't be doing this), the token and speed savings are measurable.
An MCP server turns your docs from pages into tools: searchDocs, getPage. Every Jamdesk site exposes one at /_mcp, no key required. One command makes any of our customers' docs readable from inside Claude Code:
claude mcp add --transport http jamdesk-docs https://jamdesk.com/docs/_mcp
One last trick: put instructions for agents inside the content, where every reader lands. The top of our own quickstart reads:
> **For AI agents:** the complete documentation index is at [llms.txt](/docs/llms.txt).
> Append `.md` to any page URL for its markdown version.
Any agent that lands on any page now knows about both entry points.
Summary
Your docs were already your most important marketing channel back when only humans read them. Now they're your interface to every AI assistant your users work inside, so you should focus on making them the most AI-friendly docs possible. We also built AI Score to automate exactly this audit, and scored seven platforms with it if you want a baseline.
Or, if you'd rather let someone else do it for you, every Jamdesk site comes with the whole stack by default: .md endpoints, llms.txt, llms-full.txt, and an MCP server, generated from the docs you already have.
Deploy one in five minutes, then curl your own quickstart with .md on the end.