What Is llms.txt, and Does It Work for Your Docs?

You’ve probably heard of llms.txt by now. Maybe someone on your team saw a thread calling it the new robots.txt, or an SEO tool flagged that your site doesn’t have one. The idea is simple. It’s a small Markdown file on your site that tells AI tools which pages matter most and what each one covers.

Whether it matters for your website and docs depends on what you want from it. If developers point coding agents like Claude Code at your docs, the file can help those agents find the right page with fewer wrong turns. If you’re hoping it gets you cited in ChatGPT or Google’s AI answers, the evidence so far says it won’t. For most docs sites, that makes it worth adding if your platform generates it for you, and not worth much more time than that.

We make documentation software that generates llms.txt, so we have a reason to want it to work. That’s why we went to the sources. We read the original proposal, Google’s guidance, two large studies, a benchmark whose data we recomputed, and our own docs traffic. Below is what the file is, where it helps, where it doesn’t, and how much effort it deserves.

The Short Version

• Google’s AI optimization guide says Search and its AI features don’t treat llms.txt as a special signal.
• SE Ranking studied about 300,000 domains and found no link between having llms.txt and getting AI citations.
• In Mintlify’s July 2026 benchmark, a linked llms.txt cut Claude Code’s 404s on Markdown docs from 0.79 to 0.10 per task.
• Ahrefs found that 97% of about 38,000 sites with llms.txt got no requests for it in May 2026.

What Is an llms.txt File?

llms.txt is a Markdown file that lists a site’s most useful pages, each with a one-line description, under a short introduction. Jeremy Howard, co-founder of Answer.AI, proposed it in September 2024 so models could use a website while answering someone’s question. A small one looks like this:

# Acme API

> Documentation for Acme's payments API.

## Get Started

- [Quickstart](https://docs.acme.com/quickstart.md): Make your first request
- [Authentication](https://docs.acme.com/auth.md): Create and use an API key
- [Webhooks](https://docs.acme.com/webhooks.md): Verify and handle events

The current proposal puts the file at /llms.txt or under a path such as /docs/llms.txt, where it covers the pages beneath that path. Only the project-name heading is required, but the descriptions and links are what make it useful.

The file doesn’t grant access to private content or tell crawlers what they may read. You still need sign-in for access control, robots.txt for crawl rules, and pages that answer the reader’s question. It also helps to keep two jobs apart. One is helping an assistant find your authentication guide. The other is getting your product recommended in an AI answer. Either can happen in a chat app or a coding agent, and the evidence for each is very different.

Does llms.txt Help You Get Cited by AI?

No study we found shows that it does. Google is explicit about it. Its AI optimization guide says Search, AI features included, doesn’t use llms.txt or similar files as a special mechanism. Google may crawl the file like any other URL, and that’s all.

For other answer engines, SE Ranking’s study, first published in November 2025, looked at about 300,000 domains and found no correlation between having llms.txt and getting AI citations. Its prediction model even did better once the llms.txt variable was removed. That’s observational research, so it can’t say what adding the file would do for your own site.

Traffic studies answer a smaller question. Ahrefs’ June 2026 study covered 137,210 domains and detected llms.txt on roughly 38,000 of them. Of those, 97% got no requests for the file during May, which is a lot of files waiting by the phone. Ahrefs notes that its customers are more technical and SEO-minded than the web at large.

None of that justifies selling llms.txt as a citation tactic. It also doesn’t prove the file is useless when an agent does ask for it, and answering that takes watching an agent at work.

Does llms.txt Help Agents Find the Right Page?

Yes, in the one test with public data. Mintlify’s July 2026 benchmark served docs four ways: HTML, Markdown, Markdown with a linked llms.txt, and Markdown with the index pasted into the page. Claude Code and Codex, two coding agents, answered questions across 20 Mintlify-hosted sites, for 2,400 valid attempts.

The useful comparison is Markdown with and without the index. The table counts the average requests per task that came back 404 (page not found), which covers wrong guesses at a page’s address and requests for an llms.txt that wasn’t there. We recalculated these from the released results rather than rerunning the agents, using the same 2,400 valid attempts the publisher used and leaving out the 272 that failed.

The detailed results show that most of the HTML version’s 404s were requests for .md pages or an llms.txt that didn’t exist yet, so much of the drop comes from those files appearing. On plain Markdown, more of the 404s were real wrong guesses at a page’s address.

Both agents found the right page 94% to 99% of the time in every format, so the index mostly cut the wrong turns on the way. The test measured picking the URL, not the code or answer the agent wrote afterward. It’s also a vendor’s benchmark on its own hosting, with agents told to fetch through a proxy using curl and no search or normal browsing. For docs served as Markdown, I’d still include the linked index.

Our own traffic shows the file gets used, without saying how well. From September 12 to October 12, 2026, Jamdesk-hosted docs received 160 llms.txt requests from three tools that fetch pages for a person (crawlers not counted), after we removed known audit and internal test traffic. Claude Code sent 83 of them and ChatGPT-User 62, with Claude-User sending the other 15.

Those are requests, not people, and we didn’t see the prompts or what the agent did next. Anthropic documents web search and fetching a given page as separate tools, so a request alone can’t tell you whether someone was comparing products, writing an integration or asking the agent to read the file.

How Do You Help Agents Find Your llms.txt?

Give the tool a starting point. Link the index from your quickstart or agent setup page, and give developers a prompt to paste into a browsing-enabled assistant:

Start with https://docs.example.com/llms.txt.
Find the webhook signature verification guide, then show me
how to verify an incoming request using the current Node SDK.
Cite the documentation page you used.

That points the tool straight at the index instead of hoping an answer engine finds it. The proposal also describes discovery links for every page: rel="describedby" points to the index, and rel="alternate" with type="text/markdown" points to a Markdown copy of the page. On the authentication page, that looks like this:

<link rel="describedby" href="/llms.txt">
<link rel="alternate" type="text/markdown" href="/auth.md">

Tools that support these links can follow them, but publishing them doesn’t make an agent use them. Cloudflare’s docs for agents show the idea at a larger scale, with a site-wide index plus one per product, so an agent working on Workers can start with the Workers docs. Cloudflare also offers Markdown pages, skills and MCP access as separate ways in.

What Is llms-full.txt?

llms-full.txt puts the content of your docs into one file, where llms.txt only links to pages. Mintlify says it developed the format with Anthropic, and the llmstxt.org proposal doesn’t define it. One file can be handy for a small docs set or a deliberate bulk import.

For large docs, check what the receiving tool can take in, since it may cut off a long response or cap fetches well below the model’s context window. Per-product indexes or page-by-page retrieval avoid sending the whole site every time, and a searchable docs MCP server (a service an agent can query for the pages it needs) works better for a large reference.

Whatever the format, the pages still need complete code examples, the current API version and errors explained, which our guide to designing docs for AI agents covers. Markdown helps because it keeps headings, code blocks and links without the page’s markup. Cloudflare measured one blog post at 16,180 tokens as HTML against 3,150 as Markdown, though a tool that already extracts readable text won’t save that much.

Is llms.txt Worth Adding to Your Docs?

For developer docs, yes, as long as it costs you almost nothing. I’d generate llms.txt from the same content and navigation that build the docs, link to it, and look over the output when pages move. I wouldn’t keep a second copy of the docs by hand, and I wouldn’t add it to a marketing site hoping for citations. Before spending more time on the index, check the pages it points to:

• Can a developer find the current authentication method and a working quickstart?
• Are code samples complete, with their dependencies and versions?
• Do error explanations tell the reader what to change?
• Can the tools your customers use read the content without a login, broken link or hidden tab in the way?

Then try a support question in the tool your customers use, in fresh sessions with and without the index, keeping the model and tools the same. Check which pages it finds and whether the answer is right, and repeat it, because one good response proves little. Lighthouse won’t settle it either. Its optional llms.txt audit only flags server errors, and passing it says nothing about rankings or answers.

Jamdesk generates llms.txt and llms-full.txt, Markdown pages and a docs MCP server from your content, but it can’t tell you whether AI answers will cite you. To see which of these your current docs already expose, run them through the free AI Score checker.