llms.txt vs robots.txt: two files with different jobs.
In llms.txt vs robots.txt, the jobs differ. robots.txt tells crawlers which parts of your site they may access, and major crawlers follow it. llms.txt is a proposed Markdown file that points AI agents to your useful pages, and nothing obliges anyone to read it. Every site should get robots.txt right first. llms.txt is optional, and cheap to add.
Reviewed by Anubhav Gupta, Bespoke Devs · Updated
llms.txt vs robots.txt, side by side
| Compare | robots.txt | llms.txt |
|---|---|---|
| Job | Manages which URLs crawlers may access | Gives AI agents a short, curated map of useful content |
| Status | Long established, and supported by major search engines and crawlers | A proposal, published in September 2024, open for community input |
| Format | Plain text rules for each crawler | Markdown with a title, summary and links |
| Where it lives | Site root | Site root, or a subpath such as /docs/llms.txt |
| Who reads it | Googlebot, OAI-SearchBot, ClaudeBot, PerplexityBot and others | Mostly agents and tools that choose to, particularly on developer docs |
| Can it hurt you? | Yes. One wrong line can block your site from search | Rarely. An out-of-date file is the main risk |
| Does it affect AI answers? | Yes, if it blocks a search crawler | We found no evidence that it does |
From llmstxt.org and each company’s own documentation.
What robots.txt does and doesn’t do
Google says robots.txt is “used primarily to manage crawler traffic to your site”. It also warns that it “is not a mechanism for keeping a web page out of Google”, because a blocked URL can still appear in results. To keep a page out of Search, use noindex or a password.
For AI, robots.txt is where you make the choice that really matters. OpenAI, for example, says sites opted out of OAI-SearchBot won’t be shown in ChatGPT search answers, while GPTBot is a separate setting for training. Anthropic also lists separate crawlers for model development and search. Perplexity recommends allowing PerplexityBot if you want to appear in its results.
User-triggered fetchers are an exception. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. So robots.txt isn’t a security tool. If something is private, put it behind a login.
A robots.txt example
This one allows search crawlers and opts out of two training crawlers. It’s an example, and whether to block training is your decision.
- User-agent: GPTBotDisallow: /
- User-agent: ClaudeBotDisallow: /
- User-agent: *Allow: /
- Sitemap:https://example.com/sitemap.xml
What llms.txt does and doesn’t do
The llms.txt proposal describes a small file that fits in a model’s context window and links to more detailed content. Its site says it’s used mostly for inference, when an agent is looking something up to answer a question, and less for training.
It has found a real home in developer documentation. The docs for Next.js, OpenAI, Perplexity and Claude Code each point to an llms.txt index. That’s useful when a coding agent needs to find the right page. It tells us much less about marketing sites.
Google’s position is clear for its own products: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.” We cover the evidence in does llms.txt work.
An llms.txt example
Here’s the shape of an llms.txt file for an invented product. A generator can make a first draft from your sitemap, but a good file is curated, so edit it down by hand.
- # ExampleAppThe name of your site or product, as a heading.
- > ExampleApp helps small accounting firms chase clients for documents.A one-line summary, as a quote.
- ## DocsA section heading, followed by links such as: Getting started: set up your first client.
- ## OptionalLinks that an agent can skip if it’s short of space.
llms.txt vs robots.txt: what to do
Do this on every site
- Open your robots.txt and read it
- Check that your CDN or firewall isn’t blocking search crawlers
- Make sure your sitemap is listed
- Decide on training crawlers separately
Add llms.txt if
- You have API or developer docs
- It takes less than an hour
- You can keep it up to date
- You’re not paying extra for it or expecting traffic from it
Common questions
No. robots.txt sets access rules, and llms.txt points to content. An llms.txt file can’t block anything.
No. It’s a short text file you can write by hand. A generator can give you a start, but a long list of every URL defeats the point.
OpenAI says each setting is independent, so a site can allow OAI-SearchBot for search and block GPTBot from training.
We include llms.txt and a sitemap as part of launch work on our MVPs. We don’t claim llms.txt brings traffic.
Keep reading
Tell us what you’re building.
Takes under a minute. We come back with a scope and a fixed price.
- Reply within 24 hoursA real person reads every brief.
- Fixed scope, fixed priceYou get a written proposal, no hourly billing.
- You own the codeFull ownership of code and assets at launch.