What is llms.txt, and does your website need one?

If you have read about optimizing for AI search, you have probably seen advice to "add an llms.txt file". Some of that advice treats it as a requirement, which it isn't. This guide explains what the file is, what it can and can't do, and how to decide whether it's worth the few minutes it takes.

What is llms.txt?

llms.txt is a Markdown file placed at /llms.txt on a website that summarizes the site for large language models and lists links to its most important pages. The idea is to give an AI tool a clean, short map, instead of making it work through navigation menus, scripts and ads.

It was proposed in September 2024 by Jeremy Howard, co-founder of Answer.AI, and the proposal is documented at llmstxt.org. It is a proposal that anyone can adopt, not a standard, and it is mostly aimed at the moment a user asks an AI tool about a site, rather than at training.

What does an llms.txt file look like?

It is ordinary Markdown in a fixed order: an H1 title, a short blockquote summary, optional notes, then sections of links under H2 headings.

# Example Bakery

> Example Bakery is a family bakery in Leeds selling sourdough,
> pastries and celebration cakes, with click-and-collect ordering.

Orders placed before 2 pm are ready the next morning.

## Main pages

- [Menu and prices](https://example.com/menu): full menu with current prices
- [Order online](https://example.com/order): click-and-collect ordering
- [Celebration cakes](https://example.com/cakes): sizes, flavours and lead times

## Help

- [Opening hours and location](https://example.com/visit): address, parking, hours
- [Allergens](https://example.com/allergens): allergen information per product

## Optional

- [Our story](https://example.com/about): how the bakery started

• H1 title: the name of the site or project. In the proposal, it's the only required part.
• Blockquote: one or two lines that say what the site is, with the key facts someone would need.
• Notes: optional paragraphs or lists with extra context, without headings.
• H2 sections: lists of links, each written as [name](url) and optionally followed by a colon and a short note.
• "Optional" section: links a tool can skip when it needs a shorter context.

The proposal also suggests offering clean Markdown versions of pages at the same URL with .md added. That part is less widely adopted. For a real example, see the file Join Postly publishes at joinpostly.com/llms.txt.

What is llms-full.txt?

llms-full.txt is a companion convention: one file containing the full text of a site's key pages in Markdown, so a tool can load everything at once. It became popular with documentation sites. It isn't part of the core format above, and it only makes sense when the content is stable and not too large.

How is llms.txt different from robots.txt?

robots.txt tells crawlers what they may fetch; llms.txt suggests what is worth reading. One is about permission, the other about guidance.

If you want to control AI crawlers, robots.txt is still the tool. llms.txt can't block anything.

Do search engines and AI tools use llms.txt?

At the time of writing, there is no public commitment from a major search engine to use it. Google representatives have said publicly that Google Search doesn't use llms.txt, and we are not aware of any major AI search product that documents relying on it to choose or cite sources.

Where it does get used is closer to the user. Developer tools and AI assistants can fetch a site's llms.txt when a person points them at it, and many software documentation sites publish one for that reason. That's useful, but it is a different thing from ranking.

Should your website have one?

If you can keep it accurate, adding one is cheap and low-risk. Just don't expect it to change your search traffic.

Reasons to add one

• It takes little time for a small site and costs nothing to host.
• Writing the summary forces you to say clearly what the site is, which helps your other pages too.
• Tools that do read it get an accurate, current summary in your own words.
• Documentation, API and software sites benefit most, because developers use AI assistants with docs.

Reasons not to bother (yet)

• No major search engine has confirmed using it, so it isn't a ranking factor.
• An outdated file with old prices or dead links is worse than none.
• It duplicates work your sitemap, navigation and clear page content already do.

How do you add llms.txt to your site?

1. Write the file in a text editor using the format above. Keep the summary factual: what you do, for whom, where.
2. Link to 10 to 30 pages people actually need, each with a one-line note.
3. Upload it to the root of your domain, so it opens at https://yoursite.com/llms.txt.
4. Open the URL in a browser. You should see plain text, not your site's HTML page or a 404.
5. Put a reminder in your calendar to update it when prices, products or key pages change.

How does Postly Rank check llms.txt?

The Postly Rank Site Audit checks llms.txt as part of its AI Search Readiness score, which is kept separate from the main site score.

• Validator: fetches /llms.txt and checks that it is Markdown rather than an HTML page, starts with one H1, has a blockquote summary, groups links under H2 sections and uses valid http or https links. It tests up to 20 of the links and checks whether /llms-full.txt exists.
• Generator: if you don't have a file, it drafts one from the crawl: your site name, a summary from your homepage, and your key pages grouped into sections, with legal pages moved to "Optional". Edit the draft before you publish it.
• AI crawler access: it reads your robots.txt rules for crawlers such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended, and warns only if every search-type AI crawler is blocked.

These checks are on every Postly Rank plan, including Free. More detail is on the AI search readiness page.