llms.txt explained: what it is, whether you need one and how to create it

Key Takeaway

An llms.txt is a simple Markdown file in the root of your website that gives language models a curated overview of your key pages. It is free and takes about an hour to make. However, the major AI providers have not confirmed that they use it when generating answers. Treat it as a cheap extra on top of solid content and technical SEO, not as a replacement for either.

What is an llms.txt file?

An llms.txt is a Markdown file you place at yoursite.com/llms.txt. Jeremy Howard of Answer.AI proposed it in September 2024. The reasoning: language models have a limited context window and struggle with the clutter of a typical web page, such as menus, scripts and pop-ups. A compact file with a summary and your most important links makes it easier for a model to grasp what your site is about.

It sits in the same family as robots.txt and sitemaps, but does a different job. Robots.txt tells crawlers what they may fetch, and a sitemap lists every URL. An llms.txt is a curated selection with explanations, written for a model that has to interpret the content.

Do AI search engines actually use llms.txt?

The honest answer is that this has not been shown. Google has said more than once that it does not use the file for search results or AI Overviews, and compared it to the old keywords meta tag. OpenAI and Perplexity have not confirmed that their search systems read it by default either. What does happen: developer tools and some AI agents load an llms.txt from documentation sites when asked to, and several well-known documentation sites publish one.

For an ordinary business website the expected upside is therefore small. So is the cost, which makes it a reasonable item to tick off, as long as you do not expect it to visibly change your visibility in ChatGPT or Google. What does have support is covered in my article on generative engine optimization: direct answers, concrete figures and cited sources.

What does an llms.txt look like?

The format follows a fixed Markdown structure:

  1. An H1 with the name of your site or company (required).
  2. A short summary as a blockquote: who you are and what you do, in a few sentences.
  3. Optionally some extra explanation without headings.
  4. One or more H2 sections containing link lists, each link with a short description.
  5. A section titled “Optional” for pages a model can skip when space is tight.
# Example Company

> Example Company is an online shop for sustainable bike accessories and ships across Europe.

## Products
- [Panniers](https://example.com/panniers/): all bags with materials and capacity
- [Lights](https://example.com/lights/): front and rear lights

## Help
- [Shipping and returns](https://example.com/shipping/): delivery times and return policy

## Optional
- [About us](https://example.com/about/)

How do you create an llms.txt step by step?

  1. Pick your key pages. Aim for 10 to 30: services or products, pricing, FAQs and your best explanatory articles. Leave out anything short-lived.
  2. Write one factual sentence per link. No marketing language, just what the page contains.
  3. Save it as plain text named llms.txt and upload it to the root of your domain. Many WordPress SEO plugins can also generate the file for you.
  4. Check it is reachable at yoursite.com/llms.txt and not blocked by a noindex header or a firewall.
  5. Keep it current. A file that lists deleted pages does more harm than good. Add it to your regular technical SEO check.

What should you do first?

If you want to show up in AI answers, these steps usually pay off more than an llms.txt:

  • Make sure AI crawlers are not blocked by accident in your robots.txt or by your host.
  • Write pages that answer the core question directly, backed by figures and sources.
  • Add structured data so it is clear who is behind the content.
  • Measure whether visitors arrive from AI tools, as described in measuring AI traffic.

For the bigger picture, read getting found in ChatGPT and AI Overviews.

Need help with AI visibility?

I can set up an llms.txt for you, but mostly I help with the foundations around it: technical SEO, content and measurement. See what an SEO specialist can do for you or get in touch for a no-obligation chat.

Bouke van den Berg

Written by

Bouke van den Berg

SEO specialist and AI expert with a background in journalism and copywriting. Since 2014 I have been helping organisations become more visible online, and I now build automations with n8n and AI for business owners who could use their time back.