Does llms.txt Actually Work? What We Know About This Experimental AI Standard
Executive Summary: llms.txt is a community-proposed file format from 2024 designed to provide LLMs with summary information about a website. It is not yet an official standard, and there is no verified evidence that major search or AI providers use it for ranking, attribution, or grounding. Since implementation cost is minimal, it can be added experimentally, but expectations of measurable impact should remain realistic.
What Is llms.txt and Where Did It Come From?
llms.txt is a plain text file placed in a website's root directory. Its purpose is to provide large language models (LLMs) with summarized information about the site's structure, content, and key pages. The concept emerged in 2024 from independent developers and the AI community—think of it as a robots.txt adapted for the LLM era.
The underlying rationale is straightforward: when LLMs crawl a site or retrieve content in response to user queries, they could read the llms.txt file to understand which sections are important, which pages serve as authoritative sources, and the overall content hierarchy. However, this remains a theoretical approach, and its practical effectiveness is unclear.
What Have Major Providers Said?
To date, no major AI or search provider has publicly confirmed official support for llms.txt:
- OpenAI: Documented compliance with robots.txt rules for GPTBot and OAI-SearchBot exists, but there is no official documentation indicating they read or process llms.txt.
- Google: Uses robots.txt and meta tags (nosnippet, max-snippet) for Google-Extended and Googlebot. No confirmation that llms.txt is utilized by Gemini or AI Overviews.
- Anthropic: ClaudeBot respects robots.txt; no official statement has been made regarding llms.txt support.
- Perplexity: PerplexityBot and user-initiated fetching follow robots.txt rules; no official information exists about llms.txt.
In short, llms.txt currently exists as a community proposal, and there is no evidence that major platforms actively process it. Some small-scale AI tools or experimental projects may read this file, but widespread adoption has not occurred.
How to Create an llms.txt File
The file is placed in your site's root directory as llms.txt in plain text format. The recommended structure is as follows:
# Site Name
https://example.com
## About
This site produces content about B2B software solutions.
## Key Pages
- Home: https://example.com
- Products: https://example.com/products
- Blog: https://example.com/blog
- Contact: https://example.com/contact
## Content Categories
- SEO Guides: https://example.com/blog/seo
- Technical Documentation: https://example.com/docs
- Case Studies: https://example.com/case-studies
The file uses Markdown-like syntax with a simple structure that LLMs can easily parse. It summarizes key page URLs, content categories, and the site's overall purpose.
Why Consider Adding It Anyway?
While llms.txt effectiveness remains uncertain, there are practical reasons to implement it:
- Low cost: Creation takes minutes and adds no server resource burden. It's a static text file.
- Future compatibility: If major providers adopt this format later, your file will already be in place.
- Experimental data: By adding it and monitoring your log files, you can observe which bots access this file. This provides data for future strategy.
- Internal documentation: It can serve as a simple reference summarizing your site structure for your team.
However, remember this: adding llms.txt does not guarantee appearance in AI Overviews, ChatGPT citations, or Perplexity results. This file is currently an experimental tool, and expectations of measurable SEO or GEO impact are unrealistic.
Alternative and Complementary Methods
Instead of or alongside llms.txt, we recommend using verified methods:
| Method | Purpose | Verified Support |
|---|---|---|
| robots.txt (GPTBot, ClaudeBot, etc.) | Control crawling and training access | Yes (OpenAI, Anthropic, Google) |
| Google-Extended | Control Gemini and Vertex AI training | Yes (Google) |
| nosnippet, max-snippet, data-nosnippet | Limit AI Overviews snippet usage | Yes (Google) |
| Structured data (Schema.org) | Clarify content meaning | Yes (Google, Bing, others) |
| llms.txt | Provide site summary to LLMs | No (experimental) |
Structured data (Article, FAQPage, HowTo, etc.) and clean HTML semantics are proven methods for AI systems to understand your content. Even if you add llms.txt, do not neglect these foundational practices.
Setting Realistic Expectations
After adding llms.txt, do not expect:
- More frequent ChatGPT citations
- Sudden appearance in Google AI Overviews
- Higher rankings in Perplexity results
- Measurable increases in organic traffic
Instead, treat llms.txt as a low-risk experiment. Monitor /llms.txt access in your server logs; if known AI bots regularly fetch this file, it may signal a future role. However, currently, the presence or absence of this file does not directly affect your visibility in AI systems.
Frequently Asked Questions
Can adding llms.txt harm SEO?
No. llms.txt is a static text file and does not affect Google's ranking algorithms. Unless blocked in robots.txt, the file is crawlable, but indexing or inclusion in rankings is not a concern. The only potential risk is sharing sensitive information in the file; summarize general site structure and avoid internal system details.
Which AI bots are reading llms.txt?
Currently, no major providers (OpenAI, Google, Anthropic, Perplexity) have confirmed officially reading llms.txt. Some small-scale or experimental AI tools may parse this file, but widespread usage does not exist. You can monitor GET /llms.txt requests in your server logs to see which user-agents are accessing it.
What's the difference between llms.txt and robots.txt?
robots.txt is an access protocol that controls which pages bots can access and is supported by major providers. llms.txt is a community proposal aimed at providing LLMs with summary site information and has not yet received official support. robots.txt is mandatory and functional; llms.txt is optional and experimental.
Who should update llms.txt?
Update llms.txt when your site structure changes (new major sections, important content categories added). However, frequent updates are unnecessary; reviewing once or twice annually is sufficient. Think of the file as a general site map; you don't need to add every blog post—just list main sections and permanent pages.