Understanding llms-full.txt: The Comprehensive Alternative to llms.txt
In brief: llms-full.txt is a proposed comprehensive version of llms.txt—while llms.txt contains only summary information, llms-full.txt includes complete documentation, API references, and detailed content. It is not yet an official standard, and there is no verified evidence that major AI providers use it. With no size constraints, it can generate large files; however, this doesn't guarantee that LLMs will actually read and process them.
The Emergence of the llms-full.txt Concept
When the llms.txt proposal emerged in 2024, the core idea was to provide LLMs with a summary of a site's key information. However, some developers and documentation sites realized that summaries might not be sufficient. An AI tool may need more detailed information based on user requests—for example, all API endpoints, parameter descriptions, or code examples.
This gave rise to the llms-full.txt concept: while llms.txt offers a brief guide about the site, llms-full.txt aggregates all content in plain text format. This allows an LLM or AI tool to access complete information from a single file when needed.
However, it's important to note: like llms.txt, llms-full.txt is not an official standard. There are no verified statements from major providers like OpenAI, Anthropic, Google, or Perplexity indicating they use this file for ranking, attribution, or training. Therefore, creating this file remains an experimental approach.
Key Differences Between llms.txt and llms-full.txt
| Feature | llms.txt | llms-full.txt |
|---|---|---|
| Purpose | Site summary, structure, main topics | Complete content in plain text (documentation, APIs, articles) |
| Size | Typically a few KB – a few hundred lines | Can reach tens of MB (depending on content volume) |
| Use case | Quick context, site map, general information | Detailed queries, code examples, complete reference |
| Update frequency | When site structure changes | Every time content updates (requires automation) |
In essence: llms.txt is like a table of contents, while llms-full.txt is the entire book consolidated into a single file.
Size and Format Challenges
File Size Considerations
The biggest challenge with llms-full.txt is size. A mid-sized documentation site with hundreds of pages can generate a 10–50 MB file when converted to plain text. For large e-commerce or content sites, this can exceed 100 MB.
The issue: most LLMs have limited context windows. For example, GPT-4 Turbo can handle 128,000 tokens (~96,000 words); Claude 3.5 Sonnet supports 200,000 tokens. However, this doesn't mean the entire file will be read—LLMs only retrieve the sections they need, or may not read the file at all if it's too large.
Format and Structure
llms-full.txt should be plain text. Markdown is acceptable, but structured formats like HTML, JSON, or XML are not recommended—because LLMs will read this file directly, not parse it.
Example structure:
# Product Documentation - Complete Version
## API Reference
### GET /api/users
Lists all users.
Parameters:
- limit (integer): Maximum number of results
- offset (integer): Starting point
Example response:
{
"users": [...]
}
### POST /api/users
Creates a new user.
...
## User Guide
### Installation
...
### Configuration
...
Each section should be clearly titled, with code examples and explanations presented clearly. However, if the file is too large, you cannot control which sections the LLM will read.
Updates and Automation
Manually updating llms-full.txt is impractical. You need a script that regenerates the file whenever content changes. For example:
- If your documentation site consists of Markdown files, a build script that concatenates all .md files
- For CMS-based sites, a cron job that pulls all content via API and converts it to plain text
- For static site generators (Hugo, Jekyll, Docusaurus), a custom plugin or hook
Otherwise, the file quickly becomes outdated and contains incorrect information.
When Does llms-full.txt Make Sense?
llms-full.txt isn't necessary for every site. Consider it in these scenarios:
- Comprehensive API documentation: With hundreds of endpoints, parameters, and examples, summarizing everything in llms.txt is difficult. llms-full.txt enables an AI tool to scan the entire API.
- Technical libraries or frameworks: Projects where developers frequently ask detailed questions (e.g., React, Django, Kubernetes documentation). However, these sites typically already have search and structured data; llms-full.txt would be an additional experiment.
- Internal knowledge bases: Consolidating all procedures, policies, and technical documents in a single file for internal AI assistants. Here, file size is more controllable and may fit within the LLM's context.
- Research and archival projects: If you want to make all content available to LLMs and size isn't an issue.
However, it doesn't make sense in these situations:
- If your site has thousands of blog posts, product pages, or user-generated content—the file becomes too large and LLMs cannot process it efficiently.
- If content changes frequently and you lack automation—the file must be kept constantly updated.
- If you already have structured data (schema.org), sitemaps, and a robust search API—LLMs can use these and may not need a separate file.
Do Major AI Providers Use llms-full.txt?
No, at least not as of early 2025—there is no verified usage. Companies like OpenAI, Anthropic, Google, and Perplexity have not officially announced support for llms.txt or llms-full.txt. These files are experimental approaches proposed by the community.
Therefore, there's no guarantee that creating llms-full.txt will result in better ranking or citations in ChatGPT or Gemini. However, some specialized AI tools (such as internal RAG systems) may use this file—in which case it could be beneficial.
Practical Recommendations
- Start with llms.txt first: Summarize your site's structure and main topics. This is sufficient for most use cases.
- Add llms-full.txt only when necessary: If you have detailed documentation or API references and want to make them available to LLMs.
- Control the size: If the file exceeds 10 MB, consider splitting it into sections (e.g., llms-full-api.txt, llms-full-guides.txt).
- Set up automation: Write a script that regenerates the file whenever content changes.
- Keep expectations realistic: This file may help AI tools better understand your site, but it doesn't guarantee ranking or visibility improvements.
Frequently Asked Questions
How large can an llms-full.txt file be?
There's no technical upper limit, but practically speaking, anything over 50 MB won't fit into most LLMs' context windows. If the file is too large, AI tools will only retrieve relevant sections rather than reading it entirely, or may not read it at all. If your file exceeds 10 MB, consider splitting it into sections or including only critical content.
Should I block llms-full.txt in robots.txt?
No, you actually want to keep this file accessible to LLMs. However, if the file is very large and creates server load, you can implement rate limiting. Blocking it in robots.txt defeats the file's purpose.
What's the difference between llms-full.txt and sitemap.xml?
sitemap.xml tells search engines which URLs exist; it lists links, not content. llms-full.txt provides the content itself in plain text format. Sitemaps are for crawling; llms-full.txt is designed for LLMs to read and process directly. They complement each other but don't replace one another.
Who should use llms-full.txt?
Sites offering comprehensive API documentation, technical library or framework documentation, and teams building knowledge bases for internal AI assistants. It's generally impractical for general blog sites, e-commerce platforms, or sites with frequently changing content. Start with llms.txt; only consider llms-full.txt if you have a specific need to provide detailed content.