How to Dynamically Generate llms.txt in Next.js, WordPress, and Shopify: A Practical Developer's Guide
Why Dynamic Generation of llms.txt Files Is Essential for Growing Platforms
BLUF (Bottom Line Up Front): Static llms.txt files become obsolete within hours on high-velocity content platforms, creating a critical gap between what AI agents discover and what actually exists. Dynamic generation ensures real-time accuracy, eliminates manual maintenance overhead, and guarantees that large language models always receive current site structure, content hierarchies, and resource locations—making it indispensable for enterprise WordPress installations, Next.js applications, and Shopify stores publishing dozens of pages daily.
Understanding the llms.txt Specification and Dynamic Requirements
The llms.txt file serves as a machine-readable manifest that guides AI crawlers, language models, and autonomous agents through your site's information architecture. Unlike robots.txt, which focuses on crawl permissions, llms.txt provides semantic structure, content categorization, and priority signals specifically optimized for LLM consumption.
Dynamic generation becomes critical when:
- Content velocity exceeds manual update capacity: E-commerce catalogs adding 50+ products daily, news sites publishing hourly, or SaaS documentation updating with each release cycle
- Taxonomies evolve programmatically: Auto-generated category pages, dynamic filtering systems, or user-generated content hierarchies
- Personalization layers exist: Geographic variants, language-specific routes, or authentication-gated resources requiring conditional exposure
- Multi-source content aggregation: Headless CMS architectures, microservices feeding content APIs, or federated data sources
WordPress Dynamic llms.txt Implementation
WordPress powers 43% of the web, making its llms.txt implementation patterns critically important. The following approach uses the template_redirect action to intercept requests before theme rendering, ensuring maximum performance and compatibility with caching layers.
Complete WordPress PHP Implementation
<?php
/**
* Dynamic llms.txt Generator for WordPress
* Add to theme's functions.php or create as a plugin
*/
add_action('template_redirect', 'serve_dynamic_llms_txt');
function serve_dynamic_llms_txt() {
// Only respond to /llms.txt requests
if ($_SERVER['REQUEST_URI'] !== '/llms.txt') {
return;
}
// Set appropriate headers
header('Content-Type: text/plain; charset=utf-8');
header('X-Robots-Tag: noindex');
// Prevent caching for truly dynamic content
header('Cache-Control: no-cache, must-revalidate');
// Start output
echo "# llms.txt - Dynamically Generated\n";
echo "# Generated: " . current_time('c') . "\n\n";
// Site metadata
echo "# Site: " . get_bloginfo('name') . "\n";
echo "# Description: " . get_bloginfo('description') . "\n";
echo "# URL: " . home_url() . "\n\n";
// Main pages
echo "## Primary Navigation\n\n";
$pages = get_pages(array('sort_column' => 'menu_order', 'hierarchical' => 0));
foreach ($pages as $page) {
echo "- [{$page->post_title}](" . get_permalink($page->ID) . ")\n";
}
echo "\n";
// Blog posts by category
echo "## Blog Content\n\n";
$categories = get_categories(array('hide_empty' => true, 'orderby' => 'count', 'order' => 'DESC'));
foreach ($categories as $category) {
echo "### {$category->name} ({$category->count} posts)\n\n";
$posts = get_posts(array(
'category' => $category->term_id,
'numberposts' => 10,
'orderby' => 'date',
'order' => 'DESC'
));
foreach ($posts as $post) {
$date = get_the_date('Y-m-d', $post->ID);
echo "- [{$post->post_title}](" . get_permalink($post->ID) . ") - {$date}\n";
}
echo "\n";
}
// Product integration for WooCommerce
if (class_exists('WooCommerce')) {
echo "## Product Catalog\n\n";
$product_categories = get_terms(array(
'taxonomy' => 'product_cat',
'hide_empty' => true,
'orderby' => 'count',
'order' => 'DESC',
'number' => 10
));
foreach ($product_categories as $cat) {
echo "- [{$cat->name}](" . get_term_link($cat) . ") - {$cat->count} products\n";
}
echo "\n";
}
// Documentation or custom post types
$custom_post_types = get_post_types(array('public' => true, '_builtin' => false), 'objects');
foreach ($custom_post_types as $cpt) {
echo "## {$cpt->labels->name}\n\n";
$cpt_posts = get_posts(array(
'post_type' => $cpt->name,
'numberposts' => 20,
'orderby' => 'date',
'order' => 'DESC'
));
foreach ($cpt_posts as $post) {
echo "- [{$post->post_title}](" . get_permalink($post->ID) . ")\n";
}
echo "\n";
}
exit; // Prevent WordPress from continuing execution
}
WordPress Performance Optimization Strategies
For high-traffic WordPress installations, implement these caching patterns:
- Transient API caching: Store generated output in
set_transient('llms_txt_cache', $output, 3600)with automatic invalidation on post publish hooks - Object cache integration: Leverage Redis or Memcached for distributed environments
- CDN edge caching: Set appropriate
Cache-Controlheaders (e.g.,max-age=1800) for Cloudflare or Fastly edge nodes - Selective regeneration: Hook into
save_post,created_term, anddeleted_termactions to invalidate cache only when content changes
Next.js Route Handler Implementation (App Router)
Next.js 13+ App Router provides native streaming capabilities ideal for large llms.txt files. The following implementation uses Route Handlers with TypeScript for type safety and edge runtime compatibility.
Next.js TypeScript Implementation
// app/llms.txt/route.ts
import { NextRequest, NextResponse } from 'next/server';
// Optional: Enable Edge Runtime for global distribution
// export const runtime = 'edge';
export async function GET(request: NextRequest) {
const encoder = new TextEncoder();
const stream = new ReadableStream({
async start(controller) {
// Helper function to write chunks
const write = (text: string) => {
controller.enqueue(encoder.encode(text));
};
// Header section
write('# llms.txt - Dynamic Site Map\n');
write(`# Generated: ${new Date().toISOString()}\n`);
write(`# Base URL: ${process.env.NEXT_PUBLIC_SITE_URL}\n\n`);
// Fetch data from your CMS/database
try {
// Example: Fetch from headless CMS
const pages = await fetch(`${process.env.CMS_API_URL}/pages`, {
headers: { 'Authorization': `Bearer ${process.env.CMS_API_KEY}` },
next: { revalidate: 3600 } // ISR-style caching
}).then(res => res.json());
write('## Main Pages\n\n');
pages.forEach((page: any) => {
write(`- [${page.title}](${process.env.NEXT_PUBLIC_SITE_URL}${page.slug})\n`);
});
write('\n');
// Blog posts with pagination handling
write('## Blog Articles\n\n');
let page = 1;
let hasMore = true;
while (hasMore && page <= 10) { // Limit to prevent infinite loops
const posts = await fetch(
`${process.env.CMS_API_URL}/posts?page=${page}&per_page=50`,
{ next: { revalidate: 1800 } }
).then(res => res.json());
if (posts.length === 0) {
hasMore = false;
break;
}
posts.forEach((post: any) => {
const date = new Date(post.publishedAt).toISOString().split('T')[0];
write(`- [${post.title}](${process.env.NEXT_PUBLIC_SITE_URL}/blog/${post.slug}) - ${date}\n`);
});
page++;
}
write('\n');
// Product catalog (if applicable)
const products = await fetch(`${process.env.CMS_API_URL}/products?limit=100`)
.then(res => res.json())
.catch(() => []);
if (products.length > 0) {
write('## Products\n\n');
// Group by category
const categorized = products.reduce((acc: any, product: any) => {
const cat = product.category || 'Uncategorized';
if (!acc[cat]) acc[cat] = [];
acc[cat].push(product);
return acc;
}, {});
Object.entries(categorized).forEach(([category, items]: [string, any]) => {
write(`### ${category}\n\n`);
items.forEach((product: any) => {
write(`- [${product.name}](${process.env.NEXT_PUBLIC_SITE_URL}/products/${product.slug})\n`);
});
write('\n');
});
}
// API documentation routes
write('## API Documentation\n\n');
write(`- [API Reference](${process.env.NEXT_PUBLIC_SITE_URL}/docs/api)\n`);
write(`- [Authentication Guide](${process.env.NEXT_PUBLIC_SITE_URL}/docs/auth)\n`);
write(`- [Rate Limits](${process.env.NEXT_PUBLIC_SITE_URL}/docs/limits)\n\n`);
} catch (error) {
write(`# Error generating dynamic content: ${error}\n`);
}
controller.close();
}
});
return new NextResponse(stream, {
headers: {
'Content-Type': 'text/plain; charset=utf-8',
'Cache-Control': 'public, s-maxage=1800, stale-while-revalidate=3600',
'X-Robots-Tag': 'noindex',
},
});
}
Next.js Advanced Patterns
Incremental Static Regeneration (ISR) Integration: Combine Route Handlers with ISR by setting revalidate values in fetch calls, allowing edge caching while maintaining freshness guarantees.
Parallel Data Fetching: Use Promise.all() to fetch multiple content sources simultaneously, reducing total generation time:
const [pages, posts, products] = await Promise.all([
fetchPages(),
fetchPosts(),
fetchProducts()
]);
Conditional Sections: Implement feature flags or environment-based conditionals to expose different content structures for staging vs. production environments.
Shopify Dynamic llms.txt Implementation
Shopify's architecture requires different approaches depending on your store setup. The platform's Liquid templating engine and theme structure present unique challenges and opportunities.
Method 1: Custom Liquid Template
Create a new page template in your theme:
{% comment %}
File: templates/page.llms.liquid
Create a page in Shopify admin with template suffix "llms"
Access via: yourstore.com/pages/llms-txt
{% endcomment %}
{% layout none %}
{% content_for "content_type" %}text/plain{% endcontent_for %}
# llms.txt - {{ shop.name }}
# Generated: {{ "now" | date: "%Y-%m-%d %H:%M:%S %Z" }}
# Store URL: {{ shop.url }}
## Collections
{% for collection in collections %}
{% if collection.products_count > 0 %}
### {{ collection.title }} ({{ collection.products_count }} products)
{% for product in collection.products limit: 20 %}
- [{{ product.title }}]({{ shop.url }}{{ product.url }}) - {{ product.price | money }}
{% endfor %}
{% endif %}
{% endfor %}
## Blog Articles
{% for article in blogs.news.articles %}
- [{{ article.title }}]({{ shop.url }}{{ article.url }}) - {{ article.published_at | date: "%Y-%m-%d" }}
{% endfor %}
## Pages
{% for page in pages %}
- [{{ page.title }}]({{ shop.url }}{{ page.url }})
{% endfor %}
Method 2: Shopify App Proxy
For enterprise Shopify Plus stores, implement an app proxy that generates llms.txt server-side:
- Configure App Proxy: In your Shopify app settings, set proxy path to
/apps/llmspointing to your server endpoint - Server Implementation: Build a Node.js/Python/Ruby endpoint that uses Shopify Admin API to fetch current inventory, collections, and content
- URL Rewriting: Use Shopify Scripts or theme modifications to redirect
/llms.txtto/apps/llms/generate
// Express.js app proxy handler
app.get('/generate', async (req, res) => {
const shopifyClient = new Shopify.Clients.Rest(
req.query.shop,
process.env.SHOPIFY_ACCESS_TOKEN
);
res.setHeader('Content-Type', 'text/plain');
// Fetch collections
const collections = await shopifyClient.get({
path: 'custom_collections',
});
let output = '# llms.txt\n\n## Collections\n\n';
for (const collection of collections.body.custom_collections) {
const products = await shopifyClient.get({
path: `collections/${collection.id}/products`,
query: { limit: 50 }
});
output += `### ${collection.title}\n\n`;
products.body.products.forEach(product => {
output += `- [${product.title}](https://${req.query.shop}/products/${product.handle})\n`;
});
output += '\n';
}
res.send(output);
});
Method 3: Shopify Hydrogen (Headless)
For Hydrogen storefronts, implement a server route similar to Next.js:
// app/routes/llms[.]txt.tsx
import { LoaderFunction } from '@shopify/remix-oxygen';
export const loader: LoaderFunction = async ({ context }) => {
const { storefront } = context;
const { collections } = await storefront.query(`
query LLMSData {
collections(first: 50) {
nodes {
title
handle
products(first: 20) {
nodes {
title
handle
}
}
}
}
}
`);
let output = '# llms.txt\n\n';
collections.nodes.forEach(collection => {
output += `## ${collection.title}\n\n`;
collection.products.nodes.forEach(product => {
output += `- [${product.title}](/products/${product.handle})\n`;
});
output += '\n';
});
return new Response(output, {
headers: {
'Content-Type': 'text/plain',
'Cache-Control': 'public, max-age=3600'
}
});
};
Validation and Testing Methodologies
Ensuring your dynamically generated llms.txt file meets specification requirements and performs optimally requires systematic validation.
Format Validation Checklist
- Markdown compliance: Verify proper heading hierarchy (H1 for title, H2 for sections, H3 for subsections)
- Link integrity: All URLs must be absolute, properly encoded, and return 200 status codes
- Character encoding: UTF-8 encoding with proper handling of special characters, emojis, and international text
- File size considerations: Keep under 10MB for optimal LLM processing; implement pagination or summarization for larger sites
- Update frequency indicators: Include generation timestamps and change frequency hints
Automated Testing Script
import requests
import re
from urllib.parse import urlparse
def validate_llms_txt(url):
"""Validate llms.txt format and content"""
response = requests.get(url)
# Check response headers
assert response.status_code == 200, "File not accessible"
assert 'text/plain' in response.headers.get('Content-Type', ''), "Wrong content type"
content = response.text
lines = content.split('\n')
# Validate structure
assert lines[0].startswith('#'), "Must start with title"
# Extract and validate all URLs
url_pattern = r'\[([^\]]+)\]\(([^\)]+)\)'
urls = re.findall(url_pattern, content)
print(f"Found {len(urls)} links")
# Sample validation (check first 10 links)
for title, link in urls[:10]:
parsed = urlparse(link)
assert parsed.scheme in ['http', 'https'], f"Invalid scheme: {link}"
assert parsed.netloc, f"Missing domain: {link}"
# Optional: Check if link is accessible
try:
link_response = requests.head(link, timeout=5, allow_redirects=True)
assert link_response.status_code < 400, f"Broken link: {link}"
except requests.RequestException as e:
print(f"Warning: Could not validate {link}: {e}")
# Check for required sections
assert '##' in content, "Missing section headers"
# Validate file size
size_mb = len(content.encode('utf-8')) / (1024 * 1024)
assert size_mb < 10, f"File too large: {size_mb:.2f}MB"
print("✓ Validation passed")
return True
# Usage
validate_llms_txt('https://yoursite.com/llms.txt')
Performance Monitoring
Implement these monitoring strategies to ensure dynamic generation doesn't impact site performance:
- Response time tracking: Set alerts for generation times exceeding 2 seconds
- Cache hit rate monitoring: Track percentage of cached vs. regenerated responses
- Error rate logging: Monitor failed database queries or API calls during generation
- Resource utilization: Measure CPU and memory usage during peak generation periods
Advanced Optimization Techniques
Conditional Content Exposure
Implement intelligent filtering based on user agent, geographic location, or authentication status:
// Next.js example with conditional sections
export async function GET(request: NextRequest) {
const userAgent = request.headers.get('user-agent') || '';
const isGoogleBot = userAgent.includes('Googlebot');
const isLLMCrawler = userAgent.includes('GPTBot') || userAgent.includes('Claude');
// Expose different content depths based on crawler type
const maxItems = isLLMCrawler ? 1000 : isGoogleBot ? 500 : 100;
// Generate content with appropriate limits
}
Hierarchical Prioritization
Structure content to present high-value pages first, using priority indicators:
## High Priority Content
- [Product Launch 2024](/launch) - Priority: High, Updated: 2024-01-15
- [Documentation Home](/docs) - Priority: High, Updated: 2024-01-10
## Standard Content
- [Blog Archive](/blog) - Priority: Medium
Multi-Language Support
For international sites, generate language-specific llms.txt variants:
// WordPress multi-language example
function serve_dynamic_llms_txt() {
$lang = isset($_GET['lang']) ? sanitize_text_field($_GET['lang']) : 'en';
if ($lang !== 'en') {
// Generate localized version
$posts = get_posts(array(
'lang' => $lang,
'numberposts' => 50
));
}
echo "# llms.txt ({$lang})\n\n";
// ... rest of generation
}
Security Considerations
Dynamic generation introduces potential security vectors that must be addressed:
- Rate limiting: Implement request throttling to prevent resource exhaustion attacks (e.g., 60 requests per IP per hour)
- Input sanitization: Validate and escape all dynamic content to prevent injection attacks
- Authentication bypass prevention: Never expose private content URLs in llms.txt, even if the pages themselves are protected
- Information disclosure: Avoid revealing internal system paths, API endpoints, or sensitive metadata
- DDoS mitigation: Use CDN-level protection and implement circuit breakers for upstream service failures
Maintenance and Monitoring Best Practices
Establish operational procedures to ensure long-term reliability:
- Automated testing in CI/CD: Include llms.txt validation in deployment pipelines
- Version control for templates: Track changes to generation logic with detailed commit messages
- Alerting thresholds: Set up notifications for generation failures, performance degradation, or format violations
- Regular audits: Monthly reviews of included content, broken links, and structural accuracy
- Documentation: Maintain runbooks for troubleshooting common issues and updating generation logic
Conclusion
Dynamic llms.txt generation transforms a static file into a living document that accurately represents your site's current state. By implementing platform-specific solutions for WordPress, Next.js, and Shopify, you ensure AI agents always receive authoritative, up-to-date information about your content architecture. The investment in dynamic generation pays dividends through improved AI discoverability, reduced maintenance burden, and enhanced semantic understanding of your digital properties. As LLM-powered search and discovery tools become increasingly prevalent, dynamic llms.txt files will transition from competitive advantage to baseline requirement for serious web properties.