Why AI Chatbots Couldn't Find Our Blog Posts (But Google Could)
How We Discovered the Problem: ChatGPT Didn't Know Our Blog Existed
Last month we noticed something weird. When we asked ChatGPT "What has AICompatible.com written about AI bot crawling behavior?" it didn't mention a single one of our blog posts. Yet when we checked Google Search Console, our blog pages showed as properly indexed and were getting organic traffic.
Perplexity showed the same pattern. When we asked questions about our site, it would quote from our homepage and static pages, but acted like our blog posts didn't exist. This strange inconsistency sent us down a debugging rabbit hole.
The Diagnosis: Bots Aren't Running JavaScript
First, we looked at the source code of our blog pages. Right-click → "View Page Source" in Chrome showed us this:
<div id="blog-container">
<!-- Content will load here -->
</div>
<script src="posts.js"></script>
Our entire blog was client-side rendered with JavaScript. The posts.js file pulled posts from a JSON structure and injected them into the DOM. The actual HTML source contained zero content.
We knew Googlebot could handle this—Google has been rendering JavaScript since 2015. But what about other bots? We tested with curl:
curl -A "GPTBot/1.0" https://aicompatible.com/blog/bot-crawling-guide
The returned HTML contained just an empty container. Most LLM bots—GPTBot, ClaudeBot, PerplexityBot—don't execute JavaScript. To them, our pages were literally empty.
robots.txt and Log Analysis
When we dug into our server logs, here's what we found:
- Googlebot: Visits the page, then fetches posts.js (proof of JS rendering)
- GPTBot: Only fetches the HTML page, never touches any JS files
- ClaudeBot: Same behavior—single request, no JS
- PerplexityBot: Similar pattern
The issue was clear: while modern search bots can render JS, LLM training and retrieval bots don't. This actually makes sense—when you're crawling billions of pages for training data, spinning up a headless browser for each one is prohibitively expensive.
The Fix: Generate Static HTML Pages
The solution was simple but effective: embed the content directly in the HTML for each blog post. We wrote a basic Node.js script to do this:
// generate-blog-pages.js
const fs = require('fs');
const posts = require('./posts.json');
posts.forEach(post => {
const html = `
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>${post.title} - AICompatible.com</title>
<meta name="description" content="${post.excerpt}">
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "BlogPosting",
"headline": "${post.title}",
"description": "${post.excerpt}",
"author": {
"@type": "Organization",
"name": "AICompatible.com"
},
"datePublished": "${post.date}",
"dateModified": "${post.modified || post.date}",
"publisher": {
"@type": "Organization",
"name": "AICompatible.com",
"logo": {
"@type": "ImageObject",
"url": "https://aicompatible.com/logo.png"
}
}
}
</script>
</head>
<body>
<article>
<h1>${post.title}</h1>
<time datetime="${post.date}">${post.date}</time>
${post.content}
</article>
</body>
</html>
`;
fs.writeFileSync(`./blog/${post.slug}.html`, html);
});
We run this script whenever we publish a new post or update existing content. Now every blog page has real HTML content in the source.
Adding BlogPosting Schema
As you can see in the example above, we added BlogPosting schema to each page in application/ld+json format. This helps Google show rich results and provides structured data to LLM bots.
Key fields:
- headline: The post title
- datePublished/dateModified: Date information (LLMs use this to evaluate content freshness)
- author and publisher: Clarifies the content source
Results and Lessons Learned
Two weeks after making the change, we asked ChatGPT the same question again. This time it quoted directly from our blog posts and cited them as sources. We saw similar improvements in Perplexity.
Key takeaways from this experience:
- Googlebot ≠All Bots: Being visible on Google doesn't mean LLMs can see you
- JavaScript rendering is optional: You can enhance with JS, but core content needs to be in the HTML
- Structured data is critical: Schema.org markup makes it easier for bots to understand your content
Check Your Own Site
Testing for this issue is straightforward:
- Right-click on your blog page → "View Page Source"
- Can you see your actual content in the source code?
- Or do you just see empty divs and script tags?
If your content isn't in the source, LLM bots probably can't see you.
AICompatible.com's site crawler automatically detects these issues. When we scan a page, we compare both the raw HTML and the rendered content. If significant content only loads via JS, we flag it in the report as "invisible to LLM bots." We also detect missing or incorrect schema.org markup and provide fix recommendations.
Bottom line: modern web development practices sometimes neglect backward compatibility. In the LLM era, "bot accessibility" has become a new optimization dimension. Make sure your content is accessible to both humans and all types of bots.