Why AI Chatbots Couldn't See Our Blog Posts (But Google Could)
Discovering the Problem: We Didn't Exist in Perplexity
Last month during a routine check, we asked Perplexity AI: "How do you test AI compatibility in Turkey?" It summarized content from competitor sites, but there wasn't a single mention of our detailed blog posts. Yet those same articles were getting steady traffic in Google Search Console. Here's what was weird: when we asked ChatGPT the same question, no mention of us. Claude? Same thing.
At first we thought, "maybe we haven't built enough authority yet." But when we looked at Search Console data, our posts were ranking on pages 2-3 and getting clicks. So Google was clearly seeing and indexing the content, but AI bots weren't. This inconsistency pushed us to dig deeper.
Diagnosis: The JavaScript Rendering Gap
To understand the issue, we looked at the source code of one of our blog pages. When we hit "View Source" in the browser, here's what we saw:
<div id="blog-container">
<!-- Content will be loaded via JavaScript -->
</div>
<script src="/js/posts.js"></script>The page was completely empty. All our blog content was being fetched from posts.js and written to the DOM with JavaScript. The page looked perfect in the browser because modern browsers execute JS. But in the raw HTML? Nothing.
To check Googlebot's situation, we used the "URL Inspection" tool in Search Console. When we ran a "Live Test," we confirmed that Googlebot was seeing the rendered page—it was executing JavaScript. Google has been rendering JS since 2019, so we were fine there.
How AI Bots Behave
But what about GPTBot, ClaudeBot, and PerplexityBot? We examined our server logs:
- Googlebot: Visits the page twice (first HTML, then JS render)
- GPTBot: Single request, immediate exit, no rendering
- ClaudeBot: Same pattern, one pass only
- PerplexityBot: Grabs the HTML and leaves
Most AI bots don't execute JavaScript. The reason is resource cost: when you're crawling billions of pages, spinning up a JS engine for each one is prohibitively expensive. They just grab the raw HTML and process it. Since our raw HTML was empty, to these bots our site looked like it had no content at all.
Technical Solution: Static HTML Generation
The fix was clear: we needed to embed the content directly in the HTML for each blog post. Instead of dynamic fetching, we built a system that generates static pages at build time.
Previous Structure (Problematic)
// posts.js
fetch('/api/posts/ai-compatibility')
.then(res => res.json())
.then(post => {
document.getElementById('blog-container').innerHTML = `
<h1>${post.title}</h1>
<p>${post.content}</p>
`;
});New Structure (Solution)
We generated a separate HTML file for each blog post. For example, /blog/ai-compatibility.html:
<article>
<h1>What Is AI Compatibility?</h1>
<div class="meta">
<time datetime="2024-01-15">January 15, 2024</time>
<span>Author: AICompatible Team</span>
</div>
<p>How do AI bots see your website?
In this post, we explore how GPTBot, ClaudeBot, and other AI crawlers...</p>
<!-- Full content directly in HTML -->
</article>Adding Structured Data
We added BlogPosting schema to every page. This is critical both for Google and for helping AI bots better understand the content:
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "BlogPosting",
"headline": "What Is AI Compatibility?",
"description": "Learn how AI bots see your website and how to test for compatibility.",
"author": {
"@type": "Organization",
"name": "AICompatible"
},
"datePublished": "2024-01-15",
"dateModified": "2024-01-15",
"image": "https://aicompatible.com/images/ai-compatibility.jpg",
"publisher": {
"@type": "Organization",
"name": "AICompatible",
"logo": {
"@type": "ImageObject",
"url": "https://aicompatible.com/logo.png"
}
}
}
</script>This structured data explicitly tells AI models what the content is about, who wrote it, and when it was published.
Results and What We Learned
Two weeks after deploying the changes, we asked Perplexity the same question again. This time it cited our blog. ChatGPT's web search similarly started finding our content. In the server logs, we saw GPTBot and ClaudeBot visiting more frequently and staying longer.
How Common Is This Problem?
Thousands of sites built with frameworks like React, Vue, or Angular face the same issue. Sites using Single Page Application (SPA) architecture are especially at risk. It might not be a problem for Google, but in the age of AI, if your content isn't visible to LLMs, you're missing a significant opportunity.
How to Check Your Own Site
- Source code check: Open the source code in your browser with Ctrl+U (or Cmd+U). Is your content visible there?
- cURL test: In terminal, run
curl https://yoursite.com/blog-postto get the raw HTML. Is the content there? - Disable JavaScript: Turn off JS in your browser settings and reload the page. Does the page appear empty?
AICompatible.com's crawler does this check automatically. When you scan your site, it simulates what both Googlebot and AI bots like GPTBot and ClaudeBot see, detecting JS rendering issues. We also check for missing structured data, meta tag problems, and robots.txt configuration issues.
Bottom line: being visible on Google isn't enough anymore. In the age of AI, your content needs to be readable by LLMs. And that starts with clean HTML that doesn't depend on JavaScript.