The Canonical Tag Mistake That Hurt Our Multilingual Site's Indexing
Google Thought Our 11 Languages Were Duplicate Content: The Missing Canonical Tag
How We Discovered the Problem: A Strange Drop in Search Console
About three weeks after launching AICompatible.com in 11 languages, we noticed something odd in Google Search Console: several language versions were stuck at around 40% indexing rate. We started seeing "Duplicate without user-selected canonical" warnings, especially for our German and French pages.
At first, we assumed it was an hreflang issue—that's the classic trap for multilingual sites. But our hreflang configuration was correct. The real problem was much more fundamental: over 1,200 pages had no canonical tag whatsoever.
When we caught this, we looked back and realized Google had been trying to guess which page version was "primary" on its own. Sometimes it picked the English version as canonical, sometimes Turkish. This caused other language versions to be treated as duplicate content.
Diagnosis: How Widespread Was This?
We started by manually checking a few pages. Opening Chrome DevTools and inspecting the <head> section, we saw hreflang tags but no canonical tag:
<head>
<meta charset="UTF-8">
<title>AI Tools für Marketing | AICompatible</title>
<link rel="alternate" hreflang="en" href="https://aicompatible.com/en/tools/marketing" />
<link rel="alternate" hreflang="de" href="https://aicompatible.com/de/tools/marketing" />
<link rel="alternate" hreflang="fr" href="https://aicompatible.com/fr/tools/marketing" />
<!-- no canonical tag! -->
</head>Then we ran a full site crawl with Screaming Frog. Filtering by the "Canonical" tab and listing "Missing" entries gave us the damage: 1,247 pages. Nearly all our content pages were live without canonical tags.
Why This Matters So Much
- Essential for multilingual sites: Each language version should have a self-referencing canonical pointing to its own URL. This tells Google, "This German page is the canonical version of the German page."
- Duplicate content risk: Without canonicals, Google can see the same content across 11 languages as duplicates and may only index one version.
- Lost link equity: If Google picks the wrong page as canonical, backlinks to other language versions lose their value.
The Fix: Adding Canonicals to 1,200+ Pages
We had two options: either add canonical tag support to our CMS (in our case, a custom static site generator), or bulk-edit the existing HTML files. For speed, we chose the latter.
Step 1: Analyze the Current Structure
All our HTML files lived in the /public folder, organized in subfolders by language code:
/public
/en
/tools
marketing.html
design.html
/de
/tools
marketing.html
/tr
/tools
marketing.html
...Each file had hreflang tags in the <head> but no canonical.
Step 2: Bulk Addition with a Python Script
We wrote a Python script that crawled all HTML files, detected the current URL, and added the correct self-referencing canonical:
import os
import re
from pathlib import Path
BASE_URL = "https://aicompatible.com"
PUBLIC_DIR = "./public"
def add_canonical(file_path, relative_url):
with open(file_path, 'r', encoding='utf-8') as f:
content = f.read()
# Check if canonical already exists
if ' /de/tools/marketing
relative_path = file_path.replace(PUBLIC_DIR, '').replace('.html', '')
if add_canonical(file_path, relative_path):
count += 1
print(f"✓ {relative_path}")
return count
if __name__ == "__main__":
total = process_directory(PUBLIC_DIR)
print(f"\n✓ Added canonical to {total} files.")Running the script:
$ python add_canonical.py
✓ /en/tools/marketing
✓ /en/tools/design
✓ /de/tools/marketing
...
✓ Added canonical to 1247 files.Step 3: Verify the Results
After running the script, we opened a sample file to verify:
<head>
<meta charset="UTF-8">
<title>AI Tools für Marketing | AICompatible</title>
<link rel="canonical" href="https://aicompatible.com/de/tools/marketing" />
<link rel="alternate" hreflang="en" href="https://aicompatible.com/en/tools/marketing" />
<link rel="alternate" hreflang="de" href="https://aicompatible.com/de/tools/marketing" />
<link rel="alternate" hreflang="fr" href="https://aicompatible.com/fr/tools/marketing" />
</head>Perfect. Each page now had a canonical tag pointing to its own URL. The German page pointed to the German URL, the French page to the French URL.
Step 4: Upload via FTP
We uploaded the changes to the server via FTP. With 1,200+ files, we used FileZilla's "Overwrite if newer" option and had everything updated in about 8 minutes.
Results and What We Learned
Within 10 days of making the changes, we saw dramatic improvements in Search Console:
- "Duplicate without user-selected canonical" warnings dropped by 90%
- Indexing rate for German and French pages jumped from 40% to 78%
- Organic traffic increased by 34%, especially in non-English languages
Key Takeaways
Canonical tags aren't optional for multilingual sites—they're mandatory. While hreflang tags say "these pages have alternatives," canonical tags say "but this page IS this URL." They need to work together.
This isn't just our problem—it's extremely common, especially on sites using static site generators or custom CMS platforms. If you run a multilingual site, check right now:
- Open a page in your browser, right-click → "View Page Source"
- Look for
<link rel="canonical"in the<head>section - If it's missing or pointing to the wrong URL, fix it
AICompatible.com's site crawler automatically detects these kinds of structural SEO issues (missing canonicals, incorrect hreflang, duplicate content risks). We found this problem manually, but you can discover it in minutes by running a crawl. For multilingual sites with 100+ pages, these checks take too long to do manually.
Canonical tags might seem like a small detail, but they're critical for ensuring each language version of a multilingual site gets evaluated independently. If yours are missing, don't wait for Google to decide for you—those decisions rarely go the way you want.