October 08, 2026 • By AICompatible Team • 7 min read

# When 11 Language Versions Looked Like Duplicates: The Missing Canonical Tag ## How We Discovered the Problem: A Strange Drop in Google Search Console About six weeks after launching AICompatible.com in 11 languages, we noticed something odd in Google Search Console. Our English pages were indexing normally, but several German, French, and Spanish versions were flagged with "Duplicate, Google chose different canonical than user" warnings. Even worse, some of our Turkish pages weren't being indexed at all. At first, we assumed something was wrong with our hreflang tags. But every hreflang validator we ran said everything was fine. The real problem was somewhere else entirely: **over 1,200 of our pages had no canonical tag whatsoever.** ## Diagnosis: The Chaos of Missing Canonicals To understand what was happening, we compared an English and German page pair: - `https://aicompatible.com/en/tools/chatgpt` - `https://aicompatible.com/de/tools/chatgpt` When we looked at the source code for both pages, here's what we saw: ```html ``` The hreflang tags were there, but **the canonical tag was completely missing**. Here's what Google was doing as a result: 1. It saw the same content (AI tool descriptions) in 11 different languages 2. It couldn't tell which version was the "original" 3. It tried to choose a canonical on its own (usually defaulting to English) 4. It either marked other language versions as "duplicate content" or didn't index them at all When we ran a full crawl with Screaming Frog, the situation became crystal clear: **none of our 1,247 HTML pages had a canonical tag.** Every tool page, every category page, every blog post—hundreds of pages multiplied by 11 languages—all sitting there without canonicals. ## The Technical Fix: Bulk Adding Canonicals to 1,200+ Pages Manually editing 1,200+ pages wasn't an option. We built an automated solution via FTP. Here's the step-by-step approach we took: ### 1. Analyzing the Existing Structure All our HTML files followed this structure: ``` /public_html/ ├── en/ │ ├── tools/ │ │ ├── chatgpt.html │ │ └── midjourney.html │ └── blog/ ├── de/ │ ├── tools/ │ └── blog/ └── tr/ ├── tools/ └── blog/ ``` ### 2. Self-Referencing Canonical Logic Each page needed **a canonical pointing to its own URL**. Meaning: - `/en/tools/chatgpt.html` → canonical: `https://aicompatible.com/en/tools/chatgpt` - `/de/tools/chatgpt.html` → canonical: `https://aicompatible.com/de/tools/chatgpt` - `/tr/tools/chatgpt.html` → canonical: `https://aicompatible.com/tr/tools/chatgpt` **Important:** A common mistake with multilingual sites is pointing all language canonicals to the English version. This is completely wrong! Each language should have its own self-referencing canonical—the hreflang tags already define the cross-language relationships. ### 3. Bulk Update via Python Script We wrote a Python script to connect via FTP and update all HTML files: ```python import os import re from ftplib import FTP def add_canonical(html_content, file_path): # Build URL from file path # Example: /public_html/tr/tools/chatgpt.html → /tr/tools/chatgpt url_path = file_path.replace('/public_html', '').replace('.html', '') canonical_url = f'https://aicompatible.com{url_path}' # Create canonical tag canonical_tag = f' \n' # Insert after tag (before title) pattern = r'(\s*\n)' replacement = r'\1' + canonical_tag return re.sub(pattern, replacement, html_content, count=1) # FTP connection ftp = FTP('ftp.aicompatible.com') ftp.login('username', 'password') # Scan all HTML files languages = ['en', 'de', 'tr', 'fr', 'es', 'it', 'pt', 'pl', 'nl', 'ja', 'ko'] updated_count = 0 for lang in languages: ftp.cwd(f'/public_html/{lang}') for root, dirs, files in os.walk(f'./{lang}'): for file in files: if file.endswith('.html'): file_path = os.path.join(root, file) # Download file with open('temp.html', 'wb') as f: ftp.retrbinary(f'RETR {file_path}', f.write) # Read content and add canonical with open('temp.html', 'r', encoding='utf-8') as f: content = f.read() # Skip if canonical already exists if 'rel="canonical"' not in content: updated_content = add_canonical(content, file_path) # Upload updated file with open('temp.html', 'w', encoding='utf-8') as f: f.write(updated_content) with open('temp.html', 'rb') as f: ftp.storbinary(f'STOR {file_path}', f) updated_count += 1 print(f'✓ {file_path}') print(f'\n✅ {updated_count} files updated.') ftp.quit() ``` ### 4. Result: Proper Canonical Structure After running the script, every page had this structure: ```html ``` ## Results and What We Learned Two weeks after adding the canonical tags, the "duplicate content" warnings in Google Search Console dropped by 87%. Our German and French pages started indexing properly. Our organic traffic increased by 34% within three weeks. **This isn't just our problem:** Most multilingual sites have canonical tags that are either completely missing or misconfigured. These mistakes are especially common: - All language canonicals pointing to a single language (usually English) - Canonicals exist but aren't self-referencing (pointing to a different page) - Canonical URLs contain http/https or www/non-www inconsistencies **To check your own site:** Open a page's source code in your browser (Ctrl+U) and search for `