Methodology v1.0
November 18, 2026 • By AICompatible Team • 7 min read

How to Track Your AI Search Visibility Without Spending a Dime

# AIO Tracking: How to Monitor Your AI Visibility for Free

Executive Summary: To monitor your visibility in AI search results (AI Overviews, ChatGPT, Perplexity) at zero cost, establish four core monitoring routines: manual query testing (weekly), robots.txt and bot access audits (monthly), GA4 referrer analysis (weekly), and crawl log review (monthly). Each method provides a distinct data layer; used together, they reveal which content appears in AI responses and which bots are accessing your site.

Manual Query Testing: Weekly Checkpoint

The most straightforward method to measure how frequently your brand or content surfaces in AI platforms is systematic query testing. Open a spreadsheet and list 10-15 core questions relevant to your sector. For instance, if you sell accounting software, include queries like "how to file VAT returns," "invoice requirements for small businesses," or "expense tracking best practices."

Each week, run identical queries through ChatGPT (free or Plus), Google AI Overviews (standard search on google.com), Perplexity, and Gemini. Document whether your brand, product, or content received attribution, which URLs appeared, and where in the response you were positioned (opening paragraph, bulleted list, footnote citation). Capture timestamped screenshots—you'll observe the same query yielding different responses over time.

Task Frequency What to Monitor Tools
Manual query testing Weekly Brand/URL attribution, position, specific content excerpts used ChatGPT, Perplexity, Google AI Overviews, Gemini + Excel/Sheets
GA4 referrer tracking Weekly Session/pageview volume from chatgpt.com, perplexity.ai, gemini.google.com GA4 → Reports → Traffic acquisition → Session source/medium
robots.txt and bot access audit Monthly Whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot are blocked; which directories excluded yourdomain.com/robots.txt + server access logs
Crawl log analysis Monthly Which AI bots crawled which pages, crawl frequency, HTTP status codes Server access.log or Cloudflare Analytics (free tier)

GA4 Referrer Tracking: Weekly Reporting

In Google Analytics 4, navigate to Reports → Lifecycle → Traffic acquisition → Session source/medium. Use the search function to filter for "chatgpt.com," "perplexity.ai," and "gemini.google.com" to isolate traffic from these platforms. If nothing appears, either your content isn't yet cited in AI responses or users aren't clicking through.

Create a weekly custom report: In the Explore tab, select "Session source/medium" as your dimension, then filter rows containing "chatgpt" or "perplexity." Add metrics for "Sessions," "Engaged sessions," and "Views." Save this report and review it weekly using consistent date ranges (e.g., last 7 days). Sudden spikes may indicate specific content gaining prominence in AI responses; to identify which pages are driving traffic, add Landing page + screen class as a secondary dimension.

When Referrer Data Doesn't Appear

Some AI platforms may not pass referrer headers, or if users copy URLs directly into browsers, traffic appears as "Direct." UTM parameters won't help here since AI platforms generate your URLs automatically. However, you can use Campaign URL Builder to tag links you share yourself (e.g., on social media) with utm_source=chatgpt to compare which channels drive more engagement.

Robots.txt and Bot Access Control: Monthly Audit

Each month, review your site's yourdomain.com/robots.txt file and look for these directives:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: CCBot
Disallow: /

If these lines exist, the respective bots cannot crawl your site for model training or indexing (as a courtesy protocol). However, robots.txt is not a security mechanism—compliance is voluntary. For enforcement, implement User-Agent blocking rules at the server or WAF (Web Application Firewall) level.

Google-Extended is a special case: it's a robots.txt control token, not a separate crawler. It governs training/grounding usage for Gemini applications and Vertex AI, but does not affect Google Search rankings. Google AI Overviews draw from content crawled by standard Googlebot and are managed through snippet controls (nosnippet, max-snippet, data-nosnippet HTML attributes). Blocking Google-Extended won't impact your search visibility—it only restricts third-party AI applications like Gemini from using your content.

Bot Activity in Server Logs

Download your access.log file from your hosting panel (cPanel, Plesk) or server administrator. In a text editor, search for User-Agent strings like "GPTBot," "ClaudeBot," "PerplexityBot," or "CCBot." Each line represents a request record showing date, IP, requested URL, and HTTP status code. For example:

123.45.67.89 - - [15/Jan/2025:10:23:45 +0000] "GET /blog/vat-filing-guide HTTP/1.1" 200 12345 "-" "GPTBot/1.0"

This entry shows GPTBot successfully crawled (200 status) /blog/vat-filing-guide on January 15. If GPTBot were blocked in robots.txt yet this entry still appeared, the bot ignored the directive—consider IP-based blocking. Review these logs monthly to identify which pages receive the most crawl attention and which bots are most active.

Crawl Log Analysis: Monthly Summary

If you use Cloudflare's free tier, navigate to Analytics → Traffic and enable the "Bot Traffic" filter. This reveals which bots are sending requests, their volume, and which URLs they target. Cloudflare automatically labels known bots; those marked "Verified Bot" are typically legitimate crawlers (Googlebot, Bingbot, GPTBot, etc.).

Create a monthly Excel worksheet with columns: Bot Name, Total Requests, Most Crawled Page, Average Response Time, Error Rate (4xx/5xx). For instance, if GPTBot sent 500 requests monthly and 2% returned 404 errors, you may have broken links or the bot is using an outdated sitemap. Compare this data month-over-month for trend analysis—sudden increases may indicate new content entering AI training sets, while drops could confirm robots.txt changes are effective.

Automation Tips with Free Tools

Accelerate manual testing with a Google Sheets template: Place your queries in the first column, then add columns for dates and platform names (ChatGPT, Perplexity, etc.). Conduct tests at the same time weekly (e.g., Monday mornings) and populate cells with brief notes like "Citation present," "No citation," or "URL displayed." After several months, you'll identify which queries consistently earn citations and which never appear.

Set up custom alerts in GA4: Under Admin → Custom alerts, create a rule like "If session source/medium contains chatgpt.com and session count exceeds 10, send email." This provides instant notification when AI traffic spikes.

Frequently Asked Questions

I see no AI platform traffic in GA4—what's wrong?

Two possibilities: either your content isn't yet cited in AI responses, or users aren't clicking through. Additionally, some AI platforms don't pass referrer headers, causing traffic to appear as "Direct" or "(none)." If manual testing shows your brand receives citations but GA4 shows no traffic, click-through rates are low. To improve positioning in AI responses, implement structured data (Schema.org) and use clear, authoritative statements that AI models can easily parse and cite.

I blocked GPTBot in robots.txt but still see it in server logs—why?

robots.txt is a courtesy protocol, not technical enforcement. Bots may choose to comply or not. GPTBot typically respects robots.txt, but may be operating from a stale cache or different IP range. For definitive blocking, add User-Agent rules in your server configuration (Apache .htaccess, Nginx config) or WAF. For Apache:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} GPTBot [NC]
RewriteRule .* - [F,L]

This returns a 403 Forbidden response to all requests containing the GPTBot User-Agent. However, bots can change User-Agents; for comprehensive security, implement IP-based blocking or rate limiting.

Manual testing yields different results each week—is this normal?

Absolutely. AI models continuously update, training data refreshes, and search results are dynamic. Identical queries can produce different responses across days because models have indexed new content or user feedback has influenced algorithms. Don't rely on single tests—collect 4-6 weeks of data for trend analysis. Mark consistently cited content as "strong" and produce more on similar topics. For queries that never appear, restructure your content: clearer headings, bulleted lists, and Q&A formats help AI models parse and cite your material.

I don't use Cloudflare—are there other free log analysis tools?

Yes. Hosting control panels like cPanel or Plesk typically include free log analysis tools such as Awstats or Webalizer. These automatically process access.log files to generate reports on bot traffic, most-visited pages, and error codes. Alternatively, install the open-source tool GoAccess on your server; run goaccess access.log -o report.html from the command line to generate an HTML report. If you lack technical expertise, request a "bot traffic report" from your hosting support—most providers send monthly summaries.

Test Your Own Site

Try it on your own site