Audit Scoring Methodology

A detailed overview of the mathematical formulas, crawler regulations, and structural compliance tests driving the AI Compatibility Index.

1. The Mathematical Scoring Model

To rate a website's readiness for Retrieval-Augmented Generation (RAG) processes used by ChatGPT Search, Gemini, and Perplexity, we compute a weighted index score from 0 to 100:

\[ \text{Index Score} = 0.35 \cdot C_{\text{access}} + 0.25 \cdot S_{\text{semantic}} + 0.30 \cdot M_{\text{schema}} + 0.10 \cdot P_{\text{speed}} \]

Where:

\(C_{\text{access}}\) (Bot Crawler Access): Evaluates if AI crawler bots (GPTBot, PerplexityBot, ClaudeBot, Google-Extended, Applebot-Extended) are blocked in robots.txt rules.
\(S_{\text{semantic}}\) (Semantic Content Density): Checks content readability using BLUF structures, atomic headings, and the density of factual entities.
\(M_{\text{schema}}\) (Structured Metadata): Validates the integration of entity mappings (JSON-LD Organization, Article, Product, and FAQPage schemas).
\(P_{\text{speed}}\) (Crawl Speed & Performance): Represents server response times. Delayed responses cause AI crawlers to abort crawls to conserve API token allocations.

2. Evaluation Pillar Breakdown

A. Bot Accessibility Index (\(C_{\text{access}}\))

AI bots parse server header logs and `robots.txt` files before rendering pages. We audit access permissions. If any bot is explicitly denied (`Disallow:`), the access index is penalized. If all primary commercial search bots are unblocked, the score reaches 100%.

B. Semantic Structure Index (\(S_{\text{semantic}}\))

Language models divide page content into text segments or "chunks" to retrieve answers. We check for:

BLUF Compliance: Answers to questions must be placed directly beneath the corresponding header tags.
Table & List Presence: AI models prioritize structured table rows (e.g., comparing features/pricing) and bullet lists because they are easy to extract.

C. Metadata Schema Index (\(M_{\text{schema}}\))

Without schema markup, crawlers use fuzzy heuristics to parse authors, logos, and publish dates. We check for the presence of W3C standard JSON-LD elements to verify compliance.