SEO

AI crawlers may read the page

Low impact · weight 4 · worth up to 0.8 points of the total score

Why it matters

ChatGPT search, Perplexity, Claude and Google's AI features quote pages their crawlers are allowed to read. A robots.txt rule that blocks GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot or Google-Extended keeps the page out of those answers. Blocking the training crawlers can be a deliberate choice; blocking all of them is rarely intended.

How to fix it

Remove the Disallow rules for the AI crawlers you want to be quoted by, or add a group for them that allows this path. At the least, allow OAI-SearchBot and PerplexityBot, which fetch pages to cite them rather than to train on them.

Check a page for this, and the other 70 checks at once

More SEO checks

AI crawlers are not refused at the edge

robots.txt can welcome AI crawlers while the CDN or firewall in front of the site turns them away. Cloudflare has blocked AI crawlers by default on new sites since July 2025, and many security plugins do the same. We request the page as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot and look for refusals and challenge pages. Our requests do not come from those companies' addresses, so a rule that checks the address as well may treat the real crawlers differently.

One address serves the site

The same page reachable at http and https, or with and without www, is several copies of the site as far as search engines are concerned. They pick one, split links and signals between the copies, and sometimes pick the wrong one. The other addresses should redirect permanently to the one you use.

Content is in the HTML, not only in JavaScript

GPTBot, ClaudeBot and PerplexityBot download a page's JavaScript but do not run it, so a page that builds its text in the browser is blank to them. Google does run JavaScript, but later and not always. This counts the words in the HTML as it arrives, before any script runs.

Heading levels are not skipped

Screen readers and crawlers read headings as an outline. Jumping from H2 straight to H4 implies a missing level, so the outline reports a section that does not exist.

See all checks