SEO

AI crawlers are not refused at the edge

Low impact · weight 4 · worth up to 0.8 points of the total score

Why it matters

robots.txt can welcome AI crawlers while the CDN or firewall in front of the site turns them away. Cloudflare has blocked AI crawlers by default on new sites since July 2025, and many security plugins do the same. We request the page as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot and look for refusals and challenge pages. Our requests do not come from those companies' addresses, so a rule that checks the address as well may treat the real crawlers differently.

How to fix it

In Cloudflare, open Security, then Bots, and review "Block AI bots" and "AI Crawl Control": allow at least OAI-SearchBot and PerplexityBot, which fetch pages to cite them. Check firewall and security-plugin rules that match these names as well.

Check a page for this, and the other 70 checks at once

More SEO checks

One address serves the site

The same page reachable at http and https, or with and without www, is several copies of the site as far as search engines are concerned. They pick one, split links and signals between the copies, and sometimes pick the wrong one. The other addresses should redirect permanently to the one you use.

Content is in the HTML, not only in JavaScript

GPTBot, ClaudeBot and PerplexityBot download a page's JavaScript but do not run it, so a page that builds its text in the browser is blank to them. Google does run JavaScript, but later and not always. This counts the words in the HTML as it arrives, before any script runs.

Heading levels are not skipped

Screen readers and crawlers read headings as an outline. Jumping from H2 straight to H4 implies a missing level, so the outline reports a section that does not exist.

Page links to the rest of the site

Internal links are how crawlers reach the rest of the site and how ranking signal flows between pages. A page with none is a dead end that hoards whatever authority it earns.

See all checks