Page indexing · Why pages aren't indexed
Blocked by robots.txt: what it means and how to fix it
Googlebot found the address, read the site's robots.txt before visiting, and a rule there told it to stay away. The page has not been crawled, so it is not in the index.
What the status means
Search Console puts a page under this status when Googlebot did not fetch it because a Disallow rule in robots.txt matches its path. Google knows the address from a link or the sitemap, but has never seen what is on it.
A block stops crawling, not indexing. Most blocked pages stay out of search because Google has nothing to go on, but a blocked page that enough links point to can be indexed by its address alone, and then it moves to "Indexed, though blocked by robots.txt".
Why it happens
A rule wider than meant
Rules match from the start of the path, so Disallow: /p blocks /products and /pricing too, and Disallow: /*? blocks every address with a query string.
A staging file left in place
A Disallow: / written to keep a test site out of Google went live with the site and blocks every page.
A Googlebot group
Googlebot obeys only the group that names it. Once a User-agent: Googlebot group exists, the rules under User-agent: * no longer apply to it, Allow lines included.
Pages blocked on purpose
Cart, checkout, search results, filters and account pages, blocked by the site or by the CMS. These need no fix.
How to fix it
- Go through the listed URLs and set aside the ones that should stay out of search: carts, search results, filters, accounts.
- Run the robots.txt checker below on one of the pages that should be in search: it shows the line that decides for Googlebot.
- Remove or narrow that Disallow, or add an Allow for the page's path. The longest matching rule wins, so Allow: /shop/shoes beats Disallow: /shop.
- If the file has a User-agent: Googlebot group, make the change there, since Googlebot reads nothing else.
- Publish the file and open Settings, robots.txt in Search Console to check that Google fetched the new version. It can request a recrawl of the file.
- Run the checker again on the page, then use Validate fix in the Page indexing report.
Find the rule that blocks the page
Enter a listed URL: the checker reads the site's robots.txt and shows which line decides for Googlebot.
Questions
Is "Blocked by robots.txt" an error?
No. Search Console files it under pages that are not indexed, and many of the pages there are blocked on purpose. It needs fixing only when a page you want in search results is listed.
Where did the robots.txt Tester go?
Google retired it at the end of 2023. The robots.txt report under Settings in Search Console shows the file Google last fetched and the lines it could not parse, but it does not test a URL. The checker above does: enter the page and it shows which rule decides for Googlebot.
How long until the status clears?
Google caches robots.txt for up to a day, then has to recrawl the pages, which usually takes days to a few weeks. Validate fix in Search Console asks for that recrawl and reports when it is done.
Should I use robots.txt to keep a page out of Google?
No. robots.txt controls crawling, and a blocked page can still be indexed from links. To keep a page out of search, let Google crawl it and give it a noindex robots meta tag. Use robots.txt for pages that waste crawling, such as endless filter combinations.