Page indexing · Improve page appearance
Indexed, though blocked by robots.txt: what it means and how to fix it
Google put the page in its index without ever crawling it. robots.txt kept Googlebot out, but links to the address were enough for Google to list it, usually with no description under the title.
What the status means
Search Console lists a page under this status when it is indexed although a Disallow rule in robots.txt matches it. Google always obeys robots.txt for crawling, but indexing can work from the outside: links on your site or others tell Google the page exists and what it is about.
Because Google never read the page, its result usually has no description, or says that no information is available for it. A noindex tag on the page changes nothing while the block stays: Google cannot see a tag on a page it is not allowed to fetch.
Why it happens
Links to a blocked page
Menus, internal links or other sites point at an address robots.txt blocks, so Google indexes it from the links.
noindex behind a block
The page says noindex and robots.txt blocks it too. The block hides the tag, so the page stays in the index.
Blocked after it was indexed
A Disallow added to get pages out of Google keeps them in: Google can no longer recrawl them to find out they changed or went away.
Blocked pages in the sitemap
The sitemap asks Google to index an address that robots.txt forbids it to crawl, which tells Google the page matters without letting it read the page.
How to fix it
- Go through the listed URLs and decide for each one whether it should be in search.
- Run the robots.txt checker below on a listed page to find the line that blocks it.
- If the page should be in search, remove or narrow that rule, or add an Allow for its path. Once Google crawls the page, it gets a normal title and snippet.
- If the page should not be in search, also remove the rule, and add a noindex robots meta tag or an X-Robots-Tag: noindex header. Leave the page crawlable, so Google sees the noindex and drops it.
- Take blocked and noindex pages out of the sitemap, and stop linking to the ones that should not be found.
- For a page that has to disappear now, the Removals tool in Search Console hides it for about six months while the noindex takes effect. Then use Validate fix.
Find the rule that blocks the page
Enter a listed URL: the checker reads the site's robots.txt and shows which line decides for Googlebot.
Questions
Why did Google index a page I blocked?
robots.txt stops crawling, not indexing. When links point at a blocked address, Google can index it from those links alone, without reading the page.
Will adding noindex remove the page?
Not while robots.txt blocks it. Google has to crawl the page to see the tag, so remove the Disallow first, keep the noindex, and the page drops out at the next crawl.
Is it safe to ignore these pages?
When the listed pages are filter or search URLs nobody looks for, the harm is small: they show as bare results that rarely get clicks. Some large sites keep the block on purpose to save crawling and accept the warning. For pages you want in search, or pages that must not appear, fix it.
How is this different from "Blocked by robots.txt"?
Both are pages Google may not crawl. "Blocked by robots.txt" ones stayed out of the index; these got in through links. A blocked page that gains links can move from one status to the other.