.Google.com's John Mueller addressed a concern regarding why Google.com marks webpages that are prohibited coming from crawling through robots.txt and why the it is actually safe to neglect the associated Look Console files about those crawls.Robot Traffic To Concern Specification URLs.The person inquiring the inquiry recorded that robots were generating web links to non-existent query parameter URLs (? q= xyz) to webpages along with noindex meta tags that are actually likewise obstructed in robots.txt. What triggered the inquiry is that Google.com is crawling the links to those webpages, acquiring obstructed by robots.txt (without noticing a noindex robots meta tag) then receiving turned up in Google Explore Console as "Indexed, though obstructed through robots.txt.".The individual talked to the adhering to question:." But listed here is actually the huge concern: why would certainly Google mark webpages when they can not also view the material? What is actually the advantage because?".Google's John Mueller validated that if they can not creep the webpage they can not find the noindex meta tag. He additionally makes an appealing mention of the internet site: hunt operator, recommending to disregard the results given that the "typical" users will not observe those end results.He created:." Yes, you are actually correct: if our company can not crawl the page, our company can't observe the noindex. That claimed, if our experts can not creep the web pages, at that point there's certainly not a lot for us to mark. Thus while you may observe a few of those webpages with a targeted site:- query, the typical customer won't see them, so I would not bother it. Noindex is actually additionally alright (without robots.txt disallow), it just indicates the Links are going to find yourself being crawled (and end up in the Browse Console document for crawled/not listed-- neither of these statuses induce issues to the remainder of the web site). The vital part is that you don't create all of them crawlable + indexable.".Takeaways:.1. Mueller's solution verifies the restrictions in operation the Web site: search advanced search operator for diagnostic reasons. Among those explanations is given that it is actually not linked to the normal search mark, it's a separate thing altogether.Google's John Mueller discussed the website search operator in 2021:." The short answer is that a site: inquiry is actually not suggested to become complete, nor made use of for diagnostics reasons.An internet site question is a details sort of hunt that confines the end results to a specific website. It is actually primarily just the word site, a colon, and then the website's domain.This inquiry confines the outcomes to a particular site. It's certainly not indicated to be a complete collection of all the web pages from that web site.".2. Noindex tag without using a robots.txt is fine for these type of circumstances where a robot is linking to non-existent pages that are receiving found out through Googlebot.3. Links along with the noindex tag are going to produce a "crawled/not catalogued" item in Browse Console and also those will not possess a bad result on the rest of the website.Read through the concern as well as answer on LinkedIn:.Why would certainly Google.com index web pages when they can not also view the material?Featured Graphic by Shutterstock/Krakenimages. com.