.Google.com's John Mueller responded to a concern concerning why Google.com marks webpages that are refused coming from crawling by robots.txt as well as why the it's secure to disregard the similar Explore Console files regarding those crawls.Crawler Visitor Traffic To Query Criterion URLs.The individual talking to the inquiry documented that crawlers were creating web links to non-existent query specification URLs (? q= xyz) to pages along with noindex meta tags that are actually also blocked in robots.txt. What triggered the concern is that Google.com is creeping the links to those webpages, getting blocked out through robots.txt (without envisioning a noindex robots meta tag) at that point obtaining reported in Google Search Console as "Indexed, though shut out by robots.txt.".The individual talked to the complying with question:." Yet right here is actually the significant question: why will Google.com index pages when they can't also find the material? What is actually the advantage because?".Google's John Mueller validated that if they can not creep the page they can't see the noindex meta tag. He also produces an exciting acknowledgment of the website: hunt operator, advising to ignore the results considering that the "normal" customers will not see those outcomes.He created:." Yes, you are actually appropriate: if we can not creep the page, our experts can not find the noindex. That pointed out, if our company can not creep the web pages, at that point there's certainly not a whole lot for our company to index. Thus while you might see a few of those webpages with a targeted internet site:- inquiry, the common user won't find all of them, so I definitely would not bother it. Noindex is actually likewise fine (without robots.txt disallow), it only means the Links will definitely end up being actually crawled (and also end up in the Browse Console file for crawled/not indexed-- neither of these conditions result in issues to the remainder of the internet site). The vital part is actually that you do not produce them crawlable + indexable.".Takeaways:.1. Mueller's answer affirms the constraints being used the Site: hunt accelerated hunt driver for analysis explanations. One of those explanations is given that it's certainly not connected to the frequent hunt mark, it's a different factor altogether.Google's John Mueller discussed the web site hunt driver in 2021:." The brief solution is that an internet site: question is actually certainly not indicated to be total, neither utilized for diagnostics functions.An internet site inquiry is a specific type of hunt that limits the outcomes to a specific site. It is actually primarily only the word internet site, a colon, and after that the web site's domain name.This inquiry limits the results to a particular internet site. It's certainly not suggested to be a complete assortment of all the pages from that website.".2. Noindex tag without using a robots.txt is fine for these sort of situations where a crawler is actually linking to non-existent web pages that are actually getting discovered by Googlebot.3. URLs with the noindex tag will definitely generate a "crawled/not catalogued" entry in Search Console which those will not have a damaging effect on the rest of the site.Review the concern and respond to on LinkedIn:.Why would Google index web pages when they can not even find the information?Included Photo through Shutterstock/Krakenimages. com.