.Google.com's John Mueller responded to an inquiry concerning why Google.com marks pages that are actually prohibited coming from creeping through robots.txt as well as why the it is actually secure to dismiss the associated Browse Console documents concerning those creeps.Bot Web Traffic To Question Parameter URLs.The individual inquiring the concern chronicled that robots were actually making links to non-existent question parameter Links (? q= xyz) to pages with noindex meta tags that are additionally obstructed in robots.txt. What caused the concern is that Google.com is actually creeping the web links to those web pages, getting shut out by robots.txt (without envisioning a noindex robotics meta tag) after that getting shown up in Google.com Browse Console as "Indexed, though blocked out through robots.txt.".The person inquired the observing inquiry:." Yet below is actually the big inquiry: why would certainly Google.com index webpages when they can't also view the information? What is actually the advantage because?".Google.com's John Mueller validated that if they can't crawl the page they can not see the noindex meta tag. He additionally helps make an intriguing acknowledgment of the web site: hunt driver, suggesting to ignore the outcomes since the "normal" consumers won't see those outcomes.He wrote:." Yes, you are actually proper: if our team can't crawl the page, our company can't see the noindex. That said, if our company can't creep the pages, after that there is actually not a lot for us to index. Therefore while you could see some of those pages with a targeted internet site:- inquiry, the ordinary consumer won't see all of them, so I definitely would not bother it. Noindex is actually also great (without robots.txt disallow), it simply means the URLs will certainly wind up being actually crawled (and end up in the Search Console file for crawled/not listed-- neither of these conditions induce concerns to the rest of the internet site). The fundamental part is actually that you don't produce all of them crawlable + indexable.".Takeaways:.1. Mueller's response verifies the restrictions in operation the Internet site: hunt progressed hunt operator for diagnostic reasons. Among those main reasons is actually due to the fact that it is actually certainly not connected to the routine hunt mark, it's a different point altogether.Google's John Mueller talked about the website search driver in 2021:." The short solution is that a site: question is actually certainly not implied to become total, nor utilized for diagnostics objectives.A site inquiry is actually a certain kind of search that restricts the results to a certain site. It is actually primarily only words internet site, a bowel, and then the site's domain name.This query limits the end results to a specific internet site. It's certainly not implied to become an extensive compilation of all the webpages from that website.".2. Noindex tag without using a robots.txt is alright for these sort of circumstances where a robot is actually linking to non-existent pages that are actually obtaining discovered by Googlebot.3. Links along with the noindex tag will produce a "crawled/not recorded" item in Browse Console and that those won't possess a negative result on the rest of the website.Read through the concern and also respond to on LinkedIn:.Why would certainly Google index pages when they can't even view the material?Featured Photo through Shutterstock/Krakenimages. com.