Google has published the full list of IP addresses it uses for crawling and accessing your website under the Googlebot user agents. Google said if you do not want to use reverse DNS other those types of methods for verifying Googlebot, you “can identify Googlebot by IP address by matching the crawler’s IP address to the list of Googlebot IP addresses.”
Why do I need this. If you believe someone is crawling your site, pretending to be Googlebot, and you want to verify that blocking this fake Googlebot crawler is not really Google, then you can use either the command line method or automated IP list method.
IP list. You can access the full list of Googlebot IP addresses over here in this JSON file, note that Google may update this list, so you should check the file probably daily. You can also access all other Google crawlers that match the crawler’s IP address against the complete list of Google IP addresses. Again, these IP addresses may change over time, so you should build into your mechanisms a way to update the IP addresses on your end.
Why we care. Often sites can be slowed down and potentially even go offline because of fake bots crawling and spidering the site. You rarely want to block Google from crawling your site because that can lead to indexing and ranking issues in Google Search. So knowing which bot is really Google and which is not, can help you decide which rogue bots to block from your site.
There are also third party services, like Cloudflare and others that help you manage this.
About The Author
Barry Schwartz a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns
RustyBrick, a NY based web consulting firm. He also runs
Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry's personal blog is named
Cartoon Barryand he can be followed on Twitter here.
via https://AiUpNow.com November 10, 2021 at 07:44AM by Barry Schwartz, Khareem Sudlow,
