For those of you who don’t know, web crawlers are Internet bots that browse the Internet to index pages. They’re also known as web spiders, automatic indexers and web scutters. Among all those available out there, the Googlebot is the most intriguing one.
Googlebot Goes Around More than All Other Web Spiders Combined.
When it comes to web crawlers, Google takes the lead with a staggering 60.5 percent of the total number of page crawls.
It was interesting to see Googlebot actually crawling more pages than all other search engines combined,” Incapsula points out.
Incapsula (http://www.incapsula.com/blog/googlebot-study-dr-crawlit.html) reveals in its brand new report that the bot traffic landscape is more complex than you’d originally think. Googlebot’s behavior, in short, is hard to understand with its complexity, but that doesn’t mean people don’t try.
Incapsula, for instance, observed over 400 million search engine visits to 10,000 sites over a 30-day period. During this time, 2.19 billion page crawls took place, some from Googlebot, some from other similar tools and some from impostors.
Are popular sites more visited by spiders?
One of the goals of the study was to investigate whether there was a connection between a site’s popularity and Googlebot’s crawl rates. “Simply put, the hypothesis here was “popular sites get more crawls,” Incapsula explains.
To this end, the company created a sample group of 10,000 websites and five sub-segments categorized by the number of daily human visitors.
The results came out to something which no one expected. In fact, there is almost nil significant correlation between the volume of human visits and the number of Googlebot visits. Basically, Google’s tool doesn’t play favorites.
How often does Googlebot visit?
Another goal the company had was to analyze Googlebot’s crawl patterns. After processing 210 million Googlebot sessions, the company found the bot visits a website about 187 times per day and crawls 4 pages per visit.
Of course, these are just average. In 12.5 percent of cases, for instance, Googlebot visited sites over 500 times a day, while in most extreme cases they even saw 210,000 pages crawled during a single 72-hour visit.
It seems that sites that are content-heavy and get frequent updates are the most thoroughly crawled sites. This includes forums, news sites and online shops.