The most active bots and crawlers on the web

Allowing web crawlers to scan your site is vital if you want your web pages to appear in Google, Bing and other search results.

However, unwanted traffic spikes caused by non-human visitors can be costly in terms of bandwidth, website stability, potentially leading to site outages. To help you understand web crawlers, bots and spiders visiting your site, we’ve compiled a list of the most common instances of non-human traffic we see in our data, including their User Agents for reference.

For our latest dive into the data, we looked at the numbers for Q4 2018.

The data is sourced from thousands of websites built and hosted on the goMobi web publishing platform. Below is the breakdown of all the bots we saw.

The most active crawler is Googlebot

Given their dominance of all things search, it's no surprise to see Google topping the list, driving 28.5% of all bot hits in our data.

We spotted 91 variations of Google crawlers and bots, down from the 146 individual UAs we saw over the first half of 2018.

Below are the most common Google crawlers that appeared in our Q4 2018 data, and the User Agents associated with them.

Bing almost as active as Google

Combined, the nine Bing bots we see accounted for over 22% of all bot hits in our data, just behind Google's.

Interestingly, three of Bing's mobile crawlers announce themselves as an iPhone – the very first version, released way back in 2007. The other two masquerade as a Nokia Lumia 520 and a "Generic Windows Phone".

You can read about all crawlers used by Bing, and exactly what each of them does here. Below are the user agents for the most active three Bing crawlers we saw in Q4 2018.

Facebook

Unsurprisingly, given its omnipotence, Facebook's externalhit bot is a regular visitor. We saw 6 different types of Facebook crawler in our data.

The presence of Facebook's various crawlers has dropped a lot since we last studied the data. It reached 8.7% of all bot visits in the first half of 2018, but in Q4 this didn't even reach 3%.

The main use of this crawler is page previews. When a link is pasted within the platform, Facebook crawl the target page and pull information such as title, description and preview/featured images, as below:

The Facebook crawler UAs seen most often are:

Yahoo!

One of the earliest internet pioneers, Yahoo has undergone major changes in recent years. In 1994 they were called "Jerry and David's guide to the World Wide Web", and simply listed other websites. There was no search offered, but in the year 2000 they integrated Google's product. By 2004, they had developed their own search functionality.

There is one main crawler – Yahoo! Slurp. After that, we see Yahoo Pipes, YahooCacheSystem and Yahoo Nutch, with none of those three accounting for a significant amount of traffic.

Here are the main crawlers, and their respective UAs.

Others to note

So we've looked at the "Big Four" and their crawlers and bots, and provided the most common UserAgents we see across our network. What else in the data is interesting?

With over 5% of the overall bot hit share, Sogou Spider is the 7th busiest bot on the list. It's the web crawler for Beijing-based search technology provider, Sogou.com.

New to the top spots for 2019 is Comscore's Proximic, a crawler that "enables advertising partners to determine the best matching campaign for a page's content".

Next, we have Baidu, China's answer to Google and the fourth most visited site in the world (Alexa).

With a DeviceAtlas Cloud Standard, Premium or Enterprise account, you can identify non-human traffic (robots, crawlers, checkers, download agents, spam harvesters and feed readers) in real-time. You can then decide how to act on this information, whether to block all undesired bots at the door, or just treat them in a different way to legitimate human visitors.

Read more about bot detection and how it can help your business to:

• Limit Ad-fraud
• Protect your site

We also have some handy resource lists such as a list of User Agents for the most popular smartphones and devices and the most common mobile browsers across 35 countries.