Mobile web traffic: a dive into the data

We all know that web site traffic from mobile devices is increasing rapidly. But what does it actually consist of? What devices are more popular or how large is the fraction of “non-human” traffic? There have been many reports and analysis done on web traffic in general, but there are also a couple of good reasons to look inside mobile web traffic data, that is, traffic to websites optimized for mobile device use.

One of the reasons is that mobile traffic is not all the same. There is a huge diversity of devices, from low-end feature phones equipped with WAP browsers to high-end smartphones with browsers meeting the most recent standards. Apart from browser differences, network capabilities make another set of properties varying across different devices. But nearly the most visible feature of this diversity is the wide range of device screen sizes. All the factors above can’t be overestimated when we talk about user-focused web design or user experience.

Another reason for detailed analysis is the fact that device markets change quite rapidly. Therefore, it is essential to understand the trends, in order to be able to respond adequately to users’ needs.

The analysis in this article is based on the traffic to thousands of mobile websites serving customers in many countries from each part of the world.

The data used for the analysis covers the period of July-August 2014. The classification was done based on User-Agent (UA) strings recognition using DeviceAtlas API.

Breaking it down

Let’s start with a high-level breakdown of traffic.

As one can expect, mobile devices make the largest group from several considered categories (61%), but they are not the only “visitors” to mobile sites. The next two largest groups are non-human and desktop browser traffic.

Types of visits to mobile sites

The table below contains the detailed breakdown of all considered categories.

Let’s take a look at what all the categories mean and what they contain.

Mobile devices

This group is the subject of interest to content providers, as it consists of real users visiting websites with their devices. If we want to see what type of devices are included in this group, here is another table covering this information.

Visits to mobile sites from mobile devices

Over 97% of traffic comes from the most common devices types, i.e. mobile phones and tablets.

The “mobile phones” group consists of smartphones, feature phones, or any other mobile phone that contains a browser. The share of feature phones and low-end smartphones (e.g. Nokia Asha series, older Blackberry phones, etc.) in the “mobile phones” group is about 6%. The other part consists of phones that run typical smartphone operating systems, such Android, iOS, Windows Phone, Firefox OS, etc.

The traffic generated by mobile phones is over four times higher than from tablets, but this proportion is constantly changing over the time in favor of tablets. The number of tablet models appearing on the market increased over three times during the last two years. You can find out more about device and property distribution at the Data Explorer page.

A certain level of traffic comes from devices sending generic mobile user agents, which do not contain any device specific information. The info is usually limited to an operating system and a browser. Therefore such UAs cannot be classified into any of the specified categories.

A small group of other devices can also be seen among the typical ones. This includes media players, games consoles, e-book readers and any other devices capable of making HTTP requests.

Non-human

The second largest group consists of the traffic generated by non-human visits. Robots, crawlers, checkers, feed fetchers, spam, etc. make up to 28% of the traffic. Many robot UAs pretend to be real devices and they can easily mislead device detection solutions that are commonly used for device recognition. It happens because a device user agent is often included within a robot UA apart from robot information. A large contribution to the robot traffic is made by Google crawlers. Although their visits to websites are most likely desired, they can massively change the overall picture of site visitors.

An interesting fact can be highlighted here. If we count unique User-Agents instead of visits, the breakdown shows a completely different image.

A small number of unique User-Agents show substantial activity and generate relatively large traffic. The most active robot UAs are Googlebots and other indexing agents:

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

A significant contrast between mobile and non-human traffic, in terms of the relation between visits and unique User-Agents, is also worth highlighting. Looking at values for the “mobile” group, we can see a huge diversity in the mobile User-Agents, in comparison to the number of visits they generate. This diversity can be explained by a variety of mobile devices together with a range of browsers that exist on the market.

Desktop traffic

Another significant group covers desktop traffic.

Even if we consider websites typically optimized for mobile devices, there always would be some amount of desktop browsers making requests for mobile sites. This could be a result of web testing, random redirections or User-Agent spoofing tools usage. Some users may deliberately change a browser’s User-Agent to trick a server in order to receive the content intended for desktop browsers.

Among the most popular desktop browsers are Internet Explorer (36%), Firefox (32%), Chrome (22%) and Opera (4%).

Garbage

All other User-Agent types generate a bit more than 1% of traffic. The most significant group from the remaining ones contains just garbage strings. Hashes, random character strings, deliberately altered UAs or just empty strings make about 0.6% of the traffic.

Here is a list of some interesting examples:

Mozilla/4.0
&as_qdr=all
/usr/local/websense/af_mobile/user_agent/ipad
anonymous
User-Agent
silly_that_i_have_to_do_this
‘”\'”);|]*{%0d%0a<%00>
TGVub3ZvIEEyNjlpOzEwOzIuMy42O0EyNjlpX1MwMjBfMTMxMDI0Ozg2MzgwMTAyMDc4OTMxNjsyNDcwMjEyMDE4ODgyODQ7bnVsbDs7bnVsbDtudWxsOzIzMzBhNDgwYzQ7MjMzMGE0ODBjNDtudWxsO2NvbS5hbmRyb2lkLmRlOjE1OmNvbS5hbmRyb2lkLmRlLk1lU3luY1NlcnZpY2U7MDtMViBURUxFMkZDOjk3Mw

The last UA is just an encoded (Base64) version of a UA string coming from a real device: the Lenovo A269i. The decoded string looks as follows:

Lenovo A269i;10;2.3.6;A269i_S020_131024;863801020789316;247021201888284;null;;null;null;2330a480c4;2330a480c4;null;com.android.de:15:com.android.de.MeSyncService;0;LV TELE2FC:973

As the above analysis shows, mobile traffic is not uniform to any degree. At the high level it can be broken down into different device factors such as mobile phones, tablets or even desktop browsers. But the breakdown can go even deeper into the browsers and operating systems that devices are running on. The last aspect was not covered in this discussion.

It was also shown here that a significant number of requests are made by crawlers and robots pretending to be mobile devices.

Data source and tools used

The data used for the above analysis originates from the traffic through websites created on the goMobi platform. goMobi powers thousands of websites that are optimized for mobile devices. They are spread around many countries across the world and experience a wide variety of visitors from each part of the world.

It was mentioned previously that the calculations were done with respect to users’ visits rather than hits. The visit, which can also be considered a “session”, was defined as a unique combination of IP address, User-Agent, date, and visited host. Therefore, the single visit might consist of one or more hits to the server.

The DeviceAtlas API was used for classification of User-Agent strings. In general the API allows device recognition as well as detection of device properties, such as device vendor and model, screen resolution, hardware type and others. For the purpose of this analysis some extra processing was applied in addition to the API, in order to identify User-Agents coming from mobile apps and garbage strings.

All comments welcome

The above analysis covers only a picture of a general breakdown of visits to mobile websites, but I hope it can also trigger a discussion on more specific topics. All comments are very welcome; if you have any questions and suggestions, please post them all below.