Transcoders – the technology we love to hate that will never go away

Many years ago I remember a Google product manager saying that his product, the Google mobile transcoder, was the one product that everyone wished would go away.

Alas, no such luck: Google’s mobile transcoder is still available (as Google Web Light¹), and in April 2015 Google announced a new transcoder-based web initiative in Indonesia. Facebook’s internet.org uses a web transcoder. Chrome has a bandwidth optimisation setting that uses transcoders. Opera Mini is now ten years old but going strong with over 250m users; UC Browser has over 500m users and is the number one mobile browser in China. There are many other examples of how web transcoding is still an absolutely relevant and growing technology today.

More recently still, the trend to transcode shows no signs of abating even in the face of new technologies. Rather, it seems new technologies provide new transcoding opportunities. So it was only a matter of time before someone applied transcoding tech to producing AMP web pages, but that is exactly what the Mercury HTML to AMP transcoder does. As with all automatic transcoding, the results are hit-and-miss, and this one is no exception.

So mobile transcoders are in rude health. They’re like the fax machines of the web, something that you thought would fade away years ago but that you find yourself needing nonetheless. So why are they so hated? There are a number of reasons but chief amongst them are the following:

• They can be crude, sometimes breaking mobile-friendly content in their attempts to remove excess weight.
• They tend not to allow publishers to opt out or control the process. This is probably illegal in some countries.
• Some transcoders are set up to add branding (typically of the mobile operator) to the top and bottom of pages, often interfering with the aesthetics and operation of existing pages.
• They compromise security—it is not possible to implement a fully secure HTTPS connection with a transcoder in the mix.

The inner compression mechanism of a web transcoder. Threaded bolt to left adjusts compression levels.

So why are we still using them? Surely the need for transcoders should have faded with smartphone adoption and mobile data improvements? This assumption misses a crucial fact: technology improves but it tends to improve everywhere, not just where it is needed most. So yes, mobile data rates are now faster than ever and smartphones are widespread even in poorer nations but the rest of the world didn’t stand still—so a gap remains. What qualifies as a smartphone is very broad. The “capability gap” between a smartphone sold in the US/EU and a smartphone sold in Africa is still wide, perhaps as wide as between early iOS/Android phones and their contemporaneous feature phones. Does this matter? Yes, it really does—some of today’s web sites are effectively unusable on low-end smartphones, a point that internet.org’s rules back up, insisting that sites must work without JavaScript. The same applies to data networks, which are improving globally and thus significant regional differences remain. Taken together, the gap in hardware and connectivity between countries is perhaps as wide as it ever was.

The size and shape of this gap will change but it will always exist and as long as it does it will need to be bridged. Web transcoders are in our past but also our future so we need to learn to live with them. The W3C Content Transformation Guidelines (which we at mobiForge helped to author) still makes some very good suggestions and is worth another look. Finally, note that server-side adaption can help you serve different levels of content to devices while retaining the same URL. Good device detection solutions should be able to detect these proxy-based browsers. This allows publishers to retain some control by optimising the content to avoid over-processing by transcoders. Our own DeviceAtlas has a useful guide to detecting side-loaded browsers that will help you get started.

First published: Jul 14, 2015
Updated: Jul 27, 2016, to include clarifactions on Google Web Transcoder, and Mercury HTML to AMP transcoder

^{1. The Google Web Transcoder was, until recently, available at http://www.google.com/gwt/n, but this URL has now stopped working. However, Google’s web transcoder appears to be enjoying a new lease of life as Google Web Light}