Some people call me a skeptic, but I don’t believe them…

Google intends to deprecate and freeze the User-Agent header due to privacy issues. Privacy issues, really Google? For some reason I doubt that Google is acting altruistically here. Their actions over the years suggests that there might be ulterior motives. So, with that in mind, let’s take a look at what else might be motivating Google to deprecate and freeze the User-Agent header and let’s look into how the proposal is being implemented. But first some context.

Context

– The User-Agent Header And How It Is Being Used

The User-Agent header is a central piece of web architecture and plays an important role in content negotiation. Basically (and likely most relevant to you), it’s a field that contains a piece of text (i.e. a string) that your browser sends to the server hosting the website that you are visiting.  The piece of text contains browser and other information that the server can use to enhance your overall experience of the website (e.g. show you the mobile version of the website if you’re on a mobile device).

The User-Agent header, introduced at the dawn of the web, was originally intended to be used for statistical purposes and to identify protocol violations. The web and device ecosystem have certainly developed in ways that were not contemplated in the early 90’s. As the ecosystem developed, so did the User-Agent header use case. Developers soon began leveraging the User-Agent header to identify and declare compatibility between different browsers, websites and devices – the overall aim being to enhance end user experience. Additional use cases include, user authentication, personalization and fraud prevention.

– Compatibility And Privacy

As developers can define each User-Agent string (to declare compatibility for example), some argue that the User-Agent string is limited when used to accurately identify devices. However, others highlight that it can be done accurately with sufficient resources and focus.

Regarding privacy, User-Agent strings enable browser fingerprinting. W3C notes that, “in short, browser fingerprinting is the capability of a site to identify or re-identify a visiting user, user agent or device via configuration settings or other observable characteristics.” W3C goes on to also note that “browser fingerprinting can be used as a security measure (e.g. as means of authenticating the user)” and lists best practices to mitigate the privacy impacts of browser fingerprinting, here.

– Apple Attempted To Freeze The User-Agent String

In Dec 2017, Apple published “Release Notes for Safari Technology Preview 46”. To reduce compatibility issues and prevent fingerprinting, Apple planned to freeze the User-Agent string in Safari. However, after the industry swiftly challenged and informed the company of the impacts, Apple overturned their decision. Feedback received related to concerns such as security vulnerabilities, performance issues, etc. – read some of the feedback on the Twitter thread, here.

The Intent: Google Proposes To Deprecate And Freeze The User-Agent Header

Now that we have some context, let’s take a look at Google’s proposal.

As mentioned, Google intends to deprecate and freeze the User-Agent header with the purported motivation being to enhance privacy and compatibility. Google is currently progressing with their intent despite industry concerns. An overview of the timeline for Chrome releases (where this potential change to open web standards is being entangled) can be seen below:

Source: Intent to Deprecate and Freeze: The User-Agent string

A major difference between Apple’s attempt to freeze the User-Agent string and Google’s current attempt is the proposal that Client Hints replaces the User-Agent string for content negotiation. So, now let’s look at Client Hints.

– Introduction Of Clients Hints

The aim of Client Hints is to “provide developers with the ability to perform agent-based content negotiation when necessary, while avoiding the historical baggage and passive fingerprinting surface exposed by the venerable User-Agent header.”

As a standalone tool, this sounds great – most people appear to be welcoming of Client Hints. Client Hints appears to be very useful, however, significant issues arise if it replaces the User-Agent header. Feedback and concerns regarding Client Hints are being logged here. 2 key concerns are as follows.

    1. Performance concerns

Alarmingly, the reported issue “Performance impact of moving User-Agent header to Client Hint headers” was “Closed” by Google so that Client Hints would not be blocked, noting that the performance hit would be limited to a small number of use cases. One use case impacted includes adaptive loading – there’ll be no more adaptive loading on the first response…unless you already know what device the visitor is using, but who will continue to know this information? Google of course!

This reported, and quickly closed, issue highlighted that over 80% of the top 50 websites use the User-Agent string to adapt their website to the visiting device’s capabilities. Ignoring the performance hit will likely impact the bottom line of many businesses, including Google’s competitors, while Google themselves can benefit financially as a result.

The industry continues to highlight this performance hit, by logging issues, as it appears that Google does not fully appreciate (I’d argue that they are choosing to ignore) the scale of the impact on other businesses.

    2. Privacy concerns

The overall net positive impact on privacy and fraud is still being debated. Exposing values, that are not even available in the User-Agent string, does not enhance privacy. An issue logged by Safari notes that “implementing Sec-CH-UA-Arch honestly would be a privacy regression for us”, while a comment logged by an ad network highlights concern over potentially exceeding the user’s privacy budget, while another comment logged in W3C Technical Architecture Group (TAG) issue 476 states the following: “I fail to see how this would significantly impact privacy or fingerprinting since Client Hints will enable fingerprinting equivalent or even deeper than what User-Agent currently provides.”

If Google was so concerned about privacy, they could implement changes within their own practices that could have a greater positive impact on privacy. This would be a lot easier and quicker for them to implement compared to forcing changes on open web standards. Thus, calling into question their motivation. Are they altruistically attempting to enhance privacy or are they using privacy as a veil to increase their market dominance and revenue?

If you expect to be negatively impacted by Client Hints, issues can be logged here. Similarly, if you think Client Hints will be beneficial but feel that it’s also necessary to keep the User-Agent header, feedback can be logged here.

– Google’s Approach

Google’s approach to developing and delivering on their plan leaves a lot to be desired. Listed below are 4 concerning tactics that are being used.

    1. Use dominant position to force through a change on open web standards

There have been many issues logged and many concerns raised by industry, however and unlike Apple, Google is continuing with their intent/actions to deprecate and freeze the User-Agent header.

Google, a trillion-dollar company with unparalleled technical resources, is leveraging its dominance in the browser market to implement (unless challenged further) a significant change in open web standards. A change that will negatively impact countless businesses that don’t have ample technical resources like Google, a change equivalent to actions taken by Google in the past that resulted in various EU antitrust investigations, a change that should be driven by an industry body (definitely not by a company that will profit from the change).

The W3C Technical Architecture Group (TAG) should be that body. TAG is a special working group chartered with stewardship of the Web architecture. A key focus of their mission is to “document and build consensus around principles of Web architecture and to interpret and clarify these principles when necessary”. It’s clear from the logged issues and online feedback that there is no consensus around freezing and deprecating the User-Agent.

Another key aspect of TAG’s mission is “to resolve issues involving general Web architecture brought to the TAG”. Issues relating to this proposal are being brought to TAG, here. I’d encourage anyone concerned about the impact of Google’s intent to freeze the User-Agent header to challenge the proposal now while there is still an opportunity to keep the User-Agent part of the open web standards.

    2. Lack of consultation with industry

The actual impact of this change on industry is not fully understood. As mentioned previously, understandably, the use cases for the User-Agent header has developed, along with the online and connected device landscape. It does not appear that Google has consulted industry sufficiently to try to get an appropriate understanding of the modern use cases and level of impact that a freeze to the User-Agent header will have.

Furthermore, and detailed in feedback to Google, “the burden of proof for making the case that this truly is a problem worth fixing clearly falls on the proposer of the change”. Google’s response to this was short, dismissive and vague. The response briefly noted that there’s “independent research on the subject” and a link was provided to research on browser fingerprinting (not relevant to the point raised regarding the wider impact on business) – an indication of the lack of consideration for industry.

    3. Don’t create an appropriate level of awareness

It is not clear how much effort has been put into creating awareness of this intent to deprecate and freeze the User-Agent header. The awareness generated appears to me to be mostly organic. What is clear to me, however, is that a company with the reach of Google making such an upheaval to the web (and as a result, the bottom line of many businesses) should make substantial efforts to publicize this intent and not try to force it through without due consideration, consultation and awareness.

    4. Keep up or else

Making it even more difficult for smaller players and competitors to limit the damage of this forced change by Google, the timeframe for businesses to adapt effectively is limited. Google’s dominance and resources combined with the abundance of data they have (they have the data that other companies need the User-Agent string to determine, e.g. visiting device capabilities) means that Google has the ability to push this through at a speed that maximizes the benefit to them.

The limited industry consultation and level of awareness reduces barriers to the implementation of this intended change – a positive for Google but not so great for their competitors. Google is currently proceeding, and unless challenged further, they might force through a change to open web standards that will result in many companies scrambling to limit the negative impact.

Why Is Google Planning To Deprecate And Freeze The User-Agent Header – The Skeptic’s View?

Similar to Apple’s attempt, the purported motivation behind Google’s intent to deprecate and freeze the User-Agent header relates to privacy and compatibility concerns. But is this the case? The skeptic in me says no.

We have briefly looked at the context, the proposal and how things are progressing, now let’s look at why Google is doing this.

Well, the reason it obvious: it for the $’s (clearly!). This move will increase Google’s market dominance while restricting competition, but importantly, this time they’re being more tactical in their approach to their antitrust practices.

– Avoid Antitrust Fines While Restricting Competition And Increasing Market Dominance

Google is no stranger to antitrust investigations. Google is currently under antitrust investigations in the US and in 2019 was fined €1.49 billion by the European Commission for breaching EU antitrust rules. Speaking about the breach, EU competition commissioner Margrethe Vestager, stated that “the misconduct lasted over 10 years and denied other companies the possibility to compete on the merits and to innovate.” It looks like Google is continuing with its antitrust practices but this time with a more veiled approach (with their purported motivation being privacy and compatibility).

As highlighted above, Google’s intent to deprecate and freeze the User-Agent header will have a massive impact on businesses. Let’s take a brief look at the impact that this could have within some markets where Google competes/has market dominance.

    1. Advertising industry

It’s well known that Google makes most of its revenue from advertising (over $134 billion in 2019). This move will help increase their revenue and further their market dominance. This is because Google will not require the use of Client Hints and can therefore avoid the negative performance hit that their competitors will experience. Google looks to be at risk of another antitrust fine with this kind of behavior. However, given that the €1.49 billion fine imposed on Google in 2019 equates to approx. 1% of their revenue just from advertising alone and given the pace at which this proposal is progressing, it looks like Google is willing to take the risk of receiving another fine.

Additionally, Google’s move to deprecate and freeze the User-Agent header will reduce their competitor’s ability to combat ad fraud – an issue that Google will not experience. In a comment on W3C Technical Architecture Group (TAG) issue 476, one ad network commented to Google: “Just because your organization has hundreds of millions of logged in active users across your various web properties and devices (search, chrome, android, chrome os, youtube, gmail, pixel etc.), it is significantly easier to run ad fraud analysis and protect your own ad network as the rest of us bite the dust.”

    2. Browser and analytics industry

Chrome is by far the most dominant browser worldwide with 64% market share, followed by Safari with 17% and Firefox with 5%. Further compounding Google’s browser dominance, many smaller browsers (e.g. Microsoft Edge, Opera) are based on Chromium code, which is free open-source software from Google.

Google’s browser market dominance enables them to essentially dictate what the browser market does and what developers must do to ensure that most of their website visitors receive a quality experience. Google is currently leveraging their browser dominance to force through this upheaval in the web.

Most websites / developers / users will likely benefit from Client Hints, however, competing browsers might experience a significant impact. Competing, non-Chromium based browsers will essentially be forced to follow suit with Chrome, while browsers using Chromium code might disappear from analytics results altogether – all Chromium based browsers might appear to be Chrome browsers in analytics platforms. This however is currently being discussed, with the alternative being the option to add additional Client Hints tokens, which will then likely result in similar User-Agent string characteristics (defeating the purpose of Client Hints).

An effective solution for this issue has not been provided yet.

It’s not clear what immediate impact Google’s plan will have on the browser and analytics markets, some think it’ll be positive, others think negative, but what is clear is that any limitation of data to Chrome competitors (or users of Chrome competitors) will mean that Google will be in a stronger position.

Conclusion

Without sufficient industry consultation, impact assessments, agreed objectives, robust design and without the authority, Google is leveraging their market dominance to force through a significant change on open web standards. Google should not be driving a change on open web standards that will benefit them financially, stifle competition and increase their market dominance.

The introduction of Client Hints is welcome, however, the attempt to deprecate and freeze the User-Agent header isn’t. This move appears to be another antitrust case in the making for Google. I highly doubt the thin veil of altruism will be enough to avoid another antitrust fine, however, given the potential revenue gains and already very deep pockets, the rewards appear to outweigh the risk for Google.

There is still time however to challenge and, similar to Apple’s attempt, to overturn this proposal to deprecate and freeze the User-Agent header. As mentioned above, TAG is a special working group chartered with stewardship of the Web architecture. Issues relating to the proposal can be brought to them, here.

Challenge Google’s proposal now while there is still an opportunity to keep the User-Agent part of our open web standards.