AT&T’s Misuse Of Ookla’s Speedtest Data

Earlier this month, AT&T published a news release claiming: “AT&T is the ‘Nation’s Fastest Wireless Network’ in 2019”. The claim is based on data collected by Ookla, the company behind Speedtest.net.

In the piece, AT&T writes:

With over 10 million consumer-initiated tests taken daily with Speedtest, Ookla provides invaluable insight into the performance, quality and accessibility of networks worldwide.

Ookla’s reliance on consumer-initiated tests has serious downsides. For example, consumers on different networks use different kinds of phones. Premium phones tend to have hardware that supports faster connections than cheap phones. If subscribers on Network A tend to run tests using premium phones while subscribers on Network B tend to run tests from budget phones, Network A is going to have an advantage in consumer-initiated tests that’s unrelated to underlying network performance.

Ookla’s methodology is also prone to selection bias. Consumer-initiated tests don’t occur among a randomly selected sample of subscribers on each network. Consequently, test results likely aren’t representative of typical network performance. There was a clear case of this problem when AT&T began misleadingly labeling some of its 4G service “5GE.” Here’s an excerpt from a Speedtest.net blog post:[1]

In the final week of Q1, we also observed an increase in faster tests taken on AT&T’s network. Upon investigation, we discovered that this correlated with the release of iOS 12.2 and the roll out of AT&T’s 5G E icon. We also found that the increase in tests was coming from device models that would have started to display the 5G E icon, such as the newer generations of iPhone (XR, XS Max, XS, X, 8, 8 Plus), indicating that consumers were seeing the new icon and taking a test to see what speeds they were getting.

While no method for evaluating mobile network performance is perfect, I tend to think Opensignal and RootMetrics use methodologies that are more reliable than Ookla’s. AT&T didn’t take the top spot for speeds in RootMetrics’ most recent report or Opensignal’s most recent report.

Silly criticisms

AT&T’s news release includes ridiculous digs at competitors:

Speedtest results show we increased our speeds by 45.7% year-over-year, while one of our competitors never ‘checked the box’ on speed with only a 16.5% increase year over year and the other defined what it meant to be ‘Un-speedy’ with only a 9.4% increase year over year.

Year-over-year changes in average speeds don’t on their own indicate whether networks are fast. A visual in the news release is illuminating:

Chart

AT&T performed poorly relative to T-Mobile and Verizon in early 2018. In a sense, AT&T was able to get a 45.7% year-over-year increase in speeds because it performed so poorly in 2018.

Consistency

I’d argue that AT&T’s news release ignores the most important part of Speedtest’s 2019 report. In my opinion, average speed is overrated. For most consumers, it’s far more important to consistently have decent speeds than to have high average speeds.

Ookla reports a consistency metric based on the proportion of tests that exceed a threshold of 5Mbps. Verizon takes the top spot on this metric, followed by T-Mobile, with AT&T coming in third. Verizon also beats out AT&T for coverage availability, another metric that can act as a proxy for consistency.[2]

Bias against Verizon

In Ookla’s main analyses, data is only included from “competitive geographies.” Competitive geographies only include areas where Ookla has a substantial number of test results from at least three major networks.[3] There are defensible reasons for Ookla to use the competitive geographies filter. However, it should be acknowledged that Verizon has the nation’s most extensive network and likely outperforms AT&T and other networks in non-competitive geographies.[4]

AT&T’s Claim To Being America’s Best Network

AT&T has been running an ad campaign with commercials where the company claims to offer the best network.

These commercials start with a funny skit that leads to the line, “just ok is not ok.” The commercials’ narrator then says something along the lines of: “AT&T is America’s best wireless network according to America’s biggest test.”

Here’s an example:



Alternate versions of the commercial involve ok babysitters, ok sushi, ok surgeons, and more.

AT&T bases its “best network” claim on the results of Global Wireless Solutions’s (GWS) 2018 tests. The claim is at odds with the results of many other companies’ evaluations and my own view.

The meaning of the word “best” is ambiguous, but I’d guess that a survey of professionals in the wireless industry would find that most people consider RootMetrics to be the best evaluation firm in the wireless industry. Verizon fared far better than AT&T in RootMetrics’s most recent evaluation.

It’s unclear to me what AT&T is claiming when it calls GWS’s test, “America’s biggest test.” Is it the biggest test in terms of miles driven, data points collected, area covered, or something else? GWS may have the biggest test according to one metric, but it’s not unambiguously the biggest test in the nation.

AT&T Paying $60,000,000 For “Unlimited Data” Claims

AT&T has settled with the Federal Trade Commission (FTC) and agreed to pay out $60 million to current and past customers that may have been affected by misleading claims about unlimited data. The settlement is in response to the FTC’s 2014 accustation that AT&T failed to adequately disclose that customers on unlimited data plans could have their speeds throttled substantially. Here are a few bits from the 2014 FTC complaint:

The FTC’s complaint alleges that the company failed to adequately disclose to its customers on unlimited data plans that, if they reach a certain amount of data use in a given billing cycle, AT&T reduces – or “throttles” – their data speeds to the point that many common mobile phone applications – like web browsing, GPS navigation and watching streaming video – become difficult or nearly impossible to use…AT&T’s marketing materials emphasized the ‘unlimited’ amount of data that would be available to consumers who signed up for its unlimited plans…AT&T, despite its unequivocal promises of unlimited data, began throttling data speeds in 2011 for its unlimited data plan customers after they used as little as 2 gigabytes of data in a billing period. According to the complaint, the throttling program has been severe, often resulting in speed reductions of 80 to 90 percent for affected users. Thus far, according to the FTC, AT&T has throttled at least 3.5 million unique customers a total of more than 25 million times…consumers in AT&T focus groups strongly objected to the idea of a throttling program and felt ‘unlimited should mean unlimited.’

Here’s an excerpt from the FTC’s press release from today (emphasis mine):

As part of the settlement, AT&T is prohibited from making any representation about the speed or amount of its mobile data, including that it is “unlimited,” without disclosing any material restrictions on the speed or amount of data. The disclosures need to be prominent, not buried in fine print or hidden behind hyperlinks. For example, if an AT&T website advertises a data plan as unlimited, but AT&T may slow speeds after consumers reach a certain data cap, AT&T must prominently and clearly disclose those restrictions.

I’m glad to see the FTC cracking down on misleading practices. Bogus “unlimited” plans seem to be much more common today than they were in 2014.

2,000,000 AT&T Phones Were Unlocked Illegally

It recently came out that around 2,000,000 AT&T phones were unlocked by hackers that bribed AT&T employees. Muhammad Fahd and co-conspirators allegedly bribed a handful of AT&T employees to make the unlocks possible.

As I understand it, around 2012 lists of IMEI numbers were provided to bribed employees so that devices could be fraudulently unlocked. Eventually, the crimes became more involved. Bribed employees installed malware on AT&T systems and fraudulent wireless access points in AT&T facilities.

It’s a crazy story. Several years ago, I wondered how so many third-party services managed to offer device unlocking. I suppose this story is part of the explanation.

For more details, check out Ars Technica’s article.

Cell phone in use

AT&T Prepaid 8GB Promo Plan

Disclosure: At the time of writing, I have a financial relationship with AT&T. However, I don’t believe I receive commissions on prepaid service plans.


AT&T recently began offering its prepaid plan with 8GB of data, unlimited minutes, and unlimited texts for only $300 ($25 per month) for those who are willing to purchase 360 days of service upfront. According to AT&T, this promotion is a limited time offer that’s set to end on October 21, 2019.

If you’re willing to purchase nearly a year of service all at once, this is an awfully good deal for a large data allotment on AT&T’s network. My own experience with AT&T’s prepaid service when I trialed it a few months ago was positive.

The $300 base price may not be the final price of 360 days of service. If you need to order a SIM card, that comes with an additional charge. When I went through the checkout process and indicated that I needed a SIM card shipped to Boulder, CO, my final price came to $305.43:

  • $300 for service
  • $4.99 for a SIM card
  • $0.44 in sales tax

In the grand scheme of things, the final price was awfully close to the base price. With a lot of carriers, I’d expect to see more extensive taxes and fees beyond the base price of service.

Additional details about the AT&T plan:

  • Mobile hotspot access is included. Hotspot use draws from regular data allowances.
  • Rollover data is included (only one month of unused data will rollover).
  • AT&T’s Sponsored Data deal is offered. Certain types of data use won’t count against monthly data allotments.

Comparison with Mint Mobile’s pricing

At $25 per month, a year of AT&T prepaid service with 8GB of data is competitive with the cost of a year of Mint Mobile’s 8GB service. Mint Mobile currently charges $20 per month on annual 8GB plans. Taking advantage of a current promotion, customers can get six months of Mint Mobile service for only $60.

While Mint Mobile continues to offer a lower monthly price, it may make sense for many people to spend a few extra dollars each month to access AT&T’s network. At the national level, AT&T’s network offers more extensive coverage than Mint Mobile’s service over T-Mobile’s network.

Lies, Damned Lies, and AT&T’s 5GE

There are three kinds of lies: lies, damned lies, and statistics.Benjamin Disraeli*

Fortunately, the sentiment behind this quote isn’t always accurate. Sometimes statistics can reveal lies. AT&T has recently taken a lot of heat for misleadingly branding advanced 4G networks as “5GE.” Ian Fogg at Opensignal published a post where he draws on Opensignal’s data to assess how AT&T’s 5GE-enabled phones perform compared to similar phones on other carriers. The results:[1]

In response to AT&T’s misleading branding, Verizon launched a video advertisement showing a head-to-head speed comparison between Verizon’s network and AT&T’s 5GE network.

In that video, Verizon’s 4G LTE network comes out with a download speed near 120Mbps while AT&T’s 5GE network came out around 40Mbps. That, of course, seems funny given the Opensignal data suggesting the networks deliver similar speeds on average.

A portion of the Verizon video—not long enough to show the final results—showed up in a Twitter ad. That ad led to a Twitter exchange between myself; Light Reading’s editorial director, Mike Dano; and Verizon’s PR manager, Steven Van Dinter. Dinter explained that Verizon chose to film in a public spot where AT&T’s 5GE symbol was very strong. I take Dinter’s word that there wasn’t foul play or blatant manipulation, but it is funny to see Verizon fighting misleading branding from AT&T with a misleading ad of its own.