Category: Mobile Phone Service

PCMag’s 2019 Network Tests — My thoughts

Christian 1 Comment

Summary

On June 20, PCMag published its latest results from performance testing on the major U.S. wireless networks. Surprisingly, AT&T rather than Verizon took the top spot in the overall rankings. I expect this was because PCMag places far more weight on network performance within cities than performance in less-populated areas.

In my opinion, PCMag’s methodology overweights average upload and download speeds at the expense of network reliability. Despite my qualms, I found the results interesting to dig into. PCMag deserves a lot of credit for its thoroughness and unusual level of transparency.

Testing methodology

PCMag claims to be more transparent about its methodology than other entities that evaluate wireless networks.1 I’ve found this to be true. PCMag’s web page covering its methodology is detailed. Sascha Segan, the individual who leads the testing, quickly responded to my questions with detailed answers. I can’t say anything this positive about transparency demonstrated by RootMetrics or OpenSignal.

To measure network performance, PCMag used custom speed test software developed by Ookla. The software was deployed on Samsung Galaxy S10 phones that were driven to 30 U.S. cities as they collected data.2 In each city, stops were made in several locations for additional data collection. PCMag only recorded performance on LTE networks. If a phone was connected to a non-LTE network (e.g., a 3G network) during a test, the phone would fail that test.3 PCMag collected data on six metrics:4

  • Average download speed
  • Percent of downloads over a 5Mbps speed threshold
  • Average upload speed
  • Percent of uploads over a 2Mbps speed threshold
  • Reliability (percent of the time a connection was available)
  • Latency

The Galaxy S10 is a recent, flagship device and has essentially the best technology available for high-performance on LTE networks. Accordingly, PCMag’s test are likely to show better performance than consumers using lower-end devices will experience. PCMag’s decision to use the same high-performance device on all networks may prevent selection bias that sometimes creeps up in crowdsourced data when subscribers on one network tend to use different devices than subscribers on another network.5

In my opinion, PCMag’s decision not to account for performance on non-LTE networks somewhat limits the usefulness of its results. Some network operators still use a lot of non-LTE technologies.

Scoring

PCMag accounts for networks’ performance on several different metrics. To arrive at overall rankings, PCMag gives networks a score for each metric and assigns specific weights to each metric. Scoring multiple metrics and reasonably assigning weights is far trickier than most people realize. A lot of evaluation methodologies lose their credibility during this process (see Beware of Scoring Systems).

PCMag shares this pie chart when describing the weights assigned to each metric:6

The pie chart doesn’t tell the full story. For each metric, PCMag gives the best-performing network all the points available for that metric. Other networks are scored based on how far they are away from the best-performing network. For example, if the best-performing network has an average download speed of 100Mbps (a great speed), it will get 100% of the points available for average download speed. Another network with an average speed of 60Mbps (a good speed) would get 60% of the points available for average download speed.

The importance of a metric is determined not just by the weight it’s assigned. The variance in a metric is also extraordinarily important. PCMag measures reliability in terms of how often a phone has an LTE connection. Reliability has low variance. 100% reliability indicates great coverage (i.e., a connection is always available). 80% reliability is bad. Networks’ reliability barely affects PCMag’s rankings since reliability measures are fairly close to 100% even on unreliable networks.

The scoring system is sensitive to how reliability numbers are presented. Imagine there are only two networks:

  • Network A with 99% reliability
  • Network B with 98% reliability

Using PCMag’s approach, both network A and B would get a very similar number of points for reliability. However, it’s easy to change how the same metric is presented:

  • Network A has no connection 1% of the time
  • Network B has no connection 2% of the time

If PCMag put the reliability metric in this format, network B would only get half of the points available for reliability.

As a general rule, I think average speed metrics are hugely overrated. It’s important that speeds are good enough for people to do what they want to do on their phones. Having speeds that are way faster than the minimum speed that’s sufficient won’t benefit people much.

I’m glad that PCMag put some weight on reliability and on the proportion of tests that exceeded fairly minimum upload and download speed thresholds. However, these metrics just don’t have nearly as much of an effect on PCMag’s final results as I think they should. The scores for Chicago provide a good illustration:

Despite having the worst reliability score and by far the worst score for downloads above a 5Mbps threshold, T-Mobile still manages to take the top ranking. Without hesitation, I’d choose service with Verizon or AT&T’s performance in Chicago over service with T-Mobile’s performance in Chicago. (If you’d like to get a better sense of how scores for different metrics drove the results in Chicago, see this Google sheet where I’ve reverse engineered the scoring.)

To create rankings for regions and final rankings for the nation, PCMag combines city scores and scores for suburban/rural areas. As I understand it, PCMag mostly collected data in cities, and roughly 20% of the overall weight is placed on data from rural/suburban areas. Since a lot more than 20% of the U.S. population lives in rural or suburban areas, one could argue the national results overrepresent performance in cities. I think this puts Verizon at a serious disadvantage in the rankings. Verizon has more extensive coverage than other networks in sparsely populated areas.

While I’ve been critical in this post, I want to give PCMag the credit it’s due. First, the results for each metric in individual cities are useful and interesting. It’s a shame that many people won’t go that deep into the results and will instead walk away with the less-useful conclusion that AT&T took the top spot in the national rankings.

PCMag also deserves credit for not claiming that its results are the be-all-end-all of network evaluation:7

Other studies may focus on downloads, or use a different measurement of latency, or (in Nielsen’s case) attempt to measure the speeds coming into various mobile apps. We think our balance makes the most sense, but we also respect the different decisions others have made.

RootMetrics and OpenSignal are far less modest.

Prioritized and Deprioritized Services by Network

Christian 2 Comments

Understanding networks’ prioritization policies can be tricky. Recently, I’ve spent some time digging through the major U.S. networks’ policies. I’ve created a web page for each network listing which services I expect to be prioritized and which services I expect not to be prioritized. Making sense of each network’s policies isn’t always easy, so it’s possible I’ve misclassified some services. Please contact me if you think any information is incorrect or missing.

2021 Update: Over the last few years, I’ve run a bunch of QCI tests to assess priority levels of different plans, and I’ve become more confident about my conclusions. Results from QCI tests can be found here.

“Creating Jobs”

Christian

A lot of numbers have been thrown out about how many jobs will be created or destroyed if the proposed merger between Sprint and T-Mobile goes through. Here are a few examples I found on the New T-Mobile website:

When dealing with complex systems full of feedback loops, nothing happens in isolation. This fact is often overlooked. I’ll occasionally see an article where a nutrition journalist explains that there are 3,500 calories in a pound. Using some simple math, they’ll explain that since a bag of chips has 200 calories, falling into the temptation to eat a bag of chips every day will lead a person to gain about ten pounds over six months. This is of course silly. By the same logic, eating a bag of chips each day could lead a person to gain 800 pounds over 40 years.

People can’t eat a bag of chips without affecting anything else about their body or their behavior. The body is a complex system. There are lots of feedback loops that regulate things like hunger and energy expenditure.

The economy works in a similar manner. Jobs cannot be created out of thin air. Jobs can’t be eliminated in isolation. If the New T-Mobile hires 10,000 people, it’s not reducing the number of unemployed people by 10,000. The emphasis on job creation is especially strange right now. The U.S. unemployment rate is currently lower than it’s been at any other point in my life.

SpeedTalk Mobile Incentivizing Reviews

Christian 2 Comments

I recently purchased a cheap service plan I wanted to trial from SpeedTalk Mobile. During the checkout process, I was shown the following:

This clearly violates Google’s guidelines.1

Reviews are only valuable when they are honest and unbiased. (For example, business owners shouldn’t offer incentives to customers in exchange for reviews.)

I also noticed SpeedTalk keyword stuffing in its website’s footer—another questionable strategy:2

I was pretty excited about SpeedTalk. Not anymore.

Alternatives to SpeedTalk

If you’re looking for an alternative low-cost carrier, take a look at my review of Mint Mobile or my larger list of recommended budget-friendly carriers.

Is Google Fi Worth It?

Christian

Google Fi uses an admirably simple pricing structure. A base rate of $20 per month offers subscribers unlimited talk and text. Beyond that, users are charged $10 per gigabyte of data. Single-line plans are capped at a monthly charge of $80, so subscribers that use 6GB of data will pay the same monthly price as subscribers that use 10GB of data.1 While I like the simplicity of the pricing structure, plans end up being fairly expensive. It’s my impression that Google Fi has had its current pricing structure in place for several years despite the cost per byte of data dropping in the industry at large.

Fi-enabled devices have technology that allows them to switch between T-Mobile, U.S. Cellular, and Sprint’s networks. While the technology is cool, I’m not sure I’d choose seamless switching between three networks with mediocre coverage over exclusive access to Verizon’s more reliable network.2

Fi now officially supports devices that are not Fi-enabled. When these devices are used with Fi, they’ll only have access to T-Mobile’s network. Many mobile virtual network operators use T-Mobile’s network and offer far better prices than Fi. For example, Mint Mobile’s plans blow Fi’s prices out of the water.3 Even with a Fi-enabled device, I think most people can find a better deal. A light user would pay $30 per month before taxes and fees for texts, talk, and 1GB of data on Fi’s network. You could get the same unlimited texting, unlimited talk, and 1GB of data with Verizon’s prepaid service for $30.4 RedPocket can offer those resources on any of the major networks for $19 per month.5

For heavy data users, the case against Fi is even clearer. Using 6+ gigabytes of data brings the Fi monthly bill to $80 before taxes and fees. At that cost, I expect you could purchase an unlimited, postpaid plan with any of the Big Four carriers.

Despite my negativity, I’m still a huge fan of Fi’s simplicity and remarkable international roaming policies. Hopeful Fi will revamp its prices in the near future to become more competitive with the other options on the market.

When Will 3G Be Phased Out?

Christian

Last updated: October 22, 2021

Major U.S. network operators are phasing out their 3G technologies. Below, I share my impressions about the status of the major networks’ 3G phase-outs. It’s possible some network operators won’t stick to their deadlines (plenty of early deadlines have already been pushed back).

Verizon

Verizon has twice set and then pushed back deadlines for retiring its 3G network. The date is now set at December 31, 2022. Verizon committed to not push the deadline back any further.

Verizon has stopped activating 3G-only phones and phones that don’t support HD Voice. More details about Verizon’s plans and policies can be found in an entry on Verizon’s knowledgebase.

AT&T

AT&T plans to retire its 3G network by February 2022.

T-Mobile

T-Mobile has mostly stopped activating 3G-only phones. The company plans to shut down its 3G network as of July 1, 2022.

T-Mobile plans to shut down the legacy Sprint 3G/CDMA network it acquired by March 31, 2022.

Your Mileage May Vary

Christian 2 Comments

It’s common to see some consumers giving a cell phone carrier glowing reviews while other consumers publish long rants about the same carrier. When it comes to cell phone service, your mileage may vary.

Cell phone users differ from each other in a lot of ways. It should be unsurprising that a T-Mobile customer in New York City might feel differently about the quality of his T-Mobile’s service than another customer in rural Wisconsin feels about her T-Mobile service. Even within a single geographic region, the way people feel about the quality of a given carrier will vary substantially. People use different phones and do different things with their phones. People are also pretty different. Some people don’t mind if they occasionally don’t have service while traveling in sparsely populated areas. I hate it. Some people are frustrated if their video streaming is limited to 480p quality. That doesn’t bother me.

When choosing cell service, realizing that individual experiences with a carrier will be highly variable can prevent a lot of frustration. Taking advantage of the fact that people’s mileage may vary can allow frugal consumers to save a lot on wireless service.

The frugal strategy

I would guess that there are millions of consumers paying $50-$100 per line each month for postpaid Verizon service that could switch to Mint Mobile plans costing about $20 per line each month without experiencing a recognizable decrease in service quality. While Verizon’s postpaid service is going to generally be better than Mint’s service, many consumers have great experiences with Mint.

Even if you expect to have a good experience with a premium carrier, it may be worth trying low-cost carriers beforehand. There’s a chance the low-cost carriers are capable of providing you with good service. Let’s assume a low-cost, prepaid service would cost you $20 per month while a premium service would cost you $70 per month and require a two-year commitment. If you try the prepaid service first and end up liking it enough to choose it over the premium service, the upside is huge. You can save $50 per month on each line and don’t get locked into a long contract. Over a two-year period, the financial savings come out to $1,200 per line. If you try the prepaid service and disklike it, the downside is pretty low. You experience one month of mediocre service before upgrading to something better.

Strategy outline

  • Consider getting an unlocked phone with near-universal compatibility if you don’t already have one1
  • Try a cheap, prepaid carrier for one month
  • If you love the carrier, stick with it
  • If not, try a slightly more expensive carrier
  • Keep trying more expensive carriers until you find one you like

Most people don’t use anything like this strategy. Some of them are leaving a lot of money on the table.

Google Voice Review

Christian

I love Google Voice, and I want more people to know how awesome it is. I find Voice to be most useful when paired with a wireless device, but it’s also great as a stand-alone service you can use from a desktop.

Stand-alone Google Voice

The stand-alone, personal version of Voice is free and easy to get started with (much like Google Docs or Gmail). Once you’re signed up for the web platform, you’ll be given a free Google Voice phone number. From Voice’s web interface, you can send and receive calls and texts. Calls and texts in the U.S. are free.

Here are a few examples of situations where Voice’s stand-alone service can be useful:

  • You’re selling something on Craigslist and want to list your phone number. You don’t want to share your normal phone number because you don’t want to be contacted at odd hours or receive spam calls. Listing your Google Voice number in your ad solves the problems.
  • You misplaced your phone in your house. No one else is around to call it to make it ring. After accessing Google Voice from a computer, you can call your phone yourself.
  • You’re seated comfortably at your computer, but you need to make a call. No problem! Make your call from Goole Voice.

I don’t think most people will want to use stand-alone Google Voice regularly, but it’s a super helpful tool in rare situations. I once had a phone die while traveling. Using Voice, I was able to reach out to a friend I was meeting with, confirm our plans, and let them know I wouldn’t be reachable through my usual phone number. That prevented a lot of stress.

Google Voice paired with a phone

Google Voice can be extremely useful when paired with a mobile device and a primary number. Once you’ve downloaded a Google Voice app to a device, you can send and receive calls or texts from either the Voice web platform or your mobile device. It’s a best-of-both-worlds situation. If I’m at my computer, I prefer to take calls over Voice since my computer has a solid internet connection and I often have a nice headset connected. Texting is also more pleasant with a full-sized keyboard. The Voice web interface even lets me view visual voicemails and contacts from my desktop. None of this requires me to sacrifice my mobile experience. Calls and texts will continue to come through to my phone when I’m away from my computer.

If you don’t have a primary phone number you rely on, all of this can be done free of charge. Just take the free number Google Voice gives you, pair it with your device, and treat it as your primary number going forward. Alternatively, if you have a primary number, you can port it to Google Voice for a one-time fee of $20.1 Having your number live with Google rather than a traditional carrier has amazing perks. Normally, regularly switching phone companies is kind of a pain in the ass because you need to port numbers between carriers. The porting process takes time, and issues can arise while porting. With Google Voice, no porting is necessary. I’ve used the same Google Voice number with a handful of different carriers. When I switch between carriers’ SIM cards, my primary number stays associated with the Voice app on my phone.

Voice may even allow you to lower your typical phone bill. If you make calls from your computer, you can avoid using minutes from your mobile plan. While calls made from the Google Voice app on your phone will use your plan’s minutes, you can choose to download the Hangouts app which allows you to make calls from your phone using data (Wi-Fi or cellular) instead of minutes.

Configuring Google Voice

It may take a bit of tinkering to get Google Voice working optimally. Here are a few settings I’ve selected to get the experience I want:

  • I have the option to screen calls turned off.
  • I have the option to always use my phone to place calls turned off.
  • I only pair Google Voice with one phone.2

Downsides

  • I’ve found Google Voice’s web application to have some issues when accessed via Firefox.
  • Normal video messages cannot be sent from Google Voice.
  • Occasionally, I’ve found text messages to fail to send through the web application (fortunately, the application alerts me when this happens).3

5G: Awesome But Overhyped

Christian

5G network technologies have the potential to deliver far faster wireless speeds than 4G technologies. 5G will also enable data transfer with much lower latency (delay) than 4G. The technology is awesome. However, I’ve had a nagging suspicion that aspects of 5G are being overhyped to consumers.

Earlier this month, I attended the Big 5G Event in Denver. Many executives in the telecommunications industry spoke at the conference, including speakers from each of the major U.S. networks. While I felt like some network operators had been misleading consumers about 5G, I hoped statements network operators made at the conference would be more reliable. After all, most attendees of the conference were telecommunications professionals capable of seeing through bogus claims. Unfortunately, excessive hype and silly claims showed up in many of the presentations.

That said, a couple of folks deserve applause for calling out bullshit related to 5G. From the beginning of his presentation, Ibrahim Gedeon, CTO of the Canadian operator Telus, indicated that he was going to be unusually straightforward:

How the hell am I going to spend 20 minutes looking very smart and say nothing…I should abandon that…I think 5G is letting us down.

Gedeon didn’t pull any punches when discussing AT&T’s recent shenanigans:

I feel like we have to 5G something…Maybe we can find out from some of our peers how we can fake the 5G icon on the phone…That is the cheapest way to deliver 5G…I have nothing but the ultimate respect for good marketing.

5G use cases

Prior to the conference, I was skeptical about many of the proposed use cases for 5G. After the event, I’m even more skeptical.

Nicki Palmer, a senior vice president at Verizon, explained how 5G combined with machine learning could allow people with food allergies to scan items they pass in the grocery store to detect whether items contain allergens. It’s a neat idea. It might be technically impressive. However, 5G speeds aren’t required.

Some speakers mentioned that 5G could allow augmented reality technology to be used during surgical operations. Technology of this sort might be developed eventually, but I’m awfully skeptical that we’ll see next-generation wireless revolutionizing surgery in the next decade. There are hurdles in the way of that technology being used on a large scale well beyond limitations in wireless speeds and latency.

I also heard a lot about the potential 5G has to improve education. None of it made sense to me. Schools are buildings with wired internet. Wired internet can be awfully fast. Limitations in mobile speeds are not holding back education.

When I mentioned my skepticism about 5G use cases to Chris Pearson, president of 5G Americas, he emphasized that it might not be possible to predict many of the best use cases ahead of time. It’s a good point. Over the last few weeks, I’ve been trying to think through new use cases that emerged after the 3G to 4G transition. A lot of existing applications improved in a predictable manner. E.g., video streaming became a lot better with 4G speeds. Were there a lot of new, unanticipated use cases enabled by 4G? I’m not sure.

While I have a lot of skepticism about proposed 5G use cases, I’m still awfully excited about 5G. One specific use case that the conference made me more optimistic about is fixed wireless (delivering internet to buildings via wireless transmission instead of wired transmission). In a future post, I’ll explain why I find fixed wireless so exciting.

5G & prices

Could 5G lead to drastic declines in the cost consumers pay for each gigabyte of data? Unfortunately, I heard almost nothing on the topic.

In a panel that included executives from Verizon and AT&T, the moderator asked about 5G monetization and pricing. Nicki Palmer of Verizon and Igor Glubochansky of AT&T both dodged the question with politician-like non-answers.

5G timelines

In several different presentations, representatives of U.S. network operators suggested that they would have large-scale 5G networks deployed within a year or two. If these claims are taken seriously, I think they should be interpreted as indicating that some 5G technologies will be deployed on a large scale (e.g., the 5G air interface) alongside 4G technology. I think we are still several years away from large-scale, fully 5G networks.

Network Evaluation Should Be Transparent

Christian

Several third-party firms collect data on the performance of U.S. wireless networks. Over the last few months, I’ve tried to dig deeply into several of these firms’ methodologies. In every case, I’ve found the public-facing information to be inadequate. I’ve also been unsuccessful when reaching out to some of the firms for additional information.

It’s my impression that evaluation firms generally make most of their money by selling data access to network operators, analysts, and other entities that are not end consumers. If this was all these companies did with their data, I would understand the lack of transparency. However, most of these companies publish consumer-facing content. Often this takes the form of awards granted to network operators that do well in evaluations. It looks like network operators regularly pay third-party evaluators for permission to advertise the receipt of awards. I wish financial arrangements between evaluators and award winners were a matter of public record, but that’s a topic for another day. Today, I’m focusing on the lack of transparency around evaluation methodologies.

RootMetrics collects data on several different aspects of network performance and aggregates that data to form overall scores for each major network. How exactly does RootMetrics do that aggregation?

The results are converted into scores using a proprietary algorithm.1

I’ve previously written about how difficult it is to combine data on many aspects of a product or service to arrive at a single, overall score. Beyond that, there’s good evidence that different analysts working in good faith with the same raw data often make different analytical choices that lead to substantive differences in the results of their analyses. I’m not going take it on faith that RootMetrics’ proprietary algorithm aggregates data in a highly-defensible manner. No one else should either.

Opensignal had a long history of giving most of its performance awards to T-Mobile.2 Earlier this year, the trend was broken when Verizon took Opensignal’s awards in most categories.3 It’s not clear why Verizon suddenly became a big winner. The abrupt change strikes me as more likely to have been driven by a change in methodology than a genuine change in the performance of networks relative to one another. Since little is published about Opensignal’s methodology, I can’t confirm or disconfirm my speculation. In Opensignal’s case, questions about methodology are not trivial. There’s good reason to be concerned about possible selection bias in Opensignal’s analyses. Opensignal’s Analytics Charter states:4

Our analytics are designed to ensure that each user has an equal impact on the results, and that only real users are counted: ‘one user, one vote’.

Carriers will differ in the proportion of their subscribers that live in rural areas versus densely-populated areas. If the excerpt from the analytics charter is taken literally, it may suggest that Opensignal does not control for differences in subscribers’ geography or demographics. That could explain why T-Mobile has managed to win so many Opensignal awards when T-Mobile obviously does not have the best-performing network at the national level.

Carriers advertise awards from evaluators because third-parties are perceived to be credible. The public deserves to have enough information to assess whether third-party evaluators merit that credibility.