Issues with Consumer Reports’ 2017 Cell Phone Plan Rankings

Consumer Reports offers ratings of cellular service providers based on survey data collected from Consumer Reports subscribers. Through subscriber surveying in 2017, Consumer Reports collected data on seven metrics:¹

1. Value
2. Data service quality
3. Voice service quality
4. Text service quality
5. Web service quality
6. Telemarketing call frequency
7. Support service quality

The surveys collected data from over 100,000 subscribers.² I believe Consumer Reports would frown upon a granular discussion of the exact survey results, so I’ll remain vague about exact ratings in this post. If you would like to see the full results of their survey, Consumer Reports subscribers can do so here.

Survey results

Results are reported for 20 service providers. Most of these providers are mobile virtual network operators (MVNOs). MVNOs don’t operate their own network hardware but make use of other companies’ networks. For the most part, MVNOs use networks provided by the Big Four (Verizon, Sprint, AT&T, and T-Mobile).

Interestingly, the Big Four do poorly in Consumer Reports’ evaluation. Verizon, AT&T, and Sprint receive the lowest overall ratings and take the last three spots. T-Mobile doesn’t do much better.

This is surprising. The Big Four do terribly, even though MVNOs are using the Big Four’s networks. Generally, I would expect the Big Four to offer network access to their direct subscribers that is as good or better than the access that MVNO subscribers receive.

It’s possible that the good ratings can be explained by MVNOs offering prices and customer service far better than the Big Four—making them deserving of the high ratings for reasons separate from network quality.

Testing the survey’s validity

To test the reliability of Consumer Reports methodology, we can compare MVNOs to the Big Four using only the metrics about network quality (ignoring measures of value, telemarketing call frequency, and support quality). In many cases, MVNOs use more than one of the Big Four’s networks. However, several MVNOs use only one network, allowing for easy apples-to-apples comparisons.³

• Boost Mobile is owned by Sprint.
• Virgin Mobile is owned by Sprint.
• Circket Wireless is owned by AT&T.
• MetroPCS is owned by T-Mobile.
• GreatCall runs exclusively on Verizon’s network.
• Page Plus Cellular runs exclusively on Verizon’s network.

When comparing network quality ratings between these MVNOs and the companies that run their networks:

• Boost Mobile’s ratings beat Sprint’s ratings in every category.
• Virgin Mobile’s ratings beat Sprint’s ratings in every category.
• Cricket Wireless’s ratings beat or tie AT&T’s ratings in every category.
• MetroPCS’s ratings beat or tie T-Mobile’s ratings in every category.
• GreatCall doesn’t have a rating for web quality due to insufficient data. GreatCall’s ratings match or beat Verizon in the other categories.
• Page Plus Cellular doesn’t have a rating for web quality due to insufficient data. Page Plus’ ratings match or beat Verizon in the other categories.

Taken at face value, these are odd results. There are complicated stories you could tell to salvage the results, but I think it’s much more plausible that Consumer Reports’ surveys just don’t work well for evaluating the relative quality of cell phone service providers.

Why aren’t the results reliable?

I’m not sure why the surveys don’t work, but I see three promising explanations:

• Metrics may not be evaluated independently. For example, consumers might take a service’s price into account when providing a rating of its voice quality.
• Lack of objective evaluations. Consumers may not provide objective evaluations. Perhaps consumers are aware of some sort of general stigma about Sprint that unfairly affects how they evaluate Sprint’s quality (but that same stigma may not be applied to MVNOs that use Sprint’s network).
• Selection bias. Individuals who subscribe to one carrier are probably, on average, different from individuals who subscribe to another carrier. Perhaps individuals who have used Carrier A tend to use small amounts of data and are lenient when rating data service quality. Individuals who have used Carrier B may get more upset about data quality issues. Consumer Cellular took the top spot in the 2017 rankings. I don’t think it’s coincidental that Consumer Cellular has pursued branding and marketing strategies to target senior citizens.⁴

Consumer Reports’ website gives the impression that their cell phone plan rankings will be reliable for comparison purposes.⁵ They won’t be.

The ratings do capture whether survey respondents are happy with their services. However, the ratings have serious limitations for shoppers trying to assess whether they’ll be satisfied with a given service.

I suspect Consumer Reports’ ratings for other product categories that rely on similar surveys will also be unreliable. However, the concerns I’m raising only apply to a subset of Consumer Reports’ evaluations. A lot of Consumer Reports’ work is based on product testing rather than consumer surveys.

1. “Value pertains to the quality and extent of the cell service received for the money. Data service indicates overall experience (e.g., cost, speed, reliability) with the data service. Voice refers to the quality and availability of the network for calls. Text refers to the reliability of sending and receiving text messages. Web is a rating of the availability, speed, and reliability of the provider’s cell phone internet access. Telemarketing call frequency refers to the provider’s passive or active efforts to reduce the number of telemarketing calls to the subscriber. Support indicates the overall experience with customer service.”
   From Consumer Reports Cell Phone Plan Ratings (subscription required). Accessed 2/6/2018.

   I don’t understand why data service and web service were evaluated with separate metrics. Unfortunately, Consumer Reports doesn’t offer much information about the methodology behind the survey.

2. “Scores are based on ratings from 119,772 Consumer Reports subscribers reporting on experiences with their current cell phone service provider as of Summer 2017.”
   From Consumer Reports Cell Phone Plan Ratings (subscription required). Accessed 2/7/2018.
3. I used Wikipedia’s List of United States mobile virtual network operators article (archived copy) for this information. Note that Credo Mobile is now run exclusively on Verizon’s network. I did not include it in this list since Credo Mobile used Sprint’s network until mid-2016. For more information see FierceWireless’ article on the transition (archived copy). Credo Mobile ties Verizon on two network quality metrics and loses on two. Credo Mobile beats or ties Sprint on all network metrics.
4. For example, Consumer Cellular places a lot of emphasis on its partnership with AARP.
5. Consumer Reports presents the ratings in a ranked list with different colors assigned to different scores. Further indicating that Consumer Reports intends the rankings to be used for comparative purposes, it notes: “Differences of fewer than 5 points are not meaningful.”. From Consumer Reports Cell Phone Plan Ratings (subscription required). Accessed 10/9/2018.