In October, the network evaluator Tutela released its USA State of MVNOs report. Most network evaluators only assess the performance of the Big Four carriers (AT&T, T-Mobile, Sprint, and Verizon), so it’s interesting to see Tutela assessing a wider range of carriers.
Near the beginning of the report, Tutela shares some reflections on how the MVNO landscape is changing:
The approach Tutela used to evaluate MVNOs was in line with its usual methodology. The company crowdsourced performance data from typical consumers with the help of code embedded in Tutela’s partners’ apps. In the new report, Tutela primarily considers how well MVNOs performed in regions where at least three of the big four networks offer coverage. Tutela calls these core coverage areas.
Within core coverage areas, Tutela calculates the amount of time subscribers have service that exceeds two different quality thresholds. When service exceeds the “excellent” threshold, subscribers should be able to do highly demanding things like streaming high-definition video or downloading large files quickly. When service exceeds the “core” threshold, subscribers should be able to carry out typical activities like browsing or streaming music without trouble, but performance issues may be encountered with demanding activities.
Here’s Tutela’s visualization of the main results:
A chart of median download speeds shows a similar ranking among carriers:
The results aren’t too surprising. Verizon MVNOs come out near the top of the hierarchy, while Sprint MVNOs tend to come out near the bottom. Cricket Wireless has a good score for the core threshold but does poorly in terms of the excellent threshold. That outcome makes sense since Cricket throttles maximum speeds.
Possible selection bias
I often write about how assessments of network performance that use crowdsourced data may be vulnerable to selection bias. These results from Tutela are no exception. In particular, I wonder if the results are skewed based on how high-quality phones used with different carriers tend to be. In general, newer or more expensive phones have better network hardware than older or cheaper phones.
Xfinity Mobile takes the top spot in the rankings. Xfinity Mobile is a new-ish carrier and is restrictive about which phones are eligible for use with the service. I would guess the average phone used with Xfinity Mobile is a whole lot newer and more valuable than the average phone used with TracFone. Similar arguments could be made for why Spectrum or Google Fi may have an advantage.
To Tutela’s credit, the company acknowledges the possibility of selection bias in at least one case:
Several MVNOs offer access to Wi-Fi hotspots in addition to cellular networks. I’ve been curious how much data carriers send over Wi-Fi, and Tutela’s results give an estimate. While Xfinity Mobile appears to have sent the largest share of its data via hotspots, it’s a smaller share than I expected:
Tutela also shares a graph comparing hotspot usage among different carriers:
There were a few other bits of the report that I found especially interesting. In one section, the report’s authors reflect on the fast growth of MVNOs run by cable companies:
In another part of the report, Tutela shares a map displaying the most common host network that Google Fi subscribers access. It looks like there are a decent number of areas where Sprint or U.S. Cellular provide the primary host network:
- The excerpt comes from page 1 of the report.
- Tutela’s approach may actually be biased against AT&T and Verizon, the networks with the largest coverage profiles. I expect a lot of data from areas where Verizon and AT&T dominate is ignored since those areas won’t always qualify as core coverage areas.
- The visual comes from page 5 of the report.
- From page 14 of the report.
- The excerpt comes from page 3 of the report.
- The graph comes from page 18 of the report.
- The excerpt comes from page 15 of the report.
- The visual comes from page 13 of the report.