Tag: Selection bias

Abstract map of USA

What Happened To Opensignal’s Reach Metric?

Christian

Say a country has two cellular networks. Network A offers coverage nearly everywhere. Network B offers a much smaller coverage footprint. You could crowdsource data from ordinary subscribers on the networks and track what proportion of the time users are connected to their networks.

If you compare the availability metrics for both networks, I’m unsure what insights you glean. Conventional crowdsourcing doesn’t involve random samples of people within a country. Even if Network B is crummy and has a small coverage footprint, people who use it will tend to live within its coverage footprint. People living elsewhere will opt for Network A and its larger coverage footprint. It’s a classic case of selection bias.

Even if the people you’re crowdsourcing data from represent a random sample of people within a country, a 95% availability metric wouldn’t indicate that a network covers 95% of the country’s land. People concentrate in certain areas. Although urban areas comprise a small fraction of the US, 80% of people live within them.

Opensignal’s Conundrums

Opensignal is arguably the leader in crowdsourced assessments of networks. To Opensignal’s credit, it acknowledges the limitations of its crowdsourced availability metrics:

Our availability metrics are not a measure of a network’s geographical extent. They won’t tell you whether you are likely to get a signal if you plan to visit a remote rural or nearly uninhabited region.

At some point, Opensignal started publishing another coverage metric called “Reach.” As far as I can tell, Opensignal only used this metric in the US for its 5G assessments.1

Here’s how Opensignal explains Reach:

5G Reach measures how users experience the geographical extent of an operator’s 5G network. It analyzes the average proportion of locations where users were connected to a 5G network out of all the locations those users have visited.

Reach addresses part of my concern with the crowdsourced availability metric. Interestingly, the metric doesn’t appear in the most recent reports on 5G performance in the US. I wonder why.2

Trophy

Does T-Mobile Have The Best 5G?

Christian

T-Mobile has started bragging about having the best 5G speeds in Opensignal’s latest report. Here’s an excerpt from today’s press release from T-Mobile:

New independent data from Opensignal, based on real world customer usage from millions of device measurements, shows T-Mobile customers now get the fastest 5G download speeds, fastest 5G upload speeds AND a 5G signal more often than anyone else.

I’ve been critical of selection bias issues inherent in Opensignal’s methodology. I continue to think there are serious selection bias issues with Opensignal’s latest 5G metrics. Still, I don’t think my qualms are significant enough to dismiss T-Mobile’s apparent lead in 5G speeds and 5G coverage. T-Mobile is killing it. Here’s another bit from today’s press release:

With the first and largest nationwide 5G network, T-Mobile’s Extended Range 5G covers more than 280 million people across nearly 1.6 million square miles – offering 2.5x more geographic coverage than AT&T and nearly 4x more than Verizon. With Sprint now part of T-Mobile, the Un-carrier is widening its lead, using dedicated spectrum to bring customers with capable devices download speeds of around 300 Mbps and peak speeds up to 1 Gbps. The Un-carrier’s Ultra Capacity 5G already reaches more than 1,000 cities and towns and covers 106 million people.

Early in its 5G rollout, T-Mobile relied on low-frequency spectrum around 600MHz. While this spectrum was great for coverage, it had lousy speed potential. In 2020, T-Mobile put a lot of effort into bragging about how it led the nation in 5G coverage. While the bragging was technically accurate, the whole thing was bullshit in practical terms. 5G delivered with T-Mobile’s low-frequency spectrum was often slower than a typical 4G connection.

Recently, T-Mobile started rolling out large-scale 5G deployments using mid-band spectrum. T-Mobile’s mid-band 5G now covers about a third of Americans. Mid-band 5G actually delivers speeds that are substantially better than consumers are used to with 4G.

Verizon is still crushing the competition in terms of coverage with ultra-fast, millimeter wave 5G. However, Verizon’s achievements with millimeter wave don’t have much value for consumers yet. Even Verizon’s millimeter wave coverage is lackluster, and practical applications for ultra-fast cellular speeds are rare.

While I think T-Mobile legitimately holds the top spot for 5G coverage and average 5G speeds, I also think Verizon will overtake T-Mobile as 5G rollouts reach more mature stages. In my view, the interesting thing to watch will be whether T-Mobile or AT&T ends up with the second-place spot in the 5G competition.1 T-Mobile’s spectrum holdings and financial position may give the network a significant edge over AT&T.

Woman making a skeptical face

Opensignal Released a New Report – I’m Skeptical

Christian

Opensignal just released a new report on the performance of U.S. wireless networks. The report ranks major U.S. networks in five categories based on crowdsourced data:

  • 4G availability
  • Video experience
  • Download speed experience
  • Upload speed experience
  • Latency experience

Verizon took the top spot for 4G availability and video experience. T-Mobile came out on top for both of the speed metrics. T-Mobile and AT&T shared the top placement for the latency experience metric.

Selection bias

I’ve previously raised concerns about selection bias in Opensignal’s data collection methodology. Opensignal crowdsources data from typical users. Crowdsourcing introduces issues since there are systematic differences between the typical users of different networks. Imagine that Network A has far more extensive coverage in rural areas than Network B. It stands to reason that Network A likely has more subscribers in rural areas than Network B. Lots of attributes of subscribers vary in similar ways between networks. E.g., expensive networks likely have subscribers that are wealthier.

Analyses of crowdsourced data can capture both (a) genuine differences in network performance and (b) differences in how subscribers on each network use their devices. Opensignal’s national results shouldn’t be taken too seriously unless Opensignal can make a compelling argument that either (a) its methodology doesn’t lead to serious selection bias or (b) it’s able to adequately adjust for the bias.

Speed metrics

Opensignal ranks carriers based on average download and upload speeds. In my opinion, average speeds are overrated. The portion of time where speeds are good enough is much more important than the average speed a service offers.

Opensignal’s average download speed results are awfully similar between carriers:

  • Verizon – 22.9 Mbps
  • T-Mobile – 23.6 Mbps
  • AT&T – 22.5 Mbps
  • Sprint – 19.2 Mbps

Service at any of those speeds would be sufficient for almost any activities people typically use their phones for. Without information about how often speeds were especially low on each network, it’s hard to come to conclusions about differences in the actual experience on each network.