I’ve started looking into the methodologies used by entities that collect cell phone network performance data. I keep seeing an emphasis on average (or median) download and upload speeds when data-service quality is discussed.
- Opensignal bases it’s data-experience rankings exclusively on download and upload speeds.1
- Tom’s Guide appears to account for data-quality using average download and possibly upload speeds.2
- RootMetrics doesn’t explicitly disclose how it arrives at final data-performance scores, but emphasis is placed on median upload and download speeds.3
It’s easy to understand what average and median speeds represent. Unfortunately, these metrics fail to capture something essential—variance in speeds.
For example, OpenSignal’s latest report for U.S. networks shows that Verizon has the fastest average download speed of 31 Mbps in the Chicago area. AT&T’s average download speed is only 22 Mbps in the same area. Both those speeds are easily fast enough for typical activities on a phone. At 22 Mbps per second, I could stream video, listen to music, or browse the internet seamlessly. For the rare occasion where I download a 100MB file, Verizon’s network at the average speed would beat AT&T’s by about 10.6 seconds.4 Not a big deal for something I do maybe once a month.
On the other hand, variance in download speeds can matter quite a lot. If I have 31 Mbps speeds on average, but I occasionally have sub-1 Mbps speeds, it may sometimes be annoying or impossible to use my phone for browsing and streaming. Periodically having 100+ Mbps speeds would not make up for the inconvenience of sometimes having low speeds. I’d happily accept a modest decrease in average speeds in exchange for a modest decrease in variance.5
Footnotes
- Props to Opensignal for using a more transparent methodology than either Tom’s Guide or RootMetrics.
- It’s not clear to me how Tom’s Guide arrives at its final performance ratings for each carrier, but their discussion of data-performance test results does not suggest any data-quality metrics besides average speeds are being carefully tracked.
- “After the tests are evaluated for accuracy, the results are converted into scores using a proprietary algorithm.”
From RootMetrics’ methodology page on 3/19/2019 (archived here).The only metrics I’ve seen mentioned in RootMetrics notes on data performance are median upload speed and median download speed.
- (100/22 – 100/31) * 8 = 10.6
- Opensignal, fortunately, has a 4G-availability metric that can give some hints about variance in speeds. As far as I know, RootMetrics and Tom’s Guide do not publish similar, easy-to-interpret availability metrics.
