Tutela’s Methodology

Data collection methodology

Tutela collects mobile network performance data via code included in apps it partners with. As of March 2019, Tutela claims to have partnerships with over 3,000 app developers.1 As I understand it, Tutela’s download tests typically involve transferring a 2MB file2 and upload tests involve a 1MB file.3 Given the relatively small file sizes, the speeds recorded in Tutela’s tests may tend to be lower than the speeds recorded in tests that make use of large files.4 Tutela also runs tests where tiny files are transferred to assess latency, jitter, and packet loss.5

Aggregation and results reporting

Tutela publishes periodic reports on its website. As of April 2019, the last report covering U.S. networks came out in March 2019 and discussed coverage quality in the 10 largest U.S. cities. In that report, Tutela focuses on how often a network is able to meet its thresholds for “basic” and “excellent” quality. Here are the minimum standards for each quality level:6

I’m a fan of this method for displaying results. I sometimes get frustrated when evaluators emphasize average download speed, which I believe is an overrated metric. I think Tutela’s thresholds are well-selected. At the excellent threshold, a phone should perform well for most common activities. At the basic threshold, a phone might struggle with intense activities like video streaming but should work for messaging and simple web browsing.

Based on these thresholds, Tutela can generate easy-to-interpret graphics like this one:7

Limitations

  • I don’t think Tutela can know when a user is being throttled for excessive data use. This complicates comparisons across networks.
  • Tutela’s methodology may lead to selection bias. People who use apps that conduct Tutela’s tests may differ systematically from the typical cell phone user.
  • Tutela doesn’t appear to say much about the confidence intervals around its estimates. Presumably, the estimates for small MVNOs are noisier than estimates for host operators.

Open questions

  • How often do tests take place? Does frequency vary between partner apps?
  • Do tests often take place concurrently with other data transfer unrelated to the Tutela tests?
  • Does Tutela receive data about times when tests could not occur (e.g., due to there being no connection)? If this data exists, how does it factor into aggregated results?
  • Tutela sometimes claims to have a ton of data.8 However, a recent report for the U.S. only covers the 10 largest cities, and the numbers I’ve seen about the total number of download tests Tutela conducts in the U.S. suggest Tutela may not be in a good position for assessing nationwide coverage quality.9 Does Tutela have enough U.S. data to conduct meaningful evaluations outside of large cities?

Footnotes

  1. “We have partnered with over 3000 third party mobile app developers who have embedded our data collection SDK for Android or iOS into the background of their apps.”
    From Tutela’s Methodology page on 3/30/2019.
  2. “By default, Tutela uses a 2MB file to perform download throughput speed tests.”
    From Tutela’s Download Throughput Test Methodology page as of 3/30/2019.
  3. “By default, Tutela uses a 1MB file to perform upload speed tests.”
    From Tutela’s Upload Throughput Test Methodology page on 3/30/2019.
  4. “Our results differ from other network testing companies which measure the peak performance of networks under ideal conditions (such as downloading a 200 MB file).”
    From Tutela’s March 2019 report on U.S. cities. Archived here.
  5. Tutela describes those tests here.
  6. Taken from this page of Tutela’s website (archived here).
  7. The image comes from Tutela’s March 2019 report on U.S. cities. Archived here.
  8. “Due to the way that the data is collected (in the background of popular end-user mobile applications) Tutela collects a significant number of records each day. In most countries where Tutela publishes reports, there are over 100,000 unique devices providing over 1 million records every day. This large data set enables high-resolution and detailed analysis of different geographies, operators, devices, etc.”
    From Tutela’s Data Collection Methodology page as of 3/30/2019.
  9. “Tutela has examined over three billion network measurements, taken from 1st September 2018 to 28th February 2019 in the US’s 10 largest cities. The records include over 14 million speed tests, 242 million latency tests, and 2.5 billion total records on cellular networks.”
    From Tutela’s March 2019 USA Report