Nielsen’s Methodology

Limited public information

Nielsen conducts a handful of activities to evaluate the U.S. wireless service market. Unfortunately, I’ve found it difficult to get much information and Nielsen’s methods or results. I did not receive a response when I reached out to Nielsen requesting additional information.

My understanding of Nielsen’s work in the wireless market is based largely on a handful of brief web pages published by Nielsen and a short slide deck from a few years ago that I found on the FCC’s website.

Given the lack of details and potentially out-of-date sources, my understanding of Nielsen’s methodology is mediocre at best. It’s possible that Nielsen does not currently conduct all of the types of testing I discuss on this page.

Activities

  • Drive testing — Nielsen has a fleet of vans that it drives around to collect performance data on major networks.1
  • Bill analysis — Nielsen claims to analyze cell phone bills from over 65,000 U.S. wireless customers.2
  • Analyzing carrier server logs — I don’t know what this entails.3
  • Surveying — Nielsen claims to survey consumers in a handful of different ways.4
  • Mobile video quality assessment — Nielsen has performed tests to assess mobile video quality.5
  • Actual-use panel — Nielsen claims to passively monitor about 45,000 Android devices used by consumers who voluntarily participate in a panel.6

Results

I have found almost no information about Nielsen’s results. In 2018, Sprint cited its own analysis of Nielsen data when claiming Sprint’s service was nearly as reliable as Verizon’s service.7 As of April 2019, Sprint cites its analysis of Nielsen data when making claims about Sprint’s performance in specific metro areas.8

Open questions

  • Nielsen claims that many of its mobile device metrics are “the market standard”9 and that its on-device meter is “best-in-class.”10 I am skeptical about both of these claims. Can Nielsen provide additional information in support of either claim?

  • Nielsen strikes me as far less transparent than other entities engages in similar evaluation activities. Is there a good rationale for why being transparent would be more harmful for Nielsen than other entities evaluating network performance?
  • How does Nielsen’s drive test methodology differ from RootMetrics’ drive test methodology?
  • Why doesn’t Nielsen share more results publicly?
  • Does Sprint compensate Nielsen in exchange for permission to discuss its analyses of Nielsen data publicly?

Footnotes

  1. Nielsen’s Drive Test web page on 4/1/2019 stated the following:
    “Drive Test employs 25+ vans that drive more than 1.5 million miles a year in nearly 200 U.S. markets. The vans act as a mobile laboratory, constantly testing the capability and performance of the voice, data, and VoLTE networks of the top five U.S. mobile carriers.”
  2. “Nielsen uses a broad range of measurement tools to help companies make the most of their investments in mobile, including…Analyzing the cellphone bills of more than 65,000 mobile subscribers in the U.S.”
    From Nielsen’s web page covering mobile solutions (archived here). Accessed 4/1/2019.
  3. “Nielsen uses a broad range of measurement tools to help companies make the most of their investments in mobile, including…Analyzing carrier server logs to understand feature phone usage behavior.”
    From Nielsen’s web page covering mobile solutions (archived here). Accessed 4/1/2019.
  4. “Nielsen uses a broad range of measurement tools to help companies make the most of their investments in mobile, including…Surveying mobile consumers via telephone, in-person and online surveys.”
    From Nielsen’s web page covering mobile solutions (archived here). Accessed 4/1/2019.
  5. “Today, Nielsen (NYSE:NLSN) announced the launch of Nielsen Mobile Video Performance, the industry’s first and most complete video performance evaluation solution for IP-based video content delivery. Developed for mobile operators, Internet service providers, device manufacturers and content providers, Nielsen Mobile Video Performance evaluates streaming video quality on mobile and Wi-Fi networks, benchmarks data across the industry and rates individual players on the factors which contribute most to positive customer experiences.”
    From a 12/7/17 press release. Archived here.
  6. “The product employs proprietary metering technology to passively measure a geographically representative opt-in panel of Android U.S. smartphone owners that captures over 400 million data points each month. The passive meter runs 24/7 in the background of the device, continuously capturing data speeds and hundreds of other metrics across different file sizes and applications. With a sample of 45,000 devices at the national level across the top 41 cities in the U.S., NMP measures the key metrics related to consumers’ mobile experience.”
    From Nielsen’s web page on network performance solutions (archived here). Accessed 4/1/2019.
  7. On this archived web page from 2018, Sprint cites its analysis of Nielsen data when claiming: “Our network reliability is within 1% of Verizon’s and it keeps getting better.”
  8. See the footnotes to the claims made on this page on Sprint’s website as of 4/1/2019.
  9. “Nielsen metrics for mobile devices (including “connected” devices like iPads, Kindles and tablets) are already the market standard for market share, consumer satisfaction, device share, service quality, revenue share, advertising effectiveness, audience reach and other key indicators in the mobile marketplace.”
    From Nielsen’s web page covering mobile solutions (archived here). Accessed 4/1/2019.
  10. “Using best-in-class on-device passive metering, Nielsen delivers unrivaled, accurate benchmarking of the real world network experience in the hands of the end user—your mobile consumer.”
    From Nielsen’s web page on network performance solutions (archived here). Accessed 4/1/2019.