Major T-Mobile Network Outage

A major T-Mobile network outage is going on right now.

T-Mobile’s outage

T-Mobile subscribers scattered around the U.S. have been reporting that calls and texts have been failing over the last few hours.

The outage appears to be affecting more than T-Mobile’s direct customers. Subscribers with mobile virtual network operators that run over T-Mobile’s network are also reporting problems.

T-Mobile is aware of the issues. Neville Ray, T-Mobile’s President of Technology, shared the following tweet:

Neville Ray's tweet acknowledging the outage

While Ray describes it as a “voice and data issue,” most of the reports I’ve seen so far have been about issues with calls and texts.

Other companies

The website Downdetector is recording an atypical number of issues with a bunch of wireless carriers, Facebook, Instagram, Twitch, and a handful of other internet companies. It’s unclear if Verizon and AT&T are experiencing any issues of their own. Subscribers with these carriers may only be experiencing issues when they try to contact people on T-Mobile’s network.

Prognosis

At this time, I have not seen any word on what is causing the outage. I have also not seen any clear indication about when the issue is expected to be resolved.

I plan to make updates as I learn more. This may end up being one of the largest network outages in the U.S. in years.


Monday Night Update:

At 11:03pm MT, Neville Ray tweeted that the issues have been resolved:

Voice and text services are now restored. Thank you for your patience as we fixed the issues. We sincerely apologize for any and all inconveniences.

T-Mobile’s Data Maximizer Can’t Be Turned Off On Connect Plans

Some of T-Mobile’s plans come with a setting called “Data Maximizer.” While Data Maximizer is turned on, most video traffic will be throttled to about 480p quality. Data Maximizer reduces the load on T-Mobile’s network and can help subscribers with limited data allowances to conserve their data.

T-Mobile’s Connect plans supposedly have Data Maximizer turned on by default. Subscribers can also supposedly turn off the feature. On portions of T-Mobile’s website related to the Connect plans, there are disclosures like this one:1

Video typically streams on your T-Mobile device at DVD quality (480p) with Data Maximizer. You may disable Data Maximizer at any time.

I’ve recently been testing a Connect plan. Sure enough, a test I ran with the app Wehe confirmed that some video traffic was being throttled:

Results from video throttling tests

When I tried to turn off Data Maximizer, I ran into trouble. I first tried to disable the setting from within my T-Mobile online account. While subscribers on some of T-Mobile’s other plans can turn off Data Maximizer through an online process, that didn’t seem possible with the Connect plan.

I went ahead and called T-Mobile to see if a support agent could turn off the setting. At first, the support agent looked into it and told me Data Maximizer didn’t seem to affect Connect plans. I explained that T-Mobile’s website suggested otherwise and that my video traffic appeared to be throttled. The agent seemed to agree something strange was going on. She said she’d put in a ticket to have someone at T-Mobile look into the issue.

I don’t think it would be a big deal if Connect subscribers couldn’t turn off Data Maximizer. 480p video is, in my opinion, very watchable. Conserving data while streaming can be really beneficial on plans that don’t have large data allotments. Still, the fact that I ran into this issue surprised me. The Connect plans will probably be popular. I’m surprised some sort of quality review didn’t catch this issue before the plans were released.

T-Mobile & Sprint Merger Officially Closes

To no one’s surprise, the merger between T-Mobile and Sprint finally closed this morning.1

With the closure of the merger, John Legere is stepping down from his position as T-Mobile’s CEO. Legere will be replaced by Michael Sievert, who was until now the COO of T-Mobile.

I continue to think the merger is going to be bad for consumers over the long term. However, we should see some things that are good for consumers in the short term, like the recently released T-Mobile Connect plans.

AT&T Matches T-Mobile’s $15 Plan

Last week, T-Mobile began offering a plan with unlimited minutes, unlimited texts, and 2GB of data for only $15 per month. A few days later, AT&T responded by offering its own $15 per month plan with unlimited minutes, unlimited texts, and 2GB of data.

Plan terms

I dug around to learn about the plan’s policies. Here are my impressions at the moment:

  • New subscribers need an AT&T prepaid SIM (costs $4.99).
  • Mobile hotspot and tethering are permitted.
  • Unused data rolls over for one month.1
  • Taxes and fees are not included in the $15 base price.
  • Unlimited international texting to over 100 countries is included.
  • Streaming video traffic will be throttled by default, but subscribers can turn the throttle off.
  • The plan is not eligible for AT&T’s discounts for paperless billing or automatic payments.
  • Data has a soft cap—once 2GB of regular data has been used, additional data can be used at significantly reduced speeds.

Limited time offer

AT&T has repeatedly described its new $15 plan as a promotion and a limited time offer. I don’t know when the plan will cease to be available.2

T-Mobile Connect vs. AT&T’s plan

AT&T’s $15 plan has a handful of substantial advantages over the $15 T-Mobile Connect plan:

  • AT&T has better nationwide coverage.
  • SIM cards are cheaper from AT&T ($5 vs. $10).
  • Only AT&T offers data rollover.
  • AT&T has a soft cap on data, while T-Mobile Connect has a hard cap.

However, it’s not clear how long AT&T’s plan will be around. People who take advantage of AT&T’s offer today won’t necessarily get the same great deal each month for the foreseeable future. On the other hand, it looks like the T-Mobile Connect plans will continue to be available to new and existing subscribers for years.

Update: The $15 plan from AT&T is no longer available.

Image representing the idea of a network

T-Mobile Connect Launches With $15 Per Month Service

In November 2019, T-Mobile committed to offering a new, budget plan if a merger between T-Mobile and Sprint was approved:1

[The New T-Mobile will offer a] competitive $15 per month prepaid option– half the price of the lowest T-Mobile plan today – to EVERYONE, especially lower-income consumers.

T-Mobile launched that plan last week. I didn’t expect it to be available so soon. It looks like the plan’s rollout was accelerated in response to the coronavirus:2

T-Mobile Connect was announced in November of 2019 as part of 5G for Good – the first planned Un-carrier moves for the proposed New T-Mobile – but in response to customer needs in these trying times, the Un-carrier is launching it this week.

Plan versions

The new T-Mobile Connect plans come with unlimited minutes and texts. Subscribers have two options for their data allotments:

  • 2GB plan – base price of $15 per month
  • 5GB plan – base price of $25 per month

Taxes and fees are not included in the base price of either plan. T-Mobile plans to boost data allotments by 500MB each year:3

T-Mobile Connect also has an Annual Data Upgrade, giving customers an additional 500MB of monthly data, every year, at no additional cost, for the next five years.

Plan terms

I’ve read through a lot of detail’s T-Mobile’s published about the plan. Here are my biggest takeaways from that reading:

  • New subscribers must purchase a T-Mobile SIM card for $10.4
  • International roaming is not available for T-Mobile Connect plans.
  • Data has a hard cap. Subscribers who’ve used all their regular data cannot continue to use the internet at reduced speeds.
  • Up to five T-Mobile Connect lines can be combined on a family plan. Line prices stay constant regardless of the number of people on a plan.
  • Mobile hotspot and tethering are permitted.
  • Video is throttled to 480p by default, but subscribers can turn off throttling.

Metro’s offer

T-Mobile’s flanker brand, Metro, will temporarily offer a plan similar to T-Mobile’s $15 plan:5

For the next two months, Metro is offering a $15 plan – that’s half the price of the current most affordable plan. For 60 days after customers activate, it’s just $15 per month for unlimited talk and text plus 2GB of high-speed smartphone data.

Unless I’m missing something, it seems like anyone who’s torn between the T-Mobile’s $15 plan and Metro’s plan should go with T-Mobile.

My take

T-Mobile’s Connect plans will be a great option for budget-sensitive consumers that don’t use a ton of data. Based on what I’ve seen so far, it looks like subscribers on the Connect plans will have a level of priority on par with most of T-Mobile’s postpaid subscribers. If my speculation is accurate, that may give the Connect plans a big advantage over the budget-friendly plans offered by many of the MVNOs that operate over T-Mobile’s network (e.g., Mint Mobile).

Earlier today, I placed an order for the $15 per month T-Mobile Connect plan. I’ll write more about it once I’ve had a chance to trial the service.

T-Mobile & Sprint Merger Rumored To Be Headed For Approval

Several state attorneys general have been suing to stop a merger between T-Mobile and Sprint. Rumors came out earlier this evening that the judge presiding over the case is planning to rule in favor of the merger. Here’s a bit from a Wall Street Journal article:

A federal judge is expected to approve T-Mobile US Inc.’s merger with Sprint Corp., according to people familiar with the matter, clearing the way for the two wireless rivals to combine and overcoming a state antitrust challenge.

The rumors are almost certainly correct. Sprint’s stock soared in after-hours trading. The market closed with Sprint trading at close to $4.80. Since then, the stock has been trading for almost 70% more at over $8 per share:1

Sprint Stock Price After Merger

T-Mobile’s stock experienced a more modest after-hours rise from about $85 per share to slightly over $90 per share:

T-Mobile stock price graph after merger with Sprint

I’m planning to write something more detailed once the news is made official and the companies involved release statements.

Pink-Out

Deutsche Telekom (DT), the parent company of T-Mobile, has been making legal threats against companies that use the color magenta in their branding. DT has gone after companies outside of the telecom industry. DT has even tried to force companies to stop using shades of magenta that are different from the shade it uses. TechCrunch has a good article covering the ridiculous story in more detail.

In a funny turn of events, Itamar Kestenbaum, a software engineer at one of the companies DT has threatened, released a Google Chrome extension called Pink-out. Here’s how the app is described (emphasis mine):

Experience the web according to trademark trolls. Deutsche Telekom (aka, T-Mobile’s parent) is out here telling other companies they can’t use pink…so this Chrome Extension makes sure you’re pink-compliant and removes it from all your browsing pages on the web…This extension is free – like the color pink should be.

FCC Reveals Misleading Coverage Claims

On Wednesday, the FCC released a fascinating report related to the Mobility Fund Phase II (MF-II). The MF-II is a planned program to provide federal funding for network build-outs in rural areas that are underserved by 4G coverage.

To determine which geographic areas were underserved, the FCC requested coverage maps and data from network operators. After reviewing the data and allowing outside entities to challenge the datas’ reliability, the FCC became concerned about the accuracy of the information shared by T-Mobile, U.S. Cellular, and Verizon. The FCC decided to conduct its own performance tests and compare the results of its tests to the information the network operators provided. Here’s what the agency found:1

Through the investigation, staff discovered that the MF-II coverage maps submitted by Verizon, U.S. Cellular, and T-Mobile likely overstated each provider’s actual coverage and did not reflect on-the-ground performance in many instances. Only 62.3% of staff drive tests achieved at least the minimum download speed predicted by the coverage maps—with U.S. Cellular achieving that speed in only 45.0% of such tests, T-Mobile in 63.2% of tests, and Verizon in 64.3% of tests…In addition, staff was unable to obtain any 4G LTE signal for 38% of drive tests on U.S. Cellular’s network, 21.3% of drive tests on T-Mobile’s network, and 16.2% of drive tests on Verizon’s network, despite each provider reporting coverage in the relevant area.

Incentives

When considering the accuracy of coverage maps, I try to think about the incentives network operators face. When advertising to consumers, network operators often have an incentive to overstate the extent of their coverage. However, incentives can run in the opposite direction in other situations. For example, when trying to get approval for a merger between Sprint and T-Mobile, Sprint had incentives to make its 4G coverage profile look limited and inferior to the coverage profiles of other nationwide networks.2

I’m not well-informed about the MF-II, so I don’t feel like I have a good grasp of all the incentives at play. That said, it’s not clear that all network operators would have an incentive to overstate their coverage. A network operator that claimed to offer coverage in an area it didn’t cover may limit competitors’ access to subsidies in that area. However, a network operator erroneously claiming to cover an area may prevent itself from receiving subsidies in that area.

Challenges

After network operators submitted coverage information to the FCC, a number of entities, including both governments and network operators, were allowed to challenge the validity of coverage information submitted by others. Here’s a bit more detail about the challenge process:3

After release of the map of presumptively eligible areas, mobile service providers, state, local, and Tribal government entities, and other interested parties granted a waiver were eligible to submit challenges in the challenge process via an online system operated by USAC. Challengers that requested access to the USAC MF-II Challenge Portal were able to access the provider-specific coverage maps, after agreeing to keep the coverage data confidential, and to file challenges to providers’ coverage claims by submitting speed test data. Challengers were required to conduct speed tests pursuant to a number of standard parameters using specific testing methods on the providers’ pre-approved handset models. The Commission adopted the requirement that challengers use one of the handsets specified by the provider primarily to avoid inaccurate measurements resulting from the use of an unsupported or outdated device—e.g., a device that does not support all of the spectrum bands for which the provider has deployed 4G LTE…During the eight-month challenge window, 106 entities were granted access to the MF-II Challenge Portal. Of the 106 entities granted access to the MF-II Challenge Portal, 38 were mobile service providers required to file Form 477 data, 19 were state government entities, 27 were local government entities, 16 were Tribal government entities, and six were other entities that filed petitions requesting, and were each granted, a waiver to participate.

About a fifth of the participating entities went on to submit challenges:4

21 challengers submitted 20.8 million speed tests across 37 states.

The challenge data often showed failed tests and lackluster speeds in areas where network operators claimed to offer coverage:5

During the challenge process, some parties entered specific concerns into the record. For example:6

Smith Bagley (d/b/a Cellular One) submitted maps of its service area in Arizona overlaid with Verizon’s publicly-stated 4G LTE coverage and the preliminary results of drive tests that Smith Bagley had conducted. Smith Bagley asserted that, for large stretches of road in areas where Verizon reported coverage, its drive testers recorded no 4G LTE signal on Verizon’s network. Smith Bagley argued that the ‘apparent scope of Verizon’s inaccurate data and overstated coverage claims is so extensive that, as a practical matter, the challenge process will not and cannot produce the necessary corrections.’
As part of a public report detailing its experience, Vermont published a map showing its speed test results which contradicted the coverage maps in Vermont of U.S. Cellular, T-Mobile, and Verizon, among others. This map included information on the approximately 187,000 speed tests submitted by Vermont, including download speed, latency, and signal strength. In the report, Vermont detailed that 96% of speed tests for U.S. Cellular, 77% for T-Mobile, and 55% for Verizon failed to receive download speeds of at least 5 Mbps.

After reviewing the challenges, the FCC requested additional information from the five largest network operators (AT&T, T-Mobile, Verizon, Sprint, and U.S. Cellular) to understand the assumptions involved in the networks’ coverage models.

FCC tests

Around the same time the FCC was requesting additional information from network operators, the agency also began its own testing of Verizon, U.S. Cellular, and T-Mobile’s networks. These speed tests took place in 12 states and primarily made use of a drive-testing methodology. As mentioned earlier, analyses of the FCC’s test data suggested that the on-the-ground experience with Verizon, T-Mobile, and U.S. Cellular’s network was much different than the experience that would be expected based on the information the networks provided to the FCC.

What happened?

A lot of the commentary and news articles I’ve seen in response to the FCC’s report seem to conclude that network operators are bullshitters that intentionally lied about the extent of their coverage. I have reservations about fully accepting that conclusion. Accurately modeling coverage is difficult. Lots of factors affect the on-the-ground experience of wireless subscribers. The FCC largely acknowledges this reality in its report:

Providers were afforded flexibility to use the parameters that they used in their normal course of business when parameters were not specified by the Commission. For example, the Commission did not specify fading statistics or clutter loss values, and providers were required to model these factors as they would in the normal course of business.7
Our speed testing, data analyses, and inquiries, however, suggest that some of these differences may be the result of some providers’ models: (1) using a cell edge RSRP value that was too low, (2) not adequately accounting for network infrastructure constraints, including backhaul type and capacity, or (3) not adequately modeling certain on-the-ground factors—such as the local clutter, terrain, and propagation characteristics by spectrum band for the areas claimed to be covered.8

Further supporting the idea that assessing coverage is difficult, the FCC didn’t just find that its tests contradicted the initial information submitted by network operators. The FCC data also contradicted the data submitted by those who challenged network operators’ data:

The causes of the large differences in measured download speed between staff and challenger speed tests taken within the same geographic areas, as well as the high percentage of tests with a download speed of zero in the challenger data, are difficult to determine. Discrepancies may be attributable to differences in test methodologies, network factors at the time of test, differences in how speed tet apps or drive test software process data, or other factors…Given the large differences between challenger and staff results however, we are not confident that individual challenger speed test results provide an accurate representation of the typical consumer on-the-ground experience.9

While the FCC found some of the information submitted by networks to be misleading about on-the-ground service quality, I don’t believe it ended up penalizing any network operators or accusing them of anything too serious.10 Still, the FCC did suggest that some of the network operators could have done better:

Staff engineers, however, found that AT&T’s adjustments to its model to meet the MF-II requirements may have resulted in a more realistic projection of where consumers could receive mobile broadband. This suggests that standardization of certain specifications across the largest providers could result in coverage maps with improved accuracy. Similarly, the fact that AT&T was able to submit coverage data that appear to more accurately reflect MF-II coverage requirements raises questions about why other providers did not do so. And while it is true that MF-II challengers submitted speed tests contesting AT&T’s coverage data, unlike for other major providers, no parties alleged in the record that AT&T’s MF-II coverage data were significantly overstated.11

FCC response

The FCC concluded that it should make some changes to its processes:12

First, the Commission should terminate the MF-II Challenge Process. The MF-II coverage maps submitted by several providers are not a sufficiently reliable or accurate basis upon which to complete the challenge process as it was designed.
Second, the Commission should release an Enforcement Advisory on broadband deployment data submissions, including a detailing of the penalties associated with filings that violate federal law, both for the continuing FCC Form 477 filings and the new Digital Opportunity Data Collection. Overstating mobile broadband coverage misleads the public and can misallocate our limited universal service funds.
Third, the Commission should analyze and verify the technical mapping data submitted in the most recent Form 477 filings of Verizon, U.S. Cellular, and T-Mobile to determine whether they meet the Form 477 requirements. Staff recommends that the Commission assemble a team with the requisite expertise and resources to audit the accuracy of mobile broadband coverage maps submitted to the Commission. The Commission should further consider seeking appropriations from Congress to carry out drive testing, as appropriate.
Fourth, the Commission should adopt policies, procedures, and standards in the Digital Opportunity Data Collection rulemaking and elsewhere that allow for submission, verification, and timely publication of mobile broadband coverage data. Mobile broadband coverage data specifications should include, among other parameters, minimum reference signal received power (RSRP) and/or minimum downlink and uplink speeds, standard cell loading factors and cell edge coverage probabilities, maximum terrain and clutter bin sizes, and standard fading statistics. Providers should be required to submit actual on-the-ground evidence of network performance (e.g., speed test measurement samplings, including targeted drive test and stationary test data) that validate the propagation model used to generate the coverage maps. The Commission should consider requiring that providers assume the minimum values for any additional parameters that would be necessary to accurately determine the area where a handset should achieve download and upload speeds no less than the minimum throughput requirement for any modeling that includes such a requirement.

Reflections

The FCC’s report illustrates how hard it is to assess network performance. Assumptions must be made in coverage models, and the assumptions analysts choose to make can have substantial effects on the outputs of their models. Similarly, on-the-ground performance tests don’t always give simple-to-interpret results. Two entities can run tests in the same area and find different results. Factors like the time of day a test was conducted or the type of device that was used in a test can have big consequences.

If we want consumers to have better information about the quality of service networks can offer, we need entities involved in modeling and testing coverage to be transparent about their methodologies.

VerHIDEzon – Brought To You By T-Mobile

T-Mobile just started a satirical ad campaign criticizing Verizon. T-Mobile’s CEO, John Legere, kicked the campaign off with this tweet:

Tweet from T-Mobile's CEO

The ad campaign criticizes Verizon for its decision to charge a premium for 5G service without publishing a map of areas where 5G service is available. The website for the campaign, VerHIDEzon.com, has some entertaining content:

We believe in charging a premium for 5G, without telling you where you’ll have coverage.
Why do we do this? Because we’re VerHIDEzon, and we do whatever we want…Every day we wake up with one goal in mind: charge our customers as much as possible.


T-Mobile makes a good point. It’s silly for Verizon to charge for 5G service without publishing information that indicates the extent of Verizon’s 5G coverage. Still, I find the campaign kind of odd. Neither company has much 5G coverage at the moment. Almost no one is using 5G-compatible phones yet. It may make business sense for T-Mobile to run the campaign today, but more time will need to pass before 5G has a lot of relevance for typical consumers.

T-Mobile Launches New Test Drive Program

In the last couple of days, T-Mobile has begun a new trial program. Non-customers can get a mobile hotspot along with 30 days of service and 30GB of data at no cost.

The program provides an easy option for people to test how well T-Mobile’s network could work for them. Once a hotspot is turned on, users can connect their existing phones and devices via Wi-Fi. The hotspots are compatible with LTE band 71, so hotspot users can experience the benefits of T-Mobile’s new Extended Range LTE signal.1

What’s the catch? As far as I can tell, there isn’t one. Signing up is easy, no credit card is required, and it’s not even necessary to return the hotspot at the end of the trial. I’ve joined the program, and I’ll probably post again in about a month with an update on my experience.

You can sign up for the program here. T-Mobile’s CEO, John Legere, discusses the program in more detail in the video below.