In Hindsight: T-Mobile’s Network Outage

Christian Mobile Phone Service

Yesterday, T-Mobile experienced a serious network outage beginning around 10am MT. The outage persisted through most of the day and primarily affected voice and text services.

Around 11pm MT, T-Mobile announced that outage was over:

These issues are now resolved. We again apologize for any inconvenience and thank you for your patience.

What caused the outage?

A lot of speculation was floating around yesterday. Some Twitter users, including a congressman, suggested there was a huge distributed denial-of-service attack causing trouble for U.S. networks. This story never made much sense.

Others speculated that a massive network failure was causing issues for all the major cellular operators. This idea was based on information from the website Downdetector, an entity that aggregates user complaints in real-time. Downdetector suggested that customer complaints about Verizon and AT&T were increasing at the same time that T-Mobile was experiencing issues.

Here’s how Downdectector describes its methodology:

Downdetector collects status reports from a series of sources, including Twitter and reports submitted on our websites and mobile apps. Our system validates and analyzes these reports in real-time, allowing us to automatically detect outages and service disruptions in their very early stages.

I expect the customer complaints about Verizon and AT&T were ultimately caused by T-Mobile’s network issues. Verizon and AT&T customers may have tried to call T-Mobile customers, then thought their own carriers (rather than T-Mobile) were at fault when calls failed.

Both AT&T and Verizon made statements suggesting that their networks were operating normally.

Updates from T-Mobile & the FCC

Yesterday evening, T-Mobile’s CEO, Mike Sievert, gave a vague explanation of what caused the outage:

T-Mobile has been experiencing a voice and text issue that has intermittently impacted customers in markets across the U.S…This is an IP traffic related issue that has created significant capacity issues in the network core throughout the day. Data services have been working throughout the day.

The outage was likely triggered as T-Mobile took steps to merge Sprint’s technology and/or customer base into the New T-Mobile. I haven’t heard any further information about the outage from a plausible-looking source, but details may come out soon. Ajit Pai, the FCC Chairman, shared the following tweet last night: