Tag: Outage

A red flag

Service Outage At Visible

Christian

Verizon’s flanker brand Visible has had a rough time the past few weeks. First, a security issue caused trouble for a large number of subscribers earlier this month. Then yesterday, reports of a service outage started to surface.

Several Reddit posts and dozens of comments mentioned the outage. I’m not sure how much of Visible’s subscriber base was affected.1 The issue was widespread enough for Visible to acknowledge the issue on Twitter:


Reports suggest the outage only lasted a few hours. Shortly after sharing its initial tweet, Visible noted that the outage was resolved:

I haven’t heard anything about the underlying cause of the outage.

Data Outage Affecting Some Mint Mobile Subscribers

Christian 3 Comments

A data outage has been affecting some Mint Mobile subscribers throughout the day. One of Mint’s co-founders, Rizwan Kassim, posted about the issue on Reddit:

An upstream error seems to have caused data provisioning errors for a number of subscribers.

It’s being worked, they don’t have a root cause yet, but I know this has been escalated. Down Detector showing issues on our carrier as well; not sure if it’s related or not.

I strongly think, but do not know, that this has nothing to do with the iOS 14 upgrade many of you installed today.

Based on reports I’ve read from Mint subscribers, the issue appears widespread geographically. I don’t know what proportion of Mint’s subscriber base is affected.

Kassim’s Reddit post suggests the issue may also be affecting T-Mobile. While Downdetector shows a slightly unusual level of issues associated with T-Mobile, I don’t think Downdetector’s data is consistent with a large-scale problem for T-Mobile subscribers.

I did a bit of my own digging for T-Mobile subscribers’ complaints about the networks’ performance today. I didn’t run into anything out of the ordinary.

Neville Ray’s Statement on T-Mobile’s Outage

Christian

Yesterday, Neville Ray, President of Technology at T-Mobile, shared more information about the cause of T-Mobile’s outage on Monday:

Many of our customers experienced a voice and text issue yesterday, specifically with VoLTE (Voice over LTE) calling…The trigger event is known to be a leased fiber circuit failure from a third party provider in the Southeast. This is something that happens on every mobile network, so we’ve worked with our vendors to build redundancy and resiliency to make sure that these types of circuit failures don’t affect customers. This redundancy failed us and resulted in an overload situation that was then compounded by other factors. This overload resulted in an IP traffic storm that spread from the Southeast to create significant capacity issues across the IMS (IP multimedia Subsystem) core network that supports VoLTE calls.

We have worked with our IMS (IP Multimedia Subsystem) and IP vendors to add permanent additional safeguards to prevent this from happening again and we’re continuing to work on determining the cause of the initial overload failure.

So, I want to personally apologize for any inconvenience that we created yesterday and thank you for your patience as we worked through the situation toward resolution.

In Hindsight: T-Mobile’s Network Outage

Christian

Yesterday, T-Mobile experienced a serious network outage beginning around 10am MT. The outage persisted through most of the day and primarily affected voice and text services.

Around 11pm MT, T-Mobile announced that outage was over:

These issues are now resolved. We again apologize for any inconvenience and thank you for your patience.

What caused the outage?

A lot of speculation was floating around yesterday. Some Twitter users, including a congressman, suggested there was a huge distributed denial-of-service attack causing trouble for U.S. networks. This story never made much sense.

Others speculated that a massive network failure was causing issues for all the major cellular operators. This idea was based on information from the website Downdetector, an entity that aggregates user complaints in real-time. Downdetector suggested that customer complaints about Verizon and AT&T were increasing at the same time that T-Mobile was experiencing issues.

Here’s how Downdectector describes its methodology:

Downdetector collects status reports from a series of sources, including Twitter and reports submitted on our websites and mobile apps. Our system validates and analyzes these reports in real-time, allowing us to automatically detect outages and service disruptions in their very early stages.

I expect the customer complaints about Verizon and AT&T were ultimately caused by T-Mobile’s network issues. Verizon and AT&T customers may have tried to call T-Mobile customers, then thought their own carriers (rather than T-Mobile) were at fault when calls failed.

Both AT&T and Verizon made statements suggesting that their networks were operating normally.

Updates from T-Mobile & the FCC

Yesterday evening, T-Mobile’s CEO, Mike Sievert, gave a vague explanation of what caused the outage:

T-Mobile has been experiencing a voice and text issue that has intermittently impacted customers in markets across the U.S…This is an IP traffic related issue that has created significant capacity issues in the network core throughout the day. Data services have been working throughout the day.

The outage was likely triggered as T-Mobile took steps to merge Sprint’s technology and/or customer base into the New T-Mobile. I haven’t heard any further information about the outage from a plausible-looking source, but details may come out soon. Ajit Pai, the FCC Chairman, shared the following tweet last night:

Major T-Mobile Network Outage

Christian

A major T-Mobile network outage is going on right now.

T-Mobile’s outage

T-Mobile subscribers scattered around the U.S. have been reporting that calls and texts have been failing over the last few hours.

The outage appears to be affecting more than T-Mobile’s direct customers. Subscribers with mobile virtual network operators that run over T-Mobile’s network are also reporting problems.

T-Mobile is aware of the issues. Neville Ray, T-Mobile’s President of Technology, shared the following tweet:

Neville Ray's tweet acknowledging the outage

While Ray describes it as a “voice and data issue,” most of the reports I’ve seen so far have been about issues with calls and texts.

Other companies

The website Downdetector is recording an atypical number of issues with a bunch of wireless carriers, Facebook, Instagram, Twitch, and a handful of other internet companies. It’s unclear if Verizon and AT&T are experiencing any issues of their own. Subscribers with these carriers may only be experiencing issues when they try to contact people on T-Mobile’s network.

Prognosis

At this time, I have not seen any word on what is causing the outage. I have also not seen any clear indication about when the issue is expected to be resolved.

I plan to make updates as I learn more. This may end up being one of the largest network outages in the U.S. in years.

Monday Night Update:

At 11:03pm MT, Neville Ray tweeted that the issues have been resolved:

Voice and text services are now restored. Thank you for your patience as we fixed the issues. We sincerely apologize for any and all inconveniences.
Image of a hard hat and a construction cone

Verizon Website Outage on 9/23/2019

Christian

3:20PM MT update: The outage is has ended.

VerizonWireless.com appears to be down at the moment. If you’re having trouble accessing the website right now, the issue is not on your end. As far as I can tell, the problem is not related to the browser or device visitors are using to access Verizon’s website.

At the moment, I’m being served a “Page is unavailable” message on all of the web pages I’ve tried to load:

Earlier, I was presented with a different error message:

The outage appears to be unrelated to the locations users try to access Verizon’s website from. I continued to receive error messages when accessing Verizon’s website from other locations via a VPN.

The outage appears to have started by about 1:30MT and hasn’t been resolved at the time of writing (about an hour later). Android Central and at least one Twitter user have also picked up on the outage.

Stay tuned.