In Hindsight: T-Mobile’s Network Outage

Yesterday, T-Mobile experienced a serious network outage beginning around 10am MT. The outage persisted through most of the day and primarily affected voice and text services.

Around 11pm MT, T-Mobile announced that outage was over:

These issues are now resolved. We again apologize for any inconvenience and thank you for your patience.

What caused the outage?

A lot of speculation was floating around yesterday. Some Twitter users, including a congressman, suggested there was a huge distributed denial-of-service attack causing trouble for U.S. networks. This story never made much sense.

Others speculated that a massive network failure was causing issues for all the major cellular operators. This idea was based on information from the website Downdetector, an entity that aggregates user complaints in real-time. Downdetector suggested that customer complaints about Verizon and AT&T were increasing at the same time that T-Mobile was experiencing issues.

Here’s how Downdectector describes its methodology:

Downdetector collects status reports from a series of sources, including Twitter and reports submitted on our websites and mobile apps. Our system validates and analyzes these reports in real-time, allowing us to automatically detect outages and service disruptions in their very early stages.

I expect the customer complaints about Verizon and AT&T were ultimately caused by T-Mobile’s network issues. Verizon and AT&T customers may have tried to call T-Mobile customers, then thought their own carriers (rather than T-Mobile) were at fault when calls failed.

Both AT&T and Verizon made statements suggesting that their networks were operating normally.

Updates from T-Mobile & the FCC

Yesterday evening, T-Mobile’s CEO, Mike Sievert, gave a vague explanation of what caused the outage:

T-Mobile has been experiencing a voice and text issue that has intermittently impacted customers in markets across the U.S…This is an IP traffic related issue that has created significant capacity issues in the network core throughout the day. Data services have been working throughout the day.

The outage was likely triggered as T-Mobile took steps to merge Sprint’s technology and/or customer base into the New T-Mobile. I haven’t heard any further information about the outage from a plausible-looking source, but details may come out soon. Ajit Pai, the FCC Chairman, shared the following tweet last night:

Major T-Mobile Network Outage

A major T-Mobile network outage is going on right now.

T-Mobile’s outage

T-Mobile subscribers scattered around the U.S. have been reporting that calls and texts have been failing over the last few hours.

The outage appears to be affecting more than T-Mobile’s direct customers. Subscribers with mobile virtual network operators that run over T-Mobile’s network are also reporting problems.

T-Mobile is aware of the issues. Neville Ray, T-Mobile’s President of Technology, shared the following tweet:

Neville Ray's tweet acknowledging the outage

While Ray describes it as a “voice and data issue,” most of the reports I’ve seen so far have been about issues with calls and texts.

Other companies

The website Downdetector is recording an atypical number of issues with a bunch of wireless carriers, Facebook, Instagram, Twitch, and a handful of other internet companies. It’s unclear if Verizon and AT&T are experiencing any issues of their own. Subscribers with these carriers may only be experiencing issues when they try to contact people on T-Mobile’s network.

Prognosis

At this time, I have not seen any word on what is causing the outage. I have also not seen any clear indication about when the issue is expected to be resolved.

I plan to make updates as I learn more. This may end up being one of the largest network outages in the U.S. in years.


Monday Night Update:

At 11:03pm MT, Neville Ray tweeted that the issues have been resolved:

Voice and text services are now restored. Thank you for your patience as we fixed the issues. We sincerely apologize for any and all inconveniences.
Picture of a broken phone

Is Phone Insurance Worth It?

Phone insurance usually isn’t a good deal. Companies offering phone insurance plans typically intend to make a profit. These companies profit when customers, on average, pay more into insurance programs than they get paid out.

When considering the costs of phone insurance, think in terms of a long time horizon. One of Verizon’s insurance plans, Verizon Protect, costs $17 per month. Imagine you purchase a new phone from Verizon for $600 then insure it with Verizon Protect. If you use the phone for three years and keep it insured the whole time, you’ll end up making 36 payments of $17. After three years, you’ll have spent $612 on insurance. That’s more than the original cost of the phone!

Considerations

Whether phone insurance is worth it will depend on your situation:

  • How much would you have to pay for insurance?
  • What would it cost to replace your phone without insurance?
  • What sort of deductible would your insurance plan have?
  • How risk-averse are you?
  • How careful are you about protecting your phone?
  • Would phone insurance offer any added conveniences (e.g., extra-fast repairs)?

Examples

High-end phones

I recently purchased a Samsung Galaxy S20 from Verizon. The phone has a list price of about $1,000. For $17 per month, I can cover the phone with Verizon Protect. While Verizon primarily pitches that protection plan, there’s another plan called Wireless Phone Protection that offers similar coverage for only $6.85 per month.

The Wireless Phone Protection plan comes with a $200 deductible on the S20. Since it would normally cost me $1,000 to replace the S20, the insurance plan could save me up to $800.

The protection plan’s monthly fee works out to be less than 1% of the amount I would save if I lost or destroyed my phone. I’m clumsy and tend to put my phones through a lot. There’s a greater than 1% chance I’ll break or lose my S20 in any given month. Accordingly, the insurance plan would offer me good value in the short term.

Depreciation

Today’s high-end phones will be tomorrow’s budget phones. While the S20 has a list price of about $1,000 today, it’ll be cheaper in the future. If Verizon still sells the phone in three years, it’ll cost far less.

While the replacement cost of my phone will decline over time, the rate I pay to insure it won’t. In some cases, it can make financial sense to (a) insure a high-end phone briefly after purchase and (b) drop the insurance at a later time.

Budget phones

Most companies don’t closely match the cost of insurance plans to the value of a phone. My favorite budget-friendly phone right now is the Motorola G7 Play. It costs $130 from Motorola. Verizon’s Protect plan still costs $17 per month for the G7 Play. The Wireless Phone Protection plan still costs $6.85 per month.

The G7 Play is not worth insuring. The phone has a $9 deductible. Verizon’s insurance would only save me $121 if I lost or broke a G7 Play. In just eight months, the Verizon Protect plan would cost more than a brand new device.

Self-insurance

In most cases, I recommend that people self-insure their phones. While the peace of mind you can get from an insurance plan is nice, the companies offering the plans usually come out ahead of consumers.

Some people will argue that self-insurance isn’t always reasonable. Today’s fancier phones are expensive. Many people would have trouble covering a big, unplanned hit to their finances. In my view, most people who cannot afford to self-insure are buying phones that are too expensive. Some of today’s budget phones are great. Self-insuring is easier with a low-cost device.

Verizon To Offer Student Discounts

Starting July 2, Verizon will offer student discounts on its postpaid, unlimited plans. Students enrolled in programs of higher education (undergraduate, graduate, or vocational programs) can take advantage of $10 per month off of one line of service or $25 off of two lines of service. As far as I can tell, students on plans with three or more lines will not be eligible for any discount.

Verizon website screenshot

Verizon has a page on its website about the new discount. Here are further details on the offer’s terms, per Verizon:

For eligible students actively enrolled (including online enrollment) in a U.S. secondary educational institution of higher learning, including undergraduate, graduate, and/or vocational school or institution. Approved verification documents req’d. Offer good for a max of four (4) years as long as annual eligibility evaluations are met. Discount limited to max of 2 phone lines. Eligible students must be account owner or account manager; one offer per account. Cannot be combined with most offers.

AT&T and Cricket Drop $15 Plans

In March, T-Mobile began offering its $15 per month Connect plan with 2GB of data, unlimited minutes, and unlimited texts. AT&T and AT&T’s flanker brand, Cricket, quickly came out with similar plans for $15 per month. Both AT&T and Cricket have stopped offering the plans to new customers. It’s now come out that customers who purchased these plans will not be able to renew at the same price after 7/14/2020.

Here’s an excerpt from a text I received from Cricket the other day:

We hope the $15/2GB plan has helped you during this difficult time. Starting July 15, 2020, the plan will no longer be available. You can either select a different plan or we will move you to the $30/2GB plan when your bill cycle renews after July 14, 2020.

A credible-looking Reddit thread suggested AT&T Prepaid’s $15 plan would also become unavailable for existing customers after 7/14/2020.

Reflections

Shortly after the T-Mobile Connect plan launched, I began recommending it on Coverage Critic and on the list I maintain at MrMoneyMustache.com. My decision to recommend T-Mobile’s Connect plan rather than AT&T’s $15 plan drew some criticism on Reddit and raised questions from commenters on MrMoneyMustache. After all, AT&T’s network is more expansive than T-Mobile’s network.

I was worried AT&T’s plan wouldn’t stick around. Here’s a bit I wrote in March:

It’s not clear how long AT&T’s plan will be around. People who take advantage of AT&T’s offer today won’t necessarily get the same great deal each month for the foreseeable future. On the other hand, it looks like the T-Mobile Connect plans will continue to be available to new and existing subscribers for years.

About a month ago, I saw a screenshot from a chat conversation with a Cricket customer service representative. The representative suggested that Cricket subscribers on the $15 plan would be grandfathered. I remained skeptical. I wasn’t convinced the representative’s words were authoritative. I continued not to make a strong recommendation of Cricket or AT&T’s $15 plans. Today, I feel like my skepticism was validated.

Xfinity Store

Xfinity Mobile Updates: 5G, Pricing, and Prioritization

Today, Xfinity Mobile released a handful of changes.

5G Support

Xfinity Mobile now officially offers 5G powered by Verizon’s network. The carrier’s updated network webpage has a lot of content devoted to the wonders of 5G.

Pricing

The 1GB by-the-gig plan has increased from $12 per month to $15 per month. Data add-ons for by-the-gig customers have also increased in cost from $12 per GB to $15 per GB.

The 3GB by-the-gig plan for $30, the 10GB by-the-gig plan for $60, and the $45 per line unlimited plan are still available with unchanged prices.

Prioritization and video throttling

With the last generation of Xfinity Mobile plans, subscribers were typically subject to deprioritization during congestion. It looks like subscribers on the new by-the-gig plans will not be subject to deprioritization. Here’s a screenshot from my within my Xfinity online account:1

Screenshot suggesting Xfinity Mobile is offering better prioritization on by-the-gig plans

As far as I can tell, subscribers on Xfinity Mobile’s Unlimited plan will continue to be deprioritized during periods of congestion.

Optional opt-in

Existing subscribers are not being forced to switch over to Xfinity’s new plans. At the moment, it looks like subscribers who don’t switch over will continue to experience the old price structure while missing out on new perks like 5G access.

I’m unsure whether Xfinity Mobile will let subscribers stay grandfathered on the old plans indefinitely. It’s possible subscribers will eventually be forced to switch to new plans.

My take

As an Xfinity Mobile affiliate, I got a heads up that changes were coming. It sounded like there would be a price increase on the 1GB plan, and I figured I wouldn’t be able to recommend Xfinity Mobile as strongly after the changes. I’m happy to say my expectation was wrong. The $3 increase in the 1GB plan isn’t too substantial, and the improved prioritization for by-the-gig customers is great.

I’m frustrated by how actively Xfinity Mobile is marketing the new 5G service without making it clear that (a) Verizon’s 5G coverage is extremely limited and (b) few consumers have devices compatible with Verizon’s 5G. That said, Xfinity Mobile’s marketing is less misleading than what we’re typically seeing from carriers offering 5G. While the new 5G access won’t have a meaningful effect on most subscribers today, it will become more important as Verizon expands its 5G coverage.


Considering Xfinity Mobile? Check coverage at your location with the carrier’s coverage tool.

T-Mobile’s Data Maximizer Can’t Be Turned Off On Connect Plans

Some of T-Mobile’s plans come with a setting called “Data Maximizer.” While Data Maximizer is turned on, most video traffic will be throttled to about 480p quality. Data Maximizer reduces the load on T-Mobile’s network and can help subscribers with limited data allowances to conserve their data.

T-Mobile’s Connect plans supposedly have Data Maximizer turned on by default. Subscribers can also supposedly turn off the feature. On portions of T-Mobile’s website related to the Connect plans, there are disclosures like this one:1

Video typically streams on your T-Mobile device at DVD quality (480p) with Data Maximizer. You may disable Data Maximizer at any time.

I’ve recently been testing a Connect plan. Sure enough, a test I ran with the app Wehe confirmed that some video traffic was being throttled:

Results from video throttling tests

When I tried to turn off Data Maximizer, I ran into trouble. I first tried to disable the setting from within my T-Mobile online account. While subscribers on some of T-Mobile’s other plans can turn off Data Maximizer through an online process, that didn’t seem possible with the Connect plan.

I went ahead and called T-Mobile to see if a support agent could turn off the setting. At first, the support agent looked into it and told me Data Maximizer didn’t seem to affect Connect plans. I explained that T-Mobile’s website suggested otherwise and that my video traffic appeared to be throttled. The agent seemed to agree something strange was going on. She said she’d put in a ticket to have someone at T-Mobile look into the issue.

I don’t think it would be a big deal if Connect subscribers couldn’t turn off Data Maximizer. 480p video is, in my opinion, very watchable. Conserving data while streaming can be really beneficial on plans that don’t have large data allotments. Still, the fact that I ran into this issue surprised me. The Connect plans will probably be popular. I’m surprised some sort of quality review didn’t catch this issue before the plans were released.

Mint Mobile Extends Unlimited Data Again

Mint Mobile has extended its COVID-19-related unlimited data offer. Subscribers that joined Mint prior to April 14 can continue to receive data add-ons in 3GB increments at no charge.

Mint first said this offer would be available from mid-March through mid-April, but ended up extending the offer through mid-May. The company has now extended the offer for another month.

Image showing that Mint's unlimited data offer has been extended

I don’t know if we’ll see further extensions of the offer. More information can be found on Mint’s website.

Mint Mobile Makes COVID-19 Data Add-Ons Easier

In response to COVID-19, Mint Mobile began offering unlimited data at no charge to all its customers. Initially, subscribers who ran out of their regularly allotted data had to purchase 3GB data add-ons that would later be refunded.

Mint has now streamlined the process. Here’s an excerpt from a recent Reddit post by Rizwan Kassim, a co-founder of Mint Mobile:

We heard your feedback, and once we made the decision to extend the program – we rebuilt it in a way with an experience that’s more .. well, Minty. On Apr 15, whenever you purchase the data-bolton, if you qualify, the dollar cost will be $0. No credit card charge, no wallet pull. Easy. You’ll need the latest build of the app.

I thought Mint might have intentionally put a bit of friction in the data add-on process to deter abuse and waste. It looks like there’s a better explanation:

The idea was hatched, planned, got financial and brand approval, configured, launched and messaged the unlimited bolt-on offer in 36 hours, over a weekend by an entirely WFH staff. No joke. The fastest method was not to use our billing system, but to tack on a series of jobs that ran post-charge to refund the balances.

Mint has added some sensible eligibility terms for the free data add-ons:

  • Only subscribers who were customers before April 14 are eligible for free data add-ons
  • Subscribers that downgrade their plans lose their eligibility for free data add-ons

Black Wireless & Mango Mobile Fail To Deliver

Earlier this year, two brands owned by the operator Red Pocket, Black Wireless and Mango Mobile, began offering what looked like a great deal…While the offer looked amazing, I didn’t bother writing about it. I’d previously had bad experiences with the carriers’ parent company, Red Pocket, and this recent offer looked sketchy.

That’s how I opened a retrospective post about a seemingly too-good-to-be-true plan that Black Wireless and Mango Mobile were offering. In the post, I explained why I found the carriers’ promotional deal sketchy:

  • The descriptions of the offer were confusing and possibly contradictory.
  • While Black Wireless typically offered service over AT&T’s network, the promotion was for service over T-Mobile’s network. Black Wireless was doing a terrible job of conveying that information to potential customers.
  • I couldn’t understand how Mango Mobile and Black Wireless would be able to profitably sell cheap plans with data allotments so much higher than those offered by other carriers piggybacking on the same networks.

I ended my post by suggesting that I may have been too skeptical:

I was suspicious the promotion would end up as a fiasco that looked bad for Red Pocket. However, it looks like the company has sorted things out.

New developments

It turns out my skepticism was warranted. Subscribers that took advantage of the deal are having their plans canceled. Here’s the start of a message Black Wireless sent subscribers on the plan:1

Hello, this is Black Wireless. Unfortunately, we have forfeited our contract with our vendor for T-Mobile services due to business reasons. Your service will stop by April the 10th, thus we contacted you to see whether you wish to continue the service with us and in this case we will add a web credit on your online Black Wireless account so you can use the credit to purchase a new plan with a SIM.

“Due to business reasons” is an awfully vague explanation.2 While I’m not certain what happened, I’m suspicious Black Wireless and Mango Mobile were improperly reselling T-Mobile business lines to non-business consumers.3

Damage Control

A Reddit poster associated with Black Wireless commented in a thread about the new developments.

We apologize for any inconvenience…We are providing several options for the customers affected, which include switching to our legacy Black Wireless plans and/or refunds for the balance of the time (remaining months) after the period ending April 10th. Black Wireless has been in the telecom business since 1997 and the wireless business since 2011 and we are here to assist you to make this as seamless as possible. Black Wireless is based out of New York. Thank you for your time and be safe during this trying time.

It was good to see Black Wireless joining the conversation, but the comment frustrated me. A bunch of statements in the comment are misleading or only half true.

The people who eventually created Black Wireless existed in 1997, but Black Wireless did not exist yet. Further, Red Pocket took the reigns of Black Wireless in the last several years. While Black Wireless may technically be based out of New York, I feel like the commenter was trying to mislead people about how closely the company is tied to the U.S. I’m pretty sure the company Red Pocket offloads most of its operations to is based out of Chennai.4

In a second comment, the individual associated with Black Wireless gave some level-headed insights:

We did not lose any contract, but merely the plan we had with the underlying carrier due to unforeseen circumstances. We could only do our best at this point to assist our customers in order to make sure they get the service they require or if desired a refund. In this MVNO environment you must understand there are certain criteria we must adhere to and terms we cannot state in order to stay within our business requirements and guidelines with the underlying carrier and contracts/agreements.

If we take the comment at face value, it contradicts what Black Wireless told subscribers via text message: “Unfortunately, we have forfeited our contract with our vendor for T-Mobile services due to business reasons.”

While I’ve found the handling of the whole situation unprofessional, it looks like the companies involved are taking the right steps to offer refunds to affected customers.