As of Monday afternoon, Facebook had been flat on its face for hours, suffering a simultaneous worldwide outage not only on its main site, but also at its Instagram, WhatsApp, Messenger and Oculus VR subsidiaries.
We’re aware that some people are having trouble accessing Facebook app. We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience.
— Facebook App (@facebookapp) October 4, 2021
We’re aware that some people are having trouble accessing our apps and products. We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience.
— Oculus (@oculus) October 4, 2021
The New York Times reported that Facebook’s internal communications platform, Workplace, was also dragged offline, “leaving most employees unable to do their jobs.” It’s been a thumb-twiddling afternoon, the Times reported, with two Facebook employees comparing it to a “snow day.”
On Twitter, the hashtag #facebookdown was turning up predictable hilarity, transmitting blissful relief at the notion of a rainbow-bedrenched, Facebook-less world.
Here’s the meme we all need today #facebookdown pic.twitter.com/cwgkMcJiC7
— Brandon Tozzo (@BrandonTozzo) October 4, 2021
Me enjoying all the memes on twitter after instagram and Facebook is down#facebookdown #WhatsApp#serverdown pic.twitter.com/XmmSYHuZcw
— Aleeeyyy. (@iam_aleeraza) October 4, 2021
The reasons for the outage are unclear, but judging by the error message being thrown off by Facebook’s and WhatsApp’s domains – as shown in the screen captures below – it’s a DNS problem.
As of 15:29 EDT, Instagram’s site was displaying a “5xx Server Error” error.
BGP Bye-Byes
Cloudflare CTO John Graham-Cumming said in a series of tweets that the company saw Facebook disappear from the internet “in a flurry of BGP updates” between 15:50 UTC and 15:52 UTC:
About five minutes before Facebook’s DNS stopped working we saw a large number of BGP changes (mostly route withdrawals) for Facebook’s ASN. pic.twitter.com/dMTevg6hqj
— John Graham-Cumming (@jgrahamc) October 4, 2021
In other words, Facebook’s border gateway protocol (BGP) routes were kaput, meaning that it had lost the protocols that make routing decisions based on paths, network policies or rule-sets configured by a network administrator.
Two Facebook security team members who requested anonymity told the New York Times that it’s unlikely that a cyberattack was behind the mass outages, given that “the technology behind the apps was still different enough that one hack was not likely to affect all of them at once.”
Outage Coincides with Facebook’s Media Circus
The Verge reports that Facebook’s fiefdom skidded offline just as Facebook’s global head of safety, Antigone Davis, was live on CNBC. She was there to defend her employer against a whistleblower’s accusations that Facebook values product optimization so much that it has embraced algorithms that amplify hate speech, as well as to address Facebook’s handling of research data that suggests Instagram is harmful to teens.
Saryu Nayyar, CEO of Gurucul, said that if the Facebook outage does turn out to be caused by attackers, they’re probably pissed off about Facebook’s business practices.
“As more facts about Facebook and its business practices become public, its users’ anger seems to be on the rise,” she noted via email to Threatpost on Monday. “If they are attackers, they respond by attacking – in this case, possibly a DDoS attack that flooded the company’s DNS server.”
In any event, the company is working on the problems: “We’re aware that some people are having trouble accessing our apps and products,” said Facebook police communications director Andy Stone. “We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience.”
When The Verge checked out Down Detector before publishing its 12:01 EDT report on the issues, it looked like the problems were global. Outages spiked around noon EDT and were still coming down from that high as of 15:09 EDT, but the situation clearly hadn’t completely resolved.
The Verge also reported that users of the Oculus’s virtual reality technology can load games – if they’ve already installed and the browser works – but that Oculus social features are down, and users can’t install new games.
The Internet Is Still a Fragile Web
Bill Lawrence, CISO of SecurityGate, told Threatpost on Monday that outages like this one show little progress since the distributed denial-of-service (DDoS) attack on Dyn in October 2016. That attack, which affected Twitter, GitHub and others provided lessons learned: To ward off such a scenario, many large organizations now protect against DNS loss by maintaining multiple DNS systems across different DNS providers.
Even so, five years after Dyn we still have parts of the internet that can still shatter when services like DNS get interrupted for some reason, he said. Thus, Lawrence said that it will be interesting to see what caused this lingering outage to “several jewels in the Facebook family.”
Gurucul’s Nayyar agreed with the New York Times’ sources inside Facebook’s security team who said that the company’s infrastructure is too diverse for a cyberattack to cripple. She said that it’s highly unlikely that Facebook hasn’t protected itself in similar fashion, she said.
“While the cause of Facebook’s problem isn’t yet clear, it would be amazing if they hadn’t already set up multiple DNS providers,” she commented.
Check out our free upcoming live and on-demand webinar events – unique, dynamic discussions with cybersecurity experts and the Threatpost community.