Nobody expected a social network as huge as Facebook to just “go down.” Yet at 10:30 PM last night (Vietnam time), Facebook along with Messenger, Instagram, and WhatsApp all stopped working and became inaccessible. What made this outage even more shocking was how long it lasted, almost 7 hours. This seriously affected users around the globe.
So, what exactly happened?
Facebook hasn’t officially shared the exact cause of the problem, but experts at CloudFlare offered a detailed explanation. CloudFlare is a company specializing in content delivery networks (CDN), known for services that help websites stay online even during attacks or server issues. They also run the DNS service 1.1.1.1 and the VPN tool WARP+.
BGP: The Core Issue Behind Facebook’s Outage
The internet works by connecting one computer to another. For example, when you visit Facebook, your computer connects to Facebook’s servers to exchange data. The process your computer uses to “find its way” to Facebook’s servers is called routing.
BGP, or Border Gateway Protocol, is a system that helps large networks share routing information. Routers around the world keep updated lists of the best “paths” for data packets. Think of it like a map you’d use if you got lost inside a huge shopping mall.
BGP is like the directional map you find in big malls
When Facebook went down, CloudFlare’s team noticed that Facebook’s BGP routes were withdrawn, meaning the paths to Facebook’s servers were removed. It’s like the mall map was taken down, leaving visitors confused and lost. In other words, Facebook disappeared from the internet.
Facebook’s BGP routing was updated right when the outage happened
Because Facebook’s BGP failed, the DNS (Domain Name System) couldn’t do its job of translating facebook.com into an IP address. If you’re not familiar, DNS turns domain names like facebook.com into IP addresses like 157.240.211.35. Since Facebook’s BGP routes were gone, DNS couldn’t find Facebook’s server addresses.
So why did Facebook’s BGP fail?
Now that you understand BGP and how it works, the big question is why Facebook’s BGP broke down. Doesn’t Facebook have a professional team managing this?
On Reddit, a user claiming to be part of Facebook’s incident response team shared some behind-the-scenes info.
According to this user, Facebook’s BGP issues started after a system configuration change. But the problem was so severe that it couldn’t be fixed remotely, admins had to be physically present at the data center.
However, the staff on-site at Facebook’s data center lacked the knowledge and authority to fix the issue. This caused engineers to spend much more time than usual restoring services. It’s worth noting that Facebook had reduced its on-site data center team due to COVID-19.
After sharing these details, the Reddit account was deleted.
