Facebook, Instagram, WhatsApp and Oculus are down. This is what we know

Facebook – and apparently all of the major services Facebook owns – are down today. Us first remark the problem was around 11:30 a.m. EST when some Facebook links stopped working. A little more in-depth investigation showed major DNS failures at Facebook:

The problem goes beyond Facebook’s obvious DNS failures, however. Facebook-owned Instagram was also down and its DNS services, which are hosted on Amazon rather than internal to Facebook’s own network, were up and running. Instagram and WhatsApp were accessible but instead displayed HTTP 503 failures (no server is available for the request), an indication that if DNS was working and the services load balancers were accessible, the application servers that should be powering them. load balancers were not.

A little later, Cloudflare vice president Dane Knecht reported that all BGP routes for Facebook had been removed. In the absence of BGP routes in Facebook’s network, Facebook’s own DNS servers would be inaccessible, as would the missing application servers for Facebook-owned Instagram, WhatsApp and Oculus VR.

Shortly after, Reddit user u / ramenporn reported on the r / sysadmin subreddit that the BGP peering with Facebook is down, possibly due to a configuration change that was pushed shortly before the start of breakdowns.

According to u / ramenporn – who claims to be a Facebook employee and part of the recovery effort – this is most likely a case of Facebook network engineers pushing a configuration change that inadvertently locked them down, which means that the fix must come from the technicians in the data center. with local and physical access to the routers in question. The removed routes do not appear to be the result of or related to a malicious attack on Facebook’s infrastructure.