[ad_1]
Google started the week with a big blackout that took out Gmail, Drive, and all other Workspace apps. As promised, Google now has a detailed explanation of the outage and what to do to prevent future incidents.
At a high level, the issue is related to the existing work of updating Google’s account authentication system. As the effort continued, the previous components were “left in place”. While keeping these older aspects resulted in a usage error of 0, Google instituted a grace period to delay the impact.
This fix expired and caused automated systems to respond to the error as if it were real. As the usage appeared to be at 0, the capacity of the identity management system was reduced. Although security controls are in place, they were not designed to cover the specific issue.
The issue began to affect users at 3:47 a.m. PT, and engineers were alerted a minute later. “The workspace applications were down for the duration of the incident,” as they rely on the affected infrastructure to ensure that you are logged in, authenticated and authorized to view content, such as emails and documents.
At 4:08 am, the root cause and a potential fix were identified, which led to the disabling of quota enforcement in a data center at 4:22 am. This quickly improved the situation and at 4:27 am the same mitigation was applied to all data centers which returned the error rates to normal levels at 4:33 am.
The company has plans to review, improve and evaluate its systems to avoid similar issues of this nature. Google ended its breakdown explanation with an apology:
We would like to apologize for the magnitude of the impact this incident has had on our customers and their businesses. We take any incident that affects the availability and reliability of our customers very seriously, especially incidents that span multiple regions.
The full technical explanation is available here.
FTC: We use automatic income generating affiliate links. More.
Check out 9to5Google on YouTube for more news:
[ad_2]
Source link