Incident Report

Summary

On Monday, February 10, 2025, Google services including Chat, Calendar, Gmail, Meet, and Docs experienced intermittent 'Error 500' messages and increased latency. The Google Cloud Console also encountered '500' errors and failed to load traffic in South America. These disruptions lasted for 36 minutes and primarily affected users in North and South America regions.

This is not the level of quality and reliability we strive to offer you and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

During a routine disaster recovery exercise for Google Account Infrastructure, a set of cluster resources in one of our Data Centers were drained, caused by a config change resulting in traffic being redirected to another service in another Data Center. The increased traffic load overwhelmed the service which provides authentication and other core services to Workspace products, leading to errors and increased latencies impacting traffic in North and South America regions. Engineers undrained the set of cluster resources to mitigate the issue.

Remediation and Prevention

Google engineers were alerted to the outage via internal monitoring on Monday, 10 February at 11:48 US/Pacific and immediately started an investigation. Once the nature and scope of the issue became clear at 12:12 US/Pacific, Google engineers quickly undrained the set of cluster resources which caused the traffic shifts and overloaded tasks to mitigate the issue at 12:26 US/Pacific.

Google is committed preventing a repeat of this issue in the future and is completing the following actions:

• We are working on adding automated guardrails around drain/undrain operations for the relevant services to prevent configuration changes of this type for the relevant services.
• We are making proactive adjustments to how systems handle network changes to avoid/prevent any potential disruptions.
• We are addressing a safely draining process in specific areas to ensure smooth and reliable operation and develop a workaround that will allow us to safely manage our network while maintaining uninterrupted service.

Detailed Description of Impact

On Monday, 10 February 2025 from 11:50 to 12:26 US/Pacific multiple Google Workspace products experienced intermittent '500' errors along with increased latency, and the Google Cloud Console experienced '500' errors and failed to load for traffic for some users in South America.

Google Workspace:

• Google Chat - 65k affected users.

• Google Calendar - 25k affected users

• Gmail - 77k affected users

• Google Meet - 8k affected users

• Google Docs - 1k affected users

  Google Cloud Console:

• Approximately 8k users were affected, predominantly in South America, during the outage. In North America the effect was very short lived (<2 minutes).

Errors and slowness for a portion of traffic in the Data Center were observed due to overload and cascading failure. Errors led to intermittent 500 errors on all authenticated apps. Some of the issues may have been "sticky" for users who were triply affected by broken tasks, leading to more consistent 500 errors, for about 10 minutes.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 10 February, 2025 11:50

Incident End: 10 February, 2025 12:26

Duration: 36 minutes

Affected Services and Features:

Google Workspace Services:

Google Chat, Google Calendar, Gmail, Google Meet, Google Docs

Google Cloud Services:

Google Cloud Console

Regions/Zones: Global

Description:

Multiple Google Workspace products experienced intermittent '500' errors along with increased latency, and the Google Cloud Console experienced '500' errors and failed to load for traffic in South America.

The impact lasted for a duration of 36 minutes. From the preliminary analysis the root cause of the issue was temporary overload in a single region. A full Incident Report will be published in the coming days with a detailed root cause analysis.

Customer Impact:

Google Workspace: Multiple products experienced ‘500’ errors or increased latency. The figures below represent the overall number of our customers affected globally over the course of the outage.

• Google Chat - 65k affected users.
• Google Calendar - 25k affected users
• Gmail - 77k affected users
• Google Meet - 8k affected users
• Google Docs - 1k affected users

Google Cloud Console:

• Around 70% of attempts to load the Cloud Console experienced failures or elevated latency in South America for the duration of the outage. Additionally, users in North America also experienced a small blip (<2mins), but this resolved quickly.

Summary:

Google Workspace users were experiencing elevated error rates interacting with Google Chat, Google Calendar, Gmail, and Google Meet.

Description:

Our internal metrics show service recovery for all impacted products. Our engineers are continuing to closely monitor for full recovery while investigating the cause of the issue.

We will provide an update by Monday, 2025-02-10 14:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Affected customers were experiencing intermittent errors interacting with Gmail, Google Calendar, Google Chat, and Google Meet.

Workaround: None at this time.

Summary:

Google Workspace users are experiencing elevated error rates interacting with Google Chat, Google Calendar, Gmail, and Google Meet.

Description: We are experiencing an intermittent issue with Google Chat, Google Calendar, Gmail, Google Meet beginning at Monday, 2025-02-10 11:51 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Monday, 2025-02-10 13:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Affected customers are experiencing intermittent errors interacting with Gmail, Google Calendar, Google Chat, and Google Meet.

Workaround: None at this time.