Lightning strikes at one of Google's European data centers have caused some users of the corporation's cloud services to lose data. The center, which is located in St. Ghislain, 50 kilometers southwest of the Belgium capital Brussels, was off-line for a very short period of time, but long enough to impact 0.000001% of
Google Compute Engine (GCE) disks and cause some "permanent data loss". While the center itself was not hit by lightening, a local utility grid was hit 4 times causing an interruption of power to the center.
Google's data center's are safeguarded against such incidents with failover systems that flip over to additional power sources when the main power source is unavailable. In addition, centers are also equipped with additional backup batteries. Regardless, the peculiar set of circumstances caused “extended or repeated battery drain” and some Google servers stopped operating. Google's Cloud Platform status page reported the incident as follows:
"At 09:19 PDT on Thursday 13 August 2015, four successive lightning strikes on the local utilities grid that powers our European datacenter caused a brief loss of power to storage systems which host disk capacity for GCE instances in the europe-west1-b zone. Although automatic auxiliary systems restored power quickly, and the storage systems are designed with battery backup, some recently written data was located on storage systems which were more susceptible to power failure from extended or repeated battery drain. In almost all cases the data was successfully committed to stable storage, although manual intervention was required in order to restore the systems to their normal serving state. However, in a very few cases, recent writes were unrecoverable, leading to permanent data loss on the Persistent Disk."
The status page went on, "This outage is wholly Google's responsibility. However, we would like to take this opportunity to highlight an important reminder for our customers: GCE instances and Persistent Disks within a zone exist in a single Google datacenter and are therefore unavoidably vulnerable to datacenter-scale disasters. Customers who need maximum availability should be prepared to switch their operations to another GCE zone. For maximum durability we recommend GCE snapshots and Google Cloud Storage as resilient, geographically replicated repositories for your data."
Were you impacted by Google's problems? Let us know the details. Add your comments below.
