The consequences of the power outage that Amazon in the night from Sunday on Monday, struck after a lightning strike, can still be 24 to 48 hours noticeable. Still, not all the sites are back online. In some cases, data may be lost.
Although Amazon immediately and started work with the remedy of the failure, it turns out that this is significantly longer than expected. Although the failure Sunday evening around 20.00 hours occurred, it was found that 12 hours later, not all instances of the Elastic Compute Cloud from Amazon again in the air. Cause appears to be an accumulation of problems.
The power outage occurred after a transformatorhuisje of the Amazon data center in Dublin was struck by lightning. By the impact created an explosion, followed by fire. Although a power outage typically is repacked by an emergency generator, it was found that the impact was so powerful that even the control system of the power stages was paralysed. This system ensures that the emergency generator is synchronized. Without this fasesynchronisatie can generators not be enabled automatically, so a manual action was required.
Not only took a lot of time to have all the volumes to restore it manually, also turned the available disk space is a problem. For the restore of the volumes had to Amazon an extra copy of all data, which makes almost all available storage capacity was flooding and the recovery process extra was delayed.
Throughout the day Amazon additional storage added to obviate this difficulty. Although the cloud provider claims that the lion’s share of the volumes during the day comes back, treasure the company that the 24 to 48 hours it will take before the entire process is completed.
It is expected that not every customer’s failure without loss of data can sit out. In some cases, Elastic Block Storage servers run out of power before the data were decent written. In those cases, will Amazon take a snapshot of a recovery image to restore, so maybe not entirely up to date.