Looking back, the mistakes seem so obvious: data centers located on a low-lying island (Manhattan), and back-up generators kept in basements. Now that the floodwaters have receded, it’s time ask ourselves: What have we learned from Hurricane Sandy?
Between October 24 and October 30, Hurricane Sandy wreaked havoc in eight countries from the Caribbean to Canada. It destroyed homes and livelihoods and left upwards of 200 people dead. In the United States, the storm is said to have affected 24 states. While cities as far inland as Cleveland, Ohio, felt the impact of Hurricane Sandy, it was the East Coast (New Jersey, New York, Pennsylvania, Connecticut, Massachusetts, and Maine) that bore the brunt of it. Today, the total cost of damage is estimated to be at least U.S.$50 billion. For many, a return to pre-Sandy conditions is still a long way off.
Now, five weeks after the storm, we ask ourselves, what have we learned from it? For one thing, there’s the environmental aspect. Rising sea levels and a warming planet have increased the likelihood of more frequent Hurricane Sandy-like superstorms in the future, say some. There’s also the sociopolitical aspect. The damage caused by the storm in the United States was severe. And with far fewer resources, countries in the Caribbean will have an even harder time recovering from Hurricane Sandy. Technology provides yet another lens through which to view the storm. It both played a crucial role in responding to the event and, at the same time, was made useless by it.
For example, many New York City hospitals faced the possibility of power outages and had to evacuate patients to other locations. At least one step of the harrowing journey was made smoother through technology. In this case, it was New York’s statewide use of electronic medical records, which gave doctors immediate and secure access to incoming patients’ medical charts and information. However, electronic medical records would not have been much help had the power outages been more widespread.
As it was, over seven million people in the United States lost power during Hurricane Sandy. What’s more, the outages had far-reaching secondary effects. Like at Verizon. With no power, the telecommunications provider wasn’t able to operate the safety system that keeps water out of its cables. As a result, customers lost access to Internet, TV, and the telephone. In the end, service technicians actually used thermal imaging to spot damaged cables and replace them at a fast pace. A fitting turn of events in which one technology was creatively used to repair the technology that failed in the first place.
In comparison, other companies were forced to use some very low-tech solutions to keep things running during the storm. When basement flooding took out the back-up generators at Peer 1 Hosting, a data center operator located in lower Manhattan, the company had only one hope of staying online: a single generator located on the building’s rooftop. More than 30 customers helped data center employees carry thousands of gallons of fuel, bucket by bucket, up 17 flights of stairs to the generator. Amazingly, their efforts were successful and the data center remained operational during the hurricane.
20,000 gallons of diesel fuel per day
The same outcome may have been possible with much less effort, however, if their generators and fuel tanks hadn’t been located below sea level. Since flooding is not uncommon on a low-lying island like Manhattan, the basement would seem to be the worst place to store resources that are critical for disaster response. But Peer 1 wasn’t alone in this. Internap, another data center operator in the same building, also lost its fuel pumps due to water damage. It was able to hook up its generator to a fuel truck parked outside the office, which provided the data center with up to 20,000 gallons of fuel per day for four days. Certainly an effective, but expensive, alternative to buckets.
A number of other data centers in Manhattan were forced to shut down for some amount of time during the storm, not because they didn’t have a working generator, but due to lack of fuel, or access to it. In fact, the top three reasons for generator failure, according to Gartner’s conversations with customers, all have to do with fuel. These are: failed sensors (companies thought they had fuel when they didn’t), stale fuel affecting performance and engine operation, and improper fuel storage resulting in contamination. Customers also said that fuel stored in a risky location (basements!) and inadequate supply prevented them from using generator power during an outage.
It’s surprising that so many data centers have a disaster recovery plan in place, but then neglect this crucial aspect. At least one company seems to have learned from Hurricane Sandy: Internap is considering the installation of submersible fuel pumps. For next year’s superstorm.