Common Sense Key in DR/BC Planning
Emergencies or outright disasters have a way of quickly forcing the most technology-centric and savvy into people struggling to find a way – very often a labor-intensive way – to get by.
It’s been almost a year since Superstorm Sandy hit the east coast, but two similar stories suggest how quickly people went from being masters of their (gadget) universes to struggling victims. The tales also suggest that a bit of common sense planning could avoid a tremendous amount of trouble once the emergency hits.
Major League Baseball’s media arm, MLB.com, operates from a skyscraper in midtown Manhattan. The company, according to ZDNet’s David Chernicoff, made the same disaster recovery mistake that many companies did: It placed the diesel generators that drive its servers and other equipment far too high:
For the duration of the outage it was necessary to hand carry thousands of gallons of fuel up six flights of stairs to feed the 600 Kw roof-mounted backup generator. But MLB.com kept the facility up and running throughout the 2012 disaster, keeping baseball streaming alive for their millions of users.
A similar story was told by hosting provider Peer 1, which is located a couple of miles south on Broad Street. The fuel tank was on the roof atop the 18th floor of its building. When the storm hit, only five hours of running time were left. What had to be done was clear, but by no means fun:
But Peer 1 and its datacenter tenants Fog Creek Software and Squarespace would refuse to give up, finally devising a bucket brigade to transport diesel fuel up 18 flights of stairs to the roof. For 72 hours.
ReadWrite links to a 16-minute documentary in which the folks who kept the generators running tell their story.
Ironically, the Diesel Technology Forum issued a press release last week extolling that fuel for use in emergencies. The release cites Superstorm Sandy and the Northeast Blackout of 2003, which left 50 million people in eight states and Canada without power. Diesel generators, the release says, were heavily relied on by hospitals, data centers and other critical facilities.
Both points of view are correct, of course: Diesel generators can be a huge aid when the electrical system goes down. This form of power is just as liable to the vicissitudes of events as any other form of electricity, especially if it is not deployed sensibly, as was the case with MLB.com and Peer 1.
Organizations can take many steps to increase the chances that the juice will keep flowing. In addition to no-brainers such as placing fuel tanks sensibly, generators can be connected directly to secondary power sources, such as natural gas lines. These can be disrupted as well, of course. But having secondary and even tertiary sources in place makes it more likely that disruptions will be averted.