I spend a good deal of time evaluating client disaster readiness and/or working with clients to develop and implement effective disaster recovery and business continuity plans. My view, in the aftermath of Super Storm Sandy, is that there are lessons to be learned from firms whose preparations enabled them to weather the storm with little if any ill effect as well as from those who experienced unplanned outages and worse.
2012 was one of the most active and destructive Atlantic hurricane seasons on record. The most devastating storm of the season, Sandy, hit the northeast just about two months ago, resulting in more than 120 fatalities, leaving millions without power (5 million in NY/NJ alone), and causing more than $60 billion in damage in the US.
Of the businesses that had disaster recovery plans in place (those that didn’t are a topic for another day), there were some whose plans kicked in and worked seamlessly and others whose plans didn’t. What were the differences between the two and what lessons can we learn from those differences.
Testing – How much is required
Two businesses (firm names withheld) with their primary data centers in NY City and both highly dependent on their online presence had significantly different experiences. Both firms found it necessary to exercise their disaster recovery plans. Firm A, conducted comprehensive disaster recovery tests 1-2 times per year and was able to switch processing to their disaster recovery site in advance of the storm coming ashore. The result was that operations continued seamlessly (i.e. their clients had no idea). Firm B, conducted periodic tests as well but those tests were not comprehensive. They expected to lose power during the storm and planned on their generators carrying the load until power was restored or they had to switch processing to their DR site. Unfortunately, the switch over to generator power failed. The generators were operational, fuel was available but the automatic transfer switch failed; it had not been exercised in any of their recent tests.
Lesson Learned: A comprehensive test of the entire disaster recovery plan should be conducted at least once a year.
Location, Location, Location
The decision about where to locate your company’s DR facility is critical. Much like the situation above, two other businesses on the eastern seaboard came through the storm with different experiences. One was able to transfer processing to a DR site in the mid-west and provide continuous availability of core applications. The second firm had a primary data center in NY City and a DR facility in Newark NJ. Unfortunately, both facilities experienced a loss of power.
For another perspective on location, consider the placement of critical recovery infrastructure (generators, fuel tanks, et al.). The relocation of patients from two NY City hospitals has been widely reported. In both cases, the primary issue was flooding in the basement where the generator fuel tanks were located.
Lesson Learned: Locate your disaster recovery site in a region that is unlikely to be impacted by an event that impacts your primary site. Also consider the risk to disaster recovery infrastructure when deciding where to locate them within or external to your facility.
Disaster Recovery is not Business Continuity
Restoring access to critical business systems is meaningless if no accommodations have been made for the staff that need access to those systems. One NY City based business lost access to their facility and had not provided an alternate location for critical staff nor the ability for staff to work remotely. The end result was disrupted and fairly chaotic operations as alternate arrangements were made. Another business experienced a comparable loss of facility but had made arrangements for critical staff (e.g. customer support) to work remotely and service from their homes.
Lesson Learned: Planning for staff and related work arrangements is as critical as planning for the restoration of access to IT infrastructure and application systems.
Statistics continually show that organizations in many areas (SMBs, Higher Ed, Health Care to name a few) are not rigorous with regard to disaster recovery and business continuity planning. As I look forward into the New Year, my hope is that organizations will be more disciplined in their preparations for potential disasters.