Applying the R4 Framework of Resilience: Information Technology Disaster Risk Management at Northrop Grumman

Transcription

1 Southeast Decision Sciences Institute Annual Conference Applying the R4 Framework of Resilience: Information Technology Disaster Risk Management at Northrop Grumman CIS, IS, IT, MIS & Web-Based Applications Track Michael A. King, Business Information Technology Department The R. B. Pamplin College of Business, Virginia Polytechnic Institute and State University 1007 Pamplin Hall, Blacksburg, VA 24061, , Christopher W. Zobel, Business Information Technology Department The R. B. Pamplin College of Business, Virginia Polytechnic Institute and State University 1007 Pamplin Hall, Blacksburg, VA 24061, ,

2 Southeast Decision Sciences Institute Annual Conference Applying the R4 Framework of Resilience: Information Technology Disaster Risk Management at Northrop Grumman CIS, IS, IT, MIS & Web-Based Applications Track

3 Introduction Disaster risk management and mitigation are an important component of most business s tactical and strategic objectives. However, as time passes since the occurrence of disasters such as 9/11 and hurricane Katrina, these managers are finding it much more difficult to gain budgetary approval for projects that increase disaster resilience with respect to business systems and processes. In this paper we will discuss and extend a conceptual disaster resilience framework developed by the Multidisciplinary Center for Earthquake Engineering Research (MCEER), suggest methods for quantifying organizational resilience, and apply the extended framework to an actual business disaster experienced by the Northrop Grumman Corporation. Focusing primarily on information technology functionality and infrastructure, we will provide specific examples of internal and external activities conducted by local staff members, as well as by Central Information Technology services from the Northrop Grumman corporate office that are excellent examples of organizational resilience. The intent of our future research is to supply business managers with a quantifiable approach to developing organizational resilience measurements that will enhance their business case for business continuity and resilience projects. Preliminary Definitions It is important at the outset of any discussion of disaster risk management to assure the reader understands the relevant concepts within this domain, such as risk and risk mitigation. Risk is a measure of the potential effect of a disaster which takes into account both the likelihood that the event or events will actually occur and the negative impact that would result from such an occurrence. This impact can be measured by both tangible and intangible factors, ranging from the cost of reconstruction to the loss of quality of life. Risk mitigation, on the other hand, refers to the explicit activities by social processes or individuals that buffer and attempt to lessen such risk, before a disaster event actually occurs. In a recent article, Conceptualizing and Measuring Resilience: A Key to Disaster Loss Reduction, Kathleen Tierney and Michael Bruneau propose and illustrate a conceptual framework for analyzing what they call disaster resilience (Bruneau and Tierney, 2007). The authors make a clear distinction between disaster mitigation and disaster resilience by focusing their analysis on the post-disaster responses by organizations and individuals. The authors define resilience as the ability of social units to mitigate hazards, contain the effects of disasters when they occur, and carry out recovery activities in ways that minimize social disruption and mitigate the effects of future disasters. (Bruneau, Chang, et al., 2003) It is our opinion that the definition is general enough in nature to allow for adaptability, as well as include the actions of entities such as private individuals, formal organizations, and even national economies. A Case Study Northrop Grumman Corporation is a Fortune 100 company that is diversified across several high technology markets. Although Northrop Grumman s primary focus is defense technology design, manufacturing and service, the company operates in space, information, and electronic systems technologies, ship, mission, and integrated systems and technical services. The

4 company is listed on the New York stock exchange and had over $30 billion in revenues in The company is the world s largest ship builder and the third largest defense contractor. The specific business unit that we will be discussing is Sperry Marine which operates in Northrop Grumman s electronic sector. Northrop Grumman acquired Sperry Marine, located in Charlottesville, Virginia, in 2001 because of Sperry s attractive market position and expertise in defense and commercial navigation electronics. Sperry Marine has approximately 1300 employees supporting offices in eighteen U.S. and ten international port cities. In late September 2003, Sperry Marine faced the daunting task of preparing for the landfall of Hurricane Isabel, with a projected storm track passing directly over Charlottesville. For approximately ten days the hurricane traveled over the open Atlantic waters periodically changing strengths between a category three and category five hurricane. The National Hurricane Center issued a landfall forecast for the North Carolina Virginia border vicinity. Being well inland, the headquarters of Sperry Marine, in its nearly 60 years of existence, had never experienced a hurricane. With its main data center in Charlottesville, which housed the centralized BaaN ERP system, executive and information technology management took the predicted path as a certainty. Three days before the predicted landfall, executive and information technology managers, as well as system engineers formed a disaster management team and reviewed all disaster recovery documentation and software installation kits. System engineers performed tape backups of all systems, verified and cloned the tapes and personally drove the tape sets to the offsite secure facility in Richmond, Virginia. Engineers fueled and tested the diesel generator. As an aside, the business case presentation and subsequent purchase of the diesel generator had been very contentious. The disaster recovery team tested the satellite TV in the Command Com Center, as well as verifying emergency power availability from the diesel generator. Two days before the predicted landfall, the communication staff called all 28 Sperry offices and informed them that beginning at midnight UTC, the New Orleans office would act as the designated call routing center. At this time system engineers notified the SunGard cold site and activated the business continuity service. The call to SunGard, in effect, reserved data center space, servers, tape backup hardware and telecommunication services that would allow Sperry to build a minimal data center if necessary. On the predicted landfall day, executive management requested system engineers and operations staff volunteers to oversee the Charlottesville facility for the next seventy two hours and from the volunteer list, a three day disaster team rotation schedule was created and distributed. Other than the disaster team, all employees were required to leave the premises. Additional food and supplies were collected and stock piled in the data center. See Exhibit 1 for Hurricane Isabel tracking information. By six p.m. September 18, 2003, the full force of Hurricane Isabel had reached the city limits of Charlottesville, Virginia and a citywide power loss soon followed. The diesel generator immediately engaged and all emergency circuits at Sperry Marine became active. The information technology staff noticed that the overhead lights in the data center had gone out and that the emergency lights had taken their place, and that all servers, switches, routers, and core

5 applications were running as normal. The voice and frame relay circuits bounced several times before they became silent at approximately seven p.m., but cell communication coverage continued well into the night. There was one major problem that the information technology engineers and operations staff had not planned for: a complete loss the data center air conditioning system. The air conditioning system shut down at the same moment electric power was lost to the entire building. At first it was thought that the shutdown was a normal safety recycle and in all the haste to ensure key systems were running and functioning properly, the information technology staff failed to notice that the air conditioning system did not recycle. Within thirty minutes, the data center temperature had risen to ninety five degrees Fahrenheit. The core Ethernet switch was programmed to alarm and prepare for an automatic shutdown at ninety five degrees F and it shutdown as smoothly as Cisco advertised. All servers running critical applications were directly connected to the core switch which provided network connectivity at 1 Gb. speeds to the enterprise storage area network and other application processes. At this point, what was once a controlled business continuity incident had now become a disaster recover emergency. Unknown to the information technology staff, the maintenance and operation staff had never connected the data center air conditioning system to the emergency power circuit supported by the new diesel generator. This intentional oversight was the result of the previously stated contentious situation surrounding the purchase, installation and control of the diesel generator. The information technology department sponsored the business case for the diesel generator and subsequently won executive approval for purchase and ongoing management. To say the least, the maintenance and operations staff was extremely upset because facilities, power distribution and building infrastructure were usually under their domain. The information technology department had on numerous occasions asked the maintenance and operations management to sponsor a business case for the purchase of a diesel generator and each time the department reiterated their belief that a standby diesel generator was too expensive and not actually needed. The night of September 18 th proved that belief to be wrong and extremely irresponsible. The employees did pull together and located several large turbine type fans which they moved into the data center. Within fifteen minutes of starting the fans, the data center temperature had dropped to approximately ninety two degrees F. and remained that temperature for the next three days until normal power was restored. A fan was place beside the core Ethernet switch to keep it cool enough to operate. R4 Framework Michael Bruneau, the director of the Multidisciplinary Center of Earthquake Engineering Research (MCEER), has developed an organizational resilience framework that conceptualizes and defines disaster resilience as four distinct organizational characteristics or factors. These elements are: Robustness: the ability of an economic entity to resist or forestall damaging or catastrophic events.

6 Redundancy: an organizational unit s ability to provide alternative processes for inline or critical systems. Resourcefulness: a characteristic of an entity s tenacious response to and creative solutions for a disaster related instance. Rapidity: the ability to quickly restore systems or processes. The four resilience factors of the R4 framework can be grouped into two broad descriptive categories: those which are internal to the organization and those which are external to the organization, as seen in Exhibit 2. As shown by the diagram, there is a possible continuum between internal resilience factors, defined as intra-entity activities, and external resilience factors, which could be classified as socially-focused activities. An additional insight from the framework is that rapidity is, at least conceptually, a function of the remaining resilience factors. We discuss these hypothesized linkages at the end of the paper. As with most social processes, such linkages are probably non-linear and ambiguous at best. The following section will provide examples of the R4 Framework within the context of lessons learned by Northrop Grumman. Lessons Learned After power was restored to the facility, September 21 st, the air conditioning system in the datacenter recycled and began to circulate sixty eight degrees F. air. Several staff members for the communications department surveyed the damage to external communications lines, which included data and voice transmission, and were able to quickly repair damage. Within a few hours, Sperry Marine s frame relay network was back online. Without these skilled technicians, Sperry would have had to wait approximately a week or more for the local telecommunication company to repair the damage. This is a specific example of rapidity. The information technology staff made an assessment of the datacenter and found moderate damage to the core Ethernet switch. They also discovered the BaaN ERP system had several corrupted database logs. The information technology engineers quickly restored the database rollback logs and the ERP system was back online for all offices to utilize. Systems engineers placed a service call for Cisco onsite service and repairs to the switch were completed within one day, at a cost of nearly $15,000. The use of fans for improvised cooling and the use of cloned backup tapes to restore the database logs are examples of resourcefulness. Lesson 1: Sperry Marine should attempt to redesign the enterprise network architecture with no single points of failure. The Charlottesville headquarters is logically and physically the hub of the existing frame relay network as illustrated in Exhibit 3. Within the last two years, most large telecommunication companies such as Verizon and Sprint began offering multiple protocol label switching as a new business service. This new and extremely powerful switching protocol and an essentially disruptive technology, now allows companies to design and deploy a full meshed network; the holy grail of network architecture. The reality of a fully meshed network was too cost prohibitive to even the largest corporations, until the availability of MPLS. Now with a fully meshed network, Sperry has removed Charlottesville as a major point of failure. A fully meshed network is an exceptional example of redundancy.

7 Lesson 2: The information technology department should consolidate and virtualize servers and network hardware where possible. This technology goal was actually already under before Hurricane Isabel made landfall. The hurricane just reprioritized several information technology initiatives, such as the migration to blade servers and virtualization of several racks of Wintel servers where practical. The key benefit of reducing the number of production servers was reduced process and maintenance complexity and better uptime performance. The server uptime performance metrics are logged and available 24x7 on the corporate wide intranet. The continued redesign of information technology architecture with the objective of predictability and reliability is an example of robustness. Lesson 3: Information technology management should continue to insist and enforce standard process, server and network component configurations. Before a 2004 global network and server refresh, there were numerous software and hardware permutations. A decentralized Novell network had grown into an unmanageable state and was potentially a serious security risk. An unforeseen consequence of being acquired by Northrop Grumman was the requirement that Sperry Marine move to a Microsoft Active Directory network based on Windows Server Northrop Grumman gave Sperry Marine a capital budget line item that allowed Sperry Marine to purchase and install standard Hewlett Packard Proliant servers for all servers migrated to Microsoft Active Directory. This event allowed the information technology staff to develop a cookie cutter approach for server configuration and builds. An identical hardware configuration including two servers, a router, an Ethernet switch, a power supply, a tape backup system and complete user documentation was shipped and installed at each Sperry office during The identical network hardware configuration deployed in each office is another example of redundancy. Lesson 4: The information technology staff should recommend that the data center be gracefully shut down in the face of a potentially destructive storm. Executive and information technology management have not agreed on a rule or specific process that would dictate a datacenter shutdown. Executive management decided that future shutdown decisions would be made on a case by case basis. Lesson 5: Information technology staff should practice a disaster recovery drill at the SunGard recovery site on an annual basis. This recommendation was basically ignored due to cost concerns until Hurricane Katrina made landfall near New Orleans and flooded a key depot office in the area. Boiler plate disaster recovery documentation was updated, additional detail added and the first disaster recovery practice was held during the summer of Measuring Resilience A future goal of our research is to propose an intuitive and simple method of quantifying the four resilience concepts, while at the same time maintaining some degree of scientific rigor. Although there are several resilience performance measures such as direct regional economic resilience (Rose, 2004), the calculation and application of these measures can be tedious and impractical from a business management perspective. Bruneau, et al. (2007) discuss the resilience triangle, as seen in Exhibit 4, as a graphical device to illustrate system functionality as measured over time. Exhibit 4 is modeling an instantaneous loss of functionality with an incremental recovery time. We extend this concept by considering the situation illustrated in

8 Exhibit 5, which depicts both a gradual loss and a gradual recovery of functionality. Although one of the goals of effective disaster resilience is to minimize the area of the loss triangle, Bruneau, et al. (2007) do not offer any quantitative methods for objectively measuring the loss triangle. We seek to address this need for further quantifying the loss triangle by empirically demonstrating the calculation of the four factors of resilience using actual data from catastrophic events such as the hurricane experienced by Northrop Grumman Sperry Marine. From the examples cited in the case, we would argue that Northrop Grumman was resilient, but how can an objective measure of organizational resilience be formulated? One common measurement of resilience is the classic availability equation: where MTBF is mean time between failure usually provided by the equipment manufacture and MTTR is mean time to repair. This function focuses on the component level within a specific information technology architecture. Massiglia and Marcus present an information technology complexity framework based on Herbert Simon s classic 1962 paper, Architecture of Complexity (Massiglia and Marcus, 2002). They argue that an information technology system s complexity has a significant negative effect on system resilience. They define complexity is a function of the number of components, how the components are connected and how many different types of components are involved. While the Massiglia and Marcus complexity concept can help management understand the cost associated with supporting a resilient information technology architecture, it lacks a system wide view. What we will purpose are measures that are at an aggregate organizational level as illustrated in Exhibit 6. We will seek to develop a Function of Resilience using the empirical results of quantifying the four resilience factors discussed above. We have illustrated the preliminary conceptual model associated with this effort in Exhibit 7. Numerous disaster recovery vendors advocate their own specific resiliency model, such as SunGard s Availability Risk Management program, but the Function of Resilience will be vendor neutral. The research hypotheses to be considered include the following: H1: Higher levels of robustness will be positively associated with greater gains or increased levels of organizational rapidity. H2: Higher levels of redundancy will be positively associated with greater gains or increased levels of organizational rapidity. H3: Higher levels of resourcefulness with be positively associated with greater gains or increased levels of organizational rapidity.

9 H4: Higher levels of rapidity will be positively associated with greater gains or increased levels of overall organizational or entity resilience. We postulate that with empirically-derived and objective measures of resilience in hand, organizations will have a greater opportunity to make the business cases for disaster preparedness and mitigation projects, specifically related to information technology infrastructure and processes. References Bruneau, M., S. E. Chang, et al. (2003). "A Framework to Quantitatively Assess and Enhance the Seismic Resilience of Communities." Earthquake Spectra 19(4): Bruneau, M., Tierney, K. (2007). Conceptualizing and Measuring Resilience: A Key to Disaster Loss Reduction. TR News May-June: 14. Ignatiadis, I. and J. Nandhakumar (2007). "The impact of enterprise systems on organizational resilience." Journal of Information Technology 22(1): 36. Junglas, I. and B. Ives (2007). "Recovering IT in a Disaster: Lessons from Hurricane Katrina " MIS Quarterly Executive 6(1). Massiglia, P. and Marcus, E. (2002). The Resilient Enterprise: Recovering Information Services From Disaster. Mountain View, CA: Veritas Software Corporation. Marcus, E. and Stern, H. (2003). Blueprint for High Availability. Indianapolis, IN: Wiley Publishing. Perrow, Charles (1999). Normal Accidents: Living With High Risk Technologies. Princeton, NJ: Princeton University Press. Rose, A. (2004). "Defining and measuring economic resilience to disasters." Disaster Prevention and Management 13(4): 307. Simon, Herbert A. (1962), The Architecture of Complexity, Proceedings of the American Philosophical Society 106(December):

10 Exhibit 1: Hurricane Isabel Storm Track Organizational Resilience Rapidity Robustness Redundancy Resourcefulness Internal To Entity Exhibit 2: Resilience factors External To Entity Halifax Vancouver Singapore Shanghai Hong Kong Tokyo Vlaardingen Antwerpen Hamburg Immingham Bergen New Castle Oslo Charlottesville Sprint Frame Relay New Malden 1.54 CIR All other sites 512 CIR New Malden Esbjerg Southampton Copenhagen Peterhead Miami San Diego Mobile New Orleans Cranford Houston Long Beach Seattle Va. Beach Washington DC Exhibit 3: Sperry Marine Frame Relay Logical Model

11 Percent of Functionality Percent of Functionality Time Exhibit 4: Resilience triangle Time Exhibit 5: Resilience triangle with gradual loss A Resilience Continuum Component Level Data Center Level Business Unit Level Division Level Exhibit 6: An Organizational Resilience Continuum Organization Level Robustness H1 Redundancy H2 Rapidity H4 Entity Resilience H3 Resourcefulness Exhibit 7: Conceptual model