Keys To Implementing A Successful Internal Recovery Strategy. Karl Weinberg, CBCP Rick Falvo, CBCP Chicago Mercantile Exchange (CME)

Transcription

1 Keys To Implementing A Successful Internal Recovery Strategy Karl Weinberg, CBCP Rick Falvo, CBCP Chicago Mercantile Exchange (CME)

2 Agenda CME a brief history Risk Assessment Approach to achieving the goal Analysis / RTO Defined Alternative Recovery Strategies Selling an Internal Recovery solution Recommendation Implementation The Project 2

3 CME A brief history Founded in 1898 and evolved into CME in 1919 Largest futures exchange in US Electronic trading platform (GLOBEX ) begins in 1992 First Exchange to be publicly traded in 2002 In 2003 we traded 640 million contracts with a notional value of trillion dollars 3

4 Risk and Mitigation Strategy Operational Risks: Event interrupting the operations of a data center component (hardware, software, telecom, etc) Mitigated by implementing component redundancies Facility Risks: Event interrupting the operations of an entire data center Mitigated by implementing a redundant data center Environmental Risks: Event interrupting the operations of the entire vicinity Mitigated by implementing a remote redundant data center 4

5 Some Causes of Computing Disruption Computer Hardware Malfunctions Power Surges/Outages Earthquakes Floods Fires Explosions Riots 5

6 Computer Disruptions Popular belief is that the ability to tolerate a computing disruption depends on reliable hardware. However, studies indicate that only 22% of disasters were related to a computer hardware failure. The vast majority of disruptions are caused by the other factors that were mentioned previously. This indicated that CME needed to be able to tolerate disruptions emanating from outside the four walls of the exchange. 6

7 Approach to Achieving the Goal Identified the Alternative Strategies for Data Center Disaster Recovery Defined business drivers impacting recovery needs Analyzed the business risks Identified alternative recovery strategies Researched what other exchanges are doing as it relates to back-up data centers Developed recommended strategies Performed Cost analysis/comparison of strategies Developed a set of workplans to implement the recommended Disaster Recovery strategy Performed cost analysis of strategy Developed recommendation based on cost 7

8 Establishing CME s RTO Recovery Time Objectives Trading available the next trade day Clearing in not more than 4 hours Price Quotes in not more than 15 minutes Already operating two redundant data centers, to provide rapid recoverability for critical applications. The proximity of the Exchanges two data centers created the risk of an outage if both were affected simultaneously. 8

9 Alternative Recovery Strategies Outsource with a Third Party D/R Vendor Enhance our Current Facilities Share a Remote Facility Lease a Remote Facility Retain Current Facilities and Obtain a Remote Facility 9

10 Recovery Strategy Results Outsourcing with a Third Party D/R Vendor Unable to achieve outage duration acceptable to CME Competition for resources High cost to have exclusive resources Lower capital investment for real estate/build out Enhance our Current Facilities Limited flexibility and longer planning time for space expansion Expansion cost higher due to downtown space costs Doesn t mitigate our Environmental Risks 10

11 Recovery Strategy Results Sharing a Remote Facility Limited flexibility and longer planning time for space expansion Expansion cost higher due to requirements of other tenant May compromise CME competitive strategies and alliance plans Some cost saving from sharing resources Leasing a Remote Facility Allows higher level of security and maintains integrity of strategic plans Site already in a hardened state Lower capital investment for real estate/build out Less control of space required for expansion 11

12 Recovery Strategy Results Retain our Current Facilities and obtain a Remote Facility More control of space required for expansion Most effective solution for meeting the defined RTO s Allows higher level of security and maintains integrity of strategic plans Mitigates all three risk factors Higher capital investment for real estate/build out 12

13 Recovery Strategies Are Driven by Acceptable Level of Impact Impact Restore Relocate Rebuild Vendor Provided Facilities CME Facilities Parallel Duplicate Warm Hot Cold Shell As As shown shown in in this this diagram, the the extent extent of of a a disaster, disaster, measured in in both both impact impact (usually (usually stated stated in in financial terms) terms) and and the the length length of of the the outage, outage, determine the the necessary recovery strategy. Outage Duration Minutes Hours Days Weeks Months 13

14 Selling Internal Recovery Initial presentation to Board of Directors in August 1999 Numerous presentations delivered to Management Team and BOD over the next two years Board of Directors accepted our INITIAL presentation in August 2001 after we refined our numbers 14

15 Why Internal Recovery? CME Requires Specialized equipment Hot-site providers not equipped Large server farm / Large Tandem processors Special communications requirements Recovery Time Objective can only be met with an Internal Recovery solution An Internal Recovery strategy is the only strategy that mitigates all three risk factors. 15

16 Recommendation CME should implement an Internal Recovery strategy that utilizes its current facilities and a remote facility. Utilize an active two data center recovery strategy One data center at CME corporate headquarters Another in a remote location Data center A backs up data center B and vice versa Use high speed fiber link to mirror data between the two data centers (EMC Symmetrix) 16

17 Recommendation CME Corporate Center Back up Open Outcry (Trading floor) applications in the data centers in the North and South towers. Remote Site Facility (Remote Data Center or RDC) Supports Non-Open Outcry applications GLOBEX Electronic Trading System Backup system for Clearing Market Data Network (Ticker) Allow up to four hours to reconstitute electronic trading, ticker and clearing No immediate fail-over 17

18 Recommendation Contains a fully equipped backup environment for the GLOBEX Control Center. Provides a backup facility with adequate equipment to support Clearing Staff Clearing IT Staff All other business units needed to support Electronic Trading and Clearing. Supports an Emergency Operations Center (EOC) 18

19 Recommendation Remote Site: Key Decision Criteria Distance from Primary Data Center minimum possibility of a disaster event impacting both sites different infrastructure providers for each site (eg. Power and Telecoms) Manageable Telecommunications costs requires connectivity to CME WAN requires dedicated fiber link to Wacker Drive 19

20 Recommendation Remote Site: Key Decision Criteria Adequate Space for Growth for all equipment office space for support staff and others expandability minimize construction costs Ease of Access for CME personnel for telecommunications infrastructure 20

21 Recommendation Of the 20 sites evaluated the site selected is recommended because: Mitigates all three risk categories Communications infrastructure exists thus minimizing these costs Lowest rental cost per square foot Provides maximum control/security of the facility Allows for effective use of space at both the remote site and the current facility Expandable/ Minimal incremental expense 21

22 Implementation Timing Using a phased in approach, the Exchange can begin receiving benefits from the remote site in approximately 150 working days by utilizing this site for primary operation of Electronic Trading. Continue with a phased in approach for the backup systems for Clearing, Price Reporting, etc. 22

23 Benefits of a Phased Approach Opportunity to rapidly implement redundancies of critical systems. Allows the Exchange to market a complete fault tolerance computing environment. By expediting this acquisition, we can leverage the current need for system upgrades by installing the new hardware at the remote site instead of the Wacker Drive location. A phased implementation of hardware and applications allows the opportunity to leverage experience to more efficiently implement subsequent phases. 23

24 The Project Participants Senior Management Business Continuity Technology Operations Telecommunications Physical Infrastructure Project Support Office Clearing House 24

25 The Project Implementation Sub-projects Acquire the remote site Build-Out the remote site Equip the remote site Develop fail-over plans Test/Implement the remote data center 25

26 The Project Scope Document developed Eliminated scope creep Second data center versus dedicated DR site Project Plan developed Sub plans from each discipline requested Specific time table established 26

27 Design Building The Project Site Preparation 50,000 sq. ft two story building Data Center Demolition Build out 1 st floor data center 2 nd floor business recovery space Backup power 27

28 Exciting start Telecommunications DWDM SONET Dedicated circuits PBX Equipment installation / testing Phased by application priority Business Continuity space Operations area Emergency Operations Center Technical Support area Business Unit space (hot stand by) 28

29 Winding Down Data Center ready GLOBEX production in 7 months Backup Clearing System in 17 months Business Resumption ready Utilized PR4 methodology Able to resume business at a remote site in a predominately all electronic trading environment 29

30 Continuing Obstacles Change control Applications, hardware, software possibly out of synch Testing Methodology Hardware level Software Application 30

31 Wrap Up Thanks for your time and attention! Questions comments Karl Weinberg Rick Falvo 31