Disaster Recovery & Business Continuity Dell IT Executive Learning Series

Transcription

1 Disaster Recovery & Business Continuity Dell IT Executive Learning Series Presented by Rich Armour, Debi Higdon & Mitchell McGovern THIS PRESENTATION SUMMARY IS FOR INFORMATIONAL PURPOSES ONLY AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. STATEMENTS ARE THE OPINIONS OF THE PRESENTER(S) AND DO NOT NECESSARILY REFLECT THE VIEWS OF DELL. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND Dell Inc. All rights reserved. Trademarks used in this text: Dell and the DELL logo are trademarks of Dell Inc. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell Inc. disclaims any proprietary interest in trademarks and trade names other than its own. 1

2 Disaster Recovery & Business Continuity Hardened Data Centers? What won t work? Keeping it Evergreen Do I need Continuous Operations? What s the Cost? Where is My Risk? What about that extra capacity? Invisible Failover? Is everyone in the Loop? What s most important? Leadership Build it in, Don t bolt it on Will My Systems Failover? 3of 23 Integrated Business Continuity Program Dynamic Call Routing Recoverable Applications Redundant Networking Temporary Staff Business Continuity Outsourcers & Vendors Backup Data Multiple Data Centers Contingency Facilities = Components of I/T Disaster Recovery 4of 23 2

3 Integrated Business Continuity Program Business Recoverable Continuity Planning Dynamic Applications Redundant Build Call Detailed Recovery Plans for Key Networking Routing Business Facilities & Capabilities Call Centers Factories & Distribution Centers E-Commerce Sites Temporary Service Centers Business Outsourcers Staff Consider both Continuity routine & major disruptions & Vendors Balance risk & cost of disruptions against cost of recovery capabilities Communicate Plans to all Players Backup Contingency Build Business & Multiple Data I/T Disaster Recovery capabilities Facilities Data Run periodic exercises Centers = Components of I/T Disaster Recovery 5of 23 Link to Cost & Impact of Potential Disaster Recovery Time Aug Sep Oct Nov Dec Jan Feb Recovery Time vs. Exposure Mar April May June July Aug Without DR Plan DR Implemented Oct Nov Dec Sep Financial Exposure (Lost Profit or other metric) $X Million < $X Million Financial exposure reduced 6of 23 3

4 Link to Cost & Impact of Potential Disaster Recovery Time Aug Sep Oct Nov Dec Jan Feb Recovery Time vs. Exposure Highlight Financial Benefits Without DR Plan Estimate financial exposure over the recovery time for loss of each key facility Determine best effort recovery time given level of DR capability Assess impact of program milestones Additional Data Center is operational Hardware is ordered Hardware is installed Applications are distributed & Data is replicated DR implementation starts delivering value Mar April May June July Aug long before all capabilities are fully installed & tested DR Implemented Oct Nov Dec Sep Financial Exposure (Lost Profit or other metric) $X Million < $X Million Financial exposure reduced 7of 23 Where s Your Risk? 8of 23 4

5 Where s Your Risk? Top 10 Risks 1. Fire 2. Thunderstorms 3. Utility Interruptions 4. Telecommunications Interruptions 5. Flood or Surface Water 6. Virus 7. Heavy Rain or Snow 8. Extreme Cold or Freezing Precip 9. Water or Liquid Spill or Release 10.Denial of Service Attack 9of 23 Where s Your Risk? Top 10 Risks 1. Fire 2. Thunderstorms 3. Utility Interruptions 4. Telecommunications Interruptions 5. Flood or Surface Water 6. Virus 7. Heavy Rain or Snow 8. Extreme Cold or Freezing Precip 9. Water or Liquid Spill or Release 10.Denial of Service Attack Leverage Your Insurance Carrier Expert at risk analysis Maintains extensive loss databases Motivated to reduce your risk Can drive premium reductions 10 of 23 5

6 Where s Your Risk? Other Risk Considerations Tornado Earthquake Hurricane/Typhoon Terrorist Attack Human Error Tsunami Etc.... Each Business & Location will have a unique Risk Profile Top 10 Risks 1. Fire 2. Thunderstorms 3. Utility Interruptions 4. Telecommunications Interruptions 5. Flood or Surface Water 6. Virus 7. Heavy Rain or Snow 8. Extreme Cold or Freezing Precip 9. Water or Liquid Spill or Release 10.Denial of Service Attack Leverage Your Insurance Carrier Expert at risk analysis Maintains extensive loss databases Motivated to reduce your risk Can drive premium reductions 11 of 23 Determine Criticality of I/T Resources DR Class DR Class Description RTO* II Business Critical II II Business Essential III Business Support Any outage immediately impacts the business An extended outage (i.e., > 48 hours) impacts the business An extended outage does not impact the business * Recovery Time Objective What impacts the business? Degrades business operations Impacts customer experience Results in financial loss Erodes brand value Creates unacceptable liabilities 1-4 Hrs 48 Hrs Best Effort Business Impact Analysis 1. Define Business Impact in terms relative to your business 2. Relate I/T Resources to Impact of their loss or degradation 3. Assess Applications, Data & Infrastructure 4. Ensure Single Points of Failure are identified & mitigated 5. Include components which support critical resources 6. Link to financial exposure 7. Involve & validate with business owners 8. Make DR Class part of the culture 9. Periodically update 12 of 23 6

7 Four Levels of Continuity Data Application Platform Site Site Recovery Remote or commercial recovery facilities Application Failover/ Load Balancing Continuous application access via clustering Redundant Systems Continuous server, storage, network access SAN, NAS & DAS Continuous data access Backup and Restore Tape backup, off-site storage Site/Datacenter Failover Re-route users and data to replicated sites Rapid Equipment Replacement Vendor services and financing programs High Availability System Features Hot-swappable, redundant components with mission-critical support 13 of 23 Lay the Foundation: Tier and Protect Your Data 14 of 23 7

8 Single Point of Failure (SPOF) Analysis Conduct analysis of of Single Points of of Failure (SPOF) in in Class I I (Business( Critical) Systems to to ensure Business Continuity Single Point of Failure (SPOF) A point in in a system where if if a failure occurs, there is is no redundancy to to compensate for it it Systems can have more than one single point of of failure One SPOF in in a critical component can overshadow well engineered redundancies & resilience in in the rest of of the system Removal of of all SPOFs usually not cost effective Leverage Failure Mode & Effect Analysis (FMEA) & other tools I/T I/T Resources to to be be Analyzed Client Systems Application Software Middleware Up Stream Systems Data Shared Services Network Facility Network Long-Haul Network Campus Servers Storage Environmental Power Data Center Facility 15 of 23 Build the Team... Manage the Complexity 16 of 23 8

9 Build the Team... Manage the Complexity Application teams Data Management I/T Operations & Engineering Networking & Telecom Facilities Business Customers Business Continuity Program Our Program Approach 20,000+ Tasks 2 years Dedicated PMO driving extended virtual teams Phased Program Earned Value & Scorecarding 17 of 23 Event Ice Storm Area Power Outage Don t Assume Infrastructure Will Work When Disaster Strikes Potential Impact Land-based Phones Down Highways impassable Cell & land phones down Planning Considerations Assure wireless connectivity Provide remote access capabilities Designate sites which will have power & can support e & IM, provide radios to key personnel Area Evacuation Security event Internet Worm Hazmat Event Hurricanes Building unusable no HVAC Highways jammed Network down Internet down Data Centers inaccessible All of the above total threat Data Center damage Relocation of critical users Provide remote access capabilities Ensure DR plans available off-line Maintain local copies of critical files Ensure robust remote access & system management as well as DC management capabilities Tabletop dry run of DR plans 18 of 23 9

10 Consolidate, Lock Down & Bullet Proof Your Data Centers UPS & Generator Backup Diverse Power Feeds Fire Protection Physical Security & Access Control Alarms & Sensors Dual Path Network Routing Dual Path Power Routing N+1 Redundancy Environmental Controls Expandable Manual EPO Facility Hardening Data Center Considerations Plan for the future Identify & Mitigate Single Points of Failure Use Threat Assessment & Business Impact Analysis to drive design & capabilities Balance trade-offs between DC hardening & failover capabilities 19 of 23 Leverage Always On, Active-Active, 75/75 Architecture Active/Active Workload Balanced Across Multiple Data Centers Fully Redundant Fiber Network Real-Time Data Replication Redundant, load balanced infrastructure runs as fully distributed production environment Each Data Center runs 75% of peak capacity/100% of normal Redundant Fiber Network for scalability & reliability Failover is automatic & instantaneous: : loss of capacity, not data or function Floor plan allows immediate expansion for extended recovery 20 of 23 10

11 Database Recoverability Still a Challenge Model Recovery Strategy Features/Tools Hot/ Standby App(1) Hot App(2) Standby Low Application Design Impact Recovery time: 1-4 Hours Failback Complex & Time Consuming Supported by EMC SRDF, Oracle Data Guard & other replication technologies Partitioned Active/ Active App(1) Active App(2) Active High Application Design Impact Recovery time: Instantaneous Failback Less Complex & Time Consuming Requires application logic to to support data partitioning by instance Database Active/ Active App(1) Active App(2) Active Low Application Design Impact Recovery time: Instantaneous Failback Much Less Complex & Time Consuming Not well supported by existing database products & technologies 21 of 23 Disaster Recovery Testing Scope Business Critical Systems Key systems of critical vendors Paper review on some Objectives Validate recovery procedures Demonstrate recoverability SLA Identify recoverability problems Test end-to-end failover integration Strategy Cross-Functional Planning Limited Business Disruption Not a one-time event Metrics Recovery time Procedural Errors Data Integrity Functional Availability Make Testing a Regular Cycle Test some Apps each Quarterly Test all Business Critical App Annually Q1 Q2 Each Test requires planning, staging, execution & analysis Q3 Q4 22 of 23 11

12 Disaster Recovery Testing Scope Business Critical Systems Key systems of critical vendors Paper review on some Objectives Validate recovery procedures Demonstrate recoverability SLA Identify recoverability problems Test end-to-end failover integration Strategy Cross-Functional Planning Limited Business Disruption Not a one-time event Metrics Recovery time Procedural Errors Data Integrity Functional Availability Make Testing a Regular Cycle Test some Apps each Quarterly Test all Business Critical App Annually Q1 Q2 Each Test requires planning, staging, execution & analysis Q3 If it hasn t been tested, it doesn t work Q4 23 of 23 How Mature is Your DR Capability? HIGH RISK DR PROGRAM Business continuity risks have not been quantified Mission critical Apps are not formally identified Apps are rarely tested for failover DR capability is is an after thought, not part of the development process Business Continuity Plans are independent of any I/T DR programs Data Centers are not supported by adequate power & environmental redundant systems Mission Critical Applications are hosted in in a variety of locations in in & out of Data Centers Applications must be manually failed over to backup systems or sites Where is your Disaster Recovery Program? Do you have one? MATURE DR PROGRAM Business continuity risks are quantified and validated by Audit Mission critical Apps are flagged for special handling Failover Tests are conducted regularly DR is is part of every new solution Business Continuity Plans are fully supported by I/T contingency plans & capabilities Multiple Hardened Data Centers linked with redundant networking Mission Critical Applications are hosted in in I/T Data Centers Applications are architected with transparent always on Active- Active DR Key Vendors Business Continuity Programs are regularly reviewed Key Vendors involved in in DR tests 24 of 23 12

13 Key Takeaways... Getting It Done... Start with Leadership Integrate with the Business Assess the Risks Identify the Critical Resources Balance Cost & Benefit Involve the Entire Team Harden your Data Centers Go Active-Active Assume Nothing Will Work Do a Full Failover 25 of 23 Key Takeaways... Getting It Done Start Keeping with Leadership It Evergreen Integrate with the Business Built DR Assess into the every Risks Solution Don t t Identify Break the into Critical the Piggy Resources Bank Balance Cost & Benefit Test it... Regularly! Involve the Entire Team Harden your Data Centers Make Go DR Active-Active part of your Culture Every Assume Project, Nothing Every Will App, Work Everyday Do a Full Failover 26 of 23 13

14 Questions & Answers 14