Incorporating a Risk Adjusted Recovery Time Model into the Data Center Strategy by Employing Virtualization and Cloud Services

Transcription

1 Incorporating a Risk Adjusted Recovery Time Model into the Data Center Strategy by Employing Virtualization and Cloud Services Brian Henry bhenry@bellarmine.edu Michael Mattei mmattei@bellarmine.edu Eric Satterly esatterly@bellarmine.edu

2 Why Bellarmine is so sensitive to outages? Lost for 4 days (the IT Director shortly thereafter) Ice storm no power for five days, sent students home Wind storm no power for two days Internet down five hours even with two paths off campus Lost core switch during finals two hours to restore Lightening fried entire telephone switch Student cigarette burned through fiber to large campus bldg. Water line break on floor above warm data center Lost ERP for two days

3 Foundational Definitions: Hot site an active duplication of the organization s live systems Warm site a site with systems and communications ready to use, just need to restore data Cold site a place to store data, has adequate space and infrastructure Mirrored site an exact duplication of an organization s production site Mobile site a self-contained, transportable office custom fitted with IT and communications equipment from Government Finance Review, August 2012, pg. 53

4 Few More Definitions Consequence a result or effect of an action or condition Event a thing that happens, especially one of importance Risk the possibility that something unwelcome will happen Tail risk event probability of occurrence is less than 0.03%, virtually nil (beyond three standard deviations from the mean) RTO recovery time objective RPO recovery point objective User Tolerance expected level of understanding by users to loss of services

5 Do you focus your effort on what got you here or on how you get back?

6 We Learned, Focus on Consequences not Events Brad Brekke, vice president of Assets Protection for Target Corporation, shared some of his organization s methods for preparing for the unpredictable. You can t plan for everything. Instead, we plan for consequences. What happens if you lose communication, transportation, energy? Planning for consequences, as Brekke puts it, is one way of broadening the organization s ability to respond to unlikely events.

7 Step 1: List Your 10 Most Critical Services/Apps Paretto 20% services fulfill 80% user needs Create list from user perspective Which ones generate most complaints if service/app not available to user Assume network is one of ten Is the Internet one of the ten? Don t include low level infrastructure hardware/software

8 Get a Box and Fill It A disaster will not follow a script you will have to adapt on the fly You will only have access to what is in your DR toolbox Spend your time filling your DRbox with useful tools and rummage through it often Think of DR Planning as an Agile Process Before Step 2, two powerful items for the DRbox B

9 What Exactly is The Cloud Definitions of cloud vary Generally a cloud service is a service that makes use of hardware and software in a manner that is delivered across a network Typically involves the Internet Usually involves: Virtualization Geographic Clustering Load Balancing Multi-Tenancy

10 What Types of Clouds Are There? 1. Public Amazon, Microsoft, Google, etc. 2. Private In house solutions for infrastructure, applications, and storage 3. Community A public cloud that is utilized by multiple organizations with similar requirements Usually used by government organizations and universities 4. Hybrid A system that incorporates both Public and Private cloud elements

11 Cloudy Resources Infrastructure Public: Amazon & Microsoft Private : VMWare vcloud & Microsoft Hyper V Storage Public: DropBox, Box.Net Private: Many commercial SAN providers Applications and Services Public: Lots! Private: Load balanced and replicated web services, VMware Thin App, Microsoft App-V

12 Public Cloud Overview Variable Costs Management Anytime, Anywhere Internet Dependent Hidden traps

13 Costly Services Pricing structures for cloud services vary Hosting services are often based on usage Applications are often priced per user

14 Public Cloud Case Study: Practice Fusion Affordable Care Act has provision to incentivize health care providers to move to electronic records School nurse at a school finds a service that will help do it for them called Practice Fusion for FREE Nurse requests uploading ALL student data to the cloud server from our student record system

15 Public Cloud Case Study: Practice Fusion IT Staff examine the terms and conditions and find lines such as: When you submit Content on our Services, you grant us and those we work with a worldwide, royalty-free right to store, host, reproduce, create derivative works of (such as translations, adaptations, reformatted versions and anonymized or de-identified versions), publish, publicly perform, display, use and distribute such Content

16 Security and Stability in the Public Space Security How many full time security staff do YOU have How many people have access to your data? Shared access

17 Notable Cloud Outages Some Microsoft Outage Dates: 9/9/2011, DNS Misconfiguration 2/29/2012, Leap Year Bug 7/26/2012, Cause unknown 2/11/2013, Procedural Error 2/22/2013, Expired SSL Certificate Some Amazon Outage Dates: 10/5/2009, Denial of Service Attack 4/21/2001, Cause unknown 8/10/2011, Storms knock out Dublin data center 6/30/2012, Storms knock out Virginia data center 7/3/2012, Storms and generator bugs at Virginia DC 10/22/2012, Storms take out Virginia DC 12/26/2012, Cause Unknown

18 Private Clouds Storage Infrastructure Clustered and Balanced Outage Preparation Costs

19 Sorting your Storage Network Separate Traffic Consider 10 GB Ethernet or Fiber Channel Replication Speed

20 Fix Your Single Points of Failure The most critical piece of a cloud deployment is to fix the network Anything that a Private cloud service should be designed in a N+1 setup Buy Blades or Boxes?

21 Cluster, Balance, Webalize, and Virtualize Clustered and load balanced services are by design failure resistant Web-delivered applications are easiest to cluster and load balance Infrastructure Virtualization is often leveraged as part of this to speed up deployments and disaster recovery

22 Costs of a Private Cloud Hardware Hypervisor Licensing Operating system licensing Application licensing Database licensing Manpower

23 Condensation on the Cloud The definition of the Cloud is nebulous The cloud is not 100% guaranteed It can be significantly more costly to run on a Public Cloud than you think Cloud providers often have more resources than you do You may have your own private cloud E

24 Virtualization What is it? Decoupling a system from its native environment and transplanting it in a new ecosystem Why do it? Reduce physical devices and increase utilization on the remaining devices More disaster recovery options Lower system administration workload There are different kinds of Virtualization Network Storage Server

25 Backup Strategy Let s Dive Deeper on Server Virtualization Relies heavily on a backup application and focuses on primarily on individual applications. E

26 Backup Strategy Not much improves with this type of virtualization; still heavily application dependent.

27 Backup Strategy Significant improvement in that Virtual Machines now exist and can be backed up as a single file. In Production backups remain challenging.

28

29 Hypervisor Virtualization The real power of Virtualization lies in the Hypervisor, which brings the true benefits of Virtualization to reality Hypervisor control over the hardware means more optimization possibilities Virtual Machines can be moved between host machines while in production Backups can now be accomplished with a Virtual Machine running in full production without users noticing

30 Armed with technologies (tools) like Cloud and Virtualization, it was time for Bellarmine to develop a Strategy (select best combination of tools from the DRbox)

31 Our Initial Strategy... Two data centers on campus Locate in separate buildings Virtualize all physical servers and employ multiple hosts Utilize synchronous SAN technology Mirror data centers so that one takes over if the other is lost

32 Had Some Holes If there is no true disaster but DCs lose communication with each other which one is primary? Do we need a third DC for quorum? Which layer in the stack is in charge? VM, SAN, Application? What if errant process deletes a virtual machine or application data base? In an Active-Active design, how are you certain where the freshest data are so that an accurate RPO can be understood? What if lose both data centers? There has to be a better (and cheaper) way

33 Step 2: Think Consequences, not Events & Risk 1. Campus is operating normally, everything appears normal, but top 10 services are unavailable to the users 2. Campus is operating normally, a serious event on campus in a single building that everyone can see, but it destroys the primary data center 3. Significant event damages many buildings on campus. Some offices open and about 50% of classes are being held. Primary data center and warm site are in ruins 4. Major disaster hits three square miles around campus, results in closing entire campus and sending students home for two weeks. Campus IT facilities survived but not running and no one can get to them. IT services needed to restore campus

34 Step 3: Consider Tolerance During Design Tolerance generally comes from the more spectacular event The more widespread the impact will mean more tolerance More Tolerance means more Time Recover Level of Tolerance Low Medium High Tolerance Adjusted Recovery Time for Top 10 Services Minutes Hours Days

35 Step 4: Create Consequence for Each Tier Quantify % Users Impacted Quantify Campus Operating Conditions / Level of User Tolerance Set a RTO for each cell % of Users Impacted Campus Operating Conditions (Level of User Tolerance) Normal (Low) Minor Disruption (Medium) Major Disruption (High) >70% < 1 minute 1-4 Hours 1-2 Days 30-70% 1-30 minutes 4-8 Hours 2-3 Days <30% minutes 8-16 Hours 3-4 Days

36 Step 5: Detail Strategy and Evaluate Consider how to best recover given the magnitude of the disaster, the number of people impacted, and the resulting tolerance those factors yield

37 Scenario Incorporating a Risk Adjusted Recovery Time Model into the Data Center Strategy Bellarmine Strategy for Low Tolerance This type of disaster will not be obvious to the community except in the loss of IT services. Probably an equipment failure, misconfiguration, or logic error. Users will have little patience given a lack of visual evidence that there is a problem. Strategy Manually initiated recovery to the Hot Site which is a passive data center with a replicated SAN. SAN technology allows for near realtime replication resulting is a very recent RPO.

38 Bellarmine Strategy for Low Tolerance

39 Scenario Incorporating a Risk Adjusted Recovery Time Model into the Data Center Strategy Bellarmine Strategy for Medium Tolerance This type of disaster will be obvious to many in the community, e.g. there will be emergency personnel, potential evacuations, etc. Probably a fire, flood, or threat; likely not technically initiated. Users will have some patience as they will likely be focused on gathering information about the event over completing their work. Strategy Maintain a Cold Site that will provide a recovery location. Assume that some parts of one of the campus data centers can be gathered and taken to the Cold Site.

40 Scenario Incorporating a Risk Adjusted Recovery Time Model into the Data Center Strategy Bellarmine Strategy for High Tolerance This type of disaster will be blatantly obvious to the community. Probably a wide scale Act of God type disaster large amounts of the school are rendered inoperable. All operations are impaired. Users will have a great deal of patience and will mainly be focused on wanting information. Strategy Cold site recovery strategy is the preferred approach, but if that is not possible rebuilding infrastructure in the Cloud would be the approach.

41 In Conclusion There is no Right answer Rapid changes in technology make for many options in the recovery space When developing your strategy, consider user perception and tolerance so that you can set a reasonable direction Build a framework that allows for the ability to deal with change Walk before you run, but start walking

42 Tips on DRbox Management Make sure you have access to your DRbox in a disaster The little things matter, if you are not sure about it add it to the box Have an Incident Management Program that provisions for the capture of incident details give yourself credit for even small successes and learn from failures For example A restoration of a given file for a user can serve to prove that backup and recovery are both functioning

43